Venue: NAACL 2025

# Direct Preference Optimization of Video Large Multimodal Models from Language Model Reward

Ruohong Zhang\({}^{\spadesuit}\)&Liangke Gui\({}^{*}\)\({}^{\diamondsuit}\)

Zhiqing Sun\({}^{\clubsuit}\) , Yihao Feng\({}^{\triangledown}\) , Keyang Xu\({}^{\bigtriangleup}\) , Yuanhan Zhang\({}^{\heartsuit}\) , Di Fu\({}^{\Diamond}\) ,

Chunyuan Li\({}^{\lozenge}\) , Alexander Hauptmann\({}^{\spadesuit}\) , Yonatan Bisk\({}^{\spadesuit}\) , Yiming Yang\({}^{\spadesuit}\)

\({}^{\bullet}\)CMU LTI, \({}^{\diamondsuit}\)Bytedance, UT Austin, Columbia University, NTU Project Page: [https://github.com/RifleZhang/LLaVA-Hound-DPO](https://github.com/RifleZhang/LLaVA-Hound-DPO)

Equal contribution.

###### Abstract

Preference modeling techniques, such as direct preference optimization (DPO), has shown effective in enhancing the generalization abilities of large language model (LLM). However, in tasks involving video instruction-following, providing informative feedback, especially for open-ended conversations, remains a significant challenge. While previous studies have explored using large multimodal models (LMMs) as reward models for guiding preference modeling, their ability to accurately assess the quality of generated responses and their alignment with video content has not been conclusively demonstrated. This paper introduces a novel framework that utilizes detailed video captions as a proxy of video content, enabling language models to incorporate this information as supporting evidence for scoring video Question Answering (QA) predictions. Our approach demonstrates robust alignment with OpenAI GPT-4V model's reward mechanism, which directly takes video frames as input. Furthermore, we show that applying our reward mechanism to DPO algorithm significantly improves model performance on open-ended video QA tasks.

## 1 Introduction

This paper addresses the challenge of aligning LMMs, particularly in tasks that involve video instruction following. Despite recent advancements in reinforcement learning (RL) Ouyang et al. (2022); Bai et al. (2022); Lee et al. (2023); Sun et al. (2023b) and DPO Rafailov et al. (2024); Chen et al. (2024); Hosseini et al. (2024), which have been effective in guiding LLMs towards generating more honest, helpful, and harmless content, their effectiveness in video domain remains limited. The critical obstacle lies in developing a robust reward system capable of distinguishing preferred responses from less preferred ones based on video inputs. The challenge is further complicated by the coverage and potential inaccuracies in generated content, stemming from the scarcity of alignment data across different modalities Liu et al. (2023b); Sun et al. (2023a).

While human preference data is valuable, it is challenging to scale due to its cost and labor-intensive nature, as highlighted by the LLaVA-RLHF Sun et al. (2023a) paper, which collected 10k human-evaluated instances at a considerable cost of $3000. Existing approaches for distilling preferences, such as those for image data using GPT-4V Li et al. (2023d), encounter scalability issues, especially for video inputs that require analyzing multiple frames. While Ahn et al. (2024) leverage a supervised finetuning (SFT) model for self-evaluation, the efficacy of the SFT model remains uncertain, particularly in accurately assessing the factuality of responses in relation to their corresponding videos.

To tackle the aforementioned challenges, we introduce a cost-effective reward mechanism that is both computationally and financially efficient for evaluating the quality of responses generated by video LLMs, serving as a basis for further on-policy preference optimization. We propose the use of detailed video captions as a proxy for video content, enabling a language model analyze the content and assess the quality of an LMM's response to related questions. The language model generates natural language feedback as a chain-of-thought step, and produces a numerical score as the reward, thereby creating an efficient feedback system.

However, high-quality video captions are essential for this process. To mitigate the shortage of high-quality video captions, we have developed a comprehensive video caption dataset, ShareGPTVideo, using a simple prompting technique with the GPT-4V model, comprising 900k captions that encompass a wide range of video content, including temporal dynamics, world knowledge,

object attributes, and spatial relationships. With this video caption dataset available, we verify that our reward mechanism, which utilizes video captions as a proxy, is well-aligned with evaluations derived from the more powerful, albeit costlier, GPT-4V model-generated rewards. Employing this reward mechanism as the basis for DPO algorithm, we train LLAVA-Hound-DPO that achieves an 8.1% accuracy improvement over the SFT counterpart. This marks a significant advancement in video LAM alignment and represents the first successful application of a DPO method in this domain.

Our contributions are outlined as follows:

1. We release a large-scale detailed video caption (900k) and instruction-following (900k) dataset covering a wide range of video content, which facilitates video LMM model training and research.
2. We demonstrate the effective application of DPO to improve model performance by leveraging the language model feedback as reward, which substantially improves model performance on open-ended video QA tasks.
3. We propose an automated _development_ benchmark for evaluating video instruction-following capability, serving as a cost-effective way to validate model performance.

## 2 Related Work

### Large Multi-Modal Models

LMMs Liu et al. (2023b, a); Bai et al. (2023); Chen et al. (2023); Li et al. (2023a) have enabled instruction following across modalities by utilizing LLM as backbones. In the context of video understanding, LLMs have been adapted to process video content Zhang et al. (2023a); Maaz et al. (2023); Li et al. (2023b); Luo et al. (2023); Liu et al. (2023c); Jin et al. (2024); Ahn et al. (2024); Zhang et al. (2024). Models such as Qwen2-VL Wang et al. (2024) and InternVL-2.5Chen et al. (2024c), which scale in both model size (ranging from 1 billion to 78 billion parameters) and training data volume, have demonstrated highly competitive performance in image and video understanding tasks. However, these studies fall outside the scope of this research. Our work adopts Video-LLaVA Lin et al. (2023a) backbone, focusing on model enhancement through preference modeling with the DPO technique.

### Video-text Datasets

Existing video-text datasets typically provide brief sentences or mere keywords as captions, as indicated by Bain et al. (2021); Wang et al. (2023); Yu et al. (2019); Jang et al. (2017); Xu et al. (2016). Shvetsova et al. (2023). Video-ChatGPT Li et al. (2023a)

Figure 1: Workflow diagram showing: a) the use of GPT-4V for creating a detailed caption dataset for videos; b) generating video instruction data for SFT; c) integrating captions into a feedback loop for DPO, improving the model’s performance on video instruction-following tasks.

 

2023b) employs human effort to create high-quality video instructions, albeit limited to the ActivityNet domain with only 100k instruction pairs. Short2Story Han et al. (2023) and Vript Yang et al. (2024) employ GPT-4V for video captioning, with audio details as outputs. Concurrent work Chen et al. (2024b) leverages GPT-4V to label video captions. Furthermore, previous studies Xu et al. (2017); Grunde-McLaughlin et al. (2021); Wu et al. (2024) have employed language models or predefined question templates to generate question-answer pairs. While these approaches can produce a substantial volume of questions and answers, they frequently result in low-quality questions that fail to accurately reflect real-world user inquiries. Our work leverages the GPT-4V model to generate detailed video captions and introduces a paradigm for producing question-answer pairs, which can be effectively utilized for training video LLMs. Additionally, we release this resource as a community asset for LMM training.

### Preference Modeling for LMMs

Preference modeling techniques, such as DPO Deng et al. (2024); Yu et al. (2024); Li et al. (2023d); Gunjal et al. (2023); Sun et al. (2023a) and PPO Sun et al. (2023a), have been applied to LMM alignment. More recently, Ahn et al. (2024) employed reinforcement learning based on AI feedback to enhance video LMM performance. Our work extends DPO to video LMM alignment by incorporating detailed captions as factual evidence for reward modeling.

## 3 Method

As shown in fig. 1, our methodology enhances video LMM alignment through DPO method using rewards from a language model. We elaborate on constructing a video caption dataset in section 3.1. Subsequently, in section 3.2, we discuss the generation of video instruction data and the fine-tuning process of our model. Lastly, section 3.3 details the incorporation of generated captions as a feedback mechanism for DPO method to refine our model's factual alignment in video instruction-following tasks.

### Prompting GPT-4V Model for Detailed Video Caption Distillation

The selection of dataset includes videos from three sources: WebVid (400k) and VIDAL (450k) ActivityNet (50k) datasets. WebVid and VIDAL videos are in the general domain sourced from YouTube, and ActivityNet videos focus on human activities. The three datasets together result in a comprehensive collection of 900k videos. To accommodate the requirement that GPT-4V only takes images as input, we preprocess videos by uniformly extracting ten frames per video content. These frames are then concatenated into a sequence to serve as a proxy for the video. We use GPT-4V to generate a coherent caption for the represented video based

Figure 2: Detailed illustration of the proposed factually-enhanced DPO method.

 

on the frame sequence. The prompt (fig. 19) adheres to guidelines covering temporal dynamics, world knowledge, object attributes, spatial relationships, aesthetic assessments, etc., with the goal of comprehensively understanding the video contents (examples in fig. 3 and fig. 9).

### SFT with Generated Video Instruction Data from Detailed Caption

To generate video instruction-following data for SFT, we adopt a similar methodology outlined in Video-ChatGPT Li et al. (2023b). Specifically, we first randomly sample 300k video captions and then employ ChatGPT to generate \(3\) question-answer pairs conditioned on each caption (prompt in fig. 20). We release the 900k instruction-following data to public, but we only use a random subset of 240k for our training. This approach ensures that the instructional data remains factually consistent with the content of the detailed captions.

### DPO with Language Model Reward

Acquiring high-quality on-policy preference data can be costly and labor-intensive. Although GPT-4V can be used for reward distillation, for video data, its high computation cost1, slow response, and limited accessibility hinder scalability. We propose a cost-efficient method to generate reward data for DPO using detailed video captions as supporting evidence, as shown in fig. 2.

Footnote 1: Video representation is typically encoded with 2048 tokens, while our captions only uses roughly 140 tokens.

Initially, we randomly select a subset of 20k instruction pairs from the dataset described in section 3.2. The SFT model generates six responses per input at a temperature of \(1.0\). This procedure results in 120k question-answer pairs. Subsequently, we employ ChatGPT to evaluate the model responses based on the ground truth answer and detailed description (prompt in fig. 22). ChatGPT generates an output that includes a natural language explanation as chain-of-thought step, followed by a numerical reward score on a scale from \(1\) to \(5\), indicating the overall quality.

For each video and question pair, we randomly select an answer with a score \(\geq\) 3 as positive example, and an answer with a score below \(3\) as negative. Cases where all responses are uniformly scored above or below \(3\) are excluded from the dataset. After the selection process, approximately 17k training instances are compiled for DPO training. Formally, the dataset is denoted as \(\mathcal{D}_{DOP}=\left\{\mathcal{(V},x,y_{w},y_{l})\right\}\), where \(\mathcal{V}\) is the video, \(x\) is the question, \(y_{w}\) and \(y_{l}\) are the positive and negative responses. The DPO objective is defined as below:

\[\begin{split}&\mathcal{L}_{\text{DPO}}\left(\pi_{\theta};\pi_{ \text{ref}}\right)=-\mathbb{E}_{\left(\mathcal{V},x,y_{w},y_{l}\right)\sim \mathcal{D}_{DRO}}\\ &\log\sigma\!\left(\beta\log\frac{\pi_{\theta}\left(y_{w}\mid x ,\mathcal{V}\right)}{\pi_{\text{ref}}\left(y_{w}\mid x,\mathcal{V}\right)}- \beta\log\frac{\pi_{\theta}\left(y_{l}\mid x,\mathcal{V}\right)}{\pi_{\text{ ref}}\left(y_{l}\mid x,\mathcal{V}\right)}\right)\end{split} \right),\]

where \(\pi_{\theta}\) is the policy model to be optimized and \(\pi_{\text{ref}}\) is the base reference model, both models are initialized with SFT weights. \(\sigma\) is the logistic function and \(\beta\) is set to \(0.1\).

For on-policy reward generation, our method incurs a cost of less than $20, under a pricing model of $1.5 per million tokens. In comparison, previous methods of preference data collection, such as in Sun et al. (2023a), required an expenditure of $3,000 to gather 10k human preference data points. Additionally, the method proposed by Li et al. (2023d), which employs GPT-4V for reward data labeling, incurs a significantly higher cost--$30 per million tokens--and demonstrates considerably slower inference speeds.

Figure 3: Example annotation in ShareGPTVideo: Our dataset provides more detailed video caption annotations. The red box demonstrates that textual information in the video is accurately extracted into the caption, while the blue box illustrates the connection to real-world knowledge, such as identifying the spatula’s shape as resembling a Stormtrooper helmet from Star Wars.

 



## 4 Assessment of Evaluator with GPT-4V Caption as Video Content

To assess the effectiveness of our proposed reward assignment method, we conducted a comparative analysis the GPT-4V used as a video QA evaluator. Our method utilizes detailed captions as a proxy of actual video frames, while GPT-4V directly takes in video frames as inputs. Both reward systems follow the same set of guidelines for scoring reward (prompt in fig.23).

To compare the two methods, we sample \(200\) videos from each of the WebVid, VIDAL, and ActivityNet datasets, each associated with one question and two model predictions from our SFT model, with one preferred and one dispreferred by ChatGPT. This results in \(1,200\) examples, for which we used GPT-4V to assign scores. Filtering through the Azure API backend resulted in \(196\), \(151\), and \(143\) videos from each dataset, respectively, having both answers evaluated. The average scores of all examples from ChatGPT and GPT-4V evaluations were \(2.9\) and \(3.5\) respectively, indicating a tendency of GPT-4V to yield slightly positive evaluations. The Pearson Correlation Coefficient (PCC) of \(0.47\) (\(p<0.01\)) suggests a moderate positive correlation. In fig.4 (left), the distribution of the difference between ChatGPT and GPT-4V scores reveals that majority (\(>75\%\)) of ChatGPT scores fall within one standard deviation (\(\sigma=1.31\)) of GPT-4V scores. Additionally, in fig.4 (right), the agreement on preference between ChatGPT and GPT-4V, excluding ties, exceeded \(70\%\). These findings casutiously support our benchmark's applicability in video QA evaluation. Further refinements for better alignment--such as incorporating Likert scales Zhou et al. (2023) or GPT-4 evaluation--are areas for future research.

**Human Annotation of Captions:** To evaluate the quality of the distilled captions, we conducted human annotations focusing on two aspects: coverage and accuracy (hallucination). Annotators were asked to assess each caption by identifying the number of missing items and the number of incorrect facts. The assessment was performed on a sample of 75 videos, with 25 from each domain. The results showed that annotators identified a total of 21 inaccurate items across 14 videos (accuracy: 81%) and 12 missing items across 8 videos (accuracy: 89%). Annotated examples are provided in appendixD.

## 5 Experiments

We adopt Video-LLaVA Lin et al. (2023) as the backbone of our video LMM, but our method can be applied to any other architectures as well.

**Caption Pre-training Stage (LLaVA-Hound-PT):** We use captioning data including 650k image caption data from ALLaVA Chen et al. (2024) and our distilled 900k video caption. We freeze the visual encoder and fine-tune the MLP projector and LLM, with learning rate 2e-5 and batch size 128.

**SFT Stage (LLaVA-Hound-SPT):** We use 600k image instruction data from ALLaVA and our generated 240k video instruction data, with learning rate 5e-6 and batch size 128.

**DPO training Stage (LLaVA-Hound-DPO):** We use the 17k preference data introduced in section3.3 for DPO training. Following Ivison et al. (2023), we train our policy model with full model training for \(3\) epochs with learning rate 5e-7, and a batch size of 128. All the experiments are performed on 8 A100 gpus.

Figure 4: Assessing Evaluator Quality Using Captions in Place of Frames. The left figure shows the distribution of evaluation score differences between ChatGPT (with caption as proxy) and GPT-4V (directly on frames) evaluations. The right figure shows the rate of preference agreement between ChatGPT and GPT-4V as evaluators.





### Benchmark Evaluation

Dataset and Testing EnvironmentWe evaluate model performance on four benchmark datasets: MSVD-QA (Chen and Dolan, 2011), MSRVTT-QA (Xu et al., 2016), TGIF-QA (Jang et al., 2017), and Next-QA (Xiao et al., 2021) using ChatGPT with version gpt-3.5-turbo-0611 to assess model predictions. The evaluation prompts follow (Maaz et al., 2023). In our experiment, we found that different ChatGPT versions have high impact on absolute score of metric, but the overall ranking of models is relatively stable. We select gpt-3.5-turbo-0613 due to its closeness to the reported score in Video-LLaVA paper. Further details on the selection rationale and evaluation pitfalls are discussed in Appendix A.

Baseline SelectionWe select video LMM models that have demonstrated SOTA performance with with accessible code and checkpoints at the time of paper writing, specifically including Video-LLaVA, which is also our choice of architecture. We replicate results including Video-ChatGPT (Maaz et al., 2023), LLaMA-VID (Li et al., 2023c) (7B and 13B), Chat-UniVi (Jin et al., 2023), and Video-LLaVA (Lin et al., 2023b). We copy the results from additional baselines including FrozenBiLM (Yang et al., 2022), VideoCat (Li et al., 2023b) and VideoLLaMA (Zhang et al., 2023a), sourced from their original publication.

ResultsIn table 1, our analysis shows that within the FFT models, LLaMA-VID-7B and Video-LLaVA exhibit comparable performance, with LLaMA-VID-13B performing the best. Our LLaVA-Hound-SFT model achieves comparable performance to LLaM-VID-13B. Incorporating preference modeling, LLaVA-Hound-DPO achieves an average accuracy of \(70.75\%\), surpassing LLaVA-Hound-SFT, which has an average accuracy of \(62.65\%\), by \(8.1\%\). Furthermore, LLaVA-Hound-DPO exhibits superior accuracy compared to other RL methods such as VLMRLAIF. In table 2, our model demonstrated consistent result on a relative new benchmark Next-QA.

Error AnalysisFigure 5 illustrates two examples. In the left example, LLaVA-Hound-SFT

\begin{table}
\begin{tabular}{l c c c c c c c} \hline \hline  & & \multicolumn{4}{c}{**Existing Video QA Benchmark from (Maaz et al., 2023)**} \\ \cline{3-9} Methods & **LLM Size** & \multirow{2}{*}{LCL} & \multirow{2}{*}{Acc.} & \multirow{2}{*}{Score} & Acc. & \multirow{2}{*}{Score} & Acc. & Score \\ \cline{1-1} \cline{6-7} FrozenBiLM (Yang et al., 2022)+ & & & & & & & \\ VideoL_AA (Zhang et al., 2023a)+ & & & & & & & \\ LLaMA-Adapter (Zhang et al., 2023b)+ & & & & & & & \\ VideoChat (Li et al., 2023b)+ & & & & & & & \\ BT-Adapter (Liu et al., 2023c)+ & & & & & & & \\ Video-ChatGPT (Maaz et al., 2023) & & & & & & & \\ Chat-UniVi (Jin et al., 2023) & & & & & & & \\ VideoChat2 (Li et al., 2023c) & & & & & & & \\ VideoChat2 (Li et al., 2023c) & & & & & & & \\ Video-LLaVA (Lin et al., 2023b) & & & & & & & \\ LLaMA-VID (Li et al., 2023e) & & & & & & & \\ LLaMA-VID (Li et al., 2023e) & & & & & & & \\ LLaMA-VID (Li et al., 2023e) & & & & & & & \\ VLM-RLAIF (Ahn et al., 2024)+ & & & & & & & \\ \hline LLaVA-Hound-SFT & & & & & & & \\ LLaVA-Hound-DPO & & & & & & & \\ \hline \hline \end{tabular}
\end{table}
Table 1: **Evaluation of Model Performance on Zero-Shot Video Question Answering Benchmarks Using spgt-3.5-turbo-0613. Models denoted with \(*\) have their results directly sourced from their original publications. Caution is advised when interpreting these results; see Appendix A for an in-depth analysis of evaluation challenges. All other baseline models were reproduced by our team.**

\begin{table}
\begin{tabular}{l l r} \hline \hline \multirow{2}{*}{**No. Methods**} & \multicolumn{2}{r}{**Next-QA**} \\ \cline{2-3}  & Acc. & Score \\ \hline
[1] & Video-ChatGPT (Maaz et al., 2023) & 45.23 & 2.09 \\
[2] & LLaMA-VID-7B (Li et al., 2023e) & 49.43 & 3.24 \\
[4] & Chat-UniVi (Jin et al., 2023) & 47.62 & 3.14 \\
[5] & Video-LLaVA (Lin et al., 2023b) & 48.97 & 3.25 \\ \hline
[6] & LLava-Hound-SFT & 60.60 & 3.51 \\
[7] & LLava-Hound-DPO & **74.27** & **3.74** \\ \hline \hline \end{tabular}
\end{table}
Table 2: Evaluation on Next-QA benchmark using gpt-3.5-turbo-0611 on official test set.



provides an accurate description of the video's first half but introduces a hallucination with the phrase "I'm not scared of space," absent in the video content. LLaVA-Hound-DPO yields a more accurate inference. In the right example, both LLaVA-Hound-SFT and Video-LLaVA models produce incorrect inferences, whereas LLaVA-Hound-DPO successfully correctly identifies the subject in the video.

### Open-ended QA Analysis

In this section, we conduct analysis on open-ended long-form QA with a proposed development benchmark. Specifically, we select 2,000 videos from each source: WebVidBain et al. (2021), VIDAL Zhu et al. (2023), ActivityNet Fabian Caba Heilbron and Niebles (2015), MSRVTT Xu et al. (2016), MSVD Chen and Dolan (2011), TGIF Jang et al. (2017), and Something-something V2 (SSV2) Goyal et al. (2017). For each video, ChatGPT was utilized to generate three QA pairs based on the detailed captions, and we evaluate model predictions with our reward mechanism. WebVid, VIDAL, ActivityNet are classified as indomethacin, which are involved in the model's training pipeline. MSRVTT, MSVD, TGIF, SSV2 are classified as out-of-domain.

The evaluation reveals insights into (1) the quality of long-form open-ended QA, (2) in-domain and out-of-domain generalization, and (3) Ablations on SFT and DPO experiments. Additionally, we select our best performing model on the development bench before evaluating on public benchmarks, which avoids tuning hyperparameters on

\begin{table}
\begin{tabular}{l c c c c c c c c} \hline \hline  & \multicolumn{8}{c}{**Proposed Video QA Benchmark (Out-of-domain)**} \\ \multirow{2}{*}{**Methods**} & **MSVD-QA** & **MSRVTUT-QA** & **TGIF-QA** & **SSV2-QA** \\ \cline{2-9}  & Acc. & Score & Acc. & Score & Acc. & Score & Acc. & Score \\ \hline Video-ChatGPT Maaz et al. (2023) & 34.06 & 2.20 & 25.65 & 1.98 & 31.35 & 2.09 & 19.36 & 1.75 \\ LLaMA-VID-7B Li et al. (2023) & 34.14 & 2.21 & 25.02 & 1.99 & 27.18 & 2.00 & 22.16 & 1.84 \\ LLaMA-VID-13B Li et al. (2023) & 35.81 & 2.25 & 26.34 & 2.02 & 27.58 & 2.01 & 21.98 & 1.83 \\ Chat-UnVi Jin et al. (2023) & 35.61 & 2.23 & 25.89 & 2.01 & 33.23 & 2.13 & 20.59 & 1.79 \\ Video-LLaVA Lin et al. (2023) & 39.46 & 2.37 & 30.78 & 2.15 & 32.95 & 2.18 & 24.31 & 1.90 \\ \hline LLaVA-Hound-SFT & 66.99 & 3.09 & 57.82 & 2.85 & 66.13 & 3.07 & 35.07 & 2.23 \\ LLAVA-Hound-DPO & **73.64** & **3.12** & **68.29** & **2.98** & **74.00** & **3.12** & **48.89** & **2.53** \\ \hline \hline LLaVA-Hound-PT + Image Inst. & 65.19 & 2.96 & 48.66 & 2.52 & 53.83 & 2.62 & 29.60 & 2.04 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Our proposed video QA benchmark evaluation on out-of-domain dataset using gpt-3.5-turbo-0301, with detailed captions as supporting evidence.

\begin{table}
\begin{tabular}{c l c c c c c} \hline \hline  & \multicolumn{4}{c}{**Proposed Video QA Benchmark (In-domain)**} \\ \multirow{2}{*}{**No. Methods**} & **ActivityNet-QA** & **VIDAL-QA** & **WebVid-QA** \\ \cline{2-7}  & Acc. & Score & Acc. & Score & Acc. & Score \\ \hline
[1] & Video-ChatGPT Maaz et al. (2023) & 34.17 & 2.19 & 29.35 & 2.10 & 38.88 & 2.27 \\
[2] & LLaMA-VID-7B Li et al. (2023) & 36.54 & 2.27 & 30.58 & 2.15 & 36.99 & 2.24 \\
[3] & LLaMA-VID-13B Li et al. (2023) & 37.33 & 2.29 & 32.50 & 2.18 & 39.73 & 2.30 \\
[4] & Chat-UniVi Jin et al. (2023) & 39.35 & 2.32 & 31.40 & 2.16 & 40.05 & 2.31 \\
[5] & Video-LLaVA Lin et al. (2023) & 41.35 & 2.38 & 34.30 & 2.24 & 42.47 & 2.39 \\ \hline
[6] & LLAVA-Hound-SFT & 66.62 & 3.05 & 60.50 & 2.88 & 71.07 & 3.17 \\
[7] & LLOVA-Hound-DPO & **76.62** & **3.18** & **70.06** & **3.04** & **79.82** & **3.29** \\ \hline \hline
[8] & LLAVA-Hound-PT + ImageNet Inst. & 69.31 & 3.09 & 60.57 & 2.85 & 68.03 & 3.02 \\
[9] & LLAAV-Hound-PT + VChat & 67.34 & 3.02 & 62.33 & 2.89 & 68.98 & 3.00 \\
[10] & LLAAV-Hound-DPO + training MLP & 71.89 & 3.10 & 65.57 & 2.95 & 75.37 & 3.21 \\
[11] & LLAAV-Hound-SFT + Self-play & 64.11 & 2.85 & 56.28 & 2.68 & 67.89 & 2.95 \\
[12] & LLAAV-Hound-DPO w/ lr3e-7 & 71.13 & 3.08 & 64.90 & 2.92 & 73.25 & 3.17 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Our proposed video QA benchmark evaluation on in-domain dataset using gpt-3.5-turbo-0301, with detailed captions as supporting evidence.

 

test data. Comparisons are shown in appendix E.

**Domain Generalization:** Table 3 and table 4 shows the in-domain and out-of-domain evaluation. SFT with our data tends to perform better both in- and out-of-domain, and DPO further enhances the model performance, showing the effectiveness of preference modeling.

**Video LMM without Video Instruction:**[8] in table 3 is baseline trained with only image instruction fine-tuned on LLAVA-Hound-PT, which achieves an average accuracy of \(65.97\%\), comparable to the LLAVA-Hound-SFT model's \(66.06\%\) in in-domain QA scenarios. However, its performance significantly drops in out-of-domain QA contexts (\(49.32\%\) vs. \(56.50\%\)), suggesting that Video QA training could potentially enhance generalization capabilities.

**Quality of Generated SFT:**[9] substitutes our generated video QA with the Video-ChatGPT dataset for Video-LLaVIA fine-tuning. A comparison between the findings of [9] and [6] reveals a marginal performance disparity of \(0.2\%\) in average accuracy, indicating that the quality of our generated QA closely parallels that of the existing video QA datasets. Given the similar quality in SFT data, the large gain of [6] over [5] can be reasonably concluded from large-scale pre-training on video captions.

**Unfreeze MLP:** The comparison between [10] and [7] reveals a significant decrease in performance when the MLP is unfrozen during DPO training. Despite this drop, however, the performance remains superior to that of the SFT baseline.

**Smaller Learning Rate:** The comparison between [12] and [7] reveals that using a smaller learning rate of 3e-7 (vs. 5e-7) results in a decreasing of model performance. This highlights the future improvements by finding better hyperparameters.

**Self-Play vs. DPO:**[Chen et al., 2024d] introduced a self-play methodology for DPO training, which designates ground truth answers as preferred and model-generated responses as dispreferred. When comparing the results of [11] with those in [6], a notable decrease in accuracy by \(3\%\) from the SFT model is observed, suggesting that self-play may be less effective for video LMM alignment, and introducing reward model is helpful.

**DPO Accuracy vs. Training Epochs.** The left of fig. 6 depicts the generalization performance of the model on out-of-domain video QA tasks with respect to the number of training epochs. We observe a consistent enhancement in model performance among datasets during the initial 0 to 2 epochs, with peak performance materializing at around 2.5 epochs, which corresponds to 350 training steps.

**Test-time Compute.** Previous works have demonstrated that leveraging test-time compute by generating and evaluating multiple candidates can enhance model performance (Hosseini et al., 2024). In this study, we assess the effectiveness of test-time compute and compare its performance with direct inference using our DPO model. For ranking, we employ the DPO model as a ranker to score candidate answers produced by the SFT model, operating at a temperature setting of 1.0. As illustrated on the right in fig. 6, we present the test accuracy progression when selecting the best among \(N\) candidates using the DPO ranker. Initial observations reveal that the SFT model, when set to a temperature of 1.0, attains a lower accuracy (43.3%) com

Figure 5: Examples from MSRVTTI-QA and MSVD-QA showcase that our LLAVA-Hound-DPO generates better responses, and reveal key limitations of the existing benchmark evaluation.

 

pared to greedy decoding (57.8%). A consistent improvement in performance is observed as the number of candidates increases, eventually plateauing at approximately 62% accuracy with 64 candidates. However, this performance remains inferior to direct answer generation using the DPO model, which achieves an accuracy of 68.29%. This discrepancy suggests that the DPO model exhibits stronger generalization in answer generation, despite being trained with a reward classification loss. The contrasting results compared to Hosseini et al. (2024) may stem from task differences, specifically Math QA versus Video QA. Refer to appendix F for more results.

## 6 Conclusion

We study the techniques for effective video LMM alignment. Specifically, we propose an cost-effective reward system that utilizes detailed captions as proxies for video content. We have shown the reward scores is well-aligned with the evaluation metrics of GPT-4V, and DPO training greatly enhances model performance. In addition, we have released 900k detailed video caption, 900k video instruction-following data, and 17k preference data pairs, with a complete code pipeline including pretraining for video captioning, fine-tuning for video instruction following and reinforcement learning with DPO for better LMM alignment.

## 7 Limitations

Firstly, several evaluation datasets, such as Video-MME Fu et al. (2024) featuring multiple-choice questions, were not included in our study. These datasets were available at or before the completion of our manuscript. Given our focus on enhancing open-ended question answering, multiple-choice datasets were not incorporated into our training process. Consequently, we did not retrain the model to include this data.

Secondly, the benchmark we developed is fully automated and does not incorporate human corrections for captions and QA. Human annotations indicate that caption accuracy ranges from 80% to 90%, inherently introducing errors. Therefore, we recommend using this benchmark solely for model development and hyperparameter tuning, treating performance metrics as indicative rather than definitive.

## References

* Ahn et al. (2024) Daechul Ahn, Yura Choi, Youngjae Yu, Dongyeop Kang, and Jonghyun Choi. 2024. Tuning large multimodal models for videos using reinforcement learning from ai feedback. _arXiv preprint arXiv:2402.03746_.
* Bai et al. (2023) Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023. Qwen-vl: A frontier large vision-language model with versatile abilities. _arXiv preprint arXiv:2308.12966_.
* Bai et al. (2022) Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. 2022. Constitutional ai: Harmlessness from ai feedback. _arXiv preprint arXiv:2212.08073_.
* Bain et al. (2021a) Max Bain, Arsha Nagrani, Gul Varol, and Andrew Zisserman. 2021a. Frozen in time: A joint video and

Figure 6: The left figure shows the test set accuracy of the DPO model w.r.t the number of training epochs. The right figure shows a comparison of DPO model performance as generator vs. ranker.

 

image encoder for end-to-end retrieval. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_, pages 1728-1738.
* Bain et al. (2021b) Max Bain, Arsha Nagrani, Gul Varol, and Andrew Zisserman. 2021b. Frozen in time: A joint video and image encoder for end-to-end retrieval. In _ICCV_.
* Chen and Dolan (2011) David Chen and William B Dolan. 2011. Collecting highly parallel data for paraphrase evaluation. In _Proceedings of the 49th annual meeting of the association for computational linguistics: human language technologies_, pages 190-200.
* Chen et al. (2024a) Guiming Hardy Chen, Shunian Chen, Ruifei Zhang, Junying Chen, Xiangbo Wu, Zhiyi Zhang, Zhihong Chen, Jianquan Li, Xiang Wan, and Benyou Wang. 2024a. Allava: Harnessing gpt4v-synthesized data for a lite vision-language model. _arXiv preprint arXiv:2402.11684_.
* Chen et al. (2023) Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. 2023. Shikra: Unleashing multimodal llm's referential dialogue magic. _arXiv preprint arXiv:2306.15195_.
* Chen et al. (2024b) Lin Chen, Xilin Wei, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Kang, Zehui Chen, Haodong Duan, Bin Lin, Zhenyu Tang, et al. 2024b. Sharegpt4video: Improving video understanding and generation with better captions. _arXiv preprint arXiv:2406.04325_.
* Chen et al. (2024c) Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhayang Liu, et al. 2024c. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. _arXiv preprint arXiv:2412.05271_.
* Chen et al. (2024d)Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. 2024d. Self-play fine-tuning converts weak language models to strong language models. _arXiv preprint arXiv:2401.01335_.
* Deng et al. (2024e) Yihie Deng, Pan Lu, Fan Yin, Ziniu Hu, Sheng Shen, James Zou, Kai-Wei Chang, and Wei Wang. 2024. Enhancing large vision language models with self-training on image comprehension. _arXiv preprint arXiv:2405.19716_.
* Ghaem Fabian Caba Heilbron (2015) Bernard Ghanem Fabian Caba Heilbron, Victor Escorcia and Juan Carlos Niebles. 2015. Activitynet: A large-scale video benchmark for human activity understanding. In _CVPR_.
* Fu et al. (2024) Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhua Ren, Renruia Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, Peixian Chen, Yanwei Li, Shaohul Lin, Siriu Zhao, Ke Li, Tong Xu, Xiawu Zheng, Enhong Chen, Rongrong Ji, and Xing Sun. 2024. Video-mmre: The first-ever comprehensive evaluation benchmark of multi-modal llvms in video analysis. _Preprint_, arXiv:2405.21075.
* Goyal et al. (2017) Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, Reuna Kim, Valentin Haenel, Ingio Fruend, Peter Yianilos, Moritz Mueller-Fretiag, et al. 2017. The" something something? video database for learning and evaluating visual common sense. In _ICCV_.
* Grunde-McLaughlin et al. (2021) Madeeline Grunde-McLaughlin, Ranjay Krishna, and Maneesh Agrawala. 2021. Agqa: A benchmark for compositional spatio-temporal reasoning. In _CVPR_.
* Gunjal et al. (2023) Anisha Gunjal, Jifah Yin, and Erhan Bas. 2023. Detecting and preventing hallucinations in large vision language models. _arXiv preprint arXiv:2308.06394_.
* Han et al. (2023) Mingfei Han, Linjie Yang, Xiaojun Chang, and Heng Wang. 2023. Shot2story20k: A new benchmark for comprehensive understanding of multi-shot videos. _arXiv preprint arXiv:2312.10300_.
* Hosseini et al. (2024)Arian Hosseini, Xingdi Yuan, Nikolay Malkin, Aaron Courville, Alessandro Sordoni, and Rishabh Agarwal. 2024. V-star: Training verifiers for self-taught reasoners. _arXiv preprint arXiv:2402.06457_.
* Ivison et al. (2023) Hamish Ivison, Yizhong Wang, Valentina Pyatkin, Nathan Lambert, Matthew Peters, Pradeep Dasigi, Joel Jang, David Wadden, Noah A Smith, Iz Beltagy, et al. 2023. Camels in a changing climate: Enhancing lm adaptation with tulu 2. _arXiv preprint arXiv:2311.10702_.
* Jang et al. (2017) Yunseok Jang, Yale Song, Youngjae Yu, Youngjin Kim, and Gunhee Kim. 2017. Tgif-ga: Toward spatio-temporal reasoning in visual question answering. In _CVPR_.
* Jin et al. (2023) Peng Jin, Ryuichi Takanobu, Caiwan Zhang, Xiaochun Cao, and Li Yuan. 2023. Chat-univi: Unified visual representation empowers large language models with image and video understanding. _arXiv preprint arXiv:2311.08046_.
* Jin et al. (2024) Yang Jin, Zhicheng Sun, Kun Xu, Liwei Chen, Hao Jiang, Quzhe Huang, Chengru Song, Yuliang Liu, Di Zhang, Yang Song, et al. 2024. Video-lavit: Unified video-language pre-training with decoupled visual-motional tokenization. _arXiv preprint arXiv:2402.03161_.
* Lee et al. (2023) Harrison Lee, Samrat Phatale, Hassan Mansoor, Kellie Lu, Thomas Mesnard, Colton Bishop, Victor Carbune, and Abhinav Rastogi. 2023. Rlaif: Scaling reinforcement learning from human feedback with ai feedback. _arXiv preprint arXiv:2309.00267_.
* Li et al. (2023a)Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023a. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. _arXiv preprint arXiv:2301.12597_.
* Li et al. (2023b) KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. 2023b. Videochat: Chat-centric video understanding. _arXiv preprint arXiv:2305.06355_ 

Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. 2023c. Mvbench: A comprehensive multi-modal video understanding benchmark. _arXiv preprint arXiv:2311.17005_.
* Li et al. (2023d) Lei Li, Zhihui Xie, Mukai Li, Shunian Chen, Peiyi Wang, Liang Chen, Yazheng Yang, Benyou Wang, and Lingpeng Kong. 2023d. Silkie: Preference distillation for large visual language models. _arXiv preprint arXiv:2312.10665_.
* Li et al. (2023e) Yanwei Li, Chengyao Wang, and Jiaya Jia. 2023e. Llama-vid: An image is worth 2 tokens in large language models. _arXiv preprint arXiv:2311.17043_.
* Lin et al. (2023a) Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. 2023a. Video-llava: Learning united visual representation by alignment before projection. _Preprint_, arXiv:2311.10122.
* Lin et al. (2023b) Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Li Yuan. 2023b. Video-llava: Learning united visual representation by alignment before projection. _arXiv preprint arXiv:2311.10122_.
* Liu et al. (2023a) Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2023a. Improved baselines with visual instruction tuning. _arXiv preprint arXiv:2310.03744_.
* Liu et al. (2023b) Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023b. Visual instruction tuning. _arXiv preprint arXiv:2304.08485_.
* Liu et al. (2023c) Ruyang Liu, Chen Li, Yixiao Ge, Ying Shan, Thomas H Li, and Ge Li. 2023c. One for all: Video conversation is feasible without video instruction tuning. _arXiv preprint arXiv:2309.15785_.
* Luo et al. (2023) Ruipu Luo, Ziwang Zhao, Min Yang, Junwei Dong, Minghui Qiu, Pengcheng Lu, Tao Wang, and Zhongyu Wei. 2023. Valley: Video assistant with large language model enhanced ability. _arXiv preprint arXiv:2306.07207_.
* Maaz et al. (2023) Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. 2023. Video-chatgpt: Towards detailed video understanding via large vision and language models. _arXiv preprint arXiv:2306.05424_.
* Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. _Advances in Neural Information Processing Systems_, 35:27730-27744.
* Rafailov et al. (2024) Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2024. Direct preference optimization: Your language model is secretly a reward model. _Advances in Neural Information Processing Systems_, 36.
* Shvetsova et al. (2023) Nina Shvetsova, Anna Kukleva, Xudong Hong, Christian Rupprecht, Bernt Schiele, and Hilde Kuehnze. 2023. Howtocaption: Prompting llms to transform video annotations at scale. _Preprint_, arXiv:2310.04900.
* Sun et al. (2023a) Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang-Yan Gui, Yu-Xiong Wang, Yiming Yang, et al. 2023a. Aligning large multimodal models with factually augmented rhf. _arXiv preprint arXiv:2309.14525_.
* Sun et al. (2023b) Zhiqing Sun, Yakang Shen, Hongxin Zhang, Qinhong Zhou, Zhenfeng Chen, David Cox, Yiming Yang, and Chuang Gan. 2023b. Salmon: Self-alignment with principle-following reward models. _arXiv preprint arXiv:2310.05910_.
* Wang et al. (2024) Peng Wang, Shuai Bai, Sinan Tan, Shijile Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. 2024. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. _arXiv preprint arXiv:2409.12191_.
* Wang et al. (2023) Yi Wang, Yinan He, Yizhuo Li, Kunchang Li, Jianxuo Yu, Xin Ma, Xinhao Li, Guo Chen, Xinyuan Chen, Yaohui Wang, et al. 2023. Internvid: A large-scale video-text dataset for multimodal understanding and generation. _arXiv preprint arXiv:2307.06942_.
* Wu et al. (2024) Bo Wu, Shoubin Yu, Zhenfang Chen, Joshua B Tenenbaum, and Chuang Gan. 2024. Star: A benchmark for situated reasoning in real-world videos. _arXiv preprint arXiv:2405.09711_.
* Xiao et al. (2021) Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. 2021. Next-qa: Next phase of question-answering to explaining temporal actions. In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_, pages 9777-9786.
* Xu et al. (2017) Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yueting Zhuang. 2017. Video question answering via gradually refined attention over appearance and motion. In _Proceedings of the 25th ACM international conference on Multimedia_, pages 1645-1653.
* Xu et al. (2016) Jun Xu, Tao Mei, Ting Yao, and Yong Rui. 2016. Msrtvt: A large video description dataset for bridging video and language. In _Proceedings of the IEEE conference on computer vision and pattern recognition_, pages 5288-5296.
* Yang et al. (2022) Antoine Yang, Antoine Miech, Josef Sivic, Ivan Laptev, and Cordelia Schmid. 2022. Zero-shot video question answering via frozen bidirectional language models. _NeurIPS_.
* Yang et al. (2024) Dongjie Yang, Suyuan Huang, Chengqiang Lu, Xiaodong Han, Haoxin Zhang, Yan Gao, Yao Hu, and Hai Zhao. 2024. Vript: A video is worth thousands of words. _arXiv preprint arXiv:2406.06040_ 

Tianyu Yu, Haoye Zhang, Yuan Yao, Yunkai Dang, Da Chen, Xiaoman Lu, Ganqu Cui, Taiwan He, Zhiyuan Liu, Tat-Seng Chua, et al. 2024. Rlaif-v: Aligning mlms through open-source ai feedback for super gpt-4v trustworthiness. _arXiv preprint arXiv:2405.17220_.
* Xu et al. (2019) Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, and Dacheng Tao. 2019. Activitynet-qa: A dataset for understanding complex web videos via question answering. In _Proceedings of the AAAI Conference on Artificial Intelligence_, volume 33, pages 9127-9134.
* Zhang et al. (2023a) Hang Zhang, Xin Li, and Lidong Bing. 2023a. Video-llama: An instruction-tuned audio-visual language model for video understanding. _arXiv preprint arXiv:2306.02858_.
* Zhang et al. (2023b) Renrui Zhang, Jiaming Han, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, Peng Gao, and Yu Qiao. 2023b. Llama-adapter: Efficient fine-tuning of language models with zero-init attention. _arXiv preprint arXiv:2303.16199_.
* Zhang et al. (2024) Yuanhan Zhang, Bo Li, haotian Liu, Yong jae Lee, Liangke Gui, Di Fu, Jiashi Feng, Ziwei Liu, and Chunyuan Li. 2024. Llava-next: A strong zero-shot video understanding model.
* Zhou et al. (2023) Xuhui Zhou, Hao Zhu, Leena Mathur, Ruohong Zhang, Haofei Yu, Zhengyang Qi, Louis-Philippe Morency, Yunatan Bisk, Daniel Fried, Graham Neubig, et al. 2023. Stopgia: Interactive evaluation for social intelligence in language agents. _arXiv preprint arXiv:2310.11667_.
* Zhu et al. (2023) Bin Zhu, Bin Lin, Munan Ning, Yang Yan, Jiaxi Cui, HongFaa Wang, Yatian Pang, Wenhao Jiang, Junwu Zhang, Zongwei Li, et al. 2023. Languagebind: Extending video-language pretraining to n-modality by language-based semantic alignment. _arXiv preprint arXiv:2310.01852_.

 



## Appendix A Effect of ChatGPT Version on Official Benchmark Evaluation

In Table 5, we show impact of using different ChatGPT versions on metric scores within zero-shot video question answering benchmarks. Our analysis reveals significant variations in the absolute scores across ChatGPT versions, but based on the average accuracy metric, the relative ranking of models under the same ChatGPT version shows consistency.

This comparison underscores a critical issue: many prior studies neglect to specify the ChatGPT version used, potentially leading to inaccurate conclusions during evaluation. We advocate for the explicit designation of the ChatGPT version in future evaluations. Analysis from Table 5 indicates that the version gpt-3.5-turbo-0613 aligns most closely with the performance of the Video-LLaVA Lin et al. (2023) model, serving as the benchmark for model performance comparison in our study.

\begin{table}
\begin{tabular}{l c c c c c c c c c} \hline \multirow{2}{*}{**Methods**} & \multirow{2}{*}{**LLM Size**} & \multicolumn{2}{c}{**MSVD-QA**} & \multicolumn{2}{c}{**MSRVTT-QA**} & \multicolumn{2}{c}{**TGIF-QA**} & \multicolumn{2}{c}{**Summary**} \\ \cline{3-9}  & & Acc. & Score & Acc. & Score & Acc. & Score & Avg Acc. & Rank \\ \hline \hline \multicolumn{8}{c}{gpt-3.5-turbo-0301 evaluation} \\ \hline Video-ChatGPT Maaz et al. (2023) & 7B & 78.62 & 4.00 & 71.67 & 3.63 & 56.31 & 3.45 & 68.87 & 6 \\ LLaMA-VID Li et al. (2023) & 7B & 82.57 & 4.12 & 71.94 & 3.65 & 59.00 & 3.63 & 71.17 & 4 \\ LLaMA-VID Li et al. (2023) & 13B & 83.72 & 4.16 & 73.63 & 3.68 & 59.72 & 3.66 & 72.36 & 3 \\ Chat-Univi Jin et al. (2023) & 7B & 80.52 & 4.02 & 66.92 & 3.41 & 57.73 & 3.49 & 68.39 & 7 \\ Video-LLaVA Lin et al. (2023) & 7B & 81.44 & 4.08 & 73.29 & 3.65 & 58.34 & 3.61 & 71.02 & 5 \\ LLaVA-Hound-SFT LLAVA-Hound-DPO R & 7B & 85.65 & 4.10 & 73.85 & 3.62 & 64.98 & 3.65 & 74.83 & 2 \\ LLaVA-Hound-DPO & 7B & **88.50** & **4.20** & **82.10** & **3.84** & **75.48** & **3.81** & **82.03** & 1 \\ \hline \hline \multicolumn{8}{c}{gpt-3.5-turbo-0613 evaluation} \\ \hline Video-ChatGPT Maaz et al. (2023) & 7B & 68.55 & 3.80 & 58.90 & 3.36 & 47.83 & 3.21 & 58.43 & 6 \\ LLaMA-VID Li et al. (2023) & 7B & 72.62 & 3.92 & 58.73 & 3.38 & 49.21 & 3.28 & 60.19 & 4 \\ LLaMA-VID Li et al. (2023) & 13B & 74.29 & 3.96 & 59.82 & 3.41 & 50.83 & 3.33 & 61.65 & 3 \\ Chat-UnviV Jin et al. (2023) & 7B & 70.01 & 3.79 & 53.08 & 3.14 & 46.09 & 3.12 & 56.39 & 7 \\ Video-LLaVA Lin et al. (2023) & 7B & 71.75 & 3.88 & 58.97 & 3.39 & 48.39 & 3.24 & 59.70 & 5 \\ LLaVA-Hound-SFT LLAVA-Hound-DSP TLAVA-Hound-DPO R & 7B & **80.73** & **4.07** & **70.15** & **3.66** & **61.38** & **3.46** & **70.75** & 1 \\ \hline \hline \multicolumn{8}{c}{gpt-3.5-turbo-1106 evaluation} \\ \hline Video-ChatGPT Maaz et al. (2023) & 7B & 73.02 & 4.01 & 62.09 & 3.61 & 47.76 & 3.36 & 60.96 & 6 \\ LLaMA-VID Li et al. (2023) & 7B & 75.49 & 4.08 & 62.09 & 3.61 & 51.72 & 3.47 & 63.10 & 4 \\ LLaMA-VID Li et al. (2023) & 13B & 76.97 & 4.10 & 63.16 & 3.61 & 52.53 & 3.50 & 64.22 & 3 \\ Chat-UnVi Vijn et al. (2023) & 7B & 72.22 & 3.92 & 55.02 & 3.35 & 48.16 & 3.31 & 58.47 & 7 \\ Video-LLaVA Lin et al. (2023) & 7B & 74.76 & 4.04 & 62.70 & 3.60 & 51.21 & 3.45 & 62.89 & 5 \\ LLaVA-Hound-SFT LLAVA-Hound-DPO R & 81.09 & 4.08 & 64.13 & 3.57 & 58.05 & 3.53 & 67.76 & 2 \\ LLaVA-Hound-DPO R & **86.05** & **4.23** & **76.75** & **3.85** & **70.02** & **3.71** & **77.61** & 1 \\ \hline \end{tabular}
\end{table}
Table 5: **Performance Evaluation Across ChatGPT Versions on Zero-Shot Video Question Answering Benchmarks. This table compares the performance of state-of-the-art video LMMs evaluated under different ChatGPT versions. The absolute performance metrics scored by ChatGPT vary by versions. However, the comparative ranking of models under the same ChatGPT version is relatively stable.**

Evaluation of Captioning Ability from pre-training

In Figure 8, we present the video captioning ability of models across various datasets, with a total of 900k distilled data instances. GPT-4V is employed for self-evaluation (fig. 21), serving as the upper-bound performance, while the Video-LLaVA serves for comparative analysis, establishing a baseline. Notably, Video-LLaVA is trained on 54K video QA data instances. However, our first checkpoint, utilizing only 10% of the data, is trained on 90k high-quality caption data instances, likely accounting for the observed performance disparity in the video captioning task. Our results demonstrate that incorporating more distilled data contributes to improved model performance across both in-domain and out-of-domain datasets. Despite these improvements, a performance discrepancy with the GPT-4V model remains. Further, we evaluate the generalization potential in specific data subsets, as shown in fig. 7 in the Appendix. These subsets reveal varying degrees of generalization challenges for different types of dataset. For example, the WebVid subset, which concentrates on relatively static scenes, necessitates less data for effective training compared to the VIDAL subset, which is marked by dynamic scene transitions and a diversity of video themes.

Figure 8: The video caption ability w.r.t number of training data evaluated on both in-domain and out-of-domain test videos using GPT-4V.

Figure 7: Training subsets exhibit varying levels of generalization difficulty. The WebVid subset (left) requires less data compared to the VIDAL subset (right)



GPT-4V Caption Distillation

## Appendix D Human Annotated Examples of Distilled Captions

Figure 10: Human Annotation Analysis: The video frames depict a truck driving away from the camera, but the distilled caption incorrectly describes the direction, which is identified as a hallucination. Additionally, human annotators noted that the GPT-4V model failed to recognize a man holding a cup.

Figure 9: A showcase of our dataset for detailed video caption automatically distilled from GPT-4V via our prompt technique in section 3.

 

Figure 11: Human Annotation Analysis: GPT-4V incorrectly identified a man wearing blue pants as a man standing on a blue ladder, which is labeled as a hallucination.

 

Figure 12: Human Annotation Analysis: The example demonstrates OCR hallucination and a coverage issue. The text on the woman’s t-shirt is incorrectly labeled as ‘Johnny’s BBQ’ instead of ’Jimmy’s BBQ.’ Additionally, the text ‘Step 3: Adhere Squares Together’ is missing from the description.

Figure 13: Human Annotation Analysis: The caption does not contain any hallucinations, but some text recognized by human annotators is missing, such as ‘Coca Cola’ and ‘Gopher’ on the scoreboard, as well as the time of the score match shown.

 



## Appendix A

Figure 14: Human Annotation Analysis: The figures illustrate a scoring transition that is not recognized by the GPT-4V model, highlighting the challenge of using static frames for accurate captioning.



Video QA Dataset Demonstration

To build the test dataset, we curated a dataset comprising 2,000 videos from each source: WebVid (Bain et al., 2021), VIDAL (Zhu et al., 2023), ActivityNet (Fabian Caba Heilbron and Niebles, 2015), MSRVTT (Xu et al., 2016), MSVD (Chen and Dolan, 2011), TGIF (Jang et al., 2017), and Something-something V2 (SSV2) (Goyal et al., 2017). For each video, ChatGPT was utilized to generate three QA pairs based on the detailed captions. The first three datasets (WebVid, VIDAL, ActivityNet) are classified as in-domain, since the captions and QA pairs derived from these sources are used in the model's training pipeline. Conversely, the remaining datasets (MSRVTT, MSVD, TGIF, SSV2) are classified as out-of-domain, evaluating model's zero-shot QA ability.

Appendix E compares our development benchmark with existing benchmark dataset, we identify several issues with the existing evaluation methods: (1) the auto-generated questions from current benchmarks may be grammatically incorrect or nonsensical, and (2) the answers are limited to a _single_ word, which is inadequate for evaluating LMMs in the context of long-form QA. We conduct further analysis on open-ended long-form QA with a proposed development benchmark.

We apply our reward system as described in section 4 and report scores from ChatGPT. A score of \(\geq 3\) is considered correct for accuracy calculations. The development benchmark reveals insights into (1) the quality of long-form open-ended QA, and (2) in-domain and out-of-domain generalization. Additionally, our development benchmark results correlate with existing benchmarks. We recommend that models be evaluated on the development benchmark first, followed by human evaluation.

Figure 15: Comparing testing QA in existing benchmark with that in our proposed new benchmark.

 

* [15] M. C. D'Auria, A. Gimenez, and R. J. Sorensen. The role of the video in a video? _Journal of the American Statistical Association_, 2008:139-146, 2007.
* [16] T. K. B. Wilson. _The motion of videos_. PhD thesis, University of California, Berkeley, CA (1994).
* [17] P. W. Anderson. _The motion of videos_. Cambridge university press, 1999.
* [18] L. F. Abbott. _The motion of videos_. Princeton University Press, 1992.

[MISSING_PAGE_POST]

 



## Appendix F Additional DPO Results

Figure 17: Test Set Accuracy of the DPO Model vs. Training Epochs. The figure illustrates a consistent trend in both in-domain and out-of-domain video QA, with peak performance occurring at approximately epoch 2.5, equivalent to 350 training steps.

Figure 18: Comparison of DPO Model Performance: Ranker vs. Generator. The DPO model serves as a ranker, assigning reward scores to candidate answers generated by the SFT model with a temperature setting of 1.0. Employing the DPO model directly for answer generation results in superior performance compared to its use as a ranker.



Prompts for GPT-4V and ChatGPT Queries

[FIGURE:S6.F19] GPT-4V prompt for the generation of video summary, detailed caption and conversation generation. We only use detailed caption for experiments.

 

* [16] M. C. D'Auria, A. J. L. Readers, and R. S. H. Swain. The design of the design of a single-layer perceptron. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pages 235-240, 2007.
* [17] P. F. K. DeVore, G. E. Hinton, and I. Goodfellow. Learning from tiny images. In _Advances in neural information processing systems_, volume 89. MIT press, 2007.
* [18] Y. Bengio, V. LeCun, and O. Bengio. Generative adversarial nets for large data sets. In _NIPS conference on machine learning_, volume 1030. CVPR, 2007.

[MISSING_PAGE_POST]

 

Given the following inputs:

1. **Ground Truth Video Caption****: {caption}
2. **Question Related to the Caption****: {question}
3. **Grouped Truth Answer**: {answer}
4. **Model Predicted Answer**: {prediction}

Your task is to evaluate the model's predicted answer against the ground truth answer, based on the context provided by the video caption and the question. Consider the following criteria for evaluation:

- **Relevance**: Does the predicted answer directly address the question posed, considering the information provided in the video caption?

--Accuracy**: Compare the predicted answer to the ground truth answer. Does the prediction accurately reflect the information given in the ground truth answer without introducing factual inaccuracies?

- **Clarity**: Assess the clarity of the predicted answer. Look for issues such as repetition, unclear descriptions, or any grammatical errors that could hinder understanding.

- **Completeness***: Determine if the predicted answer fully covers the scope of the ground truth answer. Does it leave out critical information or does it include all necessary details?

**Output Format**: Explanation: <brief judgement of prediction> Score: <a integers score of quality from 1-5>

Figure 23: GPT-4V Evaluation Prompt for Video Question Answering. Together with video frames input in GPT-4V API, this prompt takes in a question, and model prediction, subsequently generating an assessment of the prediction’s quality alongside a corresponding score based on predefined criteria. A score value \(\geq 3\) will be considered correct for accuracy calculation. This is used to assess the quality of ChatGPT evaluation in fig. 22.

Figure 22: ChatGPT-Evaluation Prompt for Video Question Answering. This prompt takes in a detailed caption, question, ground truth answer, and model prediction, subsequently generating an assessment of the prediction’s quality alongside a corresponding score based on predefined criteria. A score value \(\geq 3\) will be considered correct for accuracy calculation.