Direct Preference Optimization of Video Large Multimodal Models from
  Language Model Reward
Authors: name: Ruohong Zhang, name: Liangke Gui, name: Zhiqing Sun, name: Yihao Feng, name: Keyang Xu, name: Yuanhan Zhang, name: Di Fu, name: Chunyuan Li, name: Alexander Hauptmann, name: Yonatan Bisk, name: Yiming Yang

Preference modeling techniques, such as direct preference optimization (DPO),
has shown effective in enhancing the generalization abilities of large language
model (LLM). However, in tasks involving video instruction-following, providing
informative feedback, especially for detecting hallucinations in generated
responses, remains a significant challenge. Previous studies have explored
using large large multimodal models (LMMs) as reward models to guide preference
modeling, but their ability to accurately assess the factuality of generated
responses compared to corresponding videos has not been conclusively
established. This paper introduces a novel framework that utilizes detailed
video captions as a proxy of video content, enabling language models to
incorporate this information as supporting evidence for scoring video Question
Answering (QA) predictions. Our approach demonstrates robust alignment with
OpenAI GPT-4V model's reward mechanism, which directly takes video frames as
input. Furthermore, we show that applying this tailored reward through DPO
significantly improves the performance of video LMMs on video QA tasks.