VideoEspresso: A Large-Scale Chain-of-Thought Dataset for Fine-Grained
  Video Reasoning via Core Frame Selection
Authors: name: Songhao Han; name: Wei Huang; name: Hairong Shi; name: Le Zhuo; name: Xiu Su; name: Shifeng Zhang; name: Xu Zhou; name: Xiaojuan Qi; name: Yue Liao; name: Si Liu
Published: 2024-11-22 08:33:36

Abstract: The advancement of Large Vision Language Models (LVLMs) has significantly
improved multimodal understanding, yet challenges remain in video reasoning
tasks due to the scarcity of high-quality, large-scale datasets. Existing video
question-answering (VideoQA) datasets often rely on costly manual annotations
with insufficient granularity or automatic construction methods with redundant
frame-by-frame analysis, limiting their scalability and effectiveness for
complex reasoning. To address these challenges, we introduce VideoEspresso, a
novel dataset that features VideoQA pairs preserving essential spatial details
and temporal coherence, along with multimodal annotations of intermediate
reasoning steps. Our construction pipeline employs a semantic-aware method to
reduce redundancy, followed by generating QA pairs using GPT-4o. We further
develop video Chain-of-Thought (CoT) annotations to enrich reasoning processes,
guiding GPT-4o in extracting logical relationships from QA pairs and video
content. To exploit the potential of high-quality VideoQA pairs, we propose a
Hybrid LVLMs Collaboration framework, featuring a Frame Selector and a
two-stage instruction fine-tuned reasoning LVLM. This framework adaptively
selects core frames and performs CoT reasoning using multimodal evidence.
Evaluated on our proposed benchmark with 14 tasks against 9 popular LVLMs, our
method outperforms existing baselines on most tasks, demonstrating superior
video reasoning capabilities. Our code and dataset will be released at:
https://github.com/hshjerry/VideoEspresso