VideoEspresso: A Large-Scale Chain-of-Thought Dataset for Fine-Grained
  Video Reasoning via Core Frame Selection
Authors: name: Songhao Han, name: Wei Huang, name: Hairong Shi, name: Le Zhuo, name: Xiu Su, name: Shifeng Zhang, name: Xu Zhou, name: Xiaojuan Qi, name: Yue Liao, name: Si Liu

The advancement of Large Vision Language Models (LVLMs) has significantly
improved multimodal understanding, yet challenges remain in video reasoning
tasks due to the scarcity of high-quality, large-scale datasets. Existing video
question-answering (VideoQA) datasets often rely on costly manual annotations
with insufficient granularity or automatic construction methods with redundant
frame-by-frame analysis, limiting their scalability and effectiveness for
complex reasoning. To address these challenges, we introduce VideoEspresso, a
novel dataset that features VideoQA pairs preserving essential spatial details
and temporal coherence, along with multimodal annotations of intermediate
reasoning steps. Our construction pipeline employs a semantic-aware method to
reduce redundancy, followed by generating QA pairs using GPT-4o. We further
develop video Chain-of-Thought (CoT) annotations to enrich reasoning processes,
guiding GPT-4o in extracting logical relationships from QA pairs and video
content. To exploit the potential of high-quality VideoQA pairs, we propose a
Hybrid LVLMs Collaboration framework, featuring a Frame Selector and a
two-stage instruction fine-tuned reasoning LVLM. This framework adaptively
selects core frames and performs CoT reasoning using multimodal evidence.
Evaluated on our proposed benchmark with 14 tasks against 9 popular LVLMs, our
method outperforms existing baselines on most tasks, demonstrating superior
video reasoning capabilities. Our code and dataset will be released at:
https://github.com/hshjerry/VideoEspresso