FrameThinker: Learning to Think with Long Videos via Multi-Turn Frame
  Spotlighting
Authors: name: Zefeng He, name: Xiaoye Qu, name: Yafu Li, name: Siyuan Huang, name: Daizong Liu, name: Yu Cheng

While Large Vision-Language Models (LVLMs) have achieved substantial progress
in video understanding, their application to long video reasoning is hindered
by uniform frame sampling and static textual reasoning, which are inefficient
and struggle to handle visually intensive video tasks. To overcome these
challenges, in this paper, we introduce the concept of thinking with long
videos and propose a novel framework FrameThinker. Within this framework, LVLMs
are able to iteratively interrogate video content. Developing such video
reasoning capabilities in LVLMs presents notable challenges, particularly in
adapting the model to new video actions (e.g. select frame), and designing
reward functions to guide LVLMs to adopt the newly introduced action. To solve
these challenges, we propose a two-phase training strategy, first employing
Supervised Fine-Tuning (SFT) to instill fundamental action capabilities,
followed by Reinforcement Learning (RL) to optimize a strategic decision-making
policy. Notably, in this RL phase, we conduct an in-depth and comprehensive
exploration of the reward design for each action and format reward. Extensive
experiments on reasoning benchmarks like Video-Holmes, LongVideo-Reason, and
long-video understanding benchmarks such as LongVideoBench, MLVU, VideoMME, and
LVBench, demonstrate that FrameThinker achieves a significant average
improvement of +10.4% over baselines while drastically reducing the number of
processed frames. Most notably, our 7B model, FrameThinker establishes a new
state-of-the-art on LongVideo-Reason, achieving 76.1% accuracy using an average
of only 20.6 frames. This not only outperforms the competitive LongVILA-R1
(72.0%) but does so with over 20x fewer frames (vs. 512), demonstrating
unparalleled efficiency and effectiveness.