VideoRFT: Incentivizing Video Reasoning Capability in MLLMs via Reinforced Fine-Tuning
Authors: Qi Wang
Yanrui Yu
Ye Yuan
Rui Mao
Tianfei Zhou
Venue: NeurIPS 2025 poster ()
Keywords: Multimodal Large Language Models
Video Reasoning
Reinforced fine-tuning
Primary area: deep_learning

Abstract: Reinforcement fine-tuning (RFT) has shown great promise in achieving humanlevel reasoning capabilities of Large Language Models (LLMs), and has recently been extended to MLLMs. Nevertheless, reasoning about videos, which is a fundamental aspect of human intelligence, remains a persistent challenge due to the complex logic, temporal and causal structures inherent in video data. To fill this gap, we propose VideoRFT, a novel approach that extends the RFT paradigm to cultivate human-like video reasoning capabilities in MLLMs. VideoRFT follows the standard two-stage scheme in RFT: supervised fine-tuning (SFT) with chain-of-thought (CoT) annotations, followed by reinforcement learning (RL) to improve generalization. A central challenge to achieve this in the video domain lies in the scarcity of large-scale, high-quality video CoT datasets. We address this by building a multi-expert-driven, cognition-inspired CoT curation pipeline. First, we devise a cognition-inspired prompting strategy to elicit a reasoning LLM to generate preliminary CoTs based solely on rich, structured, and literal representations of video content. Subsequently, these CoTs are revised by a MLLM conditioned on the actual video, ensuring visual consistency and reducing visual hallucinations. This pipeline results in two new datasets, i.e.VideoRFT-CoT-102K for SFT and VideoRFT-RL-310K for RL. To further strengthen the RL phase, we introduce a novel semantic-consistency reward that explicitly promotes the alignment between textual reasoning and visual evidence. This reward encourages the model to produce coherent, context-aware reasoning outputs grounded in visual input. Extensive experiments show that VideoRFT achieves state-of-the-art performance on six video reasoning benchmarks.

VIDEORFT: Incentivizing Video Reasoning Capability in MLLMs via Reinforced Fine- Tuning
Qi Wang!. Yanrui Yu?, Ye Yua Rui Mao Tianfei Zhou 2.3,*
Beijing Institute of Technology, Zhuhai "Beijing Institute of Technology State Key Laboratory of Environment Characteristics and Effects for Near-space SShenzhen University
httpe///ggttuubb.omm//aiaaaaa /
Abstract
Reinforcement fine-tuning (RFT) has shown great promise in achieving human- level reasoning capabilities of Large Language Models (LLMs), and has recently been extended to MLLMs. Nevertheless, reasoning about videos, which is fundamental aspect of human intelligence, remains a persistent challenge due to the complex logic, temporal and causal structures inherent in video data. To fill this gap, we propose VIDEORFT, a novel approach that extends the RFT paradigm cultivate human-like video reasoning capabilities in MILMs. VIDEORF'T follows the standard two-stage scheme in RFT: supervised fine-tuning (SFT) with thain-of-thoughe (CoT) annotations, followed by reinforcement learning (RL) improve generalization. A central challenge to achieve this in the video domain lies in the scarcity of large-scale, high-quality video CoT datasets. We address this by building a multi-exper ciil--expert-rrit -rrive,, cognition-inspired CoT curation pipeline. First, we devise cognition-inspiin
prompting strategy to elicit a reasoning LLM to generate preliminary CoTss based solely on rich, structured, and literal representations of video content. Subsequently, these Co'Ts are revised by a MILLM conditioned on the actual video, ensuring visual consistency and reducing visual hallucinations. This pipeline results in two new datasets, i.e. VideoRFT-CCTT-002 for SFT and VideoRFT-RL-33 -L-3-R 10K for RL. To further strengthen the RL phase, we introduce a novel cemanti--coniistency reward that explicitly promotes the alignment between textual reasoning and visual evidence. This reward encourages the model produce coherent, context-aware reasoning outputs grounded in visual input. Extensive experiments show that VIDEORFT achieves ofte--of-the-at performance on six video reasoning benchmarks.
Introduction
The ability to reason about complex videos lies at the core of human cognitive development [36]. Humans, even infants, exhibit a remarkable capacity to understand videos recognizing what has happened, inferring what will happen next, and explaining why events occur. Replicating this capability in AI systems has become a central goal in video understanding, and has been extensively studied in the field of computer vision over the past decade [4, 29, 47, 57]. Despite the progress, most Al models remain limited to perceptual- level understanding and struggle to reason about video content with the depth, efficiency, and interpretability that are characteristic of human cognition.
Recently, the rapid development of video MILMs, such as Video-ChatGPT [28], VILA [24], Video- LLaVA [23], has significantly advanced the capabilities of video understanding. However, these
"Coresponning Author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Video QA
Viel Baeck
Rew VQA Data
GRPO
Quodio:: Which booksshlll festured vilco!
Structured Video
Opulons:
Reprosentation
A. Travelling: Palming, C. Blacking
D. Holling
Video RFT
CognlliveCa Generation
Aa
Ax
Ground-Tratk: A
Cross-modaal CoT Refinement
(b) Pecformance Comparison
Reward Function
CoT from VideoRFT
<hink-To deternuze the bonketlell rule fosture in Be video, let's reveww the contest step- by-slep. The video begins wilh a blesks screen dapleyinng "KILSTTI BASKETBALL FIBA TRAV ELLING RULE CHANGES, indicating that the focus on the FIBA trovelling rule.. The video ther trectioons between Spein and the USA, showing a plaper from the USA dribking the bell. Given the options video clsarly focuses
CoT Dala
F2
Fx
Rio 2016 Diympic beckethall gime
MLLMS
Policy Model
travelling mentioee video. Theredoree correct ansurer A. wellinng stiik>
options
SFT Cold Start-....
Rule-based RL
canswarss qanewar>
(a) Illustration of CoT in VideoRFT
(c) Two-Stage Training Paradigm
Overview of VIDEORFT. (a) An example of CoT derived from VIDEORFT. (b) VIDEORFT achieves leading performance in six datasets. (c) The two-stage RFT underpins the training of VIDEORFT.
Figure 1:
models are predominantly answer-driven, i.e. they produce answers without explicitly revealing the reasoning process. VoT [8] overcomes this by introducing human- like reasoning framework that structures the task of video reasoning into five predefined steps using fixed templates. Nevertheless, such a rigid, template-based approach stands in contrast to the flexibility of human cognition, which enables adaptive reasoning processes based on perceptual inputs [2, 6].
In contrast, the very recent advancements, e.g., OpenAI-ol [17], DeepSeek-R1 [11], and Kimi- 1.5 [38], have shifted focus towards building LLMs that think before answering. These models show strong proficiency in interpreting complex problems, performing multi-step reasoning, and ultimately arriving at correct answers. A key enabler to such capabilities is reinforcement fine- tuning (RFT) [27], which typically commences with a warm-up phase of supervised fine-tuning using CoTs, and subsequently refines the model through reinforcement learning algorithms (e.g.. PPO [32], GRPO [34]). Beyond the language domain, pioneering efforts have extended RFT to MLLMs to enhance image-based capabilities [15, 26, 37, 44, 49, 54], and some works [9. 20, 51] that concurrently with ours, show the potential of RFT in the video domain. However, there is a critical challenge remaining unsolved: current video CoT datasets lack the complexity and granularity necessary for advanced video reasoning, which fundamentally limits the ability of models to emulate human-level cognitive capabilities. Moreover, how to ensure that reasoning outputs are faithfully grounded in visual evidence remains underexplored in these works.
Motivated by the above analysis, we propose VIDEORF"S a novel reinforcement fine-tuning frame- work to incentivize the video reasoning capability MLLMs (see Fig. 1). To overcome the scarcity of video CoTs, we develop scalable, cogmitively inspired pipeline that integrates multiple expert models to collaboratively construct high-quality video CoT datasets. Specifically, we first employ a specialized MLLM to extract structured textual descriptions from videos, capturing fine-grained vi- sual details. These descriptions are then processed by reasoning- capable L.M (e-g., DeepSeek-RR which generates initial CoTss through blind reasoning ceasoning-relying relyyng solely on textual input. However, due to the lack of direct visual grounding, such CoTs often contain inconsistencies and hallucinations [14]. To mitigate this issue, we introduce a cross-modal revision stage, wherein MLLM refines the initial CoT's by incorporating the original video, ensuring consistency with visual evidence. Based on this pipeline, we construct two large-scale datasets, i.e., VideoRFF-CCoT-1022 and VideoRFFTRRL-3100 which together support the RFT process in VIDEORFT.
Furthermore, to strengthen the RL phase, we develop a novel semantic-consistnncy reward that explicitly enhances the visual faithfulness of reasoning outputs in MLLMs. Our key observation is that the reasoning traces of MLLMs are typically structured into three consecutive parts: question parsing, video describing. and abstract reasoning. While the question parsing and abstract reasoning components are not necessarily grounded in the visual input, the video describing part should be closely aligned with the actual visual semantics. Based on this insight, our semantic-consistency reward measures the alignment between the token representations of the video description part and the visual features of the input video. This reward is integrated into the GRPO algorithm to guide MLLMs toward generating visually grounded outputs.

---

Contributions of this work. We propose VIDEORFT. novel framework that extends RFT to MLLMs so as to emulate human-like video reasoning capabilities. To achieve this, we first establish CoT foundation for video RFT by designing cognitively inspired pipeline to curate large-scale, high- quality video CoTT annotations. Furthermore, we introduce a novel semantic-consisteney reward to explicitly guide the reasoning trajectories of MILMs grounded in visual evidence, which enhances the effectiveness of RFT in cross-modal reasoning. Built on these contributions, VIDEORFT favorably outperforms advanced competitors series of challenging video reasoning benchmarks.
2
VIDEORFT CoT Dataset
We first present the construction of VideoRFT-COT and VideoRFT-RL. to support RFT in MLLMs.
2.1 Data Collection
extensively collect video question-answer
data covering diverse modalities, task types, and cognitive skills, Given the scarcity of
AnirOA mo.
HNCEOOS loce fo
PMC yor de
Paaedi Jac.
4 on
7ad//C lact 20
high-quality video data in certain domains
Mkagl orer,
(e.g., mathematics, science), we additionally incorporate carefully curated image-based in-
Spatintemparal
STAR pea, (19)
stances. The final dataset contains 310K sam-
inghtifon
Opedkaes
Video RFT
ples in total, supporting diverse answer for- mats, including multiple-choice (mc), numer- ical (num), free-form text generation (free), optical character recognition (ocr), and re- gression (reg). As shown in Fig. the
lngitfiiee 180
12%
14.20
Malbemalic
Mddzzda
[m0, TC
(Ten,
Digxown 36)
ASVV00 Lac.
Gecesty
ples are categorized into five groups accord- ing the type of cognitive skills involved in
Figure 2: The distribution of data collection.
the reasoning process:
General: Commonsense reasoning in open-domain temporal and causal contexts.
Mathematics: Symbolic reasoning and spatial alignment for multi-step logic tasks.
Science: Domain-specific reasoning in physics, chemistry, and medicine, emphasizing causal reasoning and conceptual abstraction.
Targets structured visual parsing and information extraction from complex layouts. Spatiotemporal. Involves motion prediction, spatial transformation, and relational reasoning.
Document:
2.2 Cognitively Inspired CoT Generation
To enable MILMs to acquire human-like reasoning abilities, it is essential to construct high-quality, cognitively grounded video CoT dataset. We propose an automated pipeline for generating such CoT data. As illustrated in Fig. 3, the pipeline comprises three major stages, and all the prompts used in the pipeline are provided in the supplementary material.
Structured Video Representation. For each video we generate semantically rich textual de- scriptions by prompting GPT-40-mini [16]. The prompt Peep carefully crafted guide the model to (i) summarize video content with high-level caption, and (ii) produce analytical, frame-level metadata for uniformly sampled video frames. Each frame is structured a predefined JSON schema that includes timestamped captions and key visual elements such as objects, actions, scenes, spatial relations, and potential interactions. We denote the structured representation of S..
Cognitively Inspired CoT Generation. Given the representation S, and corresponding question we invoke a LLM, e.g., DeepSeek-RI answer the question and extract its step-by-step reasoning outputs as the initial CoT, i.e., (oT(
CoT LLM(gg S., Poog).
(I)
Here, Pang denotes composite prompt consisting of five sub-prompts ie.,, Peng - Pt. Pa. Pu. pr) each guiding a distinct stage of the reasoning trajectory in a manner that mimics human cognitive processing. Concretely, these sub pooomtromp defined as follows. (1) Simulated observation prompt

---

Cress- modal Col Refinement
Structured Video Representation
Coguitive CoT Generation
Raw Data [- 310K)
Wht de penurs to the liptop Inform
Question
Resised Cal
Jying th balf
A. Opered B. Washed C. 1ialed up D. Closed
<t quskks uakina wist
vides bey
V Video Caption:
Promellent Metadataa:
<taiio-Te quetios addeg wlot. Fist, I teel to sieadle the sidee
Iniiia
adroad The ponse bas sldte sägzlr and ae up stlondc with lodas Gives Outt laptag withe from it wan spened batorn the video bepas. Canaloeigg optseii.... caciuiice the lagog noe openid befor the penzs the
plodi las eussioocsss postorr on hed
Capties A young ths hos ec Led.
dosctppiion.
proctted
Poop
idgets [sec, gilow, compuaac caloel.
Poos
place and threes it. afte Browlng the ylloe separteo to aboud actin
Porear
gagerient with computrr.
some betrotrr)
doot) accec)
[baiceg st
compiter
:ides hina does tae and adte.... so bedeee likkg
up The qunsion
videe
dowa. the
Linnntange 9M19215.
before the video
wayte the is i. tacted ot <thilio aaaaveent <immnes
Figure 3: Illustration of the pipeline for cognitively inspired CoT generation.
(p4): Instruct the model to simulate viewing the entire video and form an initial high-level understand- ing. (2) Task understanding prompt (pt): Encourage analysis of the question to infer the task type fact, fact, reason, causal relationship). (3) Selective focus prompt (p₂): Direct attention to specific temporal segments of the video relevant to g. (4) Visual reasoning prompt (p.): Ground the reasoning process in visual content, encouraging analysis over objects, actions, spatial-temporal relations, and event transitions. (5) Reflective answering prompt (p,): ): Guide the model to derive the final answer, optionally incorporating eeffvverification or reflection to ensure reasoning quality.
Cross-modal CoT Refinement. A key limitation in the initial CoTs is that they might suffer from visual hallucinations due to the lack of visual cues in Eq. 1. To resolve this issue, we introduce cross-modal refinement strategy to revise the CoT so that it aligns better with the actual video input. Specifically, we prompt a MILM,, i.e., Qwen2.5-VL. [1], to compare the initial CoT with the video v, identify inconsistencies, and perform necessary revisions:
COT, MLLM(u, CoT .Pooos)
(2)
Here the prompt Peress is designed to guide the MLLM to: (i) verify the cross-modal alignment of Co with the content of video v, (ii) localize and explain any visual-textual inconsistencies, and (iii) revise the CoT to enhance visual grounding while preserving its original logical structure.
Finally, we apply filtering stage ensure the factual correctness of the resulting CoT annotations. For structured tasks with clear ground-truth labels, we directly exclude samples with incorrect final answers. For open-ended tasks, we remove samples exhibiting low semantic consistency (measured
by CLIP [31]) between the generated answer and the reference answer. This filtering process ensures that the resulting CoT dataset maintains high quality and factual reliability. After filtering, 102K high- high-confidence samples are retained from the initial pool of 3 310K, forming VideoRFT-CoT--02K for supervised fine-tuning.
2.3 Data Analysis
helle
hhapen
ma in
scene
Fig. presents a comparative analysis of the CoT's in our proposed VideoRFT-CoT--ONl 102K and Video-RI [9]. As shown in Fig. 4 (a), the CoTs in our dataset exhibit broader distri- bution and longer average token length com- pared to those in Video-RI (Fig. 4(C)), indi- cating that our VideoRFT-CoTT-10X contains more elaborate, fine-graince and nuanced rea- soning processes. Additionally, the word cloud in Fig. 4 (b) reveals that CoTs in VideoRFT- CoT- are dominated by dynamic, video- centric concepts such as "video", "main", "hap- pen", and "first". The lexical profile reflects an emphasis on narrative structure and temporal progression, which are the key characteristics of complex video understanding. In contrast, Rl (Fig. 4 (d)) features frequent refer-
mage
Faee
ponson Greer
Rapyy
(a) Distribation tokens'
(b) Word cloud VideoRFT-CoT-
legth in VideoRFT-CoT-10XX
102K
Plane fer ad
eent
am
person
th
lated
plant show
(c) Distribation of CoT tokens' length in Video-RI
(d) Word cloud in Video.RI
Figure 4: Comparison of CoT dataset in VideoRFT-CoT- 102K and V1deo-R1.
4

---

Reference
Reward Evaluation
<think> The question asks for une of the symbols the festiall and the options provided video begias with a drummer followed by historical segment about St. highlights
Model
Format Reward
parade,
Policy Model
Completions
tradition drinking, To derarmine the correct ansserr. need recall the symbols associated with the festival, The </answer>
Accuracy Reward
GRPO
A7
Semantic Consistency
Policy Optimization
Reward
min(1.wr man(oos(tuun) 9) 0))
Inpet video
(a) bese- Reinforcement Leaming
(b) Comantic-Cnnsiseen Reward
Figure 5: Illustrations of (a) rule-based RL, and (b) the computation of semantic-cousistencyy reward R.. The reasoning outputs are color-coded highlight question parsing (green), video description (red) and abstract reasoning (bluc). Only the red part involved in the computation Rs (see $3.2).
ences to static declarative content, e-g., "diagram", "image", "plant", and "Earth", suggesting a stronger bias toward factual descriptions rather than deep reasoning. These results highlight that VideoRFFT-CoT-IO0 102K offers greater expressiveness in reasoning depth and aligns more closely with the demands of real-world video reasoning tasks. Hence, it provides better foundation for training video MLLMs with advanced reasoning capabilities.
102K
Video Reinforcement Fine-Tume Fine-Tuning
This section presents our approach for video reinforcement fine-tuning as shown in Fig. 5. We first provide brief overview of Group Relative Policy Optimization (GRPO) [34] in $3.1, and then elaborate on the proposed rule-based reward for efficient reinforcement fine-tuning in $3.2.
3.1 Group Relative Policy Optimization
GRPO [34] is computationally efficient rule-based RL algorithm designed specifically for training large reasoning models. Unlike traditional RL. methods such PPO [32], which require four models (policy, value, reward, and reference), GRPO simplifies the approach by eliminating the value model, significantly reducing memory requirements and training complexity. GRPO operates by generating K candidate responses {0r, ... ok for each query These responses are then evaluated using defined reward functions, yielding rewards {ri, ,Kk Afterwards, these rewards normalized to calculate the advantage A, for each response as:
A mean ((1,, T2,.,,, std( (ri, rg,...,
(3)
where mean and std denote the mean and standard deviation of the rewards, respectively. Subse- quently, the model is optimized through maximization of the following objective:
(G) (o.
(o.
K
min
A;, clip
- + A. (Toll (4)
where 8 denotes model parameters to be updated, and Te.. are the current and old policy model, Tee indicates the reference policy, is the KL divergence regularization coefficient, and e regularization coefficient that prevents the policy from deviating too far from the reference model.
3.2 Rule-based Reward Modeling in VIDEORFT
The rewards in Eq. are derived from rule- mule-based reward functions, which represent foundational step in rule-based RL by simply evaluating whether model predictions exactly match ground-truth answers. Two highly resilient rule-based rewards are the Format Reward and Accuracy Reward, which are consistently utilized DeepSeek-RI and its follow-ups. However, the context of cross-modal reasoning, these rewards are insufficient to provide explicit guidance to MILMs towards visually grounded reasoning. To address this limitation, we introduce a semantie-consistency reward, which enforces the grounding of generated reasoning content in the visual input.
5

---

3.2.1 Semantic-Consisteng Reward
The reward is motivated by the observation that the reasoning trace generated by video MLLMs typically consists of three distinct parts, i.e., question parsing, video describing and abstract reasoning. as shown in Fig. 5. Among them, the video describing stage represents the model's understanding of visual content, which is the foundation for subsequent reasoning. Therefore, the reward is designed to selectively promote alignment between this stage and the input video.
Formally, to isolate the video describing sentence from the generated response, we apply a regular expression to locate the first full stop. Empirically, the text following this full stop corresponds the model's interpretation visual content. From this point, we extract a fixed-length span M tokens, denoted []] and encode it using SigLIP [48]'s text encoder: M| SigLIP (f Additionally, we uniformly sample F frames {u(0), ,..., u(F-1)) from video u, and compute the visual representation of each frame (()) via SigLIP's image encoder: w(0) SigLlPimage (v(+).. Then the final video representation
naturally obtained by averaging the frame embeddings:
F Σ - (8)),
We then define the semantic-consisteney reward as:
(5) is a scaling constant. The max(:, () ensures non-negativity of the reward, while the min(-, 1) stabilizes training by bounding the reward. This stage-aware formulation allows us to reward only the part of reasoning tied to visual comprehension, without penalizing abstract reasoning that appropriately extends beyond the visual scope. The result enhanced semantic fidelity, reduced hallucinations, and improved alignment during RL.
Rs min (1, w x max (cos (4fr,4 M), .)).
where cos(:,) denotes cosine similarity, and
3.3 Overall Reward
VIDEORFT uses three types of rewards for RL:
Format Reward. During RL, we incorporate the widely-used format reward guide the model in generating its reasoning process and final answer in structured format. This reward, denoted as R,, ensures that the model's output adheres to a predefined structure: the reasoning process must be enclosed within Sthink>..<<thinnk tags, and answers within <answer>.. </answer>>>> tags. Compliance verified via regular expression matching, and a binary reward is assigned accordingly.
Accuracy Reward. To provide reliable supervision across heterogeneous tasks, we adopt task- specific accuracy metrics: Exact Match for multiple-choice and numerical questions, ROUGE for open-ended generation, Word Error Rate for OCR tasks, and scaled relative accuracy for regression problems. These tailored evaluations ensure the reward R. aligns with each task. Comaaniic-Cnnisstrr Reward. The semantic-consiteene alignment between the reasoning text and the input visual information.
reward R,, defined in Eq. 5, promotes
The overall reward R for a sample is computed as follows:
R= + R. + RR. > 0] R.,
(6)
where [R, > 0) is the indicator function that returns 1 if R2 > 0 and 0 otherwise. This indicator function acts as gate to ensure R, is activated only when R, is non-zero, thus avoiding the reinforcement of semantically plausible but factually incorrect reasoning.
4 Experiment
4.1 Experimental Setup
Benchmark and Metric. Following previous works [9, 18, 51], we evaluate our approach on six
video reasoning and understanding benchmarks: VSI-Bench [43]. VideoMMMU [13], MMVU [55]. MVBench [19], TempCompass [25], and VideoMME [10], covering spatial reasoning, knowledge- intensive video QA, temporal logic, and general video understanding. Following conventions, we only use the subset of multiple-choice samples in MMVU, and VideoMME, is evaluated without subtitles. Average accuracy is adopted as the evaluation metric.

---

Table I: Performance Comparison. The best results are highlighted bold. !; Results obtained using
larger input resolutions, up 768 x 28 and 768 sampled frames, while ours are 256 28 x 28 and 32.
Video Reasoning
Video Understanding
Model
Pub
VSL
VideoMMMU
MMVU
MV.
TempC. VideoMME
Proprictary Models
GPT-4o [16]
34.0
61.2
75.4
71.9
Open-Source Models
LLaMA-VID [21]
ECCV 24
41.9
45.6
ShareGPT4Viden [3]
NeurIPS 24
51.2
39.9
VideoLLaMA2 [5]
arXiv 24.06
44.8
54.6
47.9
LongVA-7B [50]
TMLR
29.2
23.9
56.9
52.6
VILA-115-BB [24]
CVPR 24
28.9
20.8
58.8
LLaVA-OneVisiion-72 [18]
TMLR 24
32.4
33.8
49.2
56.7
58.2
mPLUG-OW13-8B [46]
ICLR
54.5
53.5
Qwen2.5-VI-7B [1]
mrXiv25.00
31.8
47.4
61.3
59.4
69.2
52.8
Concurrent R1-based Models
Video-R1 [9]
arXiv 25.03
35.8
52.3
63.8
63.9
73.2
59.3
TinyLLaVA- Video-RI [51]
arXiv 25.04
46.9
49.5
46.6
VideoChat-RR R [20]
arXiv 25.04
67.9
VIDEORFT
51.1
68.5
62.1
73.7
59.8
Model Training. We follow the RFT to train VIDEORFT in two stages: the warm-up SFT stage and rule-based RL stage. Specifically, the SFT stage equips the model with the ability to generate correct responses for diverse questions, and is trained based on VideoRFT-CoT--022 using the reward in Eq. 6 to optimize structured reasoning and ensure factual validity. The RL training implemented using the HuggingFace TRL library [39], and our codebase built upon Open-R1 [7].
The rule-based
RL stage is based on VideoRFT-RL-3100
Implementation Details. We use Qwen2.5-VLL-78 [1] as the base model and train VIDEORFT on 8 NVIDIA A800 GPUs, with 80GB each. For efficiency, the video input is limited to 16 frames, with each frame processed into 128 x 28 x 28 resolution during training, where 28 x 28 is the size of each image patch, and 128 denotes the number of patches. During inference, we increase the number of frames to 32 and the resolution to 256 x 28 x 28. For efficiency, we use lightweight version of SigLIP with 400M parameters in computing the semanti--cossisency reward. The entire model trained for one epoch of SFT followed by IK steps of RL.
4.2 Main Result
As shown in Table 1, we compare VIDEORFT against variety of baselines, including proprietary models (i.e., GPT-4o [16]), Open-Source MILMMS (e.g., Qwen2.5-VL. [1], VILA [24], Long VA [50]), and contemporaneous models (e.g., Video-R1 [9], TmyLLaVA-Video-P [51]. VideoChat-RI [20]).
Several key observations can be drawn from the results. First, compared our base model, i.e.. Qwen2.5-VL--B. VIDEORFT achieves significant improvements across all six benchmarks, e.g., +5.0% on VSI-Bench, +7.2% on MMVU, and +7.0% on VideoMME. This demonstrates the effectiveness of our approach in incentivizing video reasoning capabilities in MLLMs. Moreover, VIDEORFT consistently outperforms all non-RL Open-Source MLLMs. Second, VIDEORFT surpasses the proprietary GPT-4o on VSI-Bench by +2.8 % highlighting the strong potential of RFT bridging the performance gap with closed-source models in the task of video reasoning. Third, when compared to contemporaneous works, our model delivers the best overall performance, ranking first on four out six benchmarks. This validates the superiority and generalization ability of our method in comparison with recent endeavors.
4.3 Diagnostic Experiment
To gain deeper insights into VIDEORFT, conduct set of diagnostic experiments, in Table 2.
7

---

Table 2: Diagnostic experiments for VIDEORFT.
Video Reasoning
Video Understanding
Model
VSI-Bench
VideoMMMU
MMVU
MVBench
TempCompass
VideoMME
Training Data
w/o CoT Refinement
34,5
48.1
64.8
58.3
72.4
52.8
Training Paradigm
SFT only
31.7
48.5
60.5
57.0
68.4
54.1
RL only
32.1
47.4
63.5
59.2
70.8
51.9
Reward Modeling
R = RI+-R. Rs
33.2
49.1
66.4
61.1
72.4
58.5
Ry+RR
34.6
50.2
65.2
61.4
73.9
56.3
Full Model
VIDEORFT
36.8
51.1
68.5
62.1
73.7
59,8
Quatior How many whilebourd(s) are in this room?
Ground T
Fiden-RF <think-To determine number whitchoards the room, review video step-hy-step.
The video begins with a view al conference zuom featuring long table, chains, and whitebourd on the right wall. The whilebourd cuntains handwrilten notes and disgram, and green cleth is visible it. The camera then pans show window with blinds and blue door, but additional whiteboards The camera then moves to whiteboord on the wall, which has more notes and diagr ams. This whitehoard is clearly visible and distinct from fins showing various The pans around same whitehoard, with camera focusing room. The video concludes with a view of the whilehboudd and the desk, with no additional whitchoards shows. So there are two whitehoard that appear in the video.
desk computer
the desk and whitchourd. but other whiteboards present.
dingrams.
whstceboars
Wait, Let double-check it, there are two whitebourds in room: one on the right and another wall near the desk, The video not show other hiitebuurss, so the answer two. answer 2tthiink>
Canswer- Sanswer-2<lnswweer
Figure Illustration reasoning traces derived from VIDEORFT in VSI-Bench.
Training Data. To assess the impact of the cross-modal refinement in Eq. on data quality, conduct SFT+RL using the data generated by Eq. 1, denoted as CoT("), resulting in a variant, i.e., without CoT Refinement. As shown in Table 2, this consistently leads to performance drops across all six benchmarks, i.e., -2.3% on VSI-Bench, -3.7% % on MMVU, and -7.0 % on VideoMME. This demonstrates that cross cross-modal refinement effectively mitigates errors and hallucinations in the initial CoTs, thereby enhancing data quality and ultimately improving model performance.
Training Paradigm. To validate the effectiveness of the RFT training paradigm in our approach, we build two baseline approaches: i.e., SFT only and RL only. The former trains the model solely with supervised fine-tuning on VideoRFFTCooT-OON -02K, while the latter, also known as the "zero" model in DeepSeek-RR R1, relies exclusively on RL without prior SFT. As seen from Table 2, the RL only surpasses the SFT only counterpart four out of six datasets, indicating the capability of RL in stimulating more generalized reasoning capabilities. When combining both stages as RFT, our full model VIDEORFT achieves the best results, substantially outperforming the two baselines across all datasets. This highlights the complementary strengths of SFT for stable initialization and RL for reasoning enhancement in tackling video reasoning.
Reward Modeling. We further examine the effect of the reward defined Eq. Specifically, we compare two ablated variants: the first uses only the Format Reward and Accuracy Reward (R R.). while the second incorporates all three rewards directly (R R RR, + ). seen from Table 2, adding the semantic-comm conittiecccssstennyy reward R, consistently improves performance over first variant, validating its effectiveness. Finally, our full reward formulation, which conditionally activates Rg via a gating mechanism (i.e., the indicator function I RR, 0)), achieves the best overall results. Notably, brings substantial gains especially for video reasoning benchmarks, í.e., +2.2% on VSI-Bench, +3.3% % on MMVU, and +3.5 % VidcoMME.

---

Aha Moment in VIDEORFT. Fig. 6 exhibits an Aha Moment in VIDEORFT, where it behaves in a numan-like manner by pausing to double-check its inference before finalizing the answer, as seen in the phrase "Wait, let me double-check it". Such behavior suggests that the model is not simply recalling learned patterns, but instead engaging in internal feedback loops to re-evaluate evidence and refine its inference.
5 Related Work
5.1 Multimodal Reasoning in MLLMs
Enabling reasoning in MLLMs has become central objective in recent research [22, 35, 52, 56]. In the image domain, early works such as MMCoT [52] and DDCoTT [56] disentangle perception and reasoning by treating visual understanding as input prompts for subsequent inference. In the video domain, VoT [8] and STEP [30] decompose video reasoning into predefined stages, employing template-basee based prompting to facilitate multi-step inference. DoraemonGPT [45] models video understanding through symbolic memory and external tool sequences, yet still follows a modular reasoning paradigm. While these methods offer structured supervision, their rigid designs often limit generalization across diverse temporal and causal scenarios. Recently, rule-based RI. has emerged as a promising paradigm for promoting multimodal reasoning in MIL.Ms. Pioneering efforts such as Visual-RFT [26], R1-VI [49], and Reason- [37] directly adapt rule-based RI. to image perception tasks. Follow-ups like Vision-RI RI [15] and RI-One Vision [44] further demonstrate images. Concurrently, this paradigm has also been explored for video understanding [9, 20, 51J. Despite encouraging progress, these methods face a fundamental bottleneck: the lack of large-scale, high-quality video CoT datasets, which limits the full potential of RFT in the video domain. Our work addresses this gap by proposing a scalable and cognitively inspired pipeline to automatically mine high-quality CoT annotations for videos. Beyond this, we introduce a novel reward modeling strategy based on cross cross-modal modal semantic consistency, which explicitly guides MILMs to generate visually grounded reasoning traces, and proves to be highly effective in improving model performance.
its effectiveness in enabling CoT reasoning
5.2 Multimodal CoT Dataset Construction
CoT has proven effective for enhancing the reasoning capabilities of LLMs by encouraging step-by- step reasoning [41, 53). Constructing high-quality CoT data in multimodal settings, particularly for video reasoning, remains a major challenge due the temporal complexity and visual ambiguity of video data [12, 33, 40J. Recent works have explored CoT construction in both image and video domains. LL-V- [42], Vision-RI [15]. and RI RI-OneVisionn [44] simply convert visual inputs to textual descriptions before reasoning. This often leads to hallucinations and weak semantic alignment. Video-R1 [9] adopts a simplistic prompting strategy that encourages MILMs to generate CoT by inserting "let me think", "wait", etc. into responses. However, such CoTss merely mimic the surface genuine reasoning. VideoEspresso [12] generates CoT data by prompting GPT-4o with a small set of selected key frames. Due to the sparse visual context and reliance on a text-only model, the generated CoTs often lack grounding in the actual video content and are prone to hallucinations. In contrast, our CoT data combines the reasoning abilities of reasoning LLMS and the multimodal abilities of MI.LMs, ensuring the reasoning depth and visual grounding of CoT data. Moreover, we use cognition-inspil inspired prompts to enable the reasoning model to generate CoT data that is more in line with human cognition.
form of human thinking without engaging
6 Conclusion
In this work, we introduce VIDEORFT, a novel approach for incentivizing cognitive video reasoning capabilities in MILMs through reinforced fine-tuning. To accomplish this, we propose cross-modal pipeline that generates high-quality cognitive video CoT data simulating human reasoning processes, resulting in two large-scale datasets: VideoRFT-CoTT 102K and VideoRFT-RL-310K, Furthermore to strengthen the RL phase, we develop semantic-consisseneyy guided reward to explicitly encourage the alignment between reasoning traces and visual evidence. Extensive experiments across six benchmarks demonstrate that VIDEORFT consistently surpasses a variety of advanced MILMMs. We expect this work to lay foundation for future efforts in RFT-based video reasoning.

---

Acknowledgments
This work is supported by the NSFC (Grant Nos. 62576035, 62225203, 62532007), the National Key R&D Program of China (Grant No. 2022YFB2702100 100), Beijing Natural Science Foundation (L252036), CAAI-Lenovo Blue Sky Research Fund, the Beijing Municipal Science and Technology Commission and Zhongguancun Science Park Management Committee (Z231 100007433003)
References
[1] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Siho Song, Kai Dang, Peng Wang. Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report. arXiv preprint arXiv:2502. 13923, 2025.
[2]
Lawrence Barsalou. Grounded cognition. Annu. Rev. Psychol., 59(1):117-645, 2008.
[3] Lin Chen, Xilin Wei, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Zhenyu Tang. Li Yuan, et al. Sharegpe4video: Improving video understanding and generation with better captions. In NewrIPS, 2024.
[4] Tieyuan Chen, Huabin Liu, Tianyao Hc, Yihang Chen, Chaofan Gan, Xiao Ma, Cheng Zhong, Yang Zhang, Yingxue Wang, Hui Lin, et al. Meed: Unlocking multi-event causal discovery in vidco reasoning. In NeurIPS, 2024.
[5] Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, al. Videollama 2: Advancing spatial-temporal modeling and audio understanding in video-Ilms. arXiv preprint arXiv:2406.044: 2024.
[6] Jonathan St BT Evans and Keith E Stanovich. Dual-process theories of higher cognition: Advancing the debate. Perspectives on Psychological Science, 8(3):223-24) 2013.
[7] Hugging Face. Open rl: A fully open reproduction of deepseek-rl. https//gitubb.co//uuggggggggii 2025.
[8] Hao Fei, Shengqiong Wu, Wei Ji, Hanwang Zhang, Mcishan Zhang, Mong Li Lee, and Wynne Hsu. thhho-hf-thought: Step-by-step vidco reasoning from perception cognition. In ICML, 2024.
[9] Kaituo Feng. Kaixiong Gong, Bohao Li, Zonghao Guo, Yibing Wang, Tianshuo Peng. Benyou Wang, and Xiangyu Yue. Video-rl: Reinforcing video reasoning milmss. arXiv preprint arXiv:2503.21770 2025.
[10] Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li. Shuhuai Ren, Renrui Zhang. Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi- modal lmss in video analysis. In CVPR, 2025.
[11] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang. Xiao Bi, et al. Decpseek-rl: Incentivizing reasoning capability in llms via reinforcement learning, arXiv preprint arXiv:2501. 12948, 2025.
[12] Songhao Han, Wei Huang, Hairong Shi, Le Zhuo, Xiu Su. Shifeng Zhang, Xu Zhou, Xiaojuan Qi. Yue Liao, and Si Liu. Videoespresso: A large-scale chain-of-thouge dataset for fine-grained video reasoning via core frame selection. In CVPR, 2025.
[13] Kairui Hu, Penghao Wu, Fanyi Pu, Wang Xiao, Yuanhan Zhang, Xiang Yue, Bo Li, and Ziwei Liu. Video-mmmut Evaluating knowledge acquisition from musti-diseipline professional videos. arXiv preprint arXiv:2501133826 2025.
[14] Lei Huang, Weijiang Yu. Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng. Xiaocheng Feng, Bing Qin, al. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. ACM Trans. Inf. Syst., 43(2):1-5) 55, 2025.
[15] Wenxuan Huang, Bohan Jia. Zijie Zhai, Shaosheng Cao. Zheyu Ye, Fei Zhao, Xu, Yao Hu, and Shaohui Lin. Vision-rl: Incentivizing reasoning capability in multimodal large language models. arXiv preprint ar 21v:22503006745 2025.
[16] Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Go system card, arXiv preprint 2100:22400 2024.
10

---

[17] Aaron Jaech, Adam Kalai, Adam Lerer. Adam Richardson. Ahmed El-Kishky. Aiden Low, Alec Hel- yar, Aleksander Madry, Alex Beutel, Alex Camey, et al. Openai ol system card. arXiv preprint arXiv:2412.16720 2024.
[18] Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Ii, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer. Trans. Mach. Learn. Res., 2024.
[19] Kunchang Li, Yali Wang, Yinan Hc, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. Mybench: A comprehensive multi-modal video understanding benchmark. In CVPR, 2024.
[20] Xinhao Li, Ziang Yan, Desen Meng, Lu Dong, Xiangyu Zeng, Yinan He, Yali Wang, Yu Qiao, Yi Wang, and Limin Wang. Videochat-rl: Enhancing spatio-temporal perception via reinforcement fine-tuning. arXiv preprint (rXX::250400698. 2025.
[21] Yanwei Li, Chengyao Wang, and Jiaya Jia. Llama-vid: An image is worth 2 tokens in large language models. In ECCV, 2024.
[22] Zhiyuan Li, Dongnan Liu, Chaoyi Zhang, Heng Wang. Tengfei Xue, and Weidong Cai. Enhancing advanced visual reasoning ability large language models. EMNLP,, 2024.
[23] Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection. In EMNLP, 2024.
[24] Lin, Hongxu Yin, Wei Ping. Pavlo Molchanov, Mohammad Shoeybi, and Song Han. Vila: On pre-training for visual language models. In CVPR. 2024.
[25] Yuanxin Liu, Shicheng Li, Yi Liu, Yuxiang Wang, Shuhuai Ren, Lei Li. Sishuo Chen, Xu Sun, and Lu Hou. Tempcompass: Do video ilms really understand vidcos? In ACL, 2024.
[26] Ziyu Liu, Zeyi Sun, Yuhang 7 Xiaoyi Dong, Yuhang Cao, Haodong Duan, Dahua Lin, and Jiaqi Wang. Visual-rft: Visual reinforcement finc-tuning. In ICCV, 2025.
[27] Trung Quoe Luong, Xinbo Zhang, Zhanming Jie, Peng Sun, Xiaoran Jin, and Hang Li. ReFT: Reasoning with reinforced fine-tuning, In ACL, 2024.
[28] Muhammad Maaz. Hanoona Rasheed. Salman Khan, and Fahad Shahbaz Khan. Video-chatgpe Towards detailed vidco understanding via large vision and language models. In ACI., 2024.
[29] Juhong Min, Shyamal Buch, Arsha Nagrani, Minsu Cho, and Cordelia Schmid. Morevqa: Exploring modular reasoning models for video question answering. In CVPR, 2024.
[30] Haiyi Qiu, Minghe Gao, Long Qian, Kaihang Pan, Qifan Yu, Juncheng Li, Wenjie Wang, Siliang Tang, Yueting Zhuang, and Tat-Seng Chua. Step: Enhancing video-llms' compositional reasoning by spatio- temporal graph-guided self-training In CVPR, 2025.
[31] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In JCML, 2021.
[32] John Schulman, Filip Wolski, Prafulla Dhariwal, Alee Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707.0077..0644/ 2017.
[33] Hao Shao, Shengju Qian, Han Xiao, Guanglu Song, Zhuofan Zong, Letian Wang, Yu Liu, and Hongsheng Li. Visual cot: Advancing modal language models with a comprehensive dataset and benchmark for chain-of- thought reasoning. In NeurIPS, 2024.
[34] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. Xiv preprint arXiv:2402.03300 2024.
[35] Yudi Shi, Shangzhe Di, Qirui Chen, and Weidi Xie. Unlocking video-llm via agent-of-thoughts distillation. arXiv preprint arXiv:2412.0066 2024.
[36] Elizabeth S Spelke and Katherine D Kinzler. Core knowledge. Developmental science, 10(1):89-96, 2007.
[37] Huajie Tan, Yuheng Ji. Xiaoshuai Hao, Minglan Lin, Pengwei Wang, Zhongyuan Wang. and Shanghang Zhang. Reason-rft: Reinforcement fine-tuning for visual reasoning. arXiv preprint arXiv:2503.2077 2025.
11

---

[38] Kimi Team, Angang Du, Bofei Gao, Bowei Xing. Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi k1.5: Scaling reinforcement learning with Ilss. arXiv preprint arXiv:2501.12590 2025.
[39] Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching. Tristan Thrush, Nathan Lam- bert, Shengyi Huang, Kashif Rasul, and Quentin Gallouédec. Trl: Transformer reinforcement learning. https///gitabboomm/ggggigggggete 2020.
[40] Yan Wang. Yawen Zeng, Jingsheng Zheng, Xiaofen Xing, Jin Xu, and Xiangmin Xu. Videocot: A video chain-of thoughh dataset with active annotation tool. arXiv preprint arXiv:2407.05335 2024.
[41] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Deany Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. In NewrIPS, 2022.
[42] Guowei Xu, Peng Jin, Li Hao, Yibing Song, Lichao Sun, and Li Yuan. Llava-cot: Let vision language models reason step-by-step. arXiv preprint arXiv:2411. 10440, 2024.
[43] Jihan Yang. Shusheng Yang. Anjali W Gupta, Rilyn Han, Li Fei-Fei, and Saining Xie, Thinking in space: How multimodal large language models see, remember, and recall spaces. In CVPR, 2025.
[44] Yi Yang. Xiaoxuan He, Hongkun Pan, Xiyan Jiang, Yan Deng, Xingtao Yang, Haoyu Lu, Dacheng Yin. Fengyun Rao, Minfeng Zhu, et al. Rl-onevision: Advancing generalized multimodal reasoning through cross cross-modal formalization. arXiv preprint arXiv:2503. 10615, 2025.
[45] Zongxin Yang, Guikun Chen, Xiaodi Li, Wenguan Wang, and Yi Yang. Doraemongpt: Toward understand- ing dynamic scenes with large language models (exemplified as a video agent). In ICML, 2024.
[46] Jiabo Ye, Haiyang Xu, Haowei Liu, Anwen Hu, Ming Yan, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. mplug-ow13: Towards long sequenee understanding in multi-modal large language models. arXiv preprint arXiv:2408. 04840, 2024.
[47] Kexin Yi, Chuang Can, Yunzhu Li, Pushmeet Kohli, Jiajun Wu, Antonio Torralba, and Joshua B Tenenbaum. Clevrer: Collision events for video representation and reasoning. In ICLR. 2020.
[48] Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In CVPR, 2023.
[49] Jingyi Zhang, Jiaxing Huang. Huanjin Yao, Shunyu Liu, Xikun Zhang, Shijian Lu, and Dacheng Tao. RI-vI: Learning reason with multimodal large language models via step-wise group relative policy optimization. arXiv preprint arXiv:2503. 12937, 2025.
[50] Peiyuan Zhang, Kaichen Zhang. Bo Li, Guangtao Zeng, Jingkang Yang, Yuanhan Zhang, Ziyue Wang. Haoran Tan, Chunyuan Li. and Ziwei Liu. Long context transfer from language to vision. Trans. Mach. Learn. Res., 2024.
[51] Xingjian Zhang, Siwei Wen, Wenjun Wu, and Lei Huang. Tinyllva--iideo-r:: Towards smaller Imms for video reasoning. arviv preprint arXiv:2504.066/ 09641,, 2025.
[52] Zhuosheng Zhang, Aston Zhang, Mu Li, hai zhao, George Karypis, and Alex Smola. Multimodal chain-of- thought reasoning in language models. Trans. Mach. Learn. Res., 2024.
[53] Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola, Automatic chain of thought prompting in large language models. In ICLR, 2023.
[54] Jiaxing Zhao, Xihan Wei, and Liefeng Bo. RI-ommi: Explainable omni-multimodal emotion recognition with reinforcement learning. arXiv preprint arXiv:2503.0537) 2025.
[55] Yilun Zhao, Haowei Zhang, Lujing Xie, Tongyan Hu, Guo Gan, Yitao Long, Zhiyuan Hu, Weiyuan Chen, Chuhan Li. Zhijian Xu, et al. Mmvu: Measuring expert-level multi-discipline video understanding. In CVPR, 2025.
[56] Ge Zheng, Bin Yang, Jiajin Tang, Hong- Yu Zhou, and Sibei Yang. Ddcot: Duty-distinct chain-of-thought prompting for multimodal reasoning in language models. In NewrIPS, 2023.
[57] Bolei Zhou, Alex Andonian, Aude Oliva, and Antonio Torralba. Temporal relational reasoning in videos. In ECCV, 2018
12

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction clearly state the main contributions, which are supported throughout the paper, especially in Sections 3 and 4.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Our CoT data generation relies on existing reasoning LLMs and MLLMs, which may introduce biases or hallucinations, and the training requires substantial computa- tional resources.
Justification:
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and prool's
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
13

---

Answer:
[NA]
Justification:
This paper does not contain formal theoretical results or proofs
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Section 5 provides implementation details including datasets, model architec- ture, hardware used, and evaluation protocol. And the code, data, and model weights will be open- open-sourced in the future.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
14

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The constructed datasets and training code with documentation will be open- sourced upon paper acceptance
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cr
Doglii/ggides/Codeeeee
cwtemmiisssnooliiiioeooo onPolicy) for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/ /nips. wctpublic//aeccccccccccccccc /gwii/gg///////eee /Cae//ooeubbiiic Sestsubmissiooos ssionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Section 5.1 describes benchmarks, metrics, and model training settings.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Error bars and statistical are not currently reported due time and computational resource constraints, but multiple runs and more robust metrics could be included in the camera-ready version.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
15

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Section 5.1 mentions training was done on 8 A800 GPUs (80GB) and specifies input resolutions and training steps.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't fidn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https /t////nurris.pp/ wwweeuris.ccpuubbl.eeee cc/public///pcpcctcccccccccccccc http//uullc//thieeeeeeeeeeeeeeee Eette/ csGuidelinest
Answer: [Yes]
Justification: Our work complies with the NeurIPS Code of Ethics. No human subjects sensitive data were involved, and we avoid harmful generation or misuse.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The method benefits video understanding but may risk misuse in misinforma- tion surveillance if deployed irresponsibly.
Guidelines:
16

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negatíve societal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (c.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: We will release the models and datasets under licenses with usage guidelines. Video-based CoT data will be filtered to avoid unsafe content.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All models and datasets used are cited appropriately with license and source attribution, including SIGLIP, Qwen2.5-VL, gpt-4o-mini, and DeepSeek-RI.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
17

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
If assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswithcee thcode. com/ conddataset has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We will release two new datasets along with structured documentation includ- ing data format, source description, and limitations.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Our research does involve human subjects crowdsourcing. All data derived from public or synthetic video sources.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: No IRB approval is required our work does not involve human participants.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
18

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [Yes]
Justification: LLMs are used as core components our research in two ways: (1) DeepSeek- RI and GPT-4o-mini used in our video CoT data generation pipeline, and (2) Qwen2.5-VL.c as our base model for training. We will include the detailed prompts in the supplementary materials.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMMs as any important, original, non non-standard components.
Please refer to our LLM policy (https / //t/rrise..////// htt////ffeencee///eeeeeeeeeaeeee /0l.//eeee//////// /LLM) for what should or should not be described.
19

---

A
Additional Experiments
A.1
Effeet of Model Scaling on Reasoning Ability
Table 3: Performance Comparison with Small-scale Models
Video Reasoning
Video Understanding
Model
VSL
VideoMMMU
MMVU
MV.
TempC.
VideoMME
VanyLLLVA-Vioo---- RI-38
46.9
49.5
46.6
VIDEORFT-31 32.5
41.1
55.1
59.5
61.0
45.4
VIDEORFT-T7 36.8
51.1
68.5
62.1
73.7
59.8
To further assess the scalability and robustness of our method under limited computational resources, we train an additional 3B variant of our model. While its performance is naturally lower than the original 7B version, the 3B model still performs competitively across benchmarks. Crucially, when compared to TinyLLa VA- WVA-Vide-RR--BB for lightweight deployment, our 3B variant outperforms it, with gains of +8.2% on MMVU and +11.5% on TempCompass. This significant margin confirms the competitiveness of our approach even in small-scale settings, and underscores its strong generalization and reasoning abilities under resource-constrained conditions.
contemporary 3B model specifically optimized
A.2 Hyperparameter Analysis
Table 4: Hyperparameter experiments for VIDEORFT.
Video Reasoning
Video Understanding
Hyperparameter
VSI-Bench
VideoMMMU
MMVU
MVBench
TempCompass
VideoMME
1
34.2
49.2
67.9
62.6
73.5
61.1
w 2
36.8
51.1
68.5
62.1
73.7
59.8
w = 3
35.6
50.1
67.9
60.8
72.3
60.4
W 4
35,4
49.7
67.6
62.0
73.1
59.1
To evaluate the effect of the scaling factor in the semantic reward, we conducted a hyperparameter study across multiple benchmarks. As shown in Table 4, the performance varies with different values of w. with w-2 consistently achieving the best or near-best results across all benchmarks. This suggests that w=2 provides a favorable balance between semantic alignment and factual precision. In comparison, both smaller (w=1) and larger (w=3 w=4) values lead to slight decline in performance, implying that insufficient or excessive emphasis on semantic similarity may hinder the overall reasoning and understanding capabilities. These findings highlight that while performance is relatively stable across a range of scaling values, setting w=2 consistently yields optimal results, demonstrating the effectiveness of moderate semantic weighting.
EXSLANED BY IRISH RAIDERS
1903:
40GB CRRM GREEN
34 WELiON
SED NATIONAL HOLIDAY
IRISH ARLESTO
IRelAnd
PAEAUT
Question: What is special abous celebntion York according B. Dressing in green and dycing the river to green.
Options A. Hosting large parades. Ground Trudi: A
C. Drinking a lot.
D. Planting shamooks..
VibroRFT::
<thinesiao determine the suonal of New York milobmtionn let's revtew the video content. The vidco begins with drummer and trancitons segment
Patrick, including enstavemnnt comeession holiday in Ireannd 1903. The video higilights the parade in Boston 1737 and mentioes the Chicago River bern dyed green in 1962. also notes that New York has the Jarpest parade, with 2 milloon spectalons.
celctration St. Patnck's Day, noting became
The question specifiaall asks about New York's celetestion. The video statess that New York has the largest parads, with millios spectalors However, also green,
montions Chicago options, (large parades)
grewn, which signlficant (dyeing river green) are mentioned.
The
explicily
New York dyes
Chicago
Howerer if we comider the becocer coneette of colehmrpon, the video mentons that Chicage dyes its river green, which is sigmificant event. Walt, but the question is asking about a umqque special aspect of New York's teicheation, and since the video does not explicttly state that New York dyes its river green, the safest answer based on the providd information A (large paredes).
Thersfore, answeris Darge paraies), <lliinks canswrrs <jrsser>
Figure 7: Illustration of reasoning traces derived from VIDEORFT in VideoMME.
20

---

A.3 Aha Moment in VIDEORFT
Figure illustrates additional instances the "Aha Moment" behavior observed in VIDEORFT. Prior to reaching a final decision, the model exhibits human-like tendency to pause and scrutinize its intermediate reasoning steps. This reflective process, indicative of deliberative reasoning, is marked in red within the figure.
B Limitations and Future Directions
We consider this work a strong foundation for advancing video reasoning research using MILMs. Several avenues warrant further investigation in future studies:
Handling of Challenging Video Scenarios. While our model demonstrates strong perfor- mance across various benchmark tasks, its effectiveness may be affected under complex conditions such as rapid motion or severe visual occlusion. Incorporating finer-grained visual signals and higher frame-rate sampling may help mitigate these challenges, enabling more accurate and robust visual representations.
CoT Data Reliance. The quality of the generated CoT annotations is closely tied to the capabilities of the underlying reasoning language model (e.g.. DeepSeek-R1), Future work could explore leveraging more advanced reasoning models to further enhance the quality of CoT data, potentially leading improved performance of VIDEORFT.
C Potential Social Impacts
Positive Impacts. Enhanced video understanding enabled by VIDEORFT can benefit vari- ous applications. In education, it facilitates the development of intelligent tutoring systems using video content. In security, it improves the efficiency and accuracy of surveillance video analysis. Moreover, supports content moderation by aiding in the detection and filtering of inappropriate material.
Negative Impacts. There exists risk of misuse. Misinterpretation of video content due to over-reliance on automated analysis could lead the spread of misinformation. Additionally, in surveillance scenarios, the deployment of such systems may raise concerns regarding privacy and ethical use.
D Detailed Prompt Used in CoT Generation
Here, we provide detailed prompts for each step of the VIDEORFT CoT generation process.
D.1 Video CoT Generation
Structured Video Representation: For all videos, sample them at 1FPS and input them into GPT-4o-mini, and generate a structured representation according to the prompt Prep
Structured Video Representation
System Prompt:
You video analysis assistant designed to produce rich, analytical per-frame captions from video inputs.
User Prompt:
<Input Video>
</ /Input Video>
Task:
1. Overall Video Caption:
A concise 1-2 sentence (20-30 words) summary capturing the main theme or action of the video.
21

---

Structured Video Representation (Continued table)
2. Per-Frame Metadata:
Uniformly sample frames and wrap them into a JSON list. Each element must include:
timestamp: "HH: MM: SS", aligned with the sampling interval.
caption: 2-3 sentences (30-50 words) describing the scene in detail.
eey_eeemens:: an object with fields:
objects: list of detected objects/entities (strings)
actions: list of ongoing actions or movements (strings)
Scene: e.g. "kitchen", "urban street"
notable features: list of distinctive colors, textures, or patterns (strings)
spatial_relationss
relations: list of spatial relationships (e.g. "cup on table", "person left of car")
human attributes: object mull. If present, include:
gender: "male", "female", "unknown"
clothing: brief description
posture: "standing". "sitting" pooteniia_interactionns list of possible interactions (strings)
General Instructions:
Temporal Consistency: Reference continuing actions from the previous frame and highlight any changes.
Uncertainty: If confidence=00.6 or object visibility<50%% append [Uncertain] to the caption.
Implied Actions:
Describe preparatory movements (e.g. "hand reaching toward door handle" vs. "holding door handle").
Output Requirements: Wrap all per-frame objects into a single JSON list matching the number of sampled frames.
Cognitively Inspired CoT Generation: We invoke DeepSeek-R1 to answer the question and extracts its step-by-step reasoning outputs with the prompt Poog as the initial CoT.
Cognitively Inspired CoT Generation
System Prompt:
You are an AI assistant helping a user answer questions about a video. When the user asks a question, you respond by imagining you watching the video with full attention, just like human would. Your task is to reason visually and logically about the video content to answer the user's question.
Follow this multi-step reasoning approach:
1. Simulate Browsing the Video: Imagine you watching the entire video from beginning to end. Build a general sense of what is happening.
2. Understand the Question: Reflect on what the user asking. Think carefully about what kind of answer is needed (e.g., fact, reason, comparison).
3. Localize Relevant Moments: Consider which parts of the video are most related to the question. Focus those segments in your mental replay.
4. Visual Reasoning: Describe what you "see" in those segments using natural visual language (e.g., "The video shows... " "In the second half of the video... Analyze and interpret the visual content build your answer.
5. Answer Thoughtfully: Provide a clear and direct answer. Ensure your reasoning is consistent with the visual events you described.
Guidelines for Responses:
22

---

Cognitively Inspired CoT Generation Continued
answering based text information. Use statements imitating watching a video to answer.
Don't expose in the output that
Don't directly refer any textual metadata such as "captions", "description", "frame-leve metadata", "key elements", etc. If you need to mention them, use "visual evidence" instead (e.g.. "the video shows...")
It's okay double-check question yourself during the thought process reflect naturally as a human would.
Refer to moments time using broad expressions like: "at the beginning of the video", "around the middle", "toward the end".
User Prompt:
Video content: <Overall Video Caption>
Frame-level metadata: <Per-Frame Metadata>
Question: <Question>
Please think about this question as if you were a human pondering deeply. It's encouraged to include self-reflection or verification in the reasoning process.
<Corresponding Answer Format Template>
The <Corresponding Answer Format Template> in the prompt is dynamically selected from the following templates based on the question type:
Table 5: Answer Format Templates for Different Question Types
Question Type
Template
Multiple Choice
Please provide only the single option letter (e.g.. A, B, D, etc.) within the <answer> </answer> tags.
Numerical
Please provide the numerical value within the <answer> < </answer> tags.
OCR
Please transcribe text from the image/video clearly and provide your text answer within the <answer> </answer> tags.
Free-form
Please provide your text answer within the <answer> </answer> tags.
Regression
Please provide the numerical value within the <answer> </answer> tags.
We employ cross cross-modal refinement process to ensure the CoT aligns with the video content, using the prompt Penes:
Cross-modal CoT Refinement:
Cross-modal CoT Refinement
System Prompt:
You are a multimodal reasoning expert. Your task is to revise hallucinations and errors in the chain-of-thought (CoT) based the visual content of the provided video. Do not significantly alter the original CoT logic or content, and ensure the final conclusion remains the same.
Your task:
1. Carefully examine the video, the question, and the COT.
2. Identify only the reasoning steps that directly conflict with what is visually shown in the video:
Replace any text-based references with direct visual observations
Use visual phrasing such as "The video shows...". "I can see..." "From the visual sequence...".
Replace specific timestamps with broader temporal phrases.
Do not rewrite steps that are already consistent with the visual content.
Only replace correct parts that visually contradict what is shown.
23

---

Cross-modal CoT Refinement Continued
3. Ensure the rest of the CoT stays faithful to the original meaning.
User Prompt:
<Input Video> </Input Video>
Question: <Question>
Original CoT: <Original CoT>
Output format:
Strictly follow this format. Return only the revised CoT and additional explanation:
<think> [Revised CoT] </think>
D.2 Image CoT Generation
In addition to the main video data in the VideoRFT-Co11-022K, we have also designed specific prompts specifically for image data, and used the similarly CoT generation process to generate CoT.
Structured Image Representation: Due to the differences among different image datasets (the task focuses the datasets are different), we designed different prompts for each image dataset in structured image representation phase.
Common System Prompt for All Image Representations
You are vision-language expert. Your task is analyze the provided image and output detailed, modular description in JSON format.
The following prompts used for different image datasets, all sharing the above system prompt:
General Image Representation (A-OKVQA, ShareGPT4V)
User Prompt:
<Input Image> You are given a natural photograph. Output only valid JSON with two keys:
</Input Image>
1. Overall Image Caption: 2-3 sentence narrative describing scene, objects, actions, and context.
2. Image Metadata: an object containing:
objects: list objects with id, type, color, size, and bbox
text: list of text elements with content and bbox
scene_context: environment and activity
relations: list of pubjecttpredicatc--bjeee relations
CLEVR Image Representation
User Prompt:
<Input Image> </Input Image>
You are given a synthetic 3D scene. Output only valid JSON with:
1. Overall Image Caption: a brief summary of number of objects and overall layout.
2. Image Metadata:
objects: list of objects with id, shape, color, size, material, and coordinates
spatial relations: list of spatial relations between objects
spatial_relations
24

---

STEM Image Representation (Geometry3K, UniGeo, A12D)
User Prompt:
<Input Image> </Input Image>
You are given dine-drawing or diagram. Output only valid JSON with:
1. Overall Image Caption: a concise paragraph describing the diagram's purpose.
2. Image Metadata:
primitives: list geometric primitives with type, label, and bbox
annotations: list of relations between primitives
measurements: list of measurements with primitive IDs and values
OCR Image Representation (TextVQA, HME100K)
User Prompt:
<Input Image> </Input Image>
You are given an image containing printed or handwritten text. Output only valid ISON with:
1. Overall Image Caption: sentence summarizing the text context.
2. Image Metadata:
text_itee items: list of text elements with content, bbox, and style
Science Image Representation (ScienceQA, PMC-VQA. ArxivQA)
User Prompt:
<Input Image> </Input Image>
You are given munlti-panel scientific figure. Output only valid JSON with:
1. Overall Image Caption: a paragraph overviewing the figure's subject.
2.
Image Metadata:
panels: list of panels with elements and process arrows
Chart Image Representation (DVQA, PlotQA, FigureQA)
User Prompt:
<Input Image> </Input Image>
You are given chart image. Output only valid ISON with:
1. Overall Image Caption: a 1-2 sentence summary of chart type and key trend.
2. Image Metadata:
chart_type type of chart
axes: list of axis information
series: list of data series
legend: list of legend entries
Math Image Representation (Multimath-300K, TQA)
User Prompt:
<Input Image> /Input Image>
You given textbook problem image. Output only valid JSON with:
25

---

Math Image Representation Continued
1. Overall Image Caption: a summary of the problem context.
2. Image Metadata:
equations: list of equations with LaTeX and bbox
diagram_parts: list of diagram elements
givens: list of given values
Spatial Image Representation (OpenSpaces, Spacellava)
User Prompt:
<Input Image> </Input Image>
You are given an indoor scene floorplan image. Output only valid JSON with:
1. Overall Image Caption: 2-3 sentence narrative of the space.
2. Image Metadata:
rooms_op
cranss_o_seetiionn sectioos: list of room information
furniture list of furniture items with position and orientation
annotations: list of structural elements
Cognitively Inspired CoT Generation: We invoke DeepSeek-RI to answer the question and extracts its step-by-step reasoning outputs with the prompt Pog as the initial CoT.
Cognitively Inspired CoT Generation for Images
System Prompt:
You are an Al assistant helping user answer questions about an image. When the user asks question, you respond by imagining you are looking the image with full attention, just like human would. Your task is to reason visually and logically about the image content to answer the user's question.
Follow this multi-step reasoning approach:
1. Simulate Visual Perception: Imagine you are looking at the entire image carefully, Build general understanding of what is shown.
2. Understand the Question: Reflect on what the user is asking. Think carefully about what kind of answer needed (e.g., fact, reason, comparison).
3. Identify Relevant Elements: Consider which parts of the image most related to the question. Focus on those elements in your mental analysis.
4. Visual Reasoning: Describe what you "see" using natural visual language (e.g., "The image shows..." "In the upper part of the image..." "). Analyze and interpret the visual content to build your answer.
5. Answer Thoughtfully: Provide a clear and direct answer. Ensure your reasoning is consistent with the visual elements you described.
Guidelines for Responses:
Don't expose in the output that you are answering based on text information. Use statements imitating looking at an image to answer.
Don't directly refer to any textual metadata such as "captions"" "description", "metadata", etc. If you need mention them, use "visual evidence" instead (e.g., "the image shows..." ").
It's okay double-check question yourself during the thought process reflect naturally as a human would.
26

---

Cognitively Inspired CoT Generation for Images Continued
Refer to locations in the image using expressions like: "in the center", "at the top", "on the left side", or "in the background".
User Prompt:
Image content: <Overall Image Caption
Image metadata: <Image Metadata>
Question: <Question>
Please think about this question if you human pondering deeply. It's encouraged to include self-reflection or verification in the reasoning process.
<Corresponding Answer Format Template>
Cross-modal CoT Refinement: We employ a cross-modal refinement process to ensure the CoT aligns with the image content, using the prompt P
Cross-modal CoT Refinement for Images
System Prompt:
You are a multimodal reasoning expert. Your task to revise hallucinations and errors in the Mhain-of-thouglee (CoT) based on the provided image. Do not significantly alter the original CoT logic or content, and ensure the final conclusion remains the same.
Your task:
1. Carefully examine the image, the question, and the CoT.
2. Identify only the reasoning steps that directly conflict with what is shown in the image:
Replace all references to textual cues (such as "title", "bbox", "label") with direct visual observations from the image.
Use visual phrases such as "The image shows...", "I can see...", or "From the visual layout..." instead text-based observations.
Use broader spatial descriptions like "on the left", "in the background", or "in the center", instead of specific coordinates labeled boxes.
not rewrite or paraphrase steps that are already visually accurate or consistent with the image.
Only replace correct parts that visually contradict what is shown.
3. Ensure the rest of the CoT (which is either correct or visually consistent) stays faithful to the original meaning. Only revise incorrect or hallucinated steps based on visual evidence.
User Prompt:
<Input Image> </Input Image>
Question: <Question>
Original COT: <Original CoT>
Output format:
Strictly follow this format. Return only the revised CoT and no additional explanation:
<think> [Revised </think>
27


Reviews:
[{"review_id": "8Qbupq8gKk", "reviewer": "Reviewer_xRp7", "summary": "The paper introduces VIDEORFT, a novel approach that extends the RFT paradigm to cultivate human-like video reasoning capabilities in MLLMs. The approach consists of a two-stage training paradigm: supervised fine-tuning with chain-of-thought annotations, followed by rule-based reinforcement learning, enhanced by a novel semantic-consistency reward. Recognizing the lack of high-quality video CoT datasets, the authors devise an automated cognitively inspired pipeline to generate rich, visually grounded CoT annotations and construct two large-scale datasets that support both SFT and RL phases. Extensive experiments across six benchmarks demonstrate the competitiveness and generalization ability of VideoRFT compared to both open-source and proprietary baselines.", "questions": "1. Regarding the details of the prompts, can the authors provide some specific examples of cognitive heuristic prompts? At the same time, I hope to discuss the universality of this method and its advantages over other methods.\n2. Regarding the mitigation of visual hallucinations, how does the semantic consistency reward deal with situations where the language description does not completely match the visual information, or when the video evidence itself is ambiguous?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n\n1. Comprehensive Pipeline for CoT Data: This paper proposes a pipeline to solve the lack of high-quality video CoT data, while using structured video, cognitive-inspired CoT generation, and cross-modal optimization. The overall process ensures text-visual consistency on the basis of generating CoT.\n2. Rich experimental results: This paper conducts a comprehensive comparison on six representative video reasoning benchmarks. VideoRFT performs well compared with most contemporary models, and the ablation results also clearly illustrate the impact of training and rule reward design.\n3. Semantic consistency reward: A new reward mechanism is introduced to encourage the video description generated by the model to be consistent with visual evidence, helping to improve the shortcomings of the current video reasoning model.\n\nWeaknesses\n1. Innovative comparison with contemporaneous work: The unique innovations of VideoRFT in theory and method have not been fully discussed, and most of its specific contributions are more reflected in incremental combinations and improvements rather than in the proposal of new concepts.\n2. Insufficient discussion of datasets: There is limited discussion on the diversity of automatically generated datasets, and the analysis is mainly focused on vocabulary and statistics, lacking in-depth analysis of the special data type of chain thinking (CoT).\n3. Lack of details in dataset construction: The lack of specific examples of cognitive inspiration prompt strategies and cross-modal optimization processes in the text makes it difficult to understand and evaluate related methods.\n4. Evaluation result analysis: Although this method has improved performance in most contemporaneous works, it has no significant advantage over Video-R1, suggesting that further consideration should be given to the validity of the data and its actual contribution to model performance.", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the valuable feedback.\nTo begin with, we focus on the following comment:\n>Q1.The unique innovations of VideoRFT in theory and method have not been fully discussed, and most of its specific contributions are more reflected in incremental combinations and improvements rather than in the proposal of new concepts.\n\nThanks for comments. We clarify that our work contributes **a novel paradigm of reinforced fine-tuning** tailored for video reasoning, aiming at endowing MLLMs with internalized reasoning capabilities. Unlike prior works that rely on emergent abilities or heuristic prompt(e.g.\"Let think step by step\").\n\nWhile we acknowledge that several contemporaneous works share a similar motivation, these methods were publicly released only after March 1st, 2025 (Video-R1 on March 27, and TinyLLaVA-Video-R1 on April 13). Per NeurIPS 2025 guidelines, *submissions will not be rejected on the basis of comparisons to contemporaneous work.* Nevertheless, we emphasize that our method contains several unique contributions:\n\n* We develop a novel CoT prompting scheme grounded in cognitive theory, which decomposes reasoning into five structured stages—**observation, task understanding, visual focus, reasoning, and reflection**. This mirrors human cognition and enables more interpretable and human-aligned CoTs generation than others. Additionally, we introduce a cross-modal refinement module to revise CoTs with visual evidence, reducing hallucination in CoTs by 28.5% and improving performance by +6.8% on average across datasets.\n\n* We propose a new reward function-**gated semantic-consistency reward**. It computes semantic alignment between video-grounded segments of CoTs and video input, and it uses a gating mechanism to activate the reward only when answer is right, avoiding reward of incorrect reasoning. To our knowledge, it's novel for video reasoning. \n\nWe'll incorporate all discussions in next vision.\n\n>Q2.There is limited discussion on the diversity of automatically generated datasets, and the analysis is mainly focused on vocabulary and statistics, lacking in-depth analysis of the special data type of CoT.\n\nAs requested, we provide a more in-depth analysis of our CoT data. Compared with standard QA annotations, CoTs are multi-step reasoning traces where each step must be both factual and logically coherent. Therefore, evaluating CoT quality requires examining two key properties: factual correctness(i.e., hallucination) and reasoning logicality.\n\n1) **Hallucination Rate**: following [ref1], we randomly sample 200 CoTs to assess hallucination. We use GPT-4o to extract all referenced entities and actions from CoTs and verify whether they are grounded in the corresponding video/image. Ungrounded elements are manually reviewed. This result reveals our CoT annotations exhibit a low hallucination rate of 4.5%.\n\n2) **Reasoning Logicality**: To further analyze the logicality of CoTs, we compare them with those generated by a contemporaneous method(Video-R1) on the same VQA instance:\n\n|Question|Video-R1|Ours|\n|-|-|-|\n|What did the person do to the laptop before lying on the bed? A.Opened B.Washed. C.Tidied up. D.Closed. *Ground Truth*: A|The person is lying on the bed in the beginning of the video. Then, they get up and start moving around the room. Hmm, what do they do next? They approach the computer desk and start interacting with the computer. **Wait, is there any bag, table, laptop, or phone/camera involved here? Oh, I see, they are not interacting with any of those objects directly.** They are just using the computer. Let‘s break it down further. The question asks ***what happened after they lied on the bed.*** They got up and used the computer. **There is no action of opening a bag, tidying up a table, putting down a laptop, or putting down a phone/camera.**|The question is asking what the person did to the laptop before lying on the bed. The video begins with **the person already lying on the bed, with the laptop screen glowing, indicating that the computer was already on**. The person then shifts and gets up to interact with the laptop. ***Given that the laptop is active from the start, it's clear that it was opened before the video began.*** Considering the options: A. Opened: **This aligns with the observation** that the laptop was already active when the person was lying on the bed. ...(*Analysis of other answers*)... The answer is most likely A.|\n\nAs shown, the CoT from Video-R1 tends to drift toward unrelated details (e.g., bags, tables) and misunderstands the temporal cue in the question (ask what happened before lying on the bed, not after).\n\nIn contrast, **our CoT follows a coherent, visually grounded reasoning**: It anchors the scene (\"laptop screen glowing\"), infers the temporal relationship, and eliminates wrong options through visual evidence. This reflects a logically clear and cognitively aligned reasoning process, closer to how humans analyze visual scenes to reason.\n\n>Q3.1.Lack of specific examples of cognitive inspiration prompt strategies and cross-modal optimization processes.\n\n>Q3.2.Can authors provide some specific examples of cognitive prompts? Discuss universality of method and advantages over other methods.\n\n1) Sorry for the confusion. We clarify that ALL prompts used in the data generation process are presented in **[Supplementary Sec. B]**. To ease understanding, we copy key parts of prompts here. \n\n**The five sub-prompts for cognitive CoT generation in Eq. 1.**\n> Follow the multi-step reasoning approach:\n> 1. Simulate Watching: Imagine you are watching the entire video from beginning to end.\n> 2. Understand Question: Reflect on what the user is asking. Think carefully about what kind of answer is needed.\n> 3. Localize Relevant Moments: Consider which parts of the video are most related to the question. Focus on those segments in mental replay.\n> 4. Visual Reasoning: Describe what you \"see\" in those segments using natural visual language. Analyze the visual content to build answer.\n> 5. Answer Thoughtfully: Provide a clear and direct answer. Ensure reasoning is consistent with the video.\n\n**Part of prompt for cross-modal refinement**:\n> Your task is to revise hallucinations and errors in the CoT based on the visual content of the provided video.\n> 1. Carefully examine video, question, and CoT\n> 2. Identify only the reasoning steps that directly conflict with what is visually shown in the video:\n> 3. Correct CoT parts that contradict the video.\n> 4. Replace any text-based references with direct visual observations.\n> 5. Replace specific timestamps with broader temporal phrases.\n\n2) **Universality and advantages**:\n\n**Regarding universality**: Our prompting is **task-agnostic**, making it broadly applicable across diverse video reasoning scenarios. During data generation, we employ the same prompting strategy across diverse QA taks (i.e., multiple-choice, open-ended QA, OCR) and task categories (i.e., general understanding, spatiotemporal analysis, scientific reasoning, mathematical problem, and document). This shows the strong universality.\n\n**Regarding advantages**: Our method integrates complementary strengths: the visual grounding of MLLMs and the deep reasoning capabilities of reasoning LLMs. Guided by cognitively inspired prompts, the LLM generates a human-like CoT, which is then refined by a MLLM with full video access. In contrast, prior methods either use LLMs with textual visual inputs, leading to hallucinations, or rely on MLLMs to mimic reasoning via surface-level prompts, lacking internalized reasoning modules.\n\n> Q4.Although this method has improved performance in most contemporaneous works, it has no significant advantage over Video-R1, suggesting that further consideration should be given to the validity of the data and its actual contribution to model performance.\n\n1) **Performance**: **First**, we clarify that our approach shows better overall performance over Video-R1. Particularly, designed for video reasoning, more pronounced improvements on reasoning-intensive benchmarks. For instance, VideoRFT surpasses Video-R1 by +1.0% on VSI-Bench and +4.7% on MMVU. **Second**, to further address your concern, we extend the evaluation to include 5 additional video reasoning datasets. As seen, VideoRFT consistently outperforms Video-R1 by clear margins. This further confirms the superior performance of VideoRFT.\n\n|Model|LongVideo Bench|IntentQA|Next-QA|Causal-VidQA|DarkEventInfer|\n|:-|:-:|:-:|:-:|:-:|:-:|\n|Video-R1|50.2|88.2|78.3|61.7|56.0|\n|VideoRFT|**54.7**|**92.7**|**80.1**|**71.6**|**67.0**|\n\n2) **Data Contribution**: \nTo assess the actual impact of our dataset, We re-trained Qwen2.5-VL on both Video-R1 and our CoT dataset (with identical settings). Despite using only 62% samples (102K vs 165K), ours achieves better or comparable performance across:\n\n|Model|VSI.|VideoMMMU|MMVU|MV.|Temp.|VideoMME|\n|-|-|-|-|-|-|-|\n|Qwen2.5-VL + Video-R1 dataset|31.8|**47.4**|61.3|59.4|69.2|52.8|\n|Qwen2.5-VL + VideoRFT dataset|**32.7**|47.1|**63.5**|**59.9**|**69.6**|**53.0**|\n\n> Q5.How does the semantic consistency reward deal with situations where the language description doesn't completely match the visual information, or when the video evidence itself is ambiguous?\n\nTo clarify, we restate reward formulation below:\n\n$R_s=\\min(1,w\\times \\max(\\text{cos}(t,v),0)) \\quad R=R_f+R_a+\\mathbb{1}[R_a>0]\\cdot R_s$\n\n1) Regarding \"language description $t$ doesn't completely match the visual information $s$\". If completely unrelated, $\\cos(t,v)<0$ yields $R_s=0$, contributing no reward signal. If weakly related, $\\cos(t, v)$ is a small positive value, and $R_s=w \\times \\cos(t,v)$. As training proceeds, alignment improves and $R_s$ gradually increases.\n\n2) When the **video evidence is ambiguous**, the model might tend to give a wrong answer, i.e., $R_a=0$. Thus, $R_s$ is excluded from the final reward.\n\n[ref1] HalluMeasure: Fine-grained Hallucination Measurement Using Chain-of-Thought Reasoning, EMNLP2024"}, {"review_id": "3JMHM3G0D5", "reviewer": "Reviewer_CSJu", "summary": "This work designed a dataset generation pipeline to build two video reasoning CoT datasets VideoRFT-CoT-102K for SFT and VideoRFT-RL-310K for RL. Based on these datasets, a MLLM is finetuned with an extra reward (Semantic-Consistency Reward). The finetuned model is evaluated on several video datasets.", "questions": "See weakness above.", "limitations": "The limitation of this work is not discussed in the conclusion section.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n- The paper is easy to understand.\n- The produced dataset is a contribution to the community.\n\nWeakness:\n\n- From the implementation, the VideoRFT samples 16 frames from each video for training, and 32 frames for testing. It looks like this method is designed for short videos. Nowadays, for complex video reasoning tasks, the long-form videos are important part to be evaluated. Considering the recent methods shown in the VideoMME leaderboard (InternVL2.5, Qwen2-VL) that uses much more frames from a video, I doubt the long-form reasoning capability from the proposed method.\n- The proposed method doesn’t achieve the SoTA performance in some datasets. For example, in VideoMME leaderboard, the best open source and published 7B/8B size baseline is ViLAMP (ICML 2025) which achieves 67.5 without subtitle, however, the proposed method only get 59.8, which is around 8% gap.\n- For the dataset generation, high-quality annotations are required. However, I’m not sure if the current pipeline is sufficient to produce the high-quality dataset annotation.\n    - In the first step extracting the video representation, how to ensure the GPT-4o-mini produce the reliable results without hallucination?\n    - When using Qwen2.5-VL-7B to refine CoT, how to ensure this VLM can generate reliable output? Although it is designed to fix the hallucination problem from DeepSeek-R1, but how to know if Qwen2.5-VL will produce more hallucination?\n    - There is no quantitative analysis for the hallucination problem from dataset annotation generation.\n- The MLLM is finetuned based on the data produced from other LLM/VLMs, does it mean it is just a knowledge distillation from GPT-4o-mini, DeepSeek R1 and Qwen2.5-VL-7B?\n\nMinor Weakness:\n\n- Figure 2 typo: A-OKVQ → A-OKVQA\n\nI think it is an interesting work but requires more polishing. Therefore I suggest borderline reject. I’m willing to adjust my rating if my concerns are solved.", "was_revised": true, "final_justification": "See my response.", "rebuttal": "We thank the reviewer for the valuable feedback and suggestions. We begin by addressing the following concern:\n> Q 1. From the implementation, the VideoRFT samples 16 frames from each video for training, and 32 frames for testing. It looks like this method is designed for short videos. Nowadays, for complex video reasoning tasks, the long-form videos are important part to be evaluated. Considering the recent methods shown in the VideoMME leaderboard (InternVL2.5, Qwen2-VL) that uses much more frames from a video, I doubt the long-form reasoning capability from the proposed method.\n\n\nThanks for your thoughtful comments, but there might be a slight misunderstanding here. We clarify the following points:\n\n\n**First**, our frame sampling setup (16 frames for training, 32 for testing) is consistent with many recent works (Video-R1, TinyLLaVA-Video-R1 and [ref1]), and was adopted to ensure a fair comparison. \n\n**Second**, although our current evaluation uses a limited number of frames, our model is not inherently restricted to short videos. In fact, it makes no assumptions about input video length. As you noted, Qwen can handle very long-context inputs (i.e., 768 frames in the VideoMME leaderboard), and since our method is built upon Qwen, it is theoretically capable of handling such long-form inputs. However, increasing the number of frames during training and inference incurs significant GPU memory and computational overhead, which exceeded our compute constraints.\n\n**Third**,  we fully agree that long-form video reasoning is an increasingly important area. Nevertheless,  our current work focuses primarily on generic video reasoning, and accordingly, we follow the standard evaluation protocols used in closely related studies (e.g., [ref1–3]) to render more meaningful validation. As such, missing evaluations in the mentioned aspect should not be a major issue.  Despite this, to further address your concern, we quickly conduct two sets of new experiments to evaluate our method's capacity in handling long-form videos:\n\n**Experiment 1: Scaling up input frames on VideoMME**\n\nWe evaluate how the model benefits from increased frame counts during inference. As shown below, the model performance steadily improves with more frames, suggesting that VideoRFT can effectively leverage longer video contexts.\n\n|# frames|32|64|128|\n|:-|:-:|:-:|:-:|\n|VideoRFT|59.8|62.0|64.5|\n\n**Experiment 2: Evaluation on LongVideoBench**\n\nWe further evaluate our model on the LongVideoBench dataset, which is explicitly designed to test long-form video reasoning. All models are evaluated under the same condition using 32 sampled frames. As shown below, VideoRFT outperforms all baselines. This confirms the superior capability of VideoRFT in long-form video reasoning. \n\n|Model|Qwen2.5-VL|Video-R1|VideoChat-R1|**VideoRFT**|\n|:-|:-:|:-:|:-:|:-:|\n|LongVideoBench|41.8|50.2|43.1|**52.6**|\n\n\n\n\nNext, we address another follow-up concern: \n> Q 2. In VideoMME leaderboard, the best open source and published 7B/8B size baseline is ViLAMP (ICML 2025) which achieves 67.5 without subtitle, however, the proposed method only get 59.8, which is around 8% gap.\n\n\nThanks for referring this to us. However, we respectfully note that a direct comparison between ViLAMP and our model is not entirely fair, as ViLAMP adopts a significantly more resource-intensive setup. Specifically:\n* Training data: ViLAMP uses 1.3M shot videos and 500K longer videos for training, 17.6 times larger than us (only 102K videos);\n\n* Number of frames: ViLAMP supports up to 600 frames for both training and inference, whereas we use only 16 frames for training and 32 for inference.\n* Image resolution:  ViLAMP uses higher-resolution frames (384×384) compared to our model’s 224×224.\n\nTo enable a more reasonable comparison, we perform a new evaluation on the VideoMME dataset using 128 frames during inference.  p.s. this is still not strictly fair as our model was still trained on 16 frames and uses a smaller image resolution. \n\nThe results are: \n||Short|Medium|Long|\n|:-|:-:|:-:|:-:|\n|ViLAMP|75.1|64.6|53.2|\n|VideoRFT|76.3|66.1|51.1|\n\nAs seen, our model outperforms ViLAMP on the Short and Medium subsets.  While our performance is slightly lower on the Long subset, we believe this is expected, as ViLAMP is specifically tailored for long-form video reasoning, whereas our approach focuses on general-purpose video reasoning without such long-form-specific optimizations.\n\n\n\nRegarding hallucination, you mentioned:\n> Q 3.1. In the first step extracting the video representation, how to ensure the GPT-4o-mini produce the reliable results without hallucination?\n\n> Q 3.2. When using Qwen2.5-VL-7B to refine CoT, how to ensure this VLM can generate reliable output? Although it is designed to fix the hallucination problem from DeepSeek-R1, but how to know if Qwen2.5-VL will produce more hallucination?\n\n> Q 3.3. There is no quantitative analysis for the hallucination problem from dataset annotation generation.\n\nThanks for your comments. We fully acknowledge that hallucination is a well-known and ongoing challenge for large models. In our method,  we have carefully designed our data generation pipeline to minimize hallucinations throughout. Below, we address each of your specific concerns:\n\n **Regarding Q 3.1**\n\nIn the first stage of our pipeline, we adopt a **structured prompting** strategy that constrains GPT-4o-mini’s output to a strict JSON format, specifying fields such as timestamp, scene, object, and action. This structure explicitly grounds the model’s responses in observable video elements and enforces output consistency. The idea of structured prompting has been validated in prior works such as [ref4], that is, structured prompting  significantly reduces hallucination rates—for example, by 76.9% in the Wikidata list generation task. Additionally, our approach performs automatic format validation and discard ill-formed outputs.  These steps collectively help ensure that the extracted video representations from GPT-4o-mini are  properly grounded in the video with minimal hallucination. \n\n**Regarding Q 3.2**\n\nWe clarify that our pipeline use the 32B (rather than 7B) version of Qwen2.5-VL here, for refining CoT annotations. While hallucination still cannot be entirely avoided, we find that grounding the CoT refinement in visual evidence is a much more important factor in improving reliability. In our pipeline, Qwen2.5-VL-32B is provided with video-grounded representations and is tasked with validating and refining prior CoTs generated by **blind LLMs** (i.e., DeepSeek-R1). This refinement strategy improves factual consistency and reduces hallucination by anchoring reasoning to explicit visual context..\n\n**Regarding Q 3.2**\n\nTo quantitatively analyze hallucinations from dataset annotation, we consider both direct data inspection and its downstream impact on model performance.\n\nFirst, following [ref5], we randomly sample 200 CoTs from the VideoRFT-CoT dataset. GPT-4o extracts all referenced entities and actions from CoTs and checks whether they are grounded in the corresponding video/image. Potentially ungrounded items are then manually verified. Finally, we observe a small hallucination rate of **4.5%** in our final dataset. In contrast, a variant without the cross-modal CoT refinement stage showed a significantly higher hallucination rate of **33.0%**.\n\n\nSecond, we evaluate the downstream impact of hallucinated CoTs by training a model with unrefined annotations. Performance drops significantly across all benchmarks:\n\n|Model Variant|VSI-Bench|VideoMMMU|MMVU|MVbench|TempCompass|VideoMME|\n|:-|:-:|:-:|:-:|:-:|:-:|:-:|\n|w/o Cross-Modal CoT Refinement|30.2|45.0|61.8|58.2|66.4|49.7|\n|Full VideoRFT|36.8|51.1|68.5|62.1|73.7|59.8|\n\nThese results confirm the effectiveness of our dataset annotation pipeline in progressively reducing hallucination, ultimately producing high-quality CoTs with a low hallucination rate.\n\n\n\n> Q 4. The MLLM is finetuned based on the data produced from other LLM/VLMs, does it mean it is just a knowledge distillation from GPT-4o-mini, DeepSeek R1 and Qwen2.5-VL-7B?\n\n\nThank you for the question. While it's true that our model is initially fine-tuned using data generated by other LLMs/VLMs, it would be reductive to characterize our approach as merely knowledge distillation from them.\n\n\nVideo reasoning is a particularly challenging task due to the scarcity of high-quality, structured CoT annotations. To tackle this, we design an automated pipeline to distill knowledge from other LLM/VLMs. However, the distillation scheme goes far beyond conventional technique, and takes a multi-modal, multi-expert collaboration pipeline to integrate complementary strength from diverse sources:  the visual grounding of VLMs and the deep reasoning abilities of LLMs.\n\nBeyond this, we clarify that the main contribution of our method lie in a reinforced fine-tuning paradigm for video reasoning. As evidenced in Table 2,  the most substantial performance gains arise from the RL phase (rather than SFT).  As such, our method is not just a passive recipient of distilled knowledge, but rather an active framework for knowledge integration, adaptation, and generalization tailored specifically for complex video reasoning. Thanks.\n\n\n\n> Q 5. Minor Weakness: Figure 2 typo: A-OKVQ → A-OKVQA\n\nWill fix it. Thanks.\n\n> Q 6. The limitation of this work is not discussed in the conclusion section.\n\nThe limitation has been discussed in Sec. C of the supplementary material.\n\n[ref1] ShareGPT4Video: Improving Video Understanding and Generation with Better Captions, NeurIPS 2024\n\n[ref2] Chain-of-Verification Reduces Hallucination in Large Language Models, ACL 2024\n\n[ref3] HalluMeasure: Fine-grained Hallucination Measurement Using Chain-of-Thought Reasoning, EMNLP 2024"}, {"review_id": "INSOGtZnI0", "reviewer": "Reviewer_3mwX", "summary": "The paper introduces VIDEORFT, a novel framework to enhance video reasoning capabilities in MLLMs through Reinforced Fine-Tuning (RFT). The framework extends traditional RFT into the video domain by addressing two critical challenges: (1) the scarcity of large-scale, high-quality CoT datasets for video reasoning, and (2) the lack of mechanisms to ensure alignment between textual reasoning and visual evidence. VIDEORFT's contributions include: A scalable, cognition-inspired pipeline for generating high-quality CoT data for video reasoning. This results in two datasets: VideoRFT-CoT-102K for supervised fine-tuning (SFT) and VideoRFT-RL-310K for reinforcement learning. A novel semantic-consistency reward that promotes grounding of the reasoning process in visual evidence, reducing hallucinations and improving reasoning coherence. Experimental results demonstrating state-of-the-art performance across six video reasoning benchmarks.", "questions": "The leading performance of VIdeo-R1 is caused by what factors? The reward design or the data's quality?\n\nWhy didn't the author try to increase the training frames to verify the video reasoning effect under different frames? Did the author also set the inference frames to 32 when reasoning video-R1?\n\nWill the Semantic-Consistency Reward conflict with the reasoning-specific content, such as \"wait\", \"let me think\". Does it overly constrain the model’s reasoning capabilities, especially when abstract reasoning extends beyond visual input?", "limitations": "The authors did not include the limitation discussion in this paper.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. The semantic-consistency reward is a meaningful contribution that ensures reasoning is grounded in visual input, tackling a significant limitation in existing MLLMs.\n\n2. The automated pipeline for generating cognitively grounded CoT data is a notable innovation. The datasets, VideoRFT-CoT-102K and VideoRFT-RL-310K, cover diverse cognitive and reasoning tasks.\n\n3. The paper provides detailed explanations of its novel pipeline, training strategy (SFT and RL), and reward modeling.\n\nWeakness:\n\n1. The semantic-consistency reward relies on SigCLIP embeddings, which may add computational overhead during training. The authors did not evaluate the consumption of this side. \n\n2. One of the main contributions of this paper is the building of data, but the author did not show the ablation study on the dataset. I suggest that the author train their Qwen2.5-VL under the same hyperparameter setting (VideoRFT and Video-R1). It will be clearer how this dataset advanced to Video-R1.\n\n3. The authors should be clearer about the experimental details, such as what reinforcement learning training tools were used, such as VERL, Open-R1, etc., and should cite them correctly.\n\n4. The improvement is marginal compared to Video-R1, especially on the VideoMME dataset.", "was_revised": true, "final_justification": "I still stand by the rejection of this paper, for the following reasons:\n\n1. In the rebuttal state, the author showed the unrealistic speed of Siglip during the RL training, which is extremely unprofessional. \n\n2. This paper introduces the 'Semantic-Consistency Reward', which requires the structural reasoning process (e.g., describe first, then think, and finally give the answer), and the authors manually set the maximum length of the first stage to 64. This would potentially break the capability of base VLM. Whether in LLM or VLM's reasoning RL, we hope the model discovers the reasoning trace freely. However, this structural setting will limit this exploration; this limitation was already confirmed by many works.\n\n3. On the most used benchmark, VideoMME, the performance only improves 0.2%; however, it costs huge RL training resources.", "rebuttal": "We thank the reviewer for the feedback and suggestions. We address your comments below, and start with the following question:\n\n> Q1. The semantic-consistency reward relies on SigCLIP embeddings, which may add computational overhead during training. The authors did not evaluate the consumption of this side.\n\nThanks for pointing this out. First, we clarify that  for SigCLIP, we use the *siglip-so400m-patch14-384* version, which has **only  400M  parameters** - very lightweight compared to the **7B parameters of the full model**. Further, per your request, we quantify the overhead on a machine with  8×A800 GPUs. As shown in the table below, SigCLIP introduces only **0.3 GB** additional memory per GPU and adds **0.03 seconds** per training step, indicating that the computational cost from SigCLIP is minimal.\n\n|Setting|Mem per GPU (GB)|Time per Step (s)|\n|:-|:-:|:-:|\n|w/o $SigCLIP$|60.3|0.42|\n|w/ $SigCLIP$|60.6|0.45|\n\nThe table will be added to the paper.\n\n\n\n\nNext, we address your comments on evaluation (regarding both data and model):\n> Q2.1. **Data**: One of the main contributions of this paper is the building of data, but the author did not show the ablation study on the dataset. I suggest that the author train their Qwen2.5-VL under the same hyperparameter setting (VideoRFT and Video-R1). It will be clearer how this dataset advanced to Video-R1.\n\n> Q2.2. **Model**: The improvement is marginal compared to Video-R1, especially on the VideoMME dataset.\n\n\n**More results on our data**\n\nAs suggested, we train Qwen2.5-VL using data from Video-R1 and our proposed VideoRFT during the supervised finetuning stage (SFT). All configurations are kept identical except for the training data.  The results are presented in the table below. Notably, while **VideoRFT contains only 62% of the supervision data** compared to Video-R1 (102K vs. 165K), it  leads to  better overall performance. This confirms the advantage of our dataset. \n\n|Model|VSI-Bench|VideoMMMU|MMVU|MVbench|TempCompass|VideoMME|\n|:-|:-:|:-:|:-:|:-:|:-:|:-:|\n|Qwen2.5-VL + Video-R1 dataset|31.8|**47.4**|61.3|59.4|69.2|52.8|\n|Qwen2.5-VL + VideoRFT dataset|**32.7**|47.1|**63.5**|**59.9**|**69.6**|**53.0**|\n\n**Model performance against Video-R1**\n\n**First**, we clarify that our approach shows better overall performance over Video-R1. Particularly, designed for video reasoning, more pronounced improvements on reasoning-intensive benchmarks. For instance, VideoRFT surpasses Video-R1 by +1.0% on VSI-Bench and +4.7% on MMVU. **Second**, to further address your concern, we extend the evaluation to include  5 additional video reasoning datasets.  As seen,  VideoRFT consistently outperforms Video-R1 by clear margins. This further confirms the superior performance of VideoRFT. Thanks.\n\n|Model|LongVideo Bench|IntentQA|Next-QA|Causal-VidQA|DarkEventInfer-Test|\n|:-|:-:|:-:|:-:|:-:|:-:|\n|Video-R1|50.2|88.2|78.3|61.7|56.0|\n|VideoRFT|**54.7** (+4.5)|**92.7**(+4.5)|**80.1**(+1.8)|**71.6**(+9.9)|**67.0**(+11.0)|\n\nWill add the new results to the paper. \n\n\n\n>  Q3. The authors should be clearer about the experimental details, such as what reinforcement learning training tools were used, such as VERL, Open-R1, etc., and should cite them correctly.\n\nAgree. For RL training, we use the HuggingFace TRL library. The codebase is based on Open-R1. Will add proper citations in the next version. Thanks. \n\n\n\n> Q4. The leading performance of VIdeo-R1 is caused by what factors? The reward design or the data's quality?\n\n**Both**. \n\n**Reward Design**: As shown in Table 2, removing our proposed semantic-consistency reward leads to a 2.0% average accuracy drop across all benchmarks, confirming it as a key factor in leading performance.\n\n**Data Quality**: To isolate the impact of data quality, we retrain Qwen2.5-VL using either the Video-R1 dataset or our proposed VideoRFT dataset under identical training configurations. The results are shown in the above second table, even though our dataset contains only 62% of the supervision data compared to Video-R1 (102K vs. 165K), it consistently achieves better performance across six benchmarks.\n\nTo further validate the contribution of our data, we construct an alternative CoT dataset by prompting DeepSeek-R1 with our structured video representations to generate CoTs. We then train a model using this CoT data and compare it with the same model trained on our own CoTs. As shown below, our CoT data leads to consistent and notable gains:\n\n|Model Variant|VSI-Bench|VideoMMMU|MMVU|MVbench|TempCompass|VideoMME|\n|-|-|-|-|-|-|-|\n|DeepSeek-R1 output CoT|30.2|45.0|61.8|58.2|66.4|49.7|\n|Our CoT|**36.8**|**51.1**|**68.5**|**62.1**|**73.7**|**59.8**|\n\nThese results demonstrate that both our carefully designed reward and high-quality CoT data are essential contributors to the performance gains of VideoRFT, each providing measurable benefits individually.\n\n\n\n> Q5. Why didn't the author try to increase the training frames to verify the video reasoning effect under different frames? Did the author also set the inference frames to 32 when reasoning video-R1?\n\nSorry for the confusion. **First**, we clarify that our choice of the setting (16 frames for train and 32 frames for inference') strictly follows the practices in recent literature [ref1–3]. This **guarantees a fair comparison**. \n\nDue to the time limit in the rebuttal stage, we quickly examine how our model performs when increasing the number of frames during inference. As seen from the table below, when increasing the frame number from 32 to 64 to 128, the model performance progressively increases. We will continue to study the impact when increase the training frames and add the results to the next version. \n\n|Frames|32|64|128|\n|:-|:-:|:-:|:-:|\n|VideoMME|59.8|62.0|64.5|\n\n**Second**, yes, we also set the inference frames to 32 for Video-R1. \n\n\n\n\n>  Q6. Will the Semantic-Consistency Reward conflict with the reasoning-specific content, such as \"wait\", \"let me think\". Does it overly constrain the model's reasoning capabilities, especially when abstract reasoning extends beyond visual input?\n\nAs noted in Sec. 3.2.1, our CoTs typically consist of three stages: *question parsing, video description, and abstract reasoning*. During reward computation, we apply the semantic-consistency reward to a fixed-length span of M tokens starting from the second sentence of the CoT. We set M = 64 for all experiments. **This span usually corresponds to the video description part and does not cover the abstract reasoning**.\n\nTo directly address your concern, we analyze the average token index of (i) the start of the second sentence (where the reward span begins) and (ii) the first occurrence of abstract reasoning-specific content (e.g., \"wait\", \"let me think\") as following:\n\n|Metric|VSIBench|VideoMMMU|MMVU|MVbench|TempCompass|VideoMME|\n|:-|:-:|:-:|:-:|:-:|:-:|:-:|\n|Avg. Token Index of 2nd Sentence Start (Reward Span Begins)|21.29|23.24|21.62|17.65|17.25|19.41|\n|Avg. Token Index of 1st Abstract Reasoning Phrase|95.16|136.47|114.10|125.93|93.53|124.25|\n\nAs shown, the semantic-consistency reward is applied well before the onset of general or abstract reasoning. This confirms that our reward design does not interfere with the model’s abstract reasoning capabilities.\n\n\n\n> The authors did not include the limitation discussion in this paper.\n\nSorry for confusion. We have presented the limitation analysis in Sec. C of the supplementary.\n\n[ref1] ShareGPT4Video: Improving Video Understanding and Generation with Better Captions, NeurIPS 2024\n\n[ref2] Tinyllava-Video-R1: Towards Smaller LLMs for Video Reasoning. arXiv 2025.\n\n[ref3] Video-r1: Reinforcing Video Reasoning in MLLMs, arXiv 2025"}, {"review_id": "z1joSd97OH", "reviewer": "Reviewer_9Vu5", "summary": "The paper is one of the recent efforts to expand reasoning capabilities to a multi-modal large language model (MLLM), specifically for videos. For this task, one of the main hurdles is the lack of video-text datasets with step-by-step visual and text explanations. This paper discusses that such datasets are the first entry point to train MLLM with reasoning capabilities. Thus, this paper introduces an automated data pipeline for creating such a video reasoning dataset. Their proposed pipeline consists of two stages:(a) using an existing MLLM to generate a structured textual representation of the video, including captions, timestamps, etc. Then, using an existing reasoning LLM, specifically the DeepSeek model, to transform this structured text into step-by-step reasoning outputs. (b) In the second stage, another MLLM, Qwen 2.5-VL, is used to filter the generated reasoning by checking whether the reasoning traces are well-aligned with the visual content of the video. This dataset was then used to train the MLLM model, 7 B Qwen 2.5-VL. The training process follows a two-stage approach: supervised fine-tuning using the VideoRFT-CoT-102K dataset, followed by reinforcement learning via Group-Relative Policy Optimization (GRPO) using the VideoRFT-RL-310K dataset. During the reinforcement stage, the reward function combines accuracy, output formatting, and semantic consistency. The model is penalized when its reasoning trace, the portion of the response that describes video content, is misaligned with the actual video content, as measured by the SigLIP model. This penalty is applied only when the model's final answer is correct, based on the insight that even well-grounded reasoning is useless if it leads to an incorrect conclusion/answer. Their experimental results show that such reward formulation leads to performance improvements across multiple benchmarks.", "questions": "Suppose we take the structured text representation of a video, convert it into chain-of-thought (CoT) reasoning, apply your proposed filtering step, and then feed both the original video and the reasoning-based description to a pre-trained and frozen Qwen2.5-VL model. How would its performance compare to that of the fine-tuned version? This would help clarify whether additional fine-tuning is truly necessary or whether the reasoning improvements can come from better prompting and data conditioning.\n\nAssuming fine-tuning is still helpful, what tradeoffs does it introduce in terms of generalization? Specifically, how does your fine-tuned model perform on other benchmarks not included in Table 1, such as EgoSchema, LongVideoBench, VideoMME with subtitles, or MMVU when not restricted to multiple-choice questions? A comparison here would help clarify whether the fine-tuning narrows or broadens the model’s capabilities.\n\nAlso, your model is trained using inputs of 128 × 28 × 28 resolution. In Table 1, are the baseline models trained under comparable input settings? If not, how is the comparison fair?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "The paper is well-written and organized. The generated data pipeline is clearly described and reproducible, and so are the training details. The results are favorably better than baseline Qwen 2.5-VL 7B and similar approaches, such as Video-R1 on the selected six benchmarks/datasets.\n\nThe originality of this work mainly lies in fine-grained reward modifications by invoking semantic consistency only if the model is accurate during training.\n\nThe proposed automated data pipeline is also not that novel, except for prompting LLM with five sub-prompts mimicking human cognition.\n\nIn my view, these datasets and training MLLMs on them only truly matter if we can first establish that baseline models genuinely lack reasoning ability, and that further fine-tuning does not degrade their generalization. Please see my questions below.", "was_revised": true, "final_justification": "The rebuttal addressed my concerns, particularly regarding the need for fine-tuning compared to prompting and the generalization ability of the fine-tuned model. I have therefore improved my rating.", "rebuttal": "We thank the reviewer for the insights and suggestions. We address your comments below, and start with the following question:\n\n> Q1. The proposed automated data pipeline isn't that novel, except for prompting LLM with five sub-prompts mimicking human cognition.\n\nThanks for your comments. As noted in your **Summary**,  the scarcity of CoT datasets has been a major bottleneck in video reasoning. Thus, we believe that our work,  constructing such a dataset and validating its value, is itself a timely and valuable contribution to the community. \n\nRegarding the novelty of our pipeline, we'd like to clarify its distinction from prior work. Current CoT generation pipelines typically fall into two categories: 1) Blind LLM pipelines (e.g., [ref1-2]) that rely solely on reasoning-centric LLMs (e.g., DeepSeek-R1) prompted with textual video descriptions. These methods often suffer from hallucination due to the lack of visual grounding. 2) MLLM-based approaches (e.g., Video-R1 [ref3]) that use vision-language models (e.g., Qwen-VL) to simulate reasoning via surface-level cues (e.g., \"let me think…\"). However, these models lack the deep reasoning capabilities needed for complex multi-step inference.\n\nIn sharp contrast, our method introduces two major innovations:\n\nFirst, we propose **a cognitively inspired five-stage CoT reasoning structure** that goes beyond the typical approaches. Rooted in cognitive science, our method decomposes reasoning into five stages: *observation, task understanding, visual fosuc, reasoning, and reflection*. This design mirrors human reasoning processes and enables a structured transition from visual grounding to abstract thought, yielding more interpretable and accurate CoTs.\n\nSecond, our pipeline establishes a collaborative framework between complementary model types. We pair the abstract reasoning power of blind LLMs (e.g., DeepSeek-R1) with the visual grounding capabilities of strong MLLMs (e.g., Qwen2.5-VL). This multi-expert, multi-stage generation and refinement process ensures that CoTs are both logically sound and visually grounded. This is rarely explored in existing video reasoning studies. Thanks.\n\n\n\n\nNext, we address the following comments:\n> Q2.1 In my view, these datasets and training MLLMs on them only truly matter if we can first establish that baseline models genuinely lack reasoning ability.\n\n> Q2.2 Suppose we take the structured text representation of a video, convert it into chain-of-thought (CoT) reasoning, apply your proposed filtering step, and then feed both the original video and the reasoning-based description to a pre-trained and frozen Qwen2.5-VL model. How would its performance compare to that of the fine-tuned version? This would help clarify whether additional fine-tuning is truly necessary or whether the reasoning improvements can come from better prompting and data conditioning.\n\nFollowing your suggestion, we conduct additional experiments to examine whether improved prompting alone can effectively elicit the reasoning capabilities of baseline models. The conclusion is: **better prompting remains insufficient for this purpose**.\n\n**Experimental Setup:** First of all, we note that (1) the suggested pipeline, while potentially informative, is computationally expensive and impractical for real-world deployment; (2) more importantly, the reasoning-based description (i.e., CoTs) from such a pipeline would implicitly include the final answers, making the evaluation less meaningful. Therefore, we instead explore two alternative prompting strategies, and feed the video, question, and prompts to the baseline to derive answers:\n\n* **Plain Prompting:** We design a plain prompt to encourage the baseline to explicitly reason within a `<think>...</think>` block before answering, and  use reflective phrases such as \"wait\", \"let me think\", etc. The plain prompt is:\n> {Question}\n> Please think about this question as if you were pondering deeply.  Engage in an internal dialogue using expressions such as 'let me think', 'wait', 'oh, I see','let's break it down', etc.  It's encouraged to include self-reflection or verification in the reasoning process. Provide your detailed reasoning between the `<think>` and `</think>` tags, and then give your final answer between the `<answer>` and `</answer>` tags.\"\n\n* **Cognition-Aware Prompting:** Further, we extend the plain prompt into a more complex, cognition-aware prompt by explicitly guiding the baseline to mimic the 5-step human reasoning trajectory, as presented in **Sec. 2.2**. We also guide the baseline to align its reasoning with the actual video input. The key part of the prompt is:\n\n> Follow this multi-step reasoning approach:\n> 1. Browsing the Video: You watch the entire video from beginning to end. Build a general sense of what is happening.\n> 2. Understand the Question: Reflect on what the user is asking. Think carefully about what kind of answer is needed.\n> 3. Localize Relevant Moments: Consider which parts of the video are most related to the question. Focus on those segments in your mental replay.\n> 4. Visual Reasoning: Analyze and interpret the visual content and perform in-depth reasoning to build your answer.\n> 5. Answer Thoughtfully: Reflect your reasoning and provide a clear and direct answer. Ensure your reasoning is consistent with the visual events you described.\n>\n>    [Plain Prompt]\n\n\n\n**Results are:**\n|Model|VSI-Bench|VideoMMMU|MMVU|MVBench|TempCompass|VideoMME|\n|:-|:-:|:-:|:-:|:-:|:-:|:-:|\n|Plain Prompting|31.8|47.4|61.3|59.4|69.2|52.8|\n|Cognition-Aware Prompting|34.8|49.4|58.4|58.2|72.7|55.5|\n|VideoRFT|36.8|51.1|68.5|62.1|73.7|59.8|\n\nAs seen, both prompting strategies yield significantly lower performance than our VideoRFT, especially on datasets such as VSI-Bench, VideoMMMU, and MMVU, which emphasize complicated reasoning. This safely validate that **additional fine-tuning is truly necessary**.\n\n\n\nFurther, you ask some follow-up questions on model generalization: \n> Q3. Assuming fine-tuning is still helpful, what tradeoffs does it introduce in terms of generalization? Specifically, how does your fine-tuned model perform on other benchmarks not included in Table 1, such as EgoSchema, LongVideoBench, VideoMME with subtitles, or MMVU when not restricted to multiple-choice questions?\n\nSorry for the confusion.  In the current experiments, we have extensively validated that our model shows promising performance across a variety of tasks, including spatial reasoning, scientific QA, video understanding, and temporal reasoning. This already indicates compelling generalization performance. \n\nTo fully address your concern, we follow the suggestions and verify our model on a set of new and more diverse datasets. These include **LongVideo Bench, VideoMME (with subtitles), IntentQA, Next-QA, and Causal-VidQA**. Results are:\n\n|Model|LongVideo Bench|VideoMME (with subtitles)|IntentQA|Next-QA|Causal-VidQA|\n|:-|:-:|:-:|:-:|:-:|:-:|\n|Qwen2.5-VL|36.0|55.5|86.3|75.6|70.2|67.0|\n|Video-R1|50.2|62.1|88.2|78.3|61.7|56.0|\n|VideoRFT|54.7|63.3|92.7|80.1|71.6|67.0|\n\n\nThe significant improvements of VideoRFT against Qwen2.5-VL  confirms that **fine-tuning substantially broadens the baseline's capabilities in video reasoning**. Moreover, the results also show that our model  exhibits stronger generalization compared to Video-R1.\n\n\n\n> Q4. Your model is trained using inputs of 128 × 28 × 28 resolution. In Table 1, are the baseline models trained under comparable input settings? If not, how is the comparison fair?\n\nApologize for the confusion. We clarify that there are two commonly used input resolution settings in the competitors. On one hand, same to us, models such as Video-R1, VILA, and LLaMA-VID use 128 × 28 × 28 resolution (equivalent to 224 × 224). On the other hand, some baselines, including TinyLLaVA-Video-R1, mPLUG-Owl3, LLaVA-OneVision, and VideoLLaMA2, use a higher resolution of 384 × 384. \n\nTherefore, the comparison is fair and, if anything, conservative, as some competing methods benefit from larger input resolutions than ours.\n\n[ref1] Vision-R1: Incentivizing Reasoning Capability in Multimodal Large Language Models, Arxiv 2025\n\n[ref2] R1-Onevision: Advancing Generalized Multimodal Reasoning through Cross-Modal Formalization, Arxiv 2025\n\n[ref3] Video-R1: Reinforcing Video Reasoning in MLLMs, Arxiv 2025"}]

Author rebuttal:

Meta-review: The manuscript investigates reinforcement fine-tuning (RFT) for video reasoning. The proposed approach, named VideoRFT, follows the two-stage scheme in RFT (SFT with CoT and online RL). To this end, the manuscript builds a an automatic CoT curation pipeline, resulting in two new datasets: VideoRFT-CoT-102K for SFT and VideoRFT-RL-310K for RL. To consolidate the RL phase, the manuscript introduces a semantic-consistency reward that aims to improve the alignment between textual reasoning and visual evidence. The manuscript received ratings of three borderline accept and one borderline reject. Reviewers appreciated the proposed dataset and automated pipeline for generating CoT data, semantic-consistency reward. Reviewers also raised several questions including, the need for fine-tuning compared to prompting, the generalization ability of the fine-tuned model, and computational overhead during training due to SigCLIP embeddings. Post-rebuttal, most reviewers expressed thaat the initial concerns are generally resolved (e.g., questions regarding fine-tuning compared to prompting, generalization capabilities of fine-tuned model), but Reviewer 3mwX still questioned the provided computational overhead comparison with SigCLIP embeddings. This was addressed in the follow-up response from the authors, where they provided rectified computational overhead (e.g., logs) and showed no significant computational overhead. Given that three reviewers are positive, the provided rebuttal, and author-reviewer discussions, the recommendation is accept.
Decision: Accept (poster)