ReWatch-R1: Boosting Complex Video Reasoning in Large Vision-Language Models through Agentic Data Synthesis

## Abstract

Abstract While Reinforcement Learning with Verifiable Reward (RLVR) significantly advances image reasoning in Large Vision-Language Models (LVLMs), its application to complex video reasoning remains underdeveloped. This gap stems primarily from a critical data bottleneck: existing datasets lack the challenging, multi-hop questions and high-quality, video-grounded Chain-of-Thought (CoT) data necessary to effectively bootstrap RLVR. To address this, we introduce ReWatch , a large-scale dataset built to foster advanced video reasoning. We propose a novel multi-stage synthesis pipeline to synthesize its three components: ReWatch-Caption , ReWatch-QA , and ReWatch-CoT . A core innovation is our Multi-Agent ReAct framework for CoT synthesis, which simulates a human-like "re-watching" process to generate video-grounded reasoning traces by explicitly modeling information retrieval and verification. Building on this dataset, we develop ReWatch-R1 by post-training a strong baseline LVLM with Supervised Fine-Tuning (SFT) and our RLVR framework. This framework incorporates a novel Observation & Reasoning (O&R) reward mechanism that evaluates both the final answer’s correctness and the reasoning’s alignment with video content, directly penalizing hallucination. Our experiments show that ReWatch-R1 achieves state-of-the-art performance on five challenging video reasoning benchmarks. Project Page.

## 1 Introduction

While the training paradigm of Supervised Fine-Tuning (SFT) combined with Reinforcement Learning with Verifiable Reward (RLVR) [16; 35] significantly advances image reasoning in Large Vision-Language Models (LVLMs) [45; 20; 44], its application to complex video reasoning remains nascent. Recent open-source video models [14; 25; 38; 9; 32] trained with SFT+RLVR still underperform on high-difficulty benchmarks, especially for multi-step temporal tasks such as causality, state tracking, and counting events across long videos [33; 31; 10; 34; 29].

Recent efforts to apply the SFT+RLVR paradigm to video [14; 25; 38; 9; 32] typically bootstrap the SFT phase with CoT data synthesized from existing simple video QA datasets, before applying RLVR. However, this approach is fundamentally undermined by the quality of the underlying data. As illustrated in Figure [2](https://arxiv.org/html/2509.23652v2#S1.F2)(left), prevailing open-source data [14] suffers from three flaws: (1) holistic, untimestamped captions that erase temporal structure; (2) simple, perception-based QA that can be answered from short clips or textual priors; and
(3) visually unfaithful CoT that relies on commonsense knowledge and process of elimination.
This data bottleneck prevents SFT from teaching true video-grounded reasoning, and the subsequent RL phase, lacking a reliable reward signal for process correctness, struggles to penalize hallucination and improve logical fidelity [11; 21].

Figure: Figure 2: A comparative of ReWatch dataset and Video-R1 dataset on the same source video.
Refer to caption: x2.png

To address these limitations, we introduce ReWatch, a large-scale dataset explicitly designed to foster advanced video reasoning.
ReWatch is constructed through a multi-stage synthesis pipeline and comprises three tightly coupled components: ReWatch-Caption, ReWatch-QA, and ReWatch-CoT.
First, ReWatch-Caption provides temporally dense video descriptions. We employ a hierarchical captioning method to generate detailed, timestamped narratives that form a high-fidelity foundation for complex reasoning.
Second, ReWatch-QA features high-difficulty question-answer pairs. We use a contrastive generation strategy, creating questions from detailed captions that cannot be answered by concise summaries, and apply a three-tier filter to guarantee video dependency.
Finally, ReWatch-CoT promotes video-grounded reasoning. We employ a novel Multi-Agent ReAct framework to synthesize CoT that simulates a human-like "re-watching" process.
This generates reasoning traces that explicitly document information retrieval and verification against the video content. As shown in Figure [2](https://arxiv.org/html/2509.23652v2#S1.F2)(right), our ReWatch data delivers high-fidelity captions, high-difficulty QAs, and video-grounded CoTs.

Building on ReWatch, we post-train a strong LVLM in two stages to obtain ReWatch-R1. After an initial SFT phase that teaches step-by-step reasoning, we employ RLVR augmented with a novel Observation & Reasoning (O&R) reward. Unlike rewards that score only the final answer, O&R also evaluates whether intermediate observations are factually supported by the video and whether the reasoning is sufficient to recover the correct answer from those observations. This dual emphasis on process and outcome explicitly incentivizes verifiable, evidence-linked reasoning, reducing hallucinations and improving logical consistency. As summarized in Figure [1](https://arxiv.org/html/2509.23652v2#S0.F1), ReWatch-R1 sets new state of the art on five challenging video reasoning benchmarks, substantially outperforming models trained on alternative open-source data.

In summary, our contributions are:

- •
A novel, multi-stage agentic pipeline for synthesizing a large-scale, high-quality video reasoning dataset (ReWatch).
- •
A new Observation & Reasoning (O&R) reward for RLVR that improves reasoning by rewarding both final-answer correctness and the factual grounding of intermediate steps in video content.
- •
ReWatch-R1, a post-trained LVLM that achieves state-of-the-art results on five complex video reasoning benchmarks.

## 2 Data Construction: The ReWatch Dataset

To address the above data bottlenecks, we introduce ReWatch, a large, high-fidelity, high-difficulty, and video-grounded dataset for advanced video reasoning.
As shown in Figure [3](https://arxiv.org/html/2509.23652v2#S2.F3), it is constructed in three stages: Hierarchical Video Captioning, High-Difficulty QA Generation, and Multi-Agent CoT Synthesis.
The dataset contains 10k captions, 170k QA pairs, and 135k CoTs.
More details and statistics are in Appendix [A](https://arxiv.org/html/2509.23652v2#A1).

### 2.1 Stage 1: Hierarchical Video Captioning

To address the hallucination issue in LVLMs when processing long videos and to generate high-fidelity video descriptions, we propose a Hierarchical Dynamic Frame-Rate Generation pipeline for our ReWatch-Caption-10k dataset. The process is applied to our video corpus $\mathcal{V}$, sourced from five public datasets [22; 17; 27; 14; 47].

#### Semantic Segmentation.

For each video $V\in\mathcal{V}$, we first partition $V$ into $k$ semantically coherent segments $S$ using LVLM $\mathcal{M}_{\text{seg}}$, at a low-frame-rate. Each segment $s_{i}$ corresponds to a temporal interval $[t_{i}^{\text{start}},t_{i}^{\text{end}}]$, preserving event integrity.

$$ $\displaystyle S=\{s_{1},\dots,s_{k}\}=\mathcal{M}_{\text{seg}}(V)$ (1) $$

#### Detailed Description Generation.

We use a powerful LVLM $\mathcal{M}_{\text{cap}}$ to process each segment $s_{i}$ at a high frame rate and generate a detailed description $D_{i}^{\text{rel}}$, which includes $m_{i}$ distinct events $\{c_{ij}\}$ along with their relative timestamps $\{\tau_{ij}\}$.

$$ $\displaystyle D_{i}^{\text{rel}}=\{(c_{ij},\tau_{ij})\}_{j=1}^{m_{i}}=\mathcal{M}_{\text{cap}}(s_{i})$ (2) $$

#### Timestamp Realignment.

Finally, a function $\mathcal{P}$ converts relative timestamps $\tau_{ij}$ to absolute ones $t_{ij}$ by adding the segment’s start time.

$$ $\displaystyle t_{ij}=\mathcal{P}(\tau_{ij},t_{i}^{\text{start}})=t_{i}^{\text{start}}+\tau_{ij}$ (3) $$

The final video caption $C_{\text{detail}}(V)$ is the union of all timestamped descriptions.

$$ $\displaystyle C_{\text{detail}}(V)=\bigcup_{i=1}^{k}\{(c_{ij},t_{ij})\}_{j=1}^{m_{i}}$ (4) $$

This hierarchical approach generates temporally precise and semantically rich descriptions while avoiding the hallucination issues associated with LVLMs processing long videos.

### 2.2 Stage 2: High-Difficulty QA Pair Generation

Figure: Figure 3: The data construction pipeline. (a) Caption Construction. Long videos are semantically segmented to produce detailed, temporally-aware captions. (b) QA Pair Generation. A contrastive method using detailed and summary captions generates complex questions, which are then purified by a three-layer filtering mechanism. (c) CoT Synthesis. A ReAct framework with a Reasoner Agent and an Observer Agent simulates a "re-watching" process by performing targeted queries on the video caption to generate video-grounded reasoning traces.
Refer to caption: x3.png

To create our ReWatch-QA-170k dataset, we design a pipeline to generate challenging QA pairs requiring fine-grained video analysis. It combines Contrastive Prompting with Three-Layer Filtering.

#### Contrastive QA Generation.

Given a detailed caption $C_{\text{detail}}$, we first generate a concise summary $C_{\text{sum}}=\mathcal{M}_{\text{sum}}(C_{\text{detail}})$ using a lightweight LLM. Then, inspired by previous work [58; 5], our QA generator $\mathcal{M}_{\text{qa}}$ processes both $C_{\text{detail}}$ and $C_{\text{sum}}$ to create QA pairs $(Q,A)$ that are explicitly answerable from the detailed caption but not from the summary alone. This ensures questions probe fine-grained details while excluding trivial ones.

$$ $\displaystyle(Q,A)_{\text{raw}}=\mathcal{M}_{\text{qa}}(C_{\text{detail}},C_{\text{sum}})$ (5) $$

To guide generation and ensure diversity, we pre-define 10 question types.

#### Three-Layer Filtering.

Raw pairs undergo a three-layer filtering cascade to ensure quality and video-dependency:

- •
Filter 1: Answer Verification, $\mathcal{F}_{1}$: A verifier $\mathcal{M}_{\text{verify}}$ confirms the factual correctness of the answer based on $C_{\text{detail}}$.
$\displaystyle(Q,A)\text{ passes }\mathcal{F}_{1}\iff\mathcal{M}_{\text{verify}}(Q,A,C_{\text{detail}})=\text{True}$
(6)
- •
Filter 2: Text Bias Elimination, $\mathcal{F}_{2}$: Ensures the question is unanswerable from general knowledge by probing a set of LLMs $\mathbb{M}_{\text{probe}}$.
$\displaystyle(Q,A)\text{ passes }\mathcal{F}_{2}\iff\frac{1}{|\mathbb{M}_{\text{probe}}|}\sum_{\mathcal{M}\in\mathbb{M}_{\text{probe}}}\mathbf{1}(\mathcal{M}(Q)\approx A)<\theta_{\text{text}}$
(7)
- •
Filter 3: Summary Bias Elimination, $\mathcal{F}_{3}$: Similarly ensures the question is unanswerable using the summary $C_{\text{sum}}$.
$\displaystyle(Q,A)\text{ passes }\mathcal{F}_{3}\iff\frac{1}{|\mathbb{M}_{\text{probe}}|}\sum_{\mathcal{M}\in\mathbb{M}_{\text{probe}}}\mathbf{1}(\mathcal{M}(Q,C_{\text{sum}})\approx A)<\theta_{\text{sum}}$
(8)

Where $\theta_{\text{text}}$ and $\theta_{\text{sum}}$ are threshold for consensus.
The 85k pairs passing all filters are then rewritten by LLM $\mathcal{M}_{\text{rewrite}}$ into multiple-choice questions, yielding a total of 170k QA pairs.

### 2.3 Stage 3: Multi-Agent Chain-of-Thought Synthesis

To generate our ReWatch-CoT-135k dataset, we introduce a multi-agent ReAct-based framework that explicitly construct the video-grounded CoT. This method externalizes the observation process for active information retrieval.

We define two agents: a Reasoner $\mathcal{A}_{R}$ that produces thoughts $T$ and actions $Act$, and an Observer $\mathcal{A}_{O}$ that executes actions on the video caption $C_{\text{detail}}$ to return observations $Obs$.

For a given question $Q$, the agents interact in a loop. At each step $t$, the Reasoner uses the history $H_{t-1}=(Q,T_{1},Act_{1},Obs_{1},\dots,T_{t-1},Act_{t-1},Obs_{t-1})$ to decide the next step:

$$ $\displaystyle(T_{t},Act_{t})=\mathcal{A}_{R}(H_{t-1})$ (9) $$

The Observer executes the action to retrieve information from the video context:

$$ $\displaystyle Obs_{t}=\mathcal{A}_{O}(Act_{t},C_{\text{detail}})$ (10) $$

This process continues until the Reasoner produces a final answer. The core actions $Act_{t}$ simulate visual lookup:

- •
segment_retrieval(query): Finds the timestamp of an event from a natural language query.
- •
segment_query(timestamp): Retrieves the detailed description of an event from a timestamp.

This entire text-based simulation is highly efficient. The structured execution trajectory $\mathcal{T}=\{(T_{1},Act_{1},Obs_{1}),\dots,(A_{\text{final}})\}$ is then converted by LLM $\mathcal{M}_{\text{convert}}$ into a natural language CoT string $\mathcal{R}$ with explicit <action> and <observation> tags, making it ready for supervised fine-tuning and O&R reward calculation.

## 3 Post-Traing on ReWatch Dataset

As shown in Figure [4](https://arxiv.org/html/2509.23652v2#S3.F4), we use the SFT+RL paradigm to train Qwen2.5-VL. In the SFT stage, we use multi-task objectives to train to obtain ReWatch-R1-SFT. In the RL stage, based on the GRPO [16] algorithm and a novel O&R reward mechanism we propos, we obtain ReWatch-R1.

Figure: Figure 4: Our two-stage Post-Training framework. (a) A Base Model is first fine-tuned (SFT) on all ReWatch datasets, (b) then further refined as a policy via Reinforcement Learning (RL) using the ReWatch-QA dataset. (c) The "Rollout" panel illustrates the generative process of the policy: producing a purely textual chain-of-thought that simulates a Thought-Action-Observation reasoning loop through self-generated text segments. (d) We employ four verifiable reward mechanisms.
Refer to caption: x4.png

### 3.1 Supervised Fine-Tuning Stage

In this stage, we perform multi-task SFT on a base LVLM using our three datasets: ReWatch-Caption-10k ($\mathcal{D}_{\text{Cap}}$), ReWatch-QA-170k ($\mathcal{D}_{\text{QA}}$), and ReWatch-CoT-135k ($\mathcal{D}_{\text{CoT}}$). The goal is to jointly instill three core abilities: foundational video-text alignment, direct question-answering ("non-thinking" mode), and step-by-step reasoning ("thinking" mode). Crucially, we train the model to switch between these response modes using distinct instruction prompts. For detailed prompt setting during SFT, please refer to Appendix [D.2](https://arxiv.org/html/2509.23652v2#A4.SS2).

The SFT objective is to minimize a composite loss function, $\mathcal{L}_{\text{SFT}}$, which is the sum of the losses from these three tasks. Let the LVLM be denoted by a policy $\pi_{\theta}$ with parameters $\theta$. The total loss is defined as:

$$ $\displaystyle\mathcal{L}_{\text{SFT}}(\theta)=\mathcal{L}_{\text{Cap}}+\mathcal{L}_{\text{QA}}+\mathcal{L}_{\text{CoT}}$ (11) $$

where each component corresponds to a specific learning objective:

#### Video-Text Alignment.

We train the model to generate detailed captions ($C_{\text{detail}}$) from videos ($V$).

$$ $\displaystyle\mathcal{L}_{\text{Cap}}=-\mathbb{E}_{(V,C_{\text{detail}})\in\mathcal{D}_{\text{Cap}}}[\log\pi_{\theta}(C_{\text{detail}}|V)]$ (12) $$

#### Direct Question-Answering (Non-thinking).

We train the model to output a concise answer ($A$) when given a direct-answer instruction $I_{\text{direct}}$.

$$ $\displaystyle\mathcal{L}_{\text{QA}}=-\mathbb{E}_{(V,Q,A)\in\mathcal{D}_{\text{QA}}}[\log\pi_{\theta}(A|V,I_{\text{direct}},Q)]$ (13) $$

#### Chain-of-Thought Reasoning (Thinking).

We train the model to generate the full reasoning trace ($\mathcal{R}$) when given a think-step-by-step instruction $I_{\text{think}}$.

$$ $\displaystyle\mathcal{L}_{\text{CoT}}=-\mathbb{E}_{(V,Q,\mathcal{R})\in\mathcal{D}_{\text{CoT}}}[\log\pi_{\theta}(\mathcal{R}|V,I_{\text{think}},Q)]$ (14) $$

By optimizing these objectives concurrently, we produce a versatile SFT Model that is proficient in both direct answering and complex reasoning. This model then serves as the proficient initial policy for the subsequent Reinforcement Learning stage.

### 3.2 Reinforcement Learning Stage

Previous LVLMs of video reasoning [9; 14] directly utilize the accuracy of the final answer $r_{acc}$ as the reward signal for reasoning enhancement through reinforcement learning. Formally,

$$ $\displaystyle r_{acc}=\mathcal{M}_{judge}(A,A_{gt}),$ (15) $$

where $\mathcal{M}_{judge}(\cdot)$ is the judge model used to assess the consistency of inputs, which can be a rule-based verifier or an LLM.
However, the foundation of video reasoning lies in the ability to reason grounded in video content.
Such reward for mere accuracy overlooks the capabilities of video content-oriented reasoning, which may lead to potential visual or linguistic hallucinations.
To address this limitation, we design the Observation & Reasoning (O&R) reward mechanism, which encourages the model to perform appropriate reasoning grounded in the accurate understanding of video content, rather than relying on potential visual or linguistic hallucinations.
Specifically, we model the video reasoning QA process as a sequential flow:

Video+Question $\rightarrow$ Observations+Reasoning $\rightarrow$ Answer

On one hand, the model should base its reasoning on accurate observations of the video content.
Thus, we first assess the accuracy of video observations in CoT by comparing them with the detailed video caption, and use this evaluation as the observation reward.
Formally,

$$ $\displaystyle\{Act_{i},Obs_{i}\}_{i=1}^{N}={\rm Parse}(\mathcal{R}),$ (16) $\displaystyle r_{obs}={\rm mean}(\{\mathcal{M}_{judge}(C_{detail},\{Act_{i},Obs_{i}\})\}_{i=1}^{N}).$ (17) $$

Here, ${\rm Parse}(\cdot)$ denotes parsing the actions and observations from the model output.

On the other hand, the model should reason out appropriate observational actions according to the question.
Therefore, we design the reasoning reward by evaluating the accuracy of directly answering questions using the actions and observations.
If the model can provide a correct answer based on these actions and observations, the reasoning process is deemed valid and sufficient. This reward guides the model to reason appropriate observation actions that effectively address the question.
Formally,

$$ $\displaystyle A_{ao}=\mathcal{M}_{infer}(Q,\{Act_{i},Obs_{i}\}_{i=1}^{N}),$ (18) $\displaystyle r_{rea}=\mathcal{M}_{judge}(A_{ao},A_{gt}).$ (19) $$

Here, $\mathcal{M}_{infer}(\cdot)$ is an LLM used to answer the question based on the given actions and observations.
The final reward can be expressed as,

$$ $\displaystyle r_{O\&R}=r_{acc}\times(1+r_{obs}+r_{rea})+r_{fmt},$ (20) $\displaystyle r_{fmt}=\begin{cases}1,&\text{correct format}\\ 0.&\text{otherwise}\end{cases}$ (21) $$

Here, $r_{fmt}$ denotes the format reward, enabling the model to output responses in the format we desire. For example, we expect the model to enclose its actions and observations with <action>...</action> and <observation>...</observation> tags, and the answer with <answer>...</answer> tag. Finally, we employ the GRPO [16] algorithm for model optimization.

## 4 Experiments

We train Qwen2.5-VL-7B [4] on the ReWatch dataset to obtain Rewatch-R1, and then compare it with other LVLMs on five video reasoning and four video understanding benchmarks. For detailed experimental settings, please refer to the Appendix [B.1](https://arxiv.org/html/2509.23652v2#A2.SS1).

### 4.1 Main Results

Table: Table 1: Performance comparison on Video Reasoning tasks. ^∗ indicates that we reproduced the model using a training configuration with 192 frames. ^† indicates that reinforcement learning is conducted using exactly the same data as ReWatch-R1. The best results among models of the same size are indicated in bold.

Table [1](https://arxiv.org/html/2509.23652v2#S4.T1) shows the superior video reasoning performance of our model, yielding following key insights.

SOTA Performance among models of a comparable size.
In both 192-frame and 384-frame settings, the average scores of ReWatch-R1 across five reasoning benchmarks significantly surpass those of all other comparison models. This validates the effectiveness of our dataset and training methodology.

High-Quality CoT Data is Critical.
The SFT-only model ReWatch-R1-SFT (33.25%) already surpasses most competitors like Video-R1-SFT (29.74%) and LongVideoReason-SFT (26.31%), which use the same training configuration. This proves the superiority of our CoT training data.

RL Unlocks Further Potential.
Reinforcement learning further boosts performance. Our final ReWatch-R1 model improves upon the SFT version (33.25% to 35.51%). This shows that while SFT teaches the form of CoT, our RL phase imparts the spirit, enabling more logical and factually grounded reasoning.

The Efficacy of "Thinking" is Contingent on Learning "How to Think".
Enabling CoT ("Thinking" mode) is detrimental for an untrained base model (27.54% vs. 30.71%), as it can induce hallucinations. In contrast, our fully trained ReWatch-R1 excels with CoT. This proves our method successfully teaches the model how to reason.

We further evaluate performance on video understanding benchmarks in Table [3](https://arxiv.org/html/2509.23652v2#A2.T3) and performance on videos of varying durations in Figure [9](https://arxiv.org/html/2509.23652v2#A2.F9).
For detailed analysis, please refer to Appendix [B.2](https://arxiv.org/html/2509.23652v2#A2.SS2) and [B.3](https://arxiv.org/html/2509.23652v2#A2.SS3).

### 4.2 Analysis Results

Figure: (a) Ablation results using different CoT data.
Refer to caption: x5.png

High-Quality SFT Data is Foundational for RL.
An ablation study in Figure [5(a)](https://arxiv.org/html/2509.23652v2#S4.F5.sf1) shows two key findings. First, SFT is an indispensable prerequisite for RL, training without it (w/o SFT) causes a catastrophic performance drop, as RL needs a strong initial policy. Second, high-quality CoT data is vital. Replacing our ReWatch-CoT data with that from Video-R1 significantly degrades performance. This validates that our multi-agent framework produces a superior training corpus for complex reasoning.

High-quality QA data is crucial for RL.
A comparative analysis in Figure [5(b)](https://arxiv.org/html/2509.23652v2#S4.F5.sf2) shows that the quality of QA data used for RL determines final performance. Training on only baseline QA data (Video-R1-QA [14] (10k) and LongVideoReason-QA [9] (10k)) yields the lowest scores (42.0% all, 34.3% reasoning, 51.7% understanding), whereas our ReWatch-QA data provides notable improvements. This confirms that ReWatch-QA, due to its challenging nature, offers a more potent reward signal that guides the model toward robust reasoning abilities instead of overfitting to simpler patterns.

Figure: (a) A comparison of QA complexity.
Refer to caption: x7.png

Dataset Complexity & Video Dependency.
Figure [6(a)](https://arxiv.org/html/2509.23652v2#S4.F6.sf1) presents a quantitative analysis of the complexity comparison between the ReWatch-QA and Video-R1-QA datasets. The detailed experimental design can be found in Appendix [B.4](https://arxiv.org/html/2509.23652v2#A2.SS4). The results show that the ReWatch-QA dataset elicits more profound reasoning than Video-R1-QA. ReWatch requires nearly double the reasoning steps (3.31 vs. 1.82) and significantly longer responses (398.75 vs. 205.74). Critically, Video-R1 has a high Text-Only Accuracy of 68.9%, indicating questions are often solvable from text alone. In contrast, the accuracy of ReWatch is only 29.4%, near the 25% random-guess baseline. This proves our three-stage filtering is effective, eliminating textual shortcuts and forcing genuine video understanding.

RL optimizes the reasoning process, leading to more efficient yet more accurate responses.
Figure [6(b)](https://arxiv.org/html/2509.23652v2#S4.F6.sf2) shows a two-stage evolution. First, SFT teaches the model a structured reasoning format, increasing action counts and accuracy. Then, during RL, accuracy continues to improve while the average number of actions decreases. This indicates RL refines the policy to be more effective and efficient, pruning redundant steps to focus on critical actions. The model thus transitions from learning reasoning’s form (SFT) to mastering its function with efficiency (RL).

Figure: Figure 7: Impact of SFT and RL on different prompting methods. The plots show the accuracy of our ReWatch-R1 model with "thinking" (ReAct) vs. "non-thinking" (direct answering) prompting. Solid lines show performance progression during the SFT phase, dashed lines show the final performance after RL.
Refer to caption: x9.png

The thinking mode, while converging more slowly during training, ultimately achieves a significantly higher performance ceiling than the non-thinking mode.
As shown in Figure [7](https://arxiv.org/html/2509.23652v2#S4.F7), the two modes exhibit different learning dynamics. During the SFT phase (solid lines), the direct-answer "non-thinking" mode improves rapidly, whereas the "thinking" mode develops slowly. This suggests SFT primarily teaches the format of reasoning, not its logic. The subsequent RL phase (dashed lines) acts as a catalyst, causing a dramatic performance leap in the thinking mode by forcing the model to learn the causal links between reasoning and correct answers. Ultimately, the final model’s "thinking" performance surpasses the "non-thinking" mode in all tasks. This empirically proves that an explicit, step-by-step reasoning process, cultivated via our SFT-RL regimen, is optimal for complex video tasks.

## 5 Conclusion

In this work, we address the critical data bottleneck in complex video reasoning by introducing ReWatch, a large-scale dataset synthesized via a novel multi-stage agentic pipeline that generates temporally-dense captions, challenging multi-hop questions, and video-grounded Chain-of-Thought traces. We then develop ReWatch-R1 by post-training a strong LVLM using an SFT and RLVR framework, featuring our innovative Observation & Reasoning (O&R) reward that uniquely evaluates both the correctness of the final answer and the factual grounding of the reasoning process itself. The resulting model establishes a new state-of-the-art on five challenging video reasoning benchmarks. This demonstrates that our integrated approach of superior data synthesis and process-oriented reinforcement learning provides a robust and effective paradigm for complex temporal reasoning in LVLMs.

## References

- Achiam et al. [2023]
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al.
Gpt-4 technical report.
*arXiv preprint arXiv:2303.08774*, 2023.
- Arnab et al. [2025]
Anurag Arnab, Ahmet Iscen, Mathilde Caron, Alireza Fathi, and Cordelia Schmid.
Temporal chain of thought: Long-video understanding by thinking in frames.
*arXiv preprint arXiv:2507.02001*, 2025.
- Bai et al. [2025a]
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin.
Qwen2.5-vl technical report.
*arXiv preprint arXiv:2502.13923*, 2025a.
- Bai et al. [2025b]
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al.
Qwen2. 5-vl technical report.
*arXiv preprint arXiv:2502.13923*, 2025b.
- Chen et al. [2025a]
Guo Chen, Zhiqi Li, Shihao Wang, Jindong Jiang, Yicheng Liu, Lidong Lu, De-An Huang, Wonmin Byeon, Matthieu Le, Tuomas Rintamaki, et al.
Eagle 2.5: Boosting long-context post-training for frontier vision-language models.
*arXiv preprint arXiv:2504.15271*, 2025a.
- Chen et al. [2024]
Jr-Jen Chen, Yu-Chien Liao, Hsi-Che Lin, Yu-Chu Yu, Yen-Chun Chen, and Frank Wang.
Rextime: A benchmark suite for reasoning-across-time in videos.
*Advances in Neural Information Processing Systems*, 37:28662–28673, 2024.
- Chen et al. [2025b]
Yi Chen, Yuying Ge, Rui Wang, Yixiao Ge, Junhao Cheng, Ying Shan, and Xihui Liu.
Grpo-care: Consistency-aware reinforcement learning for multimodal reasoning.
*arXiv preprint arXiv:2506.16141*, 2025b.
- Chen et al. [2025c]
Yi Chen, Yuying Ge, Rui Wang, Yixiao Ge, Lu Qiu, Ying Shan, and Xihui Liu.
Exploring the effect of reinforcement learning on video understanding: Insights from seed-bench-r1.
*arXiv preprint arXiv:2503.24376*, 2025c.
- Chen et al. [2025d]
Yukang Chen, Wei Huang, Baifeng Shi, Qinghao Hu, Hanrong Ye, Ligeng Zhu, Zhijian Liu, Pavlo Molchanov, Jan Kautz, Xiaojuan Qi, et al.
Scaling rl to long videos.
*arXiv preprint arXiv:2507.07966*, 2025d.
- Cheng et al. [2025]
Junhao Cheng, Yuying Ge, Teng Wang, Yixiao Ge, Jing Liao, and Ying Shan.
Video-holmes: Can mllm think like holmes for complex video reasoning?
*arXiv preprint arXiv:2505.21374*, 2025.
- Chu et al. [2025]
Xu Chu, Xinrong Chen, Guanyu Wang, Zhijie Tan, Kui Huang, Wenyu Lv, Tong Mo, and Weiping Li.
Qwen look again: Guiding vision-language reasoning models to re-attention visual information.
*arXiv preprint arXiv:2505.23558*, 2025.
- Comanici et al. [2025]
Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al.
Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities.
*arXiv preprint arXiv:2507.06261*, 2025.
- Fan et al. [2024]
Yue Fan, Xiaojian Ma, Rujie Wu, Yuntao Du, Jiaqi Li, Zhi Gao, and Qing Li.
Videoagent: A memory-augmented multimodal agent for video understanding.
In *European Conference on Computer Vision*, pp. 75–92. Springer, 2024.
- Feng et al. [2025]
Kaituo Feng, Kaixiong Gong, Bohao Li, Zonghao Guo, Yibing Wang, Tianshuo Peng, Junfei Wu, Xiaoying Zhang, Benyou Wang, and Xiangyu Yue.
Video-r1: Reinforcing video reasoning in mllms.
*arXiv preprint arXiv:2503.21776*, 2025.
- Fu et al. [2025]
Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al.
Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis.
In *Proceedings of the Computer Vision and Pattern Recognition Conference*, pp. 24108–24118, 2025.
- Guo et al. [2025]
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al.
Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.
*arXiv preprint arXiv:2501.12948*, 2025.
- Han et al. [2025]
Songhao Han, Wei Huang, Hairong Shi, Le Zhuo, Xiu Su, Shifeng Zhang, Xu Zhou, Xiaojuan Qi, Yue Liao, and Si Liu.
Videoespresso: A large-scale chain-of-thought dataset for fine-grained video reasoning via core frame selection.
In *Proceedings of the Computer Vision and Pattern Recognition Conference (CVPR)*, pp. 26181–26191, June 2025.
- Hu et al. [2025a]
Jian Hu, Zixu Cheng, Chenyang Si, Wei Li, and Shaogang Gong.
Cos: Chain-of-shot prompting for long video understanding.
*arXiv preprint arXiv:2502.06428*, 2025a.
- Hu et al. [2025b]
Kairui Hu, Penghao Wu, Fanyi Pu, Wang Xiao, Yuanhan Zhang, Xiang Yue, Bo Li, and Ziwei Liu.
Video-mmmu: Evaluating knowledge acquisition from multi-discipline professional videos.
*arXiv preprint arXiv:2501.13826*, 2025b.
- Huang et al. [2025]
Wenxuan Huang, Bohan Jia, Zijie Zhai, Shaosheng Cao, Zheyu Ye, Fei Zhao, Zhe Xu, Yao Hu, and Shaohui Lin.
Vision-r1: Incentivizing reasoning capability in multimodal large language models.
*arXiv preprint arXiv:2503.06749*, 2025.
- Jian et al. [2025]
Pu Jian, Junhong Wu, Wei Sun, Chen Wang, Shuo Ren, and Jiajun Zhang.
Look again, think slowly: Enhancing visual reflection in vision-language models.
*arXiv preprint arXiv:2509.12132*, 2025.
- Ju et al. [2024]
Xuan Ju, Yiming Gao, Zhaoyang Zhang, Ziyang Yuan, Xintao Wang, Ailing Zeng, Yu Xiong, Qiang Xu, and Ying Shan.
Miradata: A large-scale video dataset with long durations and structured captions.
*Advances in Neural Information Processing Systems*, 37:48955–48970, 2024.
- Kumar et al. [2024]
Somnath Kumar, Yash Gadhia, Tanuja Ganu, and Akshay Nambi.
Mmctagent: Multi-modal critical thinking agent framework for complex visual reasoning.
*arXiv preprint arXiv:2405.18358*, 2024.
- Lee et al. [2025]
Daeun Lee, Jaehong Yoon, Jaemin Cho, and Mohit Bansal.
Video-skill-cot: Skill-based chain-of-thoughts for domain-adaptive video reasoning.
*arXiv preprint arXiv:2506.03525*, 2025.
- Li et al. [2025a]
Xinhao Li, Ziang Yan, Desen Meng, Lu Dong, Xiangyu Zeng, Yinan He, Yali Wang, Yu Qiao, Yi Wang, and Limin Wang.
Videochat-r1: Enhancing spatio-temporal perception via reinforcement fine-tuning.
*arXiv preprint arXiv:2504.06958*, 2025a.
- Li et al. [2025b]
Zongxia Li, Wenhao Yu, Chengsong Huang, Rui Liu, Zhenwen Liang, Fuxiao Liu, Jingxi Che, Dian Yu, Jordan Boyd-Graber, Haitao Mi, et al.
Self-rewarding vision-language model via reasoning decomposition.
*arXiv preprint arXiv:2508.19652*, 2025b.
- Lin et al. [2025]
Jingyang Lin, Jialian Wu, Ximeng Sun, Ze Wang, Jiang Liu, Yusheng Su, Xiaodong Yu, Hao Chen, Jiebo Luo, Zicheng Liu, et al.
Unleashing hour-scale video training for long video-language understanding.
*arXiv preprint arXiv:2506.05332*, 2025.
- Liu et al. [2025]
Ye Liu, Kevin Qinghong Lin, Chang Wen Chen, and Mike Zheng Shou.
Videomind: A chain-of-lora agent for long video reasoning.
*arXiv preprint arXiv:2503.13444*, 2025.
- Lu et al. [2025]
Lidong Lu, Guo Chen, Zhiqi Li, Yicheng Liu, and Tong Lu.
Av-reasoner: Improving and benchmarking clue-grounded audio-visual counting for mllms.
*arXiv preprint arXiv:2506.05328*, 2025.
- Min et al. [2024]
Juhong Min, Shyamal Buch, Arsha Nagrani, Minsu Cho, and Cordelia Schmid.
Morevqa: Exploring modular reasoning models for video question answering.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pp. 13235–13245, 2024.
- Nagrani et al. [2025]
Arsha Nagrani, Sachit Menon, Ahmet Iscen, Shyamal Buch, Ramin Mehran, Nilpa Jha, Anja Hauth, Yukun Zhu, Carl Vondrick, Mikhail Sirotenko, et al.
Minerva: Evaluating complex video reasoning.
*arXiv preprint arXiv:2505.00681*, 2025.
- Park et al. [2025]
Jinyoung Park, Jeehye Na, Jinyoung Kim, and Hyunwoo J Kim.
Deepvideo-r1: Video reinforcement fine-tuning via difficulty-aware regressive grpo.
*arXiv preprint arXiv:2506.07464*, 2025.
- Qi et al. [2025]
Yukun Qi, Yiming Zhao, Yu Zeng, Xikun Bao, Wenxuan Huang, Lin Chen, Zehui Chen, Jie Zhao, Zhongang Qi, and Feng Zhao.
Vcr-bench: A comprehensive evaluation framework for video chain-of-thought reasoning.
*arXiv preprint arXiv:2504.07956*, 2025.
- Rasheed et al. [2025]
Hanoona Rasheed, Abdelrahman Shaker, Anqi Tang, Muhammad Maaz, Ming-Hsuan Yang, Salman Khan, and Fahad Shahbaz Khan.
Videomathqa: Benchmarking mathematical reasoning via multimodal understanding in videos.
*arXiv preprint arXiv:2506.05349*, 2025.
- Shao et al. [2024]
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al.
Deepseekmath: Pushing the limits of mathematical reasoning in open language models.
*arXiv preprint arXiv:2402.03300*, 2024.
- Shi et al. [2024]
Yudi Shi, Shangzhe Di, Qirui Chen, and Weidi Xie.
Enhancing video-llm reasoning via agent-of-thoughts distillation.
*arXiv preprint arXiv:2412.01694*, 2024.
- Team et al. [2025]
V Team, Wenyi Hong, Wenmeng Yu, Xiaotao Gu, Guo Wang, Guobing Gan, Haomiao Tang, Jiale Cheng, Ji Qi, Junhui Ji, Lihang Pan, Shuaiqi Duan, Weihan Wang, Yan Wang, Yean Cheng, Zehai He, Zhe Su, Zhen Yang, Ziyang Pan, Aohan Zeng, Baoxu Wang, Bin Chen, Boyan Shi, Changyu Pang, Chenhui Zhang, Da Yin, Fan Yang, Guoqing Chen, Jiazheng Xu, Jiale Zhu, Jiali Chen, Jing Chen, Jinhao Chen, Jinghao Lin, Jinjiang Wang, Junjie Chen, Leqi Lei, Letian Gong, Leyi Pan, Mingdao Liu, Mingde Xu, Mingzhi Zhang, Qinkai Zheng, Sheng Yang, Shi Zhong, Shiyu Huang, Shuyuan Zhao, Siyan Xue, Shangqin Tu, Shengbiao Meng, Tianshu Zhang, Tianwei Luo, Tianxiang Hao, Tianyu Tong, Wenkai Li, Wei Jia, Xiao Liu, Xiaohan Zhang, Xin Lyu, Xinyue Fan, Xuancheng Huang, Yanling Wang, Yadong Xue, Yanfeng Wang, Yanzi Wang, Yifan An, Yifan Du, Yiming Shi, Yiheng Huang, Yilin Niu, Yuan Wang, Yuanchang Yue, Yuchen Li, Yutao Zhang, Yuting Wang, Yu Wang, Yuxuan Zhang, Zhao Xue, Zhenyu Hou, Zhengxiao Du, Zihan Wang, Peng Zhang, Debing Liu, Bin Xu, Juanzi Li,
Minlie Huang, Yuxiao Dong, and Jie Tang.
Glm-4.5v and glm-4.1v-thinking: Towards versatile multimodal reasoning with scalable reinforcement learning, 2025.
URL [https://arxiv.org/abs/2507.01006](https://arxiv.org/abs/2507.01006).
- Wang et al. [2025a]
Qi Wang, Yanrui Yu, Ye Yuan, Rui Mao, and Tianfei Zhou.
Videorft: Incentivizing video reasoning capability in mllms via reinforced fine-tuning.
*arXiv preprint arXiv:2505.12434*, 2025a.
- Wang et al. [2024a]
Weihan Wang, Zehai He, Wenyi Hong, Yean Cheng, Xiaohan Zhang, Ji Qi, Xiaotao Gu, Shiyu Huang, Bin Xu, Yuxiao Dong, et al.
Lvbench: An extreme long video understanding benchmark.
*arXiv preprint arXiv:2406.08035*, 2024a.
- Wang et al. [2025b]
Weiyun Wang, Zhangwei Gao, Lixin Gu, Hengjun Pu, Long Cui, Xingguang Wei, Zhaoyang Liu, Linglin Jing, Shenglong Ye, Jie Shao, et al.
Internvl3.5: Advancing open-source multimodal models in versatility, reasoning, and efficiency.
*arXiv preprint arXiv:2508.18265*, 2025b.
- Wang et al. [2024b]
Xiaohan Wang, Yuhui Zhang, Orr Zohar, and Serena Yeung-Levy.
Videoagent: Long-form video understanding with large language model as agent.
In *European Conference on Computer Vision*, pp. 58–76. Springer, 2024b.
- Wang et al. [2024c]
Yan Wang, Yawen Zeng, Jingsheng Zheng, Xiaofen Xing, Jin Xu, and Xiangmin Xu.
Videocot: A video chain-of-thought dataset with active annotation tool.
*arXiv preprint arXiv:2407.05355*, 2024c.
- Wang et al. [2025c]
Zikang Wang, Boyu Chen, Zhengrong Yue, Yi Wang, Yu Qiao, Limin Wang, and Yali Wang.
Videochat-a1: Thinking with long videos by chain-of-shot reasoning.
*arXiv preprint arXiv:2506.06097*, 2025c.
- Wei et al. [2025a]
Lai Wei, Yuting Li, Kaipeng Zheng, Chen Wang, Yue Wang, Linghe Kong, Lichao Sun, and Weiran Huang.
Advancing multimodal reasoning via reinforcement learning with cold start.
*arXiv preprint arXiv:2505.22334*, 2025a.
- Wei et al. [2025b]
Yana Wei, Liang Zhao, Jianjian Sun, Kangheng Lin, Jisheng Yin, Jingcheng Hu, Yinmin Zhang, En Yu, Haoran Lv, Zejia Weng, et al.
Open vision reasoner: Transferring linguistic cognitive behavior for visual reasoning.
*arXiv preprint arXiv:2507.05255*, 2025b.
- Yang et al. [2025]
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al.
Qwen3 technical report.
*arXiv preprint arXiv:2505.09388*, 2025.
- Yang et al. [2024a]
Dongjie Yang, Suyuan Huang, Chengqiang Lu, Xiaodong Han, Haoxin Zhang, Yan Gao, Yao Hu, and Hai Zhao.
Vript: A video is worth thousands of words.
*Advances in Neural Information Processing Systems*, 37:57240–57261, 2024a.
- Yang et al. [2024b]
Zeyuan Yang, Delin Chen, Xueyang Yu, Maohao Shen, and Chuang Gan.
Vca: Video curious agent for long video understanding.
*arXiv preprint arXiv:2412.10471*, 2024b.
- Yao et al. [2023]
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao.
React: Synergizing reasoning and acting in language models.
In *International Conference on Learning Representations (ICLR)*, 2023.
- Yuan et al. [2025]
Huaying Yuan, Zheng Liu, Junjie Zhou, Hongjin Qian, Ji-Rong Wen, and Zhicheng Dou.
Videodeepresearch: Long video understanding with agentic tool using.
*arXiv preprint arXiv:2506.10821*, 2025.
- Zeng et al. [2024]
Xiangyu Zeng, Kunchang Li, Chenting Wang, Xinhao Li, Tianxiang Jiang, Ziang Yan, Songze Li, Yansong Shi, Zhengrong Yue, Yi Wang, et al.
Timesuite: Improving mllms for long video understanding via grounded tuning.
*arXiv preprint arXiv:2410.19702*, 2024.
- Zhang et al. [2025a]
Haoji Zhang, Xin Gu, Jiawen Li, Chixiang Ma, Sule Bai, Chubin Zhang, Bowen Zhang, Zhichao Zhou, Dongliang He, and Yansong Tang.
Thinking with videos: Multimodal tool-augmented reinforcement learning for long video reasoning.
*arXiv preprint arXiv:2508.04416*, 2025a.
- Zhang et al. [2025b]
Shuyi Zhang, Xiaoshuai Hao, Yingbo Tang, Lingfeng Zhang, Pengwei Wang, Zhongyuan Wang, Hongxuan Ma, and Shanghang Zhang.
Video-cot: A comprehensive dataset for spatiotemporal understanding of videos based on chain-of-thought.
*arXiv preprint arXiv:2506.08817*, 2025b.
- Zhang et al. [2025c]
Xiaoyi Zhang, Zhaoyang Jia, Zongyu Guo, Jiahao Li, Bin Li, Houqiang Li, and Yan Lu.
Deep video discovery: Agentic search with tool use for long-form video understanding.
*arXiv preprint arXiv:2505.18079*, 2025c.
- Zhang et al. [2025d]
Yongheng Zhang, Xu Liu, Ruihan Tao, Qiguang Chen, Hao Fei, Wanxiang Che, and Libo Qin.
Vitcot: Video-text interleaved chain-of-thought for boosting video understanding in large language models.
*arXiv preprint arXiv:2507.09876*, 2025d.
- Zhang et al. [2025e]
Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li.
Llava-video: Video instruction tuning with synthetic data.
*Transactions on Machine Learning Research*, 2025e.
- Zhao et al. [2025]
Yilun Zhao, Haowei Zhang, Lujing Xie, Tongyan Hu, Guo Gan, Yitao Long, Zhiyuan Hu, Weiyuan Chen, Chuhan Li, Zhijian Xu, et al.
Mmvu: Measuring expert-level multi-discipline video understanding.
In *Proceedings of the Computer Vision and Pattern Recognition Conference*, pp. 8475–8489, 2025.
- Zheng et al. [2025]
Ziwei Zheng, Michael Yang, Jack Hong, Chenxiao Zhao, Guohai Xu, Le Yang, Chao Shen, and Xing Yu.
Deepeyes: Incentivizing" thinking with images" via reinforcement learning.
*arXiv preprint arXiv:2505.14362*, 2025.

## Appendix A Details of Dataset Construction

### A.1 Dataset Statistic

Table [8](https://arxiv.org/html/2509.23652v2#A1.F8.fig1) and Figure [8](https://arxiv.org/html/2509.23652v2#A1.F8.fig1) provide detailed statistical and distribution information of our dataset. Tabale [4](https://arxiv.org/html/2509.23652v2#A4.T4) defines the 10 types of questions that we have manually defined.

Figure: Table 2: Statistics of our dataset.
Refer to caption: x10.png

### A.2 Model Settings for data synthesis

When synthesizing ReWatch-Caption, the Semantic Segmentation model $\mathcal{M}_{\text{seg}}$ and the Detailed Description Generation model $\mathcal{M}_{\text{cap}}$ are all Gemini2.5-Flash (Non-Thinking) [12].

When synthesizing ReWatch-QA, the Summary Generation model $\mathcal{M}_{\text{sum}}$ is Gemini2.5-Flash-Lite (Non-Thinking) [12]. The Contrastive QA Generation model $\mathcal{M}_{\text{qa}}$ is Gemini2.5-Flash (Thinking) [12]. The Answer Verification model $\mathcal{M}_{\text{verify}}$ is GPT4.1 [1]. The LLMs set $\mathbb{M}_{\text{probe}}$ for Text Bias Elimination and Summary Bias Elimination includes Qwen3-235B-A22B-Instruct [46] and Qwen2.5-VL-72B-Instruct [4]. Threshold $\theta_{text}$ and $\theta_{sum}$ are equal to 1. The rewritten model $\mathcal{M}_{\text{rewrite}}$ for multiple-choice questions is Gemini2.5-Flash (Non-Thinking).

When synthesizing ReWatch-CoT, Reasoner model $\mathcal{A}_{R}$ is Gemini2.5-Flash (Thinking) [12], and Observer model $\mathcal{A}_{O}$ is GPT4.1 [1]. The model $\mathcal{M}_{\text{convert}}$ used for converting structured trajectories is Gemini2.5-Flash-Lite (Non-Thinking).

## Appendix B Detailed Experiments

### B.1 Experimental Setup

#### Benchmarks

We evaluate the model on five video reasoning benchmarks (VCR Bench [33], MINERVA [31], Video Holmes [10], Video MathQA [34], CG-AV Counting [29]) and four video general understanding benchmarks (MMVU [57], LVBench [39], VideoMME [15], VideoMMMU [19]).

#### Training Dataset Configuration

Our primary model, ReWatch-R1, is derived from Qwen2.5-VL-7B-Instruct [4] via a two-stage training pipeline. First, we create an intermediate model, ReWatch-R1-SFT, by performing SFT using a mixture of three datasets: ReWatch-Caption, ReWatch-QA, and ReWatch-CoT. Subsequently, ReWatch-R1-SFT is further refined using RL to produce ReWatch-R1. The RL phase leverages a total of 40k QA pairs, which are randomly sampled from ReWatch-QA (20k), Video-R1-QA [14] (10k), and LongVideoReason-QA [9] (10k).

#### Training Parameter Configuration

In the SFT stage, the length of the model context is 16k. The default fps is 2.0, with a maximum sampling of 192 frames, and the maximum resolution of each frame is 128*28*28. The train batch_size (per device) to be 1 and the gradient cumulative to be 4. The learning rate is 1e-6, max_grad_norm is 1.0, and the optimizer is AdamW. The number of epochs is 10. 16 H800 Gpus are used.
In the RL stage, the length of the model context is 16k. The default fps is 2.0, with a maximum sampling of 192 frames. The maximum resolution of each frame is 128*28*28. The number of rollouts is 8. The sampling temperature is 0.8 and top_p is 0.9. Both train_batch_size and ppo_mini_batch_size are 14. ppo_micro_batch_size_per_gpu is 1. The learning rate is 1e-5, max_grad_norm is 5.0, and the optimizer is AdamW. The number of epoch is 1. 16 H800 Gpus are used.
In the reward mechanism of reinforcement learning, we use Qwen3-30B-A3B-Instruct [46] as inference model $\mathcal{M}_{infer}$ and judge model $\mathcal{M}_{judge}$.

#### Baselines

We compare the performance with that of the most advanced video reasoning models in the current literature, including Qwen2.5-VL-7B [3], GLM4.1V-9B [37], InternVL3.5-8B [40], Video-R1 [14], Video-Chat-R1 [25], VideoRFT [38].
In addition, We also use two open-source datasets, Video-R1-CoT [14] and LongVideoReason-CoT [9], to reproduce Video-R1-SFT and LongVideoReason-SFT under the same training configuration of ReWatch-R1-SFT.
The RL stage for Video-R1-RL and LongVideoReason-RL utilizes an identical dataset of 40k QA pairs with ReWatch-R1.

#### Evaluation

We employ GPT-4.1 [1] to assess if model responses align with ground truth using Prompt [18](https://arxiv.org/html/2509.23652v2#A4.F18), with accuracy as the metric for all benchmarks. During inference, the maximum resolution for each frame is limited to 128*28*28 pixels, and the maximum number of frames is 192 or 384. Greedy decoding is used for Qwen2.5-VL-7B, Video-R1, Video-Chat-R1, VideoRFT, Video-R1-SFT, Video-R1-RL, LongVideoReason-SFT, LongVideoReason-RL, ReWatch-R1-SFT, and ReWatch-R1. The decoding temperature is set to 0.8 for GLM4.1V-9B and 0.6 for InternVL3.5-8B. Models utilize different prompts in "Thinking" and "Non-Thinking" modes, as detailed in the Appendix [D.1](https://arxiv.org/html/2509.23652v2#A4.SS1).

### B.2 Performance comparison on Video Understanding benchmarks

Table: Table 3: Performance comparison on Video Understanding tasks. ^∗ indicates that we reproduced the model using a training configuration with 192 frames. ^† indicates that reinforcement learning is conducted using exactly the same data as ReWatch-R1. The best results among models of the same size are indicated in bold.

Table [3](https://arxiv.org/html/2509.23652v2#A2.T3) presents a comparative analysis of the performance of our model against other models on video understanding benchmarks. The key experimental findings and insights are as follows.

Synergistic Improvement in Reasoning and Understanding Without Catastrophic Forgetting.
ReWatch-R1 achieves state-of-the-art (SOTA) performance among models of a comparable size, with an average score of 54.15% at 192 frames across four general video understanding benchmarks. This demonstrates that specialized training for complex reasoning does not impair the model’s foundational abilities. On the contrary, it enhances general understanding by facilitating a more profound analysis of video content. This positive outcome is likely attributable to the multi-task learning design implemented during the Supervised Fine-Tuning (SFT) phase. The ReWatch-Caption task preserves the model’s fundamental video-text alignment, while the ReWatch-QA (direct-answer mode) and ReWatch-CoT (reasoning mode) tasks train distinct response pathways. Together, these tasks cultivate a comprehensively capable model rather than one with a specialized or biased skill set.

RL-driven Alignment of "Thinking" and "Non-thinking" Performance.
After SFT with Chain-of-Thought, the performance of the ReWatch-R1-SFT variant still lags behind the direct-answer ("non-thinking") performance of the base model. However, with the application of RL, the resulting ReWatch-R1 model not only exhibits further performance gains on video understanding tasks but also surpasses the direct-answer performance of the base model. This indicates that the enhancements in reasoning capabilities successfully generalize to foundational understanding tasks. This finding suggests that "deep reasoning" and "shallow understanding" are not entirely discrete processes. A model proficient in complex logical thought may consequently develop more reliable fundamental observation and recognition abilities.

### B.3 Performance comparison across different video durations

Figure: Figure 9: Performance comparison across different video durations. Short: 0-3 minutes, Medium: 3-20 minutes, Long: over 20 minutes. We averaged the performance of the benchmarks for reasoning and understanding respectively, and all results were evaluated at 192 frames.
Refer to caption: x11.png

Figure [9](https://arxiv.org/html/2509.23652v2#A2.F9) presents a comparative analysis of model performance on videos of varying durations. The findings highlight two primary conclusions regarding long-video reasoning.

Superior Performance in Long-Video Reasoning.
The proposed method demonstrates a significant advantage in long-video reasoning. ReWatch-R1 substantially outperforms all other models of comparable size on reasoning tasks for long videos (>20 min). For instance, ReWatch-R1 achieves 27.46%, an absolute improvement of over 3.4 percentage points compared to the next-best model, LongVideoReason-RL (24.03%). This result provides strong evidence for the efficacy of the overall methodology. The ReWatch dataset, with its hierarchical subtitles and contrastive QA, is specifically designed to create challenges that require reasoning across extended temporal spans. The model’s success indicates that this specialized training endows it with a superior ability to locate, associate, and reason with key information embedded within lengthy and often noisy video streams.

Robustness to Performance Degradation on Long Videos. An analysis of all models reveals a consistent trend: performance on reasoning tasks declines as video duration increases. This observation confirms that long-video reasoning is a pervasive and yet-unsolved challenge for current LVLMs, a phenomenon that can be described as a "Long Video Tax." However, the key advantage of ReWatch-R1 lies in its more attenuated rate of performance degradation. For example, while its own performance drops from 40.38% (short videos) to 27.46% (long videos), its decline is less severe relative to its high baseline. This indicates that the model not only establishes a superior starting performance but also demonstrates greater resilience when confronted with the challenges of extended durations, further substantiating the robustness of the proposed method in handling long-term temporal dependencies.

### B.4 Comparative Analysis of Dataset-Induced Reasoning Complexity and Video Dependency

Figure [6(a)](https://arxiv.org/html/2509.23652v2#S4.F6.sf1) presents a quantitative analysis of the reasoning characteristics elicited by the ReWatch and Video-R1 datasets. The experiment involves using the ReWatch-R1-SFT model to perform inference on the ReWatch training set and the multiple-choice subset of the Video-R1 training set. From the outputs for each dataset, 5,000 correctly answered samples are randomly selected for analysis. Three metrics are computed for these samples: the average number of reasoning steps (<action> tags), the average response length, and the degree of video dependency. Video dependency is specifically quantified as "Text-Only Accuracy"—the accuracy of the powerful Qwen2.5-VL-7B model when answering questions with only textual input and no video. The results show that the ReWatch dataset demands more profound, multi-step inference, eliciting nearly double the number of reasoning steps (3.31 vs. 1.82) and significantly longer responses (398.75 vs. 205.74 characters). Most critically, the Text-Only Accuracy for Video-R1 is 68.9%, indicating that questions can often be answered from textual cues alone. In stark contrast, the accuracy for the ReWatch dataset is merely 29.4%, a figure close to the 25% random-guessing baseline. This provides compelling evidence that the dataset’s three-stage filtering mechanism is highly effective, successfully eliminating spurious shortcuts and ensuring that problems are solvable only through genuine video understanding.

## Appendix C Related Work

### C.1 Video QA datasets and benchmarks

A growing body of video reasoning benchmarks reveals that current LVLMs struggle on complex, multi-step temporal reasoning. Recent evaluations [33; 31; 10; 34; 29] target causal attribution, temporal ordering, state tracking, counting, and cross-modal grounding, and consistently report large performance gaps even for strong models [3; 37; 40; 14; 25; 38]. Long-video understanding suites [57; 39; 15; 19] further underscore the challenge by emphasizing hour-scale contexts and dense event structure. Collectively, these benchmarks confirm that multi-hop, evidence-driven video reasoning remains underdeveloped in LVLMs.

In contrast, the available training corpora offer limited support for developing such capabilities. Large open sources provide long videos and captions but predominantly yield holistic or coarse descriptions that lack precise temporal annotations [22; 17; 27; 47; 56; 6], or perception-centric QA that only requires simple single-step reasoning [56; 8; 7; 53; 51]. Recent video-reasoning efforts augment these resources with step-by-step traces, yet their Chain-of-Thought (CoT) is typically distilled from text-only LLMs and often resorts to commonsense or elimination rather than verifiable, video-grounded retrieval [14; 38; 42]. Such supervision is ill-suited for Reinforcement Learning with Verifiable Reward (RLVR), which requires challenging, multi-hop questions and checkable, content-grounded processes to produce reliable reward signals [11; 21]. This mismatch leaves RL methods data-starved: they can optimize answer formats and surface patterns but struggle to learn evidence-linked temporal reasoning [26].

To close this gap, we synthesize ReWatch, a dataset that couples (i) temporally precise, hierarchical captions preserving event order, (ii) high-difficulty QA generated by contrasting detailed captions against summaries to remove shortcuts, and (iii) multi-agent, video-grounded CoT that explicitly records retrieval and verification steps. This design aims to provide the process-level supervision and question difficulty necessary to unlock RLVR for complex video reasoning.

### C.2 Video Reasoning in Large Vision-Language Models

Reinforcement Learning for video reasoning emerges as a complementary path. Recent works [14; 25; 38; 9; 32] adopt RL/RFT-style training to improve reasoning, generally using final-answer accuracy as the primary reward and relying on the above training data. While promising, these pipelines inherit the limits of their supervision: weakly grounded CoT and shortcut-prone QA. Rewards remain coarse, focusing on outcomes rather than verifying intermediate observations or the sufficiency of the reasoning process. As a result, models can overfit to answer patterns, exhibit hallucinations, and fail to align intermediate steps with evidence in the video.

Agentic methods integrate reasoning with tool use to improve grounding.
Recent work extends agentic paradigms like ReAct [49] to long video understanding, enabling models to dynamically interact with video during inference to produce grounded reasoning chains [50; 54; 48; 13; 41; 23; 30; 43; 2; 18; 55]. However, these methods are often training-free, failing to internalize such reasoning abilities within the base model. Other approaches [36; 28; 24] use agents to synthesize video-based Chain-of-Thought data and then train models with SFT, but they typically generate fixed tool-use trajectories from a single planning phase, lacking the iterative "think-and-act" capability. Concurrently, the "think with video" paradigm emerges [52], which dynamically retrieves and injects video segments into the model’s context. This strategy, however, places excessive demands on context length and involves complex model context management and agentic RL training, severely limiting training efficiency.

Our work combines the strengths of the above lines while addressing their limitations: we couple agentic data synthesis with RLVR, and while maintaining dynamic interaction with long videos and evidence verification, we internalize efficient, grounded reasoning into the multimodal model, thereby overcoming key limitations of current video reasoning.

## Appendix D Propmts

### D.1 Thinking Prompts

We use different prompts to activate the thinking mode of different models.
The detailed Settings are as follows:
Qwen2.5-VL is not a reasoning model, so we use the CoT Prompt [10](https://arxiv.org/html/2509.23652v2#A4.F10). GLM4.1V itself has the thinking mode enabled by default, so we use the direct QA Prompt [16](https://arxiv.org/html/2509.23652v2#A4.F16). InternVL3.5 requires additional hints to activate the thinking mode, so we use the Prompt [15](https://arxiv.org/html/2509.23652v2#A4.F15). Video-R1 and VideoRFT use the Prompt [12](https://arxiv.org/html/2509.23652v2#A4.F12). Video-Chat-R1 uses the Promp [13](https://arxiv.org/html/2509.23652v2#A4.F13). LongVideoReason uses the Prompt [14](https://arxiv.org/html/2509.23652v2#A4.F14). Our model ReWatch-R1 uses the Prompt [11](https://arxiv.org/html/2509.23652v2#A4.F11).

Figure: Figure 10: Prompt for the Thinking mode of Qwen2.5-VL.

Figure: Figure 11: Prompt for the Thinking mode of ReWatch-R1.

Figure: Figure 12: Prompt for the Thinking mode of Video-R1 and VideoRFT.

Figure: Figure 13: Prompt for the Thinking mode of Video-Chat-R1.

Figure: Figure 14: Prompt for the Thinking mode of LongVideoReason.

Figure: Figure 15: Prompt for the Thinking mode of InternVL3.5.

### D.2 Non-Thinking Prompts

In the evaluation, all the models in this paper use the same Prompt [16](https://arxiv.org/html/2509.23652v2#A4.F16) when applying the non-thinking mode.

When training ReWatch-R1-SFT, we apply Prompt [17](https://arxiv.org/html/2509.23652v2#A4.F17), Prompt [16](https://arxiv.org/html/2509.23652v2#A4.F16), and Prompt [11](https://arxiv.org/html/2509.23652v2#A4.F11) on datasets ReWatch-Caption, ReWatch-QA, and ReWatch-CoT respectively.

Figure: Figure 16: Prompt for the Non-Thinking mode of all models in this paper.

Figure: Figure 17: Prompt for the video-text alignment.

### D.3 Answer Judge Prompt

Figure: Figure 18: Prompt for Answer judge.

**Table 4: Definitions of the 10 synthesized QA types.**
| Task Type | Definition |
| --- | --- |
| Event Localization | This task requires the LVLM to output the precise start and end times of a specific event in the video, based on a natural language query. |
| Temporal Localization | This task provides a timestamp or time interval from the video and requires the LVLM to describe what happened within that specific time. |
| Counting | This task requires the LVLM to calculate the frequency of events or actions and to perceive the number of occurrences of specific objects. |
| Cause and Effect | This task requires the LVLM to identify direct causal relationships between specific events in the video, meaning one event directly led to the occurrence of another. |
| State Changes | This task requires the LVLM to identify temporal changes in the attributes, position, behavior, or emotions of specific objects or characters in the video. |
| Reading (OCR) | This task requires the LVLM to identify and understand textual information appearing in the video frame (e.g., signs, subtitles, screen displays, document content). |
| Spatial Perception | This task requires the LVLM to understand the relative spatial positions, distances, and movement trajectories between objects, people, and their environment within the video. |
| Numerical Reasoning | This task requires the LVLM to perform all mathematical operations other than simple counting, including but not limited to comparison, calculating speed, estimating time, calculating proportions, etc. |
| Object Recognition | This task requires the LVLM to identify and name specific objects, people, or animals appearing in the video. |
| Counterfactual Reasoning | This task requires the LVLM, given the video context, to hypothesize a scenario where a certain event did not occur or occurred differently, and then infer the likely objective, verifiable consequences. This does not involve subjective feelings or pure speculation but is based on physical laws, logic, or established patterns shown in the video. |

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: -Can the authors provide concrete ablation or error-type breakdowns specifically isolating the effect of the O&R reward on hallucination rates or logical errors, perhaps through qualitative examples or confusion matrices? It is unclear if this reward is the main driver for reduced hallucinations, or if SFT data quality dominates.
-Given the extensive use of LLMs at nearly every phase, is there a risk that ReWatch-R1’s strong performance partly reflects an overfit to LLM-specific language or annotation artifacts in the data/reward pipeline?
-How sensitive is the model (and pipeline) to coarse versus fine semantic segmentation in the captioning stage? For example, does segment over-segmentation harm long-term reasoning due to context fragmentation?
soundness: 3
strengths: -The topic of video reasoning is of community’s interest and timely.
-The hierarchical segmentation and multi-agent CoT pipeline used to generate the ReWatch dataset is methodically crafted and shown to have yielded data with deeper temporal grounding and higher question complexity.
-The baselines compared are pretty up-to-date, which verifies the state-of-the-art achievement on the benchmarks tested in the paper.
summary: This submission presents ReWatch-R1, a framework for advancing complex video reasoning in large vision-language models (LVLMs). The core contributions include the creation of the large-scale, multi-stage ReWatch dataset aimed at challenging multi-hop video reasoning, a multi-agent data synthesis pipeline generating temporally precise captions, high-difficulty QAs, and video-grounded CoT traces, and a new Observation & Reasoning (O&R) reward mechanism for RL. By post-training an LVLM backbone with these elements through SFT and RL, the authors demonstrate state-of-the-art performance across five video reasoning benchmarks, alongside ablation studies.
weaknesses: -The incremental benefit of the O&R reward mechanism is not fully dissected independently from other RL contributions. In Table 1, the improvement from RL (+O&R) appears relatively modest, and there is limited analysis delineating whether specific reasoning failures or hallucinations are directly ameliorated by the O&R reward in practice.
-The paper leans heavily on LLMs (Gemini, GPT-4.1, Qwen, etc.) for practically all phases - data synthesis, answer verification, reward calculation, and benchmarking. Although this is common in the area, the cumulative propagation of LLM biases and potential “meta-overfitting” is insufficiently analyzed; for example, does repetitive LLM-based data filtering introduce subtle data shortcuts or annotation artifacts? A systematic error/robustness analysis would be welcome.
-There is little systematic exploration of failure modes, such as where ReWatch-R1 still hallucinates, or which reasoning subtypes remain unsolved - as could be illustrated by extensive qualitative error analysis or confusion matrices per reasoning dimension (section 4.2 or appendix). This is important to inform the future community on limits.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: 1) Why are the number of ReWatch-QA and ReWatch-CoT inconsistent? Shouldn't one QA data correspond to one CoT data?

2) In Section 2.3 (specifically line 247), why does a structured execution trajectory end with A_final instead of observation? (Thought-Action-Observation loop)
soundness: 3
strengths: This paper proposes a dataset construction pipeline for video reasoning, which is enlightening for the dataset construction method in this field. The detailed description of video in different time steps is introduced based on semantic segmentation, which combines with the video summary to enhance the reasoning ability. In Addition, this paper uses three-layer filtering to screen the data that can best reflect the reasoning ability. The two-stage post-training framework simulates the "Thought-Action-Observation loop", and uses the improved O&R reward for RLVR to enhance the reasoning performance of ReWatch-R1.

Therefore, this paper provides a new idea for the subsequent dataset construction method in this field and the improvement of reasoning performance.
summary: This paper mainly contributes to the following three points:

1). Proposing a novel multi-stage synthesis pipeline to synthesize ReWatch, a large-scale dataset which includes ReWatch-Caption, ReWatch-QA, and ReWatch-CoT three components, for fostering advanced video reasoning. 

2). Proposing a new Observation & Reasoning (O&R) reward for RLVR that improves reasoning by rewarding both final-answer correctness and the factual grounding of intermediate steps in video content.

3). Developing ReWatch-R1 by post-training a strong baseline LVLM with Supervised Fine-Tuning (SFT) and O&R framework, achieves state-of-the-art average performance on five challenging video reasoning benchmarks.

The dataset construction pipeline and the two-stage post-training framework proposed in this paper put the description of video change into the CoT, providing a new idea for the research of video reasoning from the perspectives of dataset and training.

In summary, this paper shows a high level in terms of method description (data construction pipeline,the O&R methods, etc.), experimental setup and writing, but for the situation that pipeline may cause error accumulation, some experimental results are not analyzed (this part will be described in detail in "Weaknesses"), which leads to the lack of analysis completeness. The quality of this paper will be improved after analyzing the missing parts.
weaknesses: The weaknesses of this paper focus on the method, experiment and writing：

In method and experiment：

1). Error accumulation: the stage of semantic segmentation and detailed description generation in the dataset construction pipeline designed in this paper may cause error accumulation. The error caused by segmentation may cause one semantic of the video to be put into multiple segments. The description generation may result in the missing description between multiple time steps, which may result in the description of an item appearing in the previous time step, but the description of the item is missing in the next step, resulting in error accumulation. This paper does not discuss and analyze this situation.

2). In Table 1, the results of ReWatch-R1-SFT and ReWatch-R1+O&R on the CG AV counting are the same, which are not analyzed in the paper.

3). Lack of comparative analysis between the vanilla RLVR and the improved O&R method under the dataset constructed in the paper.

4). In the analysis in Section 4.2 (specifically line 411), the performance degradation caused by Video-R1 replacing ReWatch-CoT does not consider the reason that SFT data and RL data is mismatch, which seems to be somewhat contradictory to the previous sentence "SFT is an independent prerequisite for RL".

5). Also in the analysis in Section 4.2 (specifically 414 lines), it seems that ablation study using different data combinations (e.g. only using ReWatch-Caption or ReWatch-QA or ReWatch-CoT) have not been carried out, and the conclusion that "The Quality of QA data used for RL determines final performance." is somewhat far fetched.

6). In Appendix C2, the performance degradation of 384 frame training is not analyzed compared with 192 frame training.

In writing：

1). In Figure 1, the annotation order of the legend is inconsistent with the display order of each LVLM in the figure.

2). The two marking symbols not appear in Table 3 of appendix C2.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. Equation (20): the design of non-format rewards lacks motivation. For example, why not use the (weighted) summation of all rewards. The design has no explanation or experimental results.
2. RL on 7B model shows promising improvements. However, the performance still lags behind the larger 32B model. Therefore, whether the same RL on 32B leads to improvement is questionable.

Some typos:
1. L255: "a novel O&R reward mechanism we propos" -> "propose"
2. L771: "Tabale 4" -> "Table 4"
soundness: 3
strengths: 1. The proposed multi-agent COT data synthesis pipeline is scalable to curate large-scale video-grounding reasoning data.
2. The reward design shows the emphasis on explicit observation and reasoning is beneficial for video reasoning.
3. The data and model design achieves state-of-the-art performance on video reasoning and understanding benchmarks in 7B-scale models. The extensive analysis shows insights on the role and importance of SFT and RL.
summary: This paper introduces **ReWatch-R1**, which tackles the challenge of video reasoning from two aspects: 
1. Data. A large synthetic dataset called **ReWatch** is collected, which includes detailed video captions, challenging question-answer pairs, and high-quality, video-grounded reasoning traces (COT data). These are generated using a multi-agent ReAct system. 
2. Model. Authors post-train QwenVL by SFT and GRPO. They introduce a new Observation & Reasoning (O&R) reward, which evaluates the accuracy of video observations and the validity of reasoning process. 
ReWatch-R1 achieves promising performance compared with other 7B models.
Analysis shows that high-quality reasoning data is crucial for RL and RL on "thinking" mode improves the reasoning efficiency.
weaknesses: 1. The information source for data synthesis is semantic segmentation and detailed video description from Gemini. The accuracy of this multi-step hierarchical captioning is not validated. Therefore there is no direct quality assessment of the synthetic data.
2. All results are based on a 7B model. The benefits of high-quality COT data and O&R reward mechanism are not validated on larger-scale models.

---

confidence: 5
contribution: 3
initial_rating: 4
presentation: 3
questions: 1. Does the proposed Observation & Reasoning (O&R) reward mechanism introduce additional reasoning overhead or inference latency compared to conventional reward designs? It would be helpful if the authors could provide quantitative results or analysis on the computational cost and efficiency trade-offs, as well as discuss potential optimization strategies to mitigate these issues.
2. The paper currently compares ReWatch-R1 with a limited set of baseline models. Have the authors considered evaluating against stronger closed-source models (e.g., Gemini 2.5 Pro/Flash, GPT-4o) or recent RL-based approaches such as TW-GRPO, VersaVid-R1? Such comparisons would help better position ReWatch-R1’s performance and highlight its relative advantages within the broader landscape of contemporary video reasoning models.
soundness: 3
strengths: 1. The paper introduces a novel multi-stage data construction pipeline that enables the creation of ReWatch, a large-scale and high-quality dataset specifically designed for video reasoning. The dataset comprehensively includes caption, QA, and chain-of-thought (CoT) annotations, providing valuable and versatile resources that can benefit future research and community development in multimodal reasoning.
2. The authors offer detailed experimental descriptions and implementation settings, ensuring strong reproducibility and facilitating independent verification and extension of their results.
summary: The paper proposes ReWatch-R1, a framework that enhances complex video reasoning in large vision-language models through agentic data synthesis and verifiable reinforcement learning. It introduces ReWatch, a large-scale, high-quality dataset built via a three-stage pipeline—hierarchical video captioning, high-difficulty QA generation, and multi-agent chain-of-thought synthesis—ensuring strong temporal grounding and reasoning diversity. Furthermore, the authors design an Observation & Reasoning (O&R) reward that jointly evaluates answer correctness and factual grounding of intermediate reasoning steps. Combined, these innovations enable ReWatch-R1 to achieve state-of-the-art performance on multiple challenging video reasoning benchmarks.
weaknesses: 1. The paper lacks qualitative experimental results, such as example responses generated by ReWatch-R1 for specific video reasoning cases. Including such examples would help readers intuitively understand the model’s reasoning style, output quality, and advantages over baseline models.
2. The selection of baseline models is relatively limited, lacking comparisons with stronger closed-source systems such as Gemini 2.5 Pro/Flash and GPT-4o, as well as with recent RL-based approaches like TW-GRPO. Including these baselines would provide a more comprehensive evaluation of ReWatch-R1’s effectiveness and competitiveness.

---

confidence: 3
contribution: 2
initial_rating: 4
presentation: 3
questions: - As shown in Tab.3, why does the trained model show marginal improvement on the most popular, general video question-answering datasets with reasoning?
- How do you distinguish reasoning video benchmarks and general video benchmarks?
soundness: 3
strengths: - This paper finds drawbacks in the current synthesized long CoT datasets, e.g., Video-R1, and synthesizes a quality-improved video reasoning dataset.
- This paper redesigns the reward shaping for video understanding.
- The trained model sets a new SOTA on several benchmarks.
summary: This paper introduces novel datasets about captions, QAs, and CoTs, with a training pipeline to improve complex video reasoning. Additionally, based on the synthesized CoT data, this paper further designs a new reward for RL training. The trained model achieves state-of-the-art performance on several video reasoning benchmarks, outperforming models trained on other open-source datasets.
weaknesses: - The contribution of this paper mainly lies in the dataset construction. Can you break down the improvement of each part, like adding timestamps into the captions, adding question difficulty, and the proposed reward design?
- The introduction about the observation is not clear. How do you define observation formerly? In common, observation is visual content, while in your settings, the observation is processed captions. Would you consider implementing the thinking-with-image style training?

---

confidence: 4
contribution: 2
initial_rating: 4
presentation: 2
questions: 1. Line 207: Why does the original set of 85K QA pairs yield over 170K multiple-choice QA pairs?  

2. From the Chain-of-Thought (CoT) example shown in the bottom-right corner of Figure 2, is retrieving explicit timestamps truly necessary? Could the reasoning path be simplified—for instance, as follows:  
> *"... So, I’ll \<action\> retrieve segments focusing on the man with blonde, curly hair on a jet ski interacting with a passenger \</action\>. \<observation\> The man on the jet ski passes a sandwich to the passenger, who then takes a bite. \</observation\> This directly answers the question. The food item passed was a sandwich..."*
soundness: 2
strengths: 1. Novel QA curation methods: ReWatch is carefully designed to enforce video dependency and multi-step reasoning through contrastive QA generation and rigorous filtering, effectively eliminating textual shortcuts and hallucination-prone supervision.

2. Innovative O&R reward mechanism: By evaluating both final answers and the fidelity of intermediate observations and reasoning steps, the O&R reward explicitly discourages hallucination and promotes evidence-based reasoning.
summary: The paper proposes ReWatch-R1, a novel approach to improve complex video reasoning in LVLMs by addressing the critical data bottleneck in existing methods. The authors introduce ReWatch, a large-scale dataset synthesized via a multi-stage agentic pipeline that includes temporally dense captions, high-difficulty multi-hop QA pairs, and video-grounded CoT traces generated through a Multi-Agent ReAct framework simulating human-like “re-watching.” They further develop an Observation & Reasoning (O&R) reward mechanism for RL, which jointly evaluates answer correctness and the factual grounding of intermediate reasoning steps.
weaknesses: 1. Formula 4 merges captions across each time interval independently, which leads to a loss of referential consistency. For example, the caption for 0–10s might be “A man…”, while that for 10–20s is again “A man…”, even though it refers to the same individual as in the earlier segment. This inconsistency compromises the overall quality of the generated captions.

2. The observation mechanism introduces inference latency. Despite yielding performance gains on certain video reasoning benchmarks, it provides only marginal improvements on most general video understanding benchmarks.

3. The paper lacks comparisons with more recent and advanced baselines, such as VersaVid-R1 and GRPO-CARE.

4. Quantitative analysis is missing: the paper does not present any concrete inference results or case studies from ReWatch-R1.


Decision: Accept (Poster)