DeepVideo-R1: Video Reinforcement Fine-Tuning via Difficulty-aware Regressive GRPO
Authors: Jinyoung Park
Jeehye Na
Jinyoung Kim
Hyunwoo J. Kim
Venue: NeurIPS 2025 poster ()
Keywords: Video Large Language Model
Post-training
GRPO
Primary area: deep_learning

Abstract: Recent works have demonstrated the effectiveness of reinforcement learning (RL)-based post-training for enhancing the reasoning capabilities of large language models (LLMs). In particular, Group Relative Policy Optimization (GRPO) has shown impressive success using a PPO-style reinforcement learning algorithm with group-normalized rewards. However, the effectiveness of GRPO in Video Large Language Models (VideoLLMs) remains underexplored. In this paper, we explore GRPO and identify two issues that hinder effective learning: (1) reliance on safeguards, and (2) vanishing advantage. To mitigate these challenges, we propose DeepVideo-R1, a video large language model trained with Reg-GRPO (Regressive GRPO) and difficulty-aware data augmentation. Reg-GRPO reformulates the GRPO loss function as a regression task that directly predicts the advantage in GRPO, eliminating the need for safeguards such as clipping and min operations. This directly aligns the model with the advantages, providing guidance to prefer better outputs. The difficulty-aware data augmentation strategy augments input prompts/videos to target solvable difficulty levels, enabling diverse reward signals. Our experimental results show that our approach significantly improves video reasoning performance across multiple benchmarks.

DeepVideo-RR Video Reinforcement Fine-Tuning via Difficult--wware Regressive GRPO
Jinyoung Park Jeehye Na
Jinyoung Kim? Hyunwoo J. Kim
Korea Advanced Institute of Science and Technology, KKorea University
[jinyoung, park, jeehyena, hyunwoojkim @kaist. ac.kr
k0121000korea.ce ac.
Abstract
Recent works have demonstrated the effectiveness of reinforcement learning (RL)- based post-training for enhancing the reasoning capabilities of large language models (LLMs). In particular, Group Relative Policy Optimization (GRPO) has shown impressive success using PPO-style reinforcement learning algorithm with group-normalized rewards. However, the effectiveness of GRPO Video Large Language Models (VideoLLMs) remains underexplored. In this paper, we explore GRPO and identify two issues that hinder effective learning: (1) reliance on safeguards, and (2) vanishing advantage. To mitigate these challenges, we propose Deep Video-R1. a video large language model trained with Reg-GRPO (Regressive GRPO) and difficulty-awan data augmentation. Reg-GRPO reformulates the GRPO loss function as a regression task that directly predicts the advantage GRPO, eliminating the need for safeguards such clipping and min operations. This directly aligns the model with the advantages, providing guidance to prefer better outputs. The difficulty-awe aware data augmentation strategy augments input prompts/videos to target solvable difficulty levels, enabling diverse reward signals. Our experimental results show that approach significantly improves video reasoning performance across multiple benchmarks.
1 Introduction
have demonstrated remarkable abilities in understanding, reasoning, and generating text across diverse domains. Their success stems from next-token prediction over vast corpora, which enables the emergence of complex reasoning patterns and world knowledge. Building on this progress, recent research has extended LLMS into the video domain, giving rise to Video Large Language Models (VideoL.L.Ms MMM)) (5)2). These models aim to unify video understanding and language generation, enabling capabilities such temporal event reasoning, video question answering, and video-to-text summarization.
Large Language Models (LLMs) 14
Despite their rapid evolution, current VideoLLMs still struggle with complex reasoning tasks, which require temporal, spatial, and semantic understanding over video sequences. Since standard su- pervised fine-tuning fits instruction data rather than reasoning processes, it is limited in improving reasoning capabilities. To address this, reinforcement learning (RL)-based post-training [10 IT] has emerged as a compelling paradigm. RL provides a mechanism to optimize models beyond likeli- hood objectives, aligning them with reward signals that encode human preference or task case-spec success. Recently, Group Relative Policy Optimization (GRPO) [12 13 has shown promise by using group-based advantages and relative preference signals to enhance reasoning capabilities.
Work was done Korca University. corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

GRPO
Policy Model
Group Responses y
Rewards Rl
Advanlages. A
GRPO
DeepVideo-R1 (Ours)
Qwen2.5-VL-3B
Qwen2.5-VL-78
Policy Felaa
1 Ruaa
Maximize
+10.1
56
+3.6
(cs Mey
4
48
53
Accuracy
Deep DeepVideo-RI
44
+8.6
50
Perturb
Policy Model
Group Responses 5
Rewards Ran gfo)
Advantages A
40
47
1 Regress
Old Policy-T@..m
ng
Implicit
36
44
Advanfage
32
L1
L2
L3
L1
L3
In-Dist.
OOD
In-Dist.
OOD
Figure 1: DeepVideo significantly improves the reasoning capabilities of VideoLLMs. Our VideoLLM, Deep Video-R1, is trained to explicitly predict the advantage A through Regressive GRPO loss. Notably, model training becomes significantly effective and achieves 10.1 performance improvement compared to GRPO.
While GRPO has achieved strong results in text-based tasks, its application to VideoLLMs remains underexplored. In this work, we investigate the application of GRPO to VideoLLMs and identify two key limitations that hinder effective training: (1) reliance on stabilizers such as minimum and clipping operations, which often suppress gradients and impede convergence, and (2) the vanishing advantage problem, where extremely easy or difficult samples yield zero advantages, thereby removing the training signal.
To overcome these limitations, we propose VideooR., video large language model trained with two key innovations: Regressive GRPO (Reg-GRPO) and difficuly-aware data augmentation. Reg-GRPO reformulates the GRPO objective as a regression problem that directly predicts group- based advantage values. This simple yet effective reformulation enables direct alignment between model outputs and the advantage values, eliminating the need for stabilizers while ensuring stable convergence. We also introduce difficulty-awwe of video-text inputs. For easy samples, we perturb the video content to inject uncertainty; for hard samples, provide auxiliary reasoning cues. This strategy diversifies the reward landscape, mitigating the vanishing advantage problem and promoting balanced learning across difficulty levels.
augmentation that dynamically adjusts the difficulty
Our experimental results demonstrate the effectiveness of Deep Video-RR on multiple challenging video reasoning benchmarks such as SEED-Bench-RI Long VideoBechh,, and NEXTGQA, achieving superior performance over recent video LLMs such as Qwen2.5-VL. 77] (Figure Notably, our model achieves consistent improvements on both in in-distribution and out-of--istributte robust generalization capabilities. These results underscore the benefits of combining a regression- based RL objective with data augmentation for training large-scale multimodal reasoning models.
tasks, indicating
Our main contributions are listed as:
We introduce Reg-GRPO, novel optimization scheme that casts GRPO regression task over grsup advantage values, eliminating heuristic stabilizers such as clipping and min operations, and mitigating the vanishing gradient issue.
We develop difficult--wware augmentation framework that modulates video-text inputs with adaptive difficulty scaling, video cue injection, and noise perturbation generate richer and more effective reward signals.
video large language model trained with two key innovations: Regressive GRPO (Reg-GRPO) and difficulty- data augmentation. Our experimental results demonstrate that Deep Video-R significantly improves the reasoning capabilities of VideoLLMs on complex vidco reasoning tasks.
We propose Deep

---

2
Related Work
Video Large Language Models (VideoLLMs). Large Language Models (LLMs) [4 have exhibited strong generalization and reasoning capabilities across a wide range of domains, including knowledge-intensiv tasks (17, mathematical reasoning I2 21 and scientific domains [22-24]
Building on these capabilities, Video Large Language Models (VideoLLMMs have extended LLM reasoning dynamic video domains, achieving notable performance across tasks (25 such as video question answering 77 [1] and video captioning [32 Despite their impressive performance, VideoLLMs remain limited on long video inputs and fine-grained video understanding tasks that require detailed spatiotemporal reasoning [35H37]. Most existing methods primarily emphasize video perception or short-contex. understanding. often relying static supervised fine- tuning objectives that fail to capture reasoning dynamics. To address these challenges, we leverage reinforcement learning- based fine-tuning approach to improve the reasoning and generalization capabilities of VideoLLMs.
have significantly improved the reasoning capabilities of LLMs through reinforcement learning (RL), including DPO 42 and RLHF (43]. Recently, variants of RL.-based fine-tuning 44 45 have explored direct reward-regression losses derived from RL objectives. A key development in this direction is Group Relative Policy Optimiza- tion (GRPO), RL algorithm proposed in [12 that computes group-wise normalized rewards to stabilize training and improve efficiency. Motivated by GRPO, several approaches have demonstrated substantial improvements in the reasoning abilities of MLLMs across image [46-53) and video tasks [54,59]. While existing approaches [55] 57 56 have primarily focused on defining appropriate
RL-based fine-tuning. Multiple works 12
13
38
reward functions tailored each visual task, some concurrent works [51 52 instead study practical issues that arise during GRPO training, aiming to further enhance model reasoning. In this work, we propose a learning algorithm that directly regresses advantages instead of increasing the likelihood of high-advantage responses. Additionally, we design difficulty-awar data augmentation to provide diverse and dense learning signals.
Methods
In this section, we present a video large language model named DeepVideo-Re which is trained with Regressive GRPO (Reg-GRPO) and difficulty-wware data augmentation for effective video context reasoning. We first introduce post-training methods for VideoLLMMs, such as proximal policy optimization and group-relative policy optimization (GRPO), and discuss the limitations of GRPO: reliance on heuristic safeguards and vanishing advantage. Then, we propose Reg-GRPO, which improves the RL-based GRPO by transforming the objective into simpler yet more effective regression loss, eliminating heuristic safeguards such as the min and clipping operations. Finally, we present a novel difficulty-aware data augmentation, which alleviates the vanishing advantage problem by modulating the difficulty of samples.
3.1 RL-based Fine-Tuning
Proximal Policy Optimization (PPO) [60] is one of the most widely used actor-critic RL algo- rithms for fine-tuning (video) large language models. For example, RLHF [61] applies the PPO algorithm. Given the input sample æ. PPO optimizes the model лy with the following objective:
Lyro (f)
Em, youte (:1m)
(yr x Ve))) To (ye 1 ye))
(Velx, yee) Toau (Jelær, 222)
e, + Ao
min
Ar, clip
where y is sampled from the policy model is hyperparameter, No... is the old model and the advantage A, is calculated with generalized advantage estimation (GAE) [2) using rewards and a trained value function Vns. Although the PPO algorithm aligns human preferences with the policy model outputs effectively, requires substantial memory and computational resources since the value function is typically another model comparable in size the policy model.
3

---

addresses the problem of PPO 600 by approx- imating the learnable value function with the average reward of multiple sampled outputs. Concretely,
Group Relative Policy Optimization (GRPO) 12
given input sample the model samples multiple output sequences ( G model Jus and then trains the policy model Ty with the following objective:
(=1 from the old policy
CORPO (0)
E
{
~ Touu (
(2) y
(8) <
x0
To
Vi
A(00)
min
clip
e, 1e
(2)
(5)
(1)
(2)
Tew
Vi
Tow
Vi
x., y211
3DLL [To]|
where 3 corresponds to a hyperparameter and DEL is the KL-divergence, Here, A is advantage calculated based the relative reward within the group, which is formulated A R( () -M+
where Pr, o, denotes the average and standard deviation values of set of rewards in the group, respectively. Although GRPO has shown its success, GRPO has two limitations that hinder the effective model optimization: reliance heuristic constraints and vanishing advantage problems.
Reliance on safeguards. GRPO optimizes the model with safeguards implemented using the min and clipping functions to avoid extreme changes in the model. However, the PPO-style clipping function induces zero gradient for samples where the value of Th (y/z) is too different from the value of Tea (3/lz). It cannot guarantee that the model (y (y)x) stays close Tee if is already far from The (y x)) [63] Similarly, GRPO also suffers from this phenomenon due to the PPO-style hard
constraints, and it deteriorates the effective model training. The analysis in [64 also shows that an
upper clipping threshold restricts the probability increase of the "exploration" token. This indicates that the safeguards in GRPO negatively influence model optimization.
Vanishing advantage problem. The vanishing advantage problem (53) indicates that the advantage of each sample within the group becomes zero, when the rewards of outputs in the group equal. problematic since the model cannot receive any signals from the training sample where the advantage is zero for every response. In particular, training samples are either too easy or too difficult for the current model. Training samples with extreme difficulty levels show worse performance than those with moderate difficulty levels.
observe that this issue often arises when
3.2 Regressive GRPO
Here, we present a Reg-GRPO (Regressive Group Relative Policy Optimizationn), which reformulates GRPO into the regression task, removing safeguards such the min and clipping functions. This reformulation enables the model to directly predict the advantages, resulting in improved alignment of the model with the preference. Following existing RL-based works 665 66 42 45), the Reg-GRPO loss function is derived from the RL objective that maximizes the expected reward with the KL constraints between Ty and Tex.
The objective of our reinforcement learning algorithm for each iteration to maximize rewards while preventing лe from making excessive changes relative Mea.
RL objective.
z arg max E E (a) R (xx y) Egg [KK.. (mT (-1x:) |To.. (-[x))].
me
where is the hyperparameter that adjusts the strength of the KL--divergence. Following prior works 42 45), the closed-form solution to the above equation (Eq. (3) can be obtained by minimum relative entropy problem:
Z Z(x) Thae (w|z) exp ( (z. v) Vz,
T (y)x)
(4)
where Z (xx) Eg Tem. ( l (z, vi) is partition function. However, since calculating the partition function Z (x) is expensive, it is hard to obtain zo exactly.

---

Reg-GRPO Loss. To address these issues, we propose Reg-GRPO (Regressive GRPO) loss, which learns the policy model to regress the advantage 1 using the reparameterizzttio, removing the normalization term Z (x). Specifically, the advantage for the i-th sample is defined as
R ar y
y Mr
A
where Mr, a, denote the average and standard deviation values of a set of rewards in the group, respectively. We can also rewrite Eq. to express the reward (ær, y) terms of the optimal model Tg. which can be formulated as
R (x,,y) = - --- (log g -10g Z (x)
3 (y) ar) Afaa (3/ a:)
log
Va, y-
(6)
Since the reward can be expressed through the optimal policy z (Eq. (6)) the advantage can be equiv- alently written as A p (2.x""))--r where p² (xr, y) is defined as p² (xe, y) log
m (ylar)
and
0,-
Tea ()]=)
1 Tp' denote mean and standard deviation of {p- (ar, (0) can see that Z (æ) is naturally removed during the reformulation.
respectively. Interestingly, we G | }
Building on this insight, define the predictive advantage, which estimates the advantage calculated by normalizing rewards within a group of samples, using the current policy Ty as
g p ( Mp
( x
p (x,, y) llog
dp
Taa (y a:
where μp, op are mean and standard deviation of {o (ae, y } respectively. Then, we optimize the policy by minimizing the gap between the target advantage and its predicted counterpart Åg using Reg-GRPO (Regressive GRPO), which is defined as:
CReg-Git GRFO (0) E ae =.,(0() - " (-m) { (A {400_
-00)
B Dx [TT]] [Tol|Tee]
(8)
Similar to GRPO, regularize the update with the KL divergence. The proposed Reg-GRPO loss serves as an effective alternative for optimizing group-level objectives, showing better performance than GRPO. In our experiments, demonstrate that this formulation leads to faster convergence and improved policy quality. The detailed derivation procedure of Reg-GRPO is in Appendix A
3.3 Difliculty-waare data augmentation
In this section, we present difficulty-aware data augmentation framework, which addresses the vanishing advantage problem in GRPO. This issue arises when training samples are either too easy or too difficult, leading to uniform rewards across multiple responses. As result, the advantage values become zero, erasing the learning signal. Our augmentation strategy mitigates this issue by modulating the difficulty of inputs incraase variance in predicted rewards, thereby preserving informative gradients for effective model optimization.
Specifically, given an input sample (v,, g). where and q denote video and a question, respec- tively, we first generate multiple responses and then compute the average reward + Σ R (æx (1) for the sample. We then measure the difficulty of æ by comparing its average reward with the average reward of samples in replay buffer B W which consists of samples Teep and their corresponding
( from the most recent W steps. Formally, the difficulty AR (x) of the input sample outputs yrop i=1 is calculated as:
G R
G y()
AR (x) = E.
( (in
1916 )
EBW
G
Trep-
(0)
1 R
Σ
xc,
(9)
1=1
j=1
Instead of using only the sample reward R (ar, y), we use the average reward of samples in the replay buffer reference value to account for model evolution.
5

---

Measure Difficulty based on Relative Reward Statisties
R(xr, (Zr,y/00) pr.
Dynamic Global Reward
R Rypoup
A(t)
dr
Reward
Response y
R(z,
Rgroep
(0.0.000. 0.0..... 0.0, 0.0)
Video LLM
Response y/
R(a.1²)
Queshion: What action Should I Tdhe next order to oet the colery?
Reccup
(11,003,.... ..,,
B W
Reploy
Response y
R(a,
Knoup
(2.0 2.0. ... 2.0.0.2000)
llerotion
Difficulty Decreasement Augmentation
Ryswe
Difficulty Increasement Augmentation
Response Ugg
(e yar)
Perturbation
Video
Response 1g
RR
What action should take next order fo the calary?
Question:
AR
GT: pick up celery
Response
c5y R(x, Mal
AR
<thimio The video shows person proparing ... To make selery you should bring from the big, dathieto- pick up celeryc/anowers
AR >
Guidance Level
Figure 2: Overview of the difficulty-aware data augmentation. First, we assess the difficulty of responses given the input video and question using Eq. (9). For hard samples, it augments the input prompts with the reasoning cues extracted from successful reasoning paths (Difficulty- decreasing augmentation), while the easy samples are perturbed with the noise -Difficutt-inccrassing augmentation). The scale of the guidance level or noise level is adaptively determined based on the difficulty of the current sample.
Building on the difficulty metric in Eq. (9) we adaptively adjust each training sample to balance the learning signal. For samples identified as too easy (i.e., high reward and low difficulty), we increase task complexity by perturbing the input-for example, by adding Gaussian noise to video inputs masking video frames-to encourage the model to attend more informative cues. Conversely, for too difficult samples (i.e., low reward and high difficulty). we inject auxiliary reasoning hints or visual cues that simplify the temporal context, helping the model focus on core reasoning paths instead of failing due to overwhelming difficulty. This dynamic modulation leads to an appropriate difficulty distribution across training, preventing the collapse of advantage values while ensuring that each update provides a meaningful gradient signal.
bifficulty-decreasing augmentation. For difficult samples (AR (a:) > 0), we ease the difficulty of the sample by providing auxiliary reasoning cues that guide the model toward generating correct reasoning. Concretely, first augment the input prompt q with the ground-truth answer and generate multiple reasoning trajectories using the VideoLLM. Among these reasoning trajectories, we select the response yg with the highest reward and extract partial reasoning trace of it. This trace is then incorporated into the original prompt to form a modified prompt i containing structured hints that guide the model's reasoning toward the correct solution. To maintain adaptive control, the guidance level is scaled according to the sample's difficulty magnitude. Harder samples receive stronger reasoning cues, while moderately difficult samples are given lighter guidance. By adaptively providing guidance for challenging inputs, this augmentation mitigates vanishing gradients in difficult cases and facilitates more stable convergence through progressively refined reasoning.
Dinticulty-inereasinng augmentation. Conversely, for easy samples (AR (x) < 0), we employ augmentation to enhance task complexity and encourage the model explore more diverse reasoning trajectories. We perturb the visual input v to create a harder input ö by introducing frame-level Gaussian noise, thereby slightly degrading perceptual fidelity while preserving overall semantic structure. The intensity of the noise is proportionally scaled by the difficulty magnitude, ensuring that easier samples receive stronger perturbations and moderately easy samples remain stable. This adaptive corruption expands the diversity of generated reasoning trajectories. By inducing distributed rewards, the augmentation ensures that even trivial samples provide informative gradients, avoiding zero learning signals throughout the optimization process.

---

Table Performance on SEED-Bench-RI validation split and LongVideoBench. In-Dist. means
in in-distribttion dataset.
SBR-LI In-Dist
SBK-L2
SBR-L3
Long Videolece Cross Tesk, Crost-Fav
Method
Closs-Eay
Cross-Tode
Cross-Eay
Daily
Daily
Daily
Hokbies
Receeation
Work
Overral
(8.15)
(180,600)
(900,500)
Overall
Baseline
33.3
33.2
26.7
28.5
30.6
27.0
27.7
35.7
43.1
21.0
22.5
26.7
1mxmmLL3-20
LmcmVLL5-88
41.4
40.8
39.2
34.6
35,0
3000
34.8
54.6
06.7
46.1
44.2
49.1
1.3m:VI.3-148
43.6
45.8
44,0
34,9
36.6
31.7
37.2
69.2
62.8
46.3
42.0
50.0
(wen2-V2-28
Qwen2-V1-28
12.9
16.4
13.0
16.3
10.4
14.4
13.8
33.0
32.7
20.5
22.2
27.2
SPT
34.1
36.2
36,9
34.6
30.0
33.9
33.8
42.6
42.3
37.0
32.2
36.3
40.4
42.0
Deeeg Viden-R1
48.9
50.3
52.4
42/7
41.1
49.2
46.3
51.4
49.7
38.5
34,8
40.1
Owen2-V2--78
Qwes2-V1-7R 1
SFT
43.8
44.1
38.3
41.0
32.2
38.6
38.2
45.0
54.7
36.7
36.4
40.0
GRPO
54.5
Video-RR
36.4
59.8
57.6
52.5
50.0
55.2
53.8
56.2
61.4
45.1
40.8
46.9
V-V-2.5-VZ-3
Q.a12.5-V1-88
25.0
50.3
SFT
35.9
39.1
39.9
31.5
29.7
31.2
33.7
51.4
52.9
36.3
35.3
39.7
GRPD
39.6
41.0
39.0
33.9
36.6
31.9
35.4
51.4
60.1
42.2
36.0
45.2
Deep Video-R1
(wen2.5:V2-7
Owm225-VL-7B
33.4
38.2
35.1
31.5
27.3
28.0
31.0
51.6
37.8
36.2
42.5
41.5
54.6
GRPO
49.1
52.1
49.4
40.7
43.2
35.2
42.2
61.1
60.8
44.4
40.8
47.7
Viepoide
$2.0
55.7
51.3
47.8
47,0
44.1
47.7
62.7
63.4
49.3
44.5
51.1
Experiments
4.1 Experimental Settings
To validate the effectiveness of the proposed method, conduct evaluations various video benchmarks, including both general video understanding tasks (e.g., SEED-Bench-RI [56), VSI- Bench, Video-MMMU, MMVU (mc), MVBench, TempCompass, Video-MME (wo sub)), long video understanding tasks (e.g.. LongViieeBBnnch (25)). and fine-grained reasoning tasks (NExTGQA (37.. More details about datasets are in Appendix [ We employ Qwen2-VL-2BA. (5) and Qwen2.5-VIL LL-3BBBLL [77] for the experiments. For the analysis, we use Qwen2.5-VL-3B default video LLM. More implementation details are in Appendix C.T
video
4.2 Experimental Results
Experimental results on SEED-Bench-RI. Table [ summarizes the performance of various baselines, supervised fine-tuning (SFT), GRPO, and our proposed DeepVideo-R the validation splits the SEED-Bench-RI (SBR) dataset. Across all settings (SBR-LI, L2, L3), DeepVideo-RI consistently achieves the best performance, demonstrating its strong capability for video reasoning under both in-distribution and cross-environment settings. Specifically, compared with Qwen2.5- VL-3B GRPO, our Video-do +8.5, +10 +10.1, and +8.6 points, respectively. Notably, the performance gains on SBR-L.2 and 1.3 (overall) exceed those on SBR-L.1, indicating that DeepVideo-R enhances generalization across cross-task and cross-environment settings. These results suggest that the proposed regression-baset optimization and reasoning-awen augmentation in Deep DeepVideo-R Video-RI RI enable more stable policy learning and improved adaptability to diverse video understanding scenarios.
improves the overall scores SBR-L1. L2, and L3 by
Experimental results on LongVideoBenchh We further evaluate DeepVideo-RI on Long VideoBench, a benchmark designed to assess long-video reasoning and temporal com- positional understanding. As shown in Table DeepVideo-Ri R1 again outperforms all baselines across varying temporal ranges, achieving an overall score of 51.1, surpassing both SFT and GRPO-trained counterparts. In particular, Deep Video-R1- R1-3B achieves substantial improvement of +7.4 over Qwen2.5-VI.c
GRPO on the longest duration range (900- 3600 s), underscoring its superior ability to reason over extended temporal contexts. This strong performance on long- duration videos highlights Deep ''s effectiveness in maintaining coherent reasoning over time, validating its robustness in complex real-world video understanding tasks.
7

---

Table Performance on various video reasoning and general benchmarks.
Video Reasoning Benchmark
Video General Benchmark
Method
VSI-Bench
Video-MMMU
MMVU (mc)
MVBench
TempCompass
Vidco-MME (wo sub)
T.aMA
67
45.6
Vade 33
54.6
LongVA-7B 68
29.2
23.9
56.9
52.6
VILA-I.5-BB 09
28.9
20.8
58.8
VILA-I_5-4BB 69
31.2
34.0
60.1
Video-UTR-7B 70
58.8
59.7
LLaVA-OneVision-TB
31
32.4
33.8
Kangarco-8B 7T
62.5
56.0
Qwen2.5-VL-3B
32.4
36.1
54.2
48.1
29.7
54.4
Vide-Roo- (Ours)
33,0
40.7
59.0
49.6
63.1
51.1
Table 3: Experimental results on NEXTGQA
Table 4: Ablation study on training schemes (Reg- GRPO and difficult-aware data augmenta- tion (DA-Aug.) in DeepVideo-RI using SEED-
Method
mloU
Acc@QA
Vision Experts
IGV
72
14.0
Bench-RI RI dataset.
Temp [CLP)
37
12.1
60.2
FroZenBiLM
73
9.6
70.8
Mothed
DA-Aug.
LI (In-Dist)
L2 (OOD)
L3(0UD)
SeViLA 74
21.7
68.1
Owen2.5VLL388
V.deof.LMMs
GRPO
29.6
41.0
35.4
VideoChat-RI RI
55
32.4
70.6
GRR)
VideoChat-R R -tthnkiing
SS
36.1
69.2
Beg GRPO
Reg-GRPO
44.2
48.1
51.1
44.2
44.0
39.5
Veep-ideo-R1-11 (Ours)
72.5
Experimental results on various video benchmarks. We further evaluate DeepVideo-P (1-do--1--3,,, built upon VLL-3B on diverse video reasoning and general benchmarks to assess its generalization ability. Following 55 we use the Video-R1 [54] training set to train the model, and report the results in T As shown in the table, VeepVideo-R1-3 consistently outperforms Qwen2.5-VL. 3B and other large-scale multimodal video models (e.g.. LLAVA-One Vision-7B. VILA-1.5-40B, Kangaroo-8B) across almost all benchmarks. In comparison the experimental results of the base model Qwen2.5-VL-77 our Deep Video-R1 consistently improves the performance on 5 out of 6 datasets. In particular, DeepVideo-Bl R1 improves performance from 29.7 to 63.1 on TempCompass. Overall, these results confirm that Deep Video-R1 R1 effectively generalizes beyond SEED-Bench-Be establishing a new performance level across both reasoning-oriemi oriented and general video understanding benchmarks.
DeepVideo-
Experimental results on NEXTGQA. Table reports the performance of DeepVideo-R compared with both vision experts (IGV, Temp[CLIP). FrozenBiLM. SeViLA) and VideoLLMs (VideoChat- For the NEXT-GQA benchmark, Deep Video-R1-1
RI, MateoChaa-RR-tthniin composite reward combining accuracy, format consistency, and IoU, aligning with the dataset's grounding-oriente oriented evaluation. As shown, Deep Video-RI R1----R--777 R1-7B achieves 36.8 mloU and 72.5 @cc@ QA, outperforming all baselines, including +4.4 mloU and +2.3 Acc Ac gain These improvements highlight DeepVideo-RR R1 's ability to enhance both spatial grounding and rea- soning precision, demonstrating its robustness diverse reward designs and its strong adaptability across grounded video reasoning tasks.
trained with
VideoChat-Ri R1.
4.3 Analysis
Ablation studies. We conduct ablation studies to explore the contribution of Reg-GRPO and difficulty-aware data augmentation (DA-Aug.) in Table 4 The table demonstrates that both Reg-GRPP and difficulty-award of Deee--deo-R1 R1. By comparing the GRPO-trained model without and with difficulty-wware data augmentation, we observe +2. -point improvement on SBR (LI), suggesting that difficulty-aware data augmentation is effective for both GRPO and Reg-GRPO. Also, Reg-GRPO (w/o DA-Aug.) yields +4.1 +4.1 -point improvement over O0wn2.5-VIL-3B--- indicates that Reg-GRPO is more effective than GRPO by directly predicting advantages.
data augmentation contribute to the performance improvement
(w/o DA-Aug.) on SBR (L.3). This

---

Table Performance comparison accord- ing to the data augmentation types. Difft indicates difficulty-nncreasing augmenta- tion. Diffj, indicates iffficultyddecraasii augmentation.
Table 5: Performance comparison reinforcement
learning algorithm.
Mothod
L1
L2
L3
Qwee2.5-VL.33 -VL-33
31.3
32.7
27.0
DPO 2+
35.8
35.2
30.8
Dill.
Dit.
L2
1.3
Online DPO
37.1
38.1
31.9
44.2
44.2
36.3
REINFFRCE 16 H2
37.0
39.5
40.0
RLOO
78
35.0
37.4
31.3
45.3
47.3
41.6
REBEL 451
41.8
43.7
38.0
48.1
51.1
44.0
Regression
GRPO
[k]]
39.6
41.0
35.4
Table 8: Performance comparison aug- mentation scaling scheme (Fixed guid- ance/noise level v.s. Adaptive guid- ance/noise level).
Regg-RROO (Ours)
44.2
44.2
39.5
Table 6: Performance comparison absolute and
relative difficulty measurement.
Diff. ref.
LI
L2
L3
Aug.
LI
L2
Absolute
47.9
50.1
40.7
No Aug.
44.2
44.2
36.3
Relative
48.1
51.1
44.0
Fixed
46.8
48.6
43.0
Adaptive
48.1
51.1
44.0
1.0
Vanishing aaaaaaavaaeeeee ratio
GRPO
GRPO
DA-Aug (Ours)
Average reward
0.6
3.6
0.4
1.4
0.2
DeepVideo-R1 (Ours)
1.0-
GRPO
0.0
Step
Step
Figure 3: Vanishing advantage ratio comparison GRPO and GRPO+DA-Aug (Difficulty-awan augmentation) (Left). Reward curves of Deep Video-RI (Ours) and GRPO (Right).
Comparison with reinforcement learning methods. Table 5 compares our Reg-GRPO with representative reinforcement fine-tuning (RFT) methods. The detailed explanations about the rein- forcement fine-tuning methods are in Appendix [ From the table, our DeepVideo-R achieves best performance among the compared RFT methods. In particular, compared to reward regression, which directly regresses reward scores, Deep Video-RI performs significantly better. This result highlights that directly aligning ad advantages is more effective than reward regression.
Impact relative difficulty measurement. We compare absolute and relative difficulty measure- ments for adaptive data augmentation in Table The relative scheme, which considers reward history statistics, consistently outperforms the absolute counterpart across all difficulty levels (L1-L3) on the SEED-Bench-Rl R1 dataset, demonstrating its superior ability to guide effective augmentation.
augmentation. In Table we compare different data augmentation schemes: difficulty-increasing and difficulty-decreasing augmentations. The table demonstrates that the model trained with both schemes achieves the best performance, yielding a +7.7-point gain on SBR (L3), which is an out-of-distribution dataset. This reveals that adjusting the sample's 's difficulty to a moderate level is important for learning with group-norman norop-oomallized ad advantages.
Impact
Impact of augmentation scaling scheme. In Table g we compare different augmentation scaling strategies: no augmentation, fixed scaling (constant guidance/noise level), adaptive scaling (difficulty- aware guidance/noise). From the table, the adaptive strategy consistently outperforms the others across all difficulty levels (I.1-I.3,, demonstrating the importance of tailoring augmentation strength based on input difficulty.

---

Question: Considering the progress shown the video and my observation shown image, what action should take order prepare berries?
move tray close tray C. open container move berries
Output the thinking process in <think </think> final answer in <answer> </answer> tags, i.e. <think reasoning process here </think><answer> answer here </answer>.
GRPO: <think The image shows person preparing food, specifically making dish with vegetables and cheese flatbread. There are berries or trays visible in image, not possible determine the steps preparing berries. options provided are related to trays containers, which relevant the current task preparing berries. </think> <answer> open container </answer>
DeepVideo-RE (aceeeoo-1(rrrrrr (Ours): <think The person in the video is currently preparing dish with berries. The next logical step would be add more berries to the dish arrange the existing berries in specific way. Since the image shows the person already handling the berries, the next action should be add more berries arrange the existing ones. </think> <answer> D. move baanser<</nswer>
Figure 4: Qualitative result of DeepVideo-R1177 in comparison of Qwen2.5-VL-7B-GRRB
Impact of augmentation. We compare the vanishing advantage ratio of GRPO and GRPO with DA-Aug (difficulty-aware data augmentation) in Figure 3 (left). The figure shows that our data augmentation effectively reduces the ratio samples causing the vanishing advantage. This indicates that the data augmentation effectively addresses the vanishing advantage problem in GRPO.
Reward curves. addition, we plot the reward curves of GRPO and our DeepVideo-RR -1, where the x-axis is training step and y-axis is the average reward in Figure 3 (right). From the figure. our DeepVideo-RR gets higher average reward with Reg- GRPO and difficulty-awer data augmentation.
Qualitative results. Figure SiareeAprreeeee presents a qualitative example from BBEDD-Bnnch-R17BB the outputs of our DeepVideo-R1 and the Qwen2.5-VLL-77 trained with GRPO. The task predict the next action given a video. Our DeepVideo-R1 correctly infers that the person will continue moving berries. While the GRPO-only model fails recognize the presence of berries. This demonstrates that Video-R has strong visual grounding and understanding capabilities.
comparing
Conclusion
We propose a video large language model, DeepVideo-R1. trained with Reg-GRPO (Regressive GRPO) and difficulty-waae data augmentation to address two key challenges in group relative policy optimization. Reg-GRPO reformulates the GRPO objective as regression task that directly aligns the model with group-normalized advantages. Difficulty-aware data augmentation modulates input difficulty to alleviate the vanishing advantage problem. Our experiments demonstrate that DeepVideo- RI consistently improves the reasoning performance diverse VideoLLMs, outperforming GRPO- based reinforcement fine-tuning.
Acknowledgment
This work partly supported by Korea Research Institute for defense Technology planning and advancement Grant funded by Defense Acquisition Program Cimnisstatio(DAPPP)/CRTTTC..... 021, 30%). the InnoCORE program of the Ministry of Science and ICT(N10250156, 30%), Virtual Engineering Platform Project (Grant No. P0022336, 30%), funded by the Ministry of Trade, Industry & Energy (MoTIE, South Korea), and Electronics and Telecommunications Research Institute(ETRR) grant funded by the Korean government 255711200 Fundamental Technology Research for Human- Centric Autonomous Intelligent Systems, 10%].
10

---

References
[1] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transfcrable visual models from natural language supervision. In JCML, 2021.
[2] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models, arXiv:2307.0922 2023.
[3] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Igge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. vrXiv:2303.08774 2023.
[4] Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean- Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: a family of highly capable multimodal models. arXiv:2312. 1805, 2023.
[5] Peng Wang, Shuai Bai, Sinan Tan, Shijic Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing rision-language model's perception of the world any resolution, arXiv:2409.12211 2024.
[6] Yi Wang, Kunchang Li, Xinhao Li, Jiashuo Yu, Yinan He, Guo Chen, Baoqi Pei, Rongkun Zheng, Zun Wang. Yansong Shi, et al. Internvide02: Scaling foundation models for multimodal video understanding. In ECCV, 2024.
[7] Shuai Bai, Kegin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge, Sibo Song. Kai Dang, Peng Wang. Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv:2502./3993, 2025.
[8] Jinguo Zhu, Weiyun Wang. Zhe Chen. Zhaoyang Liu, Shenglong Ye, Lixin Gu, Yuchen Duan, Hao Tian, Weijie Su, Jie Shao, et al. Interm13:: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv:2504. 2025.
[9] Boqiang Zhang. Kehan Li, Zesen Cheng. Zhiqiang Hu, Yuqian Yuan, Guanzheng Chen, Sicong Leng, Yuming Jiang. Hang Zhang, Xin Li, al. Videollama 3: Frontier multimodal foundation models for image and video understanding. arXiv:2501. 13106, 2025.
[10] Ruohong Zhang, Liangke Gui, Zhiqing Sun, Yihao Feng, Keyang Xu, Yuanhan Zhang, Di Fu, Chunyuan Li, Alexander Hauptmann, Yonatan Bisk, al. Direct preference optimization of video large multimodal models from language model reward. NAACL, 2025.
[11] Daechul Ahn. Yura Choi, San Kim, Youngjae Yu, Dongyeop Kang, and Jonghyun Choi. Isr-dpo: Aligning large multimodal models for videos by iterative self-retrospecttve dpo. In AAAI, 2025.
[12] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv:2402. 03300, 2024.
[13] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv:2501. 12948, 2025.
[14] Tom Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In NeurlPS, 2020.
[15] An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. technical report. a5Xi::2412.15115, 2024.
[16] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al- Dahle, Aiesha Lemman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv:2407.21783, 2024.
[17] Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, and Jiawei Han. Search-rl: Training Ilms to reason and leverage search engines with reinforcement learning. In COLM, 2025.
[18] Haoran Luo, Guanting Chen, Qika Lin, Yikai Guo, Fangzhi Xu, Zemin Kuang, Meina Song, Xiaobao Wu, Yifan Zhu, Luu Anh Tuan, et al. Graph-rl: Towards agentic graphrag framework via end-to-en -on reinforcement learning. 2rXiv:2507.21892 2025.

---

[19] Jinyoung Park, Ameen Patel, Omar Zia Khan. Hyunwoo Kim, and Joo-Kyung Kim. Graph elicitation for guiding multi-step reasoning in large language models. crXiv:2311.0772, 2023.
[20] Jinyoung Park, Minscok Joo, Joo-Kyung Kim, and Hy J Kim. Generative subgraph retrieval for knowledge graph-grounded dialog generation. In EMNIP, 2024.
[21] Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jianguang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, and Dongmei Zhang. Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruce In ICLR, 2025.
[22] Yizhen Zheng, Huan Yee Koh, Jiaxin Ju, Anh TN Nguyen. Lauren T May, Geoffrey I Webb, and Shirui Pan. Large language models for scientific discovery in molecular property prediction. Nat. Mach. Intell., pages 1-11, 2025.
[23] Lei Bai, Zhongrui Cai, Yuhang Cao, Maosong Cao, Weihan Cao, Chiyu Chen, Haojiong Chen, Kai Chen, Pengeheng Chen, Ying Chen, et al. Intern-sl: A scientific multimodal foundation model. 2xXiv:550..1766 2025.
[24] Jinyoung Park, Minseong Bae. Dohwan Ko, and Hyunwoo J Kim. Llamo: Large language model- molecular graph assistant. In NeurlPS, 2024.
[25] Haoning Wu, Dongxu Li, Bei Chen, and Junnan Li. Longvideobench: A benchmark for long-context interleaved video-language understanding. In NeurlPS, 2024.
[26] Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang. Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprchensive evaluation benchmark of multi-modal Ims in vidco analysis. In CVPR, 2025.
[27] Kunchang Li. Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo. et al. Mybench: A comprehensive multi-modal video understanding benchmark. In CVPR, 2024.
[28] Xiaoqian Shen, Yunyang Xiong. Changsheng Zhao, Lemeng Wu, Jun Chen, Chenchen Zhu, Zechun Liu, Fanyi Xiao, Balakrishnan Varadarajan, Florian Bordes, et al. Longvu: Spatiotemporal adaptive compression for long video-language understanding. In ICMI., 2025.
[29] KunChang Li, Yinan He, Yi Wang. Yizhuo Li, Wenhai Wang. Ping Luo. Yali Wang. Limin Wang. and Yu Qiao. Videochat: Chat-centric video understanding. arXiv preprint 2rXiv:2305.066 2023.
[30] Dohwan Ko, Ji Soo Lee, Wooyoung Kang, Byungseok Roh, and Hyunwoo Kim. Large language models are temporal and causal reasoners for video question answering. In EMNLP, 2023.
[31] Bo Li, Yuanhan Zhang, Dong Guo. Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer. TMLR, 2025.
[32] Haiyang Xu, Qinghao Ye, Ming Yan, Yaya Shi, Jiabo Ye, Yuanhong Xu, Chenliang Li, Bin Bi, Qi Qian, Wei Wang, et al. mplug-2: A modularized multi-modal foundation model across text, image and video. In ICML, 2023.
[33] Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li. Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, et al. Videollama Advancing spatial-temporal modeling and audio understanding in video-Ilms. arXiv:2406.07470 2024.
[34] Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. Vidco instruction tuning with synthetic data. TMLR, 2025.
[35] Jiyang Gao, Chen Sun, Zhenheng Yang, and Ram Nevatia. Tall: Temporal activity localization via language query. In ICCV, 2017.
[36] Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, and Juan Carlos Niebles. Dense-captioning events in videos. In ICCV, 2017.
[37] Junbin Xiao, Angela Yao, Yicong Li, and Tat-Seng Chua. Can trust your answer? visually grounded video question answering. In CVPR, 2024.
[38] Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. arXiv:24/2. 16720, 2024.
[39] Kimi Team, Angang Du Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du Chonghua Liao, et al. Kimi kl. 5: Scaling reinforcement learning with 1lm.. arXiv:2501.12599 2025.
12

---

[40] Ji Soo Lee, Jongha Kim, Jeehye Na. Jinyoung Park, and Hyunwoo J Kim. Vidchain: Chain-of-tasks with metric-based direct preference optimization for dense video captioning. In AAAI, 2025.
[41] Ji Soo Lee, Byungoh Ko, Jaewon Cho, Howoong Lee, Jaewoon Byun, and Hyunwoo J Kim. Captioning for text-video retrieval via dual-group direct preference optimization. In EMNLP Finddnns 2025.
[42] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsca Finn. Direct preference optimization: Your language model is secretly a reward model. In NeurlPS, 2023.
[43] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. In NeurlPS, 2022.
[44] Ke Zhu, Liang Zhao, Zheng Ge, and Xiangyu Zhang, Self-supervised visual preference alignment. In ACMMM, 2024.
[45] Zhaolin Gao, Jonathan D. Chang. Wenhao Zhan, Owen Oertell, Gokul Swamy, Kianté Brantley, Thorsten Joachims, J. Andrew Bagnell, Jason D. Lee, and Wen Sun. Rebel: Reinforcement learning via regressing relative rewards. NeurlPS, 2024.
[46] Ziyu Liu, Zeyi Sun, Yuhang Zang, Xiaoyi Dong. Yuhang Cao, Haodong Duan, Dahua Lin, and Jiaqi Wang. Visual-rft: Visual reinforcement fine-tuning. In ICCV, 2025.
[47] Yi Yang. Xiaoxuan He, Hongkun Pan. Xiyan Jiang, Yan Deng, Xingtao Yang. Haoyu Lu, Dacheng Yin. Fengyun Rao. Minfeng Zhu, et al. RI Ro-onvviioon: Advancing generalized multimodal reasoning through cross- formalization. arXiv:2503. /0615, 2025.
[48] Yufci Zhan, Yousong Zhu, Shurong Zheng, Hongyin Zhao, Fan Yang. Ming Tang, and Jinqiao Wang. Vision-rl: Evolving human-free alignment in large vision-language models via vision-guided reinforcement learning. arXiv:2503. 18013, 2025.
[49] Haozhan Shen, Peng Liu, Jingcheng Li, Chunxin Fang, Yibo Ma, Jiajia Liao, Qiaoli Shen, Zilun Zhang, Kangjia Zhao, Qiangian Zhang, et al. Vlm-rl: A stable and generalizable ri-style large vision-language model. arXiv:2304.077 200605 2025.
[50] Wenxuan Huang, Bohan Jia, Zijie Zhai, Shaosheng Cao, Zheyu Ye, Fei Zhao, Zhe Xu. Yao Hu, and Shaohui Lin. Vision-rl: Incentivizing reasoning capability in multimodal large language models. 27X7022503.06749 2025.
[51] Qihan Huang, Long Chan, Jinlong Liu, Wanggui He, Hao Jiang, Mingli Song. Jingyuan Chen, Chang Yao, and Jie Song. Boosting milm reasoning with text-debiased hint-grpo. In ICCV, 2025.
[52] Xiangyan Liu, Jinjie Ni, Zijian Wu, Chao Du, Longxu Dou, Haonan Wang, Tianyu Pang, and Michael Qizhe Shich. Noisyrollout: Reinforcing visual reasoning with data augmentation. In NeurlPS, 2025.
[53] Haozhe Wang, Chao Qu, Zuming Huang, Wei Chu, Fangzhen Lin, and Wenhu Chen. Vl-rethinker: Incentivizing self-reflection of vision-language models with reinforcement learning. In NeurlPS, 2025.
[54] Kaituo Feng, Kaixiong Gong, Bohao Li, Zonghao Guo, Yibing Wang, Tianshuo Peng, Benyou Wang, and Xiangyu Yue. Video-rl: Reinforcing video reasoning in milms. In NeurlPS, 2025.
[55] Xinhao Li, Ziang Yan, Desen Meng, Lu Dong, Xiangyu Zeng, Yinan He, Yali Wang, Yu Qiao, Yi Wang. and Limin Wang. Videochat-rl: Enhancing spatio-temporal perception via reinforcement fine-tuning. In NeurIPS, 2025.
[56] Chen, Yuying Ge, Rui Wang, Yixiao Ge, Lu Qiu, Ying Shan, and Xihui Liu. Exploring the effect of reinforcement learning on video understanding: Insights from seed-bench-rl. arXiv:2503.2 243777, 2025.
[57] Wang, Boshen Xu, Zihao Yue, Zihan Xiao, Ziheng Wang, Liang Zhang, Dingyi Yang, Wenxuan Wang, and Qin Jin. Timezero: Temporal video grounding with reasoning- guided Ivlm. X..i::2503.1372 2025.
[58] Peiran Wu, Yunze Liu, Miao Liu, and Junxiao Shen. St-think: How multimodal large language models reason about 4d worlds from ego-centric videos. M.Xii:2503.1254. 2025.
[59] Huajie Tan, Yuheng Ji, Xiaoshuai Hao, Minglan Lin, Pengwei Wang, Zhongyuan Wang, and Shanghang Zhang. Reason-rft: Reinforcement tunent for visual reasoning. In NewrIPS, 2025.
[60] Yuhui Wang, Hao He, and Xiaoyang Tan. Truly proximal policy optimization. In UAI 2020.
13

---

[61] Shengyi Huang, Michael Noukhovitch, Arian Hosseini, Kashif Rasul, Weixun Wang, and Lewis Tunstall. The implementation details of rihf with ppo: A case study on d;; dr summarization. In COLM, 2024.
[62] John Schulman. Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High-dimensional continuous control using generalized advantage estimation. In ICLR, 2016.
[63] Chloe Ching-Yuu Yun Hsu, Celestine Mendler-Dünne and Moritz Hardt. Revisiting design choices in proximal policy optimization. In ICLR, 2020.
[64] Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, ct al. Dapo: An open-source Ilm reinforcement learning system at scale. arXiv:2503. 14476, 2025.
[65] Jan Peters and Stefan Schaal. Reinforcement learning by reward-weighted regression for operational space control. In ICML, 2007.
[66] Jan Peters, Katharina Mulling, and Yasemin Altun. Relative entropy policy search. In AAAI, 2010.
[67] Yanwei Li, Chengyao Wang, and Jiaya Jia, Llama-vid: An image worth tokens in large language models. In ECCV, 2024.
[68] Peiyuan Zhang, Kaichen Zhang, Bo Li, Guangtao Zeng, Jingkang Yang. Yuanhan Zhang, Ziyue Wang, Haoran Tan, Chunyuan Li, and Ziwei Liu. Long context transfer from language to vision. TMLR. 2025.
[69] Ji Lin, Hongxu Yin, Wei Ping. Pavlo Molchanov, Mohammad Shocybi, and Song Han. Vila: On pre-training for visual language models. In CVPR, 2024.
[70] En Yu, Kangheng Lin, Liang Zhao, Yana Wei, Zining Zhu, Haoran Wei, Jianjian Sun, Zheng Ge, Xiangyu Zhang, Jingyu Wang, al. Unhackable temporal rewarding for scalable video milms.. In ICLR, 2025.
[71] Jiajun Liu, Yibing Wang, Hanghang Ma, Xiaoping Wu, Xiaoqi Ma. Xiaoming Wei, Jianbin Jiao, Enhua Wu, and Jie Hu. Kangaroo: A powerful video-language model supporting long-context video input. arXiv:2408. 15542, 2024.
[72] Yicong Li, Xiang Wang. Junbin Xiao, Wei Ji, and Tat-Seng Chua. Invariant grounding for video question answering. In CVPR, 2022.
[73] Antoine Yang, Antoine Miech, Josef Sivic, Ivan Laptev, and Cordelia Schmid. Zero-shot video question answering via frozen bidircctional language models. In NeurlPS, 2022.
[74] Shoubin Yu, Jaemin Cho, Pratoek Yadav, and Mohit Bansal. Self-chained image-language model for vidco localization and question answering. In NeurIPS, 2023.
[75] Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning. 8, 1992.
[76] Julia Kreutzer, Artem Sokolov, and Stefan Riezler. Bandit structured prediction for neural sequence-to- sequence learning. In ACL, 2017
[77] Khanh Nguyen, Hal Daumé III, and Jordan Boyd-Graher. Reinforcement learning for bandit neural machine translation with simulated human feedback. In EMNLP, 2017.
[78] Wouter Kool, Herke van Hoof, and Max Welling. Buy reinforce samples, get baseline for free! In ICLRW, 2019.
[79] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. In NeurlPS, 2019.
[80] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Huggingface's 's transformers: State-of-the-art natural language processing. In EMNLP demo, 2020.
[81] Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, Nathan Lambert, Shengyi Huang, Kashif Rasul, and Quentin Gallouédec. Trl: Transformer reinforcement lcarning. https 2020.
//gihub.. haaiuub..oom/uugggee www.//gggingaace///o///ooooo/
[82] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and lon Stoica. Efficient memory management for large language model serving with pagedattention. In SOSP, 2023.
14

---

[83] Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Antonino Furnari, Evangelos Kazakos, Jian Ma. Davide Moltisanti, Jonathan Munro, Toby Perret, Will Price, et al. Rescaling egocentric vision: Collection, pipeline and challenges epic-kitchens-100 100. IJC,, pages 1-23, 2022.
[84] Kristen Grauman, Andrew Westbury. Eugene Byme, Zachary Chavis, Antonino Furmari, Rohit Girdhar, Jackson Hamburger, Jiang, Miao Liu, Xingyu Liu, Ego4d: Around the world 3,000 hours of egocentric video. In CVPR, 2022.
[85] Chen, Yuying Yixiao Ge, Mingyu Ding, Bohao Rui Wang. Ruifeng Xu, Ying Shan, and Xihui Liu. Egoplan- bench: Benchmarking multimodal large language models for numan-leve planning. arXiv:2312.0672,, 2023.
[86] Lu Qiu, Yi Chen, Yuying Ge. Yixiao Ge. Ying Shan, and Xihui Liu. Egoplan-bench2: A benchmark for multimodal large language model planning in real-world scenarios. arXiv:2412.0444) 2024.
[87] Jihan Yang. Shusheng Yang. Aujali W Gupta, Rilyn Han, Li Fei-Fei, and Saining Xie, Thinking in space: How multimodal large language models see, remember, and recall spaces. In CVPR, 2025.
[88] Kairui Hu, Penghao Wu, Fanyi Pu, Wang Xiao, Yuanhan Zhang, Xiang Yue, Bo Li, and Ziwei Liu. Video- mmmu: Evaluating knowledge acquisition from multi dusti-diss professional videos. arXiv:2501.13826, 2025.
[89] Yilun Zhao, Lujing Xie, Haowei Zhang. Guo Gan, Yitao Long, Zhiyuan Hu, Tongyan Hu, Weiyuan Chen, Chuhan Li, Junyang Song, et al. Mmvu: Mcasuring expert-level multi multi-discipline video understanding. In CVPR, 2025.
[90] Yuanxin Liu, Shicheng Li. Yi Liu, Yuxiang Wang, Shuhuai Ren, Lei Li, Sishuo Chen, Xu Sun, and Lu Hou. Tempcompass: Do video Ilms really understand videos? In ACL-Findings, 2024.
15

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification:
In Abstract and Introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: In the appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
16

---

Justification:
We provide the assumptions and derivation in the appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: In Experiments and appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
17

---

Answer: [Yes]
Justification: In Abstract.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ /pippe.cc/pullic/guiii wwwiiccgguie//eeeeeeeeeis CoodeSubeiiieeee ssi wswonnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: In Experiment and the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [NA]
Justification:
We conduct the experiment in the single run following other works.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
18

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: In the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics sattps://nnt htttt::://nerris.... wwwpuriis.ccpppubule http:///ccpppcc//..cccccc.......
Answer:
[Yes]
Justification:
We follow NeurlPS Code Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: In the appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
19

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We do not release any datasets.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: In the appendix.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwppppswpptseeee wwworeeswthhhe....c. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
20

---

Answer: [Yes]
Justification: In the appendix.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [Yes]
21

---

Justification:
In the appendix.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy htte::////earrppcccc////f/rreees for what should or should not be described.
22

---

A Discussion and Derivation of Reg-GRPO
A.1 Deriving the optimum of the KL-constrained reward maximization optimization
We derive Eq. from Eq. 3
following [22
We optimize xa with the following objective:
arg max Es, yoma R (a, y) A 18 [DLL (ne (-ax) |Tee (-|z))]
(10)
where R is the reward function, Te.. is the old policy model, and (A 2 0) denotes a hyperparameter. We can obtain closed-form solution to the above relative-entrooy--eeulaiiee maximization problem.
T arg max Eu (- R. (x) y) Ex [[DxL. (TT (-1z) ||Te (-1x))
To (3/ a+)
arg max Eg (-
R (ar, y) A log
Ba (y/
log Te (y//c)
arg min Ex (
R (2. y)
Tiuu (r))))
(11)
(y/ xx)
arg min E. (
log Th (3/2) ( exp { R (x: 3))
( x:)
arg min ESe Ex (v)
log
log Z (x))
Z(a) Tha
exp
1 R (x.
where Z (x) (x)= function is only dependent on and the old policy maa
Ey Tew (3)a) exp ( (xx, y)) is partition function. Please note that the partition
Now let Fy be defined
To (8/z) = Z (x) Thas (y) exp exp (1 (2. (2..)))
(12)
can be seen as valid probability distribution as मe (a))== 2 0 for all y and E, no (y/lz) Since (x) is not function of y. the above minimization problem can be formulated
1.
Hee (y|z) log Z (x)
arg min En Em. log
मe (y) (az)
(13)
arg min Eg [i (To (y/(²) I| (z)=)) - log Z (x)]
Since the partition function Z (x) not dependent the optimal x² is the policy that minimizes the first KL term. Since the optimal KL-divergence achieved if and only if two distributions are identical, we have optimal solution as:
= Z (x:) Tow ()) exp (GR(2):))) Vax, y-
3 (y|æ)
(14)
A.2 Deriving the reward in terms of the optimal policy
The reward can be reorganized under the optimal policy. We can invert Eq. 14] as follows:
( R (2x (x.
म (z)(z)
exp
Z (xx)
(3) ax) Th
T ( (x:
1 R (xx 8) = logg (x) + log
(15)
Te.. (3/ (c)
R (x, 8) = A. (log (og / (x) + log
T (3) x3)
Va, y.
Te
(y) x))
23

---

A.3 Deriving the advantage in terms of the optimal policy.
The advantage A is defined as
R (z, y))))) pr
A
(16)
where denotes the average and standard deviation values of a set of rewards in the group, respectively. We can rewrite the advantage in terms of the optimal policy in Eq. (15) as follows:
A /² (x, ()) + log (x)) - (a Eÿ= p² (sc, y + log Z (ar)
Op"
(17)
0² (x, y()) up p² (x, y) = log
Te (y
0p"
Tu (y ar)
where pp-; 07 are mean and standard deviation of {pr (a, (x) (0) ) respectively. Interestingly, we can see that (x) is removed.
A.4 Reg-GRPO
Based on Eq. (7). our Reg-GRPO (Regressive GRPO) is to learn the model Ta to directly predict the advantage follows:
CReE-GRPO (0) = 18 (x ( - { -1)) - DD [Tol|TTe|) }.
(18)
1 p (2, (0) up p (x, 3) log
(y
Thas
(z)
where o are the mean and standard deviation of { (x, y Fi=p- respectively. For simplicity, we omit the KL divergence between the policy and the reference model.
Discussion. Motivated by Group-Relative Policy Optimization (GRPO), the strength of Reg-GRPO lies in its regression- based approach to advantage estimation, unlike other methods that implicitly derive policy updates from preference probabilities. By directly regressing the group-normalin noruaioom target, GRR- leads to more precise updates, as the model is not just learning which response is better, but also how much better it is, according the advantage.
In relation to Direct Preference Optimization (DPO), Reg-GRPe offers a different perspective on leveraging preference data. While DPO learns which response is preferred, Reg-GRPO attempts to capture finer-grained signal about degree of preference by directly regressing the advantages. This could be particularly beneficial in scenarios where the difference in quality between preferred and dispreferred responses varies significantly. Furthermore, the group-wise normalization inherent in GRPO, and carried into Reg-GRPO, can offer robustness when dealing with diverse and potentially inconsistently scaled preference data, which may require more carcful handling in pairwise DPO setup.
In contrast REBEL one of the novel regression based reinforcement- sentcemmntt-ie-tuning fine-tuning methods, which regresses the unnormalized pairwise reward gap differences between sampled outputs, our proposed Reg-GRPO GRPO framework directly learns to predict the group-based normalized advantage. This shift from pairwise to group-level regression is a design choice that addresses the high variance typically observed in the outputs of video L.LMs. By normalizing log-probability ratios within each group, Reg-GRPO mitigates scale discrepancies across batches and enhances the optimization during training. As a result, Reg-GRPO offers a more scalable and effective learning paradigm for fine-tuning language models using preference-based feedback.
B Reward functions
To compute verifiable rewards, we follow existing works 54 GRPO.
56
55
that fine-tune VideoI.LMMs with

---

Format reward. Following existing GRPO-based works [44, 56 55,, we employ format re- ward to ensure that the model generates outputs in the desired format. For example, the model is trained to output the thought process with <think>. </think> followed by the answer with <answer>... </ /aasswr>>. We use regular expressions to verify whether the outputs satisfy the specified format. The format reward Rormat is applied to all tasks:
f0, if output does not match the format, 1,
Rformat
(19)
if output matches the format.
Accuracy reward. For tasks such as question answering, we employ an accuracy reward, which is formulated as:
0, 1,
à # if a,
Race
(20)
where a is the ground-truth answer and à is the model prediction, which is extracted from the regular expressions with <answer>... </a.s//w/////>
IoU reward for temporal perception. We utilize an loU reward to assess the model's ability to identify the temporal segment described by the input query and localize the target within the video. The loU reward is defined as:
PP n Q
R
(21)
PP U Q
where and Q are the model prediction set and ground-trute set, respectively. For the temporal grounding task, P and Q are defined as the timestamps of events within the video.
C Detailed Experimental Settings
C.1 Implementation Details
We implement our code using the Py library [79]. We also adopt the Hugging Face Transformers library [80] and the TRL library [8] to post-train Video Large Language Models (VideoLLMs). For inference and rollout, we use vLL.M 82 For all the experiments, we fine-tune only large language model while keeping the visual encoder frozen. We use Qwen2.5-VL. 7 and Dwen2-VL. [ as base VideoLLMs. We use NVIDIA A 1100 GPUs for 3B models and NVIDIA H200 GPUs for 7B models. In addition, we use LLM-based tools for implementation and for correcting grammatical errors in the writing.
For the SEED-Bench-RI dataset, we apply KL-divergence regularizer between the model Qwen2.5-VE VL as the default base VideoLLM, and use Qwwn2.55VL-33 for all analyses. We set the number of generations in the group as for all the settings. For Deep Video-RI, I, we maintain a reward history using the most recent W 100 samples, and GRPO does not adopt safeguards based empirical study. To train the model on the SEED-Bench-RP dataset, we limit the maximum number of sampled frames per input video to 16 with frame resolution of 252 252, and then append the frame indicating the current observation as an additional image input, following LEED-Bench-A [56]. To
and the reference model गet with coefficient 0.1, following prior GRPO works [12 56). We use
train the model using NEXTGQA, we follow the experimental setups in VideoChat-RI (55
C.2 Evaluation Metrics
Accuracy. The accuracy metric measures the ratio of correct predictions that match the ground-truth answers for given questions, which as follows:
1 N
Acc
Σ =a;= N 1=11
(22)
where N is the number of samples, à; is the prediction, and a; the ground-truth answer. mloU. The mloU (i.e., mean Intersection over Union) metric calculates the average ToU over all samples, where IoU represents the similarity between the predicted and ground- -tunh timestamps,
25

---

which can be formulated as:
N
N (st (s) (s) ( e,) e8)
1
1 N
ps Qg
1 N
mloU = IoU,- N
=
(23)
Ps U
where N is the number of samples, p; (s,, e is the prediction, q; (8) { is the ground truth, and of. ex sf, E, denote the start and end timestamps of the prediction and ground truth, respectively.
@@m. 35 proposed "R@n, IoU = m" metric for the temporal grounding tasks that measures the percentage of queries where least one of the top-n predictions has an IoU higher than with the ground-truth. Following (55]. we adopt Rüm as top-1 variant of "R@n, IoU = m defined as:
1 (IoU 2 m. Dj (1)).......)...
"R@n, IoU mm"
(24)
R@m "R@1, IoU = m
Σ 1 (IoU; 2 m), where IoU, = IoU!
where N is the number of samples, m is the IoU threshold, and IoU the IoU between the j-th prediction (ranked among the top-n predictions) and the ground truth.
D RL Baselines
In this section, we describe the baseline methods used to compare against Reg-GRPO in Table of the main paper.
DPO [42) aligns model outputs with human preferences using pairwise comparisons. For DPO, we sample model outputs from fixed reference model.
Online DPO 42 also adopts direct preference optimization to learn the model. Compared to standard DPO, samples outputs from the old policy model, which evolves during training, following GRPO 12
REINFORCE 775-77 is classic policy-gradient algorithm that updates the model using the reward- weighted log-likelihood of the outputs. Generally, also samples the outputs from the old policy model Ta
REINFORCE Leave-One-Out (RLOO) 78 designed reduce the variance of gradient estimates in REINFORCE. Instead of using the reward directly as a weight, it uses Monte Carlo estimate to compute a bascline and subtracts from the reward when forming the weight. Same REINFORCE. it samples the outputs from the old policy model Tow..
directly regresses the pairwise reward gap, which motivates to apply the regression- based fine-tume tne-tuning methods. Different from our work that directly predicts the group- nomop-nommllized advantage, it regresses the unnormalized pairwise reward gap.
REBEL 45
Reward- Reggessinn (Eq. (5)) our baseline that directly regresses the reward by approximating Z (x) with Monte-Carlo sampling. Since Z (x) is not accurate and relying solely on the reward introduces high variance, performs worse than our Reg-GRPO.
E Datasets
(56] is a dataset designed to evaluate the effectiveness of post-training methods in the context of video understanding capabilities of MLLMs. Specifically, the dataset incorporates Epic-Kitchens 883 and Ego4D 84 as videos and EgoPlan- Bench [85) and EgoPlan-Bent BennhB [86] as benchmark sources to construct a hierarchical validation structure, enabling evaluation across diverse real-world scenarios.
Long Vingiideeeenh 225 contains 3,763 videos and 6,678 QA pairs, where videos are diverse in domain (e.g., Life, Movie), task (e.g., scene-referred event, object before/after text), and duration. In particular, the video durations are divided into four progressive groups, (8s, 15s]. (15s, 60s], (180s,
26

---

600s). (900s, 3600s], with an overall average of 100s, facilitating the assessment of the model's understanding of long-context interleaved multimodal inputs.
VSI-Bench 88 a dataset proposed to evaluate the visual-spatial intelligence capabilities of MILLMs, comprises over 5,000 question-answer pairs and 288 real videos. Specificallyy eight tasks (object count, relative distance, relative direction, route plan, object size, room size, absolute distance, appearance order) three types (configurational, measurement estimation, spatiotemporal) are defined within the dataset.
Video-MMMU 88 consists of 300 expert-level videos and 900 questions, targeting the evaluation of the knowledge acquisition capabilities in MLLMs. Inspired by the human process of acquiring knowledge to solve challenging problems, the questions in the dataset are auman-annotater across six disciplines (Art, Business, Science, Medicine, Humanities, Engineering) and aligned with three stages: Perception, Comprehension, and Adaptation.
MMVU comprises 3,000 expert-annotated question-answer pairs and 1,5299 specialized- -omainn videos covering 27 subjects across 4 key disciplines (Science, Healthcare, Humanities & Social sci- ences, Engineering), and aims to evaluate the expert-level, knowledge-intensiv video understanding abilities of MILMs. Following [54], report the performance on multiple-choice QA.
MVBench [27 serves as benchmark to assess temporal comprehension capabilities of MLLMs, and consists of 20 challenging video understanding tasks that require reasoning beyond a single frame. In particular, the dataset built upon videos sourced from various henchmarks, enabling the evaluation of MILMss general ability for open- world temporal understanding. Since each task contains 200 question-answer pairs, we conduct evaluation on total of 4,000 question-answer pairs.
TempCompass [00 is designed for evaluating the temporal perception ability of MLLM, based on 5 basic temporal aspects (Action, Speed, Direction, Attribute change, Event order) and 10 fine-grained sub-aspects (e.g., relative speed, camera direction, combined change). We report results on overall performance, including all four tasks (Multi-choice QA, Yes/No QA, Caption matching, Caption generation), for comparison with prior work [54]]
Video-MME 26 is dataset for evaluating the general video understanding capabilities of MLLMs, consisting of 900 videos and 2,700 question-answer pairs, where the videos are constructed with variation in both type and temporal duration. Specifically, the dataset covers 6 key domains and 30 sub-class video types, and each video is categorized as short (< 2 mins), medium (4-15 minutes), or long (30-60 minutes) depending on its duration length. We report the average performance across all temporal duration splits, without using subtitles.
NEXTGQA (37 is temporal grounding QA dataset consisting of 5,417 videos, 43,043 QA pairs, and 10,531 timestamp labels. As temporal segment annotations are available only in the validation and test splits, use the validation and test splits for the model training and model evaluation, respectively.
F Broader Impacts and Limitations
F1
Broader Impacts
We propose video large language model named DeepVideo-R1, trained with Regressive GRPO (Reg- GRPO) and difficulty-awart data augmentation. Deep Video-R1 is broadly applicable to complex video reasoning tasks. We believe that Deep Video-RI itself does not introduce new negative impacts. However, as the model is based on pretrained language and vision models, it may generate biased outputs related to race, religion, culture, and gender, which could lead to misuse. addition, training VideoLLMs may result in CO2 emissions, which contribute to global warming.
F.2 Limitations
Our DeepVideo-RP R1 is built upon large-scale pretrained video large language model and fine-tuned on video reasoning datasets to leverathe the rich world knowledge embedded in the pretrained models. However, it remains unclear whether there is any overlap between pertaining content and downstream evaluation benchmarks. This uncertainty may introduce a risk of implicit data leakage. Furthermore, since Vide--R is based on VideoLLM, it has following limitations: high computational and
27

---

memory requirements. As DeepVideo-RI is fine-tuned on top of such models, it may inevitably inherit these challenges.
28


Reviews:
[{"review_id": "Lk29z53dnO", "reviewer": "Reviewer_ZcMY", "summary": "The paper introduces DeepVideo-R1, a novel approach to enhancing the reasoning capabilities of VideoLLMs. The authors identify two key issues in GRPO used for training VideoLLMs: the reliance on heuristic stabilizers and the vanishing advantage problem. To address these, they propose a solution combining Regressive GRPO and difficulty-aware data augmentation. Through experiments across various video reasoning benchmarks, the model demonstrated significant performance improvements over existing models like Qwen2.5-VL and Intern3-VL. The results show that the DeepVideo-R1 model generalizes better to both in-distribution and out-of-distribution tasks, making it more robust than other VideoLLMs.", "questions": "I am curious about the regression objective without any clip function inside. Since the original PPO or GRPO will try to truncate the loss when there are some significant shifts. But the proposed method does not follow this and make any ablation about it. Is the training process stable? There should be some ablation curves to illustrate this. \n\nAnd predicting advantage seems more like a reweighted imitation algorithm, but there is no related works or experiments.", "limitations": "As follows.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n\n1. The work presents a well-structured and thorough explanation of the novel methods. The authors validate their approach with strong experimental results across multiple challenging benchmarks.\n\n2. The paper is clearly written and the methodology is well-explained, making it accessible to readers with a background in reinforcement learning and multimodal AI.\n\n3. The combination of Reg-GRPO with difficulty-aware augmentation is novel, presenting a fresh approach to improving video reasoning in LLMs.\n\nWeakness:\n\n1. **Limited Discussion of Computational Complexity:** While the paper provides extensive experimental results, it could further discuss the computational cost of the proposed methods, especially with large video datasets.\n\n2. **Generalizability of Difficulty-Aware Augmentation:** The difficulty-aware data augmentation strategy is an interesting approach, but it would be beneficial to explore how well this method scales across different types of video reasoning tasks beyond those tested in the paper.", "was_revised": true, "final_justification": "Resolved: 1) Compute practicality clarified—unchanged inference cost, moderate training overhead with clear gains; (2) Difficulty-aware augmentation improves results across five extra benchmarks; (3) Reg-GRPO stable without clipping (endpoint ablations); (4) Relation to reweighted imitation learning now articulated.\n\nUnresolved: Lack of training curves/variance stats; limited full compute reporting (GPU hours/memory/scaling); no direct comparison to advantage-weighted imitation baselines; limited sensitivity analyses.\n\nCamera-ready asks: Add training dynamics/variance plots, fuller compute/scaling details, a small imitation-baseline comparison, and sensitivity/error analyses.", "rebuttal": "We sincerely thank the reviewer for the positive and constructive review of our paper. We are encouraged that the reviewer recognizes multiple strengths in our paper, including: \n\n**(i) novel and fresh approaches**\n\n**(ii) well-structured and thorough explanation**\n\n**(iii) strong experimental results across multiple challenging benchmarks**\n\nWe believe that incorporating this constructive feedback significantly enhances the quality of the paper. Below, we provide the response to the given comments in detail:\n\n---\n\n**Q1. Discussion of computational complexity. [Weakness 1]**\n\n**A1.** We appreciate the reviewer’s suggestion to study the computational cost of our proposed method. To explore the computational cost, we report the average training and inference time per video across three different benchmarks (SBR-L1, SBR-L2, SBR-L3) in the table below. We use Qwen2.5-VL-Instruct 3B as a base multimodal large language model and NVIDIA A100 GPUs following the experimental setting in our supplement.\n\n|  | SBR-L1 | SBR-L2 | SBR-L3 | Inference time (sec/video) | Training time (sec/video) |\n| --- | --- | --- | --- | --- | --- |\n| GRPO | 39.56 | 41.01 | 35.35 | **3.08** | **2.97** |\n| DeepVideo-R1 (w/o D.A. Aug) | 44.20 | 44.15 | 39.52 | **3.08** | **2.97** |\n| DeepVideo-R1 | **48.05** | **51.07** | **43.98** | **3.08** | 4.56 |\n\nThe table shows that the inference time remains constant across methods since our approach focuses on training. Notably, DeepVideo-R1 significantly improves the performance of GRPO without increasing inference cost. Even though the difficulty-aware augmentation slightly increases training time, it has a superior performance gain of 6.92 on SBR-L2 compared to the model without difficulty-aware data augmentation. This demonstrates the effectiveness of the augmentation by addressing the vanishing advantage problem.\n\n---\n\n**Q2. Generalizability of difficulty-aware augmentation. [Weakness 2]**\n\n**A2.** To evaluate the generalizability of difficulty-aware augmentation on different types of video reasoning tasks, we additionally conduct the experiments by comparing the performance of DeepVideo-R1 with and without difficulty-aware augmentation on five video reasoning benchmarks. The experimental results are described in the table below. The table demonstrates that our difficulty-aware data augmentation strategy is broadly applicable and effective beyond SEED-BENCH-R1 datasets.\n\n|  | Video-MMMU | MMVU (mc) | MVBench | TempCompass | Video-MME (w/o sub) |\n| --- | --- | --- | --- | --- | --- |\n| DeepVideo-R1 (w/o D.A. Aug.) | 39.1 | 57.5 | 48.6 | 62.3 | 50.3 |\n| DeepVideo-R1 (with D.A. Aug.) | **40.7** | **59.0** | **49.6** | **63.1** | **51.1** |\n\n---\n\n**Q3. Ablation of Reg-GRPO with/without clipping function. [Question 1]**\n\n**A3.** Following the reviewer’s suggestion, we conduct additional experiments comparing DeepVideo-r1 trained with and without applying the clipping mechanism to investigate the impact of the clipping function on training stability. As NeurIPS policy does not allow figure uploads during the rebuttal phase, we provide only final performance instead of providing training curves. The experimental results are presented in the table below. The results show that the performance is comparable with and without clipping functions across all datasets, indicating that Reg-GRPO remains stable and effective without the clipping function. We will include the full training curve and analysis in our final version if the paper gets accepted. \n\n|  | SBR-L1 | SBR-L2 | SBR-L3 |\n| --- | --- | --- | --- |\n| DeepVideo-R1 (with clip. function) | 47.87 | **51.20** | 43.64 |\n| DeepVideo-R1 (without clip. function) (ours) | **48.05** | 51.07 | **43.98** |\n\n---\n\n**Q4. Relation to reweighted imitation learning algorithm. [Question 2]**\n\n**A4.** Thank the reviewer for the insightful comment regarding the reweighted imitation learning and our Regressive GRPO (Reg-GRPO). As the reviewer pointed out, our Reg-GRPO is related to prior imitation learning methods [1,2,3], which are intuited by a KL-regularized optimization problem. However, while they focus on learning a reward function relying on **”expert demonstrations”** to replicate expert actions, our Reg-GRPO does **NOT** require any expert trajectories or access to an expert policy. Instead, Reg-GRPO directly optimizes a policy by regressing onto the estimated advantages, which are derived from the model’s own rollouts. \n\nWe agree that discussing Reg-GRPO with reweighted imitation learning would improve the paper, and we will include a more comprehensive discussion and comparison in the final version. We are also happy to incorporate any additional related works the reviewer may suggest.\n\n[1] Jacq, Alexis, et al. \"Learning from a learner.\" ICML, 2019.\n\n[2] Watson, Joe, Sandy Huang, and Nicolas Heess. \"Coherent soft imitation learning.\" NeurIPS, 2023.\n\n[3] Xu, Haoran, et al. \"An Optimal Discriminator Weighted Imitation Perspective for Reinforcement Learning.\" ICLR, 2025."}, {"review_id": "VkpvaIylou", "reviewer": "Reviewer_4urM", "summary": "This work rewrites the GRPO loss to mitigate potential gradient suppression and employs difficulty-aware augmentation to address the possible vanishing advantage issue. Based on these two components, the authors propose DeepVideo-R1, which significantly enhances MLLMs' video understanding capabilities through reinforcement learning. The experimental results validate the effectiveness of the algorithm.", "questions": "1. There are instances of unclear notation in the paper. Some symbols in Section 3 lack definitions, and $\\epsilon$ appears to have different meanings in Equations 2 and 3, while $\\pi^\\star_{\\theta_{old}}$ in Equation 8 should likely be $\\pi_{\\theta_{old}}$.\n2. Minor theoretical issues: \n\n    a) In Equation 8, there **seems** to be no $\\epsilon$. This proof is overly brief in the Appendix, while the previous content derived from DPO occupies excessive length. A further elaboration on this proof would be beneficial.\n\n    b) Line 149 should be '*are mean and standard deviation of $log ρ^\\star$, respectively*'. \n\n    c) Is Equation 5  a necessary condition or an equivalent condition for  $\\pi^\\star$ to be an optimal policy? If it’s a necessary condition, how to confirm that the update direction of Equation 9 is correct? Further clarification may be needed.\n\n3. Aside from adding noise for data augmentation related to videos, other components do not seem specific to video tasks. Can Reg-GRPO be directly applied to textual tasks? Or what implicit connections exist between it and video tasks?\n\n4. A more extensive evaluation on various video benchmarks will provide a comprehensive assessment of model capabilities.\n\n5. The methodology regarding data usage seems unclear. Are SEED-Bench-R1 and NextGQA datasets mixed together for RFT or used separately? If they are used separately, can they also be combined?\n\n6. This paper claims that the clipping operation of GRPO may lead to zero gradients, while Reg-GRPO improves upon this point. It is suggested that a comparison between GRPO and Reg-GRPO regarding distances between $\\pi_\\theta$ and $\\pi_{\\theta_{old}}$ during training would better validate this claim.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\nIt is valuable to rewrite the GRPO loss term using the closed-form solution from DPO to replace the Clip operation that could lead to zero gradients. Ultimately, the loss aligns the generation probabilities with the advantages in a reasonable manner. The difficulty-aware data augmentation is reasonable for constructing a dataset without the vanishing advantage issue. The experiments reflect the efficacy of Reg-GRPO.\n  \nWeaknesses:\nReg-GRPO does not specifically target characteristics of video tasks, and there are some perplexing aspects in the theoretical section. The experimental part could be further enriched to improve overall quality. Please refer to the Questions part, and I would be willing to raise my score if the authors address my concerns.", "was_revised": true, "final_justification": "My concerns are mainly resolved throughout the rebuttal. I think this work is valuable. Please don’t forget to revise the faults in theoretical analysis and make clear statements between Reg-GRPO and video understanding.", "rebuttal": "We sincerely thank the reviewer for the valuable and constructive review. We are encouraged that the reviewer highlights multiple strengths of our work, including: \n\n**(i) A principled reformulation of GRPO**\n\n**(ii) Effective advantage alignment via Reg-GRPO**\n\n**(iii) Difficulty-aware data augmentation that addresses the key limitation of GRPO**\n\n**(iv) Comprehensive empirical validation**\n\nWe believe that incorporating this constructive feedback significantly enhances the quality of the paper. Below, we respond to the reviewer’s insightful questions below:\n\n---\n\n**Q1. Typos and clarity of notations [Question 1,2(b)].**\n\n**A1.** We appreciate the feedback. We identified the mentioned typographical issues and some notations with a lack of explanation. We will reflect them in the final version if the paper gets accepted.\n\n---\n\n**Q2. More detailed derivation of reg-grpo. [Question 2(a)]**\n\n**A2.** As requested, we specifically clarify why $\\epsilon$ is needed in Eq. (8) of the supplement as below.\n\nOur optimization objective is :\n\n$$\n\\max_{\\pi} \\ \\mathbb{E}_{ x,  y\\sim\\pi\\_\\theta (\\cdot |  x )} \\mathcal{R} ( x,  y ) - \\epsilon \\cdot \\mathbb{D}\\_{\\text{KL}} (\\pi\\_\\theta || \\pi\\_{\\theta\\_{\\text{old}}} ),\n$$\n\nwhere $\\epsilon$ is a regularization strength. \n\nFrom Section A.1 of the supplement in the DPO paper [1] and our supplement, we can obtain that the optimal policy for our objective is \n\n${\\pi}\\_{\\theta}^* ( y |  x  ) = \\frac{1}{Z ( x )} \\pi\\_{\\theta_{\\text{old}}} ( y |  x  )\\exp (\\frac{1}{\\epsilon} \\mathcal{R} ( x,  y  )  ),\\ Z(x) = \\sum_y \\pi\\_{\\theta\\_{\\text{old}}} (y|x)\\exp  (\\frac{1}{\\epsilon} \\mathcal{R}(x,y)  )$ \n\nwhere $\\mathcal{R} (x,y )$ is the reward value given $x$ and $y$.\n\nFrom this equation, the reward $\\mathcal{R} ( {x}, {y} )$ can be formulated under $\\pi_{\\theta_{\\text{old}}} ( {y} |  {x}  )$ and $\\pi_\\theta ( {y} |  {x}  )$.\n\nBy multiplying both sides of Eq. (5) by $Z ( {x} )$, we can get\n\n$Z ( x ) \\cdot {\\pi}\\_{\\theta}^* ( y|  x ) =  \\pi\\_{\\theta\\_{\\text{old}}} ( y |  x  )\\exp (\\frac{1}{\\epsilon} \\mathcal{R} ( x,  y )  ),$ \n\n$Z ( x )\\cdot \\frac {{\\pi}\\_{\\theta}^* ( {y} |  {x}  )}{\\pi\\_{\\theta\\_{\\text{old}}} ( {y} |  {x}  )} =  \\exp (\\frac{1}{\\epsilon} \\mathcal{R} ( x,  y  )  ),$  \n\nNext, we take the logarithm of both sides and then multiply both sides by $\\epsilon$ to obtain the following equation (Eq. (6) of the supplement):\n\n$\\mathcal{R} ( {x},  {y}  )  = \\epsilon \\cdot  (\\log Z ( {x}  ) + \\log  (\\frac{\\pi_{\\theta}^* ( {y} |  {x}  )}{\\pi_{\\theta_{\\text{old}}} ( {y} |  {x}  )}  ) ),$\n\nThe advantage of GRPO is defined as \n\n$A^{(i)} = \\frac{\\mathcal{R} ( {x},  {y}^{(i)}  )-\\mu_r}{\\sigma_r}$. (Eq. (7) of the supplement), where $\\mu_r, \\sigma_r$ are mean and standard deviation of a set of reward values $\\\\{\\mathcal{R} ( {x},  {y}^{(i)}  ) \\\\}_{i=1}^G$.\n\nUsing Eq.(6) of the supplement, we can reformulate the advantage by substituting $\\mathcal{R} ( {x},  {y} )$ with $\\epsilon \\cdot  (\\log Z ( {x}  ) + \\log  (\\frac{\\pi_{\\theta}^* ( {y} |  {x}  )}{\\pi_{\\theta_{\\text{old}}} ( {y} |  {x}  )}  ) )$. It can be written as\n\n$$\n\\hat{A}^{(i)} = \\frac{\\epsilon \\cdot \\log Z ( {x}  ) + \\epsilon \\cdot\\log  (\\frac{\\pi_{\\theta}^* ( {y}^{(i)} |  {x}  )}{\\pi_{\\theta_{\\text{old}}} ( {y}^{(i)} |  {x}  )}  ) - \\epsilon \\cdot \\log Z ( {x}  ) - \\epsilon \\cdot \\frac{1}{G} \\sum_{i=1}^G\\log  (\\frac{\\pi_{\\theta}^* ( {y}^{(i)} |  {x}  )}{\\pi_{\\theta_{\\text{old}}} ( {y}^{(i)} |  {x}  )}  )}{\\sigma_\\rho}\n$$\n\n$$\n\\hat{A}^{(i)} = \\frac{ \\epsilon \\cdot \\log  (\\frac{\\pi\\_{\\theta}^* ( {y}^{(i)} |  {x}  )}{\\pi\\_{\\theta_{\\text{old}}} ( {y}^{(i)} |  {x}  )}  ) - \\epsilon \\cdot \\frac{1}{G} \\sum\\_{i=1}^G\\log  (\\frac{\\pi\\_{\\theta}^* ( {y}^{(i)} |  {x}  )}{\\pi\\_{\\theta\\_{\\text{old}}} ( {y}^{(i)} |  {x}  )}  )}{\\sigma\\_\\rho} \n$$\n\n$$\n= \\epsilon \\cdot \\frac{  \\log \\rho^* ( x, y^{(i)}  ) - \\mu\\_\\rho^\\*}{\\sigma\\_\\rho^\\*},\\quad \\rho^* ( {x},  {y}  ) = \\frac{\\pi\\_{\\theta}^* ( {y} |  {x}  )}{\\pi\\_{\\theta\\_{\\text{old}}} ( {y} |  {x}  )}, \\quad  \\mu\\_\\rho^* = \\frac{1}{G}\\sum\\_{i=1}^G\\log  (\\frac{\\pi\\_{\\theta}^* ( {y}^{(i)} |  {x}  )}{\\pi\\_{\\theta\\_{\\text{old}}} ( {y}^{(i)} |  {x}  )}  ), \\quad \\sigma\\_\\rho^* = \\text{Std} (\\\\{\\log \\rho^* ( {x},  {y}^{(i)}  ) \\\\}\\_{i=1}^G  )\n$$\n\n\nThus, we can obtain Eq. (8) and $\\epsilon$ is required. It arises naturally from the KL-regularized RL objective and controls the strength of the regularization and scale of the reward. We have updated the supplement to make this derivation more explicit and easier to follow.\n\n---\n\n**Q3. Is Eq (5) in the supplement a necessary condition or an equivalent condition for** $\\pi^*$ **to be an optimal policy? [Questions 2(c)]**\n\n**A3.**  $\\pi^*$ is an optimal policy and the condition is **an equivalent (necessary and sufficient) condition** as proven from Section A in the supplement of DPO paper [1]. \n\nBased on the result, our derivation is provided in Section A.1 of our supplement. \n\nWe provide a sketch of proof here for the convenience. \n\nThe policy optimization objective is given as:\n\n$$\n\\pi_\\theta^* = \\text{arg}\\max\\_\\pi \\ \\mathbb{E}\\_{ {x},  {y}\\sim\\pi (\\cdot |  {x} )} \\mathcal{R} ( {x},  {y} ) - \\epsilon \\cdot \\mathbb{D}\\_{\\text{KL}} (\\pi\\_\\theta || \\pi\\_{\\theta\\_{\\text{old}}} )\n$$\n\nFrom our supplement and other works, the above problem can be reformulated as\n\n$$\n\\arg \\min_\\pi\\ \\mathbb{E}\\_{ {x}}\\left[\\mathbb{D}\\_{\\text{KL}} (\\pi_\\theta  ( {y} |  {x}  )||\\bar{\\pi}_\\theta  ( {y}| {x}  )  )-\\log Z ( {x} ) \\right],\n$$\n\nwhere $\\bar{\\pi}\\_{\\theta} ( {y} |  {x}  ) = \\frac{1}{Z ( {x} )} \\pi\\_{\\theta\\_{\\text{old}}} ( {y} |  {x}  )\\exp (\\frac{1}{\\epsilon} \\mathcal{R} ( {x},  {y}  )  )$. Since the partition function does not depend on $\\pi$, the optimal policy minimizes the KL regularization.\n\nThe minimum KL is achieved **if and only if** two distributions are identical, the optimal solution is Eq. (5) as follows\n\n$$\n{\\pi}\\_{\\theta}^* ( {y} |  {x}  ) = \\frac{1}{Z ( {x} )} \\pi_{\\theta_{\\text{old}}} ( {y} |  {x}  )\\exp (\\frac{1}{\\epsilon} \\mathcal{R} ( {x},  {y}  )  ), \\forall  {x},  {y}\n$$\n\nThus, Eq (5) is a necessary and sufficient (equivalent) condition for $\\pi$ to be an optimal policy under our objective. So, the update direction of Eq. (9) in our supplement is correct.\n\n---\n\n**Q4. What connection exists between Reg-GRPO and video tasks? [Question 3]**\n\n**A4.** Good question.  Reg-GRPO (Regressive Group Relative Policy Optimization) is a generic technique designed to enhance the reasoning capability of LLM-based models, particularly for complex tasks. It is not limited to video LLMs. To show the generalizability of Reg-GRPO, we evaluate it by comparing it with GRPO on multiple complex “mathematical reasoning” benchmarks in the table below.\n\n|  | AMC23 | Minerva | MATH500 | OlympiadBench |\n| --- | --- | --- | --- | --- |\n| GRPO | 47.5 | 32.35 | 80.2 | 38.96 |\n| Reg-GRPO | **60.0** | **34.48** | **80.6** | **41.93** |\n\nThe performance gain across these diverse reasoning datasets shows that Reg-GRPO is an effective post-training strategy for enhancing LLM reasoning. This generalizability extends naturally to challenging tasks such as video question answering, where complex temporal and causal reasoning is often required. Thus, Reg-GRPO strengthens the core reasoning ability of LLMs, which is essential for solving video tasks that demand high-level understanding and long-term dependencies.\n\n---\n\n**Q5. Additional performance comparisons on video benchmarks. [Question 4]**\n\n**A5.** To further validate the generalizability of our method, we compare **DeepVideo-R1** against **Video-R1** using *identical pretraining data* and **Qwen2.5-VL-3B** as the base model. Results on five video-language reasoning datasets are shown below:\n\n|  | Video-MMMU | MMVU (mc) | MVBench | TempCompass | Video-MME (w/o sub) |\n| --- | --- | --- | --- | --- | --- |\n| Video-R1 | 38.9 | 56.5 | 49.1 | **63.4** | 49.7 |\n| **DeepVideo-R1 (ours)** | **40.7** | **59.0** | **49.6** | 63.1 | **51.1** |\n\nDeepVideo-R1 outperforms Video-R1 on 4 out of 5 datasets, highlighting the robustness and generalizability of our approach across diverse video-language reasoning tasks.\n\n---\n\n**Q6. Are the SEED-Bench-R1 and NextGQA datasets combined or used separately? [Question 5]**\n\n**A6.** As noted in the supplement, SEED-Bench-R1 and NextGQA datasets are used separately for reinforcement fine-tuning. In response to the suggestion, we conducted an additional experiment using **joint fine-tuning** on the combined dataset (SEED-Bench-R1 + NextGQA). The results are summarized below:\n\n|  | SBR-L1 | SBR-L2 | SBR-L3 | NexTGQA (mIoU) | NexTGQA (ACC@QA) |\n| --- | --- | --- | --- | --- | --- |\n| SEED-Bench-R1 | 48.05 | 51.07 | 43.98 | - | - |\n| NextGQA | - | - | - | 36.8 | **72.5** |\n| SEED-Bench-R1 + NextGQA | **49.34** | **51.89** | **44.38** | **37.1** | **72.5** |\n\nThe results indicate that joint fine-tuning on both datasets slightly improves performance on SEED-Bench-R1 while maintaining comparable performance on NextGQA.\n\n---\n\n**Q7. Comparison of Reg-GRPO with GRPO regarding the gap between** $\\pi_\\theta$ and $\\pi_\\text{old}$. **[Question 6]**\n\n**A7.** Good point. As suggested, we analyze the divergence between $\\pi_\\theta (y|x )$ and $\\pi_{\\theta_{\\text{old}}} (y|x  )$. Specifically, we report the average absolute log-ratio $|\\log \\frac{\\pi_\\theta  (y|x  )}{\\pi_{\\theta_{\\text{old}}} (y|x  )}|$, which indicates instability in updates. The experimental results in the table below show that Reg-GRPO consistently yields smaller deviations than GRPO. It suggests that Reg-GRPO leads to more stable policy updates. We will include additional plots illustrating these stability metrics over training iterations in the final version to further support this analysis.\n\n|  | Avg. $\\| \\log \\frac{\\pi_\\theta} {\\pi_{\\theta_{\\text{old}}}}\\|$ | \n| --- | --- |\n| GRPO | 0.000861 |\n| Reg-GRPO (Ours) | 0.000490 |\n\n[1] Rafailov, Rafael, et al. \"Direct preference optimization: Your language model is secretly a reward model.\" NeurIPS 2023."}, {"review_id": "djbbBzzLgo", "reviewer": "Reviewer_s81Z", "summary": "DeepVideo-R1 enhances video LLMs by addressing two major limitations of GRPO, reliance on heuristic safeguards and the vanishing advantage problem. It introduces Reg-GRPO, it directly predicts advantage values, eliminating problematic safeguards like clipping and min functions. Extensive experiments demonstrate that DeepVideo-R1 significantly improves performance across multiple challenging video reasoning benchmarks, outperforming recent SOTA VideoLLMs such as Qwen2.5-VL and Intern3-VL.", "questions": "1. Predicting advantage values may introduce additional training instability if the estimation becomes noisy, \n\n2. Could the computational overhead from difficulty-aware augmentation can be quantified and compared?\n\n3. How robust is the Reg-GRPO to noisy or sparse reward signals in real-world video reasoning tasks?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n- The writing and figures are easy to follow. \n- This paper introduces a novel Reg-GRPO approach, effectively addressing vanishing advantage problem and reliance on heuristic safeguards.\n- Implements a difficulty-aware data augmentation strategy, effectively maintaining diverse training signals and enhancing the generalizability of video reasoning.\n- Demonstrates substantial improvements in performance on complex video reasoning tasks, outperforming SOTA\n\nWeakness : \n\n- Diversifying the VideoLLM baselines and analysis would strengthen the paper, one good candidate would be Video-R1 or VideoLLM with GRPO variants. https://arxiv.org/abs/2503.21776, https://arxiv.org/abs/2506.01908v1 etc\n- Potential computational overhead from dynamically adjusting data difficulty during training. I suspect significant computational resources and training time for this approach.\n- The generalizability of the proposed augmentation strategy to extremely diverse or real-world scenarios beyond the evaluated benchmarks is not fully explored.", "was_revised": true, "final_justification": "The author's response resolves my most of concerns. DeepVideo‑R1 with Reg‑GRPO and difficulty‑aware augmentation outperforms baselines across diverse benchmarks with no added inference cost, stable training, and robustness, demonstrating effectiveness and generalizability. I update my score to accept", "rebuttal": "We sincerely thank the reviewer for the positive and constructive comments. We are encouraged that **the reviewer recognized multiple strengths** in our paper, including: \n\n(i) **clear writing**\n\n**(ii) The novel Reg-GRPO approach**\n\n**(iii) Effective difficulty-aware data augmentation**\n\n**(iv) Strong empirical results, with DeepVideo-R1 outperforming prior state-of-the-art models on complex video reasoning benchmarks.**\n\nWe have tried our best to address the reviewers’ questions and concerns within the available time. We believe that incorporating this constructive feedback significantly enhances the quality of the paper. Below, we provide the response to the given comments:\n\n---\n\n**Q1. Performance on other video baselines. [Weakness 1]**\n\n**A1.** We have **already demonstrated the effectiveness of our DeepVideo-R1 compared to other GRPO-based VideoLLM such as TimeZero and VideoChat-R1** on both **Charades-STA and ActivityNet-Captions** in Table 10 of the supplement. The table shows that our DeepVideo-R1 achieves the best performance compared to these baselines across both benchmarks. These results highlight the effectiveness of our proposed Reg-GRPO and difficulty-aware data augmentation approaches. \n\nFurthermore, we measured the performance of our DeepVideo-R1 on diverse video understanding reasoning benchmarks such as VideoMMMU, MMVU (mc), MVBench, TempCompass, and Video-MME as reported in Table 9 of the supplement. As suggested, we compare our DeepVideo-R1 with Video-R1 on these benchmarks. We use training data of Video-R1 and use Qwen2.5-VL-3B as a base model. The experimental results in the table below indicate that our DeepVideo-R1 shows better performance on 4 out of 5 benchmark datasets compared to Video-R1, validating the generalizability and robustness of DeepVideo-R1 across diverse video-language reasoning tasks.\n\n|  | Video-MMMU | MMVU (mc) | MVBench | TempCompass | Video-MME (w/o sub) |\n| --- | --- | --- | --- | --- | --- |\n| Video-R1 | 38.9 | 56.5 | 49.1 | **63.4** | 49.7 |\n| **DeepVideo-R1 (ours)** | **40.7** | **59.0** | **49.6** | 63.1 | **51.1** |\n\n---\n\n**Q2. Computational costs of difficulty-aware data augmentation. [Weakness 2, Question 2]**\n\n**A2.** Good question. As suggested, we compare the computational cost of GRPO, DeepVideo-R1 without difficulty-aware data augmentation, and DeepVideo-R1 by measuring the average training and inference time per video across three different benchmarks (SBR-L1, SBR-L2, SBR-L3) in the table below. We use Qwen2.5-VL-Instruct 3B as a base multimodal large language model and NVIDIA A100 GPUs, following the experimental setting in our supplement.\n\n|  | SBR-L1 | SBR-L2 | SBR-L3 | Inference time (sec/video) | Training time (sec/video) |\n| --- | --- | --- | --- | --- | --- |\n| GRPO | 39.56 | 41.01 | 35.35 | **3.08** | **2.97** |\n| DeepVideo-R1 (w/o D.A. Aug) | 44.20 | 44.15 | 39.52 | **3.08** | **2.97** |\n| DeepVideo-R1 | **48.05** | **51.07** | **43.98** | **3.08** | 4.56 |\n\nThe table shows that the inference time remains constant across methods since both Reg-GRPO and D.A. Aug. are training approaches. Notably, DeepVideo-R1 significantly improves the performance of GRPO without increasing inference cost. Even though the difficulty-aware augmentation slightly increases training time, it has a superior performance gain of 6.92 on SBR-L2 compared to the model without difficulty-aware data augmentation, demonstrating the necessity of the augmentation.\n\n---\n\n**Q3. Generalizability of the difficulty-aware data augmentation. [Weakness 3]**\n\n**A3.** To evaluate the generalizability of difficulty-aware augmentation on different types of video reasoning tasks, we additionally conduct the experiments by comparing the performance of DeepVideo-R1 with and without difficulty-aware augmentation on five video reasoning benchmarks. The experimental results are described in the table below. The table demonstrates that our difficulty-aware data augmentation strategy is broadly applicable and effective beyond SEED-BENCH-R1 datasets.\n\n|  | Video-MMMU | MMVU (mc) | MVBench | TempCompass | Video-MME (w/o sub) |\n| --- | --- | --- | --- | --- | --- |\n| DeepVideo-R1 (w/o D.A. Aug.) | 39.1 | 57.5 | 48.6 | 62.3 | 50.3 |\n| DeepVideo-R1 (with D.A. Aug.) | **40.7** | **59.0** | **49.6** | **63.1** | **51.1** |\n\n---\n\n**Q4. Training stability of the Reg-GRPO. [Question 1]**\n\nA4. To assess the training stability of Reg-GRPO, we analyze the divergence between $\\pi_\\theta\\left(y|x\\right)$ and $\\pi_{\\theta_{\\text{old}}}\\left(y|x \\right)$. Specifically, we report the average absolute log-ratio $|\\log \\frac{\\pi_\\theta \\left(y|x \\right)}{\\pi_{\\theta_{\\text{old}}}\\left(y|x \\right)}|$ and the ratio of tokens whose likelihood ratios $\\frac{\\pi_\\theta(y_t | x, y_{<t})}{\\pi_{\\theta_{\\text{old}}}(y_t | x, y_{<t})}$deviate significantly from 1 (e.g., less than 0.5 or greater than 1.5), which indicates instability in updates. The experimental results in the table below show that Reg-GRPO consistently yields smaller deviations than GRPO across all metrics. These findings suggest that Reg-GRPO leads to more stable policy updates, which is crucial for ensuring reliable training. We will include additional plots illustrating these stability metrics over training iterations in the final version to further support this analysis.\n\n\n|  | Avg. $\\lvert \\log \\frac{\\pi_\\theta} {\\pi_{\\theta_{\\text{old}}}}\\rvert$ | \n| --- | --- | \n| GRPO | 0.000861 | \n| Reg-GRPO | 0.000490 | \n\n\n- Avg. token ratio of $\\\\lvert 1-\\frac{\\pi_\\theta(y_t | x, y_{<t})}{\\pi_{\\theta_{\\text{old}}}(y_t | x, y_{<t})}\\\\rvert>0.3$\n\n  - GRPO: 0.000972\n  - Reg-GRPO: 0.000435\n\n- Avg. token ratio of $\\\\lvert 1-\\frac{\\pi_\\theta(y_t | x, y_{<t})}{\\pi_{\\theta_{\\text{old}}}(y_t | x, y_{<t})}\\\\rvert>0.5$\n\n  - GRPO: 0.000231\n  - Reg-GRPO: 0.000073\n\n- Avg. token ratio of $\\\\lvert 1-\\frac{\\pi_\\theta(y_t | x, y_{<t})}{\\pi_{\\theta_{\\text{old}}}(y_t | x, y_{<t})}\\\\rvert>1.0$\n\n  -  GRPO: 0.000036\n  -  Reg-GRPO: 0.000009\n\n---\n\n**Q5. Robustness of the Reg-GRPO on sparse or noisy reward signals. [Question 3]**\n\n**A5.** Thank you for an insightful and important question. We have already shown the robustness of the Reg-GRPO under **sparse reward signals** where a reward signal is assigned only when the generated output is correct. This setup reflects common conditions in video QA tasks with binary correctness-based supervision. **Table 1 and 4 of the main paper** present results under this sparse reward setting. Notably, **DeepVideo-R1 equipped with Reg-GRPO significantly outperforms GRPO**, highlighting its resilience to sparse reward signals.\n\nTo further evaluate the robustness of our proposed Reg-GRPO under **noisy reward signals**, we conduct additional experiments with a newly introduced noise-injected setting. To the best of our knowledge, there has not been an experimental setting using noisy reward signals. So, we newly introduce a noisy-injected reward experimental setup, in which **the reward signal is flipped with 30% probability**: correct outputs are incorrectly assigned a reward of 0, and incorrect outputs receive a reward of 1. Under this setting, we compare GRPO and Reg-GRPO on SEED-Bench-R1 in the table below. To ensure a fair comparison that isolates the effect of the reinforcement learning algorithm, we **exclude difficulty-aware data augmentation** from all models in this experiment.\n\nThe experimental results demonstrate that **Reg-GRPO exhibits significantly lower performance degradation than GRPO under noisy rewards**, maintaining relatively stable performance across all datasets. In particular, the performance degradation of Reg-GRPO is only 1.75 on SBR-L3 dataset while GRPO suffers a substantial performance drop of 9.38. This indicates that the regression-based formulation of Reg-GRPO is inherently more robust to noise in reward signals compared to GRPO. We will include this analysis and new experimental setting in the final version of the paper if the paper gets accepted.\n\n|  | SBR-L1  | SBR-L2 | SBR-L3  |\n| --- | --- | --- | --- |\n| GRPO (noise) | 28.29 | 28.17 | 25.97 |\n| GRPO (org) | 39.56 | 41.01 | 35.35 |\n| GRPO (noise - org) | -11.27 | - 12.84 | -9.38 |\n| Reg-GRPO (noise) | 38.19  | 38.29 | 37.77 |\n| Reg-GRPO (org) | 44.20  | 44.15 | 39.52 |\n| Reg-GRPO (noise - org) | -6.01 | -5.86 | -1.75 |"}]

Author rebuttal:

Meta-review: **Summary:** This paper introduces Reg-GRPO, a regularized variant of GRPO that eliminates the need for heuristic clipping operations by reformulating the loss using a closed-form solution inspired by DPO. This mitigates the vanishing-gradient issue and provides a more principled optimization objective. Additionally, the paper proposes a difficulty-aware data augmentation strategy to ensure the model consistently receives effective training signals, thereby improving the generalization ability of Video LLMs on challenging video reasoning tasks. The authors validate their approach on multiple benchmarks, reporting consistent and substantial gains over existing methods.

**Strengths:**

- Reg-GRPO directly addresses the limitations of reliance on safeguards and vanishing advantage in GRPO.

- The difficulty-aware augmentation strategy provides a fresh perspective on dataset construction and signal diversity, and is particularly suitable for video reasoning tasks.

- The empirical results are strong, showing significant improvements across challenging benchmarks and surpassing several strong baselines.

- The paper is clearly written and well-structured, making both the methodology and motivation easy to follow.


**Weaknesses:**

- Computational complexity and scalability: Both Reg-GRPO training and difficulty-aware augmentation may introduce significant overhead, which is not thoroughly quantified or discussed.

- Generalizability of augmentation: While effective on tested benchmarks, the broader applicability to diverse or real-world datasets remains unclear.

Overall, the rebuttal was constructive and effective: it clarified theoretical concerns, acknowledged limitations, and provided convincing arguments about the practicality of the approach. After weighing the discussion, I recommend acceptance.
Decision: Accept (poster)