TempSamp-R1: Effective Temporal Sampling with Reinforcement Fine-Tuning for Video LLMs
Authors: Yunheng Li
JingCheng
Shaoyong Jia
Hangyi Kuang
Shaohui Jiao
Qibin Hou
Ming-Ming Cheng
Venue: NeurIPS 2025 poster ()
Keywords: Temporal Grounding; Multimodal Large Language Model; Reinforcement Fine-Tuning
Primary area: applications
TL;DR: TempSamp-R1 is a reinforcement fine-tuning framework that integrates off-policy supervision, soft advantage shaping, and hybrid Chain-of-Thought training to enhance the temporal grounding capabilities of MLLMs.

Abstract: This paper introduces TempSamp-R1, a new reinforcement fine-tuning framework designed to improve the effectiveness of adapting multimodal large language models (MLLMs) to video temporal grounding tasks. We reveal that existing reinforcement learning methods, such as Group Relative Policy Optimization (GRPO), rely on on-policy sampling for policy updates. However, in tasks with large temporal search spaces, this strategy becomes both inefficient and limited in performance, as it often fails to identify temporally accurate solutions. To address this limitation, TempSamp-R1 leverages ground-truth annotations as off-policy supervision to provide temporally precise guidance, effectively compensating for the sparsity and misalignment in on-policy solutions. To further stabilize training and reduce variance in reward-based updates, TempSamp-R1 provides a non-linear soft advantage computation method that dynamically reshapes the reward feedback via an asymmetric transformation. By employing a hybrid Chain-of-Thought (CoT) training paradigm, TempSamp-R1 optimizes a single unified model to support both CoT and non-CoT inference modes, enabling efficient handling of queries with varying reasoning complexity. Experimental results demonstrate that TempSamp-R1 outperforms GRPO-based baselines, establishing new state-of-the-art performance on benchmark datasets:Charades-STA (R1\@0.7: 52.9\%, +**2.7**\%), ActivityNet Captions (R1\@0.5: 56.0\%, +**5.3**\%), and QVHighlights (mAP: 30.0\%, +**3.0**\%). Moreover,  TempSamp-R1 shows robust few-shot generalization capabilities under limited data. Code is available at https://github.com/HVision-NKU/TempSamp-R1.

TempSamp-RI: Effective Temporal Sampling with Reinforcement Fine-Tuning for Video LLMs
Yunheng Li
Jing Cheng?
Shaoyong Jia
Hangyi Kuang'
Shaohui Jiao?
Qibin Hou
133
Ming-Ming Cheng 1.3
VCIP, School of Computer Science, Nankai University, "ByteDance Inc., "NKIARI, Futian, Shenzhen yunhengli @mail nankai edu.
Abstract
This paper introduces TempSamp-RI, work designed to improve the effectiveness of adapting multimodal large language models (MILMMs) to video temporal grounding tasks. We reveal that existing rein- forcement learning methods, such as Group Relative Policy Optimization (GRPO), rely on-policy sampling for policy updates. However, in tasks with large tempo- ral search spaces, this strategy becomes both inefficient and limited in performance, often fails identify temporally accurate solutions. To address this limitation, TempSamp-RI leverages ground-truth annotations as off-policy supervision to pro- vide temporally precise guidance, effectively compensating for the sparsity and mis- alignment in on-policy solutions. To further stabilize training and reduce variance reward-basee updates, TempSamp-F RI provides non- non-linear soft advantage com- putation method that dynamically reshapes the reward feedback via an asymmetric transformation. By employing hybrid Chain-of- Thought (CoT) training paradigm, TempSamp-RI optimizes single unified model to support both CoT and non-CoTT inference modes, enabling efficient handling of queries with varying reasoning complexity. Experimental results demonstrate that TempSamp-RI outperforms GRPO-based baselines, establishing new state-of-the-art performance on bench- mark datasets: Charades-STA (R1 @0.7: 7: 52.9%. +2.7%), ActivityNet Captions (R1 @0.5: 56.0%, +5.3%), and VVHighlights (mAP: 30.0%, +3.0%). Moreover, TempSamp-RR R1 shows robust few-shot generalization capabilities under limited data. Code is available at https: ttttps://gian //tpt/ggtttttttt com/ wwom/Viiiiinn wwwiiisioinuuuuS- TKUUTTmmSaap R1.
reinforcement fine-tuning frame-
Introduction
Multimodal Large Language Models (MLLMs) [6. 21, 30, 31, 34, 40, 51, 58, 64] have demonstrated impressive capabilities in comprehending video content by following general human instructions and effectively interpreting visual content. However, their application
temporal video understanding
tasks, such as temporal grounding [1, 22, 25, 35] and highlight detection [24, 43, 62], remains challenging, as these tasks require precise spatio-temporal understanding over long video sequences. A common approach is Supervised Fine-Tunine (SFT) [13, 17, 18, 28, 41, 45, 49, 54, 66), which aligns model predictions with static ground-truth timestamps using deterministic supervision. However, these methods often exhibit limited effectiveness, models tend to overfit to deterministic timestamp supervision and fail to acquire the temporal reasoning required for flexible event localization [33, 60].
Recent approaches [10, 33, 60, 72] attempt to address these challenges using reinforcement learning (RL) frameworks. In particular, Group Relative Policy Optimization (GRPO) [47] improves temporal
CCoresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Hall
On-Policy Solutions
Seft Advantages
Kroferp-i
Video:
8.0 to 17.0
-1.1
Update
a0
Bocall 184
21.9
47.5
47.5
Policy Model
person turns on the light.
13.9 22.4.
to 22,4
0.7
26-9
Query
Ground Truth
off-Policy Solution
20
11880.5
10g22
Rigos 11.0 Charddes STA
ActiviyyNet Caprions
Figure I: TempSamp-RR RI integrates high-quality off-policy solutions with on-policy sampling, com- bined with soft advantage estimation to enable stable policy updates. It outperforms GRPO, which relies solely on on-policy sampling, on both Charades- and ActivityNet Captions.
performance by updating policies via grouped comparisons of sampled solutions, mitigating over- fiuting to static annotations. GRPO-based methods, such as TimeZero [60] and VideoChat-Ri [33], optimize models via task-specific rewards (e.g., temporal Intersection-over-nniooo (IoU)) more effectively align visual dynamics with timestamped semantics in long videos. Despite these advance- ments, these methods still face critical limitation: The vast temporal search space in temporal grounding tasks severely hinders effective exploration. This issue is empirically shown in Fig. 4, when employing GRPO. purely on-policy optimization leads to low and unstable top-1 IoU rewards, particularly on ActivityNet Captions, reflecting unstable early updates, ineffective learning under sparse supervision, and premature convergence to suboptimal solutions.
Notably, most video understanding datasets provide high-quality annotations (e.g., event timestamps) that offer precise supervision for grounding tasks. However, existing GRPO methods treat these annotations solely as evaluation (e.g., computing loU) rather than dynamic learning sources, leading to suboptimal policy updates. Motivated by this observation, reinforcement finetuning framework that integrates on-policy generation with off-policy guidance facilitate more stable and efficient policy optimization. As illustrated in Fig. TempSamp- RI incorporates high-quality and instruction-aligned solutions from external sources (e.g., ground truth annotations) as off-policy guidance, providing temporally precise supervision to compensate for the sparsity and misalignment often encountered in on on-policy samplings.
propose TempSamp-R1, new
However, since off- ooflcy solutions not sampled from the on-policy model, directly using their rewards can introduce substantial discrepancies in reward distribution, resulting in biased advantage estimation. This estimation bias may suppress high-quality on-policy samplings that diverge from the off-policy solutions, thereby limiting the policy's 's ability to generalize and explore alternative solutions effectively. To mitigate this, TempSamp-RI introduces non-linear soft advantage estimation mechanism inspired by the principles of adaptive reward shaping [15, 39]. To be specific, instead of treating all rewards uniformly, our method distinguishes the learning dynamics between high- reward and low-reward solutions by compressing the advantage values of near-optimal solutions and amplifying the relative reward gaps among suboptimal ones. This asymmetric shaping generates more informative gradients and facilitates stable policy refinement.
By incorporating hybrid Chain-of-Thoue Thought (CoT) [61] training paradigm into unified model, TempSamp-RI achieves robust performance under both CoT and non-CoTT reasoning modes, which also shown to be complementary. We evaluate TempSamp-R1 through comprehensive experi- ments on temporal video understanding benchmarks, including temporal video grounding and video highlight detection. Extensive experiments show that TempSamp-RI consistently outperforms various SFT-based and GRPO-based methods. Specifically, TempSamp-RI R1 improves temporal grounding recall accuracy on Charades-STA [11] (RI @0.7: 47.9% + 52.9%) and ActivityNet Captions [22] (R1 @05:: 50.7% + 56.0%), while enhancing highlight detection in QVHighlights [24] (mAP: 27.0% - 30.0%). Notably, TempSamp-RR R1 maintains competitive performance under limited supervision, highlighting its strong generalization capacity in fho-shot scenarios.
2 Related Work
Reinforcement finetuning. Reinforcement learning has emerged as a powerful paradigm for en- hancing the reasoning capabilities of large language models and MILMSs [7-9, 12, 19, 38, 52]. For instance, OpenAl's ol model [19], DeepSeek-R1 [12] and Qwen3 [53] apply RI. to generate intermediate reasoning traces before producing final responses, thereby improving performance

---

complex reasoning tasks. Existing RL fine-tuning approaches can be broadly categorized into reward model-based methods and direct preference optimization techniques. Reward model-based methods, such as RLHF [4]. PPO [46], and RLAIF [23], rely on separately trained reward model to guide policy updates. In contrast, direct preference optimization methods, including DPO [44], IPO [2] and ORPO [16], bypass explicit reward modeling by optimizing preferences directly. GRPO [47] builds upon this paradigm by introducing on-policy sampling and groupwise preference evaluation, enabling dynamic policy refinement from richer comparative samplings. By leveraging comparisons between multiple candidate solutions, GRPO-based methods capture richer alignment supervision than stan- dard SFT, which only learns from a single reference solution [27, 48, 50, 65, 70, 71j. Nonetheless, GRPO's performance still hinges on the diversity and informativeness of samplings, as uninformative comparisons may lead to optimization bias or degraded policy exploration.
Temporal video grounding. Temporal video understanding tasks, such as temporal grounding and highlight detection, require models to accurately identify and describe events within untrimmed video sequences [5, 20, 36, 37, 42, 67, 69, 73). SFT has been the predominant approach for adapting MLLMs to these tasks [3, 32, 55-57, 74]. Models like TimeChat [45] employ video-text pre- training strategies to align fraee-level features with textual descriptions. However, these methods struggle to achieve precise temporal localization due to the limitations in modeling long-range temporal dependencies and the tendency to rely on learned language patterns over visual cues. To address these challenges, RL has been introduced as reasoning capabilities of MLLMs. Recent methods, including TimeZero [60], R1-Omni [72], and VideoChat-RR [33], utilize GRPO to fine-tune models methods emphasize the design of reward functions to guide the model toward more accurate temporal localization. Our method also builds the GRPO framework but differs from prior work in that it replaces random on-policy sampling with off-policy solutions to alleviate reward sparsity. To facilitate more efficient and stable policy optimization, we introduce a non-linear soft advantage estimation that dynamically reshapes advantage values to smooth gradient updates.
strategy enhance the temporal
tentto-le perception tasks. These
3 Methodology
To enable effective exploration beyond the limitations of on-policy learning, we propose TempSamp- R1, which combines on-policy generation with off-policy guidance and enhances training stability through soft advantage estimation. As shown in Fig. 2, our method introduces a mixed-policy training strategy based on GRPO, integrating high-quality external solutions (e.g., ground-truie -ttnhttruthhh annotations) into the policy optimization process. To stabilize training, we develop a soft advantage estimation mechanism that decouples and shapes reward to reduce gradient variance and adjust advantage bias, thereby promoting robust exploration and convergence.
3.1 Preliminaries
GRPO is samplc-efficient policy optimization algorithm designed to optimize policy models (e.g., MI..Ms) by comparing groups of solutions, thereby eliminating the need for an independent value model and reducing computational overhead. Given query q. GRPO samples group of G outputs (01, o2,..., ..) from the current policy model Ty and computes the corresponding rewards (ri, ..., r } using a task-specific reward function that evaluates output quality with respect to ground-truth annotations and predefined specific rules (e.g., IoU), The advantage for each solution o is then computed as A;- = z=- where ju and a denote the mean and standard deviation of the group rewards, respectively. This group- normalzedd advantage serves the core optimization direction in GRPO, as it adaptively amplifies preferences for outputs that exhibit relatively higher quality within the sampled group. To update the policy, GRPO reuses the same solutions sampled from the previous policy Te, and re-evaluates their likelihoods under the current policy No. The update applies importance weighting with clipping mechanism and incorporates Kl-regularization term to constrain deviation from a reference policy Teef:
[aRPo(()=
1 Σ min
TT(0sl4) (4) (Teu (o:lg) A,, clip
To (o(o:lg) Ta (o (0.
--,, (e) A. 10-300 KL [
(1)
Here, e and| /3 denote the clipping range and the Kl.-divergence penalty weight, respectively. These components collectively impose constraints policy updates and mitigate instability during the
3

---

13
Few samples
Update
01
A
GRPO
0.0
Videos
Policy Model
o2
Compute
Soft Advantage Estimation
Aa
-0.4
Reward
oa
le
0.5
Aa_
0.6
0.7
Ours
Query
Off-Policy Guidance
Og
loU
Aa
0.2
Solution
Reward
M.an=0.7
Advantage
Overview of the TempSamp-R RI framework used to fine-tune the multimodal policy model. Given few training examples, both the policy model and the off off-policy guidance used to generate solutions. Rewards are computed for each solution, and a soft advantage estimation module transforms raw rewards into standardized advantages for stable policy optimization. Right: Comparison of normalized advantages from GRPO (top) and our method (bottom), illustrating improved advantage discrimination. For clarity, the reference model and KL penalty are omitted.
Figure 2:
optimization process. Recent methods [33, 60] often adopt Te.. Tа to balance training data utilization efficiency and computational cost. Under this configuration, the importance weights collapse to unity, eliminating the need for ratio clipping while ensuring stable learning by updating the policy only once per sampling batch.
3.2 TempSamp-RI
Despite their impressive performance on general vision-language tasks, MLLMs often exhibit limited temporal grounding capabilities in video understanding [3, 56]. As a result, training under GRPO with on-policy sampling leads to slow convergence and constrained performance upper bound, as the policy model encounters significant challenges in generating temporally precise solutions.
Mix-policy sampling. To address the above limitation, we introduce mixed-policy training strategy that incorporates external off-policy solutions to provide accurate and query-specific temporal grounding. Although such high-quality solutions can be derived from expert policies, we adopt more direct and empirically effective alternative by utilizing ground-truth annotations as the external policy guidance. To unify supervision across policy sources, we normalize the advantage values using the joint distribution of on-policy and off-policy rewards. Specifically, solutions from the current policy and include one external off- off-policy solution (e.g.. ground-truth The normalized advantage for each solution with reward r is then computed as:
each query, we sample G-1
A r; mean
Ura)
std(ri, Fo.. Ura)
where denotes the reward from the external off-policy solution. While the integration of off-policy solutions enhances the diversity and quality of training supervision, it also introduces skewed reward distributions that can destabilize policy optimization. In particular, when the off off-policy solution consistently attains the highest reward, even marginal deviations among the remaining rewards (such as low inter-sample variance or tightly clustered suboptimal rewards) can render the normalization process numerically unstable.
illustrated Fig. 2. the inclusion of off-policy solutions with exceptionally high rewards elevates the intra-group mean reward. In this scenario, the original GRPO algorithm computes negative advantages for all on-policy generated solutions, including those of high quality. This misestimation adversely affects advantage calculation, disrupts gradient updates, and diminishes exploration, ultimately causing premature convergence to suboptimal solutions. To mitigate these adverse effects and ensure stable and reliable advantage estimation, propose three alternative strategies that regulate the contribution of off policy supervision.
Reward downscaling, A straightforward mitigation strategy involves explicitly bounding the off- policy reward by scaling
fixed fraction (e.g., 80%) the maximum possible value. This heuristic prevents off-policy solutions from dominating the advantage computation, mitigating distributional shift, and preserving gradient stability. However, the fixed nature of this scaling may suppress valuable learning supervision when off off-policy solutions offer genuinely optimal samplings.

---

Advantage anchoring. To leverage off off-policy supervision while mitigating distributional bias, we introduce an anchoring mechanism that decouples external solutions from on-policy advantage estimation. Specifically, off-policy samples are excluded from the computation of group statistics and do not participate in normalization. Instead, their advantage values are computed by scaling the maximum on-policy advantage within the group:
Ag = Aoff max {A, (1,2... ... GG
(3)
where Aotr = 1.2 is fixed scaling factor. This anchoring preserves the supervision from off off-policy data while constraining its influence, maintaining stability and consistency in policy gradient updates.
Non-linear reward shaping. To improve stability under skewed reward distributions, we apply non-linear transformation to the rewards prior to advantage computation. This transformation is defined as asymmetric piecewise function that compresses high rewards and expands low rewards. The shaping reward F; is defined as:
T an In ((r; - T) 1), e -
Ti 1i
TAT
602
Here, 0.8 is the reward threshold, a1 = 0.01 controls compression above the threshold, and 02 = 1 governs expansion below it. The logarithmic branch mitigates gradient spikes from optimal solutions, while the exponential branch increases contrast among suboptimal samplings.
Each of the above strategies provides an alternative mechanism to mitigate instability caused by incorporating strong off off-policy solutions. By independently adjusting rewards, advantage scales, or reward distributions, these methods offer flexible design choices to control the influence of external off- off-policy and improve the stability of policy optimization multimodal video understanding.
3.3 Training
We adopt a two-phase training scheme with task-specific reward functions to support diverse video understanding tasks. In the initialization phase, the model is optimized generate accurate final answers without explicit reasoning. Building on this, we incorporate format rewards to encourage the generation of intermediate reasoning steps alongside final outputs. To operationalize this strategy across different tasks, we define a suite of task-specific reward functions that directly guide policy optimization toward task-relevant behaviors.
IOU reward. For temporal localization tasks, the model is required to predict an event inter- given query. To quantify prediction accuracy, define reward val [tp: fll conditioned
rule based the IoU with the ground truth interval [ty. fg], computed as RioU (min(t), (y. fg) max(te (2)) /m)/mm(xxtt. (-)))m((((f, (g)) The REu is computed as the ratio between the intersection and the union of the two intervals, serving as a direct measure of temporal alignment accuracy.
Timestamp matching reward. For highlight detection tasks, the model jointly predicts temporal boundaries and associated saliency scores. To evaluate both the structural and semantic quality of these predictions, we define a composite reward: R₁ Arec F2 + Assore T+WMSE Here, the F2 score measures the temporal alignment by computing a recght-weighted F-measure over matched timestamps, emphasizing recall to better capture relevant highlights. The Weighted Mean Squared Error (WMSE) assesses the fidelity of predicted saliency scores, with weights derived from the squared ground-truth scores to emphasize high-saliency regions. We set Arec = 0.6 and Assore = 0.4 to prioritize semantic fidelity in salient regions while ensuring temporal coverage.
Format reward. To promote structured output in reasoning tasks, we introduce format reward that enforces conformity to predefined schema. The model is expected to generate reasoning enclosed in <Think> </Think> and final answers in <Answer>. ... </ Answer>, The reward is set to the output matches the required structure based on regular expression validation, and 0 otherwise.
4 Experiments
Implementation details. Our experiments are conducted using the Owen2.55-VL--7B-nnstre model [3]. To ensure fair comparison with prior efficient video fine-tuning methods [33, 60],
5

---

ActivityNet Captions Type mloU RI@0.3 RI@0.5 RI @0.7 mloU R1@0.3 @1@0.5 RI @17 MAP HIT@!
Charades-STA
QVHighlights
Method
Supervised Fine- Fine-Tuning (SFT) Methods
UnLoc-L [63]
SFT
60.8
38.4
48.3
30.2
Timechat [45]
SFT
46.7
23.7
21.7
37,9
HawkEye [59]
SFT
49.3
72.5
58.3
28.8
39.1
55.9
34.7
17.9
TRACE [14]
61.7
41.4
37.7
24.0
VideoChat-T [68]
SFT
79.4
67.1
43.0
27.0
55.3
iMOVE (29)
SFT
57.9
79.8
68.5
45.3
49.3
67.2
50.7
32,4
Reinforcement Learning (RL) Methods based on Qwen2.5-VI.
Qwen2.5-VL--7B [3]*
49.7
73.4
54.4
30.3
33.1
45.2
29.7
18.1
19.7
34.1
VideoChat-RI R1 [33]
RL
60.8
71.7
50.2
VideoChat-RR ---hhaak- [33]
59.9
70.6
47.2
TimeZero [60]
RL
83.3
72.5
47.9
68.6
47.3
26.9
(annaamp-R((o-CCoo
RL
61.7
83.3
73.6
52.2
52.1
72.8
55.4
34,2
30,0
57.6
TempSamp-RI (CoT)
RL
62.1
83.6
74.1
52.9
52.4
73.4
56.0
34,7
28.3
54,9
TempSamp-Re R1 Mixed CoT
64.2
85.0
76.0
56.3
75.7
58.7
Table 1:
Performance comparison of Charades-STA, ActivityNet Captions, and QVHighlights datasets. Our TempSamp-RR supports both CoT and no-CoT reasoning within single unified model, and achieves strong performance across all datasets. The TempSamp-R Mixed CoT selects the better prediction between CoT and no-CoT for each query.
Method
mloU
RI@003
RI@0.5
RI@0.7
SFT
20.6
30.2
16.7
7.9
GRPO
30.7
45.0
27.5
12.9
TempSamp-RI
50.9
32.2
16.2
Table 2: Out-of-domain generalization performance from Charades-STI to ActivityNet.
we standardize the input preprocessing pipeline. Specifically, all videos are temporally downsampled to frames per second (FPS) and resized to approximately 2.8 million pixels per frame. Training is performed on four NVIDIA A100 GPUs with a batch size of per GPU. For GRPO-based training. each question is associated with total of 4 solutions, consisting of G=3 on-policy samplings and off-policy solution. This setting balances computational efficiency and diversity policy learning. For Charades-STA (Tab. 1), we increase the number of solutions to 8 to better accommodate the task's 's higher compositional complexity.
Benchmarks. We evaluate our model on the temporal grounding task using the Charades-STA: ActivityNet Captions, and QVHighlights datasets. Following established practices [45, 68), we report Recall @ (RI @) at Intersection over Union (IoU) thresholds of 0.3, 0.5, and 0.7. Additionally, we compute the mean IoU (mloU) across all test samples evaluate overall localization accuracy. The QVHighlights dataset evaluates using mean Average Precision (mAP) at IoU thresholds of 0.5 and 0.75, and HIT @ which indicates whether the top-ranked clip is labeled as "Very Good."
4.1 Main results
Fine-tuning performance. We evaluate the effectiveness of our proposed method, TempSamp-R 1, across three standard benchmarks for temporal grounding: Charades-STA. ActivityNet Captions, and QVHighlights. Tab. compares our approach with wide range of baselines, including zero-shot, supervised fine-tuning (SFT) methods, and reinforcement learning (RL) based approaches. Compared to state-of-the SFT baselines, TempSamp-ER R1 achieves stronger performance. On Charades- STA, obtains 74.1% % R1 @0.5 and 52.9% Ri @..7. outperforming iMOVE by +5.6% and +7.6% respectively. In comparison to existing RL-based approaches, TempSamp- R1 achieves consistent gains across all benchmarks. For instance, it surpasses VideoChat-PP by +2.4% R1 @0.5 and TimeZero by +5.0% RI 0.7. on Charades-STA. On ActivityNet Captions, it exceeds TimeZero by +8.7% RI @0.5 and
For Qwen2.5- --V-L.B- the results Charades STA and ActivityNet Captions are reproduced from [26]. whereas the results QVHighlights obtained from our implementation.

---

50 videos
100 videos
200 videos
500 videos
Method
RI @0.5
mloU
R @00.5
mloU
RI @0.5
mloU
RI @0.5
mioU
Training Time
SFT
44.8
41.9
46.5
42.6
45.2
42.7
51.4
46.2
93 min
GRPO
36.2
38.4
39.3
40.8
43.5
43,8
55,3
49.8
338 min
TempSamp-RI (Ours)
46.7
44.7
54.0
49.1
58.2
51.8
64.0
55.1
218 min
Table 3: Few-shot performance comparison of SFT, GRPO, and our proposed method on Charades-
STA under varying training sample sizes (50, 100, 200, 500). All models were trained for 3 epochs.
Method
RI @0.33
RI @0.5
RI @.77
0.5
Mixed-polliy
0.4
Reward downscaling
GRPO (baseline)
81.2
68.9
46.0
0.3
Mixed-policy
77.8
63.0
41.3
Non- Non-linear rwann shopt
Reward downscaling
81.2
70.3
48.1
Skownos
Advantage anchoring
81.8
70.7
49.1
Non-linear reward shaping
82.9
72.1
49.6
-0.1
1500
Table 4: Ablation results comparing GRPO with en- hanced variants incorporating mixed-policy rewards and alternative advantage shaping strategies.
Figure 3: Skewness of the advantage distribu- tions during training for different variants.
+7.8% RI @0.7. We observe that CoT prompting at inference consistently improves performance on Charades-STA and ActivityNet Captions, indicating that explicit reasoning is beneficial for tasks involving complex temporal dependencies. In contrast, on QVHighlights, the TempSamp-RI with no-CoT performs better, indicating that direct prediction is more suitable for highlight detection, where explicit reasoning may be redundant or distracting. We further explore mixed variant, TempSamp-ER R1 Mixed CoT, which selects the better output between CoT and no-CoT predictions for each query. This strategy consistently outperforms either individual reasoning mode, underscoring their complementary roles. Representative examples in Fig. 6 illustrate how each reasoning mode excels under different semantic and temporal conditions.
-ut-oi- generalization. To assess the cross-dataset transferability of different approaches, we conduct out-of-domain evaluations where all models are trained Charades-STA and directly tested on ActivityNet Captions. As shown in Tab. 2, TempSamp-RI consistently outperforms both SFT and GRPO across all metrics on both datasets. On ActivityNet Captions, achieves improvements of +4.0% mloU and +4.7% RI @0.5 over GRPO. These results suggest that off- off-policy supervision and soft advantage shaping jointly enhance the cross-domain transferability of TempSamp-R1.
Few-shot performance. We evaluate our method under few-shot settings on the Charades-STA dataset, training with 50, 100, 200, and 500 videos, each for 3 epochs. Table 3 presents the performance comparison among SFT, GRPO, and our proposed method. Our method consistently outperforms both SFT and GRPO across all training sizes. Notably, with just 50 training samples, our method achieves a mloU of 44.7%, surpassing SFT by +2.8%. As the number of training samples increases, the performance gap widens. With 500 samples, our method attains RI @.5 of 64.0%, outperforming SFT by +12.6%, and GRPO by +8.7%, respectively. In terms of training efficiency, our method requires 218 minutes for training with 500 samples, which less than GRPO's 's 338 minutes. These results demonstrate that our method maintains efficient training, highlighting its practicality for real-world applications, where annotated data limited.
4.2 Component-wis analysis of TempSamp-R1
We analyze our method Charades-STI and ActivityNet Captions to assess the impact of key components in TempSamp-R1. including off- off-policy guidance and advantage shaping. Results show that these mechanisms jointly contribute more stable training and better policy optimization.
A nnalysis of advantage shaping strategies. We analyze the advantage shaping strategies introduced in Sec. 3.2 through systematic ablations within the TempSamp-RI framework here. Tab. 4 compares the GRPO baseline against four variants: mixed-policy supervision, reward downscaling, advantage anchoring, and non-lincar reward shaping. Directly injecting ground truth rewards (mixed-policy
7

---

GRPO (Baseline)
TempSamp-R R1 (Ours)
1.0
Top-1 loU Reward
0.0
633
Topa Reward
0.8
0.6
0.6
0.2
0.0
0.0
0-620
620-1240
1240-1861
1861-2481 2481-3102
0-1871
1871:3742
3742-5613 5613-7484 7484-9356
Training Step Interval
Training Step Interval
(a) Training on Charades-STA
(b) Training on ActivityNet Caption
Figure 4: Distribution of top-1 IoU rewards under GRPO and TempSamp-RI on Charades-STA and ActivityNet Captions. TempSamp-RI exhibits higher median rewards and reduced variance, indicating more stable and effective policy learning.
InU/=033
loU=0.5
InU=0.7
Sumplings
GRPO
TempSamp-RI (+A)
GRPO
TempSamp-RR R1 (+A)
GRPO
TempSamp-Ri (-A)
2
77.8
80.8 (+3.0)
60.2
67.3 (47.1)
34.4
44.6 (+10.2)
4
81.0
82.5 (+1.5)
68.9
71.6 (+2.7)
46.0
49.6 (+3.6)
6
81.3
82.6 (+1.3)
69.2
71.7 (42.5)
49.2
50.3 (+1.1)
8
81.2
82.6 (+1.4)
70.5
71.9 (+1.4)
47.1
50.8 (43.7)
Table 5: Ablation study on the number of solutions. Our method consistently outperforms the baseline GRPO across all configurations, particularly under higher IoU thresholds.
leads to degraded performance (e.g., 63.0 R1 @0.5), which can be attributed to distributional shift and reduced on-policy sampling diversity. Reward downscaling and advantage anchoring partially mitigate this issue, improving R1 @0.5 to 70.3 and 70.7, respectively. Non- reward shaping achieves the best results. To better understand these results, Fig. analyzes the skewness of the advantage distributions throughout training. GRPO exhibits persistent negative skewness, indicating dominance of low-reward solutions and weak gradient magnitude. contrast, sixeedpolicy supervision results in high positive skewness, reflecting over-reliance small set of high-reward solutions and poor policy generalization. Our proposed three shaping strategies mitigate these imbalances to varying degrees. Notably, non-linear reward shaping maintains near-zero skewness throughout training, promoting stable optimization and improved grounding performance.
Analysis of reward distribution. To better understand the learning dynamics of different methods, we analyze the distribution of top-1 IoU rewards collected during training. Fig. presents box plots comparing GRPO and methodname on Charades-STA and ActivityNet Captions. GRPO baseline exhibits low median rewards with high variance, especially on ActivityNet Captions, reflecting unsta- ble exploration and frequent convergence to suboptimal solutions. In contrast, methodname yields significantly higher median rewards and notably reduced dispersion. This indicates that the integration of off- off-policy supervision and non-linear advantage shaping not only improves reward magnitude but also enhances training stability. The compact distribution further suggests that methodname can consistently identify high-quality solutions across different video samples, in the presence of long temporal sequences and ambiguous queries.
4.3 Ablation study
We conduct ablations to evaluate the individual contributions of sampling strategy and CoT supervi- sion. Results Charades-ST, show that TempSamp-RI remains effective under limited on-policy samplings, and CoT supervision enhances temporal localization.
Impact of the number of solutions. We conduct an ablation study to evaluate how varying the number of solutions per query affects temporal grounding performance. Specifically, we assessed models with 2, 4, 6, and 8 solutions on the Charades-STA dataset. As shown in Tab. 5, TempSSmp-R outperforms the baseline GRPO across all configurations. Notably, with only solutions, TempSamp-RR I achieves remarkable 10.2% absolute improvement in RI @0.7 over GRPO, demonstrating its robustness

---

Training Prompt
Test Prompt
Method
Epochs
Think
Answer
Think
Answer
mloU
RI@0.3
RI@0.5
RI@0.7
Qwen2.5-VL-7B
29.0
44.7
24.2
11.1
(baseline)
28.1
41.8
23.4
11.1
TempSamp-Ri
1
61.1
82.6
71.9
50.8
1
/
V
54.2
75.9
63.6
40.5
61.5
82.6
51.2
2
61.2
82.4
72.8
50.6
2
V
61.8
83.3
73.6
52.2
2
62.1
83.6
74.1
52.9
Table 6: Ablation study the effects of <Think> and <Answer> prompts during training and testing under various configurations of our method on the Charades-STT dataset.
RI
@160..5
@100.7
R160
R1 690.5
R1@0.7
02
@100.3
RI60.5
@10.0.7
0.7
81.6
70.5
48.4
0.05
81.6
70.5
48.4
0.5
82.2
71.6
49.0
82,9
72.1
49.6
82,9
72.1
1.0
82,9
72.1
49.6
69.5
(a) Ablation т.
(b) Ablation on.
(c) Ablation on og-
Figure 5: Ablation study on the effects of different hyperparameters т, 01, and 02 in the non-linear reward shaping method.
under limited on-policy sampling. While increasing the number of solutions generally leads to improved performance for both methods, the gains for GRPO diminish beyond 4 solutions, whereas TempSamp-RI continues to benefit from additional on-policy samplings.
Impact of CoT supervision and inference. We perform ablation study assess the impact of incorporating <Think> and <Answer> prompts during training and testing on the Charades-STA dataset. Tab. presents the results across different combinations of training-phase supervision and inference-time prompting. The best performance is achieved when utilizing both <Think> and <Answer> prompts during training and inference. This suggests that the combination of CoT prompts enhances the model's temporal grounding capabilities. In contrast, applying <Think> prompts only during inference, without corresponding supervision during training, results in substantial performance drop, suggesting sensitivity to mismatched prompting conditions. In addition, training solely with <Answer> prompts for two epochs results in reasonable performance but does not match the effectiveness of the hybrid CoT strategy, underscoring the benefits of incorporating CoT prompts during training. Notably, Tab. (last rows) indicate that training with one non-CoT epoch followed by one CoT epoch yields stable performance regardless of test-time prompt design, indicating the model's s robustness to CoT prompt variations after hybrid CoT training.
Impact of non-linear reward shaping parameters. We analyze how each parameter in the non-linear reward shaping method influences learning dynamics, as shown Tab. 5. The threshold 7 defines the boundary between high- and low-reward regions. Moderate values maintain balanced optimization, whereas larger 7 exacerbate reward imbalance by concentrating rewards on few high-value solutions, which reduces the model's ability to learn from informatíve suboptimal solutions. The coefficient controls logarithmic scaling above the threshold. Smaller values retain gradient diversity, while larger values overly smooth the reward distribution. In contrast, 02 governs exponential scaling below the threshold. It enhances contrast among lower rewards but can cause instability when set too high. Overall, these findings indicate that the proposed non-linear shaping method is stable within broad parameter range, with 7 0.8, a = 0.01, and ag = 1.0 yielding the trade-off between reward contrast and optimization stability.
4.4 Qualitative results
To further illustrate the effectiveness our method, we present qualitative examples from the Charades-STA and ActivityNet Captions datasets, as shown in Fig. 6. Compared to the GRPO baseline, our model achieves more accurate temporal localization and generates more coherent event reasoning, particularly in challenging ambiguous cases. A key strength of TempSamp-Ri is its

---

Video:
Video:
Query: person turns on the light.
Query:: person begins sneeting
Ground Truth:
13.9
22.4
Ground Truth:
19.1
30.6
GRPO:
6.9
14.6
GRPO:
17.0 |--.e......
12.00
Tamp-S (non-CoT): 10.9
17.6
TempSamp-RI (nea-CoTy:
16.0)
29.0
TempSamp-ar
13.9
20.6
TempSamp-RR.
15.6
25.9
person entering room closing behind him. Then walks around the room and turns the lights, This can be inferred from the darkness before enters the lights. </Think>>
<Thinkk> The reading book while sitting chair. He cavers: his nose with his right hand and sneezes. After that he table and drinks
brightness
</Think>
(a) Qualitative examples where TempSamp-K RI successfully queries. CoT inference shows stronger contextual reasoning for complex queries, while non-CoT mode yiclds sharper boundaries for straightforward actions.
Video:
Video:
Query.
person starts eating.
Query: He gets up and moves the laule..
Ground Truth:
4,9
11.5
Ground Truth:
103.1
<-> 104.8
GRPO:
5.0
13.0
GRPO:
68.0
75500
TempSump-RI (non-CoT):
10.0
TempSamp-Re R1 (non-CoT):
72.0
TempSamp-RI:
6.0
19.0
TempSamp-R1:
69.0 <9.0|<------> 1
<Think> The person is sitting on a couch and cating something from bag. This can be considered as the
<Think> The man is standing on a ladder, trimming a hedge. Hc gets up from his position, moves the ladder, and continues trimming.</Thinke
person starting to cat. </Think>
(b) Failure cases mainly arise from ambiguous annotations and subtle cues that challenge accurate lncalization.
Figure 6: Qualitative comparisons on temporal grounding using TempSamp-RI. TempSamp denotes evaluation without CoT, while TempSamp-Ri R1 uses the <Think> prompt.
ability to optionally include the <Think> prompt at inference time. In cases requiring reasoning, such detecting transitions based on subtle visual cues, the CoT prompt improves boundary precision. For instance, when identifying the moment person turns on light, the model correctly associates the event with change in brightness. In contrast, for queries with clear visual markers, the model performs well without reasoning prompts, demonstrating adaptability to different reasoning demands.
However, shown in the failure cases, errors often stem from ambiguous narrowly defined ground-trutt truth annotations, which introduce uncertainty in the evaluation. Additionally, some complex queries involve subtle or overlapping visual cues that challenge precise localization, suggesting potential areas for further improvement in both annotation quality and model reasoning.
Conclusions
This paper introduces TempSamp-R1, a reinforcement learning framework designed to enhance temporal video understanding in multimodal large language models. TempSamp-Ri integrates off- policy supervision with non-lincar soft advantage mechanism to address the challenges of sparse rewards and unstable policy updates inherent in large temporal search spaces. By promoting effective policy updates, TempSamp-RI enables more accurate temporal localization of target events described in the input queries. Extensive evaluations on benchmarks such as Charades-STA, ActivityNet Captions, and HVHighlights demonstrate that TempSamp-Re consistently outperforms both SFT- based and existing GRPO-based reinforcement learning methods.
Acknowledgment. This work was supported by NSFC (No. 62495061 and No. 62225604), Shenzhen Science and Technology Program under Grant No. JCYJ202408 131 13114237044 and the Fundamental Research Funds for the Central Universities (Nankai University) under Grant 070- 63253220.
10

---

References
[1] Lisa Anne Hendricks, Oliver Wang. Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan Russell. Localizing moments video with natural language. CVPR, pages 5803-588 5212, 2017.
Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theoretical paradigm to understand learning from human preferences. In /CML. pages 4447 4447-4455 PMLR, 2024. 3
[3] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502. /3923, 2025. 3, 4, 5, 6, 14 [4] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.00 205-208224 2022. 3
[5] Zhuo Cao, Bingqing Zhang. Heming Du, Xin Yu, Xue Li, and Sen Wang. Flashvtg: Feature layering and adaptive score handling network for video temporal grounding. pages 9226-9236. IEFF, 2025. 3
[6] Sihan Chen, Handong Li. Qunbo Wang. Zijia Zhao, Mingzhen Sun, Xinxin Zhu, and Jing Liu. Vast: A omni- moni-modality foundation model and datasct. In NeurlPS, volume 36, pages
2023.
[7] Huilin Deng. Ding Zou, Rui Ma, Hongchen Luo, Yang Cao, and Yu Kang. Boosting the generaliza- tion and reasoning of vision language models with curriculum reinforcement learning. arXiv preprint
arXiv:2503.07063 2025. 2
[8] Yihe Deng, Hritik Bansal, Fan Yin, Nanyun Peng, Wei Wang, and Kai- Kei Chang. Openvithinker: An carly exploration to complex vision-language reasoning via iterative self-improvement. arXiv preprint
arXiv:2503.1 17352, 2025.
[9] Yifan Du. Zikang Liu, Yifan Li, Wayne Xin Zhao, Yuqi Huo, Bingning Wang. Weipeng Chen, Zheng Liu, Zhongyuan Wang, and Ji-Rong Wen. Virgo: A preliminary exploration on reproducing ol ol-like milm. arXiv
preprins or Xiv:2501. 019004 2025. 2
[10] Kaituo Feng, Kaixiong Gong, Bohao Li, Zonghao Guo, Yibing Wang. Tianshuo Peng, Benyou Wang, and Xiangyu Yue. Video-rl: Reinforcing video reasoning in milms. arXiv preprint arXiv:2503.21776 2025. I
[11] Jiyang Gao, Chen Sun, Zhenheng Yang, and Ram Nevatia. Tall: Temporal activity localization via language query. In ICCV, Oct 2017. 2, 15
[12] Daya Guo, Dejian Yang. Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang. Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.1294.
2025. 2
[13] Yongxin Guo, Jingyu Liu, Mingda Li, Dingxin Cheng, Xiaoying Tang, Dianho Sui, Qingbin Liu, Xi Chen, and Kevin Zhao. Vtg-Ilm: Integrating timestamp knowledge into video Ilms for enhanced video temporal
[14] grounding In AAAI, volume 39, pages 3302-3310, 2025.
Guo, Jingyu Liu, Mingda Li, Qingbin Liu, Xi Chen, and Xiaoying Tang. Trace: Temporal grounding video 1lm via causal event modeling, arXiv preprint arXiv:2410.08 05643, 2024, 6
[15] Abhishek Gupta, Aldo Pacchiano, Yuexiang Zhai, Sham Kakade, and Sergey Levine. Unpacking reward shaping: Understanding the benefits of reward engineering on sample complexity. In NeurlPS, volume 35,
pages 15281-15295, 2022. 2
[16] Jiwoo Hong, Noah Lec, and James Thome. Orpo: Monolithic preference optimization without reference model. arXiv preprint
preprint ar 2xXiv:2403.0761,, 2024. 3
[17] Bin Huang. Xin Wang. Hong Chen, Zihan Song, and Wenwu Zhu. Vtimellm: Empower Ilm to grasp video moments. In CVPR, pages 14271-14280, 2024. 1
[18] Bin Huang, Xin Wang, Hong Chen, Zihan Song, and Wenwu Zhu. Vtimellm: Empower Ilm to grasp video moments. In CVPR, pages 14271-14280, June 2024. 1 [19] Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Hel- yar. Aleksander Madry, Alex Beutel, Alex Camey, et al. Openai system card. arXiv preprint arXiv:2412. 16720, 2024, 2
[20] Jinhyun Jang, Jungin Park, Jin Kim, Hyeongjun Kwon, and Kwanghoon Sohn. Knowing where to focus:
Event-aware transformer for video grounding. In ICCV, pages 13846-13856, 2023.
[21] Peng Jin, Ryuichi Takanobu, Wancai Zhang, Xiaochun Cao, and Li Yuan. Chat-univi: Unified visual representation empowers large language models with image and video understanding. In CVPR, pages
13700-13710, June 2024.
[22] Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, and Juan Carlos Niebles. Dense-captioning events in videos. In /CCV. Oct 2017. 2, 15
[23] Harrison Lee, Samrat Phatale, Hassan Mansoor, Kellie Ren Lu, Thomas Mesnard, Johan Ferret, Colton Bishop, Ethan Hall, Victor Carbune, and Abhinav Rastogi. Rlaif: Scaling reinforcement learning from human feedback with feedback. 2023. 3
[24] Lei. Tamara L Berg, and Mohit Bansal. Detecting moments and highlights in videos via natural language queries. In NewrlPS, volume 34, pages 11846-11854 1186,1 2021. 1,2, 15
[25] Jie Lei, Licheng Yu, Tamara L Berg, and Mohit Bansal. Tvqa+: Spatio-temporal grounding for vidco
question answering. arXiv preprint arXiv: /9044/945544 /1544, 2019.
[26] Bo Li. Peiyuan Zhang, Kaichen Zhang, Fanyi Pu, Xinrun Du, Yuhao Dong. Haotian Liu, Yuanhan Zhang.
Ge Zhang, Chunyuan Li, and Ziwei Liu. Lmms-eval: Accelerating the development of large multimoal models, March 2024. 6
11

---

[27] Cheng Li, Jiexiong Liu, Yixuan Chen, and Yanqin Jia. Video-vot-rl: An efficient video inference model
integrating image packing and aoe architecture. arXiv preprint arXiv:2503. 15807, 2025. 3
[28] Hongyu Li, Jinyu Chen, Ziyu Wei, Shaofei Huang. Tianrui Hui, Jialin Gao, Xiaoming Wei, and Si Liu. Llava-st: A multimodal large language model for fine-grained spatial-temporal understanding. 2025. 1 [29] Jiaze Li, Yaya Shi, Zongyang Ma, Haoran Xu, Feng Cheng. Huihui Xiao, Ruiwen Kang. Fan Yang. Tingting Gao, and Di Zhang. imove:
aware video understanding, arXiv preprint arXiv:2502. 175994
2025. 6
[30]
KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang. Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding. arXiv preprint arXiv: 06355. 2023.
[31] Kunchang Li. Yali Wang. Yinan He, Yizhuo Li. Yi Wang. Yi Liu, Zun Wang. Jilan Xu, Guo Chen, Ping Luo, Limin Wang. and Yu Qiao. Mybench: A comprehensive multi-modal video understanding benchmark.
In CVPR, pages 22195 222066, June 2024. 1
[32] Xinhao Li. Yi Wang, Jiashuo Yu, Xiangyu Zeng, Yuhan Zhu, Haian Huang, Jianfei Gao, Kunchang Li. Yinan He. Chenting Wang. et al. addochat-flasha Hierarchical compression for long-context video modeling, arXiv prepriat arXiv:2501.00522
2024.
[33] Xinhao Li, Ziang Yan. Desen Meng. Lu Dong, Xiangyu Zeng, Yinan He, Yali Wang, Yu Qiao, Yi Wang. and Limin Wang. Videochat-r1: Enhancing spatio-temporal perception via reinforcement fine-tuning.
arxiv preprint arXiv:2504, 06958, 2025. 1, 2, 3, 6
[34] Zhaowei Li, Qi Xu, Dong Zhang. Hang Song. Yiqing Cai, Qi Qi, Ran Zhou, Junting Pan, Zefeng Li. Vu
Tu, et al. Groundinggpt: Language enhanced multi-modal grounding model. ACL, pages 6657-6678.
2024. 1
[35] Kevin Qinghong Lin, Pengchuan Zhang, Joya Chen, Shraman Pramanick, Difei Gao, Alex Jinpeng Wang. Rui Yan, and Mike Zheng Shou. Univtg: Towards unified video-language temporal grounding. In ICCV,
pages 2794-2804, 2023. 1
[36] Daizong Liu, Xiaoye Qu, and Wei Hu. Reducing the vision and language bias for temporal sentence grounding. In ACM MM, pages 4092-4101, 2022. 3
[37] Shenglan Liu, Aibin Zhang. Yunheng Li. Jian Zhou, Li Xu, Zhuben Dong, and Renhao Zhang. Temporal segmentation of fine-gained semantic action: A motion-centered figure skating dataset. In AAAI, volume 35,
pages 2163- 2021. 3
[38] Yuqi Liu, Bohao Peng. Zhisheng Zhong, Zihao Yue, Fanbin Lu, Bei Yu, and Jiaya Jia. Seg-zero: Reasoning- chain guided segmentation via cognitive reinforcement. arXiv preprint arXiv:2503. 2252200620.. 2025. 2
[39] Haozhe Ma. Zhengding Luo. Thanh Vinh Vo. Kuankuan Sima, and Tze- Yun Leong. Highly efficient
self- reward shaping for reinforcement learning. In ICLR, 2025. 2
[40] Muhammad Maaz, Hanoona Rasheed. Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt: Towards
detailed video understanding via large vision and language models. In ACL. 2024. 1
[41] WonJun Moon, Sangeek Hyun, SangUk Park, Dongehan Park, and Jac-Pil Hco. Query-dependent vidco representation for moment retrieval and highlight detection. In CVPR, pages 23023-23033, 2023, 1
[42] Fangzhou Mu, Sicheng Mo, and Yin Li. Snag: Scalable and accurate video grounding. In CVPR, pages
19930-18940,21 2024.
[43] Long Qian, Juncheng Li, Yu Wu, Yaobo Yc, Hao Fci, Tat-Seng Chua, Yucting Zhuang, and Siliang Tang. Momentor: advancing vidco large language model with fine-grained temporal reasoning. In ICML, pages
41340-41356, 2024.
[44] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning. Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. In NeurIPS, volume 36,
pages 53728 -3728-53741, 2023. 3
[45] Shuhuai Ren, Linli Yao, Shicheng Li, Xu Sun, and Lu Hou. Timechat: A time-sensitive multimodal large language model for long video understanding. In CVPR, pages 14313-14323, June 2024. 3. 6
[46] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 06347, 2017.
[47] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits mathematical reasoning in open language models. arviv preprint arXiv:2402. 03300, 2024. 1,
[48] Haozhan Shen, Peng Liu, Jingcheng Li, Chunxin Fang, Yibo Ma, Jiajia Liao, Qiaoli Shen, Zilun Zhang, Kangjia Zhao, Qianqian Zhang, ct al. VIm-rl: A stable and generalizable r1-style large vision-language
model. arXiv preprint 2xxiv:220040755 07615, 2025. 3
[49] Enxin Song, Wenhao Chai, Guanhong Wang, Yucheng Zhang, Haoyang Zhou, Feiyang Wu, Haozhe Chi, Xun Guo, Tian Ye, Yanting Zhang, al. Moviechat: From dense token to sparse memory for long video
understanding. In CVPR, pages 18221-18232, 2024. 1
[50] Huajie Tan, Yuheng Ji, Xiaoshuai Hao, Minglan Lin, Pengwei Wang. Zhongyuan Wang, and Shanghang Zhang. Reason-rft: Reinforcement fine-tuning for visual reasoning. arXiv preprint arXiv:2503.2077
2025. 3
[51] Yunlong Tang, Jing Bi, Siting Xu, Luchuan Song, Susan Liang, Teng Wang, Daoan Zhang. Jie An, Jingyang Lin, Rongyi Zhu, et al. Vidco understanding with large language models: A survey. IEEE TCSVT, 2025. I Kimi Team, Angang Du, Bofei Gao. Bowei Xing, Changjuu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du Chonghua Liao, et al. Kimi kl. 5: Scaling reinforcement learning with llms. arXiv
[52]
preprint arXiv:2501. 12599, 2025. 2
[53] Qwen Team. Qwen3, April 2025. 2
12

---

[54] Haibo Wang, Zhiyang Xu, Yu Cheng. Shizhe Diao, Yufan Zhou, Yixin Cao. Qifan Wang, Weifeng Ge, and Lifu Huang. Grounded--vddeollm: Sharpening fine-grained temporal grounding in video large language 2024.
models, arxiv preprini arX.v:2410.0322222
[55] Yi Wang. Yinan He, Yizhuo Li, Kunchang Li, Jiashuo Yu, Xin Ma, Xinhao Li, Guo Chen, Xinyuan Chen, Yaohui Wang, et al. Internvid: A large-scale video-text dataset for multimodal understanding and generation. arXiv preprint (.IXxx:33377004222 2023. 3
[56] Yi Wang. Kunchang Li. Xinhao Li, Jiashuo Yu, Yinan He, Guo Chen, Baoqi Pei. Rongkun Zheng. Zun Wang, Yansong Shi, ct al. Internvideo2: Scaling foundation models for multimodal vidco understanding.
In ECCV, pages 396-416. Springer. 2024. 4
[57] Yi Wang. Kunchang Li, Yizhuo Li, Yinan He, Bingkun Huang, Zhiyu Zhao, Hongjie Zhang, Jilan Xu, Yi Liu, Zun Wang, al. Internvideo: General video foundation models via generative and discriminative
learning. arXiv preprint arXiv:2212.0319/ 2022.
[58] Yi Wang, Xinhao Li, Ziang Yan, Yinan He, Jiashuo Yu, Xiangyu Zeng. Chenting Wang. Changlian Ma, Haian Huang, Jianfei Gao, Min Dou, Kai Chen, Wenhai Wang, Yu Qiao, Yali Wang, and Limin Wang. Intervideo2.55: Empowering video miims with long and rich context modeling. arXiv preprint wrXiv25501.23885 2025. 1
[59] Yueqian Wang, Xiaojun Meng, Jianxin Liang, Yuxuan Wang, Qun Liu, and Dongyan Zhao. Hawkeye: Training video-text Ilms for grounding text in videos, arXiv preprint arXiv:2403. 10228, 2024, 6
[60] Ye Wang. Boshen Xu, Zihao Yue, Zihan Xiao, Ziheng Wang. Liang Zhang. Dingyi Yang. Wenxuan Wang. and Qin Jin. Timezero: Temporal video grounding with reasoning-guidee Ivlm. arXiv preprint 2025. 3.
arXiv:2503. [61] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, ct al. Chain-of-thought prompting clicits reasoning in large language models. In NeurlPS, volume 35, pages 24824-24837,
2
[62] Bo Xiong. Yannis Kalantidis, Deepti Ghadiyaram, and Kristen Grauman. Less is more: Learning highlight detection from video duration. In CVPR, pages 1258- 1068-1267,2222 2019. 1
[63] Shen Yan, Xuchan Xiong. Arsha Nagrami, Anurag Arnah, Zhonghao Wang, Weina Ge, David Ross, and Cordelia Schmid. Unloc: A unified framework for video localization tasks. In ICCV, pages 13623-13633,
October 2023. 6
[64]
Dongjie Yang, Suyuan Huang, Chengqiang Lu, Xiaodong Han, Haoxin Zhang, Yan Gao, Yao Hu, and Hai Zhao. Vript: video worth thousands of words. In NeurlPS, volume 37, pages
2024. I En Yu, Kangheng Lin, Liang Zhao, Jisheng Yin, Yana Wei, Yuang Peng. Haoran Wei, Jianjian Sun, Chunrui Han, Zheng Ge, et al. Perception-rl: Pioneering perception policy with reinforcement learning. arXiv
[65]
preprint arXiv:2504.00 07954, 2025. 3
[66]
Shoubin Yu, Jaemin Cho, Prateek Yadav, and Mohit Bansal. Self-chained amage-language model for video localization and question answering. In NeurlPS, volume 36, pages 76749-76771, 2023.
[67]
Runhao Zeng, Haoming Xu, Wenbing Huang, Peihao Chen. Mingkui Tan, and Chuang Gan. Densc regression network for video grounding. In CVPR, pages 10297, 2020. 3
[68]
Xiangyu Zeng, Kunchang Li, Chenting Wang. Xinhao Li, Tianxiang Jiang, Ziang Yan, Songze Li, Yansong Shi, Zhengrong Yue, Yi Wang. Yali Wang. Yu Qiao, and Limin Wang. Timesuite: Improving milms for
long video understanding via grounded tuning. 2024. 6
[69] Songyang Zhang, Houwen Peng, Jianlong Fu, Yijuan Lu, and Jicbo Luo. Multi-scale 2d temporal adjacency networks for moment localization with natural language. IEEE TPAMI, 44(12)9903399000 2021.
[70] Xingjian Zhang, Siwei Wen, Wenjun Wu, and Lei Huang. Tinyllava-video-rl:: Towards smaller Imms for video reasoning. arXiv preprint arXiv:2504. 2504.004444 2025. 3
[71] Yi-Fan Zhang, Xingyu Lu, Xiao Hu, Chaoyou Fu, Bin Wen, Tianke Zhang, Changyi Liu, Kaiyu Jiang, Kaih- ing Chen, Kaiyu Tang, et al. RI Re-wew:rd: Training multimodal reward model through stable reinforcement learning. arXiv preprint 2.005.2.500028..
2025. 3
[72] Jiaxing Zhao, Xihan Wei, and Liefeng Bo. R1-ommi: Explainable umni-multimodal emotion recognition with reinforcement learning. arXiv preprint axXiv:2503.053::
2025. 1,
[73] Hao Zhou, Chongyang Zhang, Yan Luo, Yanjun Chen, and Chuanping Hu. Embracing uncertainty: Decoupling and de- for robust temporal grounding. In CVPR, pages 8445-8454, 2021.
[74] Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Yuchen Duan, Hao Tian, Weijie Su, Jie Shao, et al. Internv13: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint 250iv:2504. 10479, 2025.
13

---

Appendix
A Limitations and broader impact
Limitations. While TempSamp-RI demonstrates consistent improvements over existing GRPO- based methods across multiple temporal grounding benchmarks, it also presents several limitations. First, the framework currently relies on the availability of high-quality off-policy supervision (e.g.. ground-trutt truth timestamps), which may not be accessible in weakly labeled scenarios. Second, although TempSamp-Re evaluated on temporal grounding and highlight detection tasks, its effectiveness on other video reasoning tasks (e.g., multi eeulti-event tracking) remains to be explored.
Broader Impact. This work contributes to the advancement of reinforcement learning in long video understanding. By combining on-policy exploration with structured off-policy guidance, TempSamp- introduces more stable and data-efficient fine-tuning paradigm for vision-language models. We believe this direction can benefit downstream applications such as video retrieval, surveillance, and assistive robotics, where precise temporal reasoning is critical. Since our method is not designed for a specific application domain, it does not directly raise immediate societal or ethical concerns.
B Training details
We fine-tune our method on the 11-2255-VV-7---n-ee model [3] using full-paraan parameterr optimization. Tab. summarizes the full set of training configurations. The first-stage fine-tuning uses the same in-domain datasets the second stage, matching the corresponding evaluation benchmarks (e.g.. Charades-STl -TA and (VHighlights.
Trainable
Full Model
Per-device Batch Size
Gradient Accumulation
2
Epoch
Optimizer
AdamW
Deepspeed
Zero3-Offload
IR Schedule
1 x x10-6
Max Generated Sequence Length
2048
GPU Nums
4 A100 (80 GB)
Table Training configuration for TempSamp-Re
C Additional Experiments
Comparison of GRPO and TempSamp-R1 under CoT and non-CoT prompting We compare the performance of GRPO and TempSamp-RI with both CoT and non-CoT prompting strategies on the Charades-STA dataset. As shown in Tab. 8, TempSamp-R1 consistently outperforms GRPO regardless of the prompting approach. Notably, the improvements achieved with CoTT prompting are more pronounced, with gains of 1.5 and 4.8 points at RI @0.5 and RI @0.7 respectively, compared to 1.4 and 3.7 points under non-CoT prompting. These results suggest that the stabilized training and reward optimization TempSamp-RI better complement CoT prompting, enabling more effective utilization of intermediate reasoning steps. This highlights the complementary benefits of advanced reward design and CoT prompting improving temporal grounding.
D Visualizations
We present additional qualitative results to illustrate the behavior of TempSamp-Ri on temporal grounding tasks. Fig. 7 shows representative success cases from both Charades-STA and ActivityNet Captions, where TempSamp-RR accurately localizes the queried events and generates coherent reasoning under both CoT and non-CoT settings.
In contrast, Fig. 8 highlights common failure cases, which primarily stem from ambiguous visual content, repeated actions, loosely defined ground-truth annotations. tends to localize the carliest
14

---

Method
Training Prompt
Test Prompt
RI @00.3
RI@0.5
@0.7
GRPO
Non-CoT
Non-CoT Non-CoT
81.2
70.5
47.1
TempSamp-RI
Non-CoT
82.6
71.9
50.8
GRPO
CoT
CoT
83.0
72.6
48.1
TempSamp-RI
CoT
CoT
83.6
74.1
52.9
Table 8: Performance comparison of GRPO and TempSamp-RI with and without CoT prompting Charades-STh dataset.
matching interval, whereas the annotation provides only a single reference segment, leading to apparent discrepancies. These examples further underscore the challenges of precise temporal localization.
E
Datasets of training and evaluation
We train and evaluate TempSamp-RR RRR three widely-used video-language datasets, spanning two representative tasks: temporal grounding and highlight detection.
Charades-STA [11] is benchmark for temporal localization in indoor videos. Each sample consists of a natural language query and its corresponding temporal segment within a video. We follow standard splits with approximately 12.4k training and 3.7k validation examples.
License: Non-commereial use license provided by the Allen Institute for AL. https://prior.
allenai orgo http://oooecss/cceeeecooss htapjjeececccaaaeeeeeeeeee charades
URL: httpt:tt///gtttiue wwwp:/canga/ooo//////////oooo.aa TALL
ActivityNet Captions [22] provides temporally grounded captions for diverse videos. Each video contains multiple event annotations with rich semantic content. The dataset comprises approximately 20K long, untrimmed videos, with an average of 3.65 sentence-event pairs per video. Following standard practice, we adopt the official dataset splits, resulting in 37,421 training. 17,505 validation, and 17,031 test samples. Compared Charades-STA. this dataset covers broader domain and more diverse activity types.
URL: https::tp/// /sc. stanford http:///pppeeerraaaaaaaaataaaaaa
QVHighlights [24] a large-scale benchmark for evaluating query-conditioned highlight detection in long-form videos. The dataset comprises 10,148 curated video clips, each with fixed duration of 150 seconds. Each video is paired with at least one user user-issued natural language query describing salient moments, resulting in 10,310 unique queries and 18,367 annotated highlights. On average, each highlight spans approximately 24.6 seconds.
License: Attiburioo-nnnCGmmeecaaSSSeeee/ 4.0 International.
URL: htt.::/wwrgggghhha...///a//aa/ae
F
Prompts
We provide prompt templates used for temporal video grounding and video highlight detection tasks under both CoT and non-CoT settings. For CoT prompts, models are instructed to first reason step-by-step within <think> tags before outputting final predictions in <answer> tags. Non-CoT prompts, by contrast, elicit direct answers without intermediate reasoning. All prompts are designed to standardize response formats and support consistent evaluation across different tasks and inference modes. See Fig. for examples.
15

---

Video:
Vidco:
Query: there is person cooking something on the stove.
perroon
person start running through the hallway.
Ground Truth:
17.4
25.77
Ground Truth:
7.2
12.0
GRPO:
7.0
23.0
GRPO:
0.0 [-
5.7
TempSamp-RI (non-Cal): 0.0 | (e-......))))
20.0
TempSump-R. (non-CoT):
6.0 (
7.0
TempSamp-RI:
15.9
>) 25.0
TempSamp-R1:
6,0 [<--)o))))))) 11.0
<Think-> The man standing at stove, and there a pot on it. This suggests that he might cooking something. </Think>
<Think> The person is running down hallway while holding a cloth. moving quickly and his arms are swinging. </Think>
Video:
Query: After achieving his goal, the replay shows how good the man really handled the ball throw:
Query: The man opened bottle, rubbed the bottle on the ski board.
Ground Truth:
28.9
48.1
Ground Truth:
0.0
54.9
GRPO:
35.0
53.0
GRPO:
10.0 <- 30.0
TempSamp-RI (non-CoT):
30.1
55.1
TempSamp-RI (non-CoT):
20.0
|6..............................
160.9
TempSamp-RI:
29.1
50,9
TempSamp-R1:
22.0
[c...obb]] 51.00
<Think> The man uchieved his goal by throwing the ball on the field. The replay shows ball's trajectory and the distance it traveled, indicating that he threw the well, </tink>>
<Think> man opened a bottle, rubbed the bottle on the ski board. </think>
Video:
Video:
Query: A points and smiles.
Query: team are ice court playing hockey.
Ground Truth:
35.6
36.8
Ground Truth:
118.1
GRPO:
10.3
GRPO:
91.0
TempSamp-R1 (non-Col)::
9.1 [<->> 10.5
TempSamp-R1 (non-Col)):
113.9
TempSamp-RI:
34.9) <- 39.0
TempSump-RI:
106.9
<think> The caption "A guy points and smiles." suggests that there is person who points something and then smiles. In the video, we see man stilts who points at the camera and then smiles. This matches the deseription of the caption. </think>
<Think> The video starts with wide shot of an ice court where teams are playing hockey. The crowd visible the background, and the players skating around the ics court. The continues with closc-up shots of the players and their movements the ice. </Think>
Figure 7: Qualitative comparisons temporal grounding Charades-STA and ActivityNet Captions. TempSamp-ER R1 accurately localizes the queried events.
16

---

Video:
Video:
Query: the the Frishee air and the dog grabs in its mouth.
Query: Image and cltps of Mond hair stylist braiding her hair.
Ground Truth:
167.7
174.2
Ground Truth:
GRPO:
75.0
GRPO:
0.0
76.0
TempSamp-RI (non-CoT):
69.1
179.0
TempSamp-RI (non-CoT):0.0
69.9
TempSamp-Rl:
1170.1
TempSamp-Ri:
0.0
<Think> The video with opening title screen followed by collage images different hairstyles. Then, shows braiding her hair step-by-stcp.
<Think> The man throws frisbee in air and the dog eatches mouth, This happens several times throughout the video. </Think>
</Think>
Video:
Vidoo:
Query: The man changes the backgrounds as he leans forward, pushing back.
Query: A man and woman gear up and parry with each other.
Ground Truth:
80.7 <<> 117.8
Ground Truth:
88.0 |<-
139.6
GRPO:
25.3
183.0
GRPO:
33.0 74.0
TempSamp-RI RI (non-CoT))
39.1
109.0
TempSamp-RI RI (non-CoT):
26.0 |<-->> 83.0
TempSamp-Rl:
196.0 <Think> The man leans forward, pushing back againat the bull. seen in different positions, sometimes standing, sometimes leaning, and sometimes falling, The backgrounds change from green screen dosert scenc, then to cityscape, and finally to more detailed descrt soene, </hink<>
23.1|
TempSamp-RI:
67.6
<Think man and woman gear up by putting on fencing gear, then they parry with each other. </Think>
Figure Qualitative analysis of failure cases. Most prediction errors arise from ambiguous visual cues or imperfect ground-truth annotations, such as repetitive events and ill-defined temporal boundaries.
17

---

Temporal video grounding (Non-CoT):
"To accurately pinpoint the event "query" in the video, determine the precise time period of the event. Provide the start and end times (in seconds, precise to two decimal places) in the format "start time to end time" within the <answer> <lanswer> tags. For example: "12.54 17.83"."
Temporal video grounding (CoT):
"To accurately pinpoint the event "query" in the video, determine the precise time period of the event. Output your thought process within the <think> </think> tags, including analysis with either specific timestamps (xx.xx) or time ranges (xx.xX xx.xx) in <timestep> <'timestep> tags. Then, provide the start and end times (in seconds, precise to decimal places) in the format "start time to end time" within the <answer> </answer> tags. For example: "12.54 to 17.83"."
Video highlight detection (Non-CoT):
"Please find the highlight contents in the video described by a sentence query, determining the highlight timestamps and its saliency score on a scale from 1 to 5. Provide the highlight timestamps and its saliency score within the <answer> </answer> tags. The output format should be like: 'The highlight timestamps are the 82, 84, 86, 88, 90, 92, 94, 96, 98, 100 seconds. Their saliency scores are 1.3, 1.7, 1.7, 1.7, 1.7, 1.3, 1.7, 2.3, 2.3, 2.3'. Now I will give you the sentence query: 'query'. Please return the query-based highlight timestamps and salient scores."
Video highlight detection (CoT):
"Please find the highlight contents in the video described by a sentence query, determining the highlight timestamps and its saliency score on scale from 1 to Output your thought process within the <think> </think> tags, including analysis with either highlight timestamps. Then, provide the highlight timestamps and its saliency score within the <answer> </answer> tags. The output format should be like: 'The highlight timestamps are in the 82, 84, 86, 88, 90, 92, 94, 96, 98, 100 seconds. Their saliency scores are 1.3, 1.7, 1.7, 1.7, 1.7, 1.3, 1.7, 2.3, 2.3, 2.3'. Now will give you the sentence query: 'query'. Please return the query-based highlight timestamps and salient scores."
Figure Prompt templates for temporal grounding and video highlight detection tasks.
18

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The claims in the abstract and introduction accurately reflect the contributions, i.e. TempSamp-RI improves the effectiveness adapting multimodal large language models (MLLMs) to video temporal grounding tasks. The experiments detailed in Sec. 4.1 demon- strate the efficacy of our method enhancing temporal video grounding, video highlight detection and few-shot performance.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
Limitations are discussed in Sec. A.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they seale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
19

---

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proof' provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We give the implementation details and experiment hyperparameters in Sec. and Sec. B. We will make the code, training data and model weights publicly available.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (c.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
20

---

(c) If the contribution is a new model (e.g.. a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We intend make our code, training data and model weights publicly available following the paper's acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
//ipps cttpsscccc/cccc/cccccccceccccccc guwdes/dde/e/ee CoodeSobiisssseee www.rbiiissiiooooiooooooo
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We detail the implementation in Sec, 4 and Sec. B. Additionally, training and testing configurations included in the provided code.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
21

---

Answer: [No]
Justification: Based on our empirical experiments, the reproducibility of results for the same model is very high. The scores we report are reproducible and consistent across multiple trials, rather than being the best scores obtained. Moreover, we face limitations in computational resources, which prevent us from computing error bars for each experiment.
Guidelines:
The answer NA means that the paper does include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The details regarding computational resources are outlined in Sec. 4 and Sec. B.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: http///eerrrp.reeee//ee///
iines??
Answer: [Yes]
Justification: The research presented in this paper adheres to the NeurlPS Code of Ethics in all aspects.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
22

---

The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Broader impacts are discussed in Sec. A.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative socictal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymeetm of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: This paper does not engage with models that necessitate safeguards, and the datasets utilized are publicly available.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-us should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
23

---

Justification:
Licenses of datasets are put into Sec. E.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's s creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

---

Answer:
[NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: LLMs were not used as an important, original, or non-standard component of the core methods in this work.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non con-standard components.
Please refer to our LLM policy (https / //t///ners..///// wwtpc///eeeeenc///22/222222 for what should or should not be described.
25


Reviews:
[{"review_id": "9usP9U1e6m", "reviewer": "Reviewer_hB8a", "summary": "This paper presents TempSamp-R1, a reinforcement learning framework aimed at improving the performance of multimodal large language models (MLLMs) in video temporal localization tasks. It addresses the inefficiencies of existing methods like GRPO by using ground-truth annotations for precise temporal guidance and introducing a nonlinear soft advantage mechanism to stabilize training. Experiments on benchmark datasets such as Charades-STA, ActivityNet Captions, and QVHighlights demonstrate that TempSamp-R1 outperforms GRPO-based approaches, particularly in few-shot scenarios.", "questions": "Overall, I am particularly interested in the **advantage anchoring** and **non-linear reward shaping** methods regarding their generalizability across other tasks (besides Temporal Grounding) and the robustness of their hyperparameters. It remains to be seen whether these methods are universally important when combined with **mixed-policy sampling** or if they are only effective specifically for **temporal grounding**. However, this does not constitute a negative opinion of the paper.", "limitations": "yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths\n- Innovative Framework Design: TempSamp-R1 introduces a hybrid policy training strategy that combines off-policy guidance (e.g., ground-truth annotations) with on-policy generation to offer more accurate temporal supervision. This helps resolve the common issues of sparsity and misalignment in on-policy sampling. The effectiveness has been thoroughly demonstrated in the experimental parts.\n- Stable Training Mechanism: By incorporating a nonlinear soft advantage estimation mechanism, TempSamp-R1 dynamically reshapes reward feedback, thereby reducing gradient variance and adjusting advantage bias. This facilitates robust exploration and convergence.\n- Strong Experimental Results and Thorough Analysis: Experiments on multiple benchmark datasets show that TempSamp-R1 outperforms existing GRPO-based methods and exhibits strong generalization capabilities in few-shot scenarios.\n- Clear Illustrations: The paper provides numerous figures, such as Fig. 3 and Fig.4 , which significantly improve both the readability the this paper and the interpretability of the proposed methods.\n\nWeakness\n- Both **advantage anchoring** and **non-linear reward shaping** are intuitive concepts. However, aside from a set of effectiveness experiments, the paper does not include ablation studies specifically targeting these two parameters.\n- The article has significant practical value, but its overall innovativeness is somewhat limited.", "was_revised": false, "final_justification": "", "rebuttal": "## W1: Lack of ablations for advantage anchoring and non-linear reward shaping.\n\nThank you for the valuable suggestion. We have added ablations for $\\lambda_{\\text{off}}$ (used in advantage anchoring), as well as $\\tau$, $\\alpha_1$, and $\\alpha_2$ (hyperparameters within non-linear reward shaping). \n- We observe that a low $\\lambda_{\\text{off}}$ (e.g., 1.0) weakens off-policy supervision, while overly large values (e.g., 1.4) introduce distributional shifts between on- and off-policy solutions, leading to biased advantage estimation and degraded policy learning stability.\n- For non-linear reward shaping, the model shows robustness to $\\alpha_1$ and $\\alpha_2$ within reasonable ranges (e.g., $\\alpha_1 \\in [0.01, 0.02]$, $\\alpha_2 \\in [0.5, 1]$). Regarding $\\tau$, values near 0.8 generally led to improved training stability and performance in our experiments. Higher values (e.g., 0.9) skew the reward distribution toward a few high-reward solutions, limiting the model’s ability to learn from suboptimal solutions.\n\nWe will include these ablation results and analyses in the final revision.\n\n| Method           | $\\lambda_{\\text{off}}$ | R1@0.3 | R1@0.5 | R1@0.7 |\n|------------------|------|--------|--------|--------|\n| advantage anchoring | 1.0  | 81.9   | 69.7   | 48.5   |\n| advantage anchoring | **1.2**  | **81.8**   | **70.7**   | **49.1**   |\n| advantage anchoring | 1.4  | 81.8   | 68.8   | 46.5   |\n\n| Method                  |  $\\tau$     | R1@0.3 | R1@0.5 | R1@0.7 |\n|-------------------------|-------|--------|--------|--------|\n| non-linear reward shaping | 0.6   | 81.3   | 69.6   | 47.6   |\n| non-linear reward shaping | 0.7   | 81.6   | 70.5   | 48.4   |\n| non-linear reward shaping | **0.8**   | **82.9**   | **72.1**   | **49.6**   |\n| non-linear reward shaping | 0.9   | 81.1   | 69.4   | 47.5   |\n\n| Method                  | $\\alpha_1$ | R1@0.3 | R1@0.5 | R1@0.7 |\n|-------------------------|----------|--------|--------|--------|\n| non-linear reward shaping | 0.05     | 81.6   | 70.5   | 48.4   |\n| non-linear reward shaping | **0.01**     | **82.9**   | **72.1**   | **49.6**   |\n| non-linear reward shaping | 0.02     | 81.9   | 71.0   | 49.1   |\n\n| Method                  | $\\alpha_2$ | R1@0.3 | R1@0.5 | R1@0.7 |\n|-------------------------|----------|--------|--------|--------|\n| non-linear reward shaping | 0.5      | 82.2   | 71.6   | 49.0   |\n| non-linear reward shaping | **1**        | **82.9**   | **72.1**   | **49.6**   |\n| non-linear reward shaping | 2        | 81.1   | 69.5   | 46.9   |\n\n\n## W2: Significant practical value, with concerns regarding limited novelty.\nWe appreciate the reviewer’s recognition of the practical value of our work. The effectiveness of our framework stems from addressing a **key limitation** in prior RL-based approaches (e.g.,TimeZero and VideoChat-R1) for video temporal grounding, specifically unstable policy learning under **sparse** and **temporally misaligned reward supervision**.\n\nTo address these limitations, we propose two novel mechanisms:\n\n(1) Off-policy guidance. Our method introduces a novel use of ground-truth annotations by directly **integrating them into the policy update process**, rather than merely using them for **reward evaluation of on-policy samples** as in GRPO-based approaches. These instruction-aligned annotations serve as active supervision signals that enhance training stability by providing temporally accurate and semantically grounded feedback.\n\n(2) Non-linear reward transformation. To address the distributional bias introduced by incorporating off-policy rewards, we adopt a soft advantage estimation mechanism inspired by adaptive reward shaping. Rather than **treating all rewards uniformly**, our method applies an **asymmetric transformation** that compresses the advantage values of near-optimal solutions while amplifying distinctions among suboptimal ones. This strategy produces more informative gradients and mitigates the suppression of promising on-policy samples, thereby improving training stability and encouraging effective policy exploration.\n\nThese components are seamlessly integrated within a CoT-compatible framework, resulting in consistent improvements over GRPO baselines across multiple dataset. We will make this contribution more explicit and emphasize its significance in the final version.\n\n## Q1:  Generalizability and robustness across tasks.\n\nTo assess generalizability beyond temporal grounding, we conducted additional experiments on the NExT-GQA benchmark following the training and evaluation setting introduced by VideoChat-R1. This task involves spatio-temporal grounding for video-based question answering, where the model must identify both the correct answer (acc) and its supporting temporal segment (mIoU).\n\nWe observe that both advantage anchoring and non-linear reward shaping consistently outperform GRPO across multiple metrics. Notably, these results are achieved using the same hyperparameter configurations as in our temporal grounding experiments, without any task-specific tuning. This demonstrates that our proposed methods are robust and transferable to structurally different tasks that require temporal grounding with multimodal reasoning.\n\n|                          | NExTGQA |        |\n|--------------------------|---------|--------|\n|                          | mIoU    | acc    |\n| Qwen2.5-VL-7B        | 15.4    | 59.5   |\n| GRPO                     | 35.1    | 68.7   |\n| TempSamp-R1 with advantage anchoring | **35.5**    | **74.5**   |\n| TempSamp-R1 with non-linear reward shaping | **36.0**    | **76.7**   |\n\n\n### **Please feel free to leave additional comments if you have further questions or suggestions.**"}, {"review_id": "JtqfGmgWQX", "reviewer": "Reviewer_xy81", "summary": "This paper introduces a reinforcement fine-tuning framework to improve multimodal large language models (MLLMs) for video temporal grounding. Due to the vast search space for time intervals in this task, methods like GRPO can be inefficient. To address this, TempSamp-R1 incorporates ground-truth annotations as off-policy supervision, adding one off-policy solution alongside on-policy samples per query. Additionally, to prevent instability from directly using off-policy rewards, the paper proposes three strategies—reward downscaling, advantage anchoring, and non-linear reward shaping—to stabilize training. Experimental results in video temporal grounding benchmarks validate improvements over GRPO-based baselines.", "questions": "Overall, it would be valuable for this paper to provide more experimental clarification on why the proposed method works well—whether it benefits from CoT, normalization, or other factors.\n\nQ1. [See W1]\n\nQ2. [See W2]\n\nQ3. **Limited Exploration vs. Stable Convergence Trade-off:** The reduced variance in Top-1 IoU rewards for TempSamp-R1 in Figure 4 is presented as evidence of stability. However, could this tighter distribution also indicate reduced exploration of the solution space compared to the GRPO baseline? While the median reward is higher, is there a risk that TempSamp-R1 may converge to a local optimum, potentially limiting its ability to discover diverse or novel high-quality solutions?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 2, "strengths_and_weaknesses": "## Strengths\n\nS1. **Clear Presentation:** The paper is well-written and easy to follow, with clear explanations that make the context easy to understand.\n\nS2. **Clear Motivation:** The motivation is well-argued. As shown in Figure 4, while GRPO’s ability to enrich the search space with on-policy solutions is beneficial, it can also hinder the model's training stability. This is reasonable, given that pretrained MLLMs may not have sufficient knowledge of temporal grounding tasks, making it hard to generate helpful signals, especially at the beginning of training.\n\nS3. **Performance**: Experimental results (Table 1) shows that the complete pipeline clearly outperforms other SFT methods or RL methods using the same model in video temporal grounding tasks.\n\n\n\n## Weaknesses\n\nW1. **Need Clarification on Contribution (Reward design vs. CoT):** I believe the strength of GRPO in reasoning tasks lies in its ability to prompt MLLMs for additional thinking steps, even without explicitly teacher-forcing intermediate steps based on ground-truth. In this regard, I am curious about the difference between GRPO (CoT) and TempSamp-R1 (CoT). The ablation study only compares GRPO (non-CoT), TempSamp-R1 (non-CoT), and TempSamp-R1 (CoT). I wonder whether the observed improvements come primarily from the stabilized reward design, from the use of CoT itself, or the synergy of both.\n\nW2. **Heuristics on Parameters:** The specific non-linear function (Equation 4) and its parameters ($\\tau$ = 0.8, $\\alpha_1$ = 0.01, $\\alpha_2$ = 1) appear to be empirically chosen heuristics. While it is common practice to carefully select hyper-parameters for stable RL fine-tuning, I am curious about how sensitive the proposed design is to these parameters.", "was_revised": true, "final_justification": "My previous concerns on the ablation study and the parameter selection have been resolved during the rebuttal and discussion phase. Therefore I maintain my initial score, 4: borderline accept, and lean towards acceptance.", "rebuttal": "## Q1&W1: Clarifying the contribution of CoT vs. reward design.\n\nThank you for the insightful question. We conducted additional comparative experiments between GRPO and TempSamp-R1 under both CoT and non-CoT prompting strategies. The results confirm that TempSamp-R1 consistently outperforms GRPO across all settings, with or without CoT. Notably, the performance gains observed in the CoT setting (+1.5 R1@0.5 and +4.8 R1@0.7) are comparable to or even larger than those in the non-CoT setting (+1.4 R1@0.5 and +3.7 R1@0.7). This suggests that CoT prompting benefits more from stabilized training, allowing the model to more effectively exploit intermediate reasoning steps. Additionally, these results indicate that CoT prompting and the stabilized reward optimization, achieved through the combination of off-policy guidance and non-linear reward transformation, contribute complementary benefits. We will include these comparative results and further discussion in the final revision.\n\n| Method      | Training Prompt | Test Prompt | R1@0.3 | R1@0.5 | R1@0.7 |\n|-------------|-----------------|-------------|--------|--------|--------|\n| GRPO        | Non-CoT       | Non-CoT   | 81.2   | 70.5   | 47.1   |\n| TempSamp-R1 | Non-CoT       | Non-CoT   | **82.6**   | **71.9**   | **50.8**   |\n|             |                 |             |        |        |        |\n| GRPO        | CoT             | CoT         | 83.0   | 72.6   | 48.1   |\n| TempSamp-R1 | CoT             | CoT         | **83.6**   | **74.1**   | **52.9**   |\n\n\n## Q2&W2: Heuristic parameter selection and sensitivity.\n\nThank you for the thoughtful question.\nIn video temporal grounding, rewards are often sparse and temporally misaligned, making it difficult for partially correct predictions to receive useful feedback, especially when incorporating off-policy guidance that may diverge from the policy distribution. To address this, Eq. 4 is motivated by self-adaptive reward shaping techniques that have been widely applied to stabilize learning. We adopt an asymmetric formulation to better distinguish partially correct outputs from suboptimal ones. \n\nWe have added corresponding ablation experiments. Results show that the model is relatively insensitive to $\\alpha_1$ and $\\alpha_2$ within reasonable ranges (e.g., $\\alpha_1 \\in [0.01, 0.02]$, $\\alpha_2 \\in [0.5, 1]$). Regarding $\\tau$, values near 0.8 generally led to improved training stability and performance in our experiments. Higher values (e.g., 0.9) tended to exacerbate reward imbalance by concentrating reward mass on very few samples, which limits the model's ability to benefit from informative but suboptimal solutions.\n\nWe will include these ablation results and analyses in the final revision.\n\n| Method                  |  $\\tau$     | R1@0.3 | R1@0.5 | R1@0.7 |\n|-------------------------|-------|--------|--------|--------|\n| non-linear reward shaping | 0.6   | 81.3   | 69.6   | 47.6   |\n| non-linear reward shaping | 0.7   | 81.6   | 70.5   | 48.4   |\n| non-linear reward shaping | **0.8**   | **82.9**   | **72.1**   | **49.6**   |\n| non-linear reward shaping | 0.9   | 81.1   | 69.4   | 47.5   |\n\n| Method                  | $\\alpha_1$ | R1@0.3 | R1@0.5 | R1@0.7 |\n|-------------------------|----------|--------|--------|--------|\n| non-linear reward shaping | 0.05     | 81.6   | 70.5   | 48.4   |\n| non-linear reward shaping | **0.01**     | **82.9**   | **72.1**   | **49.6**   |\n| non-linear reward shaping | 0.02     | 81.9   | 71.0   | 49.1   |\n\n| Method                  | $\\alpha_2$ | R1@0.3 | R1@0.5 | R1@0.7 |\n|-------------------------|----------|--------|--------|--------|\n| non-linear reward shaping | 0.5      | 82.2   | 71.6   | 49.0   |\n| non-linear reward shaping | **1**        | **82.9**   | **72.1**   | **49.6**   |\n| non-linear reward shaping | 2        | 81.1   | 69.5   | 46.9   |\n\n\n## Q3: Trade-off between exploration and stable convergence.\n\nThank you for the valuable question. As figures cannot be included in the rebuttal, we instead report policy entropy values at different training steps. Compared to GRPO, TempSamp-R1 demonstrates higher policy entropy on ActivityNet, suggesting stronger exploratory behavior during training. This effect arises in part from the use of ground-truth annotations as off-policy guidance, since these solutions typically fall outside the model's high-probability prediction region. Integrating such informative but underexplored outputs into the learning process enables the model to expand its exploration space without compromising convergence stability. We will include the relevant results and analysis in the final revision.\n\n\n### Table: Entropy per Step for Policy Models (GRPO & TempSamp-R1)\n\n| Method/step     | 500   | 1000  | 2000  | 3000  | 4000  |\n|------------------|-------|-------|-------|-------|-------|\n| GRPO   | 0.775 | 0.730 | 0.685 | 0.675 | 0.647 |\n| TempSamp-R1 | 0.805 | 0.788 | 0.786 | 0.783 | 0.779 |\n\n### **Please feel free to leave additional comments if you have further questions or suggestions.**"}, {"review_id": "2SVUK3Et3v", "reviewer": "Reviewer_J4MX", "summary": "The paper introduces TempSamp-R1, a reinforcement fine-tuning framework that enhances the temporal grounding capabilities of multimodal large language models (MLLMs) for video understanding. It addresses the inefficiencies of prior methods like GRPO by incorporating off-policy supervision (i.e., ground-truth annotations) and non-linear soft advantage estimation to improve stability and exploration during training. The authors further integrate Chain-of-Thought (CoT) prompting into a unified model to support both CoT and non-CoT inference. TempSamp-R1 achieves state-of-the-art results on Charades-STA, ActivityNet Captions, and QVHighlights, and demonstrates strong performance in few-shot and cross-domain settings.", "questions": "See Weakness.", "limitations": "See Weakness.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1.\tThe paper critically analyzes why GRPO struggles with sparse rewards and unstable updates in large temporal search spaces. Based on this, a novel method is proposed to incorporate groud-truth annotations during RL training. This method is strengthen by a non-linear reward shaping to stabilizes the training.\n\n2.\tThe proposed method achieves generally better performance on multiple temporal grounding benchmarks. Futhur, the method demonstrates advantage over GRPO on cross-domain generalization and few-shot experiments. Extensive analysis and ablation studies also facilitate the intake of proposed method.\n\n3.\tThe writing is generally great by demonstrating the intuition and understanding behind the proposed method.\n\nWeaknesses:\n\n1.\tQuestion about off-policy output: As shown in Figure 2, during training, the model should be forced to output the ground-truth prediction so that the model weights can be updated with respective to this prediction. How was the model forced to do that? \n\n2.\tQuestion about Eq. 4: Can authors justify the design of this function? Specifically, why it has to be the format as Eq. 4 rather than other alternatives? Is it simply the best formulation from a few empirical observations?\n\n3.\tTempSamp-R1 (CoT): In L180-183, the model is trained in two stages where both stages are RL training. Since there is no CoT annotations used, how can we expect the model to output reliable CoT in the second stage? \n\n4.\tTempSamp-R1 Mixed CoT: In L234-235, how were the “better output” selected from two models? Did authors simply choose one of the two outputs that achieve higher scores of grounding metrics?\n\n5.\tTable 3: The proposed method requires much less time than GRPO. How does the proposed method save training time?\n\n6.\tFigure 3: How was the skewness score computed?\n\n7.\tThe necessity of CoT: Even though CoT achieve better results (Table 6: +0.6 mIoU between row 5 and row 8), it incurs much more computation during inference. Is it necessary for temporal grounding, given that the improvement is not significant?", "was_revised": true, "final_justification": "Thanks authors for their responses. My concerns have been addressed and I raised my score.", "rebuttal": "## Q1: How is ground-truth used for off-policy supervision?\nThank you for the clarification request. The model is not forced to output the ground-truth prediction. Instead, ground-truth annotations are incorporated into the advantage estimation as external supervision signals to stabilize learning. The policy itself is still optimized based on gradients computed from solutions and their corresponding advantage values. We will clarify this distinction more explicitly in the final version.\n\n## Q2: Justification for the functional form of Eq. 4.\n\nThank you for the insightful question. In video temporal grounding, rewards are often sparse and temporally misaligned, making it difficult for partially correct predictions to receive useful feedback, especially when incorporating off-policy guidance that may diverge from the policy distribution. To address this, Eq. 4 is motivated by self-adaptive reward shaping techniques that have been widely applied to stabilize learning. We adopt an asymmetric transformation that compresses the advantages of near-optimal outputs while amplifying differences among suboptimal ones. This encourages the model to focus on high-IoU outputs while still benefiting from partially correct predictions, thereby improving the quality of gradient feedback. We will clarify the design rationale and task-specific motivations more explicitly in the final version.\n\n## Q3: How can the model learn reliable CoT outputs?\n\nThank you for the thoughtful question. Rather than explicitly enforcing CoT generation, our framework treats CoT as an adaptive strategy that the model may utilize when beneficial for task performance. First, the base Qwen2.5-VL model already demonstrates CoT-style reasoning, and this ability is preserved after the first training stage. During the second stage, the model learns through reward-guided exploration that producing intermediate reasoning steps can improve grounding rewards. Representative examples of such outputs can be found in Figures 5, 6, and 7. We will clarify this design intent more explicitly in the final version.\n\n## Q4: How is the “better output” selected in Mixed CoT mode?\n\nIn Table 1, TempSamp-R1 Mixed CoT simply selects the output (CoT or non-CoT) with the higher grounding score to illustrate their complementarity. We also experimented with simple heuristics such as using video length for selection, but found no consistent advantage over single-mode inference. We will clarify the selection strategy more explicitly in the final version.\n\n## Q5: Why is TempSamp-R1 more time-efficient than GRPO?\n\nTempSamp-R1 achieves higher time efficiency primarily through two factors. First, our off-policy guidance allows the direct incorporation of ground-truth solutions into advantage estimation, which reduces the number of on-policy samples required per query from $G$ (in GRPO) to $G{-}1$, thereby lowering the computational cost of sampling. Second, we observe that our method produces shorter average completions during training (78 vs. 118 tokens), leading to reduced sequence generation overhead. We will clarify both aspects of this efficiency gain in the final version.\n\n## Q6: How is the skewness score computed?\n\nWe compute skewness using the standard third standardized moment:  \n\n$S = \\frac{1}{G} \\sum_{i=1}^G \\left( \\frac{r_i - \\mu}{\\sigma} \\right)^3$,\n\nwhere $r_i$ is the reward of the $i$-th sample, $\\mu$ and $\\sigma$ are the mean and standard deviation of the reward distribution, and $G$ is the total number of solutions. This computation is implemented using scipy.stats.skew, and it quantifies the asymmetry and sharpness of the reward distribution during training. We will clarify this in the final version.\n\n\n## Q7: The necessity of CoT\n\nThank you for the valuable question.  In our experiments conducted on a single NVIDIA A100 GPU, the inference time for CoT outputs remains comparable to that of non-CoT outputs (2.12s vs. 2.10s per sample), due to the relatively short length of generated reasoning steps. Furthermore, our framework is designed to accommodate both CoT and non-CoT reasoning within a unified model. As shown in Table 1, we find that CoT can improve grounding quality with minimal computational overhead in some settings (e.g., Charades-STA), making it a viable option when quality gain justifies the cost. However, in tasks where CoT offers limited performance benefits (e.g., QVHighlights), it can be omitted to reduce latency. We will clarify this point in the final version.\n\n### **Please feel free to leave additional comments if you have further questions or suggestions.**"}, {"review_id": "DuTrff5IFo", "reviewer": "Reviewer_sPEX", "summary": "This work proposes a method based on Group Relative Policy Optimization (GRPO) to fine-tune multimodal large language models (MLLMs ) on video grounding tasks. On top of GRPO, which has already been proven beneficial for video understanding and grounding tasks, this method introduces off-policy guidance, which further stabilizes the optimization process. The proposed method is evaluated on three dataset and tested against state-of-the-art models, both GRPO-based and using standard fine-tuning.", "questions": "**Q1:** **Table 1 is missing several important numbers and metrics**. While I understand that some data may not be available in the original papers, the table loses valuable information as it currently stands. For most of the models, certain metrics have been reported; therefore, when available, a trained model can be evaluated to provide the missing values. For example, if TempSamp-R1 can not be evaluated on other RL-based approaches on QVHighlights, then the comparison on this dataset is weak. \nAdditionally, I do not understand whether the *Qwen2.5-VL-7B* numbers refer to a zero-shot evaluation of the model or if they involve some level of fine-tuning. If they do not (as I assume from the -), can you explain why these numbers differ from those presented in the original paper [3]?\n\n**Q2:** Why are the **SFT and GRPO baselines** missing from Table 1? I see results in the ablation studies and the OOD experiments, but I think it is important to compare the differences across all datasets considered. \n\n**Q3:** It is not clear to me the real **contribution** of this paper. It seems that most of the mechanisms used were already used in previous work (GRPO for VTG task, iou reward, format reward, \\<think\\>). Is the main contribution only replacing random on-policy sampling with off-policy solutions to alleviate reward sparsity? The value of this, if I am not mistaken, is shown only in Table 4 and for one dataset (see Q2). The authors should elaborate on this to make their contributions stand out. \n\n**Q4:** Does the **first stage of training** use in-domain data or a different dataset? I think this information is missing in the current version of the paper. \n\n**Q5:** This is a **comment** rather than a question. I noticed that the **TimeZero numbers** you reported do not align with the current version of the paper you cited, which was updated on May 26th. I understand that this is concurrent work; if you choose to cite it and include it in your comparison, you should update the numbers accordingly.", "limitations": "yes", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths:** The paper proposes a method using  Group Relative Policy Optimization (GRPO) to fine-tune multimodal large language models (MLLMs ) on video grounding tasks, i.e., video temporal video grounding and highlight detection. This method replaces random on-policy sampling with off-policy solutions, reducing reward sparsity. The proposed approach is tested on 3 datasets. The idea follows the recent trend of using GRPO for fine-tuning MLLMs for tasks requiring temporal understanding. \n\n**Weaknesses:** While the approach obtains higher results on the baselines and previous GRPO-based approaches, it is tested on one model only (Qwen2.5-VL-7B) and shares part of the methodology with previous works (e.g., GRPO for VTG task, iou reward, format reward, \\<think\\>). As I detail in the question section below, I believe some numbers are missing from the reported tables, which weakens the overall evaluation.\n\n**Minor Weaknesses:** There are three **methodname** left in the paper by mistake (L274, L276, L279).", "was_revised": true, "final_justification": "I had multiple questions, and my initial rating was below acceptance. The authors addressed my concerns and detailed the proposed modifications they will implement. Consequently, I think this paper meets the required standard, and I have decided to raise my score.", "rebuttal": "## W1&Q2: One model only (Qwen2.5-VL-7B); missing SFT and GRPO baselines in Table 1.\n\nThank you for the valuable question. We have added comparisons on both Qwen2.5-VL-7B and Qwen2.5-VL-3B, including full baselines (SFT and GRPO). As shown in the updated table, TempSamp-R1 consistently outperforms GRPO and SFT under both model sizes, highlighting the robustness of our method beyond a single backbone.\n\nNotably, GRPO struggles to provide stable gains on the weaker Qwen2.5-VL-3B model, particularly on QVHighlights, where its performance is comparable to SFT. We attribute this to the high variance and sparse reward signals in GRPO, which are more difficult to optimize for smaller models. TempSamp-R1 effectively mitigates this limitation through off-policy supervision and stabilized reward shaping. \n\nWe will include these experimental results and analyses in the final version.\n\n### Table: Results on Charades-STA Dataset\n\n| Model              | Method      | mIoU | R1@0.3 | R1@0.5 | R1@0.7 |\n| ------------------ | ----------- | ---- | ------ | ------ | ------ |\n| Qwen2.5-VL-7B      | -           | 29.0 | 43.9   | 24.2   | 11.1   |\n| Qwen2.5-VL-7B      | SFT         | 56.8 | 79.6   | 66.7   | 43.5   |\n| Qwen2.5-VL-7B      | GRPO        | 59.0 | 81.2   | 70.5   | 47.1   |\n| Qwen2.5-VL-7B      | TempSamp-R1 | **61.7** | **83.3**   | **73.6**   | **52.2**   |\n|             |                 |             |        |        |        |\n| Qwen2.5-VL-3B      | -           | 33.8 | 51.8   | 32.1   | 15.0   |\n| Qwen2.5-VL-3B      | SFT         | 46.1 | 68.6   | 51.5   | 26.5   |\n| Qwen2.5-VL-3B      | GRPO        | 48.7 | 72.4   | 55.3   | 29.4   |\n| Qwen2.5-VL-3B      | TempSamp-R1 | **54.8** | **78.2**   | **64.8**   | **38.3**   |\n\n### Table: Results on ActivityNet Captions Dataset\n\n| Model              | Method      | mIoU | R1@0.3 | R1@0.5 | R1@0.7 |\n| ------------------ | ----------- | ---- | ------ | ------ | ------ |\n| Qwen2.5-VL-7B      | -           | 21.1 | 28.3   | 15.8   | 7.5    |\n| Qwen2.5-VL-7B      | SFT         | 39.6 | 54.5   | 37.4   | 23.0   |\n| Qwen2.5-VL-7B      | GRPO        | 47.2 | 68.6   | 47.3   | 26.9   |\n| Qwen2.5-VL-7B      | TempSamp-R1 | **52.1** | **72.8**   | **55.4**   | **34.2**   |\n|             |                 |             |        |        |        |\n| Qwen2.5-VL-3B      | -           | 17.0 | 24.2   | 12.3   | 5.3    |\n| Qwen2.5-VL-3B      | SFT         | 37.8 | 54.0   | 35.3   | 19.3   |\n| Qwen2.5-VL-3B      | GRPO        | 42.2 | 62.3   | 44.5   | 24.2   |\n| Qwen2.5-VL-3B      | TempSamp-R1 | **45.3** | **64.0**   | **46.3**   | **27.1**   |\n\n### Table: Results on QVHighlights Dataset\n\n| Model              | Method      | mAP  | HIT@1 |\n| ------------------ | ----------- | ---- | ------ |\n| Qwen2.5-VL-7B      | -           | 19.7 | 34.1   |\n| Qwen2.5-VL-7B      | SFT         | 26.5 | 34.1   |\n| Qwen2.5-VL-7B      | GRPO        | 18.0 | 35.0   |\n| Qwen2.5-VL-7B      | TempSamp-R1 | **30.0** | **57.6**   |\n|             |                 |             |        |        |        |\n| Qwen2.5-VL-3B      | -           | 13.8 | 22.8   |\n| Qwen2.5-VL-3B      | SFT         | 22.8 | 28.7   |\n| Qwen2.5-VL-3B      | GRPO        | 10.9 | 14.9   |\n| Qwen2.5-VL-3B      | TempSamp-R1 | **27.9** | **41.0**   |\n\n\n\n## W2&Q3: Clarifying the main contributions.\n\nThank you for the clarification request. Prior work, including GRPO for VTG, IoU reward, format reward, and think prompting, **serves as the foundation of our baseline**. Building upon these components, our method addresses a central limitation in video temporal grounding: unstable policy optimization under **sparse** and **temporally misaligned** reward supervision.\n\nTo address these limitations, our contribution lies in designing a more stable and generalizable reinforcement fine-tuning framework through two novel components:\n\n(1) Off-policy guidance, which for the first time incorporates high-quality, instruction-aligned supervision from ground-truth annotations directly into policy updates. Unlike prior GRPO-based methods that treat ground-truth solely as a reference for reward evaluation of on-policy samples, our approach leverages these annotations as active supervision signals, providing temporally precise guidance that improves stability and convergence during training.\n\n(2) Non-linear reward transformation, which dynamically reshapes sparse and skewed reward distributions to enable stable policy optimization. In contrast to GRPO, which treats reward values uniformly and is prone to instability under outlier-biased distributions, our transformation preserves informative gradients from partially correct predictions while mitigating the influence of extreme values.\n\n(3) As shown in Table 4, our ablation and comparative studies demonstrate the complementary effects of both components. In addition, we extend the evaluation in W1 across multiple datasets (Charades-STA, ActivityNet Captions, QVHighlights) and model scales (Qwen2.5-VL-3B and 7B), confirming the effectiveness and generalizability of our method.\n\nWe will clarify this distinction more explicitly in the final version.\n\n\n## W3&Q1: Missing values and incomplete comparisons; Qwen2.5-VL-7B setup in Table 1.\n\nWe appreciate the reviewer’s observation. As shown in the table in W1, we have added several missing metrics, including GRPO, SFT and baseline results, to support a more complete and fair comparison. Additionally, QVHighlights is a key benchmark for temporal grounding, yet was omitted in prior RL-based methods. To address this gap, we have included GRPO baselines on this dataset and extended the evaluation to multiple model sizes, demonstrating the effectiveness and generalizability of TempSamp-R1.\n\nThe reported results for Qwen2.5-VL are from zero-shot evaluation without any fine-tuning. For fair comparison with TimeZero and VideoChat-R1, we standardized the total number of input pixels per video to approximately 2.8 million, which may lead to slight discrepancies from the original paper. We will clarify this setup in the final version.\n\n## Q4: Clarification on training data for the first stage.\n\nThank you for the question. The first-stage fine-tuning uses the same in-domain datasets as the second stage, matching the corresponding evaluation benchmarks (e.g., Charades-STA and QVHighlights). We will clarify this detail in the final version.\n\n## Q5: TimeZero discrepancy.\n\nWe thank the reviewer for noting this discrepancy. We had already noted the May 26th update and will revise the TimeZero results accordingly, clarifying the cited version in the final revision.\n\n\n## Formatting concerns.\n\nThank you for pointing this out. We will fix the placeholders and move all table captions above the tables.\n\n\n### **Please feel free to leave additional comments if you have further questions or suggestions.**"}]

Author rebuttal:

Meta-review: All reviewers recommend acceptance with ratings (4, 4, 5, 5). This work presents TempSamp-R1, a reinforcement learning framework designed to enhance the performance of multimodal large language models in video temporal localization tasks. Reviewers acknowledged the novelty of the proposed method, stable training mechanism, and strong empirical results. Initial concerns regarding specific ablations and the scope of the evaluation were well addressed by the authors in the rebuttal, which included new experiments that further strengthened the paper.  The AC panel recommends accept.
Decision: Accept (poster)