Title: TempSamp-R1: Effective Temporal Sampling with Reinforcement Fine-Tuning for Video LLMs
Venue: NeurIPS 2025
Accepted: yes
Keywords: Temporal Grounding; Multimodal Large Language Model; Reinforcement Fine-Tuning
Abstract: This paper introduces TempSamp-R1, a new reinforcement fine-tuning framework designed to improve the effectiveness of adapting multimodal large language models (MLLMs) to video temporal grounding tasks. We reveal that existing reinforcement learning methods, such as Group Relative Policy Optimization (GRPO), rely on on-policy sampling for policy updates. However, in tasks with large temporal search spaces, this strategy becomes both inefficient and limited in performance, as it often fails to identify temporally accurate solutions. To address this limitation, TempSamp-R1 leverages ground-truth annotations as off-policy supervision to provide temporally precise guidance, effectively compensating for the sparsity and misalignment in on-policy solutions. To further stabilize training and reduce variance in reward-based updates, TempSamp-R1 provides a non-linear soft advantage computation method that dynamically reshapes the reward feedback via an asymmetric transformation. By employing a hybrid Chain-of-Thought (CoT) training paradigm, TempSamp-R1 optimizes a single unified model to support both CoT and non-CoT inference modes, enabling efficient handling of queries with varying reasoning complexity. Experimental results demonstrate that TempSamp-R1 outperforms GRPO-based baselines, establishing new state-of-the-art performance on benchmark datasets:Charades-STA (R1\@0.7: 52.9\%, +**2.7**\%), ActivityNet Captions (R1\@0.5: 56.0\%, +**5.3**\%), and QVHighlights (mAP: 30.0\%, +**3.0**\%). Moreover,  TempSamp-R1 shows robust few-shot generalization capabilities under limited data. Code is available at https://github.com/HVision-NKU/TempSamp-R1.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes a method based on Group Relative Policy Optimization (GRPO) to fine-tune multimodal large language models (MLLMs ) on video grounding tasks. On top of GRPO, which has already been proven beneficial for video understanding and grounding tasks, this method introduces off-policy guidance, which further stabilizes the optimization process. The proposed method is evaluated on three dataset and tested against state-of-the-art models, both GRPO-based and using standard fine-tuning.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:** The paper proposes a method using  Group Relative Policy Optimization (GRPO) to fine-tune multimodal large language models (MLLMs ) on video grounding tasks, i.e., video temporal video grounding and highlight detection. This method replaces random on-policy sampling with off-policy solutions, reducing reward sparsity. The proposed approach is tested on 3 datasets. The idea follows the recent trend of using GRPO for fine-tuning MLLMs for tasks requiring temporal understanding. \n\n**Weaknesses:** While the approach obtains higher results on the baselines and previous GRPO-based approaches, it is tested on one model only (Qwen2.5-VL-7B) and shares part of the methodology with previous works (e.g., GRPO for VTG task, iou reward, format reward, \\<think\\>). As I detail in the question section below, I believe some numbers are missing from the reported tables, which weakens the overall evaluation.\n\n**Minor Weaknesses:** There are three **methodname** left in the paper by mistake (L274, L276, L279).\n\nQUESTIONS:\n**Q1:** **Table 1 is missing several important numbers and metrics**. While I understand that some data may not be available in the original papers, the table loses valuable information as it currently stands. For most of the models, certain metrics have been reported; therefore, when available, a trained model can be evaluated to provide the missing values. For example, if TempSamp-R1 can not be evaluated on other RL-based approaches on QVHighlights, then the comparison on this dataset is weak. \nAdditionally, I do not understand whether the *Qwen2.5-VL-7B* numbers refer to a zero-shot evaluation of the model or if they involve some level of fine-tuning. If they do not (as I assume from the -), can you explain why these numbers differ from those presented in the original paper [3]?\n\n**Q2:** Why are the **SFT and GRPO baselines** missing from Table 1? I see results in the ablation studies and the OOD experiments, but I think it is important to compare the differences across all datasets considered. \n\n**Q3:** It is not clear to me the real **contribution** of this paper. It seems that most of the mechanisms used were already used in previous work (GRPO for VTG task, iou reward, format reward, \\<think\\>). Is the main contribution only replacing random on-policy sampling with off-policy solutions to alleviate reward sparsity? The value of this, if I am not mistaken, is shown only in Table 4 and for one dataset (see Q2). The authors should elaborate on this to make their contributions stand out. \n\n**Q4:** Does the **first stage of training** use in-domain data or a different dataset? I think this information is missing in the current version of the paper. \n\n**Q5:** This is a **comment** rather than a question. I noticed that the **TimeZero numbers** you reported do not align with the current version of the paper you cited, which was updated on May 26th. I understand that this is concurrent work; if you choose to cite it and include it in your comparison, you should update the numbers accordingly.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces TempSamp-R1, a reinforcement fine-tuning framework that enhances the temporal grounding capabilities of multimodal large language models (MLLMs) for video understanding. It addresses the inefficiencies of prior methods like GRPO by incorporating off-policy supervision (i.e., ground-truth annotations) and non-linear soft advantage estimation to improve stability and exploration during training. The authors further integrate Chain-of-Thought (CoT) prompting into a unified model to support both CoT and non-CoT inference. TempSamp-R1 achieves state-of-the-art results on Charades-STA, ActivityNet Captions, and QVHighlights, and demonstrates strong performance in few-shot and cross-domain settings.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1.\tThe paper critically analyzes why GRPO struggles with sparse rewards and unstable updates in large temporal search spaces. Based on this, a novel method is proposed to incorporate groud-truth annotations during RL training. This method is strengthen by a non-linear reward shaping to stabilizes the training.\n\n2.\tThe proposed method achieves generally better performance on multiple temporal grounding benchmarks. Futhur, the method demonstrates advantage over GRPO on cross-domain generalization and few-shot experiments. Extensive analysis and ablation studies also facilitate the intake of proposed method.\n\n3.\tThe writing is generally great by demonstrating the intuition and understanding behind the proposed method.\n\nWeaknesses:\n\n1.\tQuestion about off-policy output: As shown in Figure 2, during training, the model should be forced to output the ground-truth prediction so that the model weights can be updated with respective to this prediction. How was the model forced to do that? \n\n2.\tQuestion about Eq. 4: Can authors justify the design of this function? Specifically, why it has to be the format as Eq. 4 rather than other alternatives? Is it simply the best formulation from a few empirical observations?\n\n3.\tTempSamp-R1 (CoT): In L180-183, the model is trained in two stages where both stages are RL training. Since there is no CoT annotations used, how can we expect the model to output reliable CoT in the second stage? \n\n4.\tTempSamp-R1 Mixed CoT: In L234-235, how were the “better output” selected from two models? Did authors simply choose one of the two outputs that achieve higher scores of grounding metrics?\n\n5.\tTable 3: The proposed method requires much less time than GRPO. How does the proposed method save training time?\n\n6.\tFigure 3: How was the skewness score computed?\n\n7.\tThe necessity of CoT: Even though CoT achieve better results (Table 6: +0.6 mIoU between row 5 and row 8), it incurs much more computation during inference. Is it necessary for temporal grounding, given that the improvement is not significant?\n\nQUESTIONS:\nSee Weakness.\n\nLIMITATIONS:\nSee Weakness.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-19"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a reinforcement fine-tuning framework to improve multimodal large language models (MLLMs) for video temporal grounding. Due to the vast search space for time intervals in this task, methods like GRPO can be inefficient. To address this, TempSamp-R1 incorporates ground-truth annotations as off-policy supervision, adding one off-policy solution alongside on-policy samples per query. Additionally, to prevent instability from directly using off-policy rewards, the paper proposes three strategies—reward downscaling, advantage anchoring, and non-linear reward shaping—to stabilize training. Experimental results in video temporal grounding benchmarks validate improvements over GRPO-based baselines.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\nS1. **Clear Presentation:** The paper is well-written and easy to follow, with clear explanations that make the context easy to understand.\n\nS2. **Clear Motivation:** The motivation is well-argued. As shown in Figure 4, while GRPO’s ability to enrich the search space with on-policy solutions is beneficial, it can also hinder the model's training stability. This is reasonable, given that pretrained MLLMs may not have sufficient knowledge of temporal grounding tasks, making it hard to generate helpful signals, especially at the beginning of training.\n\nS3. **Performance**: Experimental results (Table 1) shows that the complete pipeline clearly outperforms other SFT methods or RL methods using the same model in video temporal grounding tasks.\n\n\n\n## Weaknesses\n\nW1. **Need Clarification on Contribution (Reward design vs. CoT):** I believe the strength of GRPO in reasoning tasks lies in its ability to prompt MLLMs for additional thinking steps, even without explicitly teacher-forcing intermediate steps based on ground-truth. In this regard, I am curious about the difference between GRPO (CoT) and TempSamp-R1 (CoT). The ablation study only compares GRPO (non-CoT), TempSamp-R1 (non-CoT), and TempSamp-R1 (CoT). I wonder whether the observed improvements come primarily from the stabilized reward design, from the use of CoT itself, or the synergy of both.\n\nW2. **Heuristics on Parameters:** The specific non-linear function (Equation 4) and its parameters ($\\tau$ = 0.8, $\\alpha_1$ = 0.01, $\\alpha_2$ = 1) appear to be empirically chosen heuristics. While it is common practice to carefully select hyper-parameters for stable RL fine-tuning, I am curious about how sensitive the proposed design is to these parameters.\n\nQUESTIONS:\nOverall, it would be valuable for this paper to provide more experimental clarification on why the proposed method works well—whether it benefits from CoT, normalization, or other factors.\n\nQ1. [See W1]\n\nQ2. [See W2]\n\nQ3. **Limited Exploration vs. Stable Convergence Trade-off:** The reduced variance in Top-1 IoU rewards for TempSamp-R1 in Figure 4 is presented as evidence of stability. However, could this tighter distribution also indicate reduced exploration of the solution space compared to the GRPO baseline? While the median reward is higher, is there a risk that TempSamp-R1 may converge to a local optimum, potentially limiting its ability to discover diverse or novel high-quality solutions?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents TempSamp-R1, a reinforcement learning framework aimed at improving the performance of multimodal large language models (MLLMs) in video temporal localization tasks. It addresses the inefficiencies of existing methods like GRPO by using ground-truth annotations for precise temporal guidance and introducing a nonlinear soft advantage mechanism to stabilize training. Experiments on benchmark datasets such as Charades-STA, ActivityNet Captions, and QVHighlights demonstrate that TempSamp-R1 outperforms GRPO-based approaches, particularly in few-shot scenarios.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- Innovative Framework Design: TempSamp-R1 introduces a hybrid policy training strategy that combines off-policy guidance (e.g., ground-truth annotations) with on-policy generation to offer more accurate temporal supervision. This helps resolve the common issues of sparsity and misalignment in on-policy sampling. The effectiveness has been thoroughly demonstrated in the experimental parts.\n- Stable Training Mechanism: By incorporating a nonlinear soft advantage estimation mechanism, TempSamp-R1 dynamically reshapes reward feedback, thereby reducing gradient variance and adjusting advantage bias. This facilitates robust exploration and convergence.\n- Strong Experimental Results and Thorough Analysis: Experiments on multiple benchmark datasets show that TempSamp-R1 outperforms existing GRPO-based methods and exhibits strong generalization capabilities in few-shot scenarios.\n- Clear Illustrations: The paper provides numerous figures, such as Fig. 3 and Fig.4 , which significantly improve both the readability the this paper and the interpretability of the proposed methods.\n\nWeakness\n- Both **advantage anchoring** and **non-linear reward shaping** are intuitive concepts. However, aside from a set of effectiveness experiments, the paper does not include ablation studies specifically targeting these two parameters.\n- The article has significant practical value, but its overall innovativeness is somewhat limited.\n\nQUESTIONS:\nOverall, I am particularly interested in the **advantage anchoring** and **non-linear reward shaping** methods regarding their generalizability across other tasks (besides Temporal Grounding) and the robustness of their hyperparameters. It remains to be seen whether these methods are universally important when combined with **mixed-policy sampling** or if they are only effective specifically for **temporal grounding**. However, this does not constitute a negative opinion of the paper.\n\nLIMITATIONS:\nyes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-05"}