Invert4TVG: A Temporal Video Grounding Framework with Inversion Tasks Preserving Action Understanding Ability
Authors: Chenzhaoyu
Hongnan Lin
Yongwei Nie
Fei Ma
Xuemiao Xu
Fei Yu
Chengjiang Long
Venue: ICLR 2026 Poster (2026)
Keywords: Video Understanding & Activity Analysis
Primary area: reinforcement learning

Abstract: Temporal Video Grounding (TVG) aims to localize video segments corresponding to a given textual query, which often describes human actions. However, we observe that current methods, usually optimizing for high temporal Intersection-over-Union (IoU), frequently struggle to accurately recognize or understand the underlying actions in both the video and query, thus reducing the effectiveness of these methods. To address this, we propose a novel TVG framework that integrates inversion-based TVG as auxiliary objectives to maintain the model's action understanding ability. We introduce three kinds of inversion TVG tasks derived from the original TVG annotations: (1) Verb Completion, predicting masked verbs (actions) in queries given video segments; (2) Action Recognition, identifying query-described actions; and (3) Video Description, generating descriptions containing query-relevant actions given video segments. These inversion tasks are entirely derived from the original TVG tasks and are probabilistically integrated with them within a reinforcement learning framework. By leveraging carefully designed reward functions, the model preserves its ability to understand actions, thereby improving the accuracy of temporal grounding. Experiments show our method outperforms state-of-the-art approaches, achieving a 7.1% improvement in R1@0.7 on Charades-STA for a 3B model.

Published as a conference paper at ICLR 2026
INVERT4TVG: A TEMPORAL VIDEO GROUNDING FRAMEWORK WITH INVERSION TASKS PRESERVING ACTION UNDERSTANDING ABILITY
Zhaoyu Chen Hongnan Lin Yongwei Niea. Fei Ma Xuemiao Xu² Fei Yu² Chengjiang Long
Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ), Shenzhen, China
School of Computer Science and Engineering, South China University of Technology, Guangzhou, China 3 DayteDance Inc., San Jose, CA 95110, USA
Other RL based LVLMs may not have good understanding of complex actions
Action Understadding
Temporal Grounding
Video description: man unbuttons his shirt, takes it off, puts different butlons up.
VD/Ace
Quory: person putting clothes and fastening the buttons.
Ground Truth
11:5s
AC/
AR/Ace
VideoChat-RI
VideoChat-RI Time-Ii R1 unable discriminate buttoning unbuttoning.
Time- RI
IGvert4IVG (ours)
6.5x
11.35
RI@0.
RI@0.3
Time-11: -hhink-Thee action of putting on clothes occurs when the person's ' hand arm the button. </think>
when person puiling on clothes and fustening the intlooes The person is seen picking up clothes from bed, putting on,
Invert4IVG (ours): <think- The task identify the precuse
Rigg0.
faxtening the </think>
Buseline
Time-RI
Inven4TVG
Figure 1: Left side: A specific example of temporal video grounding. According to the model's reasoning process,it can be seen that our method achieves better understanding of actions in the video compared VideoChat-RI and Time-RI. Right side: statistical results demonstrating that Time-R1, which is optimized solely for the IoU loss, reduces action understanding accuracy (where VC, AR, and VD are the proposed three auxiliary inversion TVG tasks measuring multi gulliigranularity action understanding ability). By introducing Inversion-TVG tasks, our method preserves action understanding ability and thus boosts TVG ability (as shown R1 @0.3, RI @0.5, and RI @0.7). Baseline is Qwcn-2.5-VL-3B.
ABSTRACT
Temporal Video Grounding (TVG) aims to localize video segments correspond- ing to given textual query, which often describes human actions. However, we observe that current methods, usually optimizing for high temporal Intersection- over-Union (IoU), frequently struggle to accurately recognize or understand the underlying actions in both the video and query, thus reducing the effectiveness of these methods. To address this, we propose novel TVG framework that inte- grates inversion-basee TVG as auxiliary objectives to maintain the model's action understanding ability. We introduce three kinds of inversion TVG tasks derived from the original TVG annotations: (1) Verb Completion, predicting masked verbs (actions) in queries given video segments; (2) Action Recognition, identifying query-described actions; and (3) Video Description, generating descriptions con- taining query-relevant actions given video segments. These inversion tasks
"qqual contribution.
Corresponding authors: nicyongwei@ @ wwu.c.ccuuccn mafci @ gml.ac.cn Thee work completed when Dr. Chengjiang Long was Meta.

---

Published as a conference paper at ICLR 2026
entirely derived from the original TVG tasks and are probabilistically integrated with them within reinforcement learning framework. By leveraging carefully designed reward functions, the model preserves its ability understand actions, thereby improving the accuracy of temporal grounding. Experiments show our method outperforms state-of-the-art approaches, achieving 7.1 % improvement in RI @0.7 on Charades-STA for a 3B model.
INTRODUCTION
Temporal Video Grounding (TVG) is crucial for long-form video understanding (Gaidon et al., 2013;
Laptev & Pérez, 2007; Darrell & Pentland, 1993). It localizes a video segment matching a textual query (Gao al., 2017; Zhang et al., 2023), enabling applications like video-text retrieval (Zhang et al., 2024) and UAV positioning (Ju et al., 2024).
Existing TVG approaches fall into three paradigms: (1) Traditional methods using hand-crafted features, sliding windows, and DETe-like networks (Shi et al., 2022; Gordeev et al., 2024); (2) LVLMs (Bai et al., 2025; Li et al., 2023) that regress segment duration via pretraining; and (3) RL- finetured LVLMs, such as Time-R1 I (Wang et al., 2025a), using Reinforcement Learning (RI.) with format rewards for structured reasoning and loU rewards for alignment.
Although significant progress has been made, we still find wrong grounding results in existing SOTA methods, and most of these wrong cases stem from incorrect action understanding. Figure left shows one case. In the video, man unbuttons his shirt, takes it off, puts on another one, and then buttons it up. The query is "A person putting on clothes and fastening the buttons", which requires localizing the actions "putting" and "fastening". Both VidcoChat-PP and Time-R1 notice hand touching a button and localize the action as "buttoning" rather than "unbuttoning", indicating that they seem focus only on the button itself without distinguishing between buttoning and unbutton- ing. We conjecture that these wrong groundings occur because video grounding models are generally optimized only for IoU, Although IoU is improved, this comes at the cost of reduced action under- standing capability, which in turn limits their overall vidco grounding performance. Figure right demonstrates this statistically. Time-R1 R1 (Wang et al., 2025a), optimized for IoU, shows improve- ment on TVG metrics (e.g., R1 @0.3/R1 @0.5/R @0.7) compared to the baseline Dwen2.55VL-33 However, exhibits degradation in action understanding tasks (VC/VD/AR), which ultimately ham- pers its TVG accuracy.
key insight of this work that training a TVG model effectively requires jointly training auxiliary tasks to preserve the model's 's action understanding capability. A naive approach to achieving this would be to train the TVG task alongside general action understanding tasks such as action recog- irnetecctioocllasffcaatiio
However, these general tasks are not specifically designed for the temporal video grounding objective, and the understanding learned from them may not align well with the precise temporal localization required in TVG.
Unlike general action understanding tasks, we design action understanding tasks that are specifically tailored for the temporal video grounding task. Specifically, by inverting the input and output of the original TVG task, we convert the localization task into understanding task, obtaining a set of Invert- TVG tasks. key advantage of these Invert-TVG tasks, compared to general action understanding tasks, is that they share the same training data as the original TVG task. On the same video-query data, our method performs both video localization (via the original TVG task) and action under- standing (via the Invert-TVG tasks). This tight coupling enables the learned action understanding be directly aligned with and supportive of the temporal grounding objective, resulting in more effective and synergistic learning.
Specifically, given video and a natural language query, the original TVG task predicts the temporal segment duration where the action occurs. Inversely, given video segment, the proposed Invert- TVG tasks infer the action-related information defined in the query from the given segment. We introduce three Invert-TVG tasks: (1) Verb Completion (VC): mask verbs (actions) in the query and then infer the verbs from video segments. (2) Action Recognition (AR): classify the actions in a given video segment where the ground-truth action is in the query. (3) Video Description (VD): generate descriptions for a given video segment, and the deseriptions should contain actions provided in the query.
2

---

Published as a conference paper at ICLR 2026
With the well-defined Invert-TVG tasks, then propose reinforcement learning framework that optimizes TVG and Invert-TVG tasks together. However, for large-scale models (c.g. 3B/7B param- eters), simultaneously optimizing multiple objectives incurs substantial memory overhead. More- over, the TVG and Invert-TVG tasks are conflict: the ground-truth video segment that TVG is re- quired to produce is precisely the input TVG task may ask for is exactly the input to the TVG task. To address this, we adopt an alternating optimization strategy, executing TVG and Invert-TVG tasks interleavingly. Besides, since temporal video grounding is our main objective while action understanding is an auxiliary objective, we op- timize the TVG task with higher probability while using lower probability for the Invert-TVG tasks.
Invert-TVGG task, and the original query that Invert-
Our contributions include:
We identify action understanding degradation in TVG from loU over-optimization, and address the problem via inversion TVG tasks.
We design three inversion TVG tasks which are self-supervised tasks re-purposing TVG annotations for action understanding, including verb completion, action recognition, and video description.
We propose reinforcement learning framework dynamically balancing TVG and Invert- TVG tasks, ensuring robust grounding and understanding.
Temporal Video Grounding. Temporal Video Grounding (TVG) (Gao et al., 2017; Hendricks et al., 2017) localizes specific segments in untrimmed videos based natural language queries. Recent methods fall into two categories: feature-based and frame-based LVLM approaches. Feature-base methods (Carreira & Zisserman, 2017; Lin et al., 2022) extract video and text features using pre- trained encoders, then predict timestamps via multimodal fusion. These rely heavily on feature qual- ity, limiting performance. Frame-based LVLM methods have recently gained traction for their strong generalization capabilities. For instance, NumPro (Wu et al., 2025) introduces a frame-numbering mechanism akin flipping a manga for efficient temporal grounding. while TimeSuite (Zeng al., 2024) employs grounded tuning to enhance Large Language Models (LLMs) for long-form video understanding. While methods like these and others (Li et al., 2024; Ren al., 2024) utilize super- vised fine-tuning to generate event sequences, they can still struggle with precise boundary detection on benchmarks like Charades-STI compared to specialized feature-based approaches. To address this, Time-R1 (Wang al., 2025a) employs reinforcement learning (RL) with IoU rewards, achiev- ing state-of-the-art TVG performance. However, its focus on temporal metrics neglects semantic alignment, constraining long-form video understanding
RL in LVLMs. RL has advanced post-training of LVLMs through Reinforcement Learning with Human Feedback (RLHF) (Ouyang et al., 2022: Yu et al., 2024) and Reinforcement Learning with Verifiable Reward (RIVR) (DeepSeek-AI, 2025; Chen et al., 2025). RLHF aligns models with hu- man preferences, improving tasks like image captioning, while RLVR enhances deterministic tasks like visual grounding (Liu et al., 2025). However, RL applications in long-form form video tasks re- main underexplored due to temporal complexity and semantic challenges. TimeZero (Wang al., 2025b), Time-RI (Wang et al., 2025a), VideoChat-RI (Li et al., 2025) apply RL to TVG but over- looks semantic understanding degradation from IoU-focused rewards. Our Invert4TVG framework addresses this by repurposing TVG data into self eeffsuuper tasks, enhancing action semantic un- derstanding and surpassing traditional RL limitations in video grounding.
2
METHOD
The TVG task aims to temporally localize video segments within long-form videos based on natural language queries. Given a video V. and a language query q, the goal is identify the temporal boundaries = [ta, te] of the segment of V that best corresponds to g, where ta te E RRt, The formal definition of the TVG task as follows:
TVG(V. 4) +
In this work, we introduce Invert4TVG, a framework designed to harness the potential Large Vision-Language Model (LVLM) for the TVG task using Reinforcement Learning (RL.) combined
3

---

Published as a conference paper at ICLR 2026
Temporal Video Grounding (TVG)
Inversion Task Verb Completion
Prompt
Query
Ground Truth Segment
Completion Prompt
Pinpoint the event Person closed door in video.
Person closed the dovr.
verb for event 'Person door based on the video.
Invert Imput and Output
LVLM
LVLM
Iall reward
Inmert mevart (VC)
Grounded Segment
Query-Related Related Output Person [openj the ahour:
2:2x
Ground Truth Segment
Action Recognition Prompt
Ground Truth Segment
Video Description Prompt
Use verb to descrihe the event bosed on the video.
Descrvbe what people have done based on the video.
LVLM
LVLM
Inivet imaard
The picuos nackia
Query-Related Output "open"
Query-Relate Rerlted Output
The person reaching for the doon open and
Yal
Prosecaion dock
Inversion Task Action Recognition
Inversion Task Video Description
We propose three Invert-TVG tasks. By partially reversing the inputs and outputs of the TVG task we obtain Verb Completion, Action Recognition and Video Description, which reuse the original TVG dataset by taking ground truth video segments input to reconstruct the target query related actions. The prompts for the three invert-TVG tasks are not identical. For VC, the verb in the query removed, and the model required complete and fill in this verb. AR asks the model to directly estimate the verb in the video. VD requires the model to describe the video content containing action verbs in the query.
Figure 2:
with TVG-inversion tasks. The Invert-TVG task is defined as (where y' denotes query-related con- tent):
IVV +
Our approach is fundamentally reinforcement learning algorithm that fine-tunes LVLMs (specif- ically the Qwen2.5-VE VL model series) by integrating both TVG and Invert-TVG tasks. In the fol- lowing, we first introduce the fundamentals GRPO (i.e., Group Relative Policy Optimization, a reinforcement learning algorithm proposed in (DeepSeek-AI, 2025)). Next, we introduce the pro- posed inversion TVG tasks, together with reward functions used to train the TVG and Invert-TVG tasks, Finally, we introduce our Invert4TVG reinforcement learning framework.
2.1 PRELIMINARY OF GROUP RELATIVE POLICY OPTIMIZATION
DeepSeek-RR (DeepSeek-AL) 2025), an early R I-ttly open-source L.M.M, uses GRPO to train policy Ty for reasoning before answers. For query it generates responses 01, 01,..., oG with score with r(.), and maximizes:
r ro:)) mean (o:))_))
(ox) Tul (ox) std (o:))
R(o)
(3)
where Th (ox) is generation probability, maw is prior state. The full objective with KL is:
max E EorTea (p) [R(o) - 3DkL (me [Ttt))
(4)
where B is scaling coefficient. We omit the clipping operation for simplicity.
4

---

Published as a conference paper at ICLR 2026
2.2 INVERT-TVG TASKS AND REWARD FUNCTIONS
In temporal video grounding, the accuracy of model inference highly depends on its understanding of both the video V and the query q. Therefore, do not rely solely on the IoU reward but also introduce additional rewards to keep or even enhance the model's action understanding ability. To this end, as illustrated Figure we design three Invert-TVG tasks and define reward functions measuring how these tasks are fulfilled. Thanks to advances in NLP. a wealth of mature linguistic toolkits (e.g., SpaCy) can now effortlessly convert verbs into their various tenses or even bring them back to the root form, making it feasible for us to compute reward values for the following inversion tasks.
Verb Completion (fine granularity). Verb Completion is task that masks verbs in the query and asks the model to recover the verbs from the ground truth video segment. For example, the original query is "Person closed the door", the masked sentence is "Person [] the door". The prompt is "Add verb for the event 'Person| [] the door' based on the video". As long as the model outputs a sentence successfully recovering the verbs of the ground truth, reward is given. For instance, if the output sentence is "Person [closes] the door" a full reward is given. Due to the randomness in the model's 's output, we are primarily concerned with whether the model comprehends the actions within the relevant segments. Therefore, we treat verbs in different tenses as equivalent using SpaCy. The reward function is as follows:
(o) = { 0 SpaCy(Upred * SpaCy(Ugt)
(5)
SpaCy (Upredd) = SpaCy(Ugt)
where Upred represents the predicted verb in the output sentence o, and Ugr represents the verb in the ground truth sentence. A full reward is obtained if the root form of the verbs are equal, where SpaCy(-) brings verb to its root form.
Action Recognition (middle granularity). Action Recognition task trains and preserves the model's action perception capability, and the output of the model is fixed single verb. We feed the model the ground truth video segment with prompt as "Use verb to describe the event based on the video", then compare the predicted verb against any verbs appearing in the ground truth query description. If the model's predicted verb is present in the reference sentence, it receives a full reward. For example, the model outputs "walk" and the ground-truth sentence "A person walks away and laughs", full reward is granted because "walk" occurs in the reference. As before, verb in different tenses are treated as equivalent. The reward function as follows:
(o) = {
0 SpaCy(Upred) & Sge (paCy(Pprod) Sgt
(6)
where Upred represents the predicted verb o, and Syt is the set of root-formed verbs in the ground truth query.
Video Description (coarse granularity). Video Description task is employed to train and maintain the model's holistic perception of events, yielding a complete segment level description. Specifi- cally, we feed the model the ground-truth video segment and prompt as "Describe what people have done based on the video". A full reward is granted as long ground-truth verbs appear in the output sentence of the model. For instance, the ground-truth verb is "jump" and the model produces "A person jumps and laughs" the reward is awarded because "jump" is present. The reward function is as follows:
= { O SpaCy(Ug) & Spred SpaCy(Umt) E Spred
(o)
where Bye represents the ground truth verb, and Spred denotes the set of root-formed verbs in the sentence predicted by the model.
2.3 IOU AND FORMAT REWARD FUNCTIONS
For the TVG task, we mainly employ the IoU reward function. Besides, we introduce a Format reward to enforce the model to output the thinking process.
IoU Reward. As stated above. TVG aims at estimating the time interval in the video that is asso- ciated with the content of given textual query. We the Intersection over Union (IoU) (Yuan
5

---

Published as a conference paper at ICLR 2026
Alternating Training Between TVG and Invert-TVG Tasks
TVG Input
Viden
Invert- TVG
TVG
TVG
TVG
Proppt
Step
Step
Step
Step
Step
Training step
VG Input
LVLM
Video Clip
T 8%
Invert- (VG(AR)
Invert- TVG
TVG
Invert-Prompe
Step
Step
Step
Step
Step
High probability (80%)
Invert TVG
probability (20%).
equal probubility.
Overview of the proposed Invert4TVG framework. The LVLM dynamically chooses between TVG tasks and Invert-TVG tasks according to different probabilities. Whenever an Invert- TVG task is selected, one of the three variants VC, AR VD is chosen with equal probability.
Figure 3:
al., 2021) between the time interval predicted by the model and the ground-truth interval as the reward function. This reward function effectively describes the accuracy of the time interval pre- dicted by the model. Given predicted [4s, 4e] and ground-truth [V,, ()] segments, the IoU reward can be calculated as follows:
(o) = [la, 4e] [1 6l
where O and U denote set intersection and union operations. I. Ml
It,, te] U
Format Reward. Recently, Time-RI (Wang al., 2025a) and VideoChat-Ri (Li et al., 2025) em- ploy Format reward making the model explicitly output its thinking process before making predic- tions. Following them, we introduce a template-based reasoning reward that incentivizes the model to generate intermediate reasoning steps prior to providing answers. The format is as following: <answer> t, to te </answer>, The reward is formulated as:
<think> + * <
Fform (0) 0, o has wrong format
(9)
1 o has correct format
2.4 INVERT4TVG REINFORCEMENT LEARNING FRAMEWORK
While a joint training approach that processes all TVG and Invert-TVG tasks simultaneously might seem straightforward, this method suffers from several critical limitations: (1) Memory inefficiency: maintaining separate computation graphs for multiple tasks drastically increases GPU memory con- sumption; (2) Optimization conflict: gradient updates from different tasks may interfere with each other, especially when their loss landscapes are not aligned; (3) Training instability: the varying convergence rates of different tasks make it challenging balance their contributions; (4) Task bias: the model may prioritize easier tasks while neglecting others. These drawbacks motivate us to adopt the training paradigm illustrated in Figure 3.
We implement a probabilistic sampling strategy where each training iteration has a high probability (80% in default) executing the primary TVG task (using IoU and format rewards) and low prob- ability of performing an Invert-TVG task. When selecting Invert-TVe we uniformly sample among VC, AR and VD. This design ensures the model maintains core action understanding capabilities while primarily focusing on temporal grounding. The asymmetric probability distribution prevents the auxiliary tasks from overwhelming the main objective while still providing regular semantic reinforcement. Formally, the reward for training the TVG task is:
(o) Trormar (o) (o)
(10)
The reward used to train an Invert-TVe task is:
Finvert Finven.TVG (VG Tiormat (0) Fine (o),
(11)

---

Published as a conference paper at ICLR 2026
where Nlav is any of rvc, FAR: and TVD. The overall reward function is defined
r(o) (r)va + + (riven/Tva(O)
(12)
where the two coefficients a and take values {0. 13. with the constraint a + B Their joint probability distribution is defined as (where 0 S p S 1):
P
if (a, 3) (1,0),
P(a,8) B) = 1-p
(a, 3) = (0,
(13)
otherwise.
As mentioned above, p 0.8 is empirically determined parameter derived from experiments.
EXPERIMENTS
We now evaluate our Invert4TVG model on the task of temporal video grounding. Code is attached.
3.1 EXPERIMENTAL SETUP
Benchmarks. We test our model on three temporal video grounding datasets: (I) Charades-STA (Sigurdsson et al., 2016) contains 6,672 long videos capturing indoor human activities. The official split for the TVG task includes 12,408 clip-query pairs for training and 3,720 for testing. (2) Activ- ityNet (Heilbron et al., 2015) comprises 20K long videos with an average of 3.65 clip-query pairs per video. We use the standard dataset splits with 37,421 training, 17,505 validation. and 17,031 test samples. (3) We further evaluate on QvHighlight (Lei et al., 2021), a high-resolution set of 10,460 long YouTTube videos paired with 48k manually annotated clip-queries. To match Charades-STA and ActivityNet formats, multi-segmentt localizations are split into single-segment tasks, forming balanced benchmark for fine-grained temporal grounding.
Implementation Details. We implement our LVLM using the Qwen2.5-VL model (Bai et al., 2025) as the backbone. To balance efficiency and memory consumption, we sample video frames at 2 FPS and resize them, resulting in approximately 2.8 million pixels per video (c.g.. a 50- second video yields 100 frames of size
96 3). Our implementation utilizes SpaCy's core web sm-3. 3. 8.0 model (12MB) to extract verhs from sentences and transform them across different tenses. For optimization, we employ the AdamW optimizer (Loshchilov & Hut- ter, 2019) with the following parameters: B1 0.9, 82 0.999, 1 x 10- weight decay of 0.0, and a learning rate x 10-5, The training time per epoch is approximately 80 hours. To ensure reproducibility, supplementary materials.
code, configuration files, and execution scripts are available in the
Evaluation metries. For TVG, adopt the "R1 @m" evaluation protocol compare with state- of-the-art models, which computes the percentage of samples where the top- 1 predicted segment has an loU greater than threshold m, with m 0.3, 0.5, 0.7. For brevity, also adopt mloU, which calculates the average IoU on all testing data as an alternative metric.
3.2 COMPARISON WITH STATE-OF-THE-ARR APPROACHES
We compare Invert4TVG with state-of-the-an TVG methods, including both traditional video- language pre-training models (VLP), recent large video-language models fine-tuned via SFT and RL-based approaches.
Comparisons on the Charades-STA dataset with fine-tuning. shown Table 1, Invert4VVG surpasses not only VLP-based and SFT-bascl models but also outperforms RL-based approaches un- der identical conditions. For example, on Tharades-STA. the 7B variant of Invert4TVG achieves an R1 @0.7 of 51.4, exceeding TimeSuite (43.0), SnAG (46.2). and Time-R1 (50.1). 1). The improvements are more pronounced for the 3B variant. Across RI @0.3, RI @0.5, and R1 @0.7, Inven4TVV's 3B model outperforms the 3B version of Time-R1.
Comparisons on the ActivityNet and QvHighlight datasets zero-shot settings. As shown in Figure 4, in zero-shot settings, Invert4TVG's 3B and 7B variants outperform Time-R1 R1 @0.3,

---

Published as a conference paper at ICLR 2026
Table 1: Performance of temporal video grounding on Charades-STA. Methods labeled FT with V were fine-tuned on the Charades-STXX training set. Methods marked with were first pre-trained on extra TVG datasets' and then fine-tuned on the Charades-STA training set, while those without are only trained on Charades-STA. We compare our method against existing 3B, 7B open-source LVLM. We highlight our results and the best-performing baselines using bold and underlining for clear comparison.
Charades- -TAde
Type
Method
Size
@1@033
@0.5
R1@0.7
2D-TAN*
57.3
45.8
27.9
Moment-DETR*
65.8
52.1
30.6
VLP
EaTR*
68.4
44.9
SnAG*
64.6
46.2
VideoChat-Flass
7B
74.5
53.1
27.6
TRACE
7B
40.3
19.4
SFT
HawkEye*
7B
72.5
58.3
28.8
TimeSuite*
7B
79.4
67.1
43.0
Time-RI(3B)
3B
74.6
53.1
26.0
RL(3B)
Time-RI*(3))
3B
78.7
64.1
36,9
Invert4TVG (ours 3B)
3B
80.8
69.0
44.0
Time-RI (7B)
7B
78.1
60.8
35.3
RL(7B)
lime-R1*(71))
7B
82.8
72.2
50.1
Invert4TVG (ours 7B)
7B
83.0
72.5
51.4
Ye-Temporal, DiDeMo, QuerYD, InterVVid, HowTo100M datasets, our method is pretrained
on those datasets.
Evalustion Meteics (RT (Acuutic))
Ti:e-R1:3))
74.1
(7B)
657
08.6.8.8
55.
52.7
46
40.0
35
31.1
25
21.4
210021.
18.1
15
9.0 9.1
R16007
R1g005
R1@2033
R1@2077
R1
R160033
ActivityNet
Qvhighlight
Figure 4: Performance of temporal video grounding on Activity and QvHighlight, We compare method with Time-RI (the best-performing among previous methods). All the models are zero- shot tested.
@I@005, and RI @0.7 over ActivityNet. On QvHighlight, where we compare single-segment pre- dictions, Invert4TVG consistently outperforms Time-RI across RI @0.3, RI @00.5, and RI @0.7. ActivityNet contains only 200 action categories, whereas QvHighlight covers significantly larger and more diverse set, with far more complex scene-action correlations. This disparity underscores the superiority of our method in understanding intricate actions.
3.3 ABLATION STUDY
We conduct detailed ablation on the Invert4TVG-3B model investigate the contribution of the design strategies.
Using different combinations of Invert-TVG tasks versus employing them in combination. As shown in Table 2, using VC. AR, or VD alone instead of jointly yields lower performance. Only-VD which emphasizes contextual understanding, peaks RI @0.3 but falls short precise localization. Only-AR, focused on immediate actions, reaches the highest RI @0.7 of 43.8. Only- VC outputs are less random than Only-VID yet less specific than Only-AR, achieving the best R1 @0.5 (68.0). The
8

---

Published as a conference paper at ICLR 2026
mixed-task Invert4TVG surpasses all three individual tasks across all three metrics, demonstrating that joint training outperforms separate use.
The combination of VC and AR improves R1@0.7 to 43.8, outper- forming either task alone (VC: 42.0; AR: 43.8), indicating complemen- tary benefits between verb comple- tion and action recognition. The VC+VD pair achieves the high- est RI @13 (80.0) among two-task setups, suggesting that video de- scription aids verb-focused localiza- tion. Invent4VVG (integrating all three auxiliary tasks) achieves the best overall results @0.7: 44.0), demonstrating that multi-task syn- ergy is maximized when all compo- nents are jointly optimized.
Table 2: Ablation study using only TVG (Time-RI), VC (verb Completion), AR (action recognition), VD (video de-
scription), and their mixed usage.
Charades-STr RI 0.5
Type
Method
RI @033
R1 @00.7
Only-TVG
78.7
64.1
36.9
Only-VD
79.1
64.3
39.4
Only-AR
78.2
65.2
43.8
Only-VC
78.8
68.0
42.0
RL
AR+VD
79.6
67.9
43.6
VC+AR
78.8
68.1
43.8
VC+VD
80.0
68.5
42.1
Invert4TVG
80.8
69.0
44.0
90
79.4
R1g00.3
80
R1600.
67.6
78.0
R1g007
70
R1 Accuracy (96
60
64.6
50
41.3
37.5
21.0
19.0
11.5
100%
Invert-TVG Task Probability (%)
Figure 5: The RI accuracy curves. Blue, orange, and green show how the three R1 metrics evolve the Invert-TVG task probability (1 p) gradually increases.
Exploiting different probabilities for the TVG and Invert-TVG tasks. As shown in Figure 5, Varying the task probability markedly alters training outcomes. At 20% Invert-TVG task probability, the model performs best, raising RI 200. from 36.9 (no Invert task) to 44.0. As the Invert-TVG task probability grows, the model increasingly emphasizes action recognition while neglecting temporal grounding. Between 60% and 80% Invert-TVG, temporal video grounding performance steadily declines, falling below the pure TVG baseline. When Invert-TVG probability reaches 100 %, the model performs only the Invert- TVG task and yields the worst results, According the experiments, we choose set p 0.8 in Eq. 13.
Binary Invert-TVG rewards sine similarity-based rewards. As shown Table 3, we observe that employing a simple binary Invert- TVG reward (0 1) during train- ing yields superior outcomes com- pared to more intricate reward mech- anisms. When training for the same Binary or
Table 3: The results using binary Invert-TVG reward or co- sine similarity reward for training.
Reward
RI @0.3
RI@0.5
R1@0.7
Cosine Similarity
76.2
62.2
39,8
80.8
69.0
44.0
two epochs, the employed Invert- TVG Reward outperforms the cosine similarity reward across all three evaluation metries (R1 @0.3, RI @0.5, RI @0.7). This advantage stems from the controllability and stability of the binary reward design, whereas cosine similarity introduces higher variance and optimization instability. For

---

Published as a conference paper at ICLR 2026
ample, in our implementation, "run" and "eat" yield cosine similarity of 0.2 despite their weak semantic link. Therefore, binary Invert-TVG reward a better choice.
4 CONCLUSION
In this work, we present Invert4TVG, an approach that introduces Invert-TVG tasks, requiring the model to generate query-related -relted content from video and its ground-truthe temporal segment. We de- sign three variants of Invert-TVG, including verb completion, action recognition, and video descrip- tion. These tasks encourage the model to retain and enhance its action understanding capabilities. We develop a Invert4TVG RL framework that jointly optimizes TVG and Invert-TVG tasks, In addi- tion standard IoU and format rewards, we introduce Invert-TVG rewards promote performance on Invert-TVe tasks. During training, the model primarily performs TVG at a high probability, while intermittently switching to Invert-TVG tasks lower probability. This balanced strategy ensures robust temporal localization while preserving semantic action-verb alignment. Our work bridges TVG-LVIMM gap, unlocking higher extensions in traditional tasks. Experiments demon- strate the effectiveness of our method over existing approaches, achieving significant improvements of grounding accuracy. The reasoning process also shows that the proposed method indeed under- stands actions better than compared approaches.
ACKNOWLEDGMENTS
This research financially supported by the Open Research Fund from Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ) under Grant No. GML-KF-24-11 Guangdong Provincial Basic and Applied Basic Research Fund under Grant No. 2025A151501 1884, and Fun- damental Research Funds for the Central Universities under Grant No. 2024ZY
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
This study exclusively utilizes publicly available open-source models and datasets; no proprictary or sensitive information is involved, and all data are free of personally identifiable content. We have strictly followed the corresponding licenses and usage guidelines. Although the present work poses no apparent ethical risks, we caution that-like like many machine learning models- its outputs could be misapplied in unforeseen contexts. We therefore advocate responsible use and encourage ongoing efforts to identify and mitigate potential biases inherent in open-source datasets.
REPRODUCIBILITY STATEMENT
We are committed to ensuring the reproducibility of our work, The models, training datasets, prompts, and hyperparameters used in our experiments are fully documented in Section 4.1 and Appendix C. These descriptions should allow researchers to replicate our experimental setup an- dresults without requiring additional resources beyond those specifed.
REFERENCES
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, and Song. Qwen2.5-VV VL technical report, 2025.
Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 629966996666 6308, Honolulu, HI, 2017. IEEE.
Liang Chen, Lei Li, Haozhe Zhao, Yifan Song, and Vinci. RI-V: Reinforcing super generalization ability in lisoun-lagguage models with less than S3, 2025. URL httpps://gititii com/ Peep-Agent// /R1-V.
Trevor Darrell and Alex Pentland. Space-Time gestures. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 315-333---00 New York, NY, USA, June 1993. IEEE.
DeepSeek-AI. DeepSeek-RR R1: Incentivizing reasoning capability in LLMs via reinforcement learn- ing, 2025. URL https: /ttp//xxvvvvve//e///ee///aaa orooor////aoo.o. abs /tt.//211..........1..111 12948.
Adrien Gaidon, Zaid Harchaoui, and Cordelia Schmid. Temporal localization of actions with Ac- toms. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(11))2782-277) 2013. doi: 2..1009/PAAMI.2033.55
Jiyang Gao, Chen Sun, Zhenheng Yang, and Ram Nevatia. TALL: Temporal activity localization via language query. In Proceedings of the IEEE International Conference on Computer Vision, pp. 5267-5275, 2017.
Aleksandr Gordeev, Vladimir Dokholyan, Irina Tolstykh, and Maksim Kuprashevich. Saliency- Guided DETR for moment retrieval and highlight detection, 2024.
Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. Activity Net: A large-scale video benchmark for human activity understanding. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 961-970, Boston, Mass., 2015.
IEEE.
Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan Russell. Localizing moments in video with natural language. In Proceedings of the IEEE International Conference on Computer Vision, pp. 5803-5812, Venice, Italy, 2017. IEEE.
Hao Ju, Shaofei Huang, Si Liu, and Zhedong Zheng. Video2BEV: Transforming drone videos to BEVs for video-based rco-localization, 2024.
Ivan Laptev and Patrick Pérez. Retrieving actions in movies. 2007 IEEE 1 1ith International Conference Computer Vision, pp. 1-8. IEEF, 2007.

---

Published as a conference paper at ICLR 2026
Jie Lei, Tamara L. Berg, and Mohit Bansal. QVHighlights: Detecting moments and highlights in videos via natural language queries. In Advances in Neural Information Processing Systems, volume 34, pp. 22611-12623, 2021.
Kunchang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang. and Yu Qiao. VideoChat: Chat-centric video understanding, 2023.
Xinhao Li, Yi Wang, Jiashuo Yu, Xiangyu Zeng, Yuhan Zhu, Haian Huang, Jianfei Gao, Kunchang Li. Yinan He, Chenting Wang, Yu Qiao, Yali Wang, and Limin Wang. Videochat-flash: Hierar- chical compression for long-context video modeling, 2024.
Xinhao Li, Ziang Yan, Desen Meng, Lu Dong, Xiangyu Zeng. Yinan He. Yali Wang, Yu Qiao, Yi Wang, and Limin Wang. VideoChat-R1: Enhancing spatio-temporal perception via reinforce- ment fine-tuning, 2025. https / a v. org/ abs 25504 06958.
Kevin Qinghong Lin, Jinpeng Wang, Mattia Soldan, Michael Wray, Rui Yan, Eric Xu, Difei Gao, Rong- Cong-Cheng Tu, Wenzhe Zhao, and Weijie Kong. Egocentric video-language pretraining. In Advances in Neural Information Processing Systems, volume 35, pp. 7575-7586, 2022.
Ziyu Liu, Zeyi Sun, Yuhang Zang, Xiaoyi Dong, Yuhang Cao, Haodong Duan, Dahua Lin, and Jiaqi Wang. Visual-RFT: Visual reinforcement fine-tuning, 2025.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference Learning Representations. OpenReview.net, 2019. URL https:// openreview net/ wor.mood-
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to fol- low instructions with human feedback. In Advances in Neural Information Processing Systems, volume 35, pp. 27744-27744. NeurIPS, 2022.
Shuhuai Ren, Linli Yao, Shicheng Li, Xu Sun, and Lu Hou. TimeChat: A time-sensitive multimodal large language model for long video understanding. In Proceedings of the TEFE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14313-14323, IEEE, 2024.
Dingfeng Shi, Yujie Zhong, Qiong Cao, Jing Zhang, Lin Ma, Jia Li, and Dacheng Tao. ReAct: Temporal action detection with relational queries. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 105-120. Springer, 2022. doi: 10. 1007/978 0-10778---0310000----
Gunnar A. Sigurdsson, Güil Varol, Xiaolong Wang. Ali Farhadi, Ivan Laptev, and Abhinav Gupta. Hollywood in homes: Crowdsourcing data collection for activity understanding. In Proceedings of the European Conference Computer Vision (ECCV), 2016.
Ye Wang, Ziheng Wang. Boshen Xu, Yang Du, Kejun Lin, Zihan Xiao, Zihao Yuc, Jianzhong Ju, Liang Zhang, Dingyi Yang, Xiangnan Fang, Zewen He, Zhenbo Luo, Wenxuan Wang. Junqi Lin, Jian Luan, and Qin Jin. Time-R1: Post-training large vision language model for temporal video grounding, 2025a. URL https: arxiv. org/ abs 25317.137777.
Ye Wang, Ziheng Wang, Boshen Xu, Yang Du, Kejun Lin, Zihan Xiao, Zihao Yue, Jianzhong Ju, Liang Zhang, Dingyi Yang. Xiangnan Fang, Zewen He, Zhenbo Luo, Wenxuan Wang, Junqi Lin, Jian Luan, and Qin Jin. TimeZero: Temporal video grounding with Reasoning- Guidedd LVLM, 2025b. URL https:://// /tte://aaxxae /aarxaivvvv.////////aaa/a/aaavva org/
13377v1.
Yongliang Wu, Xinting Hu, Yuyang Sun, Yizhou Zhou, Wenbo Zhu, Fengyun Rao, Bernt Schiele, and Xu Yang. Number it: Temporal grounding videos like flipping manga. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 13754-13765, 2025.
Tianyu Yu, Yuan Yao, Haoye Zhang, Taiwen He, Yifeng Han, Ganqu Cui, Jinyi Hu, Zhiyuan Liu, Hai-Tao Zheng. Maosong Sun, et al. RLHF-V: Towards trustworthy MLLMs via behav- ior alignment from fine-gree fine-grained correctional human feedback. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13807-13816 Seattle, WA, 2024. TEFE.
12

---

Published as a conference paper at ICLR 2026
Yitian Yuan, Xiaohan Lan, Xin Wang, Long Chen, Zhi Wang, and Wenwu Zhu. A closer look at tem- poral sentence grounding in videos: Dataset and metric. In Proceedings of the 2nd International Workshop on Human-Centrr Multimedia Analysis, pp. 13- 13-21, 2021.
Xiangyu Zeng, Kunchang Li, Chenting Wang, Xinhao Li, Tianxiang Jiang, Ziang Yan, Songze Li, Yansong Shi, Zhengrong Yue, Yi Wang, et al. Timesuite: Improving milms for long video understanding via grounded tuning. arXiv preprint arXiv:2410.19702 2024.
Gengyuan Zhang, Mang Ling Ada Fok, Jialu Ma, Yan Xia, Daniel Cremers, Philip Torr, Volker Tresp, and Jindong Gu. Localizing events in videos with multimodal queries, 2024.
Hao Zhang, Aixin Sun, Wei Jing, and Joey Tianyi Zhou. Temporal sentence grounding in videos: A survey and future directions. IEEE Transactions on Pattern Analysis and Machine Intelligence, (188):0043310488 2023. doi:
13

---

Published as a conference paper at ICLR 2026
A PRINCIPLE OF ALGORITHM FOR INVERT4TVG
Algorithm GRPO Training with Randomized Invert-TVG Task Selection
1: Input: Video V, query g, LVLM parameters 0, probability p. reward function FTvG (0). list of
Invert-TVG tasks TVV [Invert-TVG; J with corresponding rewards {ri J learning rate 7j.
2: Output: Optimized LVLM parameters 0 balancing localization accuracy and video-language alignment.
3: Define forward TVG task: TVG(V, +
4: Define a list of Invert-TVV tasks, where each (VVr)t-TGG;(,T)) +
5: while not converged do
6: Sample random value u ~ Uniform (0,
7: if' u S p then
8:
Select reward r (o) {Optimize parameters related to (localization accuracy)) Compute gradient Ver w.r.t parameters affecting T
9:
10:
else
11:
Randomly sample an Invert-TVG task Invert-TVG; ~ Timven.TVG {Select task from the list]
12:
Generate its output of + Invert-TVG.,V,7
13:
Select the corresponding reward ro( (Optimize for the sampled Invert-TVG task) Compute gradient Ver w.r.t. parameters affecting %
14:
15:
end if
16:
Update parameters: 0 + + V pr [Gradient ascent to maximize reward}
end while
18:
Result: The model retains both localization accuracy (via T) and diverse video-language align-
ment (via various gf), which are complementary.
B THEORETICAL JUSTIFICATION FOR MULTI-TASK RL IN INVERT4TVG
To demonstrate the advantages of incorporating the Invert-TVG task into the RL framework, provide theoretical analysis showing that the multi-task approach improves semantic alignment and generalization compared to single-task TVG training. We follow the Pareto optimality framework in multi-task reinforcement learning, adapted to our setting where the joint reward balances temporal localization and semantic fidelity.
Let T. denote the policy (LVLM), and D the data distribution over videos V, queries q, and ground- truth segments т. The single-task objective (TVG-only, as in prior works like Time-R1) maximizes:
max Bowns [R]va(o)] - 3xD (Tel(TTee)
(14)
where Rova(o) (o) + (o).
In our multi-task setting, we introduce the joint reward Ryomi Rivo(o) X A with A > 0 balancing the tasks. The objective becomes:
max Bo RRjiin (o)] DKI. (DRi((TTll)) (Tee))
(15)
Lemma 1 (Semantic Alignment Improvement). The Invert-TVG task minimizes semantic loss Lyem E pdd(( o 9),, where d(.,-) :) is a distance metric (e.g., verb matching or KL divergence on embeddings). Then, the joint loss satisfies Ligint Live + C for some constant C 0, as Rinven provides positive
feedbbbk feedback on semantic fidelity.
Proof. By Jensen's inequality and non non-negativity of Riveet 2 0 (binary rewards in our design), E|R 2 E[Rvva] + X min Ri 2 RTT... assuming Rinen 2 0. This implies the multi-task policy reduces semantic drift, as Invert rewards enforce alignment (e.g., verb recovery).
theorem 1 (Pareto Superiority). The multi-task policy Tipia is Pareto superior to the single-tast policy Tive if there exists 0 such that Rrva(To) Rova(Tivo)) and Rimer(To) 0.
14

---

Published as a conference paper at ICLR 2026
Proof. Consider the convex optimization formulation: minimize LIvG ALivven- Assuming Liavnt is convex (e.g., cross-entropy- semantic loss), the Pareto frontier dominates the single-task op- timum. The KL regularizer ensures the multi-task solution lies superior frontier, feedback from Invert reduces divergence: DKL. (Tjoii Ter) S regularization.
(et) for A > O from semantic
Corollary 1 (Generalization Bound), In migrating TVG LVLMs, semantic drift is reduced by Invert tasks, yielding a generalization error bound: Eriyiit S Errivo - 11,, where vi is a learning rate factor derived from multi-tast task boosting.
This analysis justifies the inclusion of Invert-TVG, showing improved alignment and generalization on the Pareto front.
C MORE IMPLEMENTATION DETAILS
We implement our model using the Qwen2.5 VL model as the backbone, selected for its robust fea- ture extraction capabilities video understanding tasks. To balance training efficiency and memory constraints, we sample video frames at 2 frames per second (FPS), adaptively resizing each frame to maintain approximately 2.8 million pixels per video. For example, a 50-second video yields 100 frames, each with a resolution of approximately 96 x 96 x pixels. During the reinforcement fine- tuning phase, we train the model for epochs with a batch size 4. All experiments are conducted cluster equipped with eight NVIDIA A100 GPUs (40GB memory each), using CUDA 11.8 and Python 3.10. For natural language processing tasks, we employ the
8 0
model from the SpaCy library (12MB) to extract verbs from sentences. Random numbers between and are generated using numpy. numpy.randou random. The model is optimized using the Adam W optimizer with parameters 31 0.9, B3 0.999, and weight decay of 0.0. The learning rate is set to 5 10 10-5 Training requires about 80 hours. All code. configurations, and preprocessing scripts are provided in the supplementary materials to ensure reproducibility.
Below, we provide detailed breakdown of the computational overhead introduced by the three auxiliary tasks (data processing, reward computation, and extra forward/backward passes), including concise cost tables.
TVG 65h
VD
VC
AR
Total
6.5h
3.5h
80h
As can be seen from the table, in our method, TVG overall consumed 65 hours (accounting for 81 81.25%), while Invert-TVG overall consumed 15 hours (accounting for 18. 18.75%), with VD, VC, and AR consuming 6.5 hours, 5 hours, and 3.5 hours, respectively. It is important to note that the Invert-TVC tasks were not used and only TVG tasks were employed, the total training time would still amount to 80 hours. The Invert-TVG task does not lead to an increase in training time, and our method just took the training time originally meant for the TVG task and used it for the Invert-TVG task.
Detailed time usage in the three auxiliary tasks per training step is shown in the following table:
Data Processing
Forward
Reward
Backward
VC Offline"
117ms
8lms
89ms
VD Offline
12lms
46ms
90ms
AR Offfine
119ms
10ms
90ms
"The data processing was fulfilled before the training.
Similarly, the VRAM costs are nearly identical to the baseline, with no additional overhead.
Model
VRAM costs
Qwen-vl-2.5-3B
146G
Our
142G
15

---

Published as a conference paper at ICLR 2026
Evaluation Metrics (mloU Accuracy)
85
Invert4TVG
75
Invert-SFT-Inveet4TVG
Qwen2.5-VL-3B
65
Invert-FFT-Q0we2255---B
57.9 57.9
55
45
35
25
18.0
17.8
15
5
--ven..--VL-3 with/witchout Invert-SFT
Invert4TVG
with/witchout Invert-SFT
Figure 6: After Invert-SFT, the mloU of Qwen2.5-VL-3B model and our Invert4TVG method compared with those without Invert SFT.
D ABLATION STUDY
Impact of Invert-SFT on Model Training. Invert-SFT refers to feeding ground-truth video clips into the model and requiring the model to output the corresponding event description based on these clips. The ground-truth clips directly cropped from annotated temporal segments in temporal video grounding datasets, while the "corresponding event" is the query to be localized. After Invert- SFT, the model's outputs become more stable, facilitating subsequent training for Invert tasks. As shown in Figure 6, for the Qwen2.5-VL-3B model initialized with Invert-SFT, the mean Intersection- over- ver-Union (mloU) slightly decreased from 18.0 to 17.8 initially. However, after sufficient training, both Invert4TVG models, with and without Invert-SFT, reached convergence, achieving identical mloU scores of 57.9.
E
QUALITATIVE RESULT
Other RL-based LVLMs cannot distinguish betweon close and open door
Query:
the person close the refrigerator door.
Ground Truth
7.1s
11.5s
VideoChat-R1
0.9s
7.0s
Time-R1
6.9s
11.39
Time-R1:thinl--T txthinko-Th action of closing the door occurs immediately after the person's a hand am touches the dlorikhh
Invert4TVG (otrr task identify precise moment person refrigerator door. The person seen opening the refrigerator door, placing items inside, and then closing door. <ithink>
Figure 7: success case 1
16

---

Published as a conference paper at ICLR 2026
Time localization of similar actions. As shown in Figure method identify similar actions, such as opening and closing doors. Many other models have insufficient understanding of similar actions, such as picking up and putting things in a box. In the model's judgment, it is likely to he classified as the same action because the model does not fully recognize the state in which the action occurs and continues. Our method help the model understand actions more deeply and distinguish similar actions.
accurately
Other RL based LVLMs may have good understanding complex events
Query: One person completely closes the door
Ground Truth
5.1s
9.6s
VideoChat-R1
0.0s
10.0s
Time-R1
0.0s
Inverl4TVVGoure)
4.8s
9.3a
Time-R1 :<think= The video provided do not match the described action of completely closes the </thikk
task to identify the precise moment when One person completely closes the door. One person stood in front the door and closed it front of the iscreen. </think>
Figure 8: success case 2
Time localization of complex actions. As shown in Figure 8, our model recognizes the comple- complex events, the time point need locate
tion of the door closing action better. For may occur at the completion of the action, rather than the beginning of the action. Other models sometimes consider the beginning of the action as the time point be located, and then proceed with subsequent positioning from this time point. Our model has clearer understanding of the start and end of the action and can effectively locate the time period when the action in such a state of completion. Meanwhile, correct understanding of the beginning and end of action is also helpful for contextual reasoning.
Our method is capable of handling situations involving multi-person
Query: person out
Ground Truch
4 29
VideoChat-R1
1.6a
0.0s
InVors)VVGouus)
Time-R1:4think-A person walked towards the door, opened it, and took out the task to identify precise moment when person took bottle. person bent down, opened the door, and took bottle. </think>
Figure 9: success case 3
Situations involving As shown in Figure 9, our method demonstrates a strong capa- bility in handling situations involving multiple persons. When two individuals appear in the frame,
17

---

Published as a conference paper at ICLR 2026
other models often susceptible interference from the secondary person, leading extended temporal localization periods. In contrast, our model possesses a more accurate understanding of actions, enabling it to precisely identify the action be localized and the state of the target person, thereby achieving superior results.
capable hundling involving non-human object
uguplani put
2.25
3.50
VilenChai-Rt
2.16
2.86
TimeR1
InveniTVSjours)
2.2:
8.4g
Time Sestiint-Sommmnt was quickly put into the fire. <tthine
(oourtddTG((ura)-ttiiii taek is to idencify the precise moment when eggplant put into fire, video shows that something, possiblyy eggplents, was put into the fire, <think>
Figure 10: success case 4
Situations involving non-human object. As shown Figure 10, our method capable of handling scenarios where person is present in the frame, such when only bonfire is shown and an eggplant is thrown into the fire. Other models, upon recognizing the keyword "fire," tend to predict very short temporal segments. Even if these predictions over Union (loU) remains low. contrast, our approach focuses on understanding the action itself, resulting in predicted segments that are longer and closer to the ground truth temporal annotations.
accurate in timing, their Intersection
Our model can handle person performing molt plle actions
QuerytA stand while coading book
Truth
VicoChat R1
Time-R1
0.0s
2.28
5.6
Time-ltt Time--R1:stiiniee A person reading book all time. </think>
Invert4TVal (ours)) info-The teek identily precise momentt when person stand up while reading book. person lying on the ground, then he stand up while reading book. <thinke
Figure 1 I: success case 5
person performing multiple actions. As shown Figure 11, the event be localized involves a person reading a book while standing up. Other models focus only on single action, namely "reading." whereas our model disambiguates the actions, accurately identifying both the reading and the simultaneous act of standing up.
person performing multiple actions. As shown in Figure 12, the query contains a temporal cue such as "and then." While other models treat the consecutive actions as single event and attend more to the earlier action, our model recognizes both actions and their temporal order, yielding more accurate localization.
person performing multiple actions. As shown in Figure 13, the event to be localized involves causal relationship: person in the video first sneezes and then takes medicine. Other models fail to accurately recognize the action of sneezing, leading them to rely on speculation and only localize the action of taking medicine. In contrast, model successfully identifies both sneezing and taking medicine, understands the causal relationship between them, and achieves superior temporal localization results.
18

---

Published as a conference paper at ICLR 2026
Our method is cupaile of handlingg autions with temporal sequence relationshins
Query: A person
and pousing wates
Ground Truis 13s
8.05
VileChai-R1
1.16
Time-1
itthe-i--tiiiott clear the
book book.<thinke
Invert4TVG (ours) <think-The lask to identify the precise moment when A reed book and then pouringg weler person then picking blenket pouring <ithik>>
Figure 12: success case 6
One moteet is cerebte ohroo actons with corel releteeniics
Quergy
VitesCraaltt
Rooet voccent
Timeo nohinia did't see snyore eneene. penson took our borte <ttinit>
InvertATVO Votoour tithee The took 3o aleniily the greaise moment A person sneezed sho soot some meediiiee person silting subicerly snooded socoulaed sho upe somnehinng.
mociknn insido
Figure 13: success case 7
Our method not sufficient semantic analysis
Query: person takes something from box again
Ground Truth
5.1s
6.6s
VideoChat-R1
1.1e
2.0s
TIme-R1
0.0s
2.18
Inverr)TVG(ours)
1.2s
2.08
task is identify the precise moment when A person takes something from box. person walked towards the box, put their hand in, and took out something, </think>
Figure 14: failure case
Deep understanding of event semanties. As shown in Figure 14, our method is not sensitive enough to some qualifiers, such as "again" representing the second occurrence of an action, which requires the model to accurately identify the action while also accurately finding the time period dur- ing which the second action occurred. Our method, as well as other models, has some shortcomings in this aspect. When locating time, may find the first occurrence time the final answer. This is because the model does not have a deep understanding of the meaning of qualifiers in the query and fully considers it when locating video time.
19

---

Published as a conference paper at ICLR 2026
F USE OF LARGE LANGUAGE MODELS
We used large language models (LLMs) limited and auxiliary manner during the preparation of this paper. Specifically, LLMs were employed improve the fluency and readability of the manuscript by polishing grammar and style, without altering the technical content. Importantly, LLMs were not involved in formulating research ideas, designing methods, conducting experiments, analyzing results, or drawing conclusions. All technical contributions of this paper are solely the work of the authors.
20


Reviews:
[{"review_id": "I0lI38PWmf", "reviewer": "Reviewer_tE1U", "summary": "This paper introduces Invert4TVG, a GRPO-based reinforcement learning framework for temporal video grounding that mitigates the loss of action understanding caused by optimizing solely for IoU. The core idea is to invert the TVG mapping via three auxiliary tasks—Verb Completion, Action Recognition, and Video Description—and interleave them with the main TVG objective through probabilistic scheduling (≈80% TVG, 20% Invert-TVG). On Charades-STA, Invert4TVG improves R1@0.7 by 7.1 points over Time-R1, demonstrating the benefit of coupling localization with semantics.", "questions": "Please refer to the Weaknesses", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) **Clear diagnosis and motivation.**\n   The paper convincingly shows that **IoU-centric optimization can erode action understanding**, with Figure 1 and accompanying evidence making the point concrete. The “buttoning vs. unbuttoning” example effectively illustrates the failure mode.\n\n2) **Strong empirical gains.**\n   The method reaches **44.0% R1@0.7 on Charades-STA**—a **+7.1** point improvement over Time-R1—and delivers **consistent improvements across Charades-STA, ActivityNet, and QVHighlights**. Notably, both the **3B** and **7B** variants benefit.\n\n3) **Practical auxiliary task design.**\n   The three **inversion tasks** (Verb Completion, Action Recognition, Video Description) **reuse existing TVG annotations** and require **no extra data collection**, making the approach pragmatic and easy to adopt.", "weaknesses": "1) **Incremental novelty via training reformulation.**\n   The main ingredients—GRPO-based RL, inversion-style auxiliary tasks, and template/format rewards—are adapted from existing ideas. The contribution lies primarily in **recasting TVG training** rather than introducing a fundamentally new algorithmic primitive.\n\n2) **Verb-centric semantics may be brittle.**\n   Reliance on **SpaCy verb lemmatization** for VC/AR/VD risks overlooking **non-verbal cues** (objects, states) and **nuanced modifiers** (e.g., adverbs, negation, ordinality), potentially limiting semantic coverage.\n\n3) **Unclear computational overhead.**\n   The paper does not quantify the **runtime/VRAM costs** introduced by the three auxiliary tasks (data processing, reward computation, extra forward/backward passes). A concise **cost table** and throughput metrics would clarify practicality.\n\n4) **Narrow baseline scope.**\n   Comparisons focus largely on **Time-R1** (concurrent work). Including **multi-task/regularization** baselines (e.g., auxiliary captioning, CLIP-style consistency, feature-level regularizers) would better contextualize the benefits of the inversion tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lSXIrLbjFR", "reviewer": "Reviewer_nj73", "summary": "This paper introduces a novel approach to address the common issue of insufficient action understanding in Temporal Video Grounding (TVG) models, often stemming from an over-optimization of Intersection-over-Union (IoU). The core contributions of the paper include identifying the degradation of action understanding due to IoU over-optimization, and subsequently proposing three self-supervised inversion TVG tasks—Verb Completion, Action Recognition, and Video Description—derived from original TVG annotations to enhance the model's semantic understanding of actions across different granularities. The framework employs a dynamically balanced reinforcement learning strategy, executing the primary TVG task with high probability and interleaving it with lower-probability inversion TVG tasks. This ensures the model maintains robust temporal localization while continuously reinforcing its action understanding. Invert4TVG consistently outperforms state-of-the-art approaches across various benchmarks, showcasing its superior capability in understanding complex actions.", "questions": "1.The core hypothesis of the paper is that reversing the task (Invert-TVG tasks) enhances the model’s action-understanding ability, which in turn improves temporal video grounding (TVG) performance. At present, however, this is only an indirect inference drawn from the final localization results. How do you plan to verify that the model’s action-understanding ability has indeed been improved, thereby boosting its TVG performance?\n\n2.The verb-centric design appears to fundamentally limit the model’s ability to handle complex language. Your own failure case (failing to understand “again”) already hints at this, and it does not seem to be a minor issue. Could you discuss more deeply how your method performs and where it breaks down when faced with compositional queries that contain:  \n1. entity attributes (e.g., “the person wearing red”);  \n2. strict temporal relations (e.g., “after A, before B”);  \n3. logical or causal relations (e.g., “tried to do something but failed”)?\n\n3.Your simplistic binary reward—full credit for matching only the verb—invites reward hacking: the model can earn a perfect score by producing meaningless sentences that happen to contain the target verb, instead of truly understanding the video. How can knowledge acquired through such \"short-cutting\" effectively benefit the main TVG task, which requires deep comprehension? Moreover, why did you opt for this minimal reward rather than metrics that better evaluate content quality (e.g., semantic similarity) to mitigate this risk?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.The paper introduces the unique \"Inversion TVG Tasks\" combined with a dynamically probabilistic reinforcement learning framework, cleverly leveraging existing data for self-supervised action understanding, providing a novel problem-solving and execution strategy for the TVG field.\n2.The methodology is rigorously designed, especially in the three multi-granularity inversion tasks and their reward functions. Experiments are comprehensive, yielding significantly superior SOTA performance across multiple benchmarks (including zero-shot settings), further validated by thorough ablation studies, making the conclusions robust.\n3.The paper is logically structured and lucidly written. High-quality visualizations are used effectively to make complex motivations and the proposed framework highly intuitive and easy to comprehend.\n4.This work not only addresses a key bottleneck in the TVG field, significantly advancing the state of the art, but its core idea of \"inversion tasks\" also holds broad inspirational value, offering a new paradigm for other multimodal alignment and understanding tasks.", "weaknesses": "1.The paper’s central claim is that \"by reversing the task, the model’s action-understanding ability is preserved and even enhanced,\" and this improvement is presented as the reason for the superior Temporal Video Grounding (TVG) performance. However, throughout the experimental section the authors only report higher TVG localization metrics (R1@m). They never directly evaluate the final Invert4TVG model on the action-understanding tasks (i.e., the three reversed tasks: VC, AR, and VD) on the test set. Consequently, one can only infer—indirectly—from the improved localization accuracy that action understanding has become stronger, which lacks direct evidence. The authors should supplement the corresponding experiments to provide conclusive proof for the core argument that \"action-understanding ability is improved.\"\n\n2.The notion of \"action understanding\" is overly simplified into superficial verb matching, and its binary reward mechanism cannot distinguish genuine semantic comprehension from opportunistic keyword generation. Such a narrow proxy disregards the deep understanding of context, entity properties, and logical relations indispensable for the TVG task. Consequently, any improvement may reflect only enhanced pattern-matching rather than true generalizable understanding, fundamentally undermining the scientific explanatory power of the core claim—\"boosting localization performance by enhancing comprehension.\"\n\n3.A fundamental limitation of this methodology is its atomistic, verb-centric design, which treats isolated verbs as the basic unit and inherently disregards the compositional nature of language. Consequently, it cannot handle complex queries that involve attributes, temporal order, or logical relations—examples include “event B happens after event A” or “the action performed by the person wearing red.” The method’s success is therefore likely confined to simple, atomic queries and fails to generalize to the richer, more complex TVG scenarios common in real-world applications. The reported inability to understand “again” is not an incidental error but a direct symptom of the approach’s intrinsic deficiency in modeling relational information, which severely restricts the universality and practical value of its contributions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "igxxUA9pvv", "reviewer": "Reviewer_voLD", "summary": "This paper introduces Invert4TVG, a framework for Temporal Video Grounding (TVG). The key motivation is that current TVG models (e.g., Time-R1) tend to over-optimize temporal IoU while neglecting action understanding, leading to semantically incorrect grounding despite high IoU scores. The authors propose three inversion-based auxiliary tasks (Invert-TVG) derived from the original TVG supervision. Experiments on some popular datasets show SOAT performance.", "questions": "Please refer to Weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "+ The idea of reversing the TVG process to construct self-supervised auxiliary objectives is conceptually fresh and well-motivated.\n+ The method is compatible with large LVLMs and scalable to different model sizes (3B and 7B).\n+ The article is written in a prominent style, making it easy for readers to grasp the core points.", "weaknesses": "+ The authors claim that “existing TVGs over-optimize IoU, leading to semantic degradation”, but this paper lacks the experimental analysis of IoU improvement.\n+ The datasets used in TVG and Invert-TVG duplicates? Is the Invert-TVG more like a video QA task?\n+ Limited performance improvement compared to Time-R1. There are no related ablation experiments for parameter p=0.8 in the main paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GyP9ok9eoV", "reviewer": "Reviewer_9Nkj", "summary": "The paper presents Invert4TVG, a novel RL-based framework for Temporal Video Grounding that introduces three inversion tasks: Verb Completion (VC), Action Recognition (AR), and Video Description (VD). These auxiliary tasks aim to preserve and enhance the model’s action understanding ability, which is often degraded when optimizing only for IoU-based rewards. The method alternates between TVG and Invert-TVG tasks with probabilistic scheduling and achieves consistent performance improvements across datasets such as Charades-STA, ActivityNet, and QvHighlight.", "questions": "see above", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Strong conceptual motivation. The paper convincingly argues that existing TVG models fail primarily due to a lack of action understanding, not just poor temporal localization. The inversion-based idea to restore semantic understanding is clear and innovative.\n2. Clear RL framework. The Invert4TVG reinforcement learning setup is well structured and explained. The probabilistic alternation between TVG and Invert-TVG tasks is intuitive and well justified experimentally.\n3. Three inversion tasks. The decomposition into VC, AR, and VD tasks provides a balanced approach across understanding levels, which are aligned with the grounding objective.\n4. Quantitative improvements. Results on Charades-STA, ActivityNet, and QvHighlight show consistent gains.\n5. Good analysis and ablations. The ablation study provides insight into the contribution of each inversion task and the effect of task probability, which helps validate the design decisions.", "weaknesses": "1. Teaser figure clarity (Fig. 1). The sequence of frames is visually unclear, and it is difficult to understand the action being performed. Consider refining the teaser fig. \n2. Limited per-component ablation. Table 2 investigates single-task vs. multi-task setups but does not test pairwise combinations (e.g., VC+AR, AR+VD, VC+VD). Adding these 3 additional experiments would more clearly quantify each component’s interaction.\n3. Missing recent related work. The paper misses recent temporal grounding works such as Number it: Temporal Grounding Videos like Flipping Manga (CVPR 2025) and TimeSuite (ICLR 2025), both relevant to grounding and MLLM temporal reasoning. These should be discussed for completeness. \n4. Limited qualitative diversity. The qualitative examples (Figs. 7–9) mostly feature human actions like “opening/closing doors.” More varied examples (e.g., multi-person or non-human object interactions) would strengthen the claim of generalizable action understanding.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)