Universal Video Temporal Grounding with Generative Multi-modal Large Language Models
Authors: Zeqian Li
Shangzhe Di
Zhonghua Zhai
Weilin Huang
Yanfeng Wang
Weidi Xie
Venue: NeurIPS 2025 poster ()
Keywords: Video Temporal Grounding
Long Video Understanding
Multi-modal Large Language Model
Primary area: applications
TL;DR: We propose a universal video grounding model based on MLLMs, which achieves superior accuracy, generalizability, and robustness.

Abstract: This paper presents a computational model for universal video temporal grounding, which accurately localizes temporal moments in videos based on natural language queries (e.g., questions or descriptions). 
Unlike existing methods that are often limited to specific video domains or durations, we propose **UniTime**, a robust and universal video grounding model leveraging the strong vision-language understanding capabilities of generative Multi-modal Large Language Models (MLLMs).
Our model effectively handles videos of diverse views, genres, and lengths while comprehending complex language queries.
The key contributions include:
(i) We consider steering strong MLLMs for temporal grounding in videos. To enable precise timestamp outputs, we incorporate temporal information by interleaving timestamp tokens with video tokens.
(ii) By training the model to handle videos with different input granularities through adaptive frame scaling, our approach achieves robust temporal grounding for both short and long videos.
(iii) Comprehensive experiments show that UniTime outperforms state-of-the-art approaches in both zero-shot and dataset-specific finetuned settings across five public temporal grounding benchmarks.
(iv) When employed as a preliminary moment retriever for long-form video question-answering (VideoQA), UniTime significantly improves VideoQA accuracy, highlighting its value for complex video understanding tasks.

Universal Video Temporal Grounding with Generative Multi-modal Large Language Models
Zeqian Li", Shangzhe Di Zhonghua Zhai?,
Weilin Huang", Yanfeng Wang', Weidi Xie
SAI, Shanghai Jiao Tong University 2ByteDance Seed
https: /ttl///zzz..// htt.//g5.itttub......... ioriiii/nniii
Abstract
This paper presents a computational model for universal video temporal grounding, which accurately localizes temporal moments in videos based on natural language queries (e.g., questions descriptions). Unlike existing methods that are often limited to specific video domains or durations, we propose UniTime, a robust and universal video grounding model leveraging the strong vision-language under- standing capabilities of generative Multi-modal Large Language Models (MLLMs). Our model effectively handles videos of diverse views, genres, and lengths while comprehending complex language queries. The key contributions include: (i) We consider steering strong MI.LMs for temporal grounding videos. To enable precise timestamp outputs, we incorporate temporal information by interleaving timestamp tokens with video tokens. (ii) By training the model to handle vidcos with different input granularities through adaptive frame scaling, achieves robust temporal grounding for both short and long videos. (iii) Compre- hensive experiments show that UniTime outperforms -tate-of-the--e art approaches in both zero-shot and dataset-specific finetuned settings across five public temporal grounding benchmarks. (iv) When employed as preliminary moment retriever for long-form video muestion-answering (VideoQA), UniTTime significantly improves VideoQA accuracy, highlighting its value for complex video understanding tasks.
approach
Introduction
Understanding long videos presents fundamental challenges due complex temporal dependen- cies and varying information granularity. Recent approaches 30 42 typically adopt a two-stage framework: first localizing relevant temporal segments, then reasoning the retrieved content. paradigm offers a structured and interpretable framework for tackling intricate video understanding tasks.
This
However, existing grounding models perform unsatisfactorily, often struggling to generalize across diverse video domains [50]. In this context, advocate for developing universal temporal ground- ing model, that is capable of accommodating videos across a wide range of viewpoints (egocentric and exocentric perspectives), topics (from cooking and daily activities sports, movies, and games), and durations (from brief clips to multi-hour videos). Such universality is crucial for real-world deployment, where video data is inherently heterogeneous and user queries are highly diverse.
Research on temporal grounding falls into two paradigms. The first includes discrininative or dual- encoder-based based approaches [4] [24], which typically rely on dataset-specific fine-tuning and lightweight architectures, such as DETR-like frameworks. While computationally efficient, these models often struggle to generalize to unseen content due to limited language encoding capabilities, hindering the understanding of complex or drer-driven queries, and constrained visual representations. In
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

(a1) Diverse Videos
(a2) Diverse Queries
Caption
Step
Question
person puts
Season
What pour
books on shelt.
with
the howf?
Long
Shoet
Exp-centric
Ego-centicc
Sport
Activity
(c) Performance Comparison
Ege4D-NLC
(b) Co Universal Video Temporal Grounding
LangViden@anch
ToCoS
Coarse grained
100:00:00, 00:20:00)
(00/09:20, 00:13:10
DAUB
Charades -STA
MLVU
624
30.2
36.4
11.4
UniTi
Fine- Fine-grained
100:08:20, 00:13110) > [c0:10.05, 00.10:30)
CG-Bench
40.3
50.0
74,3
55.5
Captions
QaEgo4
OVHighlights
Previous Top-tlers
Unitime
Figure 1: The UniTime framework empowers MILMMs with advanced universal temporal grounding capabilities. (a) UniTiime can handle diverse videos with various views, genres, and durations, as well as comprehend complex language queries.
(b) UniTime achieves universal temporal grounding through coarse-to-fine approach. (c) Performance comparison temporal grounding and video question answering benchmarks demonstrates the superior capabilities of UniTime.
contrast, recent work leverages Multi-modal Large Language Models (MLLMs) 62 which excel
at language understanding and demonstrate promising results on video temporal grounding. However, their applicability long videos hindered by short context windows and high memory demands, making them impractical for processing video inputs spanning tens of thousands of tokens.
In this paper, we further explore the potential of MLLMs towards universal temporal grounding in videos. Our proposed model, termed as UniTime, encodes temporal information by interleaving timestamp tokens with video tokens, enabling precise boundary generation for natural language queries. To handle videos of varying lengths, we introduce an adaptive frame scaling mechanism that adjusts temporal and spatial granularity based on video duration. Specifically, for long videos, the model can be adopted for multi-stage inference-time. computation: initial coarse-graing -gained sampling enables segment-level grounding. followed by fine-grained sampling within candidate regions to refine boundaries. This iterative process progressively improves grounding precision. To improve training efficiency, we adopt a video-centric training paradigm that concatenates multiple queries and their corresponding segments per video, allowing the model to process them in a single pass. This minimizes the computational overhead autoregressive decoding over repeated video tokens.
To rigorously evaluate efficacy, we pre-train UniTime on diverse, high-quality temporally grounded video datasets and assess its performance on two key tasks: temporal grounding (in both zero- shot and fine-tuning settings) and downstream video question answering (VidcoQA) for long conduct experiments on two long-video and three short-video
videos. For temporal grounding, benchmarks, where our model consistently outperforms existing methods in both straightforward and complex retrieval scenarios. For downstream VideoQA, we evaluate on four widely used long-video benchmarks using retrieval-augmentte protocol: first, our framework localizes the relevant video segment; then, an off-the-shelf VideoQA model answers the query based on the localized segment. Compared to prior approaches, our method demonstrates superior generalization, delivering more accurate and comprehensive video understanding, especially for long videos.
The remainder of the paper is organized as follows: Section 2 formulates the temporal grounding task and details our proposed method. Section presents ablation studies and comparisons validate our approach. Section 4 provides an overview of the relevant literature. Overall, our method achieves improved performance on both temporal grounding and long-video VideoQA benchmarks.
2 Methods
In this section, we introduce UniTime, novel framework for universal video temporal grounding, adapted from pre-trained Multi-modal Large Language Models (MLLMs). Our approach achieves precise temporal grounding across videos of diverse scenes, genres, and durations, and employs an iterative coarse-to- coaree-to-fine strategy to handle long videos. In the following, Section 2.1 formalizes the

---

problem setting. Section 2.2
presents our architecture and its core modules, which support flexible temporal grounding at arbitrary granularities via inference-time computation. Section 2.3 details an efficient training paradigm that equips MLLMs with universal temporal grounding capabilities.
2.1 Problem Formulation
sampled at timestamps T (41,---,43 (N and free-form text query 2 (either complex question plain description), task is to identify all temporal moments y ( e1),..., (sk, that semantically match the query, where each 8k, Ck E T denotes a start and end timestamp. respectively:
Given an untrimmed video sequence
......... fN,, E RNyxHHWx3
y PUniiimee @UnnTime QV,
With this formulation, temporal grounding achieved by identifying the frames most relevant to the query and retrieving their timestamps. By adjusting the granularity of T, our design supports multi-scale temporal predictions, enabling efficient coarse-to-fine grounding for long videos through inference-time computation, as shown in Figure I b).
2.2 Universal Temporal Grounding Model
The core innovation of UniTime lies in its capability to perform predictions at multiple temporal granularities for videos of diverse lengths. The model consists of three key components: (i) adaptive frame scaling for constructing multi-grane calti-graaular video inputs, (ii) timestamp ummestap-inttellaveed interleaved sequence construction enabling multi-scale temporal window prediction, and (iii) multi-scale prediction for coarse-to-fine temporal grounding during inference, as illustrated Figure
Adaptive Frame Scaling. A central challenge in universal temporal grounding is handling videos with highly variable durations, ranging from a few seconds to several hours. Applying fixed temporal and spatial resolutions across such diverse inputs is inefficient and often impractical, especially for MLLMs constrained by limited context windows and GPU memory.
To address this, we propose an adaptive frame scaling mechanism that dynamically adjusts per-frame token budget (Nrs) based video duration, allocating high spatial resolution to short videos and low resolution to long videos. Specifically, given a video V with constant frame rate (Nş frames in total), each frame is allocated Niss [Niaat J tokens, where Noat the total token budget.
To support varying spatial granularities, we set two thresholds, Nypha and N N as the maximum number of frames for short and long videos, respectively. Frames are then processed as:
Oproojct (dvision (Presize (J:))) E R Nix
Ny Nsshort Nghhrt < N2 < long
V,
Ccompress Oproojector @visson (fi))) E N.
where (vision(-) (-) is the vision encoder, which supports dynamic input resolutions with fixed patch size of P. and (-)) is the projection module. v/resize (-) denotes the frame resizing function, which adjusts the spatial resolution of each frame so that it can be partitioned into Niss patches of size P x P pixels. 3'compreas denotes token compression via bilinear interpolation, yielding Nis tokens as the final representation. Unlike frame resizing, which may degrade spatial details, token compression reduces redundancy the token level while better preserving key semantic information, especially for long videos with low resolutions per frame. When the number of frames exceeds iylong the video is divided into multiple clips of length / for divide-and wwide-andd-onquer processing, The ablation experiments of token budget, thresholds and the input processing strategy can be found in Appendix E.3 and E.I
Discussion. Existing MLLMs typically employ fixed frame sizes and token allocations 33 68 25 which limits their ability to encode long videos effectively. To handle such inputs, these models must resort to sparse temporal sampling, resulting in significant loss of detailed visual content. In contrast, adaptive strategy dynamically employs higher spatial resolutions for short videos and lower ones for longer videos. This allows for efficient and scalable modeling across diverse video lengths, maximizing the utility of the available token budget and model capability.
Timestamp-
Timessamm-Inteelleavee Interleaved Sequence Construction. To associate frames with their timestamps, we encode temporal information by interleaving timestamp tokens between visual tokens. For a frame f.
3

---

Coarse-to-Fine Temporal Grounding Adaptive Frame Scaling (b) Timestamp-Interleeved Sequence
From 18.0%
Long Videos:
Low Spatial Resolution
Large Language Model
Goorse-gre grained Prediction
Adaptive
Frame
560x
Scaling
Instruction
Query
Video
Vislen Encoder & Projector
Timestamps
Short Videos:
High Spatial Resolution
Fine-grained Prediction
Fromes
Frome Scaling
Adop five
1
2.0x
126 ts
791s
Figure 2: Overview of the proposed UniTime framework. (a) UniTime achieves universal temporal grounding by leveraging adaptive frame scaling construct multi-scale video inputs and then generate multi-scale predictions, allowing robust grounding across diverse video durations. (b) Within the model architecture, UniTime constructs features, which, combined with the language query. is fed into the LLM and then identifies the corresponding temporal interval from the timestamp tokens.
interleaved sequence of timestamps and scaled frame
sampled at timestamp 4. the timestamp is represented as free text T; = "timestamp: t seconds", and inserted before visual tokens of fi, forming an interleaved sequence:
S T Vi;Vas T2; Vii... TNy V Nyi Q). T, Prokenizer (T.)
where Otokenizer ( is the tokenizer. This interleaved sequence is input to the large language mod- els (LLMs), which generates temporal boundaries for given query Q, with the output formatted as A FFrrm 8% seconds to e seconds". Note that, unlike conventional methods that predict the exact moment in the original video, model instead predicts the minimal timestamp interval (#k, Cx) from the sampled timestamps set T. That is to say, we leverage the retrieval capabilities of MLLMs to read out the inserted timestamp tokens, rather than decoding dense position encodings. For a comparison of these temporal information encoding approaches, refer to the Appendix E.2
It is worth noting that direct fine fineinna temporal interval prediction on coarse -rriined long video inputs is inherently unreliable, as the lack of detailed spatial information often results in ambiguous or inaccurate predictions. This highlights the necessity of adopting predictions at different scales for videos with varying granularities. Our framework addresses this challenge by offering inherent flexibility in timestamp token insertion, making it adaptable to varying granularities. Specifically, timestamp tokens can be interleaved at different scales: before each frame for fine-grained temporal predictions, or before fixed-length segments for coarse-grained temporal reasoning.
We divide the input video into N₂ segments {S,}) each comprising L, frames. Here, for the jth segment, S, [V.y Viy+1;-... LL,+ RL,, where sy is the start index of the segment and we use the corresponding timestamp ta, at the beginning of each segment to represent its temporal location. Note that we prepend only single timestamp text token to each segment, thereby providing coarse-grained temporal supervision. We also feed the interleaved timestamps, visual tokens, and the query into the LLM retrieve the target segment.
S = TT: S S2: S.: Tw, SN, Q], T2 @wokeiizer (Tx,)
(2)
Discussion. Here, explicitly providing timestamps offers several crucial advantages: (i) Improved temporal grounding. These explicit timestamp tokens help the L.MM effectively associate visual content with its corresponding temporal position in the video, allowing it to generalize to temporal grounding tasks. (ii) Alignment-free integration. As the inserted timestamps are textual, they naturally reside in the language space and do not require realignment. In contrast, methods based on learned vectors necessitate additional embedding alignment, thereby complicating the LLM's understanding of temporal information. (iii) Generalisable temporal reasoning. When combined with adaptive frame scaling, explicitly provided timestamp tokens allow the model directly refer to the temporal cues, potentially enabling extrapolation. For example, as demonstrated in Section this design allows the model to be trained on short or medium-length videos (e.g., around minutes), while still generalizing effectively to much longer videos (e.g., 20 minutes) during inference on downstream tasks. (iv) Model-agnostic design. This method is plug-and-play and can be easily incorporated into diverse model architectures with minimal modifications, which is validated in

---

Table 1: Statistics of temporal grounding datasets. The datasets in Part are used exclusively for pre-training while those in Part Il are the training sets of benchmarks.
Dataset
#Videos
#Queries
Video Len.
Moment Len.
Views
Domain
NaQ
4.9K
1031K
413s
1.1s
Ego
Open
DiDeMo
8.4K
33,0K
29s
7.5s
Exo
Open
QuerYD
819
5.7K
278s
13.6s
Ego & Exo
Open
HiRest 61
526
4.0K
263s
18.9s
Ego & Exo
Open
COIN 533
118K
46.4K
145s
14.9s
Ego & Exo
Open
Momentor 45
6.7K
136.4K
403s
49.5s
Ego & Exo
Open
YouCook2
72
1.2K
9.6K
316s
19.7s
Ego & Exo
Cooking
Ego4D-NL.O 111
1.0K
10.0K
495s
8.2s
Ego Ego & Exo
Open
TACoS
+7
75
9.8K
287s
27.9s
Cooking
Charades-STA 51
4.8K
11.2K
318
8.1s
Exo Ego Exo Vlog/News
Activity
QVHighlights 24
7.1K
7.2K
150s
24.6s
ANet-Captions
10K
37.4K
118s
36.0s
Ego & Exo
Activity
Section 3.4
eliminating the need for architecture-speccfic adjustments such custom positional
encoding schemes 12,2)
Universal Temporal Grounding with Multi-stage Inference. Building on these designs, our framework achieves universal temporal grounding by processing input frames at different spatial resolutions and generating temporal predictions appropriate granularities, according to video duration. For long vidcos, employ a hierarchical, multi-stage inference strategy. Specifically, if video exceeds N-long frames, it is partitioned into multiple clips, each of length N Segment retrieval is first performed within each clip to identify potentially relevant segments. The retrieved segments from all clips are then aggregated and subjected to further segment retrieval, and this process can be repeated recursively if necessary. Finally, fine- ane-ggained grounding conducted within the final selected segments. As evidenced by Section 3.5 this flexible and hierarchical inference process ensures accurate and efficient temporal grounding across diverse video durations.
2.3 Training
In this section, we first describe the training objective for temporal grounding, followed by detailed explanation of the hierarchical data construction for multi caalti-granularity training. We then present video-centric training mechanism to improve training efficiency.
Training Loss. The model is trained with an auto-regressive objective over mini-batches. For each training sample (V, T, Q, 2)) 1), i.e., the video, timestamps, text query, and target, the prompt sequence constructed based on Iqquation[or for short and long videos, respectively. Then, the model minimizes the negative log-likelihood over the target tokens only:
N₂
((S.Y)
= Σ
log P(ye S, Ves:
i=1
where N, is the target length.
Training Data Construction. To support temporal grounding multiple granularities, we construct training data from videos of varying durations, using full videos for coarse-grained supervision and randomly sampling shorter segments containing ground-truth moments for fine-gra fine-grained training. All samples are combined for joint training. Since this process naturally yields more short videos than long ones, balance the data distribution by replicating long video samples with factor Neep 1.
Video-centric Training. Given dataset comprising paired videos, text queries, and corresponding moments (V))), ((k),)())) previous methods typically employ query-centric sampling: first sampling query, then loading its associated video. This approach is highly inefficient when multiple queries correspond the same video, introduces (i) I/O bottleneck from repeatedly loading long videos and (ii) redundant computation due to re-encoding identical video content.
To address this, we employ video-centric sampling, wherein each video ((k) is sampled first, and all associated query-an -newer pairs are grouped into single input sequence. This strategy minimizes
5

---

Table 2: Statistics of benchmarks. The benchmarks encompass three representative tasks Video Temporal Grounding, Grounded VideoQA, and General VideoQA. The reported statistics are based on the actual numbers of videos and queries used in our experiments.
Benchmark
#Videos
#Queries
Video Len.
Moment Len.
Views
Domain
Ego4D-NLO
1
415
4.6K
500s
10.7s
Ego
Open
TACoS 47
25
4.0K
368s
31.9s
Ego & Exo
Cooking
Charades- STA
51
1.3K
3.7K
29s
7.8s
Exo
Activity
QVHighlights
24
1.5K
1.5K
150s
32 2s
Ego & Exo
Vlog/News
ANet- N
22
4.9K
17.0K
118s
40.2s
Ego & Exo
Activity
QaEgo4D 3
148
500
487s
13s
Ego
Open
CG-Bench [5]
1.2K
12K
1661.1s
19.8s
Ego & Exo
Open
MLVU [T]
1.1K
2.2K
755.7s
Ego & Exo
Open
LonggideeBennc [57
618
1.2K
574.9s
Ego & Exo
Open
Table 3: Comparison with SoTA methods on video temporal grounding benchmarks. SP denotes the datasct-specific fine-tuning setting, and Full refers the universal pre-training setting. Best
results are bold. Improvements of UniTime- Full over SoTA are highlighted in (green).
Long-video temporal grounding benchmarks.
(b) Short-video temporal grounding benchmarks.
Method
Noo
TaCoS
Method
Charades-STA
ANet-Captions
QVHighlights
RIG.3 RIG.5
mloU
RIG.3
RIG.5 mloU
RIG
RI0.7
RI@.5 RI@
RIG.5
RI@.7
2D-TAN 66
5.04
2,02
3.53
45.61
35.77
UnLoc-L 58
60.80
38.40
48.30
30.20
66.10
46.70
VSL.Net
65
10.84
6.81
3.83
35.54
23.54
24.99
SG-DETR 10
70.20
49.50
72.20
56.60
RGNet
12.04
12.41
LD-DETR
41.56
66.80
SnAG 3r
15.87
11.26
56.44
44.86
SeAG
64.62
46.26
48.55
30.56
SG- SG-DETR 10
56.71
44.70
40.90
UniVTG 27
58.01
35.65
58.86
40.86
LD-DETR
UniVTG
21.55
UniVTG 27
7.28
3.95
4.91
51.44
34.97
33.60
TimeSuite 62
67.10
43.00
UniVTG wPT
11:74
7.54
1.88
56.11
43.44
38.63
Mr.LIP 34
69.31
49.29
$3,92
35.55
74.77
60.51
Owen2-VL-
4.14
Qwen2-VL
3.87
Qwen2.5 VL 2
1.11
0.48
0.82
7.66
3.35
5.29
Qwen2.5
60.32
34.27
16.96
8.75
UniTime SP
24.79
16.83
17.25
61.18
48.31
45.02
UniTime SP
24.33
53.71
54.81
36.62
77.76
63.29
UniTime Full
18.41
18.80
66.91
50.00
UniTime Full
56.85
35.90
(+8.8))
(+6.37)
(6.29)
(+9,30)
(+10.44)
(19.10)
(15.07)
(+735)
(.025)
(+0.35)
(+1.95)
(+2.14)
redundant video loading and computation, significantly accelerating training. To maintain consistency. we modify the attention mask to prevent cross-interaction between different query-answa pairs, and assign each query-answer sequence the same starting position index, directly following the video tokens' indices. For further implementation details, we refer readers to Appendix B.2
3 Experiments
3.1 Experimental Setups
Datasets. To support universal temporal grounding, compile diverse dataset spanning var- ied scenes, genres, durations, and query types (e.g., descriptions, questions, procedural instruc- tions), as detailed in Table For evaluation, benchmark our method across three categories:
(i) Short-video temporal grounding, including Charades-STI [51], ActivityNet-Captioo [22],
and QVHighlights (24). (ii) Long-video temporal grounding, including Ego4D-NLQ [II] and
TaCoS [4]. (iii) Video question answering (VideoQA). including two grounded benchmarks with tem- porally annotated queries (QaFgo4D 3. CG-Bench (5) and two general benchmarks (MLVU 77..
Long VideoBennn (57). The statistics of evaluation benchmarks we used are listed in Table 2
Evaluation Metrics. For temporal grounding tasks, adopt Recall @ I (R1) at multiple temporal palersection--ver-uniie (IoU) thresholds and mean loU (mIoU) as evaluation metrics. Specifically, use IoU thresholds of 0.3 and 0.5 for long-video benchmarks, and 0.5 and 0.7 for short-vides benchmarks. For general VideoQA tasks, we report standard accuracy metrics, while for grounded VideoQA tasks, also evaluate R1, mIoU, and intersection-oveeprrcccccie (IoP). More details about metrics are in the Appendix A.I
Implementation Details. Our framework is built on PyTorch, with Quen2-VL-77 56 as the base
model. All experiments are conducted with batch size of 8, using AdamW 32] optimizer with the learning rate 2c-4, and trained for one epoch with linear warmup during the first 3% of steps. The

---

Table 4: Zero-shot performance on video temporal grounding benchmarks. Zero indicates zero-shot. All models are evaluated in the zero-shot setting. Numbers in gray are sourced from the
original paper, all others are tested by using their released code and checkpoints. Improvements of Time--eeeeo over SoTA are highlighted in (green).
Ego4D-NL.
TaCo8
Charades-STA
ANel-Cuptions
QVIlighlights
Method
RI
RI@ 5 mloU RI 3 RI@
mloU
RI@.5
RI0.7
mloU R1
R10.7 mloU RI@.5
R
mloU
UniVTG 27
6.48
3.48
4.63
5.17
1.27
4.40
25.22
10.03
27.12
11.10
4.06
16.86
14.13
4.42
23.38
MrBLIP 34
6.49
3.20
5.37
24.59
14.32
17.94
VTG-LLM 12
1.71
0.46
1.36
6.87
2.92
5.27
34.11
15.81
34.93
12.32
6.74
17.86
4.13
1.55
8.60
Momentor 45
26.60
11.60
28.50
23,00
12,40
29.30
ViimeLLM
[8
1.67
2.52
9.30
8.08
34.30
34.60
26.13
11.16
29.42
Timechat HB
1.67
0.79
1.30
3.77
1.60
2,95
29.73
12.53
31.20
16.38
8.36 21.53
8.32
4.26
14.25
TimeMarker
51.90
26,90
48.40
TimeSuite
0.88
0.43
0.94
6.75
2,50
5.71
48.95
24.65
45.91
16.56
9.28
22.03
12.32
9.16
21:30
UniTime-7ezo
14.67
7.38
50,06
31.54
33.38
59.09
31.88
52.19
22.77
14.14
41.03
31.48
43.71
(-3,18)
(+3,90)
(+0.1)
(+25,47)
(+1322) (+15,64)
(+7.1))
(+1.96)
(+3,79)
(-0.23)
(+1/99)
(-1.99)
(+1450)
(+30.33)
(+14,29)
Table 5: Closed-source Model Evaluation. All models are evaluated on sampled subsets, with outputs lacking predicted timestamps excluded.
Table 6: Flexibility Verification. Improvements over baseline are highlighted in (green). All UniTime variants use the data-specific setting.
Egu4D-NLQ
Charades STA
Mechol
Egs4D-NLL
Chorale-STA
Model
R16
R10:5
mloU
R10:55
R10.7
m8eU
R10.3
R18:5
mloU
R10.5
R10.7
miot!
Qwen2-VE--B
5h
0.34
0.34
0.42
2.23
0.51
6.70
Gemini- 2,5-lmsh
8
15.29
15:29
13:93
36.46
13.54
38.72
Qwan2-VE_7BB
-Unfiime
10.50
0.48
5,811
0.46
7.2M (46.87)
05.34
42.14
57.25 (-50,55)
Gemini-2.5-je g
26.83
19.51
20.45
35.96
11.24
39.28
(+58.17)
GPT-4.1-mini
3,00
5.04
31.00
17.00
31.46
Dwm22.--VL-78
+Unii imme
21.53
L111
16.12
16.11 (+15.79)
74.19
60.32
53.98
34.27
61
52.42
(-11,20 20)
GPT-40 3
10:59
7.06
7:97
41.11
17.78
43.82
IntemV11.. 26
0,33
0.22
0.42
5.48
1.77
9.30
Seed1.5.VL VL [
19.39
10.20
13:74
90.82
63.27
73.69
latemV1...558 5-3B
+UnfTime
i
9,87
2.19
0.55
5.26
7.46 (47.04)
1.77
69:49
9.66
2.55
47.77
59103 (+-9773
17.54
UniTime-Full
25.00
17.00
17.20
81.00
61.00
70.11
+Unil Iime
16,23
9.55
12 (0.06
71.83
49.02
04).00
(+-22 96)
vision encoder is frozen, and the LLM is fine-tuned via LoRA [6] (rank = 8, alpha =). 8). We sample
frames at 2 fps, with Nyshout = 128 and N = 1024. capping input video at 16,384 tokens. The default segment length and replication factor are set to 32 and 4, respectively.
3.2 Comparison with State-of-the-arts on Video Temporal Grounding
Dataset-specifii Setting. In Table 3 we compare our model against state-of-the-an methods on two long-video benchmarks and three short-video benchmarks. Under this setting, where models are trained and tested separately on each dataset, our method (UniTime-SPP) outperforms all existing approaches (including traditional and MLLM- models), achieving absolute improvements of 6.39% on Ego4D-NLQ. 9.10% on TaCoS, 5.45% on Charades 2.88% on ANet-Captions, and 2.89% on QVHighlights over prior best results.
Universal Setting. To assess broad applicability, we train single model (UniTime-Full) on the largest collection of temporal grounding data (Table Parts I & II). Unlike dataset-specific tuning, this setting prioritizes broad competence over specialized performance. ensuring the model robustly handles diverse video types and query styles in real-world scenarios. Without venchmark-specing fine- tuning. UniTime-Full achieves superior performance across all evaluated benchmarks, demonstrating its effectiveness as a universal solution for video temporal grounding.
Zero-shot Setting. To further test the generalization capability of our method, pre-train zero-shot model (UniTime-Zero) on Part I datasets in Table [] while excluding all in-domain data (Part II). As shown in Table 4 our method achieves superior zero-shot performance, highlighting its remarkable generalization capability. The error analysis on ANet-Captions can be found in the Appendix C.2
3.3 Comparison with Closed-source models on Video Temporal Grounding
To comprehensively assess UniTime, we benchmarked mainstream dlosed-source MI.LMS to examine whether Uni Time surpasses them or narrows the gap on VTG tasks. We conducted temporal grounding experiments on a randomly sampled subset from Ego4D and Charades. As shown Table 5
SeedI.5-VI. .5LVL. (54) attained the strongest performance on short videos (72.2 mloU on Charades), while
Gemini-2.55Pro S exhibited superior grounding on long videos (20.5 mIoU on Ego4D). However,
these closed-soures models struggle to achieve high performance concurrently on both long and short
7

---

Table 7: Ablation of the proposed modules. Adaptive Scaling refers to adaptive frame scaling and Segment Retrieval indicates nuuti-granular temporal prediction.
Table 8: Robustness test on time shift and query decomposition. QD indicates query de- composition, TS denotes time shift, and Ratio TS performance relative to the original.
Adaptive Scaling
Multi-stage Inference
Segment Retrieval
Ego4D-NLL
Mechod
Charades STA
Charades IoG
Shift
R1
R1087
Ratio
R1@.3
RI@.5 5
mloU
VTimeLLM
28.53
14.65
12:35
53.67
x
14.25
7.54
9.83
x
14,00
7.51
9.83
MrBLIP
51.32
70.05
26.91
43.63
66.77
71.22
V
X
V
V
x
x
17.91
18.42
11.99
12.13
12.39
12.78
Tim:Suite
X
37.02
48.65
24,01
13.55
45.32
31,54
66.22
47.07
V
24.79
16.83
17.25
UniTime:SP
K
63.06
73.36
53.74
37,47
53.39
62.87
80.30
74.88
videos, indicating a lack of generalization. In contrast, UniTime achieves competitive performance compared to these closed-source models while maintaining more balanced capabilities for both short and long-form video grounding benchmarks, proving approach to be overall the best in terms of consistent performance across varying video types.
3.4 Flexibility Verification
As discussed Section 2.2 our approach compatible with any MLLM that processes dynamic frame inputs. To evaluate flexibility and generalization, we conduct experiments across multiple baseline MLLMs (e.g., Qwen2 Qwen2. VL, InternVL2. 5) and model scales. As shown in T larger MLLMs outperform their smaller counterparts, and our method consistently improves temporal grounding performance regardless of the underlying architecture. These results demonstrate strong generalization and flexibility, and indicate that the effectiveness of our framework continues to benefit from advances in MLLM architectures and increased model scale, enabling further gains in temporal grounding as more powerful models become available.
3.5 Ablation Studies
Effect of Individual Modules. To assess the contribution of each module, we conduct ablation studies on the Ego4D-NLQ benchmark, with results presented Table 7 The study examines three key components: "Adaptive Scaling" for video input processing: "Multi-stage Inference" implementing coarse-to-fine grounding for long videos; and "Segment Retrieval" for coarse temporal grounding through inserting timestamps before segments. The results demonstrate three key findings: (i) Adaptive frame scaling (row 2) maintains performance comparable to uniform sampling (row I) when used independently. (ii) Multi-stage inference yields consistent improvements for temporal grounding (row 3) through progressive refinement from lower to higher frame rates, and for spatial grounding (row 4) via progressive refinement from lower to higher resolution, with both coarse-to- fine grounding strategies benefiting from this hicrarchical approach. (iii) Most notably, incorporating temporal prediction (row 5), where the model first retrieves coarse relevant segment and then localizes the precise moment within it, yields substantial improvements across all metrics. These results highlight the effectiveness of the multi-stage temporal grounding framework. More details can be found in the Appendix [
Effeet of Segment Length. Our universal temporal grounding framework operates in two stages: (i) retrieving the video segment most relevant to the query; (ii) predicting the precise temporal interval within it. Segment length critically influences this process. We evaluate its impact using three metrics: Segment retrieval accuracy (R @ 1) measures the ability to identify the correct segment. Oracle grounding accuracy (oracle RI @0.3) reflects fine-grained localization given the ground-truth segment. Overall grounding accuracy (Rl @0.3) assesses the overall end-to-end temporal grounding performance. As shown in Figure 3 a), longer segments improve segment retrieval accuracy but hinder fine- mme-grained localization, resulting in lower oracle grounding performance. To balance this trade-off, we set the segment length 32 for optimal overall accuracy.
We investigate the impact of replication factor (Nrp). which balances long and short video samples during training. Figure3]bl shows that increasing Nrep initially improves segment retrieval but saturates and fluctuates later. In contrast, fine-grained grounding performance remains stable. Based on these results, we set Nrep for optimal performance.
Effeet of Replication Factor:
8

---

Relative position event
Performance
Performanne
Segment Retrimal Rei
Rig.3 24.70
Seqment Length
Replication
Relative position start
Figure 3: Ablation on hyperparameters. We decouple the effects of Segment Length (left) and Replication Factor (right) on temporal grounding performance.
Figure 4: Temporal distribution bias Charades-STA. Darker colors indicate higher data density.
3.6 Robustness Test
Generative models are known produce hallucinations in their outputs 23 To evaluate robustness, we conduct experiments on the Charades-STA benchmark from two perspectives: (i) perturbing event positions assess robustness against temporal distributional bias, and (ii) decomposing complex queries into simpler object-based questions to test the model's reliability across query types.
Time Shift Event. Previous studies [14 41 reveal significant biases in distributions within temporal grounding datasets. As seen Figure 4 annotated events in Charades-STT are heavily concentrated near the start end of videos. To assess robustness against such bias, perturb the data distribution by sampling clips that contain the event segment but place it at randomized positions. As shown in Table our method exhibits greater robustness than other MLLM-based approaches, as evidenced by the higher performance ratio relative to the unperturbed baseline.
On Decomposition of Query. Accurate localization of complex queries fundamentally depends understanding the referenced objects or events. To address this, we prompt Qwen2 [59] to decompose each Charades-STI, query into of object-based questions in the form of "When does cobject>> appear?", where <object> denotes an entity that can be grounded in the original query. For each decomposed query, compute the intersection-ovve-grrondd grouu--ggoundttutthh (IoG) between its predicted segment and the ground-truth segment. It is worth noting that this is only an approximate estimation, as entity may appear in multiple temporal segments, while annotations typically provide only single segment. As shown in Table g our approach demonstrates superior query understanding and more reliable localization, with fewer hallucinations compared to other methods.
Recently, a related work [21] focuses on prediction consistency: they also curate test data by rephras- ing original queries and shifting ground- ground-truth moments in videos to probe grounding consistency. For detailed tests and comparisons, please refer to the Appendix D
3.7 Downstream Task
To validate the effectiveness of universal temporal grounding model in facilitating long video understanding as discussed in Section [ we evaluate on four VideoQA benchmarks. As a baseline, we uniformly sample 32 frames from each video and input them the Qwen2-VL- VL-7B model for answer generation. To assess the impact of different temporal grounding approaches on downstream VideoQA, we first use each model to predict the relevant temporal segment for each question, then uniformly sample 32 frames from the predicted segment and feed them into -wen2-VL-77B for answer prediction. As shown in Table our method exhibits superior generalization compared to existing temporal grounding models, demonstrating robust grounding performance on out-of-domail data and enhancing long video understanding. Refer to Appendix A..4 for VideoQA evaluation details.
Table 9: Performance VideoQA benchmarks. loo is the mean Recall loU thresholds
[0.1 0.1 0.5). For grounded VideoQA, loP and mloU are used as grounding metrics.
QaEgo4D
CG-Bench
MLVU
Long VinchBench
Method
@..5
mloU
Acc.
TnP
R@loU
mloU
Acc.
Acc.
Acc.
Uniform Sample
3.67
49.60
1.08
33.87
60.53
54.82
UniVTG 2
9.80
6.20
7.76
50.00
1.37
4.92
3.86
34.87
62.56
54.67
VTimeL.MM
3.87
1.40
3.68
48.60
1.62
1.22
1.58
34.60
59.52
$4.30
TimeSuite 62
6.28
0.20
1.22
48.80
5.06
2.89
2.09
32.47
58.51
53.25
UniTime-Fii
25.29
19.00
18.44
55.51
16.64
17.66
11.63
40.30
66.50
56.47

---

4 Related Work
Video Temporal Grounding (VTG) tasks can typically be categorized into short- and long-duration scenarios. For short videos, SOTA methods primarily adopt DETR-lik architectures [ 24 36 35
10]. For example, Moment-DETR [24 formulates moment retrieval as a set prediction problem with wansformer-basee mncoodr-ddecoder enabling end-to-end prediction of moment coordinates and saliency. Recent works further enhance performance by introducing prior knowledge (00 and advanced attention mechanisms [36 35 Non-DETR methods, such as UMT [29] and Mr. BLIP 34. leverage multi-modal cues and large language models, respectively. While effective for short videos, these methods struggle with long videos, where sparse relevant moments create "needle-in-a haystack" challenge. Recent works [I5] 43 employ coarse-to-fine alignment and hicrarchical pipelines to address this. Despite the progress on individual benchmarks, fundamental differences between short and long videos hinder the development of unified models. While UniVTG [27] attempts to bridge this gap, its lightweight architecture limits generalization, especially in out-of- domain scenarios [50]. This motivates our work to develop a universal grounding model capable of (i) handling videos of arbitrary duration, (ii) robustly generalizing across domains, and (iii) excelling in downstream tasks like video question answering.
Temporal Grounding with Multi-modal Language Models (MLLMs). While MILMs have achieved remarkable success across various multi-modal tasks [23 44 31,, accurate temporal
grounding remains challenging. Existing temporal integration approaches in MLLMs can be grouped into three paradigms: (i) time-agnostic models: VTimeLLM [18) processes fixed-length videos (100 frames) without explicit temporal cues, predicting normalized moment positions; LITA [9 replaces the predicted textual timestamps with special time tokens. The lack of explicit temporal signals often leads to suboptimal performance. (ii) implicit amesstamp-nncoee
models: These methods
incorporate timestamp-awees -wwre encoders, such as TimeChat [8] and TimeSuite [62]. which fuse frame and temporal embeddings. VTG-LLM [12] uses absolute time embeddings, while Qwen2.5-
VL. 2 employs MRoPE for temporal modeling. Despite advances, these methods require extensive pretraining and may hallucinate timestamps due implicit temporal representations. (iii) explicit temporal marking models: Methods like Mr.BI.IP (34), TimeMarker [6]. and VideoLLLaMA3 [63] prepend textual timestamps video frames, leveraging retrieval capabilities of MILMs to match DETR-basee performance. Although these approaches have advanced short video temporal grounding, they are constrained by context windows and GPU memory, limiting scalability to long videos. To address this, we propose framework that adaptively adjusts scales of video frames, combined with a multi-scale, coarse-to-fine iterative temporal grounding strategy tailored for long video understanding.
Long Video Understanding is significantly more challenging than short video or image-based tasks due complex temporal dynamics and substantial redundancy. The large number of frames dramatically increases memory and computational demands, making dense sampling impractical. Existing video MILMs 2 56 69) often rely on uniform frame sampling, which risks omitting critical information. To address this, some approaches compress frames into minimal token sets to reduce computational costs [49] 26 671, while others focus on more effective frame sampling strategies [60] (22 [7. Despite these advancements, temporal grounding- crucial for retrieving salient events remains underexplored in the context of long videos. In this work, we leverage temporal grounding to identify key events, enhancing the understanding of forg-form video content.
5 Conclusion
In conclusion, this paper presents UniTime, a universal video temporal grounding model based on MLLMs. At its core, UniTime interleaves timestamp tokens with video tokens and retrieves relevant timestamps for natural language queries, thereby enabling precise temporal localization. By employing adaptive frame scaling, UniTime constructs inputs with varying spatial granularities for videos of different durations. By further inserting timestamps at multiple temporal granularities, the model supports multi-scale temporal prediction. As a result, it can perform coarse-to-fine temporal grounding, first coarsely localizing segments in long videos and then refining predictions to fine- grained windows in short videos. To further enhance efficiency, we propose a video-centric training paradigm that reduces redundant computation. Extensive experiments demonstrate that UniTime, pre-trained on large-scale datasets without the need for dataset- finetuning, consistently outperforms existing methods in both temporal grounding and downstream long-video question answering tasks.
multiple benchmarks
10

---

Acknowledgements
This work is supported by National Key R&D Program of China (No.2022/D0161440))
References
[1] Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan Russell. Localizing moments in video with natural language. In ICCV, 2017.
[2] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge, Sibo Song, Kai Dang, Peng Wang. Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report, arXiv preprint arXi3:2502.13992, 2025.
[3] Leonard Bärmann and Alex Waibel. Where did leave my keys?-episodi--memory--- based question answering on egocentric videos. CVPR, 2022.
[4] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV, 2020.
[5] Guo Chen, Yicheng Liu, Yifei Huang, Yuping He, Baoqi Pei, Jilan Xu, Yali Wang, Tong I lu, and Limin Wang. Cg-bench: Clue-grounded question answering benchmark for long video understanding. In ICLR, 2025.
[6] Shimin Chen, Xiaohan Lan, Yitian Yuan, Zequn Jic, and Lin Ma. Timemarker: A versatile video- Ilm for long and short video understanding with superior temporal localization ability. arXiv preprint 2rXiv2241.112111 2024.
[7] Zhe Chen, Weiyun Wang, Yue Cao. Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye. Hao Tian, Zhaoyang Liu, al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.0527), 2024.
[8] Gheorghe Comanici, Eric Bieber, Mike Schackermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with ad- vanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261, 2025.
[9] Shangzhe Di, Zhelun Yu, Guanghao Zhang, Haoyuan Li, Hao Cheng, Bolin Li. Wanggui He, Fangxun Shu, Hao Jiang, et al. Streaming video question-answeriing with in-context video kv-eache retrieval. ICLR. 2025.
[10] Aleksandr Gordeev, Vladimir Dokholyan, Irina Tolstykh, and Maksim Kuprashevich. Saliency-guided detr for moment retrieval and highlight detection. arXiv preprint 240.0114, 2024.
[11] Kristen Grauman, Andrew Westhury, Fugene Byme, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, al. Ego4d: Around the world in 3,000 hours of egocentric video. In CVPR, 2022.
[12] Yongxin Guo, Jingyu Liu, Mingda Li, Dingxin Cheng, Xiaoying Tang, Dianbo Sui, Qingbin Liu, Xi Chen, and Kevin Zhao. Vtg-Ilm: Integrating timestamp knowledge into vidco Ilms for enhanced vidco temporal grounding. In AAAI, 2025.
[13] Tanveer Hannan, Md Mohaiminul Islam, Thomas Seidl, and Gedas Bertasius. Rgnet: A unified clip retrieval and grounding network for long videos. ECCV, 2024.
[14] Jiachang Hao, Haifeng Sun, Pengfei Ren, Jingyu Wang, Qi Qi, and Jianxin Liao. Can shuffling video benefit temporal bias problem: A novel training framework for temporal grounding. In ECCV, 2022.
[15] Zhijian Hou, Wanjun Zhong, Lei Ji, Difei Gao, Kun Yan, Wing-Kwong Chan, Chong- Wah Ngo, Zheng Shou, and Nan Duan. Cone: An efficient coarse-to-fin alignment framework for long video temporal grounding. In ACI., 2023.
[16] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In ICLR, 2022.
[17] Kai Hu, Feng Gao, Xiaohan Nie, Peng Zhou, Son Tran, Tal Neiman, Lingyun Wang. Mubarak Shah, Raffay Hamid, Bing Yin, et al. M-Ilm based video frame selection for efficient video understanding. In CVPR, 2025.
11

---

[18] Bin Huang, Xin Wang. Hong Chen, Zihan Song, and Wenwu Zhu. Vtimellm: Empower Ilm to grasp video moments. In CVPR, 2024.
[19] De-An Huang. Shijia Liao, Subhashree Radhakrishnan, Hongxu Yin, Pavio Molchanov, Zhiding Yu, and Jan Kautz. Lita: Language instructed temporal-localization assistant. In ECCV, 2024.
[20] Jinhyun Jang, Jungin Park, Jin Kim, Hycongjun Kwon, and Kwanghoon Sohn. Knowing where to focus: Event-aware transformer for video grounding. ICCV, 2023.
[21] Minjoon Jung. Junbin Xiao, Byoung- Tak Zhang, and Angela Yao. On the consistency of video large language models in temporal comprehension. In CVPR, 2025.
[22] Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, and Juan Carlos Niebles. Dense-captioning events in videos. ICCV, 2017.
[23] Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Reasoning segmentation via large language model. In CVPR, 2024.
[24] Lei, Tamara Berg, and Mohit Bansal. Detecting moments and highlights in videos via natural language queries. In NeurlPS, 2021.
[25] Li, Yuanhan Zhang, Dong Guo, Renrui Zhang. Feng Li, Hao Zhang. Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer. TMLR, 2025.
[26] Xinhao Li, Yi Wang, Jiashuo Yu, Xiangyu Zeng, Yuhan Zhu, Haian Huang, Jianfei Gao, Kunchang Li, Yinan Hc, Chenting Wang, et al. Videochat-flash: Hicrarchical compression for long-context video modeling, arXiv preprint arxiv:2501.00355 2024.
[27] Kevin Qinghong Lin, Pengchuan Zhang. Joya Chen, Shraman Pramanick, Difci Gao, Alex Jinpeng Wang. Rui Yan, and Mike Zheng Shou. Univtg: Towards unified video-language temporal grounding. In ICCV, 2023.
[28] Hanchao Liu, Wenyuan Xue, Yifei Chen, Dapeng Chen, Xiutian Zhao, Ke Wang, Liping Hou, Rongjun Li, and Wei Peng. A survey on hallucination large vision-language models. arXiv preprint arXiv:2402.0022. 00253, 2024.
[29] Liu, Siyuan Li, Yang Wu, Chang- Wen Chen, Ying Shan, and Xiaohu Qie. Umt: Unified multi-modal transformers for joint video moment retrieval and highlight detection. In CVPR, 2022.
[30] Liu, Kevin Qinghong Lin, Chang Wen Chen, and Mike Zheng Shou. Vidcomind: A chain-of-lort agent for long video reasoning. arXiv preprint urXiv:2503.1 13444, 2025.
[31] Yikun Liu, Pingan Chen, Jiayin Cai, Xiaolong Jiang, Yao Hu, Jiangchao Yao, Yanfeng Wang, and Weidi Xie. Lamra: Large multimodal model as your advanced retrieval assistant. In CVPR, 2025.
[32] Ilya Loshchilov and Frank Hutter. Decoupled arXiv:1711.05101 2017.
weight decay regularization.
arXiv preprint
[33] Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. In ACI. 2024.
[34] Boris Meinardus, Anil Batra, Anna Rohrbach, and Marcus Rohrbach. The surprising effectiveness of multimodal large language models for video moment retrieval. arXiv preprint arXiv:2406. 18113, 2024.
[35] WonJun Moon, Sangeek Hyun, SuBeen Lee, and Jae-Pil Heo. Correlation-guided query-dependency calibration for video temporal grounding. arXiv preprint 0.888:231,.08835, 2023.
[36] WonJun Moon, Sangeek Hyun, SangUk Park, Dongchan Park, and Jae-Pil Heo. Query-dependent video representation for moment retrieval and highlight detection. In CVPR, 2023.
[37] Fangzhou Mu, Sicheng Mo, and Yin Li. Snag: Scalable and accurate video grounding. In CVPR, 2024.
[38] Andreca-Manne Maria Oncescu, Joao F Henriques, Yang Liu, Andrew Zisserman, and Samuel Albanie. Queryd: A video dataset with high-quality text and audio narrations. In ICASSP, 2021.
[39] OpenAL Gpt-4o system card. 2024.
[40] OpenAL. 1 system card. https////ooaaa.oooor/////////4/3 2025.
12

---

[41] Mayu Otani, Yuta Nakashima, Esa Rahtu, and Janne Heikkilä. Uncovering hidden challenges in query- based video moment retrieval. In BMVC, 2020.
[42] Junwen Pan. Rui Zhang. Xin Wan, Yuan Zhang, Ming Lu, and Qi She. Timesearch: Hierarchical video search with spotlight and reflection for human-like long video understanding. arXiv preprint arXiv:2504.01407 2025.
[43] Yulin Pan, Xiangteng He, Biao Gong, Yiliang Lv, Yujun Shen, Yuxin Peng. and Deli Zhao. Scanning only once: An end tond-to-nd framework for fast temporal grounding in long videos. In ICCV, 2023.
[44] Renjie Pi, Jiahui Gao, Shizhe Diao, Rui Pan, Hanze Dong, Jipeng Zhang, Lewei Yao, Jianhua Han, Hang Detgpt: Detect what you need via reasoning. In EMNLP, 2023.
Xu, Lingpeng Kong,
[45] Long Qian, Juncheng Li, Yu Wu, Yaobo Ye, Hao Fei. Tat-Seng Chua, Yueting Zhuang, and Siliang Tang. Momentor: Advancing video large language model with fine- geainer temporal reasoning. In ICML, 2024.
[46] Santhosh Kumar Ramakrishnan, Ziad Al-Halah, and Kristen Grauman. Naq: Leveraging narrations as queries to supervise episodic memory. CVPR, 2023.
[47] Michaela Regneri, Marcus Rohrbach, Dominikus Wetzel, Stefan Thater, Bemt Schiele, and Manfred Pinkal. Grounding action descriptions in vidcos. TACL, 2013.
[48] Shuhuai Ren, Linli Yao, Shicheng Li, Xu Sun, and Lu Hou. Timechat: A time-sensitive multimodal large language model for long video understanding. In CVPR, 2024.
[49] Xiaoqian Shen, Yunyang Xiong, Changsheng Zhao, Lemeng Wu, Jun Chen, Chenchen Zhu, Zechun Liu, Fanyi Xiao, Balakrishman Varadarajan, Florian Bordes, et al. Longvu: Spatiotemporal adaptive compression for long video-language understanding. arXiv preprint arXiv:2410.
2024.
[50] Yudi Shi, Shangzhe Di, Qirui Chen, and Weidi Xie. Enhancing video-IIm reasoning via agent-of oftho-of-thoughts distillation. In CVPR, 2025.
[51] Gunnar A Sigurdsson, Gül Varol, Xiaolong Wang. Ali Farhadi, Ivan Laptev, and Abhinav Gupta. Hollywood in homes: Crowdsourcing data collection for activity understanding. In ECCV, 2016.
[52] Xi Tang, Jihao Qiu, Lingxi Xie, Yunjie Tian, Jianbin Jiao, and Qixiang Ye. Adaptive keyframe sampling for long video understanding. In CVPR, 2025.
[53] Yansong Tang, Dajun Ding, Yongming Rao, Yu Zheng, Danyang Zhang, Lili Zhao, Jiwen Lai, and Jie Zhou. Coin: A large-scale dataset for comprehensive instructional video analysis. In CVPR, 2019.
[54] ByteDance Seed Team. Seed technical report. arXiv preprint arXiv:2505.07062 2025.
[55] Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai. Anmol Gulati, Garrett Tanzer. Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint urXiv:2403.055300 2024.
[56] Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu. Jialin Wang. Wenbin Ge, et al. Qwen2-vl: Enhancing Dision-languee model's perception the world any resolution. arXiv preprint arXiv:2409. 12191, 2024.
[57] Haoning Wu, Dongxu Li, Bei Chen, and Junnan Li. Longvideobench: A benchmark for long-context interlcaved vidco-language understanding. In NewrlPS, 2024.
[58] Shen Yan. Xuehan Xiong, Arsha Nagrani, Anurag Arnab, Zhonghao Wang. Weina Ge, David Ross, and Cordelia Schmid. Unloc: A unified framework for video localization tasks. In ICCV, 2023.
[59] An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong Haoran Wei, Huan Lin, Jialong Tang. Jialin Wang. Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng Hc, Junyang Lin, Kai Dang, Keming La, Keqin Chen, Kexin Yang. Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang. Ru Peng, Rui Men, Ruize Gao, Runji Lin, Shijie Wang. Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li. Tianyu Liu. Wenbin Ge, Xiaodong Deng. Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei. Xuancheng Ren, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zhihao Fan. Qwen2 technical report. arXiv preprint arXiv:2407.1066 10671, 2024.
[60] Jinhui Ye, Zihan Wang, Haosen Sun, Keshigeyan Chandrasegaran, Zane Durante, Cristobal Eyzaguirre. Yonatan Bisk, Juan Carlos Niebles, Ehsan Adeli, Li Fei-Fei, et al. th-thin temporal search for long-form video understanding. In CVPR, 2025.
13

---

[61] Abhay Zala, Jaemin Cho, Satwik Kottur, Xilun Chen, Barlas Oguz, Yashar Mehdad, and Mohit Bansal. Hierarchical video-moment retrieval and step-captioning. CVPR, 2023.
[62] Xiangyu Zeng, Kunchang Li, Chenting Wang. Xinhao Li, Tianxiang Jiang. Ziang Yan, Songze Li, Yansong Shi, Zhengrong Yue. Yi Wang. al. Timesuite: Improving milms for long video understanding via grounded tuning. In ICLR, 2024.
[63] Boqiang Zhang, Kehan Li, Zesen Cheng, Zhiqiang Hu, Yugian Yuan, Guanzheng Chen, Sicong Leng, Yuming Jiang, Hang Zhang, Xin Li, Videollama 3: Frontier multimodal foundation models for image and video understanding. arXiv preprint wrXiv:2501.33100 2025.
[64] Hang Zhang, Xin and Lidong Bing. Video-Ilama: instruction cecuuttto---needddeeeeeee audio-visual language model for video understanding. arXiv preprint arXiv:2306.02858 2023.
[65] Hao Zhang, Aixin Sun, Wei Jing, and Joey Tianyi Zhou. Span-base localizing network natural language video localization. In ACL, 2020.
[66] Songyang Zhang, Houwen Peng, Jianlong Fu, and Jiebo Luo. Learning temporal adjacent networks for moment localization with natural language. In AAAI, 2020.
[67] Yiming Zhang, Zhuokai Zhao, Zhaorun Chen, Zenghui Ding, Xianjun Yang, and Yining Sun. Beyond training: Dynamic token merging for zero-shot video understanding, arXiv preprint arXiv:2411.14401 2024.
[68] Yuanhan Zhang, Bo Li, haotian Liu, Yong jae Lee, Liangke Gui, Di Fu, Jiashi Feng, Ziwei Liu, and Chunyuan Li. Llava-next: A strong vero-shot video understanding model, April 2024.
[69] Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma. Ziwei Liu, and Chunyuan Li. Video instruction tuning with synthetic data. arXiv preprint 2:110:2410.02713, 2024.
[70] Pengcheng Zhao, Zhixian He, Fuwei Zhang. Shujin Lin, and Fan Zhou. Ld-detr: Loop decoder detection transformer for video moment retrieval and highlight detection. arXiv preprint arXiv:2501.10780 2025.
[71] Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Shitao Xiao, Xi Yang, Yongping Xiong, Bo Zhang, Tiejun Huang, and Zheng Liu. Mlvu: A comprehensive benchmark for multi-task long video understanding. In CVPR, 2025.
[72] Luowei Zhou, Chenliang Xu, and Jason Corso. Towards automatic learning of procedures from web instructional videos. In AAAI, 2018.
14

---

Appendix/ supplemmntalllll material
In the appendix, we provide additional experimental details (Appendix A implementation de- tails (Appendix(B) qualitative results (Appendix C grounding consistency evaluation (Appendix D)
more ablation studies (Appendix EE. discussions of limitations and future work (Appendix F. and broader impacts (Appendix G
A Experimental Details
In this section, we provide additional experimental details, including benchmark details (Ap- pendix A.I). ablation details (Appendix A.2), robustness test details (Appendix A.. and downsteam task details (Appendix A.4).
A.1 Benchmark Details
To evaluate the performance of our model on temporal grounding and VideoQA tasks, we employ three key metrics: Intersection-over- Union (loU), Intersection-over-Peeeiii Predcctttn (IoP), and Intersection- over-Ground Truth (IoG). These metrics quantify the alignment between the predicted temporal window 7ered and the ground truth temporal window 7 The formal definitions are as follows:
7
n 7
P n T
[ nT
IoU
IoP
loG
7 U T
[
7
Here, [TPreed n T denotes the duration of the overlapping region between the predicted window Tpred and the ground truth window 78, [Tpred U TT8| represents the duration of their union, while [7rrod] and |7 denote the durations of the predicted and ground truth windows, respectively.
IoU measures the overall alignment between Trd and 7 providing a balanced assessment of both precision and recall by considering the overlap relative their union. IoP emphasizes precision, indicating the proportion of the predicted window that overlaps with the ground truth. Conversely, loG focuses on recall, reflecting the proportion of the ground truth window that is covered by the prediction. Collectively, these metrics offer comprehensive evaluation of the model's ' ability to accurately localize temporal windows with respect the ground truth.
A.2
Ablation Details
A.2.1 Effect of Individual Modules
As shown in Table 7 we systematically evaluate the effect of each individual module. The detailed settings are as follows: (i) Row 1: We uniformly sample 32 frames from each video and obtain the corresponding timestamps. These frames and timestamps are then fed into UniTime for temporal grounding. (ii) Row 2: We sample the video at 2 fps and apply frame scaling. The sampled frames and their timestamps are input into UniTime for temporal grounding. (iii) Row 3: On top of uniform sampling, we introduce multi-stage inference. Specifically, after obtaining an initial prediction using the uniformly sampled frames (as in row 1), we perform second round of uniform sampling within the predicted interval and feed these frames into the model for refined localization. (iv) Row 4: Similarly, we apply multi-stage inference the adaptive frame scaling. After the initial prediction (as in row 2), we resample at 2 fps and apply frame scaling within the predicted interval, then input these frames into the model for further refinement. (v) Row 5: We adopt the coarse-to-fine temporal grounding strategy described in Section 2.2 Unlike row 4, where the model predicts fine-grained intervals in the first round due to densely sampled timestamps, row partitions the video into coarse segments and only requires the model to predict the relevant segment in the first round. In the second round, temporal grounding is performed within the selected segment localize the precise moment.
A.2.2 Effect of Segment Length and Replication Factor
we decouple the process of coarse-to-fing temporal grounding to analyze the effect of segment length and replication factor. Below, we provide detailed descriptions of the capabilities of each component after decoupling.
As shown in Figure 3
15

---

Overall, the coarse- to- lo-fine temporal grounding process can be divided into the following two stages. First, we retrieve candidate segments from the video as follows:
( UniTime 7, UnniTime (; S T2; .. ;N,, s 2)
where S, denotes the retrieved segments and N, is the number of retrieved segments. Next, perform fine-grained localization within the retrieved segments:
A (1s ...,, (sK, {k)) UniiTime (V (s,,,,, T, Q)
where each 8k, Cg E T represents the predicted start and end timestamps, respectively. Then, the evaluation metries used in Section 3.5 and Figure are defined follows:
Segment retrieval accuracy @R@1 I) indicates the accuracy of segment retrieval, i.c., the proportion of cases where any predicted segment S.. is contained within the set of ground-truth segments (S²), Oracle grounding accuracy (oracle R1 @0.3) reflects the localization performance when the model provided with the ground-truth segment as input, i.e., , U (V { T, Q). Overall grounding accuracy (R1 @0.3) measures the and-to-end temporal grounding performance, where (s:rd T, Q).
Aprd PUniTime
A.3
Robustness Test Details
A.3.1 Time Shift Event
The temporal distribution bias in Charades-STA is illustrated in Figure 4 In addition, we also conduct statistical analyses on TACoS, ANet-Captions, and QVHighlights, with the results presented in Figure 5 Significant temporal distribution bias is also observed in TACoS and ANet-Captions, whereas @Viihlighhts does not exhibit such bias.
Relative positioa of ovent ead
Relative position of
Relative position of event end
os 10
os Relative position
16
position
position
(a)
(b)
(c)
Figure 5: Temporal distribution bias in (a) ANet-Captions. (b) QVHighlights. (c) TACoS. Darker colors indicate higher data density. and the yellow curve outlines the region of greatest density.
A.3.2 Decomposition of Query
The prompts provided to Qwen2 [59] for decomposing each Charades-STT query into set of object-centric questions are as follows:
System:
You are Quen, created Alibaba Cloud. You helpful assistant.
Syatem:
Analyze the given query and:
1. Identify ONLY concrete, specific objects (nouns) that are:
Tangible physical items
Clearly named (not pronouns/ambiguuus)
STRICTLY EXCLUDE:
All human references (person, he, she, they, etc.) Ambiguous terma (something, anything, things, etc.) Pronouns (it, they, them)
Abstract concepts For each valid object, generate EXACTLY ONE question:
"When does (OBJECT] appear?"
Negative Examples (BAD)
16

---

Input: "Someone left some items the furniture"
Wrong Output:
Whhen does the furniture appear?
-heen does some items appear? # <-- AMBIGUOUS TERM SHOULD BE EXCLUDED
Positive Examples (GOOD):
Input: "The machine proceased the materials during the night"
Output:
Wheen does the machine appear?
When does the raw materials appear?
When does the night appear?
User:
Analyze: <query>
A.4
Downstream Task Details
A.4.1 VideoQA with Temporal Grounding
We use Qwen2-VL-7B as the VideoQA model for answer generation. By default, it processes long videos by uniformly sampling 32 frames. However, this sampling strategy may lead to the omission of critical information. To investigate whether temporal grounding models can compensate for this issue, we adopt the following procedure. First, we use different video temporal grounding models to localize the relevant segments for each question. Then, we crop the localized video intervals and input them into Owen2-VL-77 Specifically, for cropped video segments shorter than 32 seconds, we extend their duration from the center to 32 seconds. Within each interval, we again uniformly sample 32 frames for answer generation.
A.4.2 Prompt template for VideoQA
We use the same prompt template for all multiple-choice VideoQA benchmarks:
Syatem:
You are helpful assistant.
User:
<yideo>
Question: <question>
Options:
(A) <Option_A>
(B) <0ption_B>
(C) <0ption_C>
(D) <0ption_D>
Please only give the best option.
Best Option:
Assistant:
B Additional Implementation Details
B.1 Prompt template for UniTime
The prompt template used for coarse-grained segment retrieval as follows:
System:
You are helpful assistant.
User:
<timestamps frames>
This is sequence interleaved with timestamps and frames.
Your task is to identify the specific timestamp(s) (s) when the given query appears.
Query: <Query>
Answer:
Assistant:
The prompt template used for fine-gra finennr temporal grounding follows:
17

---

System:
You helpful assistant.
User:
stimestamps frames>
This is sequence interleaved with timestamps and frames. Your task is to identify the temporal window (start and end timestamps) when the given query appears.
Query: <Query?
Anawer:
Assistant:
B.2 Video-centric Training
In video-centric training, we first sample video from the dataset and gather all associated queries with their corresponding answers. These queries and answers are concatenated and paired with the video as input, forming sequence such as
[U1, 122 ...., UN; Q1, A1, Q2 A2...., Nampee A Namgie
As illustrated in Figure 6 we apply attention masks to prevent different queries and answers from attending to others. Additionally, each query-answer sequence is assigned the same starting Rotary Position Embedding index, immediately following the indices of the video tokens.
@
0:
Ag
RoPE Index
N+1
N+2
N+1
N+2
N+1
N+2
Figure Illustration of the video-centric training paradigm.
Qualitative Results
C.1 Qualitative Results for Ego4D-NLQ
In Figure present qualitative results of UniTime Ego4D-NLQ. In the success cases, UniTime demonstrates precise coarse-gras coarse-grained grained segment retrieval and fine-grained temporal grounding capabil- ities. Despite the limited variation in video scenes, which are primarily composed of fine-grained interactions between hands and objects, the model consistently demonstrates robust performance. In the failure cases, although the model retrieves the correct segment, it fails in fine- fine-grained localization. As shown in the specific localization frames, this error occurs because the object referred to in the query appears multiple times in the video, while the ground truth only includes of these occurrences.
C.2 Qualitative Results for ANet-Captions
As shown in Table 2. UniTime achieves substantial performance gains over competing methods across all benchmarks, except for ANet-Captions. We analyze this discrepancy below, with Figure visualizing UniTime's predictions ANet-Captions.
In successful cases (Figure & a)), UniTime achieves precise temporal grounding. Despite the limited diversity of video scenes, which predominantly feature fine-grained interactions between hands and objects, the model consistently exhibits robust performance.
However, our analysis of the failure cases indicates that most can be traced limitations ANet- Captions's 's annotation scheme. As illustrated in Figure (]))), multiple occurrences of the same event are often annotated with only a single temporal interval. This occurs because captions are ordered sequentially, causing annotators overlook recurring events.
Additionally, many failures arise from incorrect labels, as shown Figure C)c.
D Grounding Consistency Evaluation
To further evaluate UniTime's robustness in temporal grounding, we conduct grounding consis- tency assessment Charades-CON and ActivityNet-CON (21, which are curated from subsets
18

---

Query: Where did I throw the trash from the plate?
Coarse-grained Prediction
Os
480.0s
317.0s
349.05
340.6s
343.5s
317.0s
340.6s
343.6s
349.0s
Fine-grained Prediction
case
Query: Where is the paper towel?
Coarse-grained Prediction
Os
480.0s
448.0s
480.0
464.9s
467.0s
448.0s
473.0
476.0s 480.0s
Fine-grained Prediction
(b) failure case
Figure 7: Qualitative Results for long-video temporal grounding the Ego4D-NLO benchmark. Red indicates the ground truth (GT), blue denotes the results of coarse-grained segment retrieval, and green represents the results of fine-grained temporal grounding.
19

---

Query: A woman rolls towel.
13.3s
20.8s
0.0s 13.95
20.7s
115.8s
(a) Success case.
Query: man starts break dancing on the floor.
30.4s
43.4s
0.0s
127.2s
143.8s 173.6s
(b) Failure case resulting from incomplete annotations of multi-hop events.
Query: The woman then lights candle.
34.3s
93.4s
0.0s 21.7s
29.9s
190.6s
(c) Failure case caused by incorrect labels.
Figure 8: Qualitative Results for short-video temporal grounding on the ANer-Captions benchmark. Red indicates the ground truth (GT), blue denotes the results of coarse-grained segment retrieval, and green represents the results fine-grained temporal grounding.
20

---

Table 10: Grounding consistency evaluation Video-MILLMs, Relative consistency scores are in
brackets.
Charades-CON
AcdivityNet-CON
Metbod
Ground
R-Ground
S-Ground
Ground
R-Ground
S-Ground
GPT-4o
28.5
21.2 (74.3)
(32.8)
26.8
18.1 (67.5)
108.8
Gemini Flash 551
34,6
29.7 (85.7)
24.8 (7L.)
37.8
30.8 (81.4)
24.8 (65.6)
Video-LLaMA [4
14.2
(74.9)
(37.6)
12.8
8.5 (66.8)
7.2 (56.8)
Video-LLaMMA VThine
54.4
(70.))0.3)
(20.0)
33.0
24.7 (74.8)
(30.2)
TimeChat [8]
30.5
(82.
(18.5)
4.6
(.9.))111
(.0
TimeChut VTune
21
76.2
69.2 (90.8)
(47.5)
37.4
275.6
10.5 (28.3)
Uni Time-Plll
81.5
(90.7)
(71.6)
(01.5)))))
(43.4)
of CharadesSTA and ActivityNeer-Catiinns, respectively. The grounding consistency targets two complementary consistency dimensions:
i) Rephrased Grounding (R- (R-Groun)). This metric evaluates whether a model's moment predictions remain consistent across query q and its aligned rephrased variant q. Concretely, the loU between m = Tempo (q) and m = ( is computed. For each original query, the average IoU over three aligned variants is reported, reflecting the alignment consistency of predicted moments under paraphrasing.
ii) Shifted Grounding (S (S-Ground). This metric assesses whether model consistently grounds the same visual content when that content is shifted to different temporal positions. Specifically, while preserving the internal frame sequence of the ground-truth moment mgt, it is randomly shifted to new moment M,, with evaluations conducted on the resulting predictions.
Experimental results, summarized in the Table 10 demonstrate that UniTime not only exhibits strong temporal grounding performance (Ground) but also achieves excellent consistency and robustness (R-Ground and S-Ground). UniTTime attains performance comparable to VTTune (21), which extends instruction tuning with explicit consistency objectives, and in some cases surpasses
E
More Ablation Studies
E.1 Effect of Video Processing Strategies
To justify design choice in Section 2.2 i.e., resizing short videos and compressing long videos, we conduct an ablation study on the Hgo4D-N D-NLQ dataset. This study evaluates how different video processing strategies impact both grounding performance and computational efficiency.
Table 11: Ablation video processing strategies. Seg Retrieval Acc. indicates the accuracy of coarse- coarse-grained segment retireval.
Short Video
Long Video
mloU
Seg Retrieval Acc.
Training Time
Frame Resizing
Token Compression
24.79
16.83
17.25
45.28
21 h
Frame Resizing
Frame Resizing
18.53
12.67
12.88
35.30 35.36
12 h
Token Compression
Token Compression
24.06
16.18
16.71
45.28
24.5 h
Our findings indicate that: (i) For long videos, adaptive frame scaling reduces tokens per frame while keeping the frame rate fixed. Though some visual cues are lost, this fcature-Ievel token compression retains more semantic information than frame resizing, which discards input resolution. This leads to better segment retrieval accuracy and recall (row vs. row 2). (ii) For short clips with sufficient tokens, both methods perform similarly (row vs. row 3). However, token compression requires high-resolution inputs before merging, incurring higher computational costs. In UniTime-Fuil using token compression for short videos increased training time by days compared to frame resizing.
Therefore, rather than employing unified strategy, we adopt a hybrid approach to balance perfor- mance and efficiency: for short videos, we apply frame resizing to reduce computational overhead, while for long videos, we leverage token compression better preserve visual details. This design choice enables method to achieve strong performance with practical training costs.
21

---

E.2
Comparison of Temporal Information Encoding
We conduct ablation experiments on Charades-STT to compare the effectiveness of Timestamp Token Insertion versus Dense Position Encodings.
Table 12: Ablation on temporal information encoding.
Method
R1 @.55 5
R1@.7
mloU
Dense Position Encodings
48.44 74.33
27.15
44.85
Timestamp Token Insertion (UniTime-SP)
53.71
63.15
Our base model, Qwen2-VL- VL-7B, utilizes Multimodal Rotary Position Embedding (MRoPE), which decomposes position embeddings into temporal, height, and width components. After fine-tuning, temporal grounding remains suboptimal with dense position encodings. contrast, timestamp token insertion yields significant gains. We attribute this improvement to the MLLM's strong multimodal retrieval ability and the simplification of the task into (i) identifying question-relevant video segments and (ii) directly reading their corresponding timestamp tokens.
E.3 Ablation Study of Adaptive Frame Scaling Hyperparameters
In this section, conduct ablation study the threshold hyperparameters // Nyyher and the token budget Noval within Adaptive Frame Scaling, elucidating the rationale behind our parameter choices.
Threshold Nooog This parameter specifies the maximum video length that can be processed in a single pass. For videos with length < N we process the input directly; for videos with length > _ we partition the input into segments, process each segment independently, and then aggregate the predictions. shown in Table varying No has only marginal effect on accuracy because over-length inputs are segmented and subsequently fused. Moreover, Adaptive Frame Scaling maintains a fixed token budget per segment, so reducing Nrlong increases the number of segments and the total token count inference, thereby incurring higher latency. Balancing efficiency and accuracy, we set Nolong 1024.
Threshold Nehort, This hyperparameter determines whether inference is executed in a single stage via multi-stage (coarse-to-fing -one-ee) pipeline. For inputs with temporal length < Nyphon we single-stage processing; for inputs with temporal length >> Nyyhor, we employ multi-stage processing. As shown in Table I4 within the 64-128 duration range, the performance difference between multi-stage (N -64) and single-stage (Nsboot 2 128) configurations is modest. However, lower threshold increases average inference latency by routing more inputs to the multi-stage pipeline. For inputs in the 128-256 range, multi-stage processing yields substantially higher accuracy (54.57 vs. 42.90 mloU). To balance accuracy and efficiency, we set N =128.
Token budget Nootal This parameter specifies the maximum number of tokens permitted. As shown in Table [5] performance improves monotonically as the token budget increases, up to the available GPU capacity. Accordingly, we set Niotal = 16, 384, which corresponds to the upper limit supported by our hardware.
F Limitations & Future Work
While UniTime demonstrates exceptional performance on various video temporal grounding and video QA benchmarks, still has several limitations that warrant further exploration: (i) UniTime is currently constrained to temporal grounding tasks. To enable broader applications in MLLMs, requires more diverse training data with dense temporal annotations. Incorporating such data into the pretraining process of MILMs could unlock their potential for handling more temporally complex tasks, such as dense video captioning. (ii) Although UniTime enhances MILMs with temporal grounding capabilities, relying solely on temporal grounding data limits their reasoning and
22

---

Table 13:
Ablation the threshold N on Ego4D.
Table 14: Ablation of the
Table 15: Ablation of the to-
threshold Nyhort on TaCoS.
ken budget Notal on Ego4D.
RI@.3
RI@ 5
mloU
Npphre
mloU (64-1286)
mloti (128-2566)
mloU
Budget
R1 .
RI@.5
mloU
Avg.
1024
512
24.72
24.79
16.63
16.83
17.25
16.95
128
64
56.59
58.43
54.57
54.57
50.00
50.28
16384
8192
24.08
24.79
16.10
16.83
17.25
16.69
256
23.94
16.10
256
56.39
42.90
46.82
4096
23.04
15.07
15.96
question-answering abilities. The ultimate objective is to develop MLLMs that seamlessly integrate localization, reasoning, and question-answwe answering into unified framework. (iii) The current approach to processing long videos in UniTime relies on a fixed segment length, which lacks flexibility. Future work could investigate adaptive segment lengths based on the information density of videos, enabling more efficient and context-aware video processing.
G Broader Impacts Statement
Our research advances the field of video understanding by developing more accurate and efficient models for temporal grounding and video question answering. While temporal grounding with UniTime offers significant benefits for variety of video understanding applications, unexpected behaviors may occasionally result in misrepresentaatins of the underlying video content. For appli- cations that demand extremely precise temporal localization for safety-critical decisions, such as embodied intelligence tasks, it is crucial to carefully manage any such behaviors. To ensure the relia- bility of systems that rely on temporal grounding predictions, we recommend conducting thorough evaluations and implementing robust mitigation strategies to minimize potential risks. Through these precautions, the overall safety and effectiveness of these applications can substantially enhanced.
23

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction accurately reflect the main contributions.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The limitation discussed in Appendix
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
The paper provides a detailed of the experimental settings, which could be referred to Section and Appendix A
Justification:
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
25

---

Answer: [Yes]
Justification:
The code is released at /t/tttps///gtthuu.commm/
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ /pipps.ccppullc//uuuii wwwpc//ggue////eeeeeoooc CoodeSubeiiieeee ssi wsionnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The details are provided in Section 3 and Appendix Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: This paper does not provide the numerical experiment statistical significance.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
26

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please refer Section
3
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: http::///euisp..eeeee http://uurriiscccppppppp https///ccpcccctcccccccceeeeeee
Answer:
[Yes]
Justification:
We have read and strictly followed the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the broader impacts in Appendix G
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
27

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
The paper uses only open-source and publicly available datasets, which do not contain any data with high risk of misuse. Additionally, all models used in the paper open-source and do not involve any potential misuse risks.
Justification:
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We have properly cited the original paper that provided public data, models, and codes.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. If this information not available online, the authors are encouraged to reach out to the asset's 's creators.
28

---

13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes)
Justification: We have introduced a new model in this paper and have released this model at https://github: com wwt.c/ow.../ieeee Lttp/////ii/nnn
Guidelines:
The answer NA means that the paper does release assets.
Researchers should communicate the details the dataset/cooeemmodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (TRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
29

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [Yes]
Justification: Please refer Sec and Appendix.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy fattps://neurd/: /ttpa/rrpp..ccooooffeeeee/e///// for what should or should not be described.
30


Reviews:
[{"review_id": "nMwUfmqBQe", "reviewer": "Reviewer_YJyw", "summary": "This paper introduces UniTime, a versatile Multimodal Large Language Model (MLLM) designed for universal video temporal grounding. The primary goal of UniTime is to accurately localize events within videos of varying lengths, genres, and viewpoints based on natural language queries. There are three core contributions: 1) an adaptive frame scaling mechanism that dynamically adjusts frame resolution based on video duration to manage computational load while preserving semantic information; 2) a timestamp-interleaved sequence representation that explicitly integrates temporal information by interleaving textual timestamp tokens with visual frame tokens; and 3) a coarse-to-fine grounding strategy for long videos, which first identifies a broad temporal region of interest and then performs precise localization within that segment.", "questions": "Please list up and carefully describe questions and suggestions for the authors, which should focus on key points (ideally around 3–5) that are actionable with clear guidance. Think of the things where a response from the author can change your opinion, clarify a confusion or address a limitation. You are strongly encouraged to state the clear criteria under which your evaluation score could increase or decrease. This can be very important for a productive rebuttal and discussion phase with the authors.\n1.\tClarifying the Novelty of the Timestamp-Interleaved Sequences. Could you please elaborate on the specific novelty of your timestamp-interleaved sequence representation compared to prior methods?\n2.\tResolving the Contradiction in Frame/Token Compression. The manuscript states that token compression is preferable to frame resizing for preserving semantic information, which raises the question of why frame resizing is still a necessary component of your pipeline.\n3.\tJustification and Sensitivity Analysis for Adaptive Scaling Hyperparameters.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Please provide a thorough assessment of the strengths and weaknesses of the paper. A good mental framing for strengths and weaknesses is to think of reasons you might accept or reject the paper. Please touch on the following dimensions: Quality, Clarity, Significance, and Originality. For more information, please see the NeurIPS 2025 Reviewer Guidelines (https://neurips.cc/Conferences/2025/ReviewerGuidelines). You can incorporate Markdown and LaTeX into your review. See https://openreview.net/faq.\n\n1.\tStrengths\na.\tWell-Designed and Practical Method: The proposed UniTime framework effectively combines several well-motivated ideas. The adaptive frame scaling is a practical solution to the context-length limitations of MLLMs. The coarse-to-fine inference strategy is an intuitive and effective way to handle long videos hierarchically.\nb.\tComprehensive Evaluation and Exceptional Results: The experimental setup is a major strength of this work including diverse benchmarks, multiple settings and robustness tests. The paper presents very strong and consistent results across a wide array of benchmarks. UniTime substantially outperforms previous state-of-the-art models in all three evaluation settings: dataset-specific fine-tuning, universal, and zero-shot.\n2.\tWeaknesses\na.\tTimestamp-Interleaved Novelty. The paper acknowledges that other recent models like Mr.BLIP [31], TimeMarker [6], and VideoLLaMA3 [56] also use explicit textual timestamps. The manuscript should highlight specific novelty of UniTime's timestamp-interleaved sequence representation in comparison to these prior works.\nb.\tAmbiguity in the Rationale for Frame Resizing versus Token Compression: There appears to be a point of confusion in the justification for using both frame resizing and token compression. The paper states (L102-104): “Unlike frame resizing, which may degrade spatial details, token compression reduces redundancy at the token level while better preserving key semantic information…”. This statement raises the question of why frame resizing is necessary if token compression is a superior method for preserving semantic information. \nc.\tLack of Justification for Hyperparameter Choices: The adaptive frame scaling mechanism is governed by two critical thresholds, N_f^short  and N_f^long , which are set to 128 and 1024, respectively. The paper does not provide a justification or an ablation study for these specific hyperparameter choices. The model's performance may be sensitive to these values, and their selection appears somewhat arbitrary.", "was_revised": true, "final_justification": "The author resolved some of my concerns but the paper still contains many weakness. I keep the score based on the authors response.", "rebuttal": "Thank you for your constructive comments. Our responses are provided below.\n\n---\n**Q1: Novelty of Timestamp-Interleaved Sequences**\n\nThank you for the feedback. We would like to clarify the key novelty of UniTime’s timestamp-interleaved sequence representation as follows:\n\n- **Dynamic Timestamp Resolution:** Unlike prior methods that rely on fixed per-frame insertion, UniTime introduces dynamic resolution control. Timestamp tokens can be interleaved at multiple scales—either per frame for fine-grained predictions or per segment for coarse-grained reasoning. This adaptability enhances flexibility in temporal modeling (see Section 2.2, Timestamp-Interleaved Sequence Construction).\n\n- **Multi-Scale Temporal Predictions:** Coupled with adaptive frame scaling, UniTime supports coarse-to-fine inference over varying video lengths. This improves grounding accuracy on long videos, addressing limitations of prior methods that rely on sparse sampling or short clips.\n\n\n---\n\n**Q2: Explanation for Frame Resizing and Token Compression Choices**\n\nThank you for the insightful comments. To resolve the ambiguity, we conducted an ablation study on the Ego4D-NLQ dataset, evaluating the effects of different video processing strategies on grounding performance and computational efficiency:\n\n|Short Video|Long Video|R1\\@0.3|R1\\@0.5|mIoU|Seg Retrieval|Training Time|\n|-|-|-|-|-|-|-|\n|frame resize (default)|token compression|24.79|16.83|17.25|45.28|21 h|\n|frame resize|frame resize|18.53|12.67|12.88|35.36|12 h|\n|token compression|token compression|24.06|16.18|16.71|45.28|24.5 h|\n\nOur findings indicate that:\n\n- **Long Video:** Adaptive Frame Scaling reduces tokens per frame while keeping the frame rate fixed. Though some visual cues are lost, this feature-level token compression retains more semantic information than frame resizing, which discards input resolution. This leads to better segment retrieval accuracy and recall (Rows 1 vs. 2).\n\n- **Short Video:** For short clips with sufficient tokens, both methods perform similarly (Rows 1 vs. 3). However, token compression requires high-resolution inputs before merging, incurring higher computational costs. In UniTime-Full, using token compression for short videos increased training time by 5 days compared to frame resizing.\n\nTherefore, instead of a unified strategy, we adopt a hybrid approach to balance performance and efficiency: frame resizing for short videos (for computational efficiency) and token compression for long videos (to preserve visual details). This design achieves strong performance with practical training costs.\n\n---\n\n**Q3: Ablation Study of Adaptive Frame Scaling Hyperparameters**\n\nThank you for the thoughtful feedback. We have conducted a systematic ablation study on two key hyperparameters:\n\n- **Threshold $N_f^{long}$**\n    - **Role**: Determines the maximum video length that can be processed directly\n        - If video length ≤ $N_f^{long}$: Process directly.\n        - If video length > $N_f^{long}$: Split into segments, process separately, and aggregate results.\n\n    - **Evaluation on Ego4D-NLQ**:\n\n        | $N_f^{long}$   | R1\\@0.3 | R1\\@0.5 | mIoU  |\n        | -------------- | ------- | ------- | ----- |\n        | 1024 (default) | 24.79   | 16.83   | 17.25 |\n        | 512            | 24.72   | 16.63   | 16.95 |\n        | 256            | 23.94   | 16.10   | 16.68 |\n    \n    - **Key Findings**:\n        - The choice of $N_f^{long}$ has minimal impact on model accuracy because our method segments over-length videos and aggregates results. \n        - Adaptive Frame Scaling enforces a fixed token budget per segment. Consequently, a smaller $N_f^{long}$ increases inference time due to more video segments and more input tokens.\n        - **Default Choice**: 1024 for optimal efficiency-performance balance.\n\n- **Threshold $N_f^{short}$**\n    - **Role**: Decides single- or multi-stage processing:\n        - If video length ≤ $N_f^{short}$: Single-stage\n        - If video length > $N_f^{short}$: Multi-stage (coarse-to-fine)\n    - **Evaluation on TaCoS**:\n        | $N_f^{short}$ | mIoU (64-128s) | mIoU (128-256s) | Avg. mIoU |\n        | ------------- | -------------- | --------------- | --------- |\n        | 64            | 58.43 (multi)  | 54.57 (multi)   | 50.28     |\n        | 128 (default) | 56.59 (single) | 54.57 (multi)   | 50.00     |\n        | 256           | 56.59 (single) | 42.90 (single)  | 46.82     |\n    \n    - **Key Findings**:\n        - For 64-128s videos: Minimal performance difference between multi-stage ($N_f^{short}$=64) and single-stage ($N_f^{short}$>=128) processing (ΔmIoU=1.84). However, the lower threshold increases average inference time as it forces more videos into multi-stage processing.\n        - For 128-256s videos: Multi-stage processing yields significantly better results (54.57 vs 42.90 mIoU)\n        - **Default Choice**: 128 to balance performance and efficiency.\n\n---\n\nWe will incorporate the above clarifications, experiments, and analyses into our final draft."}, {"review_id": "CD25khotAC", "reviewer": "Reviewer_Zb87", "summary": "The paper proposes an approach for temporal grounding of videos where the short and long videos are treated in a unified framework. In particular, keeping the token budget fixed, spatial granularity is scaled accordingly (lower for long videos; and vice versa). The paper, in addition, leverages techniques such as interleaved timestamp tokens, coarse-to-fine search scheme, and train-time optimization to push the performance over existing MLLM-based approaches", "questions": "see weaknesses", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "### Strengths\n- Writing and presentation of the results in the paper is clear\n- Ablation study is highly informative of the design choices\n\n\n### Weaknesses\n- Highlighting the pre-training dataset (type, quantity, etc.) of the related works in Table 3 would have been helpful to draw more nuanced conclusions\n- An ablation on the design choice of lines 120-121 would be great: “we leverage the retrieval capabilities of MLLMs to read out the inserted timestamp tokens, rather than decoding dense position encodings”. Or point to an existing work that did\n- Eq 2 doesn’t contain “end” timestamps of the segments. Do the authors have an impression of whether such an input would be effective?\n\nMinor\n- Negative numbers are highlighted in green, giving a false first impression that they are positive", "was_revised": true, "final_justification": "The authors addressed my concerns on (a) reporting the pre-training datasets of prior approaches to draw head-to-head comparisons, and (b) the ablation on explicit timestamp insertion (proposed) vs. positional encodings.\n\nI suggest the authors include the above in the final version of the paper.", "rebuttal": "Thanks for your positive comments! We provide the feedback as follows.\n\n---\n\n**Q1: Pre-training Datasets of the Related Works**\n\nThank you for your valuable suggestion. Here, we summarize the pre-training datasets used by the related works. Some of the methods (like TimeMarker) are trained on a dozen datasets. Due to space constraints, we will provide the complete list in the final draft.\n\n|Method|Pre-training datasets|Sample|Duration|\n|-|-|-|-|\n|UniVTG|Ego4D, VideoCC, YouTubeHL, ...|4.3M|Long & Short\n|VTG-LLM|VTG-IT-120K, Charades, Youcook2, ...|97K|Short|\n|Momentor|Moment-10M|10M|Long|\n|VTimeLLM|InternVid-10M-FLT, VideoInstruct100K, DiDeMo, ...|170K|Short|\n|Timechat|TimeIT (DiDeMo, QueryD, HiREST, ...)|125K|Short|\n|TimeMarker|ShareGPTVideo, ShareGPT4Video, VideoChat2-IT, ...|5.4M|Long & Short|\n|TimeSuite|TimePro (DiDeMo, QueryD, HiREST, ...)|349K|Long & Short|\n|UniTime|NaQ, Momentor, Ego4d-NLQ, ...|1.3M|Long & Short|\n\n---\n\n**Q2: Timestamp Token Insertion vs. Dense Position Encodings**\n\nThank you for your insightful question. To address this, we conducted ablation experiments on Charades:\n\n|Method|R1\\@0.5|R1\\@0.7|mIoU|\n|-|-|-|-|\n|Dense Position Encodings|48.44|27.15|44.85|\n|Timestamp Token Insertion (UniTime-SP)|74.33|53.71|63.15|\n\n- Our base model, `Qwen2-VL-7B`, utilizes Multimodal Rotary Position Embedding (MRoPE), which decomposes position embeddings into temporal, height, and width components. After fine-tuning, temporal grounding remains **suboptimal with dense position encodings**. \n- In contrast, **timestamp token insertion yields significant gains**. We attribute this improvement to the MLLM's strong multimodal retrieval ability and the simplification of the task into (1) identifying question-relevant video segments and (2) directly reading their corresponding timestamp tokens.\n\n---\n\n**Q3: The “end” Timestamps of the Segments**\n\nIn the segment retrieval (coarse-grained grounding) stage, the model is provided with the starting timestamps of each segment, denoted as *{$t_1$, $t_2$, ..., $t_N$}*. The model predicts a segment boundary *$t_i$*, and the corresponding segment interval is defined as *[$t_i$, $t_{i+1}$]*. For the final segment, the end timestamp *t$_{i+1}$* is set to the total duration of the video.\n\n---\n\n**Q4: The Color of Negative Numbers**\n\nThank you for bringing this to our attention. We acknowledge the need for improved visual distinction and will use a distinct color for negative numbers in the revised manuscript.\n\n---\n\n**Q5: Inclusion of Appendices in the Main Submission**\n\nWe appreciate your question regarding appendix placement. In accordance with this year's NeurIPS submission guidelines, which state that *“authors may optionally choose to include some or all of the technical appendices in the same PDF above.”*, we have incorporated the most substantive appendices into our main submission.\n\n---\n\nWe will incorporate the above clarifications, experiments, and analyses into our final draft."}, {"review_id": "3N8g3WZItw", "reviewer": "Reviewer_A5nc", "summary": "This paper proposes UniTime, an MLLM-based model for universal video temporal grounding across diverse video scenarios. UniTime interleaves timestamp tokens with video tokens to enable precise boundary prediction and employs adaptive frame scaling to handle videos of different lengths, balancing spatial resolution and computational efficiency. For long videos, UniTime adopts a coarse-to-fine strategy—first retrieving relevant segments at low granularity, then refining boundaries within selected segments—achieving robust performance on both short and long videos. The proposed UniTime achieves superior performance on mainstream VTG and VideoQA benchmarks.", "questions": "Please refer to the above **[Weaknesses]** section. I suggest that the author provide more detailed comparisons and analysis in the experimental section. If the author can address my concerns, I would be happy to increase the score.", "limitations": "yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**[Strengths]**\n1. The paper is written clearly, fluently, and is easy to follow\n\n2. The proposed coarse-to-fine strategy, adaptive video frame sampling, and timestamp interleaving sequence methods are simple yet effective\n\n3. The proposed MLLM-based UniTime method achieves impressive performance on multiple benchmarks\n\n\n**[Weaknesses]**\n\n**Important**\n1. Lack of baseline comparison experiments. The proposed UniTime method is based on continued training of the Qwen2-VL-7B base model, but the paper does not provide the performance of Qwen2-VL-7B/Qwen2.5-VL-7B on VTG and VideoQA tasks. The authors should provide baseline experimental results and analyze how the main modules such as \"Adaptive Scaling,\" \"Multi-stage Inference,\" and \"Segment Retrieval\" improve upon the baseline\n\n2. Lack of comparison with SOTA methods. Some recent major works, such as VideoChat-Flash [23], achieved a score of 64.7 on LongVideoBench and 74.7 on MLVU, which are better than the results presented in this paper.\n\n3. No verification of whether general video understanding capabilities are maintained. UniTime utilizes the powerful video understanding capabilities of MLLM to help achieve better performance on VTG tasks, but does this training process harm the model's general video/visual understanding capabilities?\n\n(Optional)\n\n4. Flexibility verification. As a plug-and-play module, does it have generalizability across different baseline models (InternVL series) or models of different sizes (3B, 32B)? The authors should conduct more in-depth exploration of its generalizability\n\n5. Closed-source model testing. The authors should test mainstream closed-source models (such as Claude, Gemini, Doubao, etc.) to evaluate whether the proposed UniTime method surpasses or narrows the gap with closed-source models on VTG tasks", "was_revised": true, "final_justification": "The rebuttal has addressed most of my concerns including:\n\n* Regarding the preservation of general capabilities, the proposed method provides a plug-and-play module that enhances the VTG ability of MLLMs with only a small training cost. Since LoRA can be easily detached, the model's general capabilities are not harmed, which makes sense.\n\n* For the ablations of key modules, the authors gradually added IG, AS, and SR modules based on Qwen2-VL-7B, showing consistent performance improvements with +16.79 mIoU over the baseline, which demonstrates the effectiveness of the method.\n\n* Further enhancing the VTG capability of VideoQA models. Although it cannot be directly compared with VideoQA methods, the proposed approach can further improve the performance of VideoQA-type methods.\n\n* Strong generalizability. The supplementary experiments provided by the authors verified significant improvements across different model sizes in the Qwen-VL series and Intern-VL series.\n\n* Comparable to closed-source models. Compared to existing SOTA closed-source MLLMs, UniTime-Full achieved the best Avg. mIoU, approaching the corresponding SOTA closed-source models in both long and short video localization.\n\nThe author's rebuttal is very convincing, so I am willing to increase my initial score. This paper appears to be technically solid and will have a positive impact on the VTG field.", "rebuttal": "Thank you for your constructive comments. We have provided our responses below.\n\n---\n\n**Q1: General Video Understanding Capabilities**\n\n- Our goal is to **enhance temporal grounding** by leveraging their strong multimodal understanding. While task-specific fine-tuning may reduce generality, our focus is on improving temporal reasoning, which is critical for many downstream applications.\n\n- UniTime uses **pluggable LoRA parameters**, allowing the base MLLM’s original video understanding capabilities to be fully restored by detaching LoRA. This is exactly how we conduct VideoQA evaluation in Table 6 in our original submission: first perform temporal grounding with UniTime-Full, then detach LoRA and answer questions using the grounded video segment.\n\n---\n\n**Q2: Baseline Comparison Experiments**\n\nThank you for your feedback. Below, we address the raised points in detail.\n\n- **Baseline Comparison on VTG Task**\n\n    |Method|Ego4D mIoU|TaCoS mIoU|Charades mIoU|ANet-Captions|\n    |-|-|-|-|-|\n    |Qwen2-VL-7B|0.46|2.64|8.98|5.97|\n    |Qwen2.5-VL-7B|0.82|5.29|52.42|21.85|\n    |UniTime-SP (Qwen2)|17.25|45.02|63.15|52.38|\n    |UniTime-SP (Qwen2.5)|18.80|50.00|64.55|51.41|\n    |UniTime-Full (Qwen2)|18.80|50.00|64.55|51.41|\n\n    Here, we evaluate the Video Temporal Grounding (VTG) performance of `Qwen2-VL-7B` and `Qwen2.5-VL-7B` across multiple benchmarks. Both variants of our UniTime model, UniTime-SP (dataset-specific setting) and UniTime-Full (universal setting), **demonstrate significantly superior performance compared to the Qwen-VL baselines**, especially on long-form video benchmarks.\n\n- **Baseline Comparison on VideoQA Task**: Comparisons on video question-answering (VideoQA) are presented in Table 6 in our original submission:\n    - The first row, \"Uniform Sample\", denotes the `Qwen2-VL-7B` baseline that uniformly samples 32 frames.\n    - The last row, \"UniTime-Full\", also stated in Q1, first grounds the relevant video segment with UniTime-Full, detaches LoRA, and then answers questions on the grounded clip. \n    - **UniTime significantly enhances the baseline MLLM's VideoQA accuracy** (e.g., +6.43 QA accuracy on CG-Bench).\n\n- **Ablation on Main Modules**:\n    In the main draft, **Table 4** presents our module ablation study with full implementation details in **Appendix A.2.1**. Here, we are happy to systematically analyze how each component enhances VGT performance upon the baseline:\n     \n    |Configurations|R1\\@0.3|R1\\@0.5|mIoU|\n    |-|-|-|-|\n    |Qwen2-VL-7B|0.48|0.17|0.46|\n    |+Finetune|14.25|7.54|9.83|\n    |+Iterative Grounding|18.42|12.13|12.78|\n    |+Adaptive Scaling|17.91|11.99|12.39|\n    |+Segment Retrieval|24.79|16.83|17.25|\n\n    The above results were evaluated on Ego4D-NLQ:\n    - ``Qwen2-VL-7B``: The baseline sparsely samples frames from long videos, yielding poor accuracy (0.46 mIoU) caused by significant visual information loss.\n    - ``+Finetune``: While task-specific finetune delivers +9.37 mIoU gain, it retains sparse sampling limitations, remaining **suboptimal** (-7.42 mIoU compared to the last row).\n    - ``+Iterative Grounding``: Multi-stage grounding **improves accuracy** (+2.95 mIoU vs. finetuned baseline). However, it still suffers from the **omission of keyframes** due to temporally sparse sampling, resulting in the inability to generate accurate coarse-grained predictions (-4.70 mIoU compared to the last row).\n    - ``+Adaptive Scaling``: Dense sampling with token reduction slightly decreases performance (-0.39 mIoU) but is **essential for enabling Segment Retrieval**, wtih sufficient frames sampled.\n    - ``+Segment Retrieval``: Our full approach implements a *coarse-to-fine grounding* strategy that first identifies relevant video segments at a coarse level and then performs precise grounding within these segments. This delivers **substantial performance gains** (+16.79 mIoU over baseline, +4.86 mIoU over Row 4).\n\n---\n\n**Q3: Comparison with SOTA VideoQA Methods**\n\nThank you for your question. We’d like to clarify a few key points:\n- UniTime is specifically designed for video temporal grounding (VTG) and is **not directly comparable** to end-to-end VideoQA methods.\n- In Table 6 (in our original submission), we use VideoQA benchmarks solely to evaluate how effectively different VTG models identify question-relevant video segments.\n- UniTime can be **seamlessly integrated** with various Video-LLMs to enhance their performance by improving temporal grounding, reducing noise from irrelevant frames, and focusing computation on critical moments.\n\nBelow, we evaluate using `VideoChat-Flash@224` as the base VideoQA model:\n- To ensure fair comparison under identical environments, we first reproduce the baseline results.\n- UniTime's temporal grounding provides consistent improvements over the reproduced baseline (+2.1 on LongVideoBench, +1.1 on MLVU).\n\n|Method|LongVideoBench|MLVU|\n|-|-|-|\n|Reported Results|64.2|74.5|\n|Reproduced Baseline|62.2|74.8|\n|+UniTime Grounding|64.3|75.9|\n\n---\n\n**Q4: Flexibility Verification**\n\nThank you for your interest in the generalizability of our approach. In Appendix G, we provided preliminary experiments with different model sizes of `Qwen2-VL` and `Qwen2.5-VL`. To further validate flexibility, we conducted additional experiments by integrating **UniTime** with various scales of `InternVL-2.5` models:\n\n|Base Model|Ego4D mIoU|Charades mIoU|\n|-|-|-|\n|Qwen2-VL-2B|0.42|6.70|\n|Qwen2-VL-2B + UniTime|7.29 `(+6.87)`|57.25 `(+50.55)`|\n|Qwen2-VL-7B|0.46|8.98|\n|Qwen2-VL-7B + UniTime|17.25 `(+16.79)`|63.15 `(+54.17)`|\n|Qwen2.5-VL-7B|0.82|52.42|\n|Qwen2.5-VL-7B + UniTime|16.61 `(+15.79)`|63.62 `(+11.20)`|\n|InternVL-2.5-2B|0.42|9.30|\n|InternVL-2.5-2B + UniTime|7.46 `(+7.04)`|59.03 `(+49.73)`|\n|InternVL-2.5-8B|1.77|17.54|\n|InternVL-2.5-8B + UniTime|12.03 `(+10.26)`|60.50 `(+42.96)`|\n\nThe results demonstrate that our approach serves as an effective **plug-and-play module**, adapting well to different architectures and scales. For the final draft, we plan to extend testing to include a wider variety of models.\n\n---\n\n**Q5: Closed-source Model Evaluation**\n\nThank you for the valuable suggestion. To evaluate closed-source models, we conducted temporal grounding experiments on a randomly sampled subset of 100 videos from both the Ego4D and Charades datasets. It's important to note that we do not have insight into the training data used by these closed-source models, which could potentially result in data leakage.\n\nThe results are summarized below:\n\n|Model|Ego4D mIoU|Charades mIoU| Avg. mIoU|\n|-|-|-|-|\n|Gemini-2.5-flash|13.9|38.7|26.3|\n|Gemini-2.5-pro|**20.5**|39.3|29.9|\n|GPT-4.1-mini|5.0|31.5|18.3|\n|GPT-4o|8.0|43.8|25.9|\n|Doubao-1-5-thinking-vision-pro-250428|13.5|**72.2**|42.9|\n|UniTime-Full|17.2|69.5|**43.4**|\n\n- `Doubao` achieved the best performance on short videos (72.2 Charades mIoU).\n- `Gemini-2.5-pro` showed superior grounding capability for long videos (20.5 Ego4D mIoU).\n- These closed-source models struggle to achieve high performance concurrently on both long and short videos, indicating a lack of generalization.\n- UniTime achieves competitive performance compared to these closed-source models while maintaining more balanced capabilities for both short and long-form video grounding benchmarks, proving our approach to be overall the best in terms of consistent performance across varying video types.\n\n---\n\nWe sincerely appreciate these valuable comments. Based on your suggestion, we will incorporate these comprehensive evaluation results and analyses in our final manuscript revision."}, {"review_id": "HgtEVGrekM", "reviewer": "Reviewer_F2GW", "summary": "The paper presents UniTime, a framework for universal video temporal grounding using generative multimodal large language models. The approach enables precise localization of relevant video segments given natural language queries. To support long and short videos, the model introduces timestamp token interleaving, adaptive frame scaling, and a coarse-to-fine inference strategy. Experiments show strong results across five video grounding and four video question answering benchmarks, including zero-shot and fine-tuned settings.", "questions": "1. How were the thresholds and token budgets in adaptive frame scaling determined, and how sensitive is performance to these choices?\n\n2. Does relying solely on LLM-decoded textual timestamps limit fine-grained precision, and could auxiliary supervision improve it?\n\n3. What is the computational overhead of the multi-stage inference pipeline compared to one-stage baselines?\n\n4. How does the model handle ambiguous queries with multiple valid grounding regions or misleading timestamps?\n\n5. Has partial or full fine-tuning of the visual encoder been explored, and would it help under domain shift?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengths:**\n\n1. The paper aims to address an important and challenging problem of temporal grounding in both short and long videos using a unified framework.\n\n2. The writing is clear and well-structured, with detailed descriptions of the method, training strategy, and experimental setup.\n\n\n**Weakness:**\n\n1. The temporal reasoning mechanism relies entirely on textual timestamp tokens and LLM decoding, which limits its temporal resolution and introduces potential inconsistencies in fine-grained prediction, especially for closely adjacent events.\n\n2. The adaptive frame scaling policy is based on fixed token budget allocation heuristics and does not incorporate content-aware or query-guided frame importance estimation, which could lead to suboptimal grounding in visually dense regions.\n\n3. The use of interleaved timestamp tokens assumes that LLMs can resolve temporal positions through text-level retrieval, but the model lacks explicit temporal alignment supervision or contrastive objectives to ensure accurate timestamp grounding.\n\n4. The training strategy employs video-centric batching with LoRA fine-tuning on frozen visual backbones, but does not explore joint optimization of vision-language alignment, which may restrict performance in out-of-domain or visually ambiguous cases.\n\n5. The multi-stage inference pipeline improves grounding granularity but increases latency and computational cost, and the paper does not quantify this overhead or compare against single-stage efficient alternatives.", "was_revised": true, "final_justification": "I believe the paper has merit in its current form, and while there may be areas for improvement, I maintain my score based on its overall contribution.", "rebuttal": "Thanks for the constructive comments. We provide our responses as follows.\n\n---\n\n**Q1: Fine-grained Prediction**\n\nWe appreciate your insightful observations. Our method ensures fine-grained temporal prediction through the following design choices:\n\n- We utilize **2 FPS** for video processing, which offers relatively fine temporal resolution. This aligns with standard practices in video temporal grounding (VTG)—e.g., SnAG [1] (1 FPS) and UniVTG [2] (0.5 FPS).\n- For long videos, we adopt a **coarse-to-fine** grounding strategy in terms of temporal resolution to effectively distinguish closely adjacent events.\n- Our framework supports arbitrary temporal resolutions and can be adapted for finer precision. To validate this, we conducted experiments on Charades at 4 FPS, achieving slightly improved performance:\n\n    |FPS|R1\\@0.5|R1\\@0.7|mIoU|\n    |-|-|-|-|\n    |2 (default)|74.33|53.71|63.15|\n    |4|75.27|54.89|63.97|\n\n---\n\n**Q2:  Query-guided Frame Importance Estimation**\n\nWe appreciate your valuable suggestion. A lightweight module for dynamic frame importance estimation—*enabling adaptive token allocation*—is indeed a promising direction. We fully acknowledge the potential of this idea and plan to explore it in future research.\n\n---\n\n**Q3: Timestamp Retrieval vs. Explicit Temporal Alignment**\n\nWe appreciate your insightful suggestion regarding explicit temporal alignment supervision. After careful consideration, we opted not to include such supervision for the following reasons:\n\n- **Inherent Multimodal Retrieval Capabilities in MLLMs:** \n    - **Modern MLLMs exhibit strong multimodal retrieval abilities without explicit supervision**, as evidenced by tasks like *Needle-in-a-Haystack (NIAH)* [3,4], which require models to accurately retrieve target images (\"needles\") within long videos (\"haystacks\").\n    - By reformulating temporal grounding as a **timestamp retrieval** task using timestamp-interleaved sequences, we leverage and extend these inherent capabilities to temporal grounding, eliminating the need for auxiliary alignment supervision.\n\n- **Theoretical Equivalence to Explicit Alignment Methods:**\n    - Existing methods (e.g., GeLM [5]) employ alignment supervision by optimizing a similarity matrix between special tokens and visual features, ultimately enabling the token to effectively **retrieve query-relevant visual features**.\n    - Although our generative method differs from discriminative methods based on explicit alignment, it follows a **similar principle**: accurately retrieving timestamp tokens that correspond to query-relevant visual features, thereby achieving comparable grounding without requiring explicit contrastive supervision.\n\n---\n\n**Q4: Unfreezing the Visual Backbone During Training**\n\nWe appreciate your valuable suggestion. To investigate its impact, we conducted experiments where we unfroze the visual backbone of `Qwen2-VL-2B` (using LoRA) during training and evaluated performance on Charades-STA:\n\n|Vision Encoder|R1\\@0.5|R1\\@0.7|mIoU|\n|-|-|-|-|\n|Freeze|65.38|42.18|57.25|\n|Fine-tune|67.72|45.43|58.89|\n\nThese findings **validate your insight** and suggest that our current approach could potentially benefit from this modification. Accordingly, we **plan to incorporate this adjustment** when retraining our UniTime-Full model.\n\n---\n\n**Q5: Latency and Computational Cost**\n\nThank you for your valuable feedback. While Table 4 in our original submission demonstrates the performance advantages of our multi-stage approach, we have now conducted a comprehensive computational analysis to quantify the associated overhead. Using the *calflops* library [6], we measured the latency, TFLOPs, and TMACs per query for videos from the Ego4D dataset, comparing multi-stage and single-stage inference:\n\n|Method|Latency|TFLOPs|TMACs|mIoU|\n|-|-|-|-|-|\n|Multi-stage|9.54s|326.88|164.94|17.25|\n|Single-stage|1.83s|91.19|45.59|9.83|\n\n- The multi-stage approach achieves **1.8× higher mIoU** (17.25 vs. 9.83)\n- This improvement comes with **5.2× higher latency** and **3.6× greater computational cost** (TFLOPs and TMACs)\n\nThis analysis reveals the fundamental trade-off between grounding accuracy and computational efficiency in our method.  Future work could explore efficiency optimizations.\n\n---\n\n**Q6: Impact of Frame Thresholds and Token Budgets**\n\nWe appreciate your insightful questions regarding these key hyperparameters. To systematically evaluate their effects, we conducted comprehensive ablation studies:\n\n- **Threshold $N_f^{long}$**\n    - **Role**: Determines the maximum video length that can be processed directly\n        - If video length ≤ $N_f^{long}$: Process directly.\n        - If video length > $N_f^{long}$: Split into segments, process separately, and aggregate results.\n\n    - **Evaluation on Ego4D-NLQ**:\n\n        | $N_f^{long}$   | R1\\@0.3 | R1\\@0.5 | mIoU  |\n        | -------------- | ------- | ------- | ----- |\n        | 1024 (default) | 24.79   | 16.83   | 17.25 |\n        | 512            | 24.72   | 16.63   | 16.95 |\n        | 256            | 23.94   | 16.10   | 16.68 |\n    \n    - **Key Findings**:\n        - The choice of $N_f^{long}$ has minimal impact on model accuracy because our method segments over-length videos and aggregates results. \n        - Adaptive Frame Scaling enforces a fixed token budget per segment. Consequently, a smaller $N_f^{long}$ increases inference time due to more video segments and more input tokens.\n        - **Default Choice**: 1024 for optimal efficiency-performance balance.\n\n- **Threshold $N_f^{short}$**\n    - **Role**: Decides single- or multi-stage processing:\n        - If video length ≤ $N_f^{short}$: Single-stage\n        - If video length > $N_f^{short}$: Multi-stage (coarse-to-fine)\n    - **Evaluation on TaCoS**:\n        | $N_f^{short}$ | mIoU (64-128s) | mIoU (128-256s) | Avg. mIoU |\n        | ------------- | -------------- | --------------- | --------- |\n        | 64            | 58.43 (multi)  | 54.57 (multi)   | 50.28     |\n        | 128 (default) | 56.59 (single) | 54.57 (multi)   | 50.00     |\n        | 256           | 56.59 (single) | 42.90 (single)  | 46.82     |\n    \n    - **Key Findings**:\n        - For 64-128s videos: Minimal performance difference between multi-stage ($N_f^{short}$=64) and single-stage ($N_f^{short}$>=128) processing (ΔmIoU=1.84). However, the lower threshold increases average inference time as it forces more videos into multi-stage processing.\n        - For 128-256s videos: Multi-stage processing yields significantly better results (54.57 vs 42.90 mIoU)\n        - **Default Choice**: 128 to balance performance and efficiency.\n    \n- **Token Budget**\n    - **Evaluation on Ego4D**:\n        | Token Budget    | R1\\@0.3 | R1\\@0.5 | mIoU  |\n        | --------------- | ------- | ------- | ----- |\n        | 16384 (default) | 24.79   | 16.83   | 17.25 |\n        | 8192            | 24.08   | 16.10   | 16.69 |\n        | 4096            | 23.04   | 15.07   | 15.96 |\n    \n    - **Key Findings**: \n    Performance improves consistently with higher token budgets (up to GPU limits).\n\n---\n\n**Q7: Multiple Valid Grounding Regions**\n\nFor queries associated with multiple valid temporal moments (e.g., in QVHighlights), we incorporate all ground truth moments in chronological order within the training sequence. By doing so, the model learns to accurately handle ambiguous queries that may correspond to multiple grounding regions or moments. During inference, **UniTime can naturally predict multiple valid moments** for such queries, maintaining consistency with the training paradigm.\n\n---\n\nWe will incorporate the analysis, along with additional experiments, in our final draft.\n\n---\n\n[1]: Fangzhou Mu, Sicheng Mo, and Yin Li. \"Snag: Scalable and accurate video grounding.\" In CVPR, 2024.\n\n[2]: Lin, Kevin Qinghong, et al. \"Univtg: Towards unified video-language temporal grounding.\" In ICCV, 2023.\n\n[3]: Zhang, Peiyuan, et al. \"Long context transfer from language to vision.\" arXiv:2406.16852, 2024.\n\n[4]: Zhao, Zijia, et al. \"Needle in a video haystack: A scalable synthetic framework for benchmarking video mllms.\" arXiv:2406.09367, 2024.\n\n[5]: Chen, Qirui, Shangzhe Di, and Weidi Xie. \"Grounded multi-hop videoqa in long-form egocentric videos.\" In AAAI, 2025.\n\n[6]: Ye, Xiaoju. \"calflops: a FLOPs and params calculate tool for neural networks in pytorch framework.\" 2023."}]

Author rebuttal:

Meta-review: This paper introduces a multimodal large language model for universal temporal grounding, which is a practical and important problem. There are various strengths of this work:

- The techniques are well-motivated and clearly designed (Reviewer YJyw)

- Evaluation is encyclopedic and robustly validates the strengths of the proposed method (Reviewer Zb87).

In addition to these strengths, there are certain concerns regarding general video understanding capabilities of the model (Reviewer A5nc) and technical flaws (Reviewer F2GW). However, during rebuttal, authors have satisfactorily resolved these concerns. In summary, I recommend an acceptance for this work.
Decision: Accept (poster)