Empower Words: DualGround for Structured Phrase and Sentence-Level Temporal Grounding
Authors: Minseok Kang
Minhyeok Lee
Minjung Kim
Donghyeong Kim
Sangyoun Lee
Venue: NeurIPS 2025 poster ()
Keywords: Video Temporal Grounding
Moment Retrieval
Cross-modal Learning
Vision-Language Models
Temporal Localization
Primary area: deep_learning
TL;DR: DualGround mitigates EOS token bias by introducing additional phrase-aware path for fine-grained video-language alignment.

Abstract: Video Temporal Grounding (VTG) aims to localize temporal segments in long, untrimmed videos that align with a given natural language query. This task typically comprises two subtasks: \textit{Moment Retrieval (MR)} and \textit{Highlight Detection (HD)}. While recent advances have been progressed by powerful pretrained vision-language models such as CLIP and InternVideo2, existing approaches commonly treat all text tokens uniformly during cross-modal attention, disregarding their distinct semantic roles. To validate the limitations of this approach, we conduct controlled experiments demonstrating that VTG models overly rely on [EOS]-driven global semantics while failing to effectively utilize word-level signals, which limits their ability to achieve fine-grained temporal alignment. Motivated by this limitation, we propose DualGround, a dual-branch architecture that explicitly separates global and local semantics by routing the [EOS] token through a sentence-level path and clustering word tokens into phrase-level units for localized grounding. Our method introduces (1) token-role-aware cross modal interaction strategies that align video features with sentence-level and phrase-level semantics in a structurally disentangled manner, and (2) a joint modeling framework that not only improves global sentence-level alignment but also enhances fine-grained temporal grounding by leveraging structured phrase-aware context. This design allows the model to capture both coarse and localized semantics, enabling more expressive and context-aware video grounding. DualGround achieves state-of-the-art performance on both Moment Retrieval and Highlight Detection tasks across QVHighlights and Charades-STA benchmarks, demonstrating the effectiveness of disentangled semantic modeling in video-language alignment.

Empower Words: Duaun for Structured Phrase and Sentence-Level Temporal Grounding
Minseok Kang
Minhyeok Lee Yonsei University hydragon5 16 yonseiii.c.ccckkrr
Yonsei University
louis0503 yonsei.ac.kr
Minjung Kim
Donghyeong Kim
Sangyoun Lee Yonsei University sylcee@ yonsei..onnnnccc
LG Electronics minjung 2.Kimmlle.comm
Yonsei University
2donghyung87 wor.uugg@@@@@yooi.cc...k. yonsei.ac.kr
Abstract
Video Temporal Grounding (VTG) aims localize temporal segments long, untrimmed vidcos that align with a given natural language query. This task typically comprises two subtasks: Moment Retrieval (MR) and Highlight Detection (HD). While recent advances have been progressed by powerful pretrained vision-language models such as CLIP and Intern Video2, existing approaches commonly treat all text tokens uni uniformly during modal attention, disregarding their distinct semantic roles. validate the limitations this approach, we conduct controlled experiments demonstrating that VTG models overly rely on [EOS] -driven global semantics while failing to effectively utilize word-level signals. which limits their ability achieve fine- grained temporal alignment. Motivated by this limitation, we propose DualGround, dual-branch architecture that explicitly separates global and local semantics by routing the [EOS] token through a sentence-level path and clustering word tokens into phrase-level units for localized grounding. Our method introduces (1) token- role-aware cross modal interaction strategies that align video features with sentence- level and phrase-level semantics in structurally disentangled manner, and (2) joint modeling framework that not only improves global sentence-level alignment also enhances fine- grained temporal grounding by leveraging structured phrase- aware context. This design allows the model to capture both coarse and localized semantics, enabling more expressive and context-awary video grounding. DualGround achieves kate-of-ine the-art performance both Moment Retrieval and Highlight Detection tasks across QVHighlights and Charades- STA benchmarks, demonstrating effectiveness disentangled semantic modeling video-language alignment.
Introduction
Video Temporal Grounding (VTG) aims to localize segments in a video that correspond to a natural language query. VTG comprises two sub-tasks: Moment Retrieval (MR), which predicts the start and end timestamps of relevant moments, and Highlight Detection (HD), which assigns saliency scores to short video clips based query relevance. Given their structural similarity and shared objective of grounding query-relevant content, recent approaches have explored joint training of MR and HD, particularly enabled by the QVHighlights dataset [10), which provides aligned annotations for both tasks. Furthermore, the use of pretrained vision-language models (VLMss, such as CLIP 22 and Intern Video2 29 has improved query-video alignment through rich cross-modal representations.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Query: guy with red jacket getting interviewed
MAP
[EOS]]
+0.57
Foreground
Tolen
Ctp
57
S8
59
80
61
62
63
(608)
61
-0.30
gelling
Irttrviw
CG-DETR
TR-DETR
FlashVTG
DualGround
(E08)
(b)
(a) Comparison of token input configurations (Word only, [EOS] only, Full (Word + [EOS])) on QVHighlights set. DualGround reduces over-reliance on [EOS] and better utilizes word-level cues. (b) Cross-attention map from FT [ showing weak attention to word tokens and dominant focus on [EOS], suppressing localized semantics.
Figure 1:
Both CLIP and Intern Video2 tokenize input text queries with special tokens such as [SOS] and [EOS], which are positioned at the beginning and end of the token sequence, respectively. Crucially, the [EOS] token is designed not merely as a positional marker but as a summary representation of the entire sentence. It is trained to embed holistic semantic information derived from all preceding tokens, and thus becomes highly influential in downstream tasks.
Despite the inherent semantic differences between word-level tokens and the sentence-level [EOS] token, prior VTG models (24 19 18 D treat all text tokens uniformly during cross-moda modal attention. Since VTG aims to align video segments with the overall sentence intent, attention tends to concentrate on the [EOS] token, which encapsulates global semantics. This design, however, risks underutilizing localized word-level cues that are essential for fine-grained grounding.
We empirically investigate how current VTG models utilize textual representations through a com- parison of three input configurations -1) word tokens only, (2) the [EOS] token only, and (3) full sequences-on the QVHighlights validation set using Intern Vide22 features. As shown in Figure [ models achieve comparable or even superior performance when using only the [EOS] token compared to the full input. Attention visualizations further reveal that even for video clips unrelated to the input query, the model exhibits predominant focus on the sentence-leve [EOS] representation, while salient word tokens (e.g., "red 'red jacket") that provide visually meaningful cues are largely underutilized. This phenomenon arises because the [EOS] token, produced by off-the-shelf pretrained text en- coder, is designed to summarize the entire sentence independently of the visual context. Consequently, the [EOS] token may fail to reflect textual cues that are visually salient and critical for accurate grounding. These observations underscore the importance of incorporating fine-gratn aeee-rraned word-level semantics for precise and context-aware moment localization. To substantiate this observation at scale, we provide a quantitative correlation analysis in Appendix B.1, demonstrating that prior VTG models exhibit consistently high alignment between the [EOS] and word-token attentions.
These findings reveal that existing VTG models are biased toward global sentence- semanties, indicating the need for an approach that can more accurately ground videos where such bias hinders fine-grained alignment. To address this issue, we introduce a dual-branch architecture that jointly models global and local textual semantics for robust video-text alignment. This design retains the strong grounding capabilities of the sentence-level representation while incorporating phrase-level path that clusters contextually coherent words into semantically meaningful units. The dual-path

---

structure allows the model to balance coarse global alignment and fine-grained local interactions, capturing nuanced word dependencies that are often diluted in flat token sequences.
Within the sentence-level path, we adopt Adaptive Cross Attention (ACA) to strengthen alignment between the sentence embedding and video clips. ACA incorporates learnable dummy tokens to absorb irrelevant attention, guiding semantically aligned clips to focus on the [EOS] token. This strategy mitigates the limitations of single-token attention and promotes stable sentence-level grounding with reduced interference from noisy textual inputs.
For the phrase-level path, cluster word tokens into semantically coherent phrases based their representational correlation in the feature space. Given that word semantics are context- context-dependent and emerge through interactions with neighboring words, phrase-level abstraction offers more coherent representation for aligning with visual content. Inspired by recent multimodal reasoning researches 7 20), use these phrase approach as structured intermediate units that support fine- grained alignment. Initial phrase groupings are formed using a Recurrent Phrase Generator (RPG), which composes each phrase by attending over word tokens, conditioned on global semantics and prior phrase context. These groupings are then refined through a Slot Attention module, which disentangles overlapping meanings and enhances semantic purity through iterative updates.
Unlike prior works that treat textual features as flat sequences, our model explicitly captures interac- tions between each phrase and each video clip. We compute dense phrase-clip context embedding via Hadamard product between projected phrase and video embeddings, followed by temporal self- attention maintain consistency across time. This design enables the model reason about semantic relevance at a fine temporal resolution. By unifying global and localized semantics in a structured manner, model achieves This joint modeling not only improves retrieval accuracy but also enhances robustness across varying query complexities.
precise and context-aware alignment between language and video.
Building upon these observations and designs, our main contributions can be summarized as follows:
We empirically identify strong bias in existing VTG models toward the global [EOS] representation, which leads to underutilization of word-level semantics crucial for fine- grained grounding.
We propose DualGround, dual-path architecture that jointly models global sentence- level and localized phrase-level semantics, enabling balanced and context-aware video-text alignment.
Our approach achieves precise and robust grounding by integrating dual-level textual se- mantics, achieving state-of-the-art performance on the QVHighlights and Charades-STA benchmarks.
2 Related Work
Video Temporal Grounding. VTG has been extensively studied through its two core sub-tasks: moment retrieval (MR) and highlight detection (HD). Early MR approaches can be categorized into either proposal-based generate candidate temporal proposal-free paradigms. Proposal-based methods (6 33 37 first
-typennlttyyiallll via sliding windows or anchor mechanisms and then rank them according to their relevance the query. While effective, these methods often suffer from redundancy and coarse temporal boundaries. In contrast, proposal-free methods [10 20) directly regress start and end timestamps use attention- based localization, allowing end-to-end optimization and more flexible fine-grained reasoning. As result, proposal-free frameworks have become dominant due to their efficiency and compositional flexibility.
major milestone for unified VTG research was the introduction of the QVHighlights dataset [T]. which provides aligned annotations for both MR and HD tasks over shared video-query pairs. This dataset enabled models to jointly optimize coarse -aaneeeg temporal localization and fine- -rniirai clip-level saliency estimation, bridging the two previously disjoint tasks and facilitating cross-task supervision.
Building on this foundation, recent VTG models increasingly adopt DETR frameworks 10 19 24 26 H, where learnable decoder queries replace heuristic proposals to achieve end-to-end training and global reasoning. However, such methods rely a limited number of decoder queries,
18
3

---

Decoding
City iise Attention
Phrase-
Conv
Self Attention
Goa
Adiettivn Atlention
x
Phrase-level Path
Sentence-lovel Path
Sallency
Opce
Moment Boundary
Phrase Generator
Encoder
Background
Foreground
Moment Retrieval
f Text Encoder
Video
Sallency Score
Highlight Detection
"A white puppy 15 eating cake..
Feature Extraction
Figure 2: Overall architecture of DualGround. The model processes sentence-level semantics and phrase-level signals through separate pathways to capture both global intent and localized context. These two representations then fused and used to perform moment retrieval and highlight detection with fine temporal precision.
restricting temporal granularity and making difficult to capture short or densely overlapping events. To address these limitations, follow-up studies have introduced multi-scale temporal modeling (e.g.. R2-Tuning [13] and Flash [., which construct temporal pyramids or hierarchical representa- tions to improve localization accuracy across diverse segment durations. This design paradigm draws inspiration from multi-scale feature encoding methods temporal action localization, as in Action- Former [35] Other lines of research enrich the multimodal representation space by incorporating additional modalities, including audio signals 14
25 or by leveraging external knowledge from improve grounding generalization and cross-domain
large language models (LLMs) [27]] 16 robustness.
Vision-Language Foundation Models. Following the release of QVHighlights, VTG has entered new stage driven by large-scale lasgunel pretraining. The success of CLIP 22 demon- strated that contrastive multimodal representation learning could effectively align visual and textual semantics open domains. Subsequent video- language models such Video-LLaMA [36] and Intern Inter2Video2 (29), further extended this paradigm to spatiotemporal contexts, enabling generic video encoders to serve as universal feature extractors for VTG. By leveraging these pretrained represen- tations, recent works have achieved remarkable transferability and data efficiency on downstream MR and HD benchmarks. This transition toward lasion-language foundation model-based VTG has shifted the focus from purely architectural innovations to representation learning and cross-modal alignment quality, paving the way for unified and scalable grounding frameworks.
Dext-Centric Approach. Complementary to architectural advances, recent studies have explored improving textual representations for accurate grounding. Woo et al. (30) proposed a holistic query understanding framework that employs a global text anchor to regulate visual attention, demonstrating the importance sentence-level semantics in filtering irrelevant clips. Keyword- DETR (26 instead emphasizes visually salient keywords through token-level attention, highlighting the role of linguistically informative words temporal grounding. These works share our motivation to strengthen textual cue utilization for precise vidco-text alignment.

---

3 Method
3.1 Model Overview
An overview of DualGround is provided in Figure DualGround adopts dual-path framework that integrates sentence-level and phrase-level semantics for video temporal grounding. Section introduces the sentence-level path, which leverages the [EOS] token to capture global alignment with the video. Section 3.3 presents the phrase-level path, which clusters word tokens into localized phrases and models their interaction with the video. The decoding module that fuses these signals for moment retrieval and highlight detection is descrihed in Section 3.4. Finally, the overall training objectives are detailed in Section 3.5
3.2 Sentence-Level Path
In our design, we isolate the sentence-level representation using only the [EOS] token, resulting in single-token key sequence. While this token captures global semantics, such a minimal representation inherently incompatible with standard attention mechanisms, which rely on multiple key tokens to compute contrastive similarity distributions. To address this, we incorporate the Adaptive Cross Attention (ACA) mechanism introduced in CG-DETR [18), appending learnable dummy tokens to the key sequence. These dummy tokens are conditioned on the sentence and act semantic attractors for video clips that are weakly aligned with the query, absorbing their attention and preventing interference with the alignment signal encoded in the [EDS] token. Conversely, clips that are semantically relevant to the query attend more directly to the [EOS] token, enabling sharp and robust global alignment. This design allows our model to simulate full-sequence sentence-level attention using only a compact representation, reducing reliance on noisy or less informative word-level tokens.
Hummy-Enhancee Sentence Attention. We represent the sentence-level embedding as the [EOS] token {[es) Rd. and introduce La learnable dummy tokens (d1, .e....................... dLa} IR We stack them to form D {d, [ E TR. Ladd To contextualize them, we concatenate and the [EOS] token to obtain E = [D; e [pos)] E (l and pass this sequence through a lightweight Transformer encoder fene. which corresponds to the dummy encoder illustrated in Fig, 2] From the encoder output, we retain the first La rows the updated dummy embeddings D' {dd } 1-=1- and append the original epens] to construct the attention input sequence E' DD' [ces]] R (Tx+1)xd where the [EOS] token is placed at the final index.
To compute cross-attention, video features V = {w:}1= E R are projected into queries Q = {q J. and E' projected into keys K = (h:, and values {uy} using learnable linear layers. The attention weight for the i-th video clip is computed with respect to the [EOS] token's 's key vector
4s -kj Vd
a₁ softmax
ACA(e:) 04 Up
j=but1
Here, qi the query vector for the i-th video clip, and ky, uy are the key and value vectors position j. This setup allows each video clip feature v to selectively attend to sentence-level semantics encoded in the [EOS] token, while dummy tokens act as attention sinks for noisy or irrelevant content. To enhance temporal coherence, we apply a stack of self-attention layers along the clip (temporal) dimension, producing the final untence-guuded video representation / g
3.3 Phrase-Level Path
Recurrent Phrase Generation. To generate initial phrase representations, we aim to cluster the input sentence into fixed number N of semantically coherent units. We design a Recurrent Phrase Generation (RPG) module that incrementally composes phrases by attending over word tokens, conditioned on both the global sentence semantics and the previously generated phrases. Inspired by the sequential phrase grouping strategy in LGI 200, this formulation helps form contextually coherent and robust phrase-level representations. addition, to encourage the grouping of adjacent words, we inject positional embeddings into the word tokens during phrase composition.
5

---

E ( denote the word-level embeddings excluding the [EOS] token. We generate N initial phrase representations recurrent manner, where each phrase computed by using a guide vector to softly aggregate word-level embeddings. At each step n, the guide vector g(r) is constructed from the sentence-level embedding [toss and the previously generated phrase P4 (n-1) For the first - -where previous phrase exists- we instead use zero-initialized placeholder vector. The transformation ((-) used to produce the guide vector is implemented as a lightweight MIP followed by GELU activation. Once the guide vector is computed, it attends to the word tokens via scaled dot-product attention to produce the corresponding phrase embedding. The resulting phrase set is denoted as P. .!....!!!! RNxx
Let
L-1 softmax
(ghh) Vd
Φ (W e [tens], 0)
if n =
Es, where g g(m)
P
Φ W ( 1--1) if n 2 2
(2)
Phrase Refine & Global Token Reconstruction. These initial groupings refined via Slot Attention based module. While the initial clustering captures coarse semantic groupings, it may not fully disentangle overlapping noisy meanings due to its limited capacity and sequential generation. To address this, we adopt a refinement mechanism that allows phrase embeddings to be iteratively updated in more context-aware manner.
Slot attention 15] is particularly well-suited for refining initial phrase representations, as treats latent slot that selectively aggregates semantically aligned word-level
each phrase embedding features. However, its effectiveness can be sensitive to how the slots are initialized, making informed initialization important for stable refinement. Since framework already generates context-aware phrase embeddings through sequential clustering, it naturally provides reliable initialization for slot attention, mitigating the sensitivity to slot quality. We then apply slot-attention layer, where each phrase attends to word tokens treated as key-value inputs. The module employs slot-wise softmax followed by input-wise normalization to enable semantically coherent refinement without the need to discover clusters from scratch.
Let the refined phrase set be denoted P (p").....p!")) Nxd_ To further promote global coherence and inter-slot interaction, we append a learnable token P[EoS] to the phrase set and pass [P: Preas]] through lightweight self-attention transformer block. Although the phrase-level path primarily models localized semantics, we introduce P(EaS] to explicitly consolidate the global meaning from phrase-level cues, enabling the model to maintain sentence-level semanties without relying on the { As detailed in Phrase-Guie Guided Aggregation, Prenst summarizes the overall phrase semantics and produces importance weights for aggregating the phrase-clip context.
Phrase-Clip Context. We model the semantic relevance between each phrase and video clip through crrrase-conditioned interaction implemented as Hadamard product over projected representations. Given the refined phrase set E IR xd and video features V E Txxx this process yields the phrase-clip context embeddings.
C fax @ fo(V)) NxTxd
All functions fp. fur and fax are implemented MLPs with GELU activations, enabling expressive modeling of interactions. To ensure temporal consistency, we apply stacked self-attention over clip dimension T within each phrase stream.
Phrase-Guided Aggregation. The final step of our phrase-level path aggregates the phrase-clip context embedding C E NxTxd into unified phrase guided representation V, xd where each clip feature Up. integrates information across all phrases.
We compute the semantic importance of each phrase by measuring its similarity to the reconstructed sentence-level token Ptans] Rdd These attention weights are then used to aggregate across phrases for each time step t:
N
(WyPrEos) W -nnt
Up.t
softmax
(4)
Vd

---

This aggregation process allows the model to emphasize different phrases depending on their align- ment with the global sentence-level intent, yielding compositional representation that reflects phrase-aware relevance at each clip.
3.4 Decoding Path for Temporal Grounding
We follow the decoding strategy proposed in FlashVTG [ and R2- R2-Tuning [3]. which replaces standard -ETR-style decoding [0] with multi-scale prediction framework. Unlike DETR, which predicts moment spans using fixed number of learnable queries, predictions over the fused video-text feature F = V + V = {ft]-= E RTxd, which processed through temporal feature pyramid constructed via stacked ID convolutions. This design enables the model to capture moments of varying durations more effectively by making predictions at multiple temporal resolutions.
method directly performs
Moment boundaries are predicted each scale using a shared prediction head. The multi-scale outputs are then concatenated and passed through classification and regression heads produce moment confidence scores and normalized start/end points. Highlight detection is performed at the base resolution using a saliency scoring head that combines global and local context through Hadamard interaction.
3.5 Training Objectives
We adopt standard training objectives widely used in VTG literature. The overall loss consists of three components for Moment Retrieval (MR), Highlight Detection (HD), and Phrase-Level supervision.
Moment Retrieval Loss. We employ classification loss (Focal loss 12) and boundary regres- sion loss (LI) to supervise the moment prediction. The total moment retrieval loss is defined as Lm Lets + Lreg.
Highlight Detection Loss. Following prior work, the highlight detection loss is defined as the sum of four components: the ranking loss and contrastive loss over the clip-level saliency scores S, and the ranking loss and contrastive loss over the sentence-level attention weights a. The overall
highlight detection loss is expressed as Las C (S) + >aut (C + (()) (
Phrase-Level Loss. To ensure that the phrase representations both semantically disentangled and consistent with the overall sentence representation, we introduce a phrase-level objective Cphrase composed of two complementary terms.
(1) Distinct Query Attention (DQA) Loss. We encourage semantic diversity across phrases by regularizing their attention distributions to remain orthogonal. Let BBxNx denote the attention weights over (L- 1) word tokens for N phrases across batch of size B. The DQA loss is defined using the Frobenius norm ||F as:
- E1,
1 Σ
LDAA =
(5)
where is scaling coefficient that controls the strength of self-correlation along the diagonal, determining how strictly each phrase attention is encouraged to be distinct.
(2) EOS Reconstruction Loss. To maintain alignment between the sentence-leve semantics, we introduce a reconstruction objective that aligns the reconstructed global token P Prens] with the original [eea] embedding. Here, is temperature hyperparameter and cos(-.)) denotes cosine similarity. We employ an InfoNCE 21
representations and
loss:
exp(cos( CrppcoosPtteaaie
P(eos) e (eoss) )/T)
CEOS = - llg Σ exp(cos(P(een)) e )/T
(6)
The final training loss is computed as weighted sum of three components: the moment retrieval loss Cur, the highlight detection loss Lhd, and the phrase-level supervision loss, which includes the distinct query attention loss LDQA and the [EOS] reconstruction loss LEOS. Formally, the total loss is given by Liotal AmrLmr + Ana + Aphraae (LDQA + LEOS); where each controls the relative weight of the corresponding term.
Total Loss.
7

---

Table 1: Video moment retrieval (MR) and highlight detection (HD) results QVHighlights Test split. Bold: best overall, Underline: second best overall. SF+C denotes CLIP text features with SlowFast and CLIP video features; IV2 denotes Intern features for both modalities.
Method
Backhone
@0.5
@100
mAP
mAP@0.5
mAP@0.75
VG-mAP
VG-Hit
CG-DETR
18
SF+C
65.43
48.38
42.86
64.51
42.77
40.33
66.21
TR-DETR
SF+C
64.66
48.96
42.62
63.98
43.73
39.91
63.42
UVCOM
R1
SF+C
63.55
47.47
43.18
63.37
42.67
39.74
64,20
R2-Tuning
13
C
68.03
49.35
46.17
69.04
47.56
40.75
64.20
FlashVTG
SF4C
66.08
50.00
48.70
67.99
47.59
41.07
66,10
DualGround
SF+C
68.20
51.72
49.02
69.23
47.71
41.15
66,30
FlashVTG
IV2
70.69
53.96
52.00
72.33
53.85
44.09
71.00
DualGround
IV2
71.87
56.94
52.73
72.41
54.38
44.02
70.80
Table 2: Performance on QVHighlights Validation split using Intern Video2 features for fair compari-
son.
Method
Backbone
Ri @0.5
@10.07
mAP
mAP@0.5
mAP@0.75
VG-mAP
VG-Hit@
CG-DETR TR-DETR
70.06
55.87
48.93
69.85
49.56
42,30
68.71
71.72
55.93
50.14
43.74
70.84
FIashVTG
71.48
56.06
52.61
72.37
55.03
44.08
71.48
DualGround
IV2
73.48
58.97
53.26
72.99
56.35
44.12
71.62
Experimental Results
4.1 Datasets & Evaluation Metrics
We evaluate method on three benchmarks: QVHighlights [00. Charades-STA (6), and TV- Sum 233. These cover both moment retrieval and highlight detection, across diverse domains including open-domain YouTube videos, indoor activities, and web videos. We adopt standard data splits and evaluation metrics used in prior works [19 T0 [3] including Recall and mAP for moment retrieval, and mAP and HIT @ for highlight detection. Detailed dataset statistics and metrics are provided in the Appendix A.2.
4.2 Implementation Details
We utilize pretrained encoders for feature extraction: CLIP 22] SlowFal [
InternVideo2 (2 for
QVHighlights [10] and Charades-Sa STA [6]. and I3D I for TVSum (23 Features are extracted without fine-tuning. Detailed feature extraction settings, full training setups and hyperparameters described in the Appendix A.4.
4.3 Experiment Results
We evaluate our model on the QVHighlights [10 dataset, which supports both Moment Retrieval (MR) and Highlight Detection (HD). Test and validation results are shown in Table and Table 2 respectively. Across both backbones CLIP+SlowFFat and Intern Mideo2--DulGGroo surpasses prior methods [3] 18
consistently 24 31 in MR metrics, especially at higher IoU thresholds (e.g., @.07), highlighting its strength in precise moment localization. Table ensures fair comparison by using Intern Video2 for all methods. Even under this strong backbone, DualGround achieves the best performance across all metries, confirming that the improvements stem from architecture, not just feature quality. Notably, we improve R1 @0.7 by 1.72% % with CLIP+SlowFast and 2.98 % with Intern Video2.
In Table we further evaluate on Charades- STA benchmark on CLIP and Internvideo2 backbone feature. DualGround again shows consistent gains in both R @0.5 and R @0.7, under both backbone settings. This reinforces the generalization ability of our model across different domains and video types.

---

Query: guy red jacket getting
(e)
p!
p2
pa
p
9u
61 Idtervl
-V
RestrTo
Th cemn
Jack
(b)
CT Saliency
DuslGround
FinahVTG
TT--ET
getting
(c)
Interview
(d)
.ed
Figure Visualization results on the QVHighlights validation split. (a) Moment retrieval predictions and (b) Highlight detection scores are compared across models. (c) 1.2 norm activation map of phrase- level embeddings, (d) L2 norm activation map of sentence-level embeddings, and (e) Phrase-to-word attention map visualizations from our proposed DualGround model, highlighting how it captures localized semantics and structured alignment.
4.4 Qualitative Analysis
Figure 3 presents a qualitative result on the QVHighlights validation set, comparing our model with CG-DETR [8] TR-DETR (24). and FTashVTG [1] In the visualization, baseline models relying heavily on sentence-level [EOS] representations tend to focus on the prolonged interviewing scene from the beginning of the video, underutilizing the local semantic cue of the phrase "red jacket". In contrast, our model leverages phrase-level representations that preserve word-level semantics, allowing it to accurately localize the intended moment corresponding the described visual concept.
In addition, the visualization highlights how our phrase grouping mechanism effectively clusters contextually related words into coherent units. These phrase clusters align well with localized visual cues, demonstrating the benefit disentangled phrase representations for fine-gra mne-grained temporal localization. Additional visualization results be found in the Appendix E.
4.5 Ablation & Detailed Analysis
Ablation Study. Table presents the results of our ablation study conducted on the HVHighlights validation split. Setting (a) corresponds the baseline model FTashVTG (I], which employs full-token sequence setting, including both word and the [EOS] token. All subsequent settings (b)-() are based on our proposed dual-path architecture, where sentence-level and phrase-level branches are jointly optimized. We observe consistent performance gains in MR when replacing flat word-level modeling with structured phrase-level representations. Specifically, introducing phrase clustering significantly improves both RI @0.7 and mAP compared to directly modeling context between individual word tokens and video clips. Furthermore, incorporating the Distinct Query Attention loss (Lpqa), which encourages semantic separation across phrases, yields additional gains, demonstrating its effectiveness enhancing phrase-level disentanglement and improving temporal localization.
While the improvements are clear for MR, we observe that exhibits more nuanced behavior. Certain settings without proper phrase regulation (e.g., when omitting IDQA or Leos) lead to slight degradations HD metries, suggesting that the unregulated phrase-level path can inject noisy fine-grade fine-grained cues into the sentence-level representation. This highlights the importance of phrase

---

Table 3: Ablation study on QVHighlights-val split. RPG, LDQA and LEOS denote Recurrent Phrase Generation, Distinet Query Attention Loss, and [EOS] Reconstruction Loss, respectively.
Settings
RPG
Slot
LOQA
Leos
RI @0.7
mAP
VG-Hit@
VG-mAP
(a)
56.13
52.24
70.88
44.04
(h)
56.99
52.46
71.12
43.97
(c)
57.83
53.02
71.29
44.09
(d)
56.17
52.30
70.07
43.34
(e)
v
58.02
53.11
70.28
43.51
(1)
56.55
52.53
71.77
44.10
(g)
58.97
53.26
71.62
44.12
Table 4: MR mAP on HVHighlights val split by query length using IV2 backbone.
Table 5: MR Performance on Charades test set under different backbones.
Method
Backbone
R1
RI 6P0.7
Words
Method
UniVTG CG-DETR 18
C+SF
59.25
36.64
0-10
10-15
15-20
2020
C+SF
58.41
36.32
TR-DETR
C+SF
57.61
33.52
CG-DETR
48.61
50.88
47.80
35.21
FlashVTG
1
C+SF
61.08
37.89
DualGround
C+SF
61.11
38.52
TR-DETR
24
49.26
49.15
50.26
35.33
CG-DETR 18
IV2
70.40
48.40
FTashVTG
53.24
54.78
50.58
43.46
TR-DETR 24
IV2
69.73
46.33
FT II
IV2
70.32
49.87
DualGround 54.12
54.33
51.96
48.92
DualGround
IV2
70.67
50.33
regulation in preserving global semantic consistency during clip-level saliency modeling. Overall, the results confirm that our dual-path design enhances MR performance while maintaining comparable or improved HD performance across most settings.
Across different query lengths. We further analyze the impact of query length performance, as shown in Table While existing methods show clear performance drop as queries become longer, our DualGround maintains robust performance even for queries exceeding 20 words. This suggests that our phrase-level path plays critical role in handling complex queries where the global [EOS] token alone is insufficient capture the full semantic structure.
5 Conclusion, Limitation, and Future Works
Conclusion. We propose DualGround, a dual-branch architecture for Video Temporal Grounding that separates sentence-level and phrase-level semanties. Unlike prior models relying the sentence-level [EOS] token, ours introduces structured phrase modeling to recover fine-grained local cues. This enables richer video-text alignment by capturing both global intent and local context. We hope this work lays the groundwork for future research in semantic disentanglement and multimodal grounding.
Limitations & Future Works. Our model assumes fixed number of phrases per query, requiring manual adjustment across datasets. Additionally, it does not leverage audio features, which may limit performance in audio-visual grounding scenarios. Adaptive phrase segmentation based query structure and extension to audio signals for richer multimodal grounding are promising directions for future research.
Acknowledgement
This research was supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No. RS-2024-00340745) and the Yonsei Signature Research Cluster Program of 2025 (2025-22-0013), and the the Korea Institute of Science and Technology (KIST)
Institutional Program (Project No. No.2E33312-25-016)
10

---

References
[1] Zhuo Cao, Bingqing Zhang, Heming Du, Xin Yu, Xue Li, and Sen Wang. Flashvtg: Feature layering and adaptive score handling network for video temporal grounding. 2025 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 9226-9236. IEEE, 2025.
[2] Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? new model and the kinetics dataset. In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6299-6308, 2017.
[3] Houlun Chen, Xin Wang, Xiaohan Lan, Hong Chen, Xuguang Duan, Jia Jia, and Wenwu Zhu. Gurriculum-listene Consisteney-und and complementarity aware audio-enhanced temporal sentence grounding. In Proceedings of the 3 ACM International Conference on Multimedia, pages 3117-3128, 2023.
[4] Yi-Wen Chen, Yi-Hsuan Tsai, and Ming- Hasan Yang. End-to-end mudal video temporal grounding. Advances in Neural Information Processing Systems,
2021.
[5] Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. Proceedings of the IEEE/CVE international conference on computer vision, pages 6202-6211, 2019.
[6] Jiyang Gao, Chen Sun, Zhenheng Yang, and Ram Nevatia. Tall: Temporal activity localization via language query. In Proceedings of the IEEE international conference computer vision, pages 5267-5275, 2017.
[7] Shuting He and Henghui Ding. Decoupling static and hierarchical motion perception for referring video segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13332-133411 2024.
[8] Jinhyun Jang, Jungin Park, Jin Kim, Hycongjun Kwon, and Kwanghoon Sohn. Knowing where to focus: Event-aware transformer for video grounding. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 13846-13856, 2023.
[9] Pilhyeon Lee and Hyeran Byun. Bam-detr: Boundary-aligned moment detection transformer for temporal sentence grounding in videos. In European Conference on Computer Vision, pages Springer, 2024.
[10] Jie Lei, Tamara L Berg, and Mohit Bansal. Detecting moments and highlights in videos via natural language queries. volume 34, pages 1 11846-11855 2021.
[11] Kevin Qinghong Lin, Pengchuan Zhang, Joya Chen, Shraman Pramanick, Difei Gao, Alex Jin- peng Wang, Rui Yan, and Mike Zheng Shou. Univtg: Towards unified video-language temporal grounding. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2794-2804, 2023.
[12] Tsung-Yl Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pages 2980 2988, 2017.
[13] Ye Liu, Jixuan He, Wanhua Li, Junsik Kim, Donglai Wei, Hanspeter Pfister, and Chang Wen Chen. R 2-tuning: Efficient amage-to-video transfer learning for video temporal grounding. In European Conference on Computer Vision, pages 421-438. Springer. 2024.
[14] Ye Liu, Siyuan Li, Yang Wu, Chang Beang-Wen e Chen, Ying Shan, and Xiaohu Qie. Umt: Unified multi- modal transformers for joint video moment retrieval and highlight detection. In Proceedings of the IEEF/CVF Conference Computer Vision and Patlern Recognition, pages 3042-3051, 2022.
[15] Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention. Advances in neural information processing systems, 33:11525-11538 2020.

---

[16] Weiheng Lu, Jian Li, An Yu, Ming- Ching Chang, Shengpeng Ji, and Min Xia. Llava-mr: Large lunguage-and-vision assistant for video moment retrieval. arXiv preprint arXiv:2411.14500 2024.
[17] Boris Meinardus, Anil Batra, Anna Rohrbach, and Mareus Rohrbach. The surprising effec- tiveness of multimodal large language models for video moment retrieval. arXiv preprint arXiv:2406. 18/13, 2024.
[18] WonJun Moon, Sangeek Hyun, SuBeen Lee, and Jae-Pil Heo. Correlation-guided query- dependency calibration for video temporal grounding, arXiv preprint arXiv:2311.08833 2023.
[19] WonJun Moon, Sangeek Hyun, SangUk Park, Dongchan Park, and Jae-Pil Heo. Query- dependent video representation for moment retrieval and highlight detection. In Proceedings of the IEEE/CVF conference on computer vision and patter recognition, pages 23023-23033, 2023.
[20] Jonghwan Mun, Minsu Cho, and Bohyung Han. Local-global video-text interactions for temporal grounding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattem Recognition, pages 10810-10819, 2020.
[21] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv: 1007.037777000 2018.
[22] Alec Radford, Jong Wook Kim, Chris Hallacy, Adiyya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning,
pages 3748- PmL.R, 2021.
[23] Yale Song, Jordi Vallmitjana, Amanda Stent, and Alejandro Jaimes. Tysum: Summarizing web videos using titles. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 5179-5187, 2015.
[24] Hao Sun, Mingyao Zhou, Wenjing Chen, and Wei Xie. Tr-detr: Task-reciprocal transformer for joint moment retrieval and highlight detection. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 4998-5007, 2024.
[25] Yapeng Tian, Jing Shi, Bochen Li, Zhiyao Duan, and Chenliang Xu. Audio-visual event localization in unconstrained videos. In Proceedings of the European conference on computer vision (ECCV), pages 247-263, 2018
[26] Sung Jin Um, Dongjin Kim, Sangmin Lee, and Jung Uk Kim. Watch video, catch keyword: Context- keyword attention for moment retrieval and highlight detection. Proceedings of the AAAI Conference Artificial Intelligence, volume 39, pages 7473-7481, 2025.
[27] Haibo Wang, Zhiyang Xu, Yu Cheng, Shizhe Diao, Yufan Zhou, Yixin Cao, Qifan Wang, Weifeng Ge, and Lifu Huang. Grrundddd-vddeollm:: Sharpening fine-grained temporal grounding in video large language models. arXiv preprint arXiv:2410.03299. 2024.
[28) Lezi Wang, Dong Liu, Rohit Puri, and Dimitris N Metaxas. Learning trailer moments in full-length movies with co-contrastive attention. In European Conference on Computer Vision, pages 300-316. Springer, 2020.
[29] Yi Wang, Kunchang Li, Xinhao Li, Jiashuo Yu, Yinan He, Guo Chen, Baoqi Pei, Rongkun Zheng, Zun Wang. Yansong Shi, et al. Internvide02: Scaling foundation models for multimodal video understanding. In European Conference on Computer Vision, pages 396 416. Springer, 2024.
[30] Jongbhin Woo, Hyeonggon Ryu, Youngjoon Jang, Jae Won Cho, and Joon Son Chung. Let me finish my sentence: Video temporal grounding with holistic text understanding. In Proceedings of the 32nd ACM International Conference on Multimedia, pages 8199-8208, 2024.
[31] Yicheng Xiao, Zhuoyan Luo, Yong Liu, Yue Ma, Hengwei Bian, Yatai Ji, Yujiu Yang, and Xiu Li. Bridging the gap: A unified video comprehension framework for moment retrieval and highlight detection. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 18709-18719, 2024.
12

---

[32] Bo Xiong, Yannis Kalantidis, Deepti Ghadiyaram, and Kristen Grauman. Less is more: Learning highlight detection from video duration. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1258-1267, 2019.
[33] Huijuan Xu, Kun He, Bryan A Plummer, Leonid Sigal, Stan Sclaroff, and Kate Saenko. Multilevel language and vision integration for text-to-clip retrieval. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 9062-9069, 2019.
[34] Minghao Xu, Hang Wang. Bingbing Ni, Riheng Zhu, Zhenbang Sun, and Changhu Wang. Cross- category video highlight detection via set-based learning. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 7970-7979, 2021.
[35] Chen-Lin Zhang, Jianxin Wu, and Yin Li. Actionformer: Localizing moments of actions with transformers. In European Conference on Computer Vision, pages 492-510 Springer, 2022.
[36] Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tunet audio-visual language model for video understanding. arXiv preprint arXiv:2306. 02858,, 2023.
[37] Songyang Zhang, Houwen Peng, Jianlong Fu, and Jiebo Luo. Learning 2d temporal adjacent networks for moment localization with natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 22870-18777 2020.
13

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification:
Abstract and introduction clearly state the bual-b architecture and how it improves moment localization through disentangled semantics, aligning with empirical results.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Limitations are explicitly discussed in the final section, including fixed phrase count and lack of audio modality.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "I "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and prool's
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
14

---

Answer:
[NA]
Justification: The paper focuses on architecture and empirical evaluation without theoretical theorems or proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Full architectural details, loss formulations, and ablation setups are described; additional information is in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
15

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [No]
Justification:
We will release the code after acceptance of the paper.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cr cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ priips.cc/publicc/uuieeee CoodeSumeeieeeeee ssi iessonPolll mollicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Training setup, hyperparameters, datasets, and backbones are summarized in the paper; full details are in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: We simply have reported and compared the evaluations without statistical analysis, because our experiments stable across multiple runs.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
16

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [No]
Justification: While GPU type and training epochs are mentioned, we did not detail memory, runtime, or compute hours.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: The work does not involve human subjects sensitive data and adheres to ethical research principles.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Although this work focuses on foundational research in video-language ground- ing without direct deployment, we acknowledge potential dual-use concerns. The proposed VTG technology could be applied to large-scale content analysis or surveillance systems, raising privacy misuse issues if used irresponsibly. We emphasize that contribu- tions are intended solely for advancing multimodal understanding research and should be developed and deployed under ethical and srivacy-preservee guidelines.
Guidelines:
17

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negatíve societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (c.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The model does not involve high-risk generative tasks or language generation; misuse risk is minimal.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All datasets (QVHighlights, Charades, TVSum) are cited with original papers;
license terms are respected.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
18

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the datasets.paperssiii paperswi thcode. www.coo//aaaaa datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not introduce any new assets
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: No human subjects or crowdsourced data were involved.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Not applicable since the research does involve human participants.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
19

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [No]
Justification: No large language models were used in model design; LLMs were only used for language editing.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https://neurl)) hatt::///eerrppccc/oofffeeeeeeee /c/////feeeeeees//225////.. for what should or should not be described.
20

---

Appendix
The appendix is organized follows.
Dataset, Metric, Implementation Details: We describe the datasets used, including high- light detection results on TVSum, the evaluation metrics employed, and key implementation details.
Token Dependency Analysis: We examine the model's reliance on the [EOS] token through attention correlation analysis, performing ablations under various token-condittoning settings with CLIP- and Intern Video2-based encoders.
Analysis Phrase Segment: We investigate how the optimal number of segmented phrases varies depending the dataset and backbone features, analyzing its impact model performance.
We compare multiple strategies for fusing clip-level embed- dings from the phrase and sentence paths.
Ablation on Fusion Method:
Additional Visualization: We include supplementary visualizations to better illustrate the model behavior and support claims made in the main paper.
A Dataset, Metric, Implementation Details
A.1 Dataset Description
QVHighlights QVHighlights [T] is large-scale benchmark for joint video moment retrieval and highlight detection. It contains 10,148 videos collected from YouTube, spanning various domains including daily life, travel, and news. Each video is paired with natural language queries and annotated with corresponding highlight segments.
Charades-STA Charades-STi [6] extends the Charades dataset by adding temporal moment annotations aligned with text queries. It consists of 9,848 short videos depicting indoor human activitics and provides 16,128 annotated query-moment pairs. The dataset is commonly used for evaluating moment retrieval performance and is provided with standard train/test split.
TVSum TVSum 23 is a video summarization dataset comprising 50 videos from 10 different cat- egories such as documentary, sports, and travel. Each video annotated with frame-level importance scores gathered through crowd-sourced annotations. Following prior work, we adopt 4:1 train-test split and use video titles as textual queries in the highlight detection setting. Although originally intended for summarization, TVSum is widely repurposed for highlight detection due to the similarity between the two tasks.
A.2 Evaluation Metrics
We employ standard metrics commonly used in moment retrieval and highlight detection tasks. Recall@1 1 is measured at multiple Intersection over Union (IoU) thresholds (e.g., 0.5 and 0.7), indicating whether the top-ranked prediction sufficiently overlaps with any ground-truth segment. Mean Average Precision (mAP) is computed by averaging the precision across multiple loU thresholds, capturing both retrieval quality and temporal localization accuracy. Hit@| evaluates whether the top-scoring prediction exactly matches one of the ground-truth highlights, serving as strict top-1 correctness measure. Additionally, mean IoU (mIoU) reports the average overlap between predicted and annotated segments.
We report Recall @ (0.5/0.7), mAP, and Hit@| on QVHighlights, Recall@ I (0.5/0.7) and mean IoU on Charades-STA, and top-5 mAP and Hit@ I on TVSum.
A.3 Experiment Results on TVSum Dataset
Table AT presents the highlight detection performance on the TVSuu val split across video categories. Our method achieves the highest average mAP of 88.1, outperforming existing base-
21

---

Table Al: Experimental results on the TVSum val dataset.
Method
VT
VU
GA
MS
PK
PR
FM
BK
BT
DS
Avg
LIM-S
55.9
42.9
61.2
54.0
60.4
47.5
43.2
66.3
69.1
62.6
56.3
Trailer 28
61.3
54.6
65.7
60.8
59.1
70.1
58.2
64.7
65.6
68.1
62.8
ML-Modulc
86.5
68.7
74.9
86.2
79.0
63.2
58.9
72.6
78.9
64.0
73.3
UMT 14
87.5
81.5
88.2
78.8
81.4
87.0
76.0
86.9
84.4
79.6
83.1
QD-DETR
88.2
87.4
85.6
85.0
85.8
86.9
76.4
73.7
85.0
UVCom 31
87.6
91.6
91.4
86.7
86,9
86.9
76.9
92.3
87.4
75.6
86.3
CG-DETR 08
86.9
88.8
94.8
87.7
86,7
89.6
74.8
93.3
89.2
75.9
86.8
TR-DETR 24
89.3
93.0
94.3
85.1
88.0
88.6
80.4
91.3
89.5
81.6
88.1
VTG 88.3
94.3
91.5
87.7
87.1
91.1
74.7
93.4
90.3
81.7
88.0
DualGround 89.7
93.2
90.7
87.4
88.3
91.3
75.6
92.4
90.6
Table A2: Implementation details across datasets. From top to bottom, we list the hyperparameters and architectural configurations for QVHighlights (QVH.), Charades (Ch.), and TVSum (TVS.). In the Feat column, SF+C denotes the use of SlowFast and CLIP features, IV2 refers to Intern Video2, and I3D indicates I3D features. From left to right, bs the batch size, E is the number of training epochs, and Ir is the learning rate. Ld and N represent the counts of dummy tokens and phrase segments, respectively. D.Ene specifies the depth dummy encoders, ACA is the number of adaptive cross-attention layers, and P-SA indicates the number of slot attention layers in the phrase-level path. P.Enc and S.Enc denote self-attention layers applied along the clip axis in the phrase-level and sentence-level paths, respectively. AMR, λuD, Aphrase are loss weights for moment retrieval, highlight detection, and phrase-level supervision. VDQA coefficient controlling the orthogonality regularization in the DQA loss.
Hyperparameter
Layer #
Loss
Datasel
Feat
bs
Ld
D.Emne
ACA
P-SA
P.Emne
S.Ene
AMR
AHp
Aphrrase
TDQA
QVH.
SF+C
64
le-
2
QVH.
IV2
64
150
le
2
3
2
2
5
1
0.3
Ch.
SF+C
128
2.5e
2
5
0.3
Ch.
IV2
128
50
2.5e
2
3
2
2
5
1
0,3
TVS.
I3D)
4
600
le
2
5
0.3
lines including TR-DETR and FlashVTG, Notably, our model exhibits strong consistency across Parade(PR), Attempting Bike Trick(BT), and Dog Show(DS).
A.4 Implementation Details
Table A2 summarizes the training configurations across datasets. We vary the backbone features (SF+C,IV) IV2, 13D) depending on the dataset and adopt consistent architectural settings. Specific hyperparameters, layer numbers, and loss coefficients are detailed in the table.
Each model uses a hidden dimension 256 and is optimized with the Adam optimizer. Transformer layers follow a post-norm architecture with attention heads. For post-processing non-maximum suppression (NMS) is applied with a threshold of 0.7. All experiments are conducted on a machine equipped with Ryzen 3960X 24-core CPU and single NVIDIA RTX 3090 GPU.
For the Intern Video2 (IV2) 29 setting, we employ the pretrained model released by OpenGVLab The video encoder corresponds to the 1B parammeteer version of Intern aideo2-staged while the text encoder is Stage2-CLIn version (Intern to enhance cross modal representation quality. This configuration follows the official CLIP training pipeline and maintains consistent alignment between visual and textual embeddings.
B Token Dependency Analysis
We quantitatively evaluate the model's dependency on the [EOS] token by measuring correlations of cross-modal attention pattern across tokens, which reveal the degree of over-reliance by the [EOS]
22

---

Table A3: Performance of VTG models using the SF+C backbone across token conditions.
Method
Word
EOS
Full
R1@0.5
R1@0.7
mAP
mAP@0.5
mAP@0.75
CG-DETR
64.84
49.68
43.18
65.27
44.16
CG-DETR
62.19
46.13
41.87
64.01
42,43
CG-DETR
66.90
50.32
43,47
65.48
44.79
TR-DETR
66.32
50,45
43.99
65.74
44.89
TR-DETR
64.00
47.74
41.78
64.25
42.45
TR-DETR
66.48
50.71
44.53
65.43
44.98
VTG
68.85
53.81
48.42
67.83
51.50
FlashVTG
65.62
52.60
45.32
67.12
50,49
FTashVTG
69.03
54.06
49.85
68.44
52.12
DualGround
68.20
54.11
48.51
68.02
51.83
DualGround
65.91
52.31
45.44
67.24
50.33
DralGround
69.25
54.87
49.96
68.62
52.30
Table A4: Performance VTG models using the IV2 backbone across token conditions.
Method
Word
EOS
Full
RI 00.5
R1 @..7
mAP
mAP@0.5
mAP@0. 0.75
CG-DETR
70,06
55.55
48.84
69.71
49.66
CG-DETR
71.35
56.65
49.36
70.08
50.67
CG-DETR
69.74
56.45
48.97
69.18
50.46
TR-DETR
70.65
55.94
48.80
69.52
49.57
TR-DETR
73.35
58.84
50.19
72.02
52.20
TR-DETR
72.06
57.03
49.23
70.45
50.83
FlashVTG
70.72
55.90
51.33
70.92
52.80
FlashVTG
72.23
56.51
52.19
72.34
55,60
FlashVTG
72.32
56.89
52.26
72.39
55.21
DualGround
72.20
57.71
$1.70
72.28
55.29
DualGround
72.11
56.55
52.24
72.31
55.33
DralGround
73.48
58.97
53.26
72.99
56.35
token. We then analyze how varying textual token conditions [Word only, [EOS] only. and Full (Word [EOS]] affect the performance of VTG models under two backbone settings: CLIP-based (SF+C) and Intern Video2-based (IV2). Table A3 and Table A44 present the moment retrieval results on the QVHighlights val set across these conditions.
B.1 Generalization of the EOS Over-Reliance
To verify that the over-reliance on the [EOS] token is prevalent phenomenon, we conduct quantita- tive correlation analysis across the entire dataset. Specifically, we measure the statistical correlation between the attention weights assigned to the [EDS] token and those assigned to individual word tokens during cross-modal interaction. We adopt both the Pearson and Spearman correlation coefficients, which evaluate linear and rank-based relationships, respectively, ensure robustness.
Pearson Correlation.
Pearson correlation coefficient between two variables and is defined as:
ΣN (z; - x () y) 1 (z; 8)2 MN (y: 7
where x, and yi denote individual data points, and J., y are their mean values. higher r indicates stronger lincar relationship between and y.
23

---

Spearman Correlation.
Spearman rank correlation assesses monotonic relationships based on
ranked values:
6 60= d
p p=
(8)
N(N2-1)
where d; is the rank difference between the paired values, and is the number of data points.
Measurement Procedure. The overall computation process is summarized as follows:
1. Cross-modal attention extraction:
For each sample in the training and validation sets, we extract the cross-modal attention map A E R- where N, and N,, denote the number text tokens and video clips, respectively.
2. Token isolation: We separate the attention vector of the [EOS] token, aços. and those of the remaining N; word tokens (a:).-i
3. Token-wise correlation: For each word token, we compute Pearson and Spearman correla- tions between a, and afos, yielding (N₂ - 1) correlation values per sample.
4. Averaging: We average the correlations across tokens and then across all samples within the subset, reporting mean Pearson and Spearman values for both training and validation sets.
Results.
Table
A5 summarizes the results for representative VTG models.
Table A5: Average Pearson and Spearman correlations between [EOS] and word-token attentions.
Model
Train
Val
Pearson
Spearman
Pearson
Spearman
CG-DETR
0.8960
0.8914
0.5962
0.7622
TR-DETR
0.8110
0.7753
0.6021
0.6340
FlashVTG
0.9745
0.9801
0.6771
0.7800
Discussion. Across all models, the correlation values remain consistently high (close to 1.0) for both Pearson and Spearman metrics, indicating that word tokens exhibit attention patterns highly similar to that of the [EOS] token. This confirms that prior VTG models show generalized over-reliance on [EOS], where word-level semantics are largely overridden by global sentence-level alignment cues.
Furthermore, when relating these findings to the results in Table we observe that models with weaker attention correlations tend yield higher performance under the the single [EOS] token setting, when compared with Full-token setting. This suggests that in current model architectures, suppressing the local semantic contributions of individual word tokens may lead to a more optimized training trajectory.
B.2 Impact of Backbone Semantics
Under the SF+C backbone (Table Λ3). using the [EOS] token alone consistently yields lower performance than using word tokens across all models. contrast, the IV2 backbone (Table (A4) shows the opposite trend: in all models except ours, the [EOS]-only setting achieves either the best performance (e.g., CG-DETR, TR-DETR) or results comparable to other configurations (e.g.. FlashVTG).
We attribute this discrepancy to the difference in feature dimensionality between the backbones. CLIP encodes each token 512-dimensional vector, while InternVide02 produces (096-dimensional embeddings. This higher capacity allows IV2's [EOS] token to carry richer sentence-level semantics, enabling strong alignment even without word-level information.Com Conversely, CLIP's limited [EOS] capacity cannot fully represent complex queries, leading models to fall back on word tokens for localized cues. However, this reliance arises not from intentional design but side effect of the [EOS] token's 's limitations. Treating all tokens uniformly in flat sequence still ignores their distinct semantic roles, leading suboptimal alignment.
Our proposed method alleviates this issue by separating sentence-level and phrase-level semantics. As shown in Table A3 it achieves robust performance even with CLIP-based features, validating its
24

---

QV Highlights- Interavideo2
QV Highlights SlowFast CLIP
$1.5
Phrase
Charades Internvideo2
Charades SlowFast CLIP
38.1
50.5
38.7
$0.1
38.3
48.7
37.9
Figure A1: Ablation study on the number of phrase segments.
effectiveness despite the limited capacity of the [EOS] token. As vision-language models (VLMs) evolve with increasingly powerful text encoders, the [EOS] token will likely play an even greater role, making proper treatment of token-level semanties critical consideration for future VTG architectures.
Table A6: Average query length per dataset.
Table A7: Ablation on Fusion Method
Dataset
Split
Query Length
Option
R.5
R1@0.7
mAP
Viiihhighhs
train
10.46
10.49
Add
73.48
58.97
53.26
Charades
train
6.21
Hadamard
71.71
55.25
51,27
6.23
Gate
73.51
58.71
53.24
TVSum
train
7.55
val
7.70
Concat-mlp
73.66
58.19
52.91
B.3
Architectural Influence on Token Utilization
As shown in Table CG-DETR and TR-DETR achieve better performance when using only the [EOS] token, compared to word or full token inputs. This suggests that word tokens may act as noise in these architectures. In CG-DETR, the clip-word distillation loss emphasizes alignment with individual words. which can suppress the rich global semantics of the [EOS] token. In TR-DETR. the global textual feature used for regulation computed by mean-pooling over all word tokens. This strategy may dilute the semantic strength of the [EOS] token and introduce noise from weakly aligned or irrelevant words. both cases, using only the [EOS] token avoids such noise and leads to better aligmment.
These results suggest that the integration token-leve inputs should account for distinct semantic roles of word and [EOS] tokens. Word tokens are most effective when they complement the global sentence representation without interfering with it. Our DualGround framework supports this balance by explicitly disentangling global and local semantics.
25

---

C Analysis of Phrase Segment
C.1 Ablation on Phrase Segment Number
To determine the optimal number of phrase segments, we conduct an ablation study on the phrase segmentation parameter N, which defines the number of semantic units extracted from the input query. We experiment with different values of N on both the Charades and QVHighlights datasets using two backbones: SlowFast CLIP and Internvideo2.
As shown in Fig.AA] Charades achieves the best performance at N = 3, while QVHighlights yields the highest accuracy N This difference further analyzed in the next subsection. We also observe a performance drop when N becomes large. Excessive segmentation divides queries into overly short spans, which may fail capture complete semantic units and lead fragmented or diluted phrase representations. This prevents effective alignment with video content and undermines the benefits of phrase-level modeling.
C.2 Effect of Query Complexity
We hypothesize that the optimal number of phrase segments is influenced by the complexity text queries. Intuitively, queries with greater semantic richness benefit from finer phrase decomposition, as they contain more diverse word-level information that can he aligned with visual content.
To investigate this, we analyze the average query length across datasets, as shown in Tab. A6 Queries from HVHighiights are substantially longer than those from Charades or TVSum, indicating higher semantic complexity. This aligns with ablation results, where QVHighlights achieves the best performance at 4, while Charades performs best at N 3. These observations suggest that phrase segmentation should be tailored to the dataset's linguistic characteristics.
D Ablation on Fusion Strategy
We evaluate four different strategies for integrating the sentence-level (V,) and phrase-level (V₂) features into unified representation F' = V, V,, which used for downstream prediction (see Sec. 3.4). The following options are compared in Tab. A7
Add: Element-wise addition of V, and Vn. This is our default configuration due to its
simplicity and efficiency.
Hadamard: Ellement-wise multiplication of V. and Vp,, emphasizing shared dimensions.
Gate: A learnable sigmoid gate is applied such that the fused feature F -
This allows the model adaptively weight sentence and phrase contributions.
Concat-mlp: The two features concatenated and passed through linear projection layer to match the original dimensionality.
As shown in Tab. A7 the Add method achieves the best overall performance considering both effec- tiveness and computational simplicity. While Concat-mip slightly improves R100.., its performance on RI @0.7 and mAP is inferior to Add. The Gate mechanism performs comparably but introduces additional parameters and complexity. We thus adopt addition our default fusion strategy due to its favorable trade-off between accuracy and efficiency.
E
Additional Visualization
We provide additional qualitative results in Figure A2 The visualizations demonstrate how semanti- cally aligned word tokens are clustered into meaningful phrases, as illustrated in A2 e). This grouping provides localized, clip-wise information that complements the global sentence- representation, particularly in cases where fine-grained cues are difficult to capture. As result, it enables more accurate and context-aware temporal grounding.
26

---

Query: group running and walks towards and rides In it
(c)
pt
group
o1 Interme
are
running
(a)
then
stope
and
DustGround
GT Sallency
wal
(b)
RashVTT
and
(c)
ides
(d)
GT Momont
DualGround
Query: black speaking moving train.
(c)
p1
pa
pa
p*
man
(8)
speaking
(b)
(c)
moving
(d)
Query: Woman flexes her muscles for photos.
pa
p*
Woman
her
(b)
cloo
(0)
photos
(d)
27

---

Query: examining grilled sandwich that ordered.
(c)
p
p²
p
18
DT steeve
innin
Illed
(b)
cho
sand
(c)
that
ordered
(d)
Figure A2: Additional Visualization results the QVHighlights validation split. (a) Moment retrieval predictions and (b) Highlight detection scores are compared across models. (c) L2 norm activation map of phrase-level embeddings, L2 activation map of sentence-level embeddings, and (e) Phrase-to-word attention map are visualizations from our proposed DualGround model, highlighting how it captures localized semantics and structured alignment.
28


Reviews:
[{"review_id": "BvVia3GSne", "reviewer": "Reviewer_Ys6A", "summary": "This paper argues that Video Temporal Grounding (VTG) models over-rely on the global [EOS] token, which neglects word-level cues and hinders fine-grained localization. To address this, it proposes DualGround, a dual-branch architecture that disentangles semantics. A Sentence-Level Path handles global alignment using the [EOS] token, while a Phrase-Level Path clusters words to capture local context. Fusing these paths aims for more accurate grounding, validated by state-of-the-art results on multiple benchmarks.", "questions": "### 1. On the Fundamental Motivation \nThe paper's motivation hinges on the premise that relying on the [EOS] token is a flaw. However, since the [EOS] token is trained to encapsulate the entire sentence's meaning, one could argue it is the ideal representation for a sentence-level task like VTG.  \n- Could you please elaborate on why and in which specific scenarios \"fine-grained temporal alignment\" is critical? Providing a concrete example with sequential actions (e.g., \"A man picks up a ball and then throws it\"), where the [EOS] token alone would be insufficient, would greatly strengthen your motivation. A convincing answer here is critical. If the authors can prove that a significant portion of the VTG task requires more than global semantics, my assessment of the work's Significance would increase substantially.  \n\n### 2. On the Phrase-Level Path's Mechanism \nThe effectiveness and interpretation of this core contribution are unclear, which impacts the perceived novelty and quality of the work.  \n- The paper claims to group words into \"semantically coherent phrases,\" but visualizations suggest the model attends to individual keywords. Could you clarify this discrepancy? Is the goal to form syntactic phrases or to disentangle core concepts?  \n- Your analysis shows the optimal $N$ depends on query length, suggesting the fixed-$N$ approach is a limitation. Have you considered methods for dynamically adapting $N$ based on query complexity? A discussion on their potential advantages or disadvantages would be insightful. Clarifying these points is essential for understanding your main technical contribution. A satisfactory response would significantly improve my assessment of the paper's Originality and Quality.  \n\n### 3. On the Sentence-Level Path's Mechanism \nThe paper claims that dummy tokens act as \"semantic attractors\" for irrelevant content, which is an interesting architectural choice.  \n- Could you provide a visualization (e.g., attention maps) to support this claim? Showing how irrelevant video clips attend more to dummy tokens while relevant clips focus on the [EOS] token would be necessary to validate this mechanism. Providing this evidence would increase my confidence in the paper's architectural Quality.  \n\n### 4. On Experimental Clarity and Justification Several details are needed to fully assess the validity of the experiments.  \n- For the \"Ours\" model in Figure 1(a), what was the exact configuration for the \"EOS only\" condition? Also, given that compared models have different architectures, is it fair to conclude the universal importance of word-level semantics from this experiment?  \n- What is the exact architecture for the baseline setting (a) in Table 3? Please confirm if it is a model composed solely of the sentence-level path. These clarifications are necessary for reproducibility. Answering them satisfactorily would increase my confidence in the experimental results and thus the paper's overall Quality.\n\n### 5. Positioning within Related Work on Holistic Understanding \n\"Let Me Finish My Sentence: Video Temporal Grounding with Holistic Text Understanding\" (Woo et al., MM '24), addresses the importance of holistic query understanding in VTG. While your paper separates global ([EOS]) and local (phrase) semantics, Woo et al. use a global text anchor to create a gating mechanism.\n- Could you discuss how your DualGround approach is positioned relative to this work? Specifically, what are the conceptual advantages and disadvantages of your dual-path separation compared to their global anchor-based gating mechanism? Situating your work in relation to highly relevant, concurrent research is crucial for clarifying its unique contribution. A thoughtful comparison would significantly strengthen the \"Related Work\" section and my assessment of the paper's Originality.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "### Strengths  \n- The proposed DualGround architecture is an intuitive solution. Its separation of global and local semantics is a well-motivated design that directly addresses the identified problem. \n- The method is validated with extensive experiments, achieving consistent state-of-the-art results across multiple benchmarks.\n\n### Weaknesses  \n- Questionable Core Motivation: The paper's central premise—that reliance on the [EOS] token is a flaw—is debatable. Since VTG is a sentence-level task, using a token designed to represent the entire sentence seems logical, not problematic. The paper needs a stronger justification for why this is a fundamental limitation.\n- The central claim of learning \"semantically coherent phrases\" is weakly supported. In visualizations like Figure 3(e) and Figure A2, the model appears to focus on individual keywords (e.g., 'man', 'speaking', 'train') rather than composing meaningful phrases (e.g., \"A man in black\"). This raises doubts about whether the phrase-level path is truly learning phrasal semantics or is acting more like a keyword-weighting mechanism. \n- The use of a fixed $N$ for all queries, regardless of their length or complexity, is a major limitation. The authors' own analysis in the supplementary material (Sec. C), which shows that the optimal $N$ varies by dataset based on average query length, ironically argues against a fixed $N$ and highlights the need for a dynamic approach.\n- While the overall architecture of DualGround is original, several of its core components are adapted from prior work. For instance, the ACA in the sentence-level path is from CG-DETR, the decoding path is from FlashVTG, and the idea for phrase generation is inspired by LGI. This slightly diminishes the paper's originality, as its main contribution lies more in the creative combination and design of effective existing modules rather than the invention of entirely new ones.", "was_revised": true, "final_justification": "The rebuttal clarifies many of the concerns in the original review. I raise my rating from 3 to 4.", "rebuttal": "# [W1, Q1] EOS Reliance and Motivation  \nA core motivation behind our work is not simply that attention concentrates on the [EOS] token, but that word tokens become functionally indistinguishable from [EOS] during cross-modal alignment. This phenomenon arises from two factors in prior VTG models.\n\n**1. Shared Projection for Cross-Modal Interaction:**  Most VTG models use a shared projection layer for all text tokens before cross-modal attention. As [EOS] is explicitly trained to represent global semantics, it becomes the dominant alignment target during training.  \n\n**2. Semantic Collapse:**  Since grounding loss can be optimized by aligning [EOS] with relevant video clips, word tokens are implicitly drawn toward [EOS] in embedding space. Their attention patterns converge, leading to semantic redundancy. (**For further clarification, please refer to Reviewer sNK1-[W3]**)\n\nThis convergence results in a semantic collapse where word tokens lose their individual meaning, acting as redundant surrogates of [EOS]. However, [EOS] is derived solely from the pretrained text encoder without visual grounding and may **overlook key visual cues** such as objects or actions. This limitation undermines accurate grounding in cases where localized semantics are essential.  \n\nOur model overcomes this challenge by leveraging both the strong sentence-level representation of the [EOS] token and the local semantics of the individual words. While [EOS] is still important for global understanding, our approach ensures that word-level semantics are preserved to localize precise moments that may be missed when relying solely on [EOS] for alignment.  \n\nThe above problem and our contribution are evident in Supplementary Fig. A2. In the query “A guy with red jacket is getting interviewed,” both baselines and DualGround’s sentence path (d) retrieve the interview scene but miss the visual cue “red jacket”. The phrase path (c), however, focuses on the correct clip and retrieves the precise moment. Similarly, third sample in Fig. A2 “Woman flexes her muscles for photos,” baseline models and sentence-level predictions attend to the photo-taking scene, while the phrase path correctly identifies the moment of muscle flexing.\nTo support this claim, we examine the attention weights of [EOS] token from InternVideo2 final self-attention layer. The tokens **“red jacket”** and **“muscles”** receive low attention scores, indicating that [EOS] underrepresents visually grounded content, especially when such cues are critical for localization.  \n\n| index | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 |\n|:---:|---|---|---|---|---|---|---|---|---|---|---|---|---|\n|**token**| [sos] | A | guy | _y | with | **red** | **jack** | **_et** | is | getting | **interview** | **_ed** | [eos] |\n| **[eos]** | 0.0908 | 0.0007 | 0.1205 | 0.0366 | 0.0083 | **0.0068** | **0.0037** | **0.0148** | 0.0284 | 0.0232 | **0.1064** | **0.2633** | 0.2959 |\n\n| index | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 |\n|:---:|---|---|---|---|---|---|---|---|---|---|\n| **token**| [sos] | Woman | flex | _es | her | **mus** | **_cles** | for | **photos** | [eos] |\n| **[eos]** | 0.0632 | 0.0429 | 0.0165 | 0.0906 | 0.0161 | **0.0961** | **0.0669** | 0.0301 | **0.2668** | 0.3103 |\\\n***\n\n# [W2, Q2] Interpretation of Phrase  \n\nOur use of the term **“semantically coherent phrases”** may have suggested the formation of linguistically well-formed expressions (e.g., \"a man in black\"). However, we would like to clarify intended meaning.\nOur approach focuses on grouping semantically related (**i.e semantically coherent**) word tokens based on their representational similarity in the feature space. Through our RPG and Slot Attention modules, each slot attends to a subset of tokens that tend to represent related concepts. While it is true that the visualizations suggest the model appears to focus strongly on individual keywords (e.g., 'man', 'speaking', 'train'), we interpret this behavior as a natural outcome of the model's tendency to first anchor on a core token, with surrounding words providing complementary context. \n\nWe recognize that in some cases, the phrase-level path may resemble keyword weighting, where individual word tokens dominate attention. However, this behavior is not unintended; it reflects the way the model clusters semantically related tokens in the feature space, leading to a **soft clustering** of related concepts rather than rigid syntactic units. This explains why certain words receive stronger attention, with the meaning of nearby surrounding words being integrated naturally to form coherent phrase-level representations.\n\nIn this sense, while the model’s behavior may sometimes appear similar to keyword weighting, the goal of grouping semantically related words into phrases is still upheld, and the representation effectively captures the relevant semantics for moment retrieval tasks.\n\n***\n# [W3, Q2] Static N Constraint \nWe agree that highlighting the limitations in our work is important. This question overlaps with **Reviewer 3BT3-[W1]**, and due to space constraints, we kindly refer the reviewer to our response to **Reviewr 3BT3-[W1]**.\n\n***\n# [W4] Module Originality Concern  \n\nWe acknowledge that core components in our design are inspired by prior works (e.g., LGI, CG-DETR, FlashVTG).  \nHowever, our aim was not to simply reuse existing modules, but to structurally reframe them within a novel dual-path architecture that mitigates their individual limitations. During phrase grouping, we incorporate position embeddings to reflect the sequential and contextual nature of linguistic phrases, an aspect often overlooked in prior approaches. We then apply Slot Attention to enhance semantic coherence and guide the resulting phrase representations toward the sentence intent via a reconstruction loss. These design choices collectively contribute to a more interpretable representation, which is central to our dual-path architecture.\n***\n# [Q3] Dummy Token as Attractor\n\nWhile we cannot include additional visualizations, we provide a theoretical explanation of the role of dummy tokens as **“semantic attractors”**.\n\nIn our sentence-level path, we introduce an auxiliary loss that regresses the attention weight assigned to the [EOS] token toward the highlight detection (HD) score of each video clip. (**This is outlined in Section 3.5 HD Loss**) Given that HD scores reflect the semantic relevance of each clip with respect to the input query, this supervision encourages the model to assign higher attention to the [EOS] token for salient clips. In contrast, non-salient clips—those with low HD scores—are not encouraged to attend to [EOS] and thus distribute their attention toward the dummy tokens.  \nAs a result, dummy tokens function as attractors for irrelevant content.\n***\n# [Q4] On Experimental Clarity and Justification\n1. **\"EOS only\" in Figure 1(a)**:  \n   This variant uses only the sentence-level path with the [EOS] token as input. The phrase-level path is completely disabled.  \n   \n2. **Purpose of the experiment**:  \n   The goal of this ablation is to justify the observed underutilization of word tokens in standard VTG architectures. More direct discussion of word semantics and their contribution to performance is provided in our response to **[W1, Q1]**.\n\n3. **Baseline (a) in Table 3**:  \n   This setting uses the sentence-level path with the full token sequence (i.e., including all word tokens and the [EOS] token). We acknowledge that this was not clearly distinguished from the \"EOS only\" setting in Figure 1(a), and we will revise the manuscript for clarity.\n***\n# [Q5] Comparison to Holistic Query Models\n\n**Conceptual Overlap:**\nWoo et al.[1] introduces a global anchor via mean pooling over text tokens, which gates noisy frames. This aligns conceptually with our Adaptive Cross Attention (ACA), where [EOS]—serving as a sentence-level anchor—is used to inform dummy tokens that absorb attention from less relevant clips.\n\n**Key Architectural Differences:**\nUnlike the approach of Woo et al., our design explicitly separates global and local semantics. We use the [EOS] token as the sole sentence-level representation and apply grouping modules to remaining word tokens into phrase-level clusters. This results in a dual-path architecture that treats sentence and phrase-level semantics as structurally distinct, enabling finer control and interpretability.\n\nThis distinction leads to several advantages.\n- **Interpretable Design with Localized Cue Awareness**:  \n   By separating [EOS] and word tokens into global and local paths, our model preserves token-level distinctions and captures localized visual cues, leading to stronger interpretability and fine-grained video-text interactions.\n\n- **Direct Gating Supervision**:  \n   Unlike Woo et al., our ACA’s attention weights are explicitly supervised using highlight scores, encouraging alignment between sentence relevance and video importance in a more grounded and controllable manner.\n\n- **Stability in Cross-Modal Matching**:  \n   Using a dummy token for gating provides a discrete mechanism to filter irrelevant clips, whereas feature-wise gating based on continuous similarity scores can be more sensitive to alignment noise between video and text embeddings.  \n\nHowever, we acknowledge trade-offs in this approach.\n- **Computation Overhead**:  \n   The dual-path architecture introduces additional components, increasing model complexity and training cost.\n\n- **Semantic Granularity Gap**:  \n   While the phrase path enhances fine-grained localization by focusing on localized semantics, this path may diverge from the globally consistent representation that highlight detection relies on. As a result, its influence on clip-wise saliency scoring can be limited.  \n\n[1] Woo, Jongbhin, et al. \"Let me finish my sentence: Video temporal grounding with holistic text understanding.\" Proceedings of the 32nd ACM International Conference on Multimedia. 2024."}, {"review_id": "lepJzxoqs2", "reviewer": "Reviewer_sNK1", "summary": "The paper first highlights a characteristic of existing video grounding methods whereby they focus more on the EOS token representing the full sentence, rather than focusing on the fine-grained details of the sentence. The method, DualGround, aims to improve on previous Video Moment Retrieval and Highlight Detection approaches by implementing a phrase-level path alongside a sentence-level path. The phrase-level path generates phrase-level representations for interaction with the video features, replacing joint word-level and EOS token interaction. This phrase-level path seems to be the key novelty of the approach. The method achieves improved performance over state-of-the-art on Moment Retrieval and also strong performance on Highlight Detection.", "questions": "- In the supplementary, Table A3 and A4, is it understood why EOS alone is better in many cases with IV2 features, but with SF+C features, it is consistently worse with just EOS? This suggests that there is a fundamental difference in the features that is causing this effect rather than it being simply a case of the models not utilising the word tokens. As the claim that these models are not using the word tokens is a key part of the motivation of the method, it would be good to have a better understanding of this behaviour, as it does not seem to be true for the SF+C features, with which many of these models were originally designed (at least CG-DETR and TR-DETR were, and these are the ones that show the biggest discrepancy with IV2 features)\n- It's not clear to me that using a phrase-level path is better than using a word-level path alongside a sentence-level path. So I am wondering if a similar improvement could be achieved in this manner using a word-level path. I understand this is likely not feasible in the rebuttal period but I wonder if any work has been done by the authors previously in this regard? If not, it could be interesting to test in the future (I'm not asking for it in the rebuttal period if it hasn't already been done). It would demonstrate further the effectiveness of the approach.\n- Overall  my rating is positive as I think the method is logical and it shows generally strong performance, even if the performance gains are not extremely significant. I do, however, still believe that the motivation of the paper could be improved, hence keeping the rating at only borderline accept.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths\n- The base-level motivation of the paper seems good, where it highlights that existing models treat word tokens and EOS tokens the same\n- The idea of using phrase-level information seems sensible, and the manner in which this is implemented seems well thought through and makes sense.\n- The performance is strong on Moment Retrieval, achieving reasonable, even if not extremely strong, performance gains, and performance is also competitive on Highlight Detection.\n- The ablations are insightful, Table 3 shows the importance of each component and displays that the different components and losses work well in tandem with each other\n- Table 4 is interesting, showing the particularly improved performance on longer sentence queries.\n- It is appreciated that performance is tested with both SF+CLIP and IV2 features.\n- The paper is generally well-written and the ideas are clearly described.\n\nWeaknesses\n- While the method appears to work well, achieving some performance gains, and the methodology largely makes sense, the main weaknesses that I see are related to the analysis of existing methods and the understanding of the motivation.\n- I worry that Figure 1(b) could be somewhat misleading. In my estimation, it would not be that surprising or alarming that a video grounding model would pay more attention to the overall sentence representation than it would to the individual words. It would be more surprising to me if the model paid more attention to an individual word. Rather than comparing the attention of individual words, I would be more interested to see the comparison between the EOS token attention vs the summation of the attention from the other words. If the model was paying more attention to the EOS token than all of the word tokens combined that would be more of a clear issue and sign of bias in my mind.\n- It would also be helpful if this could be summarised across a number of examples (ideally averaged across all examples) rather than just a single example. A single example does not provide much evidence for this behaviour. I think it would be beneficial for the paper if this motivation was more clear.\n- Following from this, it would be nice to see something similar to Figure 1(b) for the DualGround model, to show that the model is actually paying more attention to the phrase features. Seeing as this is part of the motivation for the approach, it would be helpful to see that the method works to alter that aspect of the model behaviour. Admittedly, I'm not sure if there's a direct equivalent possible in the DualGround model due to the manner in which phrase information is incorporated into the video features. Perhaps the attention weights in the phrase-guided aggregation. If something like this was possible it would be nice to see in the supplementary.\n- I think it's possible to say that it is debatable whether the model really achieves improved performance on Highlight Detection, as on QVHighlights test, FlashVTG does marginally better with IV2 features, while DualGround does marginally better with SF+C features. On QVHighlights val, DualGround does marginally better with IV2 features. \n- Minor point: It would be helpful to associate f_enc in the text with the \"dummy encoder\" in Fig. 2 to make it a bit more clear that these are the same thing", "was_revised": true, "final_justification": "The rebuttal has addressed my main concerns. The motivation for the phrase-level path is maybe not quite as clear as I think it could be, although it still broadly gets the point across. Overall I think the phrase-level path is shown to be useful when the sentence-level representation does not fully capture the semantics of the query, the method produces strong enough results, and the method makes sense within the context of the use of the phrase-level path. I adjust my rating to accept.", "rebuttal": "# [W1, Q3] Core Motivation\nFor a detailed explanation of our core motivation, we kindly refer the reviewer to our response to **Reviewer Ys6A’s [W1, Q1]**, due to space constraints.\n***  \n\n# [W2] Fairness of Attention Comparison  \nWe understand the reviewer’s concern and recognize that our intent with Figure 1(b) may not have been fully conveyed.  \nOur goal was not only to emphasize [EOS] as dominant in weight, but also to expose a deeper issue:  \n**In current VTG architectures, word tokens tend to lose their grounding and act as redundant surrogates of [EOS]**.   \nIn Figure 1(b), we highlight the phrase **“man in gray suits”**. Although the object appears clearly in both background and foreground clips (clips 38–44), the corresponding word tokens remain largely inactive in background clips. This indicates that word tokens fail to maintain their semantic grounding and function as weak surrogates of the [EOS] token. This pattern is consistent with our controlled experiments in Figure 1(a), where excluding word tokens (“EOS only”) yields performance that matches or exceeds full-sequence input in IV2 feature.\n***\n# [W3] Generalization of the EOS Over-Reliance\nWe provide a quantitative analysis across the entire dataset to validate the prevalence of the [EOS] over-reliance phenomenon. Specifically, we measure the statistical correlation between the attention weights assigned to the [EOS] token and those assigned to the individual word tokens during cross-modal interaction. To ensure robustness, we adopt both **Pearson and Spearman correlation coefficients**—widely used metrics for evaluating linear and rank-based relationships, respectively.  \n\nThe Pearson correlation coefficient is given by:\n\n$$\nr_{xy} = \\frac{\\sum_{i=1}^{n} (X_i - \\bar{X})(Y_i - \\bar{Y})}{\\sqrt{\\sum_{i=1}^{n} (X_i - \\bar{X})^2 \\sum_{i=1}^{n} (Y_i - \\bar{Y})^2}}\n$$\n\nWhere $r_{xy}$ measures the **linear relationship** between $X$ and $Y$, with $X_i$ and $Y_i$ as the individual data points, and $\\bar{X}$ and $\\bar{Y}$ as the mean values of $X$ and $Y$.\n\nThe Spearman rank correlation is computed as:\n\n$$\n\\rho_{xy} = 1 - \\frac{6 \\sum_{i=1}^{n} d_i^2}{n(n^2 - 1)}\n$$\n\nWhere $\\rho_{xy}$ evaluates the **monotonic relationship** between $X$ and $Y$, with $d_i$ representing the **rank differences** and $n$ as the number of data points.\n\nThe full measurement process is as follows:\n\n1. **Cross-modal attention map**: For each sample in the training and validation set, we extract the cross-modal attention matrix of shape $[L,T]$, where $L$ is the number of text tokens and $T$ is the number of video clips.\n2. **Isolate attention vectors**: We treat the attention matrix $A \\in \\mathbb{R}^{L \\times T}$ as a set of $T$-dimensional weight arrays, and separate the [EOS] token’s attention vector $A_{EOS} \\in \\mathbb{R}^{T}$ and those of the remaining $L-1$ word tokens $A_{word} \\in \\mathbb{R}^{(L-1) \\times T}$ for further correlation analysis.\n3. **Compute token-wise correlations**: For each of the $(L-1)$ word tokens' weight arrays, we compute the Pearson and Spearman correlation with $A_{EOS}$, resulting in $(L-1)$ correlation values.\n4. **Average correlation**: The average of the $(L-1)$ correlation scores is computed for each sample, and then the mean correlation is calculated across all samples in the subset, reporting values for both training and validation sets.\n\n| Model | Train | | Val | |  |  |  |  |\n|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n|  | **Pearson** | **Spearman** | **Pearson** | **Spearman** |  |  |  |  |\n| **CG-DETR** | 0.8960 | 0.8914 | 0.5962 | 0.7622 |  |  |  |  |\n| **TR-DETR** | 0.8110 | 0.7753 | 0.6021 | 0.6340 |  |  |  |  |\n| **FlashVTG** | 0.9745 | 0.9801 | 0.6771 | 0.7800 |  |  |  |  |  \n\nA high correlation (close to 1.0) indicates that word tokens exhibit nearly identical attention patterns as [EOS].\nPrior VTG models exhibit consistently high correlation values across both Pearson and Spearman metrics, indicating that the attention distributions of word tokens closely resemble that of the [EOS] token. This suggests that word-level semantics are highly overridden, resulting in identical alignment centered around the [EOS] token in general.\n***\n# [W4] Phrase-level Attention in DualGround\n| Model | Train |  | Val |  |\n|:---:|:---:|:---:|:---:|:---:|\n|  | **Pearson** | **Spearman** | **Pearson** | **Spearman** |\n| **DualGround** | 0.2218 | 0.2371 | 0.1185 | 0.1760 |\n\nWhile the updated NeurIPS policy prohibits additional visualizations in rebuttal, we instead provide a quantitative alternative to demonstrate that DualGround indeed leverages phrase-level information in a distinct manner.\n\nTo address this, we compute the Pearson and Spearman correlation between the sentence-level representation $V_s \\in \\mathbb{R}^{T \\times d}$ and the phrase-clip context vector $C \\in \\mathbb{R}^{N \\times T \\times d}$. Specifically, we first compute the L2 norm of each vector along the $d$-dimension to reduce both $V_s$ and $C$ to scalar activation maps over clips. For $C$, we obtain $N$ isolated activation maps per phrase and compute their individual correlations with the $V_s$. The average of these $N$ correlation scores is reported as the sample-level correlation.\n\nThese scores are substantially lower than the [EOS]-word token attention correlations observed in baseline models (**detailed in [W3]**). While not directly comparable due to architectural differences—our model structurally separates [EOS] from word tokens—the result indicates that the two paths in DualGround exhibit compensatory behavior. This suggests that the phrase-guided mechanism helps mitigate EOS over-reliance by preserving localized semantics.  \nWe hope these results help clarifying the functional role of each path and support the reviewer’s understanding of how DualGround leverages phrase-level information.\n***\n# [W5] Marginal Performance in Highlight Detection  \n\nWe acknoledge that the improvement in Highlight Detection (HD) is relatively marginal compared to the significant gains observed in Moment Retrieval (MR). We believe this difference stems from the fundamental nature of each task.  \n\nMR involves retrieving discrete temporal segments based on semantic matching. Unlike MR, HD requires regression of highlight scores for each clip. While our dual-branch architecture enables more accurate and semantically grounded moment retrieval—evidenced by the notable improvement in R1@0.7 metrics—it poses different challenges in HD. We hypothesize that the two paths in our model capture complementary yet distinct aspects of video-text alignment, and an uncalibrated fusion of the two could introduce inconsistencies in score prediction. As future work, we plan to explore adaptive mechanisms, such as gating, to selectively balance global (sentence-level) and local (phrase-level) information for more precise and robust highlight scoring.\n***\n# [W6] Clarifying the Dummy Encoder  \n\nWe thank the reviewer for pointing this out. We agree that explicitly linking f_enc in the main text with the \"dummy encoder\" depicted in Figure 2 would improve clarity. We will revise the manuscript to clarify this correspondence in both the figure caption and relevant textual descriptions.\n***\n# [Q1] Discrepancy between IV2 and SF+C\n\nThe performance gap between IV2 and SF+C in the [EOS]-only setting stems from differences in representation capacity and pretraining objectives:\n\n1. **Feature dimensionality**: Text encoder of IV2 produces 4096-dimensional embeddings, offering richer sentence-level semantics than the 512-dimensional CLIP features used in SF+C.\n2. **Pretraining domain**: IV2 is trained on video-text pairs, making its [EOS] token more aligned with temporal and action-related cues than CLIP, which is trained on image-text data.\n\nAs a result, IV2 feature can rely more on [EOS], while SF+C feature benefit from token-level cues due to weaker global representations. However, we view that the stronger reliance on word tokens in SF+C models does not reflect an intentional architectural design to utilize token-level semantics. Instead, it compensates for the weaker sentence-level representation of the [EOS] token in this setting. This is further supported by the fact that our model outperforms all baselines in the SF+C setup. By explicitly modeling phrase-level semantics, our method addresses the structural limitation of existing models that rely implicitly on token-level information without architectural support.\n\nLooking forward, as pretrained video-language models continue to improve and sentence-level representations become increasingly powerful (exemplified by IV2), the tendency of VTG models to over-rely on [EOS] is likely to intensify. We believe our approach offers a principled step toward mitigating this risk in future VTG research.\n***\n# [Q2] Phrase vs. Word-level Path\nTo leverage the localized semantics of word tokens in VTG, we cluster them into coherent phrase units. This phrase-level abstraction offers two advantages over a word-level path.  \n\nFirst, representing a sentence using N phrases (where N < L-1, the number of word tokens) reduces the computational overhead in subsequent processing while preserving semantic richness.  \n\nSecond, visually grounded concepts are often better represented by groupings of multiple word tokens rather than isolated words. Our phrase slots tend to cluster semantically related tokens in the feature space by residual grouping and slot attention. This often leads to tighter alignment with specific visual targets. For instance, in Fig. 1(b), tokens such as “man”, “gray”, and “suits” may individually activate across many irrelevant clips. When aggregated, their combined representation can focus attention more precisely on the intended entity, improving grounding precision.  \n\nWe agree that a direct comparison would further validate the effectiveness of our phrase-based design. We appreciate the reviewer’s suggestion and will explore this direction in future work."}, {"review_id": "c48WUf0MCZ", "reviewer": "Reviewer_ffuB", "summary": "This paper addresses the task of Video Temporal Grounding (VTG), which involves localizing video segments corresponding to a natural language query. The authors begin by identifying a fundamental limitation in prior work: existing models treat all text tokens uniformly, leading to an over-reliance on the global semantics encoded in the special [EOS] token while under-utilizing more granular, word-level information essential for precise alignment. To solve this, the paper proposes DualGround, a novel dual-branch architecture that explicitly disentangles global and local semantics. One branch, the \"sentence-level path,\" isolates the [EOS] token to handle coarse, global alignment. The other, more innovative \"phrase-level path,\" clusters word tokens into semantically coherent phrases and models their fine-grained interactions with video clips. This path uses a Recurrent Phrase Generator and a Slot Attention module to create structured, context-aware phrase representations. By fusing the outputs of these two paths, DualGround achieves a more robust and precise video-text alignment. The method demonstrates state-of-the-art performance on standard VTG benchmarks, including QVHighlights and Charades-STA, validating the effectiveness of its disentangled semantic modeling approach.", "questions": "**1. On Comparison with Relevant Keyword-Centric Approaches**\n\nThe core idea of the phrase-level path is to identify and ground semantically important word groups. This concept shares similarities with recently proposed methods like 'Keyword-DETR' [1](Um et al., AAAI 2025), which also focuses on identifying key textual elements for grounding. Could the authors comment on the relationship between DualGround's phrase-level path and Keyword-DETR? A discussion of the key similarities and differences in the Related Work section, and potentially a comparative analysis, would help to better situate the novelty of your approach.\n\n[1] Um, S. J., et al. \"Watch Video, Catch Keyword: Context-aware Keyword Attention for Moment Retrieval and Highlight Detection.\" AAAI 2025. \n\n**2. Ablation Study Results for Highlight Detection**\n\nTable 3 provides a clear ablation study on the contributions of the proposed components for the Moment Retrieval (MR) task. However, the corresponding ablation results for the Highlight Detection (HD) task are not presented. Given that the model is designed for joint MR and HD, could the authors provide the ablation results for HD as well? This would be crucial to confirm that the proposed components are beneficial for both sub-tasks.\n\n**3. Robustness of the EOS Reconstruction Loss**\n\nThe EOS Reconstruction Loss uses an InfoNCE objective, which works by treating other samples in the batch as negatives. This assumes that sentences from different video-query pairs are semantically distinct. However, in datasets that may contain many paraphrases or visually similar events described with slightly different sentences, this objective might incorrectly push apart the representations of semantically similar queries. Could the authors comment on the robustness of this loss component to datasets with high semantic similarity between different queries?\n\n**4. On the Quantitative Analysis of Computational Cost**\n\nThe proposed phrase-level path, with its RPG and Slot Attention modules, introduces additional complexity compared to single-path architectures. While the paper demonstrates superior performance, a quantitative analysis of the computational cost is not provided. Could the authors provide details on the model's efficiency, for instance, by reporting metrics such as FLOPs and inference latency, and comparing them against key baselines like FlashVTG and TR-DETR? This information would be crucial for understanding the practical trade-offs of the `DualGround` architecture.", "limitations": "Yes. The authors have explicitly and clearly addressed the limitations of their work in the final \"Conclusion, Limitation, and Future Works\" section. They correctly identify the fixed number of phrases per query and the lack of audio features as key limitations and suggest them as promising directions for future research. This is excellent practice.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengths:**\n\n- **Excellent Motivation and Problem Formulation:** The paper is exceptionally well-motivated. It starts with a clear and insightful empirical analysis (Figure 1) that convincingly demonstrates a core weakness in previous state-of-the-art models—their over-reliance on the [EOS] token. This strong diagnostic work makes the subsequent proposal of the `DualGround` framework highly compelling and easy to appreciate.\n- **Novel and Elegant Architecture:** The proposed dual-branch architecture is a novel and elegant solution to the identified problem. The core idea of disentangling sentence-level (global) and phrase-level (local) semantics is powerful. The phrase-level path is particularly sophisticated, combining a Recurrent Phrase Generator (RPG) for initial grouping and a Slot Attention module for refinement. This demonstrates a deep understanding of structured representation learning and applies it effectively to the VTG task.\n- **Thorough and Convincing Evaluation:** The experimental results are state-of-the-art and robustly validated. `DualGround` outperforms strong baselines across multiple standard benchmarks (QVHighlights, Charades-STA) and with different vision backbones. The comprehensive ablation studies (Table 3) clearly demonstrate the contribution of each novel component, and the analysis on varying query lengths (Table 4) provides strong evidence that the phrase-level path is key to the model's robustness on complex queries.\n\n**Weaknesses:**\n\n- **Architectural Complexity:** The phrase-level path, with its Recurrent Phrase Generator, Slot Attention module, and phrase-guided aggregation, is quite complex. While shown to be effective, this complexity might present implementation challenges and increase computational overhead. The paper could benefit from an analysis of the trade-offs between this complexity and performance.\n- **Lack of Discussion on Negative Societal Impact:** In the NeurIPS checklist, the authors state that the work is foundational research without direct deployment risks and thus omit a discussion on negative societal impacts. While the direct risks may be low, VTG technology could be a component in large-scale content monitoring or surveillance systems. A brief acknowledgment of this common dual-use concern is generally good practice in the field.", "was_revised": true, "final_justification": "I thank the authors for their thorough and convincing rebuttal, which has successfully addressed my primary concerns. The new quantitative analysis of computational cost provides a clear picture of the practical trade-offs of the proposed architecture, and the additional ablation study for the Highlight Detection task confirms the benefits of the model's components for both sub-tasks. The authors have also provided a thoughtful comparison with related work and have agreed to include a discussion on societal impact. Given that my main reservations have been resolved, I believe the paper's strengths in its motivation and novel architecture are significant. Therefore, I have raised my score from Borderline Accept to Accept.", "rebuttal": "# [W1,Q4] Parameter Analysis \nWe appreciate the reviewers’ concerns regarding the computational overhead introduced by our phrase-level architecture.  \nTo address this, we provide a quantitative comparison of model complexity and runtime efficiency against prior state-of-the-art baselines. The table below summarizes parameter counts, FLOPs, and inference throughput:\n\n| Model | Params(M) | GFLOPs | Thorughput (samples/sec) | MR (mAP) | HD (VG-mAP) |\n|:---:|:---:|:---:|:---:|:---:|:---:|\n| **CG-DETR** | 942 | 0.48 | 78 | 48.93 | 42.30 |\n| **TR-DETR** | 737 | 0.40 | 116 | 48.93 | 43.74 |\n| **FlashVTG** | 754 | 0.46 | 103 | 52.61 | 44.08 |\n| **DualGround** | 883 | 0.51 | 84 | **53.26** | **44.12** |\n\nAll measurements were conducted on a single NVIDIA RTX 3090 GPU, using a text token extracted via InternVideo2 (video feature length 768, text feature length 4096). For baselines (CG-DETR, TR-DETR, FlashVTG), we used their official base configurations including encoder and decoder layers. DualGround’s configuration follows Supplementary Table A2.  \n\nWhile DualGround has the highest FLOPs (0.51G), the increase over FlashVTG is only +0.05G, and the model still achieves real-time inference at 84 samples/sec. Given the consistent performance gains across Moment Retrieval and Highlight Detection, we believe this trade-off is justified. In addition, we carefully designed the architecture to offset complexity where possible:\n\n- **Sentence Path Efficiency:** Unlike previous works that apply cross-modal attention across all text tokens, DualGround passes only a single [EOS] token into the sentence path. This design significantly reduces attention computation in this branch.\n- **Fewer Dummy Tokens:** CG-DETR and FlashVTG use 45 and 40 dummy tokens, respectively, to maintain representational richness. In contrast, our EOS-centric sentence formulation requires only 3 dummy tokens, further reducing memory and compute cost.\n- **Efficient Phrase Operations:** The Recurrent Phrase Generator compresses (L-1) word tokens into N=4 phrase units, lowering the number of semantic elements involved in phrase-level operations. All downstream computations scale with N rather than (L-1), which ensures controlled overhead.\n\nLastly, we note that the primary bottleneck in VTG pipelines typically lies in the pretrained VLM encoders (e.g., InternVideo2, CLIP). The incremental cost introduced by our dual-branch structure is marginal in the context of end-to-end latency and does not affect practical deployment.\nWe believe DualGround maintains a favorable balance between architectural complexity and performance benefit. Our design provides interpretability and robustness with minimal additional cost, making it both effective and practically viable.\n\n***\n# [W2] Negative Societal Impact\n\nWe thank the reviewer for raising this point. While our work focuses on foundational modeling and is not directly intended for deployment, we acknowledge that Video Temporal Grounding (VTG) technology may be used as a component in broader systems with potential surveillance or content monitoring applications. We agree that it is good practice to recognize such negative societal impact, and we will revise our NeurIPS checklist to include a brief note on the potential societal implications and risks associated with downstream use.\n***\n# [Q1] Comparison with Keyword-DETR\n\nKeyword-DETR [1] emphasizes visually sparse individual word tokens as key elements for alignment. The underlying intuition is that words referring to visually rare or distinctive concepts are more likely to serve as effective grounding cues. While this approach successfully highlights semantically informative words, it operates to weighting individual token, where alignment targets may be ambiguous— isolated word tokens such as “man” or “gray” can appear in a wide range of contexts, leading to imprecise grounding. This issue is amplified by tokenization, where a single meaningful word may span multiple subword tokens, further complicating visual alignment.  \n\nOur approach is instead motivated by the observation that visually grounded concepts often correspond to multi-word expressions that convey a unified semantic meaning. Rather than assigning importance to individual word tokens, we focus on modeling interactions by grouping semantically related word tokens in the feature space. This approach captures compositional semantics more effectively and allows the model to attend to cohesive visual-linguistic patterns. We believe this design better supports the nature of semantic alignment required in moment retrieval tasks. \n\nIn summary, while both approaches share a common goal of identifying linguistically salient cues for temporal grounding, they differ in their granularity and modeling strategies.\n\n- **Key Similarities**: \n  - **Salient textual cues**: Keyword-DETR and our approach aim to identify semantically significant cues in the text for grounding. Both methods recognize the importance of focusing on certain words or phrases that play a pivotal role in connecting visual content with textual cues.  \n   \n\n- **Key Differences**: \n  - **Granularity**: Keyword-DETR isolates individual word tokens, especially those that refer to visually rare or distinctive concepts. Our approach focuses on grouping semantically related word tokens into compositional phrases, which we believe captures compositional semantics more effectively.\n  - **Modeling Strategy**: Our model integrates **sentence-level interpretability** with **fine-grained token-level analysis**, allowing it to both capture broad semantic meanings and effectively handle detailed, context-dependent cues in visual grounding.\n\n[1] Um, Sung Jin, et al. \"Watch Video, Catch Keyword: Context-aware Keyword Attention for Moment Retrieval and Highlight Detection.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 39. No. 7. 2025.\n***\n# [Q2] Ablation Study on Highlight Detecion  \n| Settings | RPG | Slot | L_DQA | L_EOS | MR-R1@0.7 | MR-mAP | HD-VG mAP | HD- VG Hit@1 |\n|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n| (a) |  |  |  |  | 56.13 | 52.24 | 44.04 | 70.88 |\n| (b) |  | ✔ | ✔ | ✔ | 56.99 | 52.46 | 43.97 | 71.12 |\n| (c) | ✔ |  | ✔ | ✔ | 57.83 | 53.02 | 44.09 | 71.29 |\n| (d) | ✔ | ✔ |  |  | 56.17 | 52.30 | 43.34 | 70.07 |\n| (e) | ✔ | ✔ | ✔ |  | 58.02 | 53.11 | 43.51 | 70.28 |\n| (f) | ✔ | ✔ |  | ✔ | 56.55 | 52.53 | 44.10 | 71.77 |\n| (g) | ✔ | ✔ | ✔ | ✔ | **58.97** | **53.26** | **44.12** | **71.62** |\n\nWe have added HD evaluation metrics (VG-mAP and VG-Hit@1) to our existing ablation table to assess the effectiveness of each proposed component across both MR and HD tasks. This addition confirms that the proposed components contribute positively to both MR and HD tasks, validating the effectiveness of our dual-path design across settings. While MR aims to retrieve a discrete moment that best matches the sentence, HD requires assigning a saliency score to every clip based on the global sentence meaning.  \n\nNotably, we observe that using the phrase-level path without phrase regulation ($L_{phrase}$, including $L_{DQA}$ and $L_{EOS}$) results in degraded HD performance. This degradation is particularly evident in the $L_{EOS}$ metric, which reflects the phrase path’s ability to preserve sentence-level semantics. Removing $L_{EOS}$ introduces noise into the sentence-level path by injecting unregulated fine-grained features. This highlights the importance of phrase regulation in preserving global semantic consistency during clip-level saliency modeling.  \n\nWhile our proposed components yield consistent performance gains in both MR and HD, the improvement on HD metrics is comparatively modest. We attribute this to the partial misalignment between fine-grained local cues captured by the phrase-level path and the global saliency requirements of HD. The phrase-level path helps discover fine-grained moments, but these do not always aggregate effectively into global saliency estimates. We believe this limitation can be addressed by introducing adaptive mechanisms (e.g., gating) that dynamically balance global and local semantics. We will incorporate these HD ablation results and the above discussion into the revised manuscript.\n***\n# [Q3] Robustness of the EOS Reconstruction Loss  \n\nWe appreciate the reviewer’s thoughtful question. We acknowledge that InfoNCE loss relies on the assumption that query sentences within a batch are semantically distinct, and that in datasets with frequent paraphrastic overlap, this may lead to false negatives, (i.e. semantically similar queries being treated as negatives) thereby incorrectly pushing their representations apart.  \nIn our current training setup, this issue is mitigated in practice by the diversity of sentence queries, which are derived from natural video descriptions that tend to vary across samples. Nonetheless, we recognize that in more paraphrase-rich datasets, this assumption may not always hold.  \nTo improve robustness in such cases, a practical extension involves leveraging a pretrained sentence encoder to compute semantic similarity between queries, thereby excluding highly similar pairs from the negative set. We believe this simple yet effective refinement can further enhance the stability and generalization of the InfoNCE objective."}, {"review_id": "yTgY4w4i0O", "reviewer": "Reviewer_3BT3", "summary": "This paper proposes DualGround, a dual-branch architecture for Video Temporal Grounding (VTG), which aims to localize relevant moments in untrimmed videos given a natural language query. Existing VTG models often over-rely on the [EOS] token for global semantics, overlooking fine-grained word-level cues and limiting temporal precision. To address this, DualGround combines a sentence-level branch that leverages the [EOS] token with Adaptive Cross Attention, and a phrase-level branch that clusters and refines word tokens into structured phrases for localized grounding. Experiments on QVHighlights and Charades-STA demonstrate that DualGround achieves state-of-the-art performance on both Moment Retrieval and Highlight Detection tasks, showing enhanced robustness to complex queries.", "questions": "While the proposed dual-branch design is motivated by semantic disentanglement, the phrase-level branch is still trained to reconstruct the [EOS] embedding. This raises concerns about potential semantic redundancy between the two branches.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The proposed DualGround introduces a clear and intuitive dual-branch architecture that separates global sentence-level and local phrase-level semantics, addressing a key blind spot in prior VTG models.\n2. The paper convincingly demonstrates that existing methods over-rely on the [EOS] token through controlled ablation experiments and attention map analysis.\n3. DualGround achieves state-of-the-art performance on both QVHighlights and Charades-STA benchmarks across multiple VTG metrics (e.g., R1@0.7, mAP, VG-Hit@1).\nWeaknesses:\n1. The model relies on a fixed number of phrases per query, introducing a dataset-specific hyperparameter that may hinder generalization and require manual tuning across different datasets and scenarios.\n2. While the architecture separates global and local semantics, potential semantic redundancy between the sentence-level and phrase-level representations is not examined. The paper lacks analysis on whether the two branches contribute complementary information or overlap in functionality.\n3. DualGround adds considerable architectural complexity—including dual semantic branches, Slot Attention, Recurrent Phrase Generation, and multiple loss functions—yet the performance gains over strong baselines like FlashVTG and TR-DETR are relatively modest (typically 1–2% mAP). Moreover, the paper does not provide any discussion of runtime cost, model size, or inference efficiency.", "was_revised": true, "final_justification": "The authors have provided an explanation of the complementarity between the two existing branches and offered a comparison in terms of model parameters. The explanation regarding a fixed number of phrases is also reasonable.", "rebuttal": "# [W1] Fixed Number of Phrases\n\nAs noted in our paper’s Limitation section, using a fixed number of phrase slots N across all queries is a constraint of our current design. Figure A1 in the supplementary material shows that the optimal N differs by dataset, reflecting variations in query length and linguistic complexity. This supports the reviewer’s suggestion that a dynamic strategy may offer better adaptability.  \n\nUsing a stable setting of **N=4**, our method consistently outperforms all baselines across both datasets, demonstrating robust and generalizable performance. For example, N=4 yields the best results on QVHighlights and causes only marginal drops on Charades (0.22 with InternVideo2, 0.09 with SF+C in R1@0.7).  \n\nWe also explored **dynamic clustering algorithms** such as FINCH, which operate directly on word tokens. However, these methods often struggled to form stable groupings, leading to optimization instability and inferior performance compared to our RPG-based formulation.  \n\nA promising direction for future work is a hybrid approach that **sets a maximum number of clusters and dynamically allocates them based on query complexity**. While such designs may increase computational cost and pose challenges in defining and supervising query complexity, we believe they offer strong potential for further improving adaptability and performance across datasets.\n\n***\n# [W2, Q1] Complementarity of Dual Branches  \n\nWe clarify that the two branches are designed to capture complementary semantics, not redundant information.  \n\nEmpirical evidence for the functional distinction between phrase and sentence branches is provided in visualization results ㅡL2 norm activation maps shown in Figure 3(c) (phrase-level) and Figure 3(d) (sentence-level) of the main paper, as well as Figure A2 in the supplementary material. The activation values indicate the degree of alignment between each pathway’s representation and the input video clips. These visualizations reveal that the two branches exhibit distinct activation patterns across time.  \n\nTo quantify distinction, we compute the correlation of clip-wise activation between phrase and sentence branches. The results, detailed in our responses to **Reviewer sNK1 [W3] and [W4]**, confirm that the two paths operate in complementary fashion.  \n\nMoreover, EOS reconstruction in phrase path is not performed via direct supervision to replicate the [EOS] token embedding. Instead, contrastive InfoNCE loss is adopted, where the [EOS] embedding is treated as a positive target and contrasted against other samples in the batch. This design encourages the phrase-level representation to capture semantically aligned information sufficient to distinguish the correct [EOS], but it does not enforce exact replication of the [EOS]. As such, the phrase branch retains its own fine-grained compositional structure while remaining semantically coherent.\nTogether, these findings support our claim that DualGround performs structured dual-path reasoning, rather than inducing semantic redundancy.\n***\n# [W3] Parameter Observation  \nWe provide a quantitative comparison of the parameter size, FLOPs, and inference throughput in the table below:  \nAll measurements were conducted on a single NVIDIA RTX 3090 GPU, using a text token extracted by InternVideo2 (video feature length 768, text feature length 4096). For baselines (CG-DETR, TR-DETR, FlashVTG), we used their official base configurations including encoder/decoder layer. DualGround’s model setting follows the QVHighlights dataset configuration described in Supplementary Table A2.  \n\n| Model | Params(M) | GFLOPs | Thorughput (samples/sec) | MR (mAP) | HD (VG-mAP) |\n|:---:|:---:|:---:|:---:|:---:|:---:|\n| **CG-DETR** | 942 | 0.48 | 78 | 48.93 | 42.30 |\n| **TR-DETR** | 737 | 0.40 | 116 | 48.93 | 43.74 |\n| **FlashVTG** | 754 | 0.46 | 103 | 52.61 | 44.08 |\n| **DualGround** | 883 | 0.51 | 84 | **53.26** | **44.12** |\n\nAlthough DualGround shows the highest FLOPs (0.51G), the increase over FlashVTG is only +0.05G, and inference remains real-time capable at 84 samples/sec. We argue that this slight increase is a reasonable trade-off, justified by the consistent performance gains in both Moment Retrieval and Highlight Detection.\nWhile the phrase-level path introduces additional computation, we design the architecture to offset this by reducing cost elsewhere:\n\n- Unlike prior models that apply cross-modal attention to full text tokens, DualGround routes only a single [EOS] token into the sentence-level path, effectively lightening the attention computation in that branch.\n- Both CG-DETR and FlashVTG utilize 40+ dummy tokens (45 and 40, respectively) to enhance representation richness across all token positions. In contrast, our method introduces only 3 dummy tokens (Ld=3), owing to the EOS-centric formulation of the sentence path, further reducing memory and compute.\n- At the word level, we perform lightweight clustering (e.g., N=4) via the Recurrent Phrase Generator, which processes (L-1) word tokens into N phrases. This drastically reduces the number of semantic units involved in phrase-level operations and ensures that the cost scales with N, not (L-1).\n\nFurthermore, the most time and computation consuming components in VTG pipelines are typically the pretrained vision-language encoders (e.g., InternVideo2, CLIP). In this context, the modest increase in FLOPs and parameter count introduced by our dual-branch architecture has negligible impact on the overall end-to-end inference latency.  \nTherefore, we believe that the computational trade-off is practically acceptable."}]

Author rebuttal:

Meta-review: This authors propose to disentangle textual semantics to address the problem of over-reliance on EOS token, which results in limited fine-grained localization. This work demonstrates intuitive solutions and validate them with extensive experiments and ablation studies. Even though there are several concerns regarding illustrative figures and technical complexities, the authors elegantly handle them. In overall, I recommend this paper being accepted.
Decision: Accept (poster)