Title: Empower Words: DualGround for Structured Phrase and Sentence-Level Temporal Grounding
Venue: NeurIPS 2025
Accepted: yes
Keywords: Video Temporal Grounding

Moment Retrieval

Cross-modal Learning

Vision-Language Models

Temporal Localization
Abstract: Video Temporal Grounding (VTG) aims to localize temporal segments in long, untrimmed videos that align with a given natural language query. This task typically comprises two subtasks: \textit{Moment Retrieval (MR)} and \textit{Highlight Detection (HD)}. While recent advances have been progressed by powerful pretrained vision-language models such as CLIP and InternVideo2, existing approaches commonly treat all text tokens uniformly during cross-modal attention, disregarding their distinct semantic roles. To validate the limitations of this approach, we conduct controlled experiments demonstrating that VTG models overly rely on [EOS]-driven global semantics while failing to effectively utilize word-level signals, which limits their ability to achieve fine-grained temporal alignment. Motivated by this limitation, we propose DualGround, a dual-branch architecture that explicitly separates global and local semantics by routing the [EOS] token through a sentence-level path and clustering word tokens into phrase-level units for localized grounding. Our method introduces (1) token-role-aware cross modal interaction strategies that align video features with sentence-level and phrase-level semantics in a structurally disentangled manner, and (2) a joint modeling framework that not only improves global sentence-level alignment but also enhances fine-grained temporal grounding by leveraging structured phrase-aware context. This design allows the model to capture both coarse and localized semantics, enabling more expressive and context-aware video grounding. DualGround achieves state-of-the-art performance on both Moment Retrieval and Highlight Detection tasks across QVHighlights and Charades-STA benchmarks, demonstrating the effectiveness of disentangled semantic modeling in video-language alignment.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes DualGround, a dual-branch architecture for Video Temporal Grounding (VTG), which aims to localize relevant moments in untrimmed videos given a natural language query. Existing VTG models often over-rely on the [EOS] token for global semantics, overlooking fine-grained word-level cues and limiting temporal precision. To address this, DualGround combines a sentence-level branch that leverages the [EOS] token with Adaptive Cross Attention, and a phrase-level branch that clusters and refines word tokens into structured phrases for localized grounding. Experiments on QVHighlights and Charades-STA demonstrate that DualGround achieves state-of-the-art performance on both Moment Retrieval and Highlight Detection tasks, showing enhanced robustness to complex queries.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The proposed DualGround introduces a clear and intuitive dual-branch architecture that separates global sentence-level and local phrase-level semantics, addressing a key blind spot in prior VTG models.\n2. The paper convincingly demonstrates that existing methods over-rely on the [EOS] token through controlled ablation experiments and attention map analysis.\n3. DualGround achieves state-of-the-art performance on both QVHighlights and Charades-STA benchmarks across multiple VTG metrics (e.g., R1@0.7, mAP, VG-Hit@1).\nWeaknesses:\n1. The model relies on a fixed number of phrases per query, introducing a dataset-specific hyperparameter that may hinder generalization and require manual tuning across different datasets and scenarios.\n2. While the architecture separates global and local semantics, potential semantic redundancy between the sentence-level and phrase-level representations is not examined. The paper lacks analysis on whether the two branches contribute complementary information or overlap in functionality.\n3. DualGround adds considerable architectural complexity—including dual semantic branches, Slot Attention, Recurrent Phrase Generation, and multiple loss functions—yet the performance gains over strong baselines like FlashVTG and TR-DETR are relatively modest (typically 1–2% mAP). Moreover, the paper does not provide any discussion of runtime cost, model size, or inference efficiency.\n\nQUESTIONS:\nWhile the proposed dual-branch design is motivated by semantic disentanglement, the phrase-level branch is still trained to reconstruct the [EOS] embedding. This raises concerns about potential semantic redundancy between the two branches.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-22"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the task of Video Temporal Grounding (VTG), which involves localizing video segments corresponding to a natural language query. The authors begin by identifying a fundamental limitation in prior work: existing models treat all text tokens uniformly, leading to an over-reliance on the global semantics encoded in the special [EOS] token while under-utilizing more granular, word-level information essential for precise alignment. To solve this, the paper proposes DualGround, a novel dual-branch architecture that explicitly disentangles global and local semantics. One branch, the \"sentence-level path,\" isolates the [EOS] token to handle coarse, global alignment. The other, more innovative \"phrase-level path,\" clusters word tokens into semantically coherent phrases and models their fine-grained interactions with video clips. This path uses a Recurrent Phrase Generator and a Slot Attention module to create structured, context-aware phrase representations. By fusing the outputs of these two paths, DualGround achieves a more robust and precise video-text alignment. The method demonstrates state-of-the-art performance on standard VTG benchmarks, including QVHighlights and Charades-STA, validating the effectiveness of its disentangled semantic modeling approach.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n- **Excellent Motivation and Problem Formulation:** The paper is exceptionally well-motivated. It starts with a clear and insightful empirical analysis (Figure 1) that convincingly demonstrates a core weakness in previous state-of-the-art models—their over-reliance on the [EOS] token. This strong diagnostic work makes the subsequent proposal of the `DualGround` framework highly compelling and easy to appreciate.\n- **Novel and Elegant Architecture:** The proposed dual-branch architecture is a novel and elegant solution to the identified problem. The core idea of disentangling sentence-level (global) and phrase-level (local) semantics is powerful. The phrase-level path is particularly sophisticated, combining a Recurrent Phrase Generator (RPG) for initial grouping and a Slot Attention module for refinement. This demonstrates a deep understanding of structured representation learning and applies it effectively to the VTG task.\n- **Thorough and Convincing Evaluation:** The experimental results are state-of-the-art and robustly validated. `DualGround` outperforms strong baselines across multiple standard benchmarks (QVHighlights, Charades-STA) and with different vision backbones. The comprehensive ablation studies (Table 3) clearly demonstrate the contribution of each novel component, and the analysis on varying query lengths (Table 4) provides strong evidence that the phrase-level path is key to the model's robustness on complex queries.\n\n**Weaknesses:**\n\n- **Architectural Complexity:** The phrase-level path, with its Recurrent Phrase Generator, Slot Attention module, and phrase-guided aggregation, is quite complex. While shown to be effective, this complexity might present implementation challenges and increase computational overhead. The paper could benefit from an analysis of the trade-offs between this complexity and performance.\n- **Lack of Discussion on Negative Societal Impact:** In the NeurIPS checklist, the authors state that the work is foundational research without direct deployment risks and thus omit a discussion on negative societal impacts. While the direct risks may be low, VTG technology could be a component in large-scale content monitoring or surveillance systems. A brief acknowledgment of this common dual-use concern is generally good practice in the field.\n\nQUESTIONS:\n**1. On Comparison with Relevant Keyword-Centric Approaches**\n\nThe core idea of the phrase-level path is to identify and ground semantically important word groups. This concept shares similarities with recently proposed methods like 'Keyword-DETR' [1](Um et al., AAAI 2025), which also focuses on identifying key textual elements for grounding. Could the authors comment on the relationship between DualGround's phrase-level path and Keyword-DETR? A discussion of the key similarities and differences in the Related Work section, and potentially a comparative analysis, would help to better situate the novelty of your approach.\n\n[1] Um, S. J., et al. \"Watch Video, Catch Keyword: Context-aware Keyword Attention for Moment Retrieval and Highlight Detection.\" AAAI 2025. \n\n**2. Ablation Study Results for Highlight Detection**\n\nTable 3 provides a clear ablation study on the contributions of the proposed components for the Moment Retrieval (MR) task. However, the corresponding ablation results for the Highlight Detection (HD) task are not presented. Given that the model is designed for joint MR and HD, could the authors provide the ablation results for HD as well? This would be crucial to confirm that the proposed components are beneficial for both sub-tasks.\n\n**3. Robustness of the EOS Reconstruction Loss**\n\nThe EOS Reconstruction Loss uses an InfoNCE objective, which works by treating other samples in the batch as negatives. This assumes that sentences from different video-query pairs are semantically distinct. However, in datasets that may contain many paraphrases or visually similar events described with slightly different sentences, this objective might incorrectly push apart the representations of semantically similar queries. Could the authors comment on the robustness of this loss component to datasets with high semantic similarity between different queries?\n\n**4. On the Quantitative Analysis of Computational Cost**\n\nThe proposed phrase-level path, with its RPG and Slot Attention modules, introduces additional complexity compared to single-path architectures. While the paper demonstrates superior performance, a quantitative analysis of the computational cost is not provided. Could the authors provide details on the model's efficiency, for instance, by reporting metrics such as FLOPs and inference latency, and comparing them against key baselines like FlashVTG and TR-DETR? This information would be crucial for understanding the practical trade-offs of the `DualGround` architecture.\n\nLIMITATIONS:\nYes. The authors have explicitly and clearly addressed the limitations of their work in the final \"Conclusion, Limitation, and Future Works\" section. They correctly identify the fixed number of phrases per query and the lack of audio features as key limitations and suggest them as promising directions for future research. This is excellent practice.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper first highlights a characteristic of existing video grounding methods whereby they focus more on the EOS token representing the full sentence, rather than focusing on the fine-grained details of the sentence. The method, DualGround, aims to improve on previous Video Moment Retrieval and Highlight Detection approaches by implementing a phrase-level path alongside a sentence-level path. The phrase-level path generates phrase-level representations for interaction with the video features, replacing joint word-level and EOS token interaction. This phrase-level path seems to be the key novelty of the approach. The method achieves improved performance over state-of-the-art on Moment Retrieval and also strong performance on Highlight Detection.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- The base-level motivation of the paper seems good, where it highlights that existing models treat word tokens and EOS tokens the same\n- The idea of using phrase-level information seems sensible, and the manner in which this is implemented seems well thought through and makes sense.\n- The performance is strong on Moment Retrieval, achieving reasonable, even if not extremely strong, performance gains, and performance is also competitive on Highlight Detection.\n- The ablations are insightful, Table 3 shows the importance of each component and displays that the different components and losses work well in tandem with each other\n- Table 4 is interesting, showing the particularly improved performance on longer sentence queries.\n- It is appreciated that performance is tested with both SF+CLIP and IV2 features.\n- The paper is generally well-written and the ideas are clearly described.\n\nWeaknesses\n- While the method appears to work well, achieving some performance gains, and the methodology largely makes sense, the main weaknesses that I see are related to the analysis of existing methods and the understanding of the motivation.\n- I worry that Figure 1(b) could be somewhat misleading. In my estimation, it would not be that surprising or alarming that a video grounding model would pay more attention to the overall sentence representation than it would to the individual words. It would be more surprising to me if the model paid more attention to an individual word. Rather than comparing the attention of individual words, I would be more interested to see the comparison between the EOS token attention vs the summation of the attention from the other words. If the model was paying more attention to the EOS token than all of the word tokens combined that would be more of a clear issue and sign of bias in my mind.\n- It would also be helpful if this could be summarised across a number of examples (ideally averaged across all examples) rather than just a single example. A single example does not provide much evidence for this behaviour. I think it would be beneficial for the paper if this motivation was more clear.\n- Following from this, it would be nice to see something similar to Figure 1(b) for the DualGround model, to show that the model is actually paying more attention to the phrase features. Seeing as this is part of the motivation for the approach, it would be helpful to see that the method works to alter that aspect of the model behaviour. Admittedly, I'm not sure if there's a direct equivalent possible in the DualGround model due to the manner in which phrase information is incorporated into the video features. Perhaps the attention weights in the phrase-guided aggregation. If something like this was possible it would be nice to see in the supplementary.\n- I think it's possible to say that it is debatable whether the model really achieves improved performance on Highlight Detection, as on QVHighlights test, FlashVTG does marginally better with IV2 features, while DualGround does marginally better with SF+C features. On QVHighlights val, DualGround does marginally better with IV2 features. \n- Minor point: It would be helpful to associate f_enc in the text with the \"dummy encoder\" in Fig. 2 to make it a bit more clear that these are the same thing\n\nQUESTIONS:\n- In the supplementary, Table A3 and A4, is it understood why EOS alone is better in many cases with IV2 features, but with SF+C features, it is consistently worse with just EOS? This suggests that there is a fundamental difference in the features that is causing this effect rather than it being simply a case of the models not utilising the word tokens. As the claim that these models are not using the word tokens is a key part of the motivation of the method, it would be good to have a better understanding of this behaviour, as it does not seem to be true for the SF+C features, with which many of these models were originally designed (at least CG-DETR and TR-DETR were, and these are the ones that show the biggest discrepancy with IV2 features)\n- It's not clear to me that using a phrase-level path is better than using a word-level path alongside a sentence-level path. So I am wondering if a similar improvement could be achieved in this manner using a word-level path. I understand this is likely not feasible in the rebuttal period but I wonder if any work has been done by the authors previously in this regard? If not, it could be interesting to test in the future (I'm not asking for it in the rebuttal period if it hasn't already been done). It would demonstrate further the effectiveness of the approach.\n- Overall  my rating is positive as I think the method is logical and it shows generally strong performance, even if the performance gains are not extremely significant. I do, however, still believe that the motivation of the paper could be improved, hence keeping the rating at only borderline accept.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper argues that Video Temporal Grounding (VTG) models over-rely on the global [EOS] token, which neglects word-level cues and hinders fine-grained localization. To address this, it proposes DualGround, a dual-branch architecture that disentangles semantics. A Sentence-Level Path handles global alignment using the [EOS] token, while a Phrase-Level Path clusters words to capture local context. Fusing these paths aims for more accurate grounding, validated by state-of-the-art results on multiple benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths  \n- The proposed DualGround architecture is an intuitive solution. Its separation of global and local semantics is a well-motivated design that directly addresses the identified problem. \n- The method is validated with extensive experiments, achieving consistent state-of-the-art results across multiple benchmarks.\n\n### Weaknesses  \n- Questionable Core Motivation: The paper's central premise—that reliance on the [EOS] token is a flaw—is debatable. Since VTG is a sentence-level task, using a token designed to represent the entire sentence seems logical, not problematic. The paper needs a stronger justification for why this is a fundamental limitation.\n- The central claim of learning \"semantically coherent phrases\" is weakly supported. In visualizations like Figure 3(e) and Figure A2, the model appears to focus on individual keywords (e.g., 'man', 'speaking', 'train') rather than composing meaningful phrases (e.g., \"A man in black\"). This raises doubts about whether the phrase-level path is truly learning phrasal semantics or is acting more like a keyword-weighting mechanism. \n- The use of a fixed $N$ for all queries, regardless of their length or complexity, is a major limitation. The authors' own analysis in the supplementary material (Sec. C), which shows that the optimal $N$ varies by dataset based on average query length, ironically argues against a fixed $N$ and highlights the need for a dynamic approach.\n- While the overall architecture of DualGround is original, several of its core components are adapted from prior work. For instance, the ACA in the sentence-level path is from CG-DETR, the decoding path is from FlashVTG, and the idea for phrase generation is inspired by LGI. This slightly diminishes the paper's originality, as its main contribution lies more in the creative combination and design of effective existing modules rather than the invention of entirely new ones.\n\nQUESTIONS:\n### 1. On the Fundamental Motivation \nThe paper's motivation hinges on the premise that relying on the [EOS] token is a flaw. However, since the [EOS] token is trained to encapsulate the entire sentence's meaning, one could argue it is the ideal representation for a sentence-level task like VTG.  \n- Could you please elaborate on why and in which specific scenarios \"fine-grained temporal alignment\" is critical? Providing a concrete example with sequential actions (e.g., \"A man picks up a ball and then throws it\"), where the [EOS] token alone would be insufficient, would greatly strengthen your motivation. A convincing answer here is critical. If the authors can prove that a significant portion of the VTG task requires more than global semantics, my assessment of the work's Significance would increase substantially.  \n\n### 2. On the Phrase-Level Path's Mechanism \nThe effectiveness and interpretation of this core contribution are unclear, which impacts the perceived novelty and quality of the work.  \n- The paper claims to group words into \"semantically coherent phrases,\" but visualizations suggest the model attends to individual keywords. Could you clarify this discrepancy? Is the goal to form syntactic phrases or to disentangle core concepts?  \n- Your analysis shows the optimal $N$ depends on query length, suggesting the fixed-$N$ approach is a limitation. Have you considered methods for dynamically adapting $N$ based on query complexity? A discussion on their potential advantages or disadvantages would be insightful. Clarifying these points is essential for understanding your main technical contribution. A satisfactory response would significantly improve my assessment of the paper's Originality and Quality.  \n\n### 3. On the Sentence-Level Path's Mechanism \nThe paper claims that dummy tokens act as \"semantic attractors\" for irrelevant content, which is an interesting architectural choice.  \n- Could you provide a visualization (e.g., attention maps) to support this claim? Showing how irrelevant video clips attend more to dummy tokens while relevant clips focus on the [EOS] token would be necessary to validate this mechanism. Providing this evidence would increase my confidence in the paper's architectural Quality.  \n\n### 4. On Experimental Clarity and Justification Several details are needed to fully assess the validity of the experiments.  \n- For the \"Ours\" model in Figure 1(a), what was the exact configuration for the \"EOS only\" condition? Also, given that compared models have different architectures, is it fair to conclude the universal importance of word-level semantics from this experiment?  \n- What is the exact architecture for the baseline setting (a) in Table 3? Please confirm if it is a model composed solely of the sentence-level path. These clarifications are necessary for reproducibility. Answering them satisfactorily would increase my confidence in the experimental results and thus the paper's overall Quality.\n\n### 5. Positioning within Related Work on Holistic Understanding \n\"Let Me Finish My Sentence: Video Temporal Grounding with Holistic Text Understanding\" (Woo et al., MM '24), addresses the importance of holistic query understanding in VTG. While your paper separates global ([EOS]) and local (phrase) semantics, Woo et al. use a global text anchor to create a gating mechanism.\n- Could you discuss how your DualGround approach is positioned relative to this work? Specifically, what are the conceptual advantages and disadvantages of your dual-path separation compared to their global anchor-based gating mechanism? Situating your work in relation to highly relevant, concurrent research is crucial for clarifying its unique contribution. A thoughtful comparison would significantly strengthen the \"Related Work\" section and my assessment of the paper's Originality.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}