Generalized Video Moment Retrieval
Authors: You Qin
Qilong Wu
Yicong Li
Wei Ji
Li Li
Pengcheng Cai
Lina Wei
Roger Zimmermann
Venue: ICLR 2025 Poster (2025)
Keywords: video moment retrieval
diverse query types
model versatility
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: In this paper, we introduce the Generalized Video Moment Retrieval (GVMR) framework, which extends traditional Video Moment Retrieval (VMR) to handle a wider range of query types. Unlike conventional VMR systems, which are often limited to simple, single-target queries, GVMR accommodates both non-target and multi-target queries. To support this expanded task, we present the NExT-VMR dataset, derived from the YFCC100M collection, featuring diverse query scenarios to enable more robust model evaluation.
Additionally, we propose BCANet, a transformer-based model incorporating the novel Boundary-aware Cross Attention (BCA) module. The BCA module enhances boundary detection and uses cross-attention to achieve a comprehensive understanding of video content in relation to queries. BCANet accurately predicts temporal video segments based on natural language descriptions, outperforming traditional models in both accuracy and adaptability. Our results demonstrate the potential of the GVMR framework, the NExT-VMR dataset, and BCANet to advance VMR systems, setting a new standard for future multimedia information retrieval research.

Published as a conference paper at ICLR 2025
GENERALIZED VIDEO MOMENT RETRIEVAL
You Qin 12x Qilong 2u
Yicong List, Wei Ji Shawn Li Pengcheng Cai', Lina Wei", Roger Zimmermann' Nannjng University, NNational University of Singapore, Shanghai Al Laboratory *University of Southern Califormia, "Zhejiang University
{ginyou, gilong wu) nus. edu, 1yc071719@gmaill com
wei jiu.iiuuduuu..n 1i 1 1 www.:ii00succcommm edu, rogerzecomp. nus edu. sg
Code: http://gttbbbbooom/2222iignngngn
ABSTRACT
In this paper, we introduce the Generalized Video Moment Retrieval (GVMR) framework, which extends traditional Video Moment Retrieval (VMR) to handle a wider range of query types. Unlike conventional VMR systems, which are often limited to simple, single-target queries, GVMR accommodates both non-target and multi-target queries. To support this expanded task, present the NEXT-VME dataset, derived from the YFCC collection, featuring diverse query scenarios enable more robust model evaluation. Additionally, we propose BCANet transformer- based model incorporating the novel Boundary-aware Cross Attention (BCA) module. The BCA module enhances boundary detection and uses cross- attention to achieve comprehensive understanding of video content in relation to queries. BCANet accurately predicts temporal video segments based on natural language descriptions, outperforming traditional models in both accuracy and adaptability, Our results demonstrate the potential of the GVMR framework, the VMR dataset, and BCANet to advance VMR systems, setting a new standard for future multimedia information retrieval research.
INTRODUCTION
Video moment retrieval (VMR) has rapidly evolved as a pivotal task in the domain of video under- standing, garnering significant attention the fields of information retrieval Ji et al. (2025) and multi-modal understanding Li et al. (2024a 2023a). At its core, VMR involves identifying specific segment or moment within a video that corresponds to a given query, usually in the form of a natural language (Zhao et al.) 2023 Li et al.) 20236 Ji et al. 2023b a). VMR is beneficial to series of downstream tasks, such as video question answering (Li et al. 2022a b Xiao et al. 2022 Ji et al.
2025 Li et al. 2023c Xiao al. 2025). video relationship detection (Shang et al. 2017 2021 T
Li al. 2021 and video dialog (Chu et al. 2021 PPham et al., 2022, Li et al. 2024b). and other multimodal tasks (Liu al. 2023b 2025 2024). This task holds immense potential across various applications, including interactive media, surveillance systems, educational tools, etc. The promise of VMR lies in its ability to refine and expedite the retrieval of video moments based on complex queries, thereby elevating user engagement and paving the way for groundbreaking advancements in multimedia information retrieval technologies.
Despite substantial progress, current VMR models face notable challenges, particularly in handling diverse and complex query scenarios. Traditional VMR approaches typically focus on simple, direct queries that correspond practicality of current VMR models, as real-world queries often encompass more nuanced and varied requirements, such as identifying multiple moments from single query processing queries with no relevant targets in the video. As shown in Fig. [ the retrieved video clips generated by VMR models can be multiple (" ("There are two dogs positioned in front a grassy field with woman nearby") ") or none ("Two cattle are positioned logether, a dog standing on a grassy field nearby") in real applications, which means the query can indicate any number of target video clips. Moreover,
single target moment within video. This restriction limits the
Equul contribution. 'Corresponding authors.

---

Published as a conference paper at ICLR 2025
Video
Timeline 05
3s
6.5
9.
125
15c
18s
21.5
24s
276
30
33%
376
14.0%
25.5s
28.9
No Target
Two containers are front of two adult, two dogs standing on a grassy field with standing nearby.
There two doys positioned In front on grassy field with woman nearby.
Twotle are positioned together, dog standing on grassyy field nearby.
(a) ne
(b) O ne-tooMMulti
(c) No Target
Query
Figure 1: Illustration of Generalized Video Moment Retrieval (GVMR). Each block in different colors represents different video clips and words with red underline represent the salient objects in the clip. Compared with current VMR datasets (a) that only define one clip as ground truth for each query, our GVMR output description-relevan video clips in various numbers, including multiple (b) or even none (c), which is more in line with real applications.
addressing these challenges necessitates a more generalized VMR model that can handle a broader spectrum of query types, including both one-to-multi and no-target queries, thereby enhancing the model's versatility and applicability in real-world scenarios.
Motivated by these challenges and inspired by Liu al. (2023a), introduce an innovative framework termed Generalized Video Moment Retrieval (GVMR). GVMR is designed to adeptly navigate the complexities associated with multitude of query types, extending the conventional VMR paradigm encompass both multiple and no-target queries. By harnessing comprehensive dataset that encapsulates broad range of query scenarios, GVMR aspires to deliver robust and flexible solution for video moment retrieval, significantly enhancing the model's interpretative prowess, precision, and retrieval efficiency.
Motivated by this, we build specialized dataset, NEXT-VMR, which is derived from the YFCC100M dataset (Thomee et al., 2016) after meticulous construction and analysis. This dataset tailored specifically for GVMR, featuring diverse array of query types, including one-to-multi and no no-target queries. The dataset is meticulously labeled and processed to reflect real-world complexities, with a balanced distribution of various query types. This rich dataset serves the foundation for training and evaluating the VMR model generalized scenario, providing a comprehensive understanding of its performance in a more general setting.
Besides, we also propose new Video Moment Retrieval (VMR) model named BCANet, designed enhance the performance on the challenging task of VMR, where the model needs to accurately predict variable numbers of temporal timestamps based on natural language descriptions. BCANet leverages sophisticated transformer encoder-decoder architecture, which harmonizes vision and language features through advanced pre-trained models, thereby creating a unified dimensional space for feature integration. The cornerstone of BCANet the Boundary module, an inventive component that utilizes contrastive learning to produce boundary-awand video features. This module further employs query-region cross attention, ensuring a global and uniform perception of video features by decoder queries, which significantly enhances the precision and accuracy in predicting the start and end times of video segments, along with generating evidence scores for each prediction. The BCA module is particularly notable for its dual components: the Boundary- Contrastive Learning (BCL) and the Query-region Cross Attention (QCA). BCL focuses utilizing ground-truth labels contrast region features with sentence features, fostering the
Cross Attention (BCA)
2

---

Published as a conference paper at ICLR 2025
generation of video features that acutely aware of boundary information. This is complemented by QCA, which leverages these enhanced features to ensure that decoder queries have a comprchensive and uniform understanding of the video content, enabling precise retrieval of relevant video moments.
In summary, our contributions are summarized below:
We first introduce a novel Generalized VMR task that significantly expands the scope of traditional VMR, catering to a wider range of query types and enhancing the model's practical utility.
We present a meticulously curated dataset, NEXT-VMR, specifically designed for GVMR task, which includes a variety of query types and scenarios, thereby enabling more robust and versatile model training and evaluation.
To deal with GVMR, we propose the BCANet model, which incorporates the innovative Boundary-aware Cross Attention and Query-region Cross Attention modules. This model is underpinned by the Boundary-awe modeling of interactions and flexible adaptation diverse video features.
Contrastive Learning strategy, enabling effective
RELATED WORKS
2.1
VIDEO MOMENT RETRIEVAL DATASETS
Current video moment retrieval (VMR) datasets vary in their focus, ranging from everyday activities to specific scenarios, each contributing uniquely to the advancement of VMR models. Here summarize the key features and challenges presented by notable VMR datasets.
Diverse Scenarios and Complexities in VMR Datasets: Datasets like DiDeMo (Anne Hendricks et al.] 2017) and TEMPO (Anne Hendricks et al., 2018). originating from vast video collections like YFCC100M present scenarios of varied human activities. DiDeMo (Anne Hendricks al. 2017). with its structure of five-second video segments, offers a simplified retrieval challenge with fixed-length segments. TEMPO (Anne Hendricks et al. 2018)extends this by incorporating more complex, genarrated language queries, pushing models towards a better understanding of natural language in VMR. contrast, Starades-STA (Sigurdsson et al.] (016), derived from the Charades dataset, delves into daily indoor activities, enriching VMR with semi-automatically generated annotations that align detailed activity labels with video descriptions.
Challenges in Scale and Annotation Diversity: ActivityNet Captions (Caba Heilbron et al. 2015)
based on the ActivityNet benchmark, stands out for its scale and application in dense video captioning, VMR resource. It offers broad spectrum of human activities, crucial for training models in diverse, dynamic environments. TACoS (Regneri et al., (013a),, focusing cooking activities, brings scene- spennff challenges, combining fine tineigrinedd activity labels with natural language descriptions. This specificity in scene offers a unique testbed for VMR models to understand and process scene-specific activities and their linguistic descriptions.
effectively doubling
Advancing VMR with Long-form Content and Unbiased Annotations: The MAD dataset (Soldan al. 2022) marks a significant evolution in VMR, shifting towards mainstream, long-form movies. It challenges conventional VMR approaches with its vast scale and nuanced, descriptive queries. Unlike other datasets, MAD addresses the issue of hidden biases in VMR, providing a more accurate and unbiased annotation framework. This shift towards fong-form content opens up new avenues in VMR, particularly understanding and retrieving moments from extensive, continuous video streams.
Synthesis of Real-world Application and Research: Collectively, these datasets offer comprehen- sive landscape for VMR research, each introducing distinct challenges from fixed-length segments in DiDeMo to complex, multi-scene activities in ActivityNet Captions, and the narrative-driven. long-form content of MAD. The diversity in video content, query complexity, and annotation methods across these datasets underscores the multifaceted nature of VMR challenges, driving the development of more sophisticated and versatile VMR models. To make VMR closer to real application, propose the generalized video moment retrieval task with clips in arbitrary numbers as retrieval results for each query in this paper.
3

---

Published as a conference paper at ICLR 2025
3 PROPOSED DATASET
Query: Two adults are pushing each other in green field with white house in the background.
Human Check & Modification
Follow these steps and only output one refined sentence:
1. Remove the letters(" A "" ...) and change(like cat A in front of cat B" --> two cats") in F Foreground.
2.Add Background scene 5 words to the end of Foreground with preposition.
@ OpenAI
Combine Refine
GPT-3.5 Turbo
3. Make sure the output sentence coherent and grammar correct,
ChatGPT API
Background: green field with white house in the background.
adult A pushing the adult C, and the adult C is pushing the adult A.
Foreground:
& LLaVA
Scene Caption
Prompt: Describing background scene(excluding foreground objects) within 30 words.
Extract Key Frame
Processed Q Query
Figure 2: The overall pipeline for Query Generation in NExT-VMR dataset. With the help of current LLM models, such as GPT and LLaVa, we can generate queries fine-grain semantic, to benefit the video content understanding to the next stage.
3.1 TASK DEFINITION
This section defines two primary challenges in video moment retrieval tasks: no-target queries and one-to-many queries. These task definitions crucial for understanding and improving video moment retrieval approaches.
No-Target. In no-target queries (n 0), the query statements have no corresponding target moments in the video content. This scenario is common in real-world applications, as user queries may not relate the video content. The key to evaluating this task the model's ability correctly identify these queries as no-target and respond with corresponding empty timestamps.
One-to-Multi. In one-to-multi queries, we focus on how a single query statement corresponds to multiple moments in a video. Unlike traditional retrieval tasks which map query to only one moment, this task reflects more complex and realistic scenario where a single query may pertain to several relevant segments within the content. This reflects real use cases where inquiry could span multiple, non-consecutive events or scenes that share a thematic or narrative link. The evaluation of such queries necessitates model that can not only retrieve multiple distinct moments but also understand the broader context that binds these moments together. Success in this task is indicative of model's 's advanced comprehension capabilities. allowing it provide comprehensive response that encapsulates all pertinent instances within the video, thereby enhancing the user's 's search experience by acknowledging the multifaceted nature of their request.
4

---

Published as a conference paper at ICLR 2025
3.2 DATA SOURCE
We utilizes the YFCC100M| I dataset as the primary data source. As the largest publicly collection, the YFCC100M dataset offers wealth of video content. We extend application to video moment retrieval tasks, focusing particularly on one-to-many queries and no-target queries.
3.3 DATA PREPARATION
We manually annotated the original YFCCI00M video collection with (subject, predicate, object) relation tuples and their corresponding timestamps {(t,, t. then transformed the tuples into grammatically correct sentences to serve as query statements. Here, indicates the number of correct relation tuples in query statement. Next, we create arbitrary combinations of relation tuples for and n 3 scenarios. For instance, for 3, we intersect the timestamps of three relation tuples to form a new query statement. These query statements are used to match corresponding moments videos.
3.4 DATA LABELING & PROCESSING
3.4.
No-TARGET GENERATION
In constructing the No-Target part of the dataset, we specifically generated such query statements using the following methods:
Randomly Modifyyng Query Statements. We select (s, relation tuples from n = 1 scenarios, randomly modifying the subject (s) or object (o) to make them unrelated the video content. We also randomly change the predicate (p) to create new query statements.
Ensuring Grammatical Diversity, We use a dictionary mapping to ensure the generated query state- ments are both grammatically correct and incorrect, approximately in a 1:1 ratio, adding complexity the task.
Cross-Video Query Statements. Other query statements come from non-target videos, i.c., n query statements randomly extracted from other videos the dataset.
3.4.2 OUULTO-MULTI GENERATION
When constructing the One-to-Multi part of the dataset, we followed these steps:
Constructing Single Relation Pair Queries (n 1). We first extract (s, o) relation tuples from the original YFCC100M dataset and convert them into grammatically correct sentences, forming single relation pair queries. Here, n = 1 indicates that query statement contains only one correct relation tulle, which can correspond to a precise moment in the video (i.e., the ground truth moment's timestamps).
Combining Query Statements (n 1). Next, we combine different single relation pair queries to create 2 and 3 scenarios. Specifically, we look for intersections in the timestamps of relation tuples from different query statements. For example, in the case of n 3, there are three original relation pairs, and their respective ground truth timestamps are (t) t.), (t. ( and (t2, f2).
When these relation pairs have a temporal intersection, we define this intersection as (t',, f.) this combine these relation pairs into query statement and define it as the = 3 scenario.
Moreover, during the pairing phase, we implement a 5% threshold to curtail the creation of intersecting queries originating from abundance of ephemeral moments. This measure is instituted diminish the potential for superfluous noise within the query set. In addition, we synthesize the extracted bubject-predicate-obyee
(s, o) relational pairs by amalgamating multiple intersecting pairs of (s, o) using the conjunction ",and", This approach facilitates the construction of more streamlined queries that cohesively represent the combined elements of the foreground.
Videos from YFCC100M (Thomee al. 2016 directly crawled from Flickr.
5

---

Published as a conference paper at ICLR 2025
3.4.3 SCENE CAPTION & QUERY REFINEMENT WITH LLM
Figure 2 iillustrates detailed overview of our whole query generation and refinement pipeline. These initial queries are derived from rich, contextuall--wware scene graph generated for each video. To enhance the descriptiveness and semantic richness of the background scene graph, we employ the advanced Ilava-v1.5-76 model with Int8 Quantization, which is renowned for its efficiency and accuracy in scene understanding and caption generation tasks.
Subsequent to the background scene graph captioning, we integrate the scene's background in- formation with the dynamic foreground elements. This holistic approach ensures that each query encapsulates a comprehensive view of the video content, accounting for both the primary subjects of interest in the foreground and the contextual details provided by the background.
The final and critical phase of our pipeline involves the refinement of these merged queries. For this purpose, we leverage the powerful natural language processing capabilities of the gpt-3.5-turbo API. The API's nuanced understanding of language nuances allows us to polish the queries to meet high standards of clarity and relevance, thus greatly enhancing the overall quality and effectiveness of our video moment retrieval system as depicted in the process flow of Figure Additional statistical analyses of the dataset are provided in Section A.1
Table 1: Comparison of NEXT-GVMR with various existing VMR datasets. Two columns of Multi- target and No-target indicate if the dataset could be used for Multi-target No-target Moment Retrieval. indicates that the senario which excludes no target queries.
Dotuse
# Videos
Queries
Avg. Query/Vidco
Arg. Segiuutty
Domain
Multi-torget
No-target
ANes-C ANet-Captions
Caba Heilhron et al. 2015
14,926
71,953
4.83
Open
Chandos-SA Stqunssoon 2 2010
9,818
27,847
2.83
Indoor activities
DiDeMo
Amme Heenireks
10.461
40,513
3,87
Open
TEMPO
10,464
Open
MAD Solaas at -0zz
650
ts: 384,600
5.9
Movics
QVEigngen ragns Let 2 2
10,148
18,367
1,02
1.8
Vlog / News
TAC&S
Rognen
127
18,818
143.52
Cooking
NEXT-VMR (Ours)
9,957
Open
NEXT VMR (Ous)
9.957
153,191
15.39
1.51
Open
3.5 DATASET COMPARISON
Finally, we compared our dataset with several other popular video moment retrieval datasets. This comparison mainly focuses on data diversity, difficulty, and practical application value. As shown in the Table | our dataset, GVMR, stands out in several aspects. Unlike other datasets, GVMR has two versions: one with focus multi-target instances and another that includes no-target instances, enhancing the complexity and variability in video moment retrieval tasks. GVMR contains 9,957 videos with over 123,238 queries in version and 153,191 in another, indicating substantial number of unique query-video pairings. This volume is comparatively higher than most datasets listed, with the exception of MAD, which has a high number of queries per video due to its movic domain focus. However, GVMR provides a more extensive average number of segments per query. which can be more challenging for retrieval algorithms. Moreover, GVMR's domain open, making applicable to a wide range of real-world scenarios, unlike others that are confined to specific domains such as indoor activities, cooking, or movies. The design of GVMR, which includes nulti-target and no-target instances, not only increases its relevance for practical applications but also sets a higher bar for the development of vidco moment retrieval systems by requiring them to discern more nuanced aspects of video content.
PROPOSED MODEL.
In the GVMR task, the input comprises an untrimmed video, represented as V = (v.) i e {1..... M) and natural language description, denoted W (w,).. {l,..., NY. where M and indicate the total number of frames in the video and words in the description, respectively. The objective is accurately predict a of temporal timestamps {(1 {8) k Z, with f, and

---

Published as a conference paper at ICLR 2025
Test Encodor
Text Feeture
Query-region Cross Allention (QCA)
Fy. gg
Quary with Ground Trusk:
Pooting 84
Quen: degs pertlcii
Thes leo daage postboaed or greay fald wilk noceaa seally.
Boundary Aware Contrastive carning (BCL)
seatty
Query-Test ttention
Transfrmet Encoder
without Ground Truhh
Self Atenton
Query: Tho Ted og For peojAs
Transformer Decader
Pooling
Ray
Prony Loss
Video Encoder
Fv
Query
Figure 3: The architecture of our proposed BCANet, primarily comprising the Boundary- Contrastive Learning and Query-region Cross Attention modules. These components enable learnable queries to uniformly perceive and analyze intricate details within video segments, facilitating robust time span predictions. Additionally, confidence scores are utilized for ranking final predictions and filtering in no-target detection samples.
te representing the start and end timestamps, respectively. Unlike previous approaches, this task introduces an increased level of complexity as the number of predicted timestamps is variable. This means that natural language description may correspond none, one, or multiple segments within the video. Our evaluation metrics penalize both over-prediction and under-prediction of target objects, underscoring the need for precise temporal localization response to the given descriptions.
4.1 MODEL OVERVIEW
To tackle the GVMR task more effectively, we employ transformer encoder-decoder model (Figure 3). The video V is processed by two pre-trained models-SlowFas (Feichtenhofer et al., 2019) and the CLIP video encoder (ViT-B/32) extract vision features Fv E RmXC. where the temporal length and C. the feature dimension. For language input, the CLIP text encoder extracts token features FQ * (with tokens and Cg the hidden dimension). Two linear layers then align these features unified dimension C, yielding Fy E Rm and Fq E B Following (Lei jet al. 2021). concatenated video and sentence features are fed into a transformer encoder to produce hidden features H that interact with decoder queries via -ross-attention for boundary prediction. These queries are generated by the Boundary-aware Cross Attention (BCA) module (Section (4.2), where Fq is average-pooled and refined by MLPs to yield the sentence feature S; combining S with boundary features from Fv forms positive and negative pairs for boundary-aware contrastive learning. Subsequently. L learnable queries interact with the enhanced features via query-region attention, supervised by proxy loss to ensure accurate local perception. The final queries predict start and end times T= = {(th } and evidence scores k through three-layer MLP. No-target detection uses threshold 6, classifying samples no-target if Ve E {e*). < 8. Both T and evidence scores are supervised by the matching cost introduced (Carion et al. 2020).
4.2 BOUNDARY-AWARE RELATION MODELING
This section introduces the Boundary-aware Cross Attention (BCA) module, which combines two key components: (I) Boundary-aware contrastive learning, leveraging ground-truth labels to produce boundary-aware video features, and (2) Query-region cross attention, ensuring decoder queries uniformly perceive these enhanced features for more accurate predictions.
Boundary-awane Contrastive Learning (BCL): Unlike previous boundary modeling approaches (Wang et al.) 2019), which directly perform binary boundary classification on the features before grounding MLP. our component leverages ground-truth labels to aggregate region features and
7

---

Published as a conference paper at ICLR 2025
contrast these with sentence features generate boundary-aware video features. Specifically, it utilizes ground-trutt label information (tx,t) to collate video features within each labeled duration. The feature indices in the temporal space are denoted as (1))))) By applying average pooling, region feature unit (Ro,,) is obtained, formulated as:
R AxgPoolingg(F
Here, represents the concatenation operation, and a indicates the index corresponding to the j-th ground-truth of the i-th sample in training batch. For non-target samples, the entire video feature set is utilized negative sample:
Raco AvgPooling(Fr)
(2)
Subsequently, the sentence feature S, contrasted with the above-mentionee region features to stretch the feature space for contrastive learning. In each batch, the positive sample is the region features (Ra.y) corresponding to the sentence S/s ground-truth label, with other batch indices forming negative pairs. This forms the basis for symmetric contrastive learning loss calculation:
E, exp (cos (Ro.. S.) /T)
CB llog
(3)
Σ E, exp (cos (Rou.s St) /T)
E, exp (cos (Si, Rois) /T)
LS-IR
loog
Σ E, exp (cos (St, Roya) /T)
Here, r denotes the exponential temperature that controls the sharpness of the loss function. Notably, for no-target sample, positive sample contrastive learning loss is not computed due to the absence of positive samples.
Query-region Cross Attention (QCA): After enhancing features through contrastive learning to obtain Fv, after the further cross-attention with text word-level feature Fg and self-attention to produce Fy the QCA mechanism applies cross-attention between Fy and learnalle queries Q E BLXC Attention maps computed
A (Boimax E R4xm
(5)
where is the GELU activation function. These L span attention maps focus on different video regions, guided by proxy loss. Region-aware features, Q, Ano(Fv W.j). which, alongside Fy and Fo. is input to the transformer decoder. The proxy loss (Errery) identifies queries as positive (yr = 1) or negative based alignment with ground-truth spans:
Sproxy
log(y:(AAt)
This guides the decoding queries to learn features from correct spans, improving prediction accuracy.
4.3 LOss FUNCTION
To facilitate moment localization, we employ set prediction loss, grounded bipartite matching as outlined in (Lei al. 2021). This process involves calculating the optimal assignment based on the similarity of timestamps and their respective confidence scores using the Hungarian algorithm. The ground truth moment timestamps, denoted as T, E [o, 1 consist of normalized center coordinates and width. The optimal assignment is determined follows:
8

---

Published as a conference paper at ICLR 2025
Table Comparison of results of various SOTA models and BCANet on the NEXT-VMR dataset, where la represents the IoU thresholds to categorize predicted moment as "correct". The bold value denotes the highest performance in each prediction type category for each metric. "*" indicates that non-target scenarios are ignored, considering only queries with at least one ground-truth annotation.
Model
Pred. Type
@t
mloU
Avg.
N-ace:
T-acc
0.3
0.5
0.7
0.25
0..
0.75
Moment-DETF DETT Lei et
Multi
56.75
30.36
44.41
62.90
47.03
47.45
94.28
QD DETE (Mods et ai 2023
Mulä
62.27
47.70
35.10
48.94
69.98
53.87
37.05
$3.64
77.68
92.67
EaTR Jang
Multi
63.85
44.81
34.50
49.44
72.94
55.17
36,91
55.67
76.44
95.68
Prompt
Chat UniVi
Uin et al A1241
Prompt
46.15
27.52
13.81
29.62
42.13
23.71
9.95
25.26
71.55
68.48
BCANet (Our))
Multi
65.06
48.61
37.10
50.24
72.75
56.29
41,22
56.75
77.92
95.79
N
8' arg min d'EWN
Σ I- AcPe.or(s) + C(Ts, Ta(()))
(7)
C(T,, (a(())) A, (a((s) + Aiou (Te, Ta((s)),
(8)
where A Aiou, and d are balancing parameters. Using the optimal assignment Ø', the moment localization loss is defined as:
N Σ
mmmmnnt
do logPe, + +C(Te (o))
(9)
The overall objective then formulated as:
Loverall = Lmoment + Aoot + SprozyCrrozzy
(10)
In this formula, Abot and Aprozy are additional balancing parameters, and Lhst CR-$ L This comprehensive approach ensures a balanced and effective mechanism for moment localization within the context of the GVMR task.
Inference. During the inference phase, the computation of both Boundary-aware Contrastive Learning (BCL) and Proxy predictions will not be conducted.
EXPERIMENT
5.1 EVALUATION METRICS
We mainly evaluate the performance with the following metrics:
R1@ IoU and mAp@X. We use RI @ IoU and mAP@X primary metrics. For multi-moment retrieval, mAP is averaged over IoU thresholds [0.25, 0.5, 0.75] (following (Caba Heilbron et al. 2015)). For single moment retrieval, Recall @ IoU (R1 @ μ)) counts prediction as positive if its loU 18 at least y (with set to 0.3, 0.5, or 0.7).
N-Accuracy and T-Accuracy. N-Accuracy (N-acc.) measures no-target identification, where no-target sample with no prediction is a true positive (TP) and any prediction is false negative (FN), computed N-acc. N- T-Accuracy (T-acc.) assesses target classification T-ace. TN
TP
TN+FP. with TN and FP denoting true negatives and false positives, respectively.
5.2 IMPLEMENTATION DETAILS
We utilize the pre-trained SlowFast (Feichtenhofer et al. 2019) and CLIP (Rafford et al., 2021 video encoder model extract video features with sampling rate 0.2, and extract token-level

---

Published as a conference paper at ICLR 2025
text features by pre-trained CLIP (Radford et al. 202 text encoder. We uniformly sampled 128 clips from each video as feature imput. During training. AdamW Myy Loshchilov, (2099) optimizer with weight decay lc-4 is adopted; the batch size is set at 32 for training and 128 for testing; the hidden dimension C = 256. We configured our transformer encoder and decoder with layers each, denoted as T 2. The hyperparameter settings were determined as follows:
L = 10, Ae 4, Åt, = 10, Aiou = 1, Abot Aprozy 0.1, for optimal performance. For the no-target threshold set as 6 0.7 which is experimentally balance for target and no-target generalization.
5.3 COMPARISON WITH STATE- OF-TEE-AFTETAT METHODS
Table 2 presents the performance SOTA methods on the QV-highlight benchmark, re-implemented with unified feature backbone and default hyperparameters. For Moment-DETT (Lei et al. (2021).
QD-DETR (Moon al. 2.23), and EaTR (Jang et al. 2023), we excluded the highlight detection loss and applied the same threshold 5 to filter no-target predictions. These models exhibit strong performance, effectively handling target and no-target samples, our model's explicit multi-region detection significantly outperforms these baselines, achieving over 5% improvement in overall performance. Additionally, our model marginally surpasses SOTA methods in no-target scenarios. Furthermore, conducted Video-Larg Language Models (Vid-LLMs) zero-shot evaluation (Jim et al. 2024) Maaz et al.) [2024] [in et al.] (2024). among which only Jin et al. (2024) supports long video imputs. While Vid-LLMs show potential in single-segment and no-target detection, they struggle with multi-segment predictions, highlighting a critical area for improvement. Detailed results and analysis are included Appendix A.3
5.4 ABLATION STUDY
Effectiveness of proposed BCA module. As illustrated in Table comprehensive ablation ex- periments were conducted explore the impact model performance when either the Boundary- aware Contrastive Learning (BCL) or the Query-region Cross Attention (QCA) module is omitted. Notably, omitting QCA only requires the removal of the proxy loss, as BCL. module built on QCA. The results in the table clearly show that both modules contribute incre- mentally to the model's performance. The most substantial improvement is seen when both modules used to- gether, highlighting their synergistic effect. Further ablation studies on query length and threshold selection can be found in Appendix A.2
Table 3: Performance Comparison of BCANet With and Without the Boundary-aware Contrastive Learning (BCL) and Query-region Cross Attention (QCA) Modules.
Model
RI @0.5
@00777
mAP@0.5
m.P@0.7
w/o Both
42.31
30.41
47.06
33.42
w/o QCA
46.76
36.88
55.03
38.97
w/o BCL
46.20
35.33
54.54
36.42
BCANet
48.61
37.10
56.29
41.22
CONCLUSION
In this paper, we have introduced and developed the concept of Generalized Video Moment Retrieval (GVMR), which aims to handle complex query scenarios such as one-to-many and no-tan po-target queries. By constructing and analyzing comprehensive dataset NEXT-VMR, we provided a solid foundation for training and evaluating our model's performance across a variety real-world query types. Our BCANet model, equipped with advanced techniques like Boundary Aware Cross Attention, demonstrated a significant leap in performance, handling multiple moments and no-target queries with greater accuracy and efficiency. The model's 's adaptability and robustness and validated through extensive evaluations, setting a new benchmark in the field of video moment retrieval.
rigorously tested
10

---

Published as a conference paper at ICLR 2025
7 ACKNOWLEDGEMENT
This work is supported by the Advanced Research and Technology Innovation Centre (ARTIC), the National University of Singapore under Grant (project number: ELDT-RP2).
REFERENCES
Lisa Anne Hendricks, Oliver Wang. Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan Russell. Localizing moments in video with natural language. In ICCV, pp- 5803-5812, 2017.
Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan Russell. Localizing moments in video with temporal language. In arXiv: 2018.
Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles, Activitynet: A
large-scale video benchmark for human activity understanding. In CVPR, pp. 961-970, 2015.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers, 2020.
Yun-Wei Chu, Kuan-Yer Lin, Chao-Chun Hsu, and Lun-Wei Ku. End-to-end recurrent cross-modality attention for vidco dialogue. TASLP, 2021.
Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition, In Proceedings of the IEEE/CVF international conference on computer vision, pp. 6202-6211, 2019.
Jiyang Gao, Chen Sun, Zhenheng Yang, and Ram Nevatia. Tall: Temporal activity localization via language query. ICCV, pp. 5267-5275, 2017.
Junyu Gao and Changsheng Xu. Fast video moment retrieval. In ICCV, 2021.
Frank Hutter Ilya Loshchilov. Decoupled weight decay regularization. In Int. Conf. Learn. Represent., 2019.
Jinhyun Jang, Jungin Park, Jin Kim, Hyeongjun Kwon, and Kwanghoon Sohn. Knowing where focus: Event-aware transformer for video grounding. In ICCV, pp. 13846-13856, 2023.
Wei Ji, Renjie Liang, Lizi Liao, Hao Fei, and Fuli Feng. Partial annotation-based video moment retrieval via iterative learning. In Proceedings of the 31th ACM international conference on Multimedia, 2023a.
Wei Ji, Renjie Liang, Zhedong Zheng, Wenqiao Zhang, Shengyu Zhang, Juncheng Li, Mengze Li, and Tat-seng Chua. Are binary annotations sufficient? video moment retrieval via hierarchical uncertainty-based active learning. Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 220122 2023b.
Wei Ji, Li, Zheqi Lv, Wenqiao Zhang, Mengze Li, Zhen Wan, Wenqiang Lei, and Roger Zimmer- mann. Backpropagation-fii multi-modal on-device model adaptation via cloud-device collabora- tion. ACM Trans. Multimedia Comput. Commun. Appl., 22),,, January 2025. ISSN 1551-6857.
doi: 10. 10.1145/3706422.
Peng Jin. Ryuichi Takanobu, Wancai Zhang, Xiaochun Cao, and Li Yuan. Chat-univi: Unified visual representation empowers large language models with image and video understanding. In Proceedings of the JEEF/CVE Conference on Computer Vision and Pattern Recognition, pp- 13700-13710) 2024.
Jie Lei, Tamara I. Berg, and Mohit Bansal. Qvhighlights: Detecting moments and highlights in videos via natural language queries. In NeurlPS, 2021.
Li Li, Chenwei Wang, You Qin, Wei Ji, and Renjie Liang. Biased-predicet annotation identifica- tion via unbiased visual predicate representation. ACM MM, pp. 4410-4420, 2023a. ISBN 9798400701085 doi: 10.1 0055/3581783.333 1847.
11

---

Published as a conference paper at ICLR 2025
Li Li. Wei Ji, Yiming Wu, Mengze Li, You Qin, Lina Wei, and Roger Zimmermann. Panoptic scene graph generation with semantics prottottess learning. AAAI, (8(44)3145-315 Mar. 2024a. doi: 10.1609/aaii 1a.16609aaai.v33422889999
Pandeng Li, Chen-Wai Xie, Hongtao Xie, Liming Zhao, Lei Zhang, Yun Zheng, Deli Zhao, and Yongdong Zhang. Momentdiff: Generative video moment retrieval from random to real. In Adv. Neural Inform. Process. Syst., 2023b.
Yicong Li, Xun Yang, Xindi Shang, and Tat-Seng Chua. Interventional video relation detection. In Proceedings of the 29th ACM International Conference on Multimedia, pp. 4091 1099,, 2021.
Yicong Li, Xiang Wang, Junbin Xiao, and Tat-Seng Chua. Equivariant and invariant grounding for video question answering. In Proceedings of the 30th ACM International Conference on Multimedia, pp. 4724-4722, 2022a.
Yicong Li, Xiang Wang, Junbin Xiao, Wei Ji. and Tat-Seng Chua. Invariant grounding for video question answering. In CVPR, pp. 292879937, 2022b.
Yicong Li, Xiang Wang, Junbin Xiao, Wei Ji, and Tat-Seng Chua. Transformer-eppoweeed invariant grounding for video question answering. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023c.
Yicong Li, Junbin Xiao, Chun Feng. Xiang Wang, and Tat-Seng Chua. Discovering spatio-temporal rationales for video question answering. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 13869-13878 2023d.
Yicong Li, Na Zhao, Junbin Xiao, Chun Feng, Xiang Wang, and Tat-seng Chua. Laso: Language- guided affordance segmentation on 3d object. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 14251-14260, 2024b.
Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. Video-LLaVA: Learning united visual representation by alignment before projection. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. Computational Linguistics. URL
Miami, Florida, USA, November 2024. Association for wtgg//22220 emnlp caannppmminn main
342
Chang Liu, Henghui Ding, and Xudong Jiang, Gres: Generalized referring expression segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 23592-23601, 2023a.
Zhiyuan Liu, Sihang Li, Yanchen Luo, Hao Fei, Yixin Cao, Kenji Kawaguchi, Xiang Wang, and Tat- Seng Chua. Molca: Molecular graph-language modeling with cross-modal projector and uni-modal adapter. In EMNLP, 2023b. URL BRL
Zhiyuan Liu, An Zhang, Hao Fei, Enzhi Zhang, Xiang Wang, Kenji Kawaguchi, and Tat-Seng Chua. Prott3: Protein-to-text generation for text-based protein understanding. In ACL. Associa- tion for Computational Linguistics, 2024. URL ZmI jOPi12b
net forum?id
Zhiyuan Liu, Yanchen Luo, Han Huang, Enzhi Zhang, Sihang Li, Junfeng Fang, Yaorui Shi, Xiang Wang, Kenji Kawaguchi, and Tat-Seng Chua. NEXT-MOL: 3d diffusion meets Id language modeling for 3d molecule generation. In The Thirteenth International Conference on Learning Representations, 2025. htRhstt:: /ttp:s:///ppeeee///
Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. In Proceedings the 62nd Annual Meeting of the Association for Computational Linguistics (ACL 2024), 2024.
WonJun Moon, Sangeek Hyun, SangUk Park, Dongchan Park, and Jae-Pil Heo. Query-dependent video representation for moment retrieval and highlight detection. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 23023-23033 2023.
12

---

Published as a conference paper at ICLR 2025
Hoang-Anh Pham, Thao Minh Le, Vuong Le, Tu Minh Phuong, and Truyen Tran. Video dialog as conversation about objects living in space-time. In ECCV, pp. 710-726. Springer, 2022.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 9748-8763. PMLR, 2021.
Michaela Regneri, Mareus Rohrbach, Dominikus Wetzel, Stefan Thater, Bernt Schiele, and Manfred Pinkal. Grounding action descriptions in videos. ACL, 1:25-36, 2013a.
Michaela Regneri, Mareus Rohrbach, Dominikus Wetzel, Stefan Thater, Bernt Schiele, and Manfred Pinkal. Grounding action descriptions in videos. ACL, 1:25-36, 2013b.
Xindi Shang, Tongwei Ren, Jingfan Guo, Hanwang Zhang, and Tat-Seng Chua. Video visual relation detection. In ACM Multimedia, pp. 1300-1308, 2017.
Xindi Shang. Yicong Li, Junbin Xiao, Wei Ji, and Tat-Seng Chua. Video visual relation detection via iterative inference. In Proceedings of the 29th ACM international conference on Multimedia, pp. 3654-3663, 2021.
Gunnar A. Sigurdsson, Güil Varol, Xiaolong Wang, Ali Farhadi, Ivan Laptev, and Abhinav Gupta. Hollywood in homes: Crowdsourcing data collection for activity understanding. In ECCV, 2016.
Mattia Soldan, Alejandro Pardo, Juan León Alcázar, Fabian Caba Heilbron, Chen Zhao, Silvio Giancola, and Bernard Ghanem. Mad: A scalable dataset for language grounding in videos from movie audio descriptions. In CVPR, 2022.
Bart Thomee, David A Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. Y0cc100m: The new data in multimedia research. Communications of the ACM, 59(2):64-73, 2016.
Jingwen Wang, Lin Ma, and Wenhao Jiang. Temporally grounding language queries in videos by contextual boundary-aware prediction, 2019. URL http:p:://a......o///aa///////// 05010
Junbin Xiao, Angela Yao, Zhiyuan Liu, Yicong Li, Wei Ji, and Tat-Seng Chua. Video as conditional graph hierarchy for multi- multi-granular question answering. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 2804-282220 1144-2812, 2022.
Junbin Xiao, Nanxin Huang. Hangyu Qin, Dongyang Li, Yicong Li, Fengbin Zhu, Zhulin Tao, Jianxing Yu, Liang Lin, Tat-Seng Chua, et al. Videoqa in the era of Ilms: An empirical study. International Journal Computer Vision, pp. 1-24, 2025.
Hao Zhang, Aixin Sun, Wei Jing, and Joey Tianyi Zhou. Span-based localizing network for natural language video localization. In ACL, pp. 6543-6554, 2020a.
Songyang Zhang, Houwen Peng, Jianlong Fu, and Jiebo Luo. Learning 2d temporal networks networss for moment localization with natural language. AAAI, volume 34, pp. 12870-12877, 2020b.
Henghao Zhao, Kevin Qinghong Lin, Rui Yan, and Zechao Li. Diffusionvmr: Diffusion model for video moment retrieval, 2023.
13

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 STATISTICAL ANALYSIS
We conducted detailed statistical analysis of the dataset, including the types of query statements, the distribution of moments, and the proportion of query statements with different values.
Query Distribution. In our expanded YFCC100M dataset, queries are designed cover range from no-target to multti-tareet scenarios.
These queries are categorized into four types: no target (n 0), single relation 1), double relation (n = 2), and triple relation (n = 3). Figure 4 shows the distribution of these queries in the train, validation and test sets. This distribution reflects the diversity and balance of the dataset. Notably, one-to-many queries (n = 2 and
3) occupy significant proportion in the training set, providing rich data for model training to understand and handle more complex query situations.
Analysis of No Target Queries. For no-target queries (n 0), we further categorize them into grammatically correct (right) and incorrect (wrong) queries to reflect the diversity of queries in the real world. Figure 4 shows the distribution of these two types of queries in the train, validation and sets. This categorization particularly important for training and evaluating video moment retrieval models. The model needs to accurately process and differentiate these two types of no-target queries, ensuring a good response to various query types that may be encountered real-world applications.
Timestamp Counts of Queries. As depicted in Figure (a), the data showcases pronounced long tail distribution in the counts of queries as they relate to the number of ground-truth times- tamps. There is clear trend where queries associated with smaller number of timestamps dominate the dataset. This trend sharply de- clines as the number of timestamps increases, indicating that instances with higher count of ground-truth moments are significantly rarer. logarithmic scale highlights
2900
Xx Turpt Type
Figure 4: Distribution of (a) query numbers rele- vant to n clips and (b) Query Type.
The graph the steep drop-off in frequency as the num- ber of timestamps grows, reinforcing the rarity of queries with extensive temporal references. This distribution pattern underlines the challenge in handling queries with multiple timestamps, they not only less common but may also represent more complex retrieval scenarios.
Query Length and Normalized Moment Distribution. We analyze different percent- age of each query length and compare them with Charades-STA, ActicityNet-Captions and
Nomber of Quertes (Log Scale)
Poreentag
TACoS- 2DTAN. As shown in Figure [S] (b),
the GVMR dataset predominantly consists of
queries with shorter length, with vast ma- jority falling within the 0-10 words range. This
is in contrast to the other datasets, which dis- play a more varied distribution across different query lengths. Charades-STI and ActivityNet- Captions have a more substantial representation in the 10-20 words range, while TACoS-2DTAN features longer queries, with no presence in the shortest category. The concentration of shorter queries in GVMR suggests that the dataset is designed for concise and focused retrieval tasks, which may present unique challenge for video moment retrieval systems that are tuned to datasets with more extended deseriptions. The suceinet na- ture of GVMR's 's queries may require algorithms to be highly efficient at understanding and matching brief textual prompts to video content.
Number Greund-Truth Timestapp
Query Length Words)
(9)
(h)
Figure 5: (a) Statistics Distribution for Query Num- ber and Each Corresponding Segments Ground Truth Timestamps tion for Query Length with comparison with other datasets.
Log Scale. (b) Distribu-
14

---

Published as a conference paper at ICLR 2025
And also count non-empty normalized ground truth times- tamps for each query and visu- alize the start and end times- tamps of each moment. As shown in Figure 6, train and test sets all exhibit similar dis- tributions and share a common characteristic: these moments spread widely across the up- per triangle, even though many moments are concentrated in the upper-left region, i.e., a long tail distribution exists among them.
Train
Val
Text Set
Start
STarT
Start
Figure 6: Moment annotation distributions of NEXT-VMR dataset, where "Start" and "End" axes represent the normalized start and end time points, respectively.
A.2 ABLATION STUDY
Length of Queries:
Substantial ablation studies were conducted to determine the optimal length for queries, varying from 5 to 15.
was observed that an increase in the number of queries enhances the model's perceptual sensitivity, albeit
Table 4: Comparative Evaluation of BCANet Performance with Varied Initial Query Configurations
at the cost of higher computational demands. As depicted in Table 4 the model's performance initially im- proves with an increase in the number of queries, but subsequently dimin- ishes beyond certain point. Based on these findings, we have chosen an optimal configuration for our query length setting, denoted as T, 10. This length strikes a balance between detailed perception and computational efficiency, thereby maximizing the overall efficacy of the model.
Query num. (L)
R1 @0055
R1 @00.77
mAP@0.5
mAP@0.7
35.27
53.38
39,27
48.61
37.10
56,29
41.22
48.83
37.01
55.34
40,70
15
47.97
36.53
55.64
40.13
Strategy for No-target Threshold: In our study, presented in Table ablation experiments focusing on selection of the no-target threshold were conducted.
Interestingly, we observed that the N- accuracy (Non-target Accuracy) in- creases with the rise in threshold value, while the T-accuracy (Target Accuracy) exhibits inverse relation- ship. Furthermore, there's a gradual decrease in the precision of target pre- dictions. evident that when no- target filtering is not applied any of the samples, the prediction accuracy reaches peak. This attributed to the fact that the unfiltered proposal candidates cover a broader range of potential predictions. A higher thresh- old leads the model to preferentially predict no-target outcomes; conversely, many target samples incorrectly generalized as no- no-target predictions. To balance the performance between these two aspects, we set the no-tan target detection threshold at 0.7, which forms the cornerstone our strategy for identifying no-target instances.
Table 5: Performance comparison of BCANet in AP and NorTet Detection with different thresholds.
Threshold (6)
m.P@0.7
N-ace.
T-acc.
0.5
42.30
58.55
98.94
0.6
41.35
62.30
98.54
0.7
41.22
77.92
95.79
0.8
38.55
80.20
87.60
0.9
35.44
83.28
75.29
15

---

Published as a conference paper at ICLR 2025
A.3 INDEEPTH-ANALYSS
Experimental Results on Traditional VMR Datasets. evaluate the performance of the proposed
BCANet, we conducted experiments on traditional video moment retrieval (VMR) datasets, including
Charades (Gao al. 2017) and TACoS (Regneri et al., 2013b). The results are presented in Table (6]
alongside comparisons with state-of-the-arr (SOTA) methods; VSLNet (Zhang et al. 2020a), 2D-
TAN (Zhang et al. 2020b), Moment-DETR (Lei al. 2021), QD-DETR (Moon al. 2023 and
EaTR (Jng al. 2023).
Table 6; Performance comparison of BCANet and other state-of-the-art (SOTA) methods on the Charades and TACoS datasets.
Charades
TACoS
Models
Pred. Type
RI@0.33
@00.5
RI @.0.
mIoU
RI
RI 20055
RI@0.7
mloU
2D-TAN
Single Single
58.76
46.02
27.40
41.25
40.01
27.99
12.92
27.22
VSLNet
64.30
47.31
30.19
45.15
29.61
24.27
20.03
24.11
Moment-DETR QD-DETR
Multi Multi
65.83
52.07
30.59
45.54
37.97
24.67
11.97
25.49
57.31
32.55
40.32
25.13
12.47
27.86
EaTR
Multi
72.50
58.55
37.07
52.18
33,42
21.74
10.62
22.63
BCANet (Ours)
Multi
72.01
59.47
37.46
51.97
42.81
27.23
13.11
30.01
On the Charades dataset, BCANet achieves the highest scores across all metrics, demonstrating significant improvements in RI @0.5 and RI @0.7 compared EaTR and other SOTA methods. Similarly, on the TACoS dataset, BCANet surpasses its competitors, particularly excelling in R @0.3 and mloU. This showeases its robustness in precise moment localization and overall retrieval quality.
These results underscore the effectiveness of our boundary-aware cross-attention mechanism and its ability generalize across datasets with diverse challenges, solidifying BCANet as a robust solution for video moment retrieval tasks.
Efficient Analysis. To further analyze the efficiency and impact of BCANet, following Gao & Xu (2021), we conducted a component wise evaluation on the NEXT-VMR dataset. Table provides detailed breakdown of the performance across key components: text embedding (TE), cross-modal learning (CML.), their combined inference time (ALL), and the overall retrieval accuracy (ACC). For TE, we used BERT features for 2D-TAN and VSLNet, and CLIP word-level features for all other models. The reported values represent the average inference time (in milliseconds) for single query-video pair. All speed tests were conducted on a single NVIDIA RTX A40 GPU.
As shown in Table 7 BCANet achieves competitive performance across all components while maintain- ing high efficiency. It outperforms EaTR in overall accuracy (ACC) due to its effective cross-modal representa- tion learning. Additionally, BCANet strikes balance between the contri- butions of TE and CML, resulting in strong combined inference time (ALL). These results demonstrate the model's capability to cohesively inte- grate text and video features, paving the way for more efficient and accu- rate video moment retrieval frame- works.
Table 7: Component-wise performance comparison of BCANet and other SOTA methods on the NExT-VMR dataset. Metries include TE (Text Embedding), CML (Cross- modal Learning), ALL (TE CML), and ACC (Accuracy).
NExT VMR
Models
TE (ms)
CML (ms)
ALL (ms)
ACC (%)
2D-TAN
4.01
8.67
12.68
44.74
VSLNet
4.01
0.32
4.33
48.31
Moment-DETR
3.60
1.44
5.04
47.03
QD-DETR
3.60
1.68
5.28
53.87
EaTR
3.60
2.10
5.70
55.17
BCANet (Ours)
3.60
2.03
5.63
56.29
models, after thorough investigation, selected Chat-UniVi (Jin et al., 2024) bascline model. This decision was based on its unique ability to encode muluple frames simultaneously. making it particularly suitable for our dataset, where videos often contain hundreds even thousands of frames.
Vid-LLMs Prompting. To evaluate the capabilities large
16

---

2, top p=None, and Published as a conference paper at ICLR 2025
In contrast, other models like VideoLLaVA (Lin et al. 2024) and Video-ChatGPP ChatGPT (Maaz al. 2024 are limited to encoding up to 8 frames, which is insufficient for long video understanding. Chat- UniVi effectively addresses this limitation. For our experiments, uniformly sampled 100 frames from each video for processing and understanding and set num_beams=1 for stable inference.
Stage 1: To check whether the query exists in the video.
This is a Iduration: .2f) second video clip.
Step 1: Does the query; 'qquery'' cerrespond to any segment in the video? Respond with 'yes' or 'no' on the first line. Step 2: If your response in Step is 'yes', provide only one time interval formatted on the second line, where 'a' and 'b' are numbers such that <= < (huration:.2f),
"[a, bl'.
If your response in Step 1 is 'ne', leave the second line blank. Do not include any additional explanation text beyond the specifled format.
Stage 2: If the query exists, extract the specific time interval and calculate IoU.
This is (duration: 2ff second video clip.
Provide a single time interval within the range [0, (duration: formatted as ' [a, b]', that correspends to the segment of the video best matching the query: { query).
Respond with only the numeric interval.
To perform the zero-shot evaluation, we used this two-stage prompting approach.
17


Reviews:
[{"review_id": "MPehAL5waJ", "reviewer": "Reviewer_feEk", "summary": "This paper first introduce a novel GVMR framework that significantly expands the scope of traditional VMR, catering to a wider range of query types and enhancing the model’s practical nutility and  present a meticulously curated dataset. To deal with GVMR, they propose the BCANet model. Totally speaking, the contributions about extend task and new dataset is valuable.", "questions": "1. The model relies on the LLaVa to generate scene captions to divide the backgrounds and foregrounds. Will it brings noisy due to the diversity of generated samples?\n2. Did you compare your model with the finetuinng model based on multimodal LLM ?\n3. With using the LLM to generate captions , what about the time cost and memory usage?\n4. Will the dataset be open source? If so ,it will be an important contribution.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper introduces a extend task about TVG which covers more situations about video localiztion.\n2. They build a new dataset for their introduced dataset which helps the deeper researches in future.\n3. They propose a model based on LLM generate queries in fine-grain semantic, to benefit the video content understanding to the next stage", "weaknesses": "1. The boundary awareness is a common question in this field and many works like[1] focus on this problem. The author didn't analyse the differences between them to show their novelty.\n\n\n*[1]J. Wang, L. Ma, and W. Jiang, “Temporally grounding language queries in videos by contextual boundary-aware prediction,” in AAAI, 2020.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aMPEIbURhB", "reviewer": "Reviewer_Zk91", "summary": "This paper introduces the Generalized Video Moment Retrieval (GVMR) framework, which extends traditional Video Moment Retrieval (VMR) to handle a wider range of query types. To support this expanded task, it presents the NExT-VMR dataset, derived from the YFCC100M collection, featuring diverse query scenarios to enable more robust model evaluation. Additionally, a novel transformer-based model named BCANet is proposed and achieves outperforms when compared with traditional VMR approaches.", "questions": "Please refer to the weakness part.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. A novel GVMR framework is first introduced that significantly expands the scope of traditional VMR, catering to a wider range of query types and enhancing the model's practical utility.\n\n2. A meticulously curated dataset, NExT-VMR, is presented. The dataset is specifically designed for GVMR task, which includes a variety of query types and scenarios, thereby enabling more robust and versatile model training and evaluation.", "weaknesses": "1. As mentioned in the paper, \"GVMR aspires to deliver a robust and flexible solution for video moment retrieval, significantly enhancing the model's interpretative prowess, precision, and retrieval efficiency.\" Would the enhancement of the model on interpretative prowess, precision, and retrieval efficiency be proved and presented in experiments?\n\n2. More statistics of the proposed dataset are needed to be listed in Table 1, such as the entire/average lengths of videos and queries and the number of one-to-multi and no-target samples in the dataset.\n\n3. Is it possible to demonstrate the evaluation of this method on traditional VMR datasets (e.g., Charades, ActivityNet, and TACoS) for one-to-one (different from the mentioned one-to-multi or no-target) VMR tasks? \n\n4. The BCANet pipeline presented in Figure 1 is confusing and difficult to interpret. The respective variable symbols should be labeled in Figure 1. Moreover, the processes for the BCL and QCA modules, along with query-text cross-attention, are also not clearly depicted in the figure. Moreover, what's the meaning of the \"region\" and \"region features\" in this paper? Dose the \"region\" mean a video clip?\n\n5. The BCANet proposed in this paper does not seem to include specific designs for handling one-to-multi cases. Compared to traditional VMR methods, what advantages does BCANet offer in dealing with such data?\n\n6. Since the retrieval speed is also an important metric in VMR, for example, [a] analyze the speed of various VMR approaches. I recommend the authors conducting speed analysis in this paper. In addition, is GVMR slower than traditional VMR?\n[a] Fast video moment retrieval, ICCV 2021.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9HvlIWs5s1", "reviewer": "Reviewer_4Qqv", "summary": "This paper introduces the Generalized Video Moment Retrieval framework, which expands traditional VMR to handle complex, non-target, and multi-target queries. To support this, the authors present the NExT-VMR dataset and BCANet, a transformer-based model with a Boundary-aware Cross Attention module that improves boundary detection and query understanding. \nThe results show that GVMR, NExT-VMR, and BCANet significantly enhance VMR performance, setting a new benchmark for future research in multimedia information retrieval.", "questions": "- Can't the previous model achieve multi-target query well after training on the standard dataset? Lack of relevant experiments to prove the author's motivation.\n\n- Can the model in this paper use a single query to query multiple related clips in the video? This is a more worthy of attention.\n\n- YFCC100M is an excellent large-scale video dataset, but why did the author only select 9957 videos from YFCC100M? How was it selected?\n- Will the author open source the dataset and code?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "The problem the authors solve is interesting.\n\nThis paper provides a good moment retrieval dataset.\n\nThe paper is clearly written and the method can be understood.", "weaknesses": "- The NExT-VMR dataset is not large enough to be used for large-scale training. In addition, why not divide the test set on NExT-VMR for experiments?\n\n- Figure 3 is confusing. There are many colored blocks around the Query-Text Cross Attention module. I don't understand what it wants to express? Lack of technical details.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Dx7CG8IfID", "reviewer": "Reviewer_LXSe", "summary": "This paper introduces the Generalized Video Moment Retrieval (GVMR) framework, which expands traditional Video Moment Retrieval (VMR) to accommodate both non-target and multi-target queries, moving beyond the usual single-target focus. To support this, the authors present the NExT-VMR dataset, derived from the YFCC100M collection, featuring diverse query scenarios for robust model evaluation. They also propose BCANet, a transformer-based model that incorporates the Boundary-aware Cross Attention (BCA) module to enhance boundary detection and improve understanding of video content in relation to queries. Experiments demonstrate the potential of the NExT-VMR dataset and BCANet to advance VMR capabilities.", "questions": "- BCANet is a supervised learning framework. How does it address the issue of inconsistent target quantities for each query in a batch during the training process? Additionally, if a query corresponds to three moments, are all three segments' timestamp information used in the loss function calculation?\n- For queries labeled as non-target, what information do they provide during training?\n- Why does the paper utilize the YFCC100M dataset as the primary data source instead of existing VMR datasets such as TACoS, Charades-STA, MAD, or ActivityNet-Captions?\n- From the statistics in Table 1, it can be seen that the average number of segments corresponding to a query is less than 2, which contradicts the paper's motivation for creating multiple-target queries. Additionally, the proportion of non-target queries is very low. Based on the dataset statistics, it is unclear where the challenges of this new dataset compared to existing ones lie.\n- From Table 2, it can be seen that BCANet shows a significant advantage only in the single-moment retrieval task, while its performance in multi-target and non-target scenarios is not as pronounced. This raises questions about the overall effectiveness of the method.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This paper introduces a new task called Generalized Video Moment Retrieval (GVMR), which is designed to adeptly handle the complexities of various query types, extending the traditional Video Moment Retrieval (VMR) paradigm to include both multiple-target and no-target queries.\n- This paper presents a specialized dataset, NExT-VMR, which is meticulously constructed and analyzed from the YFCC100M dataset. Tailored specifically for GVMR, this dataset features a diverse range of query types, including one-to-multiple and no-target queries.\n- This paper introduces a model called BCANet, aimed at improving performance on the challenging task of GVMR, where the model must accurately predict a variable number of temporal timestamps based on natural language descriptions.", "weaknesses": "- Why does the paper utilize the YFCC100M dataset as the primary data source instead of existing VMR datasets such as TACoS, Charades-STA, MAD, or ActivityNet-Captions?\n- From the statistics in Table 1, it can be seen that the average number of segments corresponding to a query is less than 2, which contradicts the paper's motivation for creating multiple-target queries. Additionally, the proportion of non-target queries is very low. Based on the dataset statistics, it is unclear where the challenges of this new dataset compared to existing ones lie.\n- From Table 2, it can be seen that BCANet shows a significant advantage only in the single-moment retrieval task, while its performance in multi-target and non-target scenarios is not as pronounced. This raises questions about the overall effectiveness of the method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)