Not All Inputs Are Valid: Towards Open-Set Video Moment Retrieval using Language

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-26 02:02:49

Abstract: As a significant yet challenging multimedia task, Video Moment Retrieval (VMR) targets to retrieve the specific moment corresponding to a sentence query from an untrimmed video. Although recent respectable works have made remarkable progress in this task, they implicitly are rooted in the closed-set assumption that all the given queries as video-relevant1. Given a video-irrelevant OOD query in open-set scenarios, they still utilize it for wrong retrieval, which might lead to irrecoverable losses in high-risk scenarios, e.g., criminal activity detection. To this end, we creatively explore a brand-new VMR setting termed Open-Set Video Moment Retrieval (OS-VMR), where we should not only retrieve the precise moments based on ID query, but also reject OOD queries. In this paper, we make the first attempt to step toward OS-VMR and propose a novel model OpenVMR, which first distinguishes ID and OOD queries based on the normalizing flow technology, and then conducts moment retrieval based on ID queries. Specifically, we first learn the ID distribution by constructing a normalizing flow, and assume the ID query distribution obeys the multi-variate Gaussian distribution. Then, we introduce an uncertainty score to search the ID-OOD separating boundary. After that, we refine the ID-OOD boundary by pulling together ID query features. Besides, video-query matching and frame-query matching are designed for coarse-grained and fine-grained cross-modal interaction, respectively. Finally, a positive-unlabeled learning module is introduced for moment retrieval. Experimental results on three challenging datasets demonstrate the effectiveness of our OpenVMR.

Main paper: 
 introduction: 
 Video Moment Retrieval (VMR) is a challenging yet crucial task in multi-modal retrieval [8, 34, 66, 71, 82] , which has attracted significant attention in recent years due to its vast potential applications Query: He is holding a bowl of noodles and broth.The person gets out a plate.The person gets out a plate.The man cleans again with soap and water the car for the second time. in areas such as human activity retrieval [21, 54, 77, 84] . As illustrated in Fig. 1 (a), its main objective is to identify and retrieve the relevant video moment corresponding to a given sentence query.Obviously, most of the video content is query-irrelevant, and only a very short video segment matches the query. It is substantially more challenging since a well-designed model should not only model the complex multi-modal interaction between videos and queries, but also capture complicated context information for cross-modal semantics alignment. The target model requires recognizing objects/activities and identifying which visual content is sufficient to retrieve the accurate moment expressed in free-form natural language, accounting for the fact that the accurate moment may occupy only a tiny portion of the entire video. To achieve this, both videos and queries must be deeply integrated to distinguish the subtle details of adjacent frames, thereby enabling accurate determination of moment boundaries. Most existing VMR works [5, 15, 35, 72, 73, 76, 79] are under fully-supervised setting, where each frame is manually labeled as the query-relevant or query-irrelevant frame. Instead of using such dense frame annotations, some recent works try to explore a weakly-supervised setting [6, 33, 40, 55, 83] with only the videoquery correspondence to alleviate the reliance on a certain extent. However, their performance is less satisfactory. Although the above VMR methods have made exciting headway, they refer to the closeset assumption that we can obtain a moment in the untrimmed video for any given query. In the real-world open-set environment, we often input a random or irrelevant query for moment retrieval. As shown in Fig. 1(b) , given an irrelevant query, previous methods still retrieve a wrong moment as the model output, which will lead to irrecoverable losses in high-risk scenarios, e.g., criminal activity detection. It is unacceptable for our society to classify normal activity as criminal and to treat criminal activity as normal. In real-world multimedia applications, we have a small-scale set of video-query pairs, and many unannotated videos and many video-irrelevant queries. Since labeling video-query annotation is very expensive and time-consuming, it is unrealistic to manually annotate all the queries as video-relevant (i.e., ID) or video-irrelevant (i.e., OOD).Therefore, we explore a novel and challenging task: open-set VMR (OS-VMR). Given an untrimmed video, our OS-VMR task aims to not only temporally retrieve the specific moment semantically corresponding to the ID query, but also reject the OOD query shown in Fig. 1(b) . Different from previous closed-set settings that only align video and query representations for moment retrieval, our OS-VMR task suffers from three major challenges: 1) how to accurately learn the distribution of ID queries? 2) how to precisely reason the separating boundary of ID and OOD queries? 3) how to fully interact with video and ID queries? In this paper, we propose a novel OpenVMR framework for the challenging OS-VMR task. Specifically, we first design a multi-layer coupling block to construct the normalizing flow for learning ID query distribution based on the multi-variate Gaussian distribution assumption. Besides, we reason the ID-OOD separating boundary by a well-designed uncertainty score and the log-likelihood distribution of each query. Moreover, we pull ID query features together to refine the ID-OOD boundary based on a triplet loss for OOD query detection. After that, for the video and ID query, we conduct the cross-modal interaction for video-query matching and frame-query matching. Finally, we design a simple yet effective positive-unlabeled learning module with pre-defined proposals to retrieve the target moment.Our main contributions are summarized as follows:• To the best of our knowledge, we make the first attempt at the open-set Video Moment Retrieval (OS-VMR) task, which is fundamentally more challenging but highly valuable in open-set settings. In this setting, we should not only retrieve the video moment for ID queries, but also reject OOD queries.  
related work: 
 Video moment retrieval. The goal of Video Moment Retrieval (VMR) is to recognize and temporally localize all the action instances in an untrimmed video [5, 43, 79] . Most of the existing VMR methods [3, 12, 47, 72, 81, 82] refer to the fully-supervised setting where all video-query pairs are annotated in detail, including corresponding moment boundaries. The above methods heavily rely on datasets that require numerous manually labeled annotations for training. To ease the human labeling efforts, several recent works [6, 33, 55, 83] consider a weakly-supervised setting that only accesses the information of matched video-query pairs without accurate moment boundaries. However, their performance is significantly worse than fully-supervised methods.Open-set recognition. Open-set Recognition (OSR) aims to classify in-distribution (ID) samples and reject out-of-distribution (OOD) samples. Previous OSR methods [18, 28, 32, 36, 39, 41, 56, 60, 69, 70, 74] can be divided into four types: classification-based methods [17, 27, 29, 32] , density-based methods [23, 52] , distance-based methods [28, 57] and reconstruction-based methods [67, 86] . [9, 19, 20, 38, 45, 49, 61] in the image datasets. These OSR methods cannot effectively understand video and query in our VMR task. However, it is the uniqueness of the moment retrieval in an open-set setting that makes the OS-VMR problem even more challenging and valuable in practice. 
our proposed openvmr: 
 Setup. Given an untrimmed video 𝑉 and corresponding language query 𝑄, the traditional Video Moment Retrieval (VMR) task aims to retrieve the query-described activity moment from the video. However, existing VMR methods refer to a closed-set setting, which lacks their understanding of video-irrelevant queries, limiting their real-world information retrieval applications.To this end, we investigate a more practical but challenging setting, called open-set VMR (OS-VMR), which not only conducts video grounding by the video-relevant query but also rejects the video-irrelevant query. Given an untrimmed video and a sentence query, the OS-VMR task aims to retrieve the moment location from the video (if the query is video-relevant, i.e., ID query), or reject the query (if the query is video-irrelevant, i.e., OOD query). Therefore, our posed OS-VMR is brand-new and more challenging than VMR. Fig. 2 illustrates the framework of our proposed method.Video encoder. Following previous VMR works [79, 81, 82] , given a video with 𝑁 𝑣 frames, we first extract its frame-wise features by a pre-trained C3D network [58] , and then employ a multi-head self-attention [59] module to capture the long-range dependencies among video frames. We denote the extracted video features as𝑉 = {𝑣 𝑖 } 𝑁 𝑣 𝑖=1 ∈ R 𝑁 𝑣 ×𝑑, where 𝑑 is feature dimension. Query encoder. Similarly, given a query with 𝑁 𝑤 words, we also follow previous VMR works [79, 81, 82] to utilize the Glove [48] embedding to encode each word into dense vector. We further employ the Bi-GRU [7] layers to extract word-level query feature In module (III), we further refine the ID-OOD boundary by maximizing the margin between ID and OOD boundaries. Finally, we utilize the ID query for final moment retrieval by: a) we align video and ID query by video-query matching and frame-query matching, b) treating query-relevant frames as positive data and query-irrelevant frames as unlabeled data, we introduce positive-unlabeled learning to predict the coarse-grained moment boundary, and c) a retrieval head is used for fine-grained moment retrieval. Best viewed in color.𝑊 = {𝑤 𝑗 } 𝑁 𝑤 𝑗=1 ∈ R 𝑁 𝑤 ×𝑑 .whole query, we utilize the Skip-thought parser [24] to extract the sentence-level query feature 𝑞 ∈ R 𝑑 .When it comes to detecting OOD queries, we aim to find an OODindependent separating boundary to distinguish ID queries and OOD queries. Thus, we extract the simplified distribution of ID query features. We use normalizing flow [10, 11, 26, 46, 68] to learn the distribution of ID query features. In a VMR dataset, for a video, we select 𝑁 𝑖𝑑 ID queries and 𝑁 𝑜𝑜𝑑 OOD queries. We refer to the sentence-level query features extracted by the query encoder as input features for normalizing flow. We denote these features asQ = Q 𝑖𝑑 ∪ Q 𝑜𝑜𝑑 , where Q 𝑖𝑑 = {𝑞 𝑖𝑑 𝑖 } 𝑁 𝑖𝑑 𝑖=1 and Q 𝑜𝑜𝑑 = {𝑞 𝑜𝑜𝑑 𝑗 } 𝑁 𝑜𝑜𝑑 𝑗=1are ID and OOD query features, respectively.Φ 𝜔 : Q ∈ R 𝑑 → X ∈ R 𝑑 as our normalizing flow,where 𝜔 is a learnable parameter and X denotes the latent space. Especially, a coupling block with multiple coupling layers [10] is leveraged such thatΦ 𝜔 = Φ 𝐶 • • • • • Φ 2 • Φ 1 ,where 𝐶 is the total layer number. Defining 𝑑-dimensional input and output features of normalizing flow as 𝑘 0 = 𝑞 ∈ Q and 𝑘 𝐶 = 𝑥 ∈ X, the output of the 𝑐-th latent layer is 𝑘 𝑐 = Φ 𝑐 (𝑘 𝑐 -1 ), where {𝑘 𝑐 } 𝐶 -1 𝑐=1 are the intermediate outputs. By the change of variables formula, the input distribution estimated by model 𝑝 𝜔 (𝑞) can be formulated as:EQUATIONwhere 𝐽 Φ 𝑐 (𝑘 𝑐 -1 ) = 𝜕Φ 𝑐 (𝑘 𝑐 -1 )/𝜕𝑘 𝑐 -1 is the Jacobian matrix of Φ 𝑐 at 𝑘 𝑐 -1 , and det means determinant. Besides, we can approximate the feature distribution 𝑝 Q with 𝑝 𝜔 (𝑞) by the normalizing flow. By optimizing the log-likelihoods across the training distribution 𝑝 Q , we can obtain the set of parameters 𝜔 as follows:EQUATION3.2.2 Learning ID Query Feature Distribution. Then, by maximum likelihood optimization, we leverage the normalizing flow to learn ID query feature distribution. In general, the latent variable distribution 𝑝 X (𝑥), 𝑥 ∈ R 𝑑 can be assumed to obey the following multi-variate Gaussian distribution [16] :EQUATIONwhere 𝜇 and 𝜎 are the mean and the covariance, respectively. For simplicity, we assume the latent variables for the ID query feature to obey the standard normal distribution during training. By replacing 𝑝 X (𝑥) = (2𝜋) -𝑑 2 exp(-1 2 𝑥 𝑇 𝑥) in Eq. ( 1), we rewrite the optimization objective in Eq. ( 2) as follows:EQUATION)Since 𝑑/2log(2𝜋) is constant, we remove it in Eq. ( 4). To learn the ID query feature distribution, we define the maximum likelihood loss function as follows: EQUATIONBased on the learned ID query feature distribution, we can search an explicit and compact separating boundary between ID and OOD query. To reduce the computational cost from high-dimensional query features, we consider searching the boundary based on the uncertainty score. Since the log-likelihoods generated by the normalizing flow can be equivalently converted to uncertainty scores, we select the boundary on the log-likelihood distribution.3.3.1 Uncertainty Score. By the normalizing flow, we can estimate the exact log-likelihood log𝑝 (𝑞) for each query feature 𝑞:EQUATIONWith the estimated log-likelihood log𝑝 (𝑞), we utilize the exponential function to convert it to likelihood. Since we aim to maximize log-likelihoods for normal features in Eq. ( 5), the likelihood can directly measure the uncertainty. Thus, we can obtain the following uncertainty score:𝑢 (𝑞) = max 𝑞 ′ ∈ Q (exp(log𝑝 (𝑞 ′ ))) -exp(log𝑝 (𝑞)),where 𝑢 (𝑞) is the uncertainty score of query 𝑞. The log-likelihood can be equivalently converted to the uncertainty score since the exponential function is monotonic. Therefore, the boundary in uncertainty score distribution is equivalent to the separating boundary in log-likelihood distribution.Reasoning ID-OOD Separating Boundary. With log-likelihood distribution, we can search the separating boundary as follows:1) Firstly, we search the ID log-likelihood distribution by ID loglikelihoods P 𝑖𝑑 = {log𝑝 𝑖 } 𝑁 𝑖𝑑 𝑖=1 based on the log-likelihood estimation formulation in Eq. ( 6). 2) Then, we can approximate the loglikelihood distribution of all ID queries by P 𝑖𝑑 . 3) After that, we introduce a position hyper-parameter 𝛼 to determine the boundary. Specially, we set the 𝛼-th percentile (e.g., 𝛼 = 5) of sorted ID loglikelihood distribution as the ID boundary 𝑏 𝑖𝑑 , which also serves as the upper bound of the ID false positive rate is 𝛼%. 4) To enhance the robustness of our model, we employ the margin hyper-parameter Δ and define an OOD boundary as 𝑏 𝑜𝑜𝑑 = 𝑏 𝑖𝑑 -Δ.With the explicit ID-OOD boundary, we design an ID-OOD boundary refinement module for discriminative feature learning. Then, we use the boundary 𝑏 𝑖𝑑 as the contrastive target. Specially, we push OOD query features whose log-likelihoods are larger than 𝑏 𝑜𝑜𝑑 apart from 𝑏 𝑖𝑑 at least beyond the margin Δ, while pulling together ID query features whose log-likelihoods are smaller than 𝑏 𝑖𝑑 . Thus, we introduce the following triplet loss:EQUATION)Since any log-likelihood log𝑝 𝑖 fallen into the margin region (𝑏 𝑜𝑜𝑑 , 𝑏 𝑖𝑑 ) will increase the value of L 2 , we can encourage the sparse loglikelihood distribution in the margin region (𝑏 𝑜𝑜𝑑 , 𝑏 𝑖𝑑 ). The loglikelihoods can range from a large region (-∞, 0], which makes it difficult to select the satisfactory hyper-parameter Δ. To normalize the log-likelihoods to the small range (e.g., [-1, 0]), we introduce a large enough normalizer ℎ 𝑖𝑑 (e.g., ℎ 𝑖𝑑 = 100). Since these loglikelihoods can be easily divided into OOD queries, the extremely small log-likelihoods (less than -1) can be excluded outside the loss in Eq. ( 7). Thus, minimizing the loss in Eq. ( 7) will encourage all log-likelihoods P to distribute in the regions[-1, 𝑏 𝑜𝑜𝑑 ] or [𝑏 𝑖𝑑 , 0].Given a query, if its log-likelihood falls into [𝑏 𝑖𝑑 , 0], we treat it as ID. Otherwise, it is OOD.3.5.1 Video-query Matching. The frame-level video and word-level query representations are denoted as 𝑉 ∈ R 𝑁 𝑣 ×𝑑 and 𝑊 ∈ R 𝑁 𝑤 ×𝑑 , respectively. Specially, we first extract the global representation by an attentive pooling:𝑣 ′ = 𝑁 𝑣 𝑖=1 𝑓 𝑣 𝑖 𝑣 ′ 𝑖 , 𝑓 𝑣 = 𝜉 𝑠𝑜 𝑓 𝑡𝑚𝑎𝑥 (𝑉 𝑀 𝑣 ), 𝑞 ′ = 𝑁 𝑤 𝑗=1 𝑓 𝑞 𝑗 𝑞 ′ 𝑖 , 𝑓 𝑞 = 𝜉 𝑠𝑜 𝑓 𝑡𝑚𝑎𝑥 (𝑊 𝑀 𝑞 ), where 𝑀 𝑣 ∈ R 𝑑 ×1 and 𝑀 𝑞 ∈ R 𝑑 ×1 are learnable matrices; 𝑣 ′ and 𝑞 ′ are global video and query representations, respectively. To evaluate the video-query matching score, we introduce the following cosine similarity:EQUATIONwhere ∥ • ∥ 2 denotes the L2-norm of a vector. In a training batch with 𝑁 𝑏 video-query pairs {𝑉 𝑖 , 𝑄 𝑖 } 𝑁 𝑏 𝑖=1 , we adopt the multi-modal features {𝑣 ′ 𝑖 , 𝑞 ′ 𝑖 } 𝑁 𝑏 𝑖=1 for cross-modal semantic alignment. For positive and negative video-query pairs, we utilize the video-query matching score 𝑆𝑖𝑚(𝑣 ′ , 𝑞 ′ ) for the following alignment loss:EQUATIONwhere 𝜂 is a temperature parameter.To further better understand the given video 𝑉 and ID query 𝑄 𝑖𝑑 , we design a Frame-Query Matching (FQM) module to estimate the matching videos and queries based on the likelihood of each frame about queries. Specially, the FQM module includes two linear layers followed by two functions (sigmoid and tanh), which is defined as: FQM(𝑓 𝑣 𝑖 , 𝑞 ′ ) = 𝜉 𝑠𝑖𝑔𝑚𝑜𝑖𝑑 (𝜉 𝑡𝑎𝑛ℎ (𝑓 𝑣 𝑖 𝑀 1 )𝑞 ′ 𝑀 2 ), where 𝜉 𝑠𝑖𝑔𝑚𝑜𝑖𝑑 (•) and 𝜉 𝑡𝑎𝑛ℎ (•) denote sigmoid and tanh functions, respectively; 𝑓 𝑣 𝑖 denotes the 𝑖-th frame feature in video 𝑉 ; 𝑀 1 ∈ R 𝑑 ×1 and 𝑀 2 ∈ R 𝑑 ×1 are learnable matrices. We denote the likelihood of each frame about 𝑞 ′ as 𝑃 (𝑓 𝑣 𝑖 | 𝑞 ′ ). Also, we can obtain the probability: 𝑃 (𝑓 𝑣 𝑖 | 𝑞 ′ ) = FQM(𝑓 𝑣 𝑖 , 𝑞 ′ ). Then, we calculate the frame-aware attention scores 𝑎 by the softmax function as: 𝑎 = 𝜉 𝑠𝑜 𝑓 𝑡𝑚𝑎𝑥 (𝑝 is the fused feature, and 𝑀 3 , 𝑀 4 and 𝑀 5 are learnable weight matrices.Learning. Since most of the video content is background, it is significant to predict the foreground frames that are relevant to the ID query. We can treat query-relevant frames as positive data, whereas query-irrelevant frames are unlabeled data. We can transfer the VMR problem to a semi-supervised learning problem, positive-unlabeled learning (PUL) [1] . Thus, we design a simple yet effective PUL module to predict the target moment.Specifically, we follow previous VMR works [82] to retrieve the target moment with pre-defined moment proposals based on 𝑓, where we sample 𝑡 proposals for each frame. Then, we utilize Eq. EQUATIONBy Eq. ( 10), we can push the probably pure background proposals far away from positive proposals. Although the PUL method is simple, the learned similarity scores are enough for distinguishing the foreground and background proposals.As the boundaries of pre-defined proposals are coarse, we employ a retrieval regression loss for calibrating the retrieval. We calculate the regression loss for every positive sample:EQUATIONwhere L 𝑠𝑚𝑜𝑜𝑡ℎ is the smooth 𝐿 1 loss; 𝑡 𝑠 , 𝑡 𝑒 are the ground-truth start and end timestamps; 𝑡 ′ 𝑠 , 𝑡 ′ 𝑒 are the predicted timestamps. For convenience, we set L 4 = L BCE + L reg . Therefore, we can obtain the following overall loss:EQUATIONwhere 𝜆 1 , 𝜆 2 and 𝜆 3 are the balanced weight hyper-parameters.Inference. Given a video and a sentence query, we first feed them into our model to detect if the query is ID or OOD. If the query is ID, we can obtain the fused cross-modal feature 𝑓. Then, we predict the moment boundary (𝑡 ′ 𝑠 , 𝑡 ′ 𝑒 ) in Eq. (11) . "Top-n (R@n)" moment candidates will be selected with non-maximum suppression [44] . 
experiments 4.1 datasets and evaluation metrics: 
 Datasets. We experiment on four VMR datasets with various characteristics: Charades-STA [12] , Activitynet-Captions [25] and TACoS [50] . In this paper, we conduct experiments under two settings: closed-set and open-set, whose split principle is illustrated in Table 1 . In the closed-set setting, for each dataset, we utilize its queries and its videos for both training and testing. In the open-set setting, we train each dataset based on its queries and videos, while we conduct testing on its videos and queries from all three datasets. ActivityNet Captions. ActivityNet Captions is introduced by [25] , which contains about 20k untrimmed videos and 100k descriptions with diverse open-domain activities. Following the split principle in [81] , in the closed setting, we utilize 37,417 video-query pairs for training, and 34,536 pairs for testing, respectively. Charades-STA. It is built on Charades [53] by [12] , including 9,848 videos of indoor scenarios. In the closed setting, we use 12,408 and 3,720 video-sentence pairs for training and testing, respectively. TACoS. TACoS is a challenging dataset collected from the MPII Cooking Composite Activities [51] , which contains 127 long videos of cooking scenarios. In the closed setting, we follow the standard split used in [12] and obtain 10,146, and 8,672 video-sentence pairs as training and testing dataset, respectively. Evaluation metrics. There are two challenges in our method: ID query for moment retrieval and OOD query detection. Thus, we use two types of metrics for comprehensive performance evaluation:1) ID query for moment retrieval. Following [12, 79] , we adopt "R@n, IoU=m" as the evaluation metric, which denotes the percentage of language queries having at least one result whose Intersection over Union (IoU) with ground truth is larger than m in top-n retrieved moment. In our experiments, we use 𝑛 ∈ {1, 5} for all datasets, 𝑚 ∈ {0.5, 0.7} for Charades-STA, 𝑚 ∈ {0.3, 0.5} for ActivityNet Captions and TACoS.2) OOD query detection. To evaluate open-set performance, we introduce two popular metrics to evaluate the performance of detecting ID and OOD queries for a given video: the Area Under the Receiver Operating Characteristic (AUROC) curve and the Area Under the Precision-Recall (AUPR) [37] .For all the metrics, the larger value denotes better performance. Implementation details. For the video encoder, we apply C3D [58] to encode the videos on ActivityNet Caption and I3D [2] on Charades-STA and TACoS. Since some videos are overlong, we set the length of frame sequences 𝑀 to 64, 64 and 200 for Charades-STA, Activi-tyNet Captions and TACoS, respectively. For the query encoder, we utilize GloVe 840B 300d [48] to embed each word as word features. We sample 800 moment proposals for TACoS and 384 for Charades-STA and ActivityNet Captions similar with [82] . We train our model for 200 epochs with a batch size of 128 and an early stopping strategy. Parameter optimization is performed by Adam [22] optimizer with a learning rate of 0.0008. We conduct our experiments on a single Nvidia TITAN XP GPU. To evaluate efficiency, we only choose the open-source compared methods that are grouped into two categories: (i) Fully-supervised (FS) setting [12-14, 30, 31, 34, 64, 72, 75, 80-82] ; (ii) Weaklysupervised (WS) setting [4, 62, 63, 65, 83, 85] . For convenience, we denote our proposed "open-set setting" as "OS". Following [42, 78] , we directly cite the results of compared methods from corresponding works. Note that no weakly-supervised method reports its results on TACoS. The best results are bold. Based on these experimental results, we list several notable observations as follows: 1) Compared with other state-of-the-art methods, our proposed model achieves the highest performance over all metrics on three datasets, which demonstrates the superiority Particularly, our model beats the best compared fully-supervised method G2L by 5.22% in terms of "R@5, IoU=0.5". As for the best compared weakly-supervised method CNM, our model outperforms it by 14.17% over "R@1, IoU=0.3". It shows that our model has excellent generalization ability in more complex and diverse real-world scenarios by frame-query matching and video-query matching. 3) On the Charades-STA dataset, based on the reported settings of previous methods, our proposed method reaches the new state-ofthe-art on all the metrics. In terms of "R@1, IoU=0.3", our model outperforms the best compared fully-supervised method Moment-Diff by 4.59%, and the best compared weakly-supervised method LCNet by 19.19%. 4) On the TACoS dataset, our model surpasses other state-of-the-art methods by a large margin. Compared with best compared method, our model improves the performance by 10.66% and 9.04% in terms of "R@1, IoU=0.3" and "R@1, IoU=0.5", which may stem from the inherent characteristics of this dataset.Open-set Evaluation. Similarly, we conduct open-set experiments on all three datasets. Table 5 , 6 and 7 illustrate the open-set VMR performance on ActivityNet Captions, Charades-STA, and TACoS, respectively. Obviously, our model achieves the best performance on three datasets over all the metrics. Open-set results on ActivityNet Captions: We report its openset comparison in Table 5 . Particularly, our model beats the best compared fully-supervised method MMN by 15.79% in terms of "R@1, IoU=0.5". As for the best compared weakly-supervised method Open-set results on Charades-STA: In Table 6 , we further evaluate our open-set performance on the Charades-STA dataset. Our method achieves the best results in all the cases. For example, our model outperforms the best compared fully-supervised method DRN by 14.49% and 11.48% in terms of "R@1, IoU=0.3" and "R@1, IoU=0.5", respectively. Besides, our model surpasses the best weaklysupervised LCNet by 21.10% and 12.19% over "R@1, IoU=0.3" and "R@1, IoU=0.5", respectively. This is because the Charades-STA dataset is under the indoor scenarios, which are irrelevant to outdoor queries in other two datasets. Previous methods directly retrieve outdoor moments based on outdoor queries, leading to unreasonable retrieval results for the open-set VMR task.Open-set results on TACoS: As shown in Table 7 , we also evaluate the open-set performance of our model on the TACoS dataset. Particularly, our model outperforms the best compared method MomentDiff by 24.14% and 12.21% over "R@1, IoU=0.3" and "R@1, IoU=0.5", respectively. The main reason for the significant performance improvement is that all the videos on TACoS are cookingrelated, which only corresponds to cooking-related queries. Given some cooking-irrelevant OOD queries from other two datasets, previous methods mistakenly utilize these OOD queries for wrong moment retrieval, which severely limits their performance. In the closed-set setting, our model achieves better retrieval performance than previous state-of-the-art methods (MMN and WSTAN). This is because our model can fully conduct cross-modal interaction by both video-query matching and frame-query matching.For the challenging open-set setting, our model can reason the right ID-OOD boundary for OOD query recognition, thus rejecting video-irrelevant queries.Main ablation studies. To demonstrate the effectiveness of each component in our model, we conduct ablation studies regarding the components (i.e., ID knowledge acquisition module, uncertaintyaware OOD boundary reasoning module and ID-OOD boundary refinement module). In particular, we remove each key individual module to investigate its contribution. For convenience, we design three ablation models: 1) Ours(a). We remove the ID knowledge acquisition module while keeping the other two modules. 2) Ours(b). We remove the uncertainty-aware OOD boundary reasoning module while keeping the other two modules. 3) Ours(c). We remove the ID-OOD boundary refinement module while keeping the other two modules. Besides, we use our full model as the baseline: Ours(full). Since we focus on the open-set setting, we conduct corresponding experiments in Table 5 , 6 and 7. Based on these tables, we can observe that all three modules contribute a lot to the final performances on all three datasets, demonstrating their effectiveness in the open-set VMR task. As the core module for detecting OOD Training process of different ablation models. Following [33] , we analyze the training process and retrieval performance of different ablation models in Fig. 4 . We can obtain the following representative observations: (i) During training, Our(full) outperforms other ablation models, which further demonstrates the effectiveness of each module. (ii) Our(full) converges faster than ablation models, which shows that our full model is more efficient on timeconsuming. For instance, Our(full) converges within 120 epochs on both ActivityNet Captions and Charades-STA, while Our(c) converges after 160 epochs. Thus, our full model can process these challenging datasets more efficiently. Effect of ID knowledge acquisition. In the ID knowledge acquisition (IKA) module, we design a multi-layer coupling block and utilize the multi-variate Gaussian distribution (MGD). We implement different variants of the IKA module in Table 9 . Obviously, we can achieve the best performance when we utilize the five-layer coupling block and MGD to learn ID distribution. It demonstrates that the five-layer coupling block can sufficiently construct the normalizing flow and learn ID query feature distribution based on multi-variate Gaussian distribution assumption. Analysis on uncertainty-aware OOD boundary reasoning.Similarly, we conduct ablation on the uncertainty-aware OOD boundary reasoning (UOBR) module. As shown in Table 9 , we change the value of 𝛼 to search the best results. Thus, we set 𝛼 = 5 in all the experiments. Influence of ID-OOD boundary refinement. Also, we conduct ablation study on the ID-OOD Boundary Refinement module. Table 9 illustrates the ablation results on the Charades-STA dataset. Obviously, when Δ = 0.04, we can obtain the best performance.Analysis on cross-modal interaction. As shown in Table 10 , we further evaluate the importance of three components in the crossmodal interaction module: video-query matching, frame-query matching and positive-unlabeled learning. We find that the videoquery matching component achieves the largest improvement, which is because wrong video-query matching will directly lead to retrieval failure. Moreover, positive-unlabeled learning brings significant improvement since it can recognize the proper moment proposals by the binary cross-entropy loss. In addition, frame-query matching provides fine-grained vision-language alignment for more precise retrieval. Thus, all three components in our cross-modal interaction module are significant. Parameter analysis. As shown in Table 11 , we conduct experiments on the ActivityNet Captions dataset under the open-set setting, and present the ablation study on the hyper-parameters (𝜆 1 , 𝜆 2 , 𝜆 3 , 𝜂). We can find that, their performance only varies in a small range, indicating that our model is insensitive to these parameters. Finally, we choose 𝜆 1 = 0.6, 𝜆 2 = 0.3, 𝜆 3 = 0.7, 𝜂 = 0.2. 
conclusion: 
 In this paper, we pose a brand-new and challenging task: open-set video moment retrieval (OS-VMR). Given a video and a query, the OS-VMR task aims to conduct retrieval if the query is video-relevant, otherwise reject the query. To address it, we propose a novel method for this special task. Experiments on three challenging benchmarks demonstrate the effectiveness of our method.  
 
 Appendix: In this paper, we treat "video-relevant query" as "in-distribution (ID) query" and "video-irrelevant query" as "out-of-distribution (OOD) query". 


Decision: All reviewers have reached an agreement on the acceptance. The paper introduces a novel Open-Set Video Moment Retrieval (OS-VMR) task, addressing the challenge of handling out-of-distribution (OOD) queries in video retrieval. The proposed OpenVMR model effectively distinguishes between ID and OOD queries using normalizing flow technology and demonstrates significant performance improvements over existing methods across multiple datasets. Despite some concerns about methodology and experimental setup, the paper's contributions and experimental validations sufficiently justify its acceptance.

Reviews: review: paper_summary: paper_summary: This paper proposes a new setting of Video Moment Retrieval (VMR) called Open-Set Video Moment Retrieval (OS-VMR), where the model should not only localize the temporal boundaries of language queries that are relevant to the video content, but also reject those irrelevant queries describing non-existing visual activities in the video. A new method named OpenVMR is also proposed for the task of OS-VMR. Experiments are conducted to verify the effectiveness of the proposed OpenVMR method.
strength_weakness: strength_weakness: 1.The basic motivation of this work is reasonable, i.e., the proposed OS-VMR task is a relatively more practical version of the original VMR task where the model should be capable of recognizing and rejecting irrelevant queries.

2.A relatively comprehensive list of experiments are conducted, e.g., performance comparisons under closed-set and open-set scenarios are reported.

3.The authors give some visualization results of the proposed setting and method, which helps the readers to more intuitively grasp the points of the paper.
limitations: limitations: 1.Although the conceptual idea of this paper is basically correct, there are actually considerable flaws in terms of technical soundness and experimental setup. Firstly, this work proposes to learn a distribution from the ID query features and use an OOD boundary to separate the ID queries and OOD queries. However, it is unreasonable to determine OOD queries by merely utilizing the information from the query side, because the relevance between queries and the video can only be correctly acquired based on their semantical matching degree. The OOD query identification is unsolvable if the model has no access to the cross-modal information but only the query information.

2.Secondly, in the open-set experiments of this work, the OOD queries for each video are simply selected from the other datasets. However, there are easily identifiable characteristics among queries from the three datasets in use, i.e., ActivityNet Captions, Charades-STA, and TACoS, since they respectively focus on the open-domain content, indoor activities and cooking scenarios. This provides an obvious but incorrect shorcut for the model to learn and I think this is very likely the reason why the current model performs well on the OOD detection metrics.

3.For the video moment retrieval modules for localizing relevant queries, the designs are short of novelty and lack sufficient insights. Specifically, the video-query matching and frame-query matching module utilize the video-sentence contrastive learning and frame-wise attention gating operations and these are common techniques well investigated in early existing works [1,2,3]. In addition, it seems unclear why the Positive-unlabeled Learning module should adopted for training. Given the reliable ground-truth labels, e.g., the temporal IoU between ID queries and moment proposals, why should we need to convert the training into a semi-supervised learning process that relies on the unreliable cross-modal similarities learned by the model?

4.There are some confusing expressions and results in the paper. For example, in Table 2, Table 3 and Table 4, the type of the proposed model is written as "OS". Combining my personal understanding and the content in Table 1, the closed-set model proposed in this work adopts the same setup as the previous fully-supervised methods, so what additional meaning is expressed by "OS" in the above tables? 
Besides, it is a bit confusing to me that the proposed model which is not sophisticatedly designed for the fully-supervised setting turns out to outperform the previous state-of-the-arts by a very significant margin, especially on the ActivityNet Captions and TACoS datasets. Necessary explanations from authors should be added to clarify these confusing points.

Reference

[1] Niluthpol Chowdhury Mithun, Sujoy Paul, Amit K. Roy-Chowdhury. Weakly Supervised Video Moment Retrieval From Text Queries. In CVPR 2019.

[2] Hao Zhang, Aixin Sun, Wei Jing, Joey Tianyi Zhou. Span-based Localizing Network for Natural Language Video Localization. In ACL 2020.

[3] Hao Zhang, Aixin Sun, Wei Jing, Guoshun Nan, Liangli Zhen, Joey Tianyi Zhou, Rick Siow Mong Goh. Video Corpus Moment Retrieval with Contrastive Learning. In SIGIR 2021.
paper_summary: paper_summary: ● This paper focuses on Video Moment Retrieval (VMR), which involves retrieving relevant video segments or moments corresponding to natural language queries. 
● Specifically, it addresses the challenges of identifying in-domain (ID) queries and out-of-domain (OOD) queries, as well as refining the boundary between them for more accurate retrieval.
● The paper proposes a comprehensive framework comprising several modules, including ID knowledge acquisition, uncertainty-aware OOD boundary reasoning, and ID-OOD boundary refinement.
strength_weakness: strength_weakness: ○ The paper introduces the novel concept of Open-set Video Moment Retrieval (OS-VMR), which extends traditional video moment retrieval by requiring the system to handle out-of-domain (OOD) queries in addition to in-domain (ID) queries. This innovation is particularly significant for applications in open-set environments.
○ The paper provides a clear theoretical explanation of how the OpenVMR framework operates. 
○ The paper conducts several experiments that validate the effectiveness of the proposed method, demonstrating its ability.
limitations: limitations: Need for Additional Supporting Material:
1.   On line 89, why can this solution be directly obtained?   The author needs to provide more supporting material to substantiate this claim.   Obviously, most of the video content is irrelevant to the query, and only a very short video segment matches the query.
Misunderstandings in the Experimental Section Tables:
2.   Table 11: In the parameter ablation experiment, what are the default values of the other parameters when one parameter is fixed?   The author needs to clarify.
3.   Table 10: The author may need to provide experiments combining VQM, FQM, and PUL to further demonstrate the effectiveness of the method
Questions:
4.   What phenomena might occur if a large number of videos are used to create an open-domain video dataset tethered with closed-domain queries?
Some parts of the article are written in an obscure and difficult-to-understand manner.
5.   "Ours(a), Ours(b), and Ours(c)" in Tables 5, 6, and 7 are mentioned later in subsequent ablation experiments, but there is no explanation in Tables 5, 6, and 7 caption, which may hinder readability.
paper_summary: paper_summary: Video Moment Retrieval (VMR) is a challenging multimedia task that aims to localize a specific moment based on an untrimmed video and a sentence query. Existing VMR methods refer to a close-set assumption that all the queries are video-relevant (ID). However, the authors notice that real-world users might provide a video-irrelevant query (OOD query) in realistic applications, which will lead to unreasonable results. Based on this, the authors wisely pose a novel setting, open-set VMR, which first distinguishes ID and OOD queries, and then conducts moment retrieval based on ID queries. Extensive experiments demonstrate the effectiveness of the proposed method.
strength_weakness: strength_weakness: +The paper is easy to understand, and the methodology is technologically sound. The normalizing flow can learn the query distribution, and the multi-variate Gaussian distribution is very consistent with real-world applications. Authors equivalently convert the log-likelihood to the uncertainty score u(q) for simplifying the computation, which is very ingenious to reason the ID-OOD boundary. 

+The experiments are sufficient. The performance improvement is very impressive, which illustrates the effectiveness of the proposed method. Both close-set and open-set settings are considered in this paper. Besides, the ablation study is profound.

+The open-set VMR task is very interesting, novel and consistent with reality. Previous methods mechanically utilize the given query for VMR.
limitations: limitations: -Authors should try other pretrained video or query encoders in Multimodal LLM for better performance.

-Could the proposed method serve as a plug-and-play module for existing VMR works? If the proposed method can help previous works deal with the open-set problem, it will be very exciting.

-Some VMR related work should be included for comparison and analysis, such as “Video Moment Retrieval With Cross-Modal Neural Architecture Search, TIP 2022 ”, “Deconfounded Video Moment Retrieval with Causal Intervention, SIGIR 2021”
paper_summary: paper_summary: This paper poses a novel and realistic multi-modal task, open-set video moment retrieval, which aims to not only retrieve the precise moments based on ID query, but also to reject OOD queries. Unlike previous methods based on a closed-set assumption that all the given queries as ID, the proposed model OpenVMR can first distinguish ID and OOD queries based on the normalizing flow technology, and then conduct moment retrieval based on ID queries. Extensive experiments on open-set and closed-set settings show the effectiveness of OpenVMR.
strength_weakness: strength_weakness: 1. The open-set setting is interesting and novel. The motivation for the open-set setting is clear, and the proposed method is easy to understand. The technical novelty is enough, and the proposed model is sound.

2. This paper is well-written and easy to follow, and the structure is carefully organized. Figures and tables are well-illustrated. The experimental results are impressive.

3. Extensive experiments under both closed-set and open-set settings are conducted. For each core module, the corresponding ablation study is well-designed to fully analyze its effectiveness. Experimental results show that the proposed method can beat other state-of-the-art approaches.
limitations: limitations: 1. The authors employ GloVe to extract the textual feature. I am wondering if transformer-based contextualized embedding models, such as BERT, can perform better.

2. In Section 3.2.2, the authors assume that p_X(x) obeys the Multi-variate Gaussian Distribution. Why? Have the authors tried any other multi-variate distributions? e.g., Multivariate Dirichlet Distribution
paper_summary: paper_summary: This paper addresses the limitations of existing Video Moment Retrieval (VMR) methods that operate under a closed-set assumption, assuming all input queries are relevant to the video content. The authors introduce a novel task, Open-Set Video Moment Retrieval (OS-VMR), which aims to not only retrieve video moments for in-distribution (ID) queries but also reject out-of-distribution (OOD) queries. They propose a model named OpenVMR, which employs normalizing flow technology to distinguish between ID and OOD queries and subsequently conducts moment retrieval for ID queries. The model involves several key components: learning the ID query distribution, introducing an uncertainty score to separate ID and OOD queries, refining the ID-OOD boundary using a triplet loss, and performing video-query and frame-query matching for accurate moment retrieval. Extensive experiments on three datasets (ActivityNet Captions, Charades-STA, and TACoS) demonstrate the effectiveness of OpenVMR, showing significant improvements over state-of-the-art methods in both closed-set and open-set settings.
strength_weakness: strength_weakness: This paper introduces the novel task of Open-Set Video Moment Retrieval (OS-VMR), addressing an important gap in the current VMR research. By moving beyond the closed-set assumption, the authors tackle a significant real-world problem where irrelevant queries may be presented. The use of normalizing flow technology to differentiate between in-distribution (ID) and out-of-distribution (OOD) queries is innovative and well-executed. Additionally, the comprehensive evaluation across multiple challenging datasets, including ActivityNet Captions, Charades-STA, and TACoS, provides robust empirical evidence supporting the proposed method's effectiveness. The model's performance improvements over state-of-the-art methods are particularly noteworthy.
limitations: limitations: However, the paper is dense and challenging to follow in some sections, particularly in the methodological explanations. Improving clarity and structuring the content more logically could enhance readability. While the paper compares its results with state-of-the-art methods, it would benefit from a more detailed analysis of why the proposed method outperforms these baselines. Providing more insights into the failure modes of existing methods when dealing with OOD queries could strengthen the argument.

The model's performance is impressive on the selected datasets, but the paper lacks a discussion on the generalizability of the approach to other types of videos or queries outside the tested domains. Although the paper includes ablation studies, they could be more detailed. Specifically, analyzing the contribution of each component in varying contexts and datasets would provide a deeper understanding of the model’s robustness.

The paper lacks supplementary materials that could provide additional insights and experimental results. Including detailed experiment logs, hyperparameter configurations, and extended results in the supplementary materials would enhance the transparency and reproducibility of the study. This is a common issue among the state-of-the-art methods cited, which often lack comprehensive supplementary materials, making it difficult to fully validate and reproduce the reported results. Addressing this gap would not only strengthen the current paper but also set a higher standard for future research in this area.
paper_summary: paper_summary: This paper introduces open-set video moment retrieval, which retrieves specific moments based on query sentences and can reject OOD queries. The proposed model uses normalizing flow technology to distinguish between ID and OOD queries.  Moreover, it refines the ID-OOD boundary and incorporates cross-modal interactions for video-query and frame-query matching. Additionally, a positive-unlabeled learning module is introduced for moment retrieval. Experimental results on three datasets demonstrate its effectiveness.
strength_weakness: strength_weakness: 1. This paper proposes the open-set Video Moment Retrieval (OS-VMR) task, which is interesting and valuable in open-set settings.

2. This paper presents an OpenVMR framework to distinguish ID and OOD queries and utilizes ID queries for moment retrieval.

3. Extensive experiments are conducted on three popular datasets.
limitations: limitations: 1. How to construct OOD queries?

2. The initial focus of this paper appears to address an OOD detection challenge, suggesting numerous methods could be employed. Why does the paper opt for normalizing flow? Are there alternative strategies considered?

3. The ID queries and video matching rely on commonly employed modules, lacking novel technical insights. Similarly, algorithms for the OOD and ID separation appear conventional. Overall, the paper's contribution appears limited.

4. While the consideration of OOD queries is pertinent, why restrict the discussion solely to OOD queries rather than also addressing OOD videos?