MomentDiff: Generative Video Moment Retrieval from Random to Real
Authors: Pandeng Li
Chen-Wei Xie
Hongtao Xie
Liming Zhao
Lei Zhang
Yun Zheng
Deli Zhao
Yongdong Zhang
Venue: NeurIPS 2023 poster (2023)
Keywords: Video Moment Retrieval
Diffusion Model
TL;DR: We demonstrate the effectiveness and generalization of our generative video moment retrieval model.

Abstract: Video moment retrieval pursues an efficient and generalized solution to identify the specific temporal segments within an untrimmed video that correspond to a given language description.
To achieve this goal, we provide a generative diffusion-based framework called MomentDiff, which simulates a typical human retrieval process from random browsing to gradual localization.
Specifically, we first diffuse the real span to random noise, and learn to denoise the random noise to the original span with the guidance of similarity between text and video.
This allows the model to learn a mapping from arbitrary random locations to real moments, enabling the ability to locate segments from random initialization.
Once trained, MomentDiff could sample random temporal segments as initial guesses and iteratively refine them to generate an accurate temporal boundary.
Different from discriminative works (e.g., based on learnable proposals or queries), MomentDiff with random initialized spans could resist the temporal location biases from datasets.
To evaluate the influence of the temporal location biases, we propose two ``anti-bias'' datasets with location distribution shifts, named Charades-STA-Len and Charades-STA-Mom.
The experimental results demonstrate that our efficient framework consistently outperforms state-of-the-art methods on three public benchmarks, and exhibits better generalization and robustness on the proposed anti-bias datasets. 
The code, model, and anti-bias evaluation datasets will be released publicly.

MomentDiff: Generative Video Moment Retrieval from Random to Real
Pandeng Li Chen-Wei Xie", Hongtao Xie' Liming Zhao?,
Lei Zhang', Yun Zheng", Deli Zhao?, Yongdong Zhang' University Science and Technology of China, Hefei, China
Alibaba Group
1pd@mail. ustc. edu. en, fhtxie, leizh23, 2hyd73) edu. cn (eniac. xcw, lingchen zlm, chengyun.zy: ibaba-babaaeee inn com thaodeli@gmal..o
Abstract
Video moment retrieval pursues an efficient and generalized solution to identify the specific temporal segments within untrimmed video that correspond to given language description. To achieve this goal, we provide generative diffusion- based framework called MomentDiff, which simulates a typical human retrieval process from random browsing to gradual localization. Specifically, we first diffuse the real span to random noise, and learn denoise the random noise the original span with the guidance of similarity between text and video. This allows the model learn a mapping from arbitrary random locations real moments, enabling the ability to locate segments from random initialization. Once trained, MomentDiff could sample random temporal segments as initial guesses and iteratively refine them to generate an accurate temporal boundary. Different from discriminative works (e.g., based on learnable proposals or queries), MomentDiff with random initialized spans could resist the temporal location biases from datasets. evaluate the influence of the temporal location biases, we propose two "anti-bias" datasets with location distribution shifts, named Charades-STA-LL. and Charades- STA-Mom. The experimental results demonstrate that our efficient framework consistently outperforms state-of-th-aat methods on three public benchmarks, and exhibits better generalization and robustness on the proposed anti- bias datasets. The code, model, and
evaluation datasets available https:: :/g1 thub Shap//MCCreetrioall/ommettt
Introduction
Video understanding 1.9 is a crucial problem in machine learning [0-16 which covers various video analysis tasks, such video classification and action detection. But both tasks above limited to predicting predefined action categories. A more natural and elaborate video understanding process is the ability for machines match human language descriptions specific activity segments in complex video. Hence, a series of studies 17H22 are conducted on Video Moment Retrieval (VMR), with the aim of identifying the moment boundaries í.e. the start and end time) within given video that best semantically correspond to the text query.
As shown in Fig. aa), early works address the VMR task by designing predefined dense video proposals (ie. sliding windows 23 25), anchors (26] and 2D map [2)] ). Then, the prediction segment determined based the maximum similarity score between dense proposals and the
Interns at Alibaba Group t Corresponding author
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Predefined video proposals
N learnable queries
N randomn spans
end
crd
start
Video
Video
Video
score spar
soore span
spun
Decoder
Denoiser
Text
Text
Text
oventiiee
(a) Dense: 2DTTA
(b) Sparse: MomentDETR
(c) Random: MomentDiff
(d) Inference
person is
Fusion canbedding
swakening.
Timeline
Query
Denoising
text
Danoising
Denoising
Retking
VMD
VMD
VMD
Top-1
Video
Random spans intensity i
Noise
Prodict-ppans
incensity m-1
Noise
Preduct spans
intensily
Noiie
Result
Figure 1: (a) Proposal-predefing occurring moments in the dataset. (b) Proposal- methods. (c) Our generative method. (d) We model the VMR task as a process of gradually generating real temporal span from random noise.
methods. Yellow highlights in the timeline represent frequently
query text. However, these methods have a large redundancy of proposals and the numbers of positive and negative proposals are unbalanced, which limits the learning efficiency (28). To deal with this problem, a series of VMR works (29]36) have recently emerged, mainly discussing how to reduce the number of proposals and improve the quality of proposals. Among them, a promising scheme
(Fig. [[b) is to use sparse and learnable proposals (34] or queries [37 38 (i.e., soft proposals) to model the statistics of the entire dataset and adaptively predict video segments. However, these proposal-learnatee
methods rely on a few specific proposals or queries to fit the location distribution of ground truth moments. For example, these proposals or queries may tend to focus on video segments where locations in the dataset occur more often (i.e., yellow highlights in Fig. (b)). Thus, these methods potentially disregard significant events that transpire in out-of-sample situations. Recent studies (39 40) indicate that VMR models [35] 27 32) may exploit the location biases present in dataset annotations (23], while downplaying multimodal interaction content. This leads to the limited generalization of the model, especially in real-world scenarios with location distribution shifts.
To tackle the above issues, we propose a generative perspective for the VMR task. As shown in Fig. ! (c) and (d), given an untrimmed video and the corresponding text query, we first introduce several random spans as the initial prediction, then employ random spans by conditioning on similarity relations between the text query and video frames. A heuristic explanation of our method is that, it can be viewed as a way for humans to quickly retrieve moments of interest in video. Specifically, given an unseen video, instead of watching the entire video from beginning to end (which too slow), humans may first glance through random contents to identify a rough location, and finally iteratively focus on key semantic moments and generate this way, we do not rely on distribution-specife proposals or queries (as mentioned in the above discriminative approaches) and exhibit more generalization and robustness (Tab. 5 and Fig. when the ground truth location distributions of training and test sets are different.
iiffusion-based denoiser to iteratively refine the
temporal coordinates.
To implement our idca, we introduce a generative diffusson-based framework, named MomentDiff. Firstly, MomentDiff extracts feature embeddings for both the input text query and video frames. condition generator. This generator modulates the video embeddings with text embeddings to produce text-video fusion embeddings. The fusion embeddings contain rich semantic information about the similarity relations between the text query and each video frame, so we can use them as a guide to help us generate predictions. Finally, we develop Video Moment Denoiser (VMD) that enhances noise perception and enables efficient generation with only a small number of random spans and flexible embedding learning. Specifically, VMD directly maps randomly initialized spans into the multimodal space.
Subsequently, these text and video embeddings are fed into a similarity-
2

---

taking them as imput together with noise intensities. Then, VMD iteratively refines spans according to the similarity relations of fusion embeddings, thereby generating true spans from random real.
Our main contributions are summarized as follows. 1) To the best of our knowledge, we are the first to tackle video moment retrieval from generative perspective, which does not rely on predefined or learnable proposals and mitigates temporal location biases from datasets. 2) We propose new framework, MomentDiff, which utilizes diffusion models to iteratively denoise random spans the correct results. 3) We propose two "anti-bias" datasets with location distribution shifts to evaluate the influence of location biases, named Seaardes-STA-LLEe experiments demonstrate that MomentDiff is more efficient and transferable than state-of- of-te-----eeatt methods on three public datasets and two anti-bias datasets.
and SEarrdes-STA-MMe
Extensive
2 Related Work
Video Moment Retrieval. Video moment retrieval 32
33 43 46 is a newly researched subject that emphasizes retrieving correlated moments in a video, given natural language query. Pioneering works are proposal-bas based approaches, which employ a "proposal-rank two-stage pipeline. Early methods [23] 25 [32 usually use handcrafted predefined proposals to retrieve moments. For example, CTRL 23 and MCN [25] aim to generate video proposals by using sliding windows of different scales. TGN 26 emphasizes temporal information and develops multi-scale candidate solutions through predefined anchors. 2DTAN
27 designs a 2D temporal map to enumerate
proposals. However, these dense proposals introduce redundant computation with large number of negative samples [28 36). Therefore, two types of methods are proposed: 1) Proposal-fre methods [1731]
[7 do not use any proposals and are developed to directly regress start and end boundary values probabilities based on ground-truth segments. These methods usually much faster than proposal-basedd methods. 2) Proposal-learnable methods that use proposal prediction networks (6 or learnable queries [37 38 to model dataset statistics and adaptively prediet video segments. QSPN [35) and APGN 34 adaptively obtain discriminative proposals without handcrafted design. LPNet (36) uses learnable proposals to alleviate the redundant calculations in dense proposals. MomentDETR (37 can predict multiple segments using learnable queries. Since proposal-learnable methods adopt two-stage prediction 34 or implicit iterative (77 design, the performance is often better than that of proposal -fee methods. However, proposal-learnable methods explicitly fit the location distribution of target moments. Thus, models are likely to be inclined to learn location bias in datasets [39] 40). resulting in limited generalization. We make no assumptions about the location and instead use random inputs to alleviate this problem.
Diffusion models. Diffusion Models [48]5]] are inspired by stochastic diffusion processes in non- equilibrium thermodynamies. The model first defines Markov chain of diffusion steps to slowly add random noise to the data, and then learns the reverse diffusion process to construct the desired data samples from the noise. The diffusiin-basel generation has achieved disruptive achievements in tasks such as vision generation (52458) and text generation [59] Motivated by their great success in generative tasks [60]. diffusion models have been used in image perception tasks such as object detection [61] and image segmentation [62]. However, diffusion models are less explored for video- text perception tasks. This paper models maniilartty-warrr multimodal information as garse-grained cues, which can guide the diffusion model to generate the correct moment boundary from random noise in a gradual manner. Unlike DiffusionDet 61,, we avoid large number of Region of Interest (ROI) features and do not require additional post- post-processing techniques. To our knowledge, this is the first study to adapt the diffusion model for video moment retrieval.
3 Method
In this section, we first define the problem in Sec. 3.1 introduce our framework in Sec. 3.2 and describe the inference process in Sec. 3.3
3.1 Problem Definition
Suppose untrimmed video V {vx J==1 is associated with natural text description T [t,),,² where N, and N, represent the frame number and word number, respectively. Under this notation definition, Video Moment Retrieval (VMR) aims to learn model Ω to effectively predict the moment 2o = (čo, up) that is most relevant to the given text description: a = 22( V), where ćs and wo represent the center time and duration length of the temporal moments, iee. predicted spans.
3

---

Leim
umr
Timeline
Predicted spans
Video encoder
MLP
fusion embedkings
Noise intensity
Similarity- aware
cundition
Video Moment
Condition Generator
Denoiser
Timeline
Ground Iruth
Timeline
Random spans
a person is awakening.
Text encoder
Noise intensity
Figure Our MomentDiff framework, which includes Similarity-awe aware Condition Generator (SCG) and Video Moment Denoiser (VMD). The diffusion process is conducted progressively in VMD.
3.2 The MomentDiff Framework
Fig. 2 sheds light on the generation modeling architecture our proposed MomentDiff. Concretely, we first extract frame-level and word-level features by utilizing pre-trained video and text backbone networks. Afterward, we employ a similarity-awan condition generator to interact text and visual features into fusion embeddings. Finally, combined with the fusion embeddings, the video moment denoiser can progressively produce accurate temporal targets from random noise.
3.2.1 Visual and Textual Representations.
Before performing multimodal interaction, we should convert the raw data into a continuous feature space. To demonstrate the generality of our model, we use three distinct visual extractors 32
to obtain video features V: 1) 2D visual encoder, the VGG model (33 2) 3D visual encoder, the
C3D model [64). 3) Cross-modal pre-train encoder, the CLIP visual model [65]. However, due to the absence of temporal information in CLIP global features, we additionally employ the SlowFast model [66] to extract features, which concatenate CLIP features. Besides, to take full advantage the video information (38], we try incorporate audio features, which are extracted using pre-trained PANN model (67]. To obtain text features, we try two feature extractors: the Glove model [68] and the CLIP textual model extract 300-d and 512-d text features T, respectively.
3.2.2 Similarity-aware Condition Generator
Unlike generation tasks (69 that focus on the veracity and diversity results, the key to the VMR task is to fully understand the video and sentence information and to mine the similarities between text queries and video segments. To this end, we need to provide multimodal information to cue the denoising network to learn the implicit relationships in the multimodal space.
natural idea is to interact and aggregate information between video and text sequences with a multi- layer Transformer [70]. Specifically, we first use two multilayer perceptron (MIP) networks to map feature sequences into the common multimodal space: V E XD and T E N. where D is the embedding dimension. Then, we employ two cross-attention layers to perform interactions between multiple modalities, where video embeddings V are projected as the query Q,, text embeddings T
are projected as key K, and value V,: : V softmax (Q,KT) V. + Q,,, where V NRN.D To
help the model better understand the video sequence relations, we feed V into 2-layer self-attention network, and the final similarity-aware fusion embedding is F softmax (? V Q.-
where Q K V. is the matrix obtained from V after three different projections respectively.
In the span generation process, different text queries are very different. Since the fusion embedding F serves as the input condition of the denoiser, the quality of F directly affects the denoising process. To learn similarity relations for F in the multimodal space, we design the similarity loss Liim, which contains the pointwise cross entropy loss and the pairwise margin loss:
for the same video, the correct video segments corresponding to
1 Σ max (0,8 + Sn, -8p,) (1) N.
Lsim
N. Σ y: log(s.) + (1 y,) * log(1 -8:)

---

where & E R. is the similarity score, which is obtained by predicting the fusion embedding F through the MLP network. y E R N. is the similarity label, where y; if the i-th frame is within the ground truth temporal moment and y, = 0 otherwise. 8p; and Sn; are the randomly sampled positive and negative frames. N₂ is the number of samples and the margin & = 0.2. Although fesim may only help the fusion embedding retain some graiied similarity semantics, this still provides indispensable multimodal information for the denoiser.
3.2.3 Video Moment Denoiser
Recent works 39 40 have revealed that previous models 35 27 32 may rely on the presence of location bias in annotations to achieve seemingly good predictions. To alleviate this problem, instead of improving distribution-speciif proposals or queries, we use random location spans to iteratively obtain real spans from a generative perspective. In this section, we first introduce the principle of the forward and reverse processes in diffusion models. Then, we build the diffusion generation process in the video moment denoiser with model distribution (ro) to learn the data distribution (2x))
Forward process. During training, we first construct for- ward process that corrupts real segment spans æo q(20)) to noisy data Zm. where m is the noisy intensity. Specif- ically, the Gaussian noise process of any two consecu- tive intensities [49] can be defined as: q (Im Im-1)=
fa- B prodicted spans Diffuse
Scores Aas
PC
PC
2x
Intensity-aware Attention
N (Im: v1- Bm 1m. Bm. where / is the variance schedule. In this way, Em can be constructed by 2n: q (m1 x = II 9 (æ; 1). Benefiting from the reparameterization technique, the final forward process is simplified to:
Kf
intenaty
Position
enooding
3m am + v1 Em:
(2)
fusson
embeddings
Random spant
where the noise 6m ~ NN(O)) I) and a'm- = II=-
3c).
Figure 3: Video moment denoiser. For simplicity, we only draw the intensity- aware attention structure that is different from the general Transformer.
Reverse process. The denoising process is learning to remove noise asymptotically from I'm to Fo, and its tradi- tional single-step process can be defined as:
Po (#m-1 (mm) W (Im-11 1: (Im. m), o
where o associated with Bm and pgg (mm, m) is the predicted mean. In this paper, we train the Video Moment Denoiser (VMD) to reverse this process. The difference is that we predict spans from the VMD network fo (Im: m, F) instead of Heo (Im. m).
Denoiser network. As shown in Fig. 3. the VMD network mainly consists of 2-layer cross attenntt Transformer layers. Next, we walk through how VMD works step by step. For clarity, the input span and output prediction presented below a single vector.
Span normalization. Unlike generation tasks, our ground-trute truth temporal span 2g is defined by two parameters co and Wo that have been normalized to [0, IJ. where co and wo are the center and length of the span Øo. Therefore, in the above forward process, we need to extend its scale to |-X, A] to stay close to the Gaussian distribution [ 61]. After the noise addition is completed, we need to clamp I.m to [-X, >] and then transform the range to [0, 1]: Im = (æm/A + 1)/2, where A = 2.
Span embedding. To model the data distribution in multimodal space, we directly project the discrete span to the embedding space through the Fully Connected (FC) layer: x'n FC(#m) R.." Compared to constructing ROI features in DiffusionDett [61], linear projection is very flexible and decoupled from conditional information (i.e., fusion embeddings), avoiding more redundancy.
8 Intensity-aware attention. The denoiser needs to understand the added noise intensity m during denoising, so we design the intensity-aware attention to perceive the intensity magnitude explicitly. In Fig. 3 we use sinusoidal mapping for the noise intensity m to obtain em. E IR in the multimodal space and add it to the span embedding. We project '''n + em as query embedding and the positional embedding pos,, m E IR is obtained by sinusoidal mapping of Im. We can obtain the input query: + em), ). Similarly, The input key is K , = (Proat(Proj(F)) pos 3) and the input value is V f = Proj(F), where Proj(-) is the projection function and pos, E R is the standard position embedding in Transformer [70]. Thus, the srensity-awart attention is:
Q. = Concat(Proj((
Qm = softmax (Qm (0mkt) V + Qm
5

---

4 Denoising training. Finally, the generated transformer output is transformed into predicted spans 3m-1 (cm-1, 1mm-1) and confidence scores 2m which are implemented through simple FC
layer, respectively. Following [7]. the network prediction should be as close to ground truth æo as possible. In addition, inspired by [7 72). we define the denoising loss as:
Cymr (zo, (20: m. F)) AL1 - 8m-1ll
Xiou Liou (zo, 3m + XceCce ( (2m-1),
(5)
where ALI Anau and Ace are hyperparameters. Lim is a generalized IoU loss 73 fice is a cross-entropy loss. Note that the above procedure is a simplification of training. Considering that there may be more than one ground truth span in the dataset [37], we set the number of input and output spans to Ny. For the input, apart from the ground truth, the extra spans are padded with random noise. For the output, we calculate the matching cost of each predicted span and ground truth according to Cume (i.e., the Hungarian match (72), and find the span with the smallest cost calculate the loss. In Les, we set the confidence label to 1 for the best predicted span and 0 for the remaining spans.
3.3 Inference
After training, MomentDiff can be applied to generate temporal moments for video-text pairs including unseen pairs during training. Specifically, we randomly sample noise &.m from Gaussian distribution (0, I), the model can remove noise according to the update rule of diffusion models [69]]
o Bm Va fo m. F)
8m =v am a. m. F) + V 0m-1
+ J Cm-
(6)
Om
As shown in Fig dd) we iterate this process continuously obtain &o from coarse to fine. Note that in the last step we directly use ʃe ( 1, F)) as &gg. In Lg. we choose the span with the highest confidence score in ên as the final prediction. To reduce inference overhead, we do not employ any post- post-processing techniques, such as box renewal in DiffusionDet [61] and self-condition [71].
4 Experiments
4.1 Datasets, Metrics and Implementation Details
Datasets. We evaluate the efficacy of our model by conducting experiments on three representative datasets: Charades-STI, 23), QVHighlights (37] and TACoS 774. The reason is that the above three datasets exhibit diversity. Charades STA comprises intricate daily human activities. HVHighlights contains a broad spectrum of themes, ranging from everyday activities and travel in lifestyle vlogs to social and political events in news videos. TACoS mainly presents long-form videos featuring culinary activities. The training and testing divisions are consistent with existing methods 28 38
Metrics. To make fair comparisons, we adopt the same evaluation metrics as those used in previous 19 27 namely RI @n, MAP@n. and MAP Specifically, RJ @n is defined as the percentage of testing queries that have at least one correct retrieved moment (with an intersection over union (loU) greater than n) within the top-1 results. Similarly, MAP@n n is defined as the mean average precision with an IoU greater than n, while MAP aey is determined as the average MAP@n across multiple IoU thresholds [0.5: 0.05: 0.95].
works [38] 37 23 75 29
Implementation details. For a fair comparison 44
20
30
we freeze the video encoder and
C3D (44 or SlowFast+CLLIP (SF+C) [66] 65). we extract video features every 1/6s, 1s or 2s. So the frame number N, is related to the length of the video, while the max text length N, set to 32, Note that since the videos in TACoS are long, we uniformly sample the video frame features and set the max frame number Ne to 100 in TACoS. We set the hidden size D 256 in all Transformer layers. In SCG, we also use variant of the pairwise margin loss called InfoNCE loss (76). The number of random spans N, is set to 10 for QVHighlights, 5 for Charades-STI STA and TACoS. We use the cosine schedule for For all datasets, we optimize MomentDiff for 100 epochs on one NVIDIA Tesla A100 GPU, employ Adam optimizer 777 with le-4 weight decay and fix the batch size as 32. The learning rate is set to le-4. By default, the loss hyperparameters \L. 10, Xion 1 and A = 4. The weight values for Lsim and Lomr are and To speed up the sampling process during inference, we follow DDIM (69 and iterate 50 times.
text encoder and use only the extracted features. For VGG 63

---

Table 1: Performance comparisons (%) the Charades-SS -Trades-STAAAA. dataset. "x" denotes that implement the method under the same training scheme. "A" stands for using audio data.
Charades-STA
Method
Type
RI 1005
0.0.
MAP@0.5
MAP@0.75
MAPavg
MAN 3
VGG, Glove
41.21
20.54
RaNet"
75
VGG, Glove
42.91
25.82
53.28
24.41
28.55
2DTAN
27
VGG, Glove
41.34
23.91
54.68
24.15
29.26
DORi 778
VGG, Glove
43.44
26.37
CBL.N 44
VGG, Glloee
47.94
28.22
DCM 40
VGG, Glove
47.80
28.00
MMN
28
VGG, Glove
46.93
27.07
58.85
28.16
31.58
MomenIDETR
VGG, Glove
50.54
28.01
57.39
25.62
29.87
MomentDiff
VGG, Glove
51.94
28.25
59.86
29.11
31.66
UMT 38
VGG+A. Glove
48.44
29.76
58.03
27.46
30.37
MomentDiff
VGG+A, Glove
52.62
60.69
29.74
31.81
7
C3D, Glove
37.39
17.69
LPNet 36
C3D, Glove
40.94
21.13
VSI.Net" [30
C3D, Glove
48.67
30.33
56.88
25.79
30.16
MomentDETR
37
C3D, Glove
50.49
29.95
56.27
26.08
29.92
MomentDiff
C3D, Glove
53.79
30.18
59.32
29.85
31.89
MomentDETR
37
SF4C, C
53.22
30.87
58.86
26.43
30.43
MomentDiff
SF4C, Glove
55.42
32.17
60.93
32.47
32.59
MomentDiff
SF+C, C
55.57
32.42
61.07
32.51
32.85
4.2 Performance Comparisons
Comparison with state-of-the-art methods. prove the effectiveness of MifentDiff, we compare the retrieval performance with 17 discriminative VMR methods. Tab. D Tab. 2 and Tab. show @n, MAP@n, and MAP aag results on Charades-STI QVHighlights and TACoS. Compared MomentDiff achieves significant improvements on Charades-STr regardless whether 2D features (VGG), multimodal features (VGG+A), 3D features (C3D), multimodal pre-trained features (SF+C) are used. This proves that MomentDiff is universal generative VMR method. In the other two datasets (QVHighlight and TACoS), we still have highly competitive results. Specifically, compared to MomenIDETR [37], MomentDiff obtains 2.35%, 3.86% and 13.13% average gains in @0.5 on three datasets. It is worth noting that TACoS contains long videos of cooking events where different events only slightly different in terms of cookware, food and other items. The learnable queries in MomentDETR may not cope well with such fine-grained dynamic changes. We attribute the great advantage of MomentDiff to fully exploiting similarity-awarr condition information and progressive refinement denoising.
with SOTA methods [28]
38
75
30
37
these methods
Table 2: Performance comparisons (%) on QVHighlights Table 3: Performance comparisons (%) with SF+C video features and CLIP text features. "*" de-on TACoS. We adopt C3D features notes that we re-implement the method with only segment encode videos. MDE is the abbreviation moment labels. "f" stands for using audio data. MDE the of MomentDETR [37] abbreviation of MomentDETR
TACoS
Method
HVHighlights
RT#0.1
RI 2T03
RI60.3
Method
@100.5
MAP@0.5
MAP@0.75
MAP
CTRL
23
24.32
18.32
13.30
11.41
24.94
10.67
SCDM
32
26.11
21.17
CAL
25.49
23.17
81
41.83
44.63
38.84
29.07
XML+
46.69
47.89
34.90
CBLN
441
49.16
38.98
27.65
MDE
53.56
53.97
28.65
29.39
FVMR
12
$3.12
41.48
29.12
RaNet
75
43,34
33.54
Mofen:Diff
57.42
39.66
54.02
35.73
35.95
M
38
56.26
40.31
$2.77
36.82
35.79
MDE
37
41.16
32.21
20.55
MMN
51.39
39.24
26.17
MiftntDift
58.21
41.48
$4.57
37.21
36.84
33.68
Transfer experiments. To explore the location bias problem, we first organize the Out of Distri- bution (OOD) experiment following [82), which repartitions Charades-STA 23 and ActivityNet-
Captions 83 datasets according to moment annotation density values [82 In Tab. exceed

---

Table 4: Transfer experiments (%)
Charades-CD (VGG,, Glove) and ActivityNet-CD (C3D, Glove).
Charades-CU
ActivityNet-CD
Method
@0.3
@10.
@1@0.7
MAP axg
@1
RI@0.5
RI@0.7
MAP aug
2DTAN 27
49.71
28.95
12.78
12.60
40.04
22.07
10.29
12.77
MMN 28
55.91
34.56
15.84
15.73
44.13
24.69
12.22
15.06
DETR 37
57.34
41.18
19.31
18.95
39.98
21.30
10.58
12.19
MomentDiff
67.73
47.17
22.98
22.76
45.54
26.96
13.69
16.38
Table 5: Transfer experiments (%) on anti-bias datasets with VGG and Glove video-text features.
Charades-STA-LLe
Charades- SAonssSTTMMMoo RI@0.5
Method
@00.3
RI @0.5
@1@0.7
MAP uny
RI @0.3
@100.7
MAP avg
2DTAN 127
39.68
28.68
17.72
22.79
27.81
20.44
10.84
17.23
MMN 28
43.58
34.31
19.94
26.85
33.58
27.20
14.12
19.18
METE 37
42.73
34.39
16.12
24.02
29.94
22.16
11.56
18.66
MomeniDiff
51.25
38.32
23.38
28.19
48.39
33,59
15.71
21.37
MomentDETR by large margin on Charades-CD and ActivityNet-CD, These results prove the robustness of the model in dealing with OOD scenarios.
To further explore the impact of a single factor (wo or Co ) on the location bias problem, we organize moment retrieval on two anti-bias datasets with location distribution shifts: O Charades-STA-Len We collect all video-text pairs with wy S 10s and randomly sample pairs with wg > 10s in the original training set of Charades-STA. which account for 80% and 20% of the new training set, respectively. On the contrary, we collect all pairs with wo > 10s and randomly sample pairs with wo < 10s from the original test set, accounting for 80% and 20% of the new test set. @ Charades-STA-MGR Similarly, we collect all video-text pairs with the end time + wo/2 2 Σ 15s and sample pairs with the start time Co wa/22 > 15s as the training set, which accounts for 80% and 20%, respectively. Likewise, the construction rules for the test set are the opposite of those for the training set. Dataset statistics can refer to Train/Test in Fig. and the supplementary material.
In Tab. 5 the proposed MomentDiff shows much more robustness than previous state-of-the-art method MomentDETR 37 Concretely, compared with the experiment in Tab. T the performance gap between MomentDiff and MomentDETR gets larger on Charades STaess-------L -ennLendee and Charades-STA Mom. Fig. 4 also demonstrates that the distribution of our prediction closer the one of the test set. We conjecture that it is because MomentDiff discards the learnable proposals that fit the prior distribution of the training set. Moreover, 2DTAN (27 and MMN (2X perform worse than MomenDETTR on the original Charades-STA dataset, but they achieve better or comparable results
than MomentDETR in Tab. 5 This shows that predefined proposals [27 28 may be better than learnable proposals in dealing with the location bias problem, but they take up more time and space overhead. Differently, our method performs well on both public datasets and anti-bias datasets.
4.3
Ablation Study
To provide further insight into MomentDiff, we conduct critical ablation studies on Charades- STA.
Charades- STA-Len
Charades-STTA STA-Mom
1.0
Thin
L.0
Thxin
0.8
Tert
0.8
Ted
Samples
MDE
(Samples
MDE
0.66
Oux
0.6
0.4
0.4-
0.2
0.2-
0.0-
10
20
0
10
15
20
25
50
35
40
Duration Lengh
Position Time (6)
Figure 4: Statistical distributions on anti-bias datasets with location distribution shifts. We count the length or moment of each video-text pair, and finally normalize the counts (# Samples) for clarity. Train/Test: the count distribution of true spans on the training or test set. MDE: the count distribution of t predicted spans by using MomentDETR on the test set.
8

---

Table Ablation study (%) on the Charades-STD dataset with SF+C video features and CLIP text features. We report RI @0.5, R.@0.7 and MAPav, Default settings are marked in blue
(a) Different span embedding types.
(b) Effect scale X.
(c) VMD and noise intensity m.
Type
R1@0.5
R1@0.7
aAP
scale
R1
R1
MAPary
Type
R1
R10007
MAP...
ROI
PC
55.57
50.21
32.42
28.85
32.85
29.12
2.0
55.57
54.33
50.12
28.08
32.42
32.17
32.85
30.47
w/o
50.41
32.42
29.73
28.72
(e) Rffect of span numher N,-
(f) Model performance vs. speed.
(d) Different loss designs.
N, R1@0.5
R1
MAP
Step
R160.5
R1
MAPa..
FPS
Livr
Coerr
R.8005
R1
MAP.us
50.83
26.17
26.98
531,4
55.57
40.94
31.53
24.65
32.42
16.52
23.76
32,85
17.89
10
5
53.89
55.57
32.42
30.48
27.53
32.85
30.27
10
55.41
55.57
55.62
31.92
32.42
52.16
32.85
33.54
32.99
465.0
338.2
148.8
Query: The person starts drinking glass of coffee.
Query: Kinds checking out their goodles and chocolater chrring Easter.
MBB
20.0
35,0s
42.0
16.5
MemonDDit
2001
GEDX
2001
25.0
2.0
$6.0s
Figure 5: Visualization of the diffusion process on Charades- (Left) and HVHigllights (Right). For clarity, we show 3 random spans (æ55) with Gaussian initialization, and progressively get the top-1 result (2n) according to the confidence score. Green box: right segment. MDE: MomentDETR (37]
Span embedding type. Regarding the way discrete spans are mapped to the embedding space, we compare the ROI strategy [61] with our linear projection (FC) in Tab. a)a). For the ROI strategy, we slice the fusion embeddings F corresponding to random spans, followed by mean pooling on the sliced features. Tab. (b)a shows that ROI does not work well. This may be due to two points: 1) ROI is a hard projection strategy, while the importance of each video frame is quite different. FC is similar to soft ROI, and its process can be trained end-to-end. 2) FC decoupled from F, which allows the model to focus on modeling the diffusion process and avoid over-dependence on F.
Scale A. À the signal-to-nn eggnal-t--ooise ratio [61] of the diffusion process, and its effect is shown in Tab, T)b. We find that the effect of larger À drops significantly, which may be due to the lack of more hard samples for denoising training when the proportion of noise is small, resulting poor generalization.
Video Moment Denoiser (VMD) and noise intensity m. In Tab. ([c,, we first remove the denoiser and the diffusion process (w/o VMD). After training with the same losses, find that predicting with only fusion embeddings F leads to a drastic drop in results, which reveals the effectiveness of denoising training. Then we remove the noise intensity (w/o m.) and the result is reduced by 5.16% on RI @0.5. This shows that explicitly aggregating noise intensity with random spans improves noise modeling. Combined with VMD and m, the diffusion mechanism can fully understand the data distribution and generate the real span from coarse to fine.
Loss designs. In Tab. d), we show the impact of loss functions. In Cyims we use pointwise and pairwise constraints to guide token-wiig wise interactions between multimodal features, while ensuring reliable conditions for subsequent denoising, In Lemr, the model can leam to accurately localize exact segments. Adequate multimodal interaction and denoising training procedures are complementary.
Span number. In Tab. 6(e). we only need 5 random spans to achieve good results. Unlike object detection (61, the number correct video segments corresponding to text query is small. Therefore, large number of random inputs may make the model difficult to train and deteriorate the performance.

---

Model performance vs, speed. In Tab. f),, we explore the effects of different diffusion steps. When step=2. good results and fast inference speed have been achieved. Subsequent iterations can improve the results of high loU (iee, R1 @0.7), which shows the natural advantages of diffusion models.
4.4 Qualitative Results
We show two examples of the diffusion process in Fig. We can find that the retrieved moments by MomentDiff are closer to the ground truth than those by MomentDETR. The diffusion process can gradually reveal the similarity between text and video frames, thus achieving better results. Besides, the final predictions corresponding to spans with multiple random initial locations are close to the ground truth. This shows that our model achieves a mapping from arbitrary locations to real segments.
5 Limitation and Conclusion
Limitation. Compared to existing methods 27 37 the diffusion process requires multiple rounds of iterations, which may affect the inference speed. As shown in Tab. fi,, we reduce the number of iterations, with only small sacrifice in performance. In practical usage, we suggest choosing a reasonable step number for a better trade-off between performance and speed.
This paper proposes a novel generative video moment retrieval framework, MofentDiff, which simulates typical human retrieval style via diffusion models. Benefiting from the denoising diffusion process from random noise to temporal span, we achieve the refinement of prediction results and alleviate the location bias problem existing in discriminative methods. MomentDiff demonstrates efficiency and generalization on multiple diverse and anti-bias datasets. We aim to stimulate further research on video moment retrieval by addressing the inadequacies in the framework design, and fimmly believe that this work provides fundamental insights into the multimodal domain.
Conclusion.
6 Acknowledgement
This work is supported by the National Key Research and Development Program of China (2022YFB3104700, the National Nature Science Foundation of China (62121002, 62022076, 62232006).
References
[1] Han, T., W. Xie, A. Zisserman, Self-supervised co-training for video representation learning. In NeurlPS, pages 5679-5690. 2020.
[2] Chen, Y. Y. Tsai, M. Yang. End-to-en multi-modal video temporal grounding. NeurlPS, pages 28442-28453, 2021.
[3] Kim, M., H. Kwon, C. Wang, et al. Relational self-attention: What's missing in attention for video understanding. In NeurlPS, pages 8046-8059. 2021.
[4] Xie. C.-W., S. Sun, X. Xiong, et al. RA-CLIP: Retrieval augmented contrastive language-iman pre-training. In CVPR, pages 19265-19274 2023.
[5] Zhao, L., K. Zheng, Y. Zheng, et al. RLEG: vision-language representation learning with diffusion-based embedding generation. In ICML, pages 42247-42258. 2023.
[6] Jin, P., H. Li, Z. Cheng, al. Text-video retrieval with disentangled conceptualization and set-to-set alignment. In IJCAL. 2023.
[7] Ging, S., M. Zolfaghari, H. Pirsiavash, et al. Coot: Cooperative hierarchical transformer for video-text representation learning, In NeurlPS, pages
2020.
[8] Li, P., C.-W. Xie. L. Zhao, et al. Progressive spatio-temporal prototype matching for text-video retrieval. In ICCV, pages 4100-4110. 2023.
[9] Li, P., H. Xie, J. Ge, al. Dual-stream mowleegge-preserving hashing for unsupervised video retrieval. In ECCV, pages 181-197. 2022.
10

---

[10] Miech, J.-B. Alayrac, L. Smaira, et al. End-to-end learning of visual representations from uncurated instructional videos. CVPR, pages 9879-9889. 2020.
[11] Jin, P., J. Huang, F. Liu, et al. Expecation-maxxiizatton contrastive learning for compact video-and-language representations. In Neur/PS, pages 30291-30306. 2022.
[12] Jin, P., H. Li, Z. Cheng, et al. Diffusionret: Generative text-video retrieval with diffusion model. In ICCV. 2023.
[13] Jin, J. Huang, P. Xiong, et al. Video-text as game players: Hierarchical banzhaf interaction for cross-modal representation learning. In CVPR, pages 2472-2482. 2023.
[14] Li, P., H. Xie, Min, et al. Deep fourier ranking quantization for semi-supervised image retrieval. TIP, pages 5909-5922, 2022.
[15] Zhang, A., H. Fei, Y. Yao, al. Transfer visual prompt generator across llms. In NeurIPS. 2023.
[16] Li, P., Li, H. Xie, et al. Neighbothoood-ddapiie structure augmented metric learning. In AAAI, pages 1367-1375. 2022.
[17] Yuan, Y., T. Mei, Zhu. To find where you talk: Temporal sentence localization in video with attention based location regression. In AAAI, pages 9159-9166. 2019.
[18] Cao, M., T. Yang. J. Weng, et al. Loevtp: Video-text pre-training for temporal localization. In ECCV, vol. 13686, pages 38-56. 2022.
[19] Zhang, M., Y. Yang, X. Chen, et al. Multi-stage aggregated transformer network for temporal language localization in videos. In CVPR, pages 12669-12678. 2021.
[20] Soldan, M., A. Pardo, J. Alcázar, al. MAD: A scalable dataset for language grounding in videos from movie audio descriptions. In CVPR, pages 5016-5025. 2022.
[21] Kim, D., J. Park, J. Lee, et al. Language-free training for zero-shot video grounding. In WACV, pages 2539-2548. 2023.
[22] Moon, W., S. Hyun, S. Park, et al. Query-dependent video representation for moment retrieval and highlight detection. In CVPR. 2023.
[23] Gao, C. Sun, Z. Yang, al. Tall: Temporal activity localization via language query. ICCV, pages 5267-5275. 2017.
[24] Liu, M., X. Wang. L. Nie, et al. Attentive moment retrieval in videos. In SIGIR, pages 15-24. 2018.
[25] Anne Hendricks, L., O. Wang, E. Shechtman, et al. Localizing moments in video with natural language. In ICCV, pages 5803-5812. 2017.
[26] Chen, J., X. Chen, I.. Ma, et al. Temporally grounding natural sentence in video. In EMNLP, pages 162-171. 2018.
[27] Zhang. S., H. Peng, J. Fu, et al. Learning 2d temporal adjacent networks for moment localization with natural language. AAAI, pages 12870- 2020.
[28] Wang, Z., L. Wang, T. Wu, et al. Negative sample matters: A renaissance of metric learning for temporal grounding. In AAAI, pages 2613-2623. 2022.
[29] Zhao, Y., Z. Zhao, Z. Zhang, al. Cascaded prediction network via segment tree for temporal video grounding. In CVPR, pages 4197-4206. 2021.
[30] Zhang, H., A. Sun, W. Jing, et al. Span-based localizing network for natural language video localization. In ACL. pages 6543-6554, 2020.
[31] Zeng, R., H. Xu, W. Huang, et al. Dense regression network for video grounding. CVPR, pages 10287-10296. 2020.

---

[32] Yuan, Y., L. Ma, J. Wang, et al. Semantic conditioned dynamic modulation for temporal sentence grounding videos. NeurIPS, 32, 2019.
[33] Zhang, D., X. Dai, X. Wang, et al. MAN: moment alignment network for natural language moment retrieval via iterative graph adjustment. In CVPR, pages 1247-1257. 2019.
[34] Liu, D., X. Qu, Dong, al. Adaptive proposal generation network for temporal sentence localization in videos. In EMNLP, pages 9292-9301. 2021.
[35] Xu, H., K. He, B. A. Plummer, et al. Multilevel language and vision integration for text-to-clip retrieval. In AAAI, pages 9062-9069 2019.
[36] Xiao, S., L. Chen, J. Shao, et al. Natural language video localization with learnable moment proposals. In EMNLP, pages
2021.
[37] Lei, J.. T. L. Berg, M. Bansal. Qvhighlights: Detecting moments and highlights in videos via natural language queries. In NeurlPS. 2021.
[38] Liu, Y., S. Li, Y. Wu, al. UMT: unified multi-modal transformers for joint video moment retrieval and highlight detection. CVPR, pages 3032-3041. 2022.
[39] Otani, M., Y. Nakashima, Rahtu, et al. Uncovering hidden challenges in query-based video moment retrieval. In BMVC. 2020.
[40] Yang, X., Feng, W. Ji, et al. Deconfounded video moment retrieval with causal intervention. In SIGIR, page 1-10. 2021.
[41] Nan, G., R. Qiao, Y. Xiao, et al. Interventional video grounding with dual contrastive learning. In CVPR, pages 2765-2775. 2021.
[42] Gao, J., C. Xu. Fast video moment retrieval. In ICCV, pages 1523-1532. 2021.
[43] Wang, H., Z.-J. Zha, L. Li, et al. Structured multi-level interaction network for video moment localization via language query. In CVPR. 2021.
[44] Liu, D., X. Qu, J. Dong, et al. Context-aware biaffine localizing network for temporal sentence grounding. In CVPR, pages 11235-11244 2021.
[45] Zhang, H., A. Sun, W. Jing, et al. Towards debiasing temporal sentence grounding in video. arXiv preprint arXiv:2111.0432 2021.
[46] Zeng, Y., D. Cao, X. Wei, et al. Multi-modal relational graph for cross-modal video moment retrieval. In CVPR, pages 2215-2224, 2021.
[47] Mun, J., M. Cho, B. Han. Local-global video-text interactions for temporal grounding. In CVPR, pages 10807-10816. 2020.
[48] Sohl-Dickstein, J., A. Weiss, N. Maheswaranathan, al. Deep unsupervised learning using nonequilibrium thermodynamics. In ICML, vol. 37, pages
2015.
[49] Ho, J., A. Jain, P. Abbeel. Denoising diffusion probabilistic models. In NeurIPS. 2020.
[50] Austin, J., D. D. Johnson, J. Ho, al. Structured denoising diffusion models in discrete state-spaces. In NeurlPS, pages 17981-17993. 2021.
[51] Ho, T. Salimans, A. A. Gritsenko, et al. Video diffusion models. In NeurIPS. 2022.
[52] Gu, S., D. Chen, J. Bao, et al. Vector quantized diffusion model for text-to-image synthesis. In CVPR, pages 10686-10694 2022.
[53] Hao, S., K. Han. S. Zhao, et al. ViCo: Detail-preserving visual condition for personalized text-to-image generation. arXiv preprint arXiv:2306.0m 00971, 2023.
[54] Xue, 7., G. Song, Q. Guo, et al. Raphael: Text-to-image generation via large mixture of diffusion paths. arXiv preprint 11X.:22305..8295. 2023.
12

---

[55] Liu, Z., R. Feng, K. Zhu, et al. Cones: Concept neurons in diffusion models for customized generation. arXiv preprint arXiv:2303.05115, 2023.
[56] Chen, X., L. Huang, Liu, et al. Anydoor: Zero-shot object-level image customization. arXiv preprint arX1v:2307.09481, 2023.
[57] Zhao, S., D. Chen, Y.-C. Chen, al. Uni-ControlNet: All-in-one control text-to-image diffusion models. In NeurIPS. 2023.
[58] Wang, X., H. Yuan, S. Zhang, et al. VideoComposer: Compositional video synthesis with motion controllability. NeurIPS. 2023.
[59] Li, X. L., J. Thickstun, I. Gulrajani, et al. Diffusion-hi improves controllable text generation. In NeurlPS, 2022.
[60] Popov, V., Vovk, Gogoryan, et Grad-tts: diffusion probabilistic model for text-to- speech. In ICML, vol. 139, pages 8599-8608, 2021.
[61] Chen, S., P. Sun, Y. Song, et al. DiffusionDet: Diffusion model for object detection. In ICCV, pages 19830-19843. 2023.
[62] Liang, C., W. Wang, J. Miao, et al. GMMSeg: Gaussian mixture based generative semantic segmentation models. In NeurIPS, 2022.
[63] Simonyan, K., A. Zisserman. Very deep convolutional networks for large-scale image recogni- tion. In ICLR. 2015.
[64] Tran, D., L. D. Bourdev, R. Fergus, et al. Learning spatiotemporal features with 3d convolutional networks. ICCV, pages 4489-4497, 2015.
[65] Radford, A., J. W. Kim, C. Hallacy, et al. Learning transferable visual models from natural language supervision. In ICML, vol. 139, pages 4748-8763 2021.
[66] Feichtenhofer, C.,, H. Fan, J. Malik, et al. Slowfast networks for video recognition. In ICCV, pages 6201-6210. IEEE, 2019.
[67] Kong, Q., Y. Cao, T. Iqbal, al. Panns: Large-scale pretrained audio neural networks for audio pattern recognition. IEEE Trans Audio Speech Lang Process, 2020.
[68] Pennington, J., R. Socher, C. D. Manning. Glove: Global vectors for word representation. In EMNLP, pages 1532-1543. 2014.
[69] Song, J., C. Meng, S. Ermon. Denoising diffusion implicit models. In ICLR. 2021.
[70] Vaswani, A., N. Shazeer, N. Parmar, et al. Attention is all you need. In NeurIPS. 2017.
[71] Chen, T., R. Zhang, G. E. Hinton. Analog bits: Generating discrete data using diffusion models with self-conditioning. In ICLR. 2023.
[72] Carion, N., F. Massa, G. Synnaeve, et al. End-to-end object detection with transformers. In ECCV, vol. 12346, pages 213-229. 2020.
[73] Rezatofighi, H., N. Tsoi, Gwak, et al. Generalized intersection over union: A metric and loss for bounding box regression. In CVPR, pages 658-666. 2019.
[74] Regneri, M., M. Rohrbach, D. Wetzel, et al. Grounding action descriptions in videos. TACL, 1:25-36, 2013.
[75] Gao, J., X. Sun, M. Xu, et al. Relation-aware video reading comprehension for temporal language grounding. In EMNLP, pages
2021.
[76] Chen, T., S. Kornblith, M. Norouzi, al. simple framework for contrastive learning of visual representations. In ICML. 2020.
[77] Kingma, D. P., J. Ba. Adam: A method for stochastic optimization. In ICLR. 2015.
13

---

[78] Sedriiuuz-Opazzz C., E. Marrese- Marrese-Taylor, B. Fernando, et al. DORi: Discovering object
relationships for moment localization of natural language query video. In WACV. 2021.
[79] Lu, C., L. Chen, C. Tan, et al. Debug: dense bottom-up grounding approach for natural language video localization. In EMNLP. 2019.
[80] Escorcia, V., M. Soldan, Sivic, et al. Temporal localization of moments in video collections with natural language. arXiv preprint
2019.
[81] Lei, J., L. Yu, T. L. Berg, et al. TVR: A large-scale dataset for video-subtitle moment retrieval. In ECCV, pages 447-463. 2020.
[82] Yuan, Y., X. Lan, X. Wang, al. A closer look at temporal sentence grounding in videos: Dataset and metric. In HUMA. 2021.
[83] Krishna, R., K. Hata, F. Ren, et al. Dense-captioning events in videos. In ICCV, pages 706-715. 2017.
[84] Paszke, A., S. Gross, F. Massa, et al. Pytorch: An imperative style, high-performance deep learning library. In NeurlPS, 2019.
14

---

This supplementary material provides more details of our MomentDiff framework:
1. Implementation details.
2. Inference efficiency of MomentDiff.
3. More experiment results.
4. Broader impacts.
A Implementation details
A.1 Datasets
Public datasets. Charades-STA 23 serves benchmark dataset for the video moment retrieval task and is built upon the Charades dataset, originally collected for video action recognition and video captioning. The Charades-STA dataset comprises 6,672 videos and 16,128 video-query pairs, allocated for training (12,408 pairs) and testing (3,720 pairs). On average, the videos in this dataset have duration of 29.76 seconds. Each video annotated with an average of 2.4 moments, with each moment lasting approximately 8.2 seconds. QViighlghhts [37] contains 10, 148 videos, each 150 seconds long and annotated with at least one text query describing its relevant content. These videos are from three main categories, daily vlogs, travel vlogs, and news events. On average, there are approximately 1.8 non-overlapping moments per query, annotated on non-overlapping clips. The dataset contains a total of 10,310 queries with 18,367 annotated moments. The training set, validation set and test set include 7,218, 1,550 and 1,542 video-text pairs, respectively. TACoS (74 is compiled specifically for video moment retrieval and dense video captioning tasks. It is comprised of 127 videos that depict cooking activities, with an average duration of 4.79 minutes. TACoS contains a total of 18,818 video-query pairs. In comparison video segments that are temporally annotated with queries per video. On average, each video contains 148 queries. Additionally, the TACoS dataset is known for its difficulty, the queries it contains limited to only a few seconds or even just a few frames. To ensure impartial comparisons, we use the same dataset split B1] which consists of 10,146, 4,589, and 4,083 video-query pairs for the training, validation, and testing sets, respectively. ActivityNet-Captionn [83 comprises of 20,000 videos and 100,000 descriptions that encompass wide range of contexts. Similar (27]. we designate val 1 as the validation set and val 2 as the testing set. The dataset contains 37,417, 17,505, and 17,031 pairs of moments and corresponding sentences for training, validation, and testing respectively.
the Charades-STA dataset, TACoS has more
Table 7: Training and test sets two anti-bias datasets.
Charades-STA--Ln
Charades-STA-Mom
Dataset
Wo S 10s
We > 10s
Total
Co + wo/2 < 15s
Cy -wo/2 > 15s Total
Training
9307
2326
11633
5330
1332
6662
Test
197
788
985
259
1038
1297
Anti-bias datasets. To explore the location bias problem, construct two anti-bias datasets with location distribution shifts based on the Charades-STA dataset. In video moment retrieval, length and position important parameters of spans.
Therefore, we first investigate the effect of span length wo- As shown in Tab. 7 in the training set of Charades-STA, STA-Len, collect 9,307 video-text pairs with span length wo S 10s and 2.326 video-text pairs with wo > 10s, accounting for 80% and 20% the total training set. In contrast, in the test set, select 197 video-text pairs with wa < 10s and 788 video-text pairs with w, > 10s, accounting for 20% and 80% of the total test set.
Then, we design the dataset Charades- Baarades-STA-Merr STA-Mom based on the span's end time + wo/2 2 and start time Co wo/2 /2 the training set of Charades-STA-MAr
we collect 5,330 video-text pairs with Co + w < 15s and 1,332 video-text pairs with wo/2 > 15s, accounting for 80% and 20% of the total training set. contrast, the test set, we select 259 video-text pairs with Co + wo/2 S 15s and 788 vidco-text pairs with Co wo/2 2 15s, accounting for 20% and 80% of the total test set.
15

---

Algorithm 1: MomentDiff Training in PyTorch-like style.
# Video features: v_feats E ND
# Text features: t_feats RMXD
# Ground truth spans: gt_spans [
cupha_cu (m) cumulative product of ai
# Fully-connected layer: FC()
Video Noment Denoiser: VMD()
def train (v_feets,, t_feats, gt_spans):
Similarity- aware Condition Generator f f_feats = (c (cG(v_eets, t_feats) # x x D
# Span normalization
ps pad_spans (gt_ (gt_spans) Pad gt_spans to [Nr,
ps = (ps 2 -1) * Signal scaling m randint M) Noise intensity noi = normal (mean=0, std=1) Noise ps_m sqrt( cumprodd ps +
sqrt alpha_cumppail (m)) noi # Noisy span ps_m (ps_m/A +1) Normalization
Span embedding ps_emb_r m FC(ps_m)
Intensity-aven attention
output_m VMD(ps_emb_m, m, f_feats)
Output embedding
# Denoising training
hat_ps_m Span_pred (output_m)
Predicted span
hat_cs Score_predd (output_m)
# Confidence score
Computing loss
loss = loss_simme (f_feats) loss_vmi vmr (hat_ps_m, hat_cs_m, gt_spans)
return loas
A.2 Pseudo Code of MomentDiff
Algorithm provides the pseudo-code of MomentDiff Training in PyTorch-like style.
The inference procedure of MomentDiff is a denoising sampling process from noise temporal spans. Starting from spans sampled in Gaussian distribution, the model progressively refines its predictions, as shown in Algorithm
B Inference Efficiency of MomentDiff
Inference time. Inference efficiency is critical for machine learning models. We test 2DTAN (27). MMN (28), MomentDETR [37] and MomentDiff on the Pytorch framework in Tab. We test all models with one NVIDIA Tesla A100 GPU.
and MMN 28 MomentDiff (Step=1) not only achieves the best results on recall, but also improves the inference speed by 5-7 times. This is because 2DTAN and MMN predefine a large number of proposals, which may be redundant and increase computational overhead. Compared to MomentDETR (37, MomentDiff (Step=10) achieves better results with similar inference time. The possible reasons are that we adopt fewer random spans, very simple network structures, and avoid post-processing.
Compared with 2DTAN
16

---

Algorithm 2: MomentDiff inference in PyTorch-like style.
Video features: v_feats E ND
# Text features: t_feats RMXD
# Video Moment Denoiser: VMD()
def (est(v_Ba) (v__fats,,, t_feats, sampling.nun)
# Similarity-awar Condition Generator
f_feats SCG(v_feats, t_feats) # N₂ D
Noisy span: 2
ps_m normal (mean=0, std=1)
# uniform sample intensity ceversed(Linesacee M, sampling_num)
[(M-1, M-2), (M-2, M-3), (1, 0), (0, -1)]
intensity_pairs list (iit(iii(intensity/ [1-1], intensity[1:]
for intensity_nov _nw, intensity_next in zip(intensity.paizz
# predict ps_0 from ps_m
output_m (ps_m f_feats, intensity.now)
Predicted span
hat_ps_m (pa__ppte
Update ps_m
ps_m = ddim_update (hat_ps_m, ps_m, intensity_nov, intensity_next)
Confidence score
_a__ Score pred
return hat_ps_m, hat_cs_m
Table 8: The inference time of 2DTAN [27], MMN [28], MomentDETR [37 and MomentDiff on Charades-STA with VGG video features and Glove text features. We report RI @0.5, RI @0.7 and MAP aug. Default settings marked in blue
Charades- STA
Inference time (second)
Method
RI @1@0.5
R1 @0.7
MAP aeg
2DTAN 27
41.34
23.91
29.26
42.
MMN 28
46.93
27.07
31.58
53.42
MomentDETR 37
50.54
28.01
29.87
12.42
MomentDiff (Step=1)
49.17
26.39
29.12
7.56
MomentDiff (Step=2)
50.81
27.84
31.27
8.23
MomentDiff (Step=10)
52.36
28.08
31.75
11.01
MomentDiff' (Step=50)
51.94
28.25
31.66
20.74
MomentDiff (Step=100)
52.21
28.84
31.01
34.35
C More Experiment Results
C.1 Experiment on ActivityNNe-Captions
The results of our method on ActivityNet-Capiine are shown in Tab. 9 training time and the inference time for test samples in one NVIDIA Tesla A100 GPU. Our model has been improved compared with baseline (Momen(DETR), Compared to SOTAs, we still have competitive results. Compared with MMN, our method 25 times faster training time and 7.24 times faster in testing time.
We record the epoch
17

---

Table 9: Performance comparisons (%) ActivityNee--aatiins with C3D video features and Glove
text features.
Method
@I00.33
@1@0.5
@0.7
MAP cvg
Training time
Inference time
2DTAN
59.92
44.63
27.53
27.26
1.1h
523.74g
MMN
28
65.21
48.26
28.95
28.74
1,5h
662. 128
MomEntDETR 37
61.87
43.19
25.74
25.63
0.05h
52.72s
MomentDiff
62.79
46.52
28.43
28.19
0.06h
91.43s
VGG, Glove
C3D, Glove
SF+C, C
70
50
30
RI@0.5
R @l0.0.7
MAP@0.5
MAP@0.75
MAP __aag
Figure 6: Performance fluctuations (%) corresponding different features and multiple random seeds on the Charades-STA dataset.
C.2 Error bars
Fig. shows the performance fluctuation of the model on the Charades-STA dataset. We different random seeds (seed= 2023, 2022, 2021, 2020, 2019) and different features (VGG, Glove; C3D, Glove; SF+C, C:) organize experiments. This shows that the model always converges and achieves stable results for different initializations. This phenomenon demonstrates the ability of the model to learn to generate real spans from arbitrary random spans.
C.3 Ablation study
Different sampling strategies. Denoising Diffusion Probabilistic Models (DDPM) 49 and Denois- ing Diffusion Implicit Models (DDIM) [69] are popular and classic diffusion models. We show the results of both strategies in Tab. ao We find that DDIM and DDPM perform similarly, but DDIM samples faster. Therefore we adopt DDIM as the default technology.
Effect of the schedule of B. The schedule of B3 determines the weighting ratio different intensity b)) we find that the cosine schedule works better in experiment. The cosine schedule makes the noisy spans change slowly at the beginning and end during the diffusion process, and the generation effect is more stable. So we the cosine schedule as the default.
noises. In Tab. b)]b),
post-processing technique in DiffusionDet [6I. Tab. c) c) shows that Box Renewal can indeed slightly improve the results. To keep the inference process as simple as possible, we do not use Box Renewal by default.
Effect of the Box Renewal. Box Renewal
Effect of the batch size. As shown in Tab. 10 d), we set the batch size to 32 to achieve the best results.
Effect of the layer number. In Tab. T0je), we show the effect of the layer number of Similarity-aware Condition Generator (SCG) and Video Moment Denoiser (VMD). The default setting (2+2:2): SCG contains 2 cross-attention and 2 self-attention layers and VMD contains 2 cross-attention layers. The default setting works best.
Effect of the weight value on Lzim- We set the weight values of Leim and Lemr to and 1, respectively. Keeping the weight value of Lumr unchanged, organize the weight influence experiment of Luims as shown in Tab. f). The default setting works best.
Effect of the number of spans on R1 0.5. Our training and inference are decoupled. Our simple framework allows us input any number of random noises. Tab. TO gg) shows that there is slight improvement when testing with more noise boxes.
18

---

Table 10: Ablation study (%) on the Charades-STA dataset with SF+C video features and CLIP text features. We report RI @0.5, RI @0..7 and MAPavy Default settings are marked in blue.
(a) Different sampling strategies.
(b) Effect of the schedule of [3.
Sampling
R @100.5
RI @.07
MAP
Schedule
R1 @1.55.
R1 @0.7
MAPavy
DDPM 499
55.62
32.37
32.48
Linear
54.76
31.43 32.42
31.59
DDIM 69
55.57
32.42
32.85
Cosine
55.57
32.85
(d) Effect batch size.
(c) Effect of the Box Renewal [61].
batch size
RI@0.5
R1@0.7
MAP avy
Schedule
@0055
RI @0.7
MAP ang
52.42
30.81
30.14
w/o Box Renewal 55.57
32.42 32.64
32.85
55.57
32.42
32.85
w/ Box Renewal 56.03
33.18
64
53.78
32.25
31.93
Effect of the layer number.
(f) Effect of the weight value Caim-
SCG:VMD
@I@0.5
@00.7
MAPavy
Weight
@.055
RI @0.7
MAP ava
1+1:1
51.74
28.97
29.82
52.34
29.63
29.98
2+2:2
55.57
32.42
32.85
53.74
30.82
31.15
343:3
54.39
32.16
32.54
55.57
32.42
32.85
(g) Effect of the number of spans on RI @0.5.
test
3
5
10
20
train
50.83
50.91
50.97
50.87
50.82
3
53.98
54.12
54.19
54.23
54.21
5
55.36
55.41
55.57
55.69
55.51
10
53.71
53.84
53.86
53.89
53.93
20
53.16
53.38
53.40
53.44
53.42
D Broader Impacts
First, our work does not involve private data. Second, we believe that AI is a double-edged sword, and our model is no exception. For example, when users websites use model, only natural language is needed to locate video moments and collect desired video material, which improves the productivity of society. However, this may have negative impact if the natural language entered by the user contains words related to violence, pornography, etc. We will consider these scenarios and implement secure VMR model.
19


Reviews:
[{"review_id": "UWFpaKIDcF", "reviewer": "Reviewer_oTs9", "summary": "This paper first proposes a diffusion model for video moment retrieval to overcome proposal-based moment retrieval and distribution-specific methods.", "questions": "I want to get answers about the weaknesses above in the rebuttal.", "limitations": "See in the weakness.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "[+] First work bridging generative frameworks into deterministic task as video moment retrieval.\n[+] Illustrative presentation\n[+] writing is easy to follow", "weaknesses": "[Motivation]\n[1] There are many solutions (i.e. 2DTAN, VSLNet) without relying on moment proposals. Fully-supervised VMR does not concern about proposal generation, as it is frame-level supervision is available, where the previous works already design the regression-based (i.e., Attention Based Localization Regression) methods.\n\n[2] As this paper proposes a new framework (diffusion framework) for the VMR, what the previous frameworks of VMR are relying on the distribution-specific proposals? This paper refers to the 2DTAN, the framework of 2D map with respect to start-time and end-time can allow all the possible moments, where the framework is not relying on the distribution-specific bias. I think the authors may understand the applying the masking in the map as resulting distribution-specific proposals, which is more related to the heuristic filtering by empirical experiments, not a bias problem.\n\n[3] Why the diffusion (generative framework) more can be better than the deterministic models? In fact, the authors' proposed method is presented to overcome the proposal-learnable methods, however, current VMR method is not relying on proposals. (rather, the weakly-supervised method is dependent on proposals). Furthermore, location bias problem is sourced from dataset distributions, proposed diffusion framework does not correlate to mitigating bias.\n\n[Method]\n\n[1] Preliminary section is recommended in the paper or appendix about the forward-backward process of diffusion framework (e.g., denoising diffusion probabilistic model, conditional diffusion, sampling) to enhance the readability.\n\n[2] x_{0} is a 2-dimensional vector of center point and width. Does this paper truly add Gaussian noise on to the 2 values and denoise them?\n\n[Experiment]\n\n[1] I can not trust the performances in Table. Is there any experimental qualitative evidence why denoising frameworks can guarantee more performance than previous work?\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments! We hope our answers can address you concerns.\n\n[Motivation]\n\nThank you for your very professional review. \nIn fact, our understanding is basically the same, but there are two points that need to be clarified before solving your doubts.\n\n**Definition of proposal.** There are indeed many methods that do not require proposal generation. However, the proposal we refer to is a broad concept, including dense anchors in 2DTAN and learnable queries in MomentDETR. Like [1], we regard both 2DTAN and MomentDETR as  implicit proposal-based methods.\n\n[1] CONE: An Efficient COarse-to-fiNE Alignment Framework for Long Video Temporal Grounding\n\n**Promising DETR-based VMR framework.** Recent VMR methods are mainly divided into three types: dense proposal-based methods (e.g., 2DTAN, MMN), regression-based methods (e.g., ABLR, VSLNet) and DETR-based methods (e.g., MomentDETR, UMT). The disadvantage of dense proposal-based methods is that there is a large amount of proposal redundancy and the calculation speed is slow (refer to Q1's reply in Reviewer uAtb). Regression-based methods can alleviate this problem, but the results are often not good enough. Recently, DETR-based methods have weighed efficiency and performance. MomentDETR exceeds MMN 7 times in terms of inference speed (see Table 2 in Supplementary Material), and it also exceeds many regression-based methods in terms of performance.\nFor example, MomentDETR can achieve R1@0.5 50.49 in Charades-STA, surpassing VSLNet(48.67) [18].\nTherefore, we further explore VMR tasks based on DETR, which is a very promising genre.\n\n**Q1 many proposal-free methods.**\nThe emerging DETR-based VMR method is an important genre, and we are solving the location bias problem existing in the DETR series.\n\n**Q2 motivation about 2DTAN.** \nSorry for the misunderstanding.\nWe do not think that 2DTAN suffers from severe location bias problems.\nWe introduced 2DTAN mainly to illustrate the shortcomings of dense proposals:\nthese methods have a large redundancy of proposals and the numbers of positive and negative proposals are unbalanced.\n\nBesides, in Table 2 in the supplementary material, we also show the inference efficiency comparison between 2DTAN (42.18s) and our method (20.74s). \nThis proves that dense proposals do affect model efficiency.\n\n**Q3.1 & Q6 why can the generative framework reach SOTA results? Qualitative analysis?** \nWe improved based on MomentDETR. Both MomentDETR and UMT have SOTA performance, refer to Table 1 in our manuscript. \nSpecific model differences include iterative denoising paradigm and training details. \n1. Iterative denoising paradigm. In Table 5(c) w/o VMD, we find that removing the denoiser for model optimization, the final result drops by 8.88% in R1@0.5. According to Table 5(f), the result (53.31) of only 1-step denoising is about 2% lower than the best result (55.62). \nWhen the model performs 1-step denoising, it is actually similar to MomentDETR.\nThe above results prove the effectiveness of the iterative denoising paradigm.\n\n    **Qualitative analysis.** In Figure 5 of our manuscript, we visualize the prediction results of MomentDETR and MomentDiff. **In Figure 1 of the rebuttal pdf file, we further present the results generated by our method in a single step.** The result of single-step generation still has a certain deviation from the correct video segments, especially the example on the right of Figure 1. Through continuous iterative diffusion denoising, our model can finally obtain better results than MomentDETR.\n\n\n2. Encoder and loss functions. \nCompared with MomentDETR, we use additional cross-attention layers as encoder, which strengthens modality information interaction. We use point cross-entropy loss, which focuses on finer-grained and comprehensive positive and negative video frames.\n\nDefinitely, we will present the above results, analysis and codes in the revised version.\n\n\n**Q3.2 reason about alleviating bias:**\nA simple idea is to solve it at the data level, which can be sampled from biased data sets or constructed with more evenly distributed data for training.\nBut we hope to solve the problem from a deeper and essential perspective, that is, the model itself.\nSpecifically, learnable queries in MomentDETR may tend to focus on video segments where locations in the dataset occur more often.\nTherefore, we directly replace the queries with data-independent random noise, which can alleviate the above-mentioned bias to a certain extent.\nIn addition, more experimental results can refer to the reply of reviewer uAtb Q2. We achieve impressive performance on the Out of Distribution (OOD) evaluation dataset (Charades-CD and ActivityNet-CD).\n\n\n\n[Method]\n\n**Q4 preliminary section about diffusion process.** \nThanks for your constructive suggestions. \nIn the supplementary material, we have given the pseudo codes of diffusion training and inference in Algorithm 1 and 2 for the convenience of readers.\nTo make it clearer, we will add the basic background, flowchart and more details of common diffusion models in detail in the appendix.\n\n**Q5 add noise to the span.** \nYes，we add Gaussian noise according to the following formula:\n$\\boldsymbol{x}_m=\\sqrt{\\bar{\\alpha}_m} \\boldsymbol{x}_0+\\sqrt{1-\\bar{\\alpha}_m} \\boldsymbol{\\epsilon}_m$ and code:\n```python\n #python3\nnoise = torch.randn(self.query_num, 2).cuda() \nsqrt_alphas_cumprod_m = extract(self.sqrt_alphas_cumprod, m, x_0.shape) \nsqrt_one_minus_alphas_cumprod_m = extract(self.sqrt_one_minus_alphas_cumprod, m, x_0.shape)\nx_m =  sqrt_alphas_cumprod_m * x_0 + sqrt_one_minus_alphas_cumprod_m * noise\n```\nIn addition, the denoising process is inspired by the diffusion model that can exploit random noise to generate images in specified semantics.\nWe generate noise into the temporal spans corresponding to the query semantics, which can be fully realized through the guidance of video-text fusion embeddings and appropriate loss constraints. \nWe will publish code and models."}, {"review_id": "vOe6y7a7nR", "reviewer": "Reviewer_KdBK", "summary": "This paper proposes a novel generative approach, MomentDiff, to address the Video Moment Retrieval (VMR) task. It replaces traditional dense or learnable proposals with random spans and a diffusion-based denoiser to refine predictions, mimicking the human process of identifying key video moments. This reduces the impact of temporal location biases and improves the system's generalizability. The authors also introduce two \"anti-bias\" datasets, Charades-STA-Len and Charades-STA-Mom, for evaluation. Experiment results showed that MomentDiff outperforms existing methods in efficiency and transferability.", "questions": "1. Clarification on Visual and Textual Representations: It would be helpful if the authors could elaborate on why they chose the specific visual and textual extractors, like VGG, C3D, CLIP, Glove, etc. Are there specific reasons these were chosen over other potential extractors?\n2. Elaboration on Span Generation Process: In the span generation process, it is mentioned that for the same video, the correct video segments corresponding to different text queries are very different. Could you elaborate more on this? Is there a way to address this challenge?\n3. Justification for Hyperparameter Choices: Could the authors provide further clarification on the selection of the hyperparameters used in the model? How were these optimized, and what was the impact on model performance?\n4. Scalability of the Model: Could the authors discuss how this model scales with larger, more complex datasets? Can the method efficiently handle real-world scenarios with high volumes of data, and if so, are there any limitations or performance degradation?\n5. Use of Pre-Trained Models: What are the implications of using several pre-trained models? How does it affect the generalizability of the proposed method across diverse datasets, especially ones that differ significantly from the datasets these pre-trained models were trained on?\n6. Computational Resources: Could the authors provide details about the computational resources required for the model to run both in the training and inference stages? This is crucial for evaluating the practicality of the proposed model.\n7. No motivation is provided in Similarity-aware Condition Generator, i.e., why specific modality features are selected as Query, Key, and Values? Why not any other combination? \n", "limitations": "1. Authors have provided limited limitations.\n2. Code is not provided in supplementary that can help with in more detailed understanding. \n", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed method creatively combines pre-trained video and text backbones for feature extraction, a similarity-aware condition generator, and a video moment denoiser. This composite approach takes existing tools and blends them in a unique way. The inclusion of audio data as a feature, alongside visual and textual data, also represents an innovative approach to video moment retrieval.\n2. The paper showcases a high-quality approach by incorporating various feature extractors, utilizing a multilayer transformer for multimodal interaction, and deploying a similarity-aware fusion embedding. The fact that the paper also discusses the limitations of the proposed method speaks to its quality and rigor.\n3. The proposed methodology is outlined clearly and in a structured manner. Each part of the system, from feature extraction to the inference process, is explained with sufficient detail. However, some areas could benefit from additional explanation (e.g., the impact of the quality of fusion embeddings on the denoising process), which could further enhance clarity.\n4. The paper tackles the important problem of video moment retrieval, which has broad implications in fields like media indexing, recommendation systems, and video summarization. The solution proposed in the paper, especially with the inclusion of audio features, can be significant in improving the efficiency and effectiveness of video moment retrieval tasks. By outlining its method clearly and discussing potential limitations, the paper contributes to further research and improvement in the field.", "weaknesses": "1. The proposed method relies heavily on the effectiveness of the chosen feature extractors. Although they have tested multiple feature extraction models, the paper does not discuss the impact of these choices on the final results in detail. Additionally, the models chosen for feature extraction could potentially limit the generalizability of the approach to datasets significantly different from those on which the models were trained.\n2. The paper does not provide a clear comparison with existing methods in terms of computational resources. This makes it hard to gauge the improvement the proposed method offers over current techniques.\n3. The paper mentions multiple hyperparameters but does not discuss how they are selected or tuned. This could impact the replicability and robustness of the model across different datasets.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1 the reason and impact of extractors.**\nThank you for suggestions.\nWe chose these extractors for two reasons:\n1. We want to prove that our method is general to different types of extractors, so different models are used: 1)2D encoder, VGG. 2)3D encoder, C3D. 3)Cross-modal encoder, CLIP.\nMany methods only use a single extractor.\n2. The extractors involved in the VMR task include VGG, C3D, CLIP. For the fairness of the comparison, we use the consistent pre-trained models.\n\nThe impact of different feature extractors:\n\nWe agree with your profound point: different pretrained models generalize differently on downstream tasks. \nVGG, C3D or CLIP are pre-trained on ImageNet, Sports-1M, or 4B image-text pairs respectively.\nHowever, since our video data is quite different from the above datasets, the impact of pre-training model may not lie in whether the data has been seen, but in the representation ability of pre-training model itself.\nVGG is an early image backbone, and its representation ability in VMR may be weaker than that of C3D and CLIP. \nTherefore, on Charades-STA, VGG is not as effective as C3D or CLIP.\n\nWe will emphasize the impact of model choices in the revised version.\n\n**Q2 computing resources in training and testing.**\nWe count the training time (Tr) of an epoch and the inference time (In) on Charades-STA with VGG features.\n\nCharades-STA:\n|Method|R1@0.5|R1@0.7|Tr|In|  \n|:---|---:|---:|---:|---:|\n|MMN [16]|46.93|27.07|479.63s|53.42s| \n|MomentDETR [25]|50.54|28.01|40.74s|12.42s|\n|Ours-step 1|49.17|26.39|48.12s|7.56s|  \n|Ours-step 2|50.81|27.84|48.12s|8.23s|\n|Ours-step 10|52.36|28.08|48.12s|11.01s|\n|Ours-step 50|51.94|28.25|48.12s|20.74s|\n\n1. Compared with MMN, MomentDiff (Step=2) achieves better results and improves training speed by 10 times and inference speed by 7 times.\nBecause MMN predefines lots of proposals, which increase computational overhead.\n2. All experiments are performed on one A100 GPU.\nMemory usage is correlated with number and dimension of features . \nFor VGG, C3D or SF+C, we extract video features every 1/6s, 1s or 1s on charades-STA.\nWe use VGG (4096-d), C3D (500-d), or SF+C (2816-d) features with 40.1G, 3.86G, or 4.02G memory. \nEven using C3D features, MomentDiff outperforms SOTA models.\nWe can flexibly select extractors according to our own resources.\n\n**Q3 the impact of hyperparameters.** \nThank you for good suggestion. \nFirst, we provide the ablation study about scale factor, span number and batch size in Table 5 of our main paper and Table 3 of supplementary material.\n\nNext, we show more results on Charades-STA (SF+C features), including the loss weight $\\lambda_{L1},\\lambda_{iou},\\lambda_{ce}$ and the weights $\\lambda_{L_{sim}}$ of $L_{sim}$, Transformer layer number.\n\n1. We show the results of different weights and $[\\lambda_{L1}=10,\\lambda_{iou}=1,\\lambda_{ce}=4, \\lambda_{L_{sim}}=4]$ is best.\n\n| $\\lambda_{L1}$ | $\\lambda_{iou}$ | $\\lambda_{ce}$ |  $\\lambda_{L_{sim}}$ |R1@0.5 | R1@0.7| $MAP_{avg}$|\n|:---|---:|---:|---:|---:|---:|---:|\n|10|1|4|4|55.57|32.42|32.85\n|10|1|4|2|55.03|31.94|32.22\n|10|1|4|1|54.36|31.14|31.09\n|5|1|4|4| 53.74|30.25| 30.48|\n|10|2|4|4|55.42|32.29|32.57|\n\n2. We show the effect of the layer number of Similarity-aware Condition Generator (SCG) and Video Moment Denoiser (VMD).\nThe default setting: SCG contains 2 cross-attention and 2 self-attention layers and VMD contains 2 cross-attention layers.\nThe default setting works best:\n|SCG|VMR|R1@0.5|R1@0.7|$MAP_{avg}$| \n|:---|---:|---:|---:|---:|\n|1+1|1|51.74|28.97|29.82|\n|2+2|2|55.57|32.42|32.85|\n|3+3|3|54.39|32.16|32.54|\n\n**Q4 elaboration on Span Generation Process.**\nIn datasets, there are multiple segment-text pairs for the same video. \nThe content of text is different, it will correspond to different segments.\nWe designed two solutions to this issue:\n1. We use cross-attention layers in SCG. The video and text in the cross-attention layers are fully interacted, so that the model can perceive different video-text pairs.\n2. We design $\\lambda_{L_{sim}}$ to optimize the fusion embedding $F$.\nNote that $F \\in \\mathbb{R}^{N_v * d}$, where N_v is the number of frames. \nThe purpose of loss is to pay attention to the frames within ground truth spans.\nIn this way, for different segment-text pairs of the same video, the model can focus on positive frames to generate accurate spans.\n\nWe will add explanations.\n\n**Q5 using complex datasets.** \nThanks for valuable question. We organize experiments on ActivityNet Captions, which contains complex activities.\nThere are 37417, 17505, and 17031 pairs for training, validation, and testing. The results on ActivityNet Captions (C3D features):\n|Method|R1@0.3|R1@0.5|R1@0.7|$MAP_{avg}$|\n|:---|---:|---:|---:|---:|  \n|MomentDETR|61.87|43.19|25.74|25.63| \n|Ours|62.79|46.52|28.43|28.19|  \n\nCompared with baseline, our method still has a certain improvement.\n\nTo verify performance in real world, we refer to the Out of Distribution (OOD) division on ActivityNet-CD[1]. The results are:\n|Method|R1@0.3|R1@0.5|R1@0.7|$MAP_{avg}$|\n|:---|---:|---:|---:|---:|  \n|MomentDETR|39.98|21.30|10.58|12.19|\n|Ours|45.54|26.96|13.69|16.38|\n\nThe results prove model robustness.\nWe will add results in the revised version.\n\n[1] A Closer Look at Temporal Sentence Grounding in Videos: Dataset and Metric.\n\n**Q6 why specific modality features are selected as Q, K, V?**\nBecause the task purpose is to locate the start and end positions.\nTherefore, we hope that the features after multi-modal fusion are frame-level features, e.g., $ F \\in \\mathbb{R}^{N_v * d}$, which is convenient for frame-level loss design based on ground truth span.\n\nAccording to the attention formula in Transformer, the dimension of output embedding is consistent with the dimension of Query.\nTherefore, we need to design video features $ V \\in \\mathbb{R}^{N\\_v * d} $ as Query and text features $ T \\in \\mathbb{R}^{N_t*d}$ as Key and Value. \nIf $T$ is Query, $F$ is word-level, which may not be suitable for video to locate spans."}, {"review_id": "2qVWN5VSZQ", "reviewer": "Reviewer_2CFW", "summary": "This paper first tackles video moment retrieval from a generative perspective, and proposes a novel framework called MomentDiff based on recently proposed technique Diffusion Models. MomentDiff can generate correct results from random spans, which can resist the temporal location biases. The experiments on three public datasets and two anti-bias datasets proposed by the authors demonstrate the effectiveness of MomentDiff.", "questions": "See the Weaknesses. ", "limitations": "The author points out that too many iterations will slow down the inference speed, and the solution is to reduce the number of iterations and make a trade-off between performance and speed. It is recommended that the author give the iteration round parameters on all datasets to show better trade-off.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThe generative perspective for video moment retrieval is novel.\n2.\tThe designed MomentDiff is effective for the location bias problem and easy to reproduce.\n3.\tExperimental results on three public datasets and two anti-bias datasets demonstrate the effectiveness of the proposed method.\n", "weaknesses": "1.\tFrom a generative perspective, traditional generative models like GANs can also be applied to the video moment retrieval task. Do the authors believe that these methods can be used, and if so, what is the difference between GAN and Diffusion models in this task? If not, please provide a reason.\n2.\tIt is unclear how other methods have solved the problem of location biases. It would be helpful for the authors to compare and contrast the advantages of the proposed method with existing solutions. \n3.\tThe total loss function is missing. Please clarify whether both loss functions L_{sim} and L_{vmr} are weighted 1.  \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your constructive comment. We answer your questions point by point.\n\n**Q1 Using GAN in VMR.**\nWe tend to use diffusion models instead of GANs to build generative VMR frameworks for three reasons:\n1. The VMR task requires the model to locate fine-grained moments, so a single-step generative model like GAN may not work well.\nIn our experiments, the effect of MomentDiff at step=1 is significantly lower than that of subsequent iterative generation results.\n2. In addition, the diffusion model is very flexible and easy to calculate, we can choose any number of spans and steps to solve the VMR task.\n3. The training process of GAN is difficult and unstable. \n\n**Q2 existing methods about location bias.**\nTo remove the harmful location bias, DCM [1] first disentangles the moment representation and applies causal intervention on the multimodal model input to remove the confounding effects of moment location.\n[2] samples from the training set and constructs the uniform dataset, and reduces the gradient of biased samples to achieve an unbiased model.\nWe propose a new diffusion VMR scheme from a generative perspective, which mitigates location bias by replacing learnable queries with random noise.\nWe believe that generative methods will bring new insights to the field.\nWe will carefully emphasize these methods in related work.\n\n[1] Deconfounded Video Moment Retrieval with Causal Intervention --SIGIR 2021\n\n[2] Towards Debiasing Temporal Sentence Grounding in Video --arXiv:2111.04321\n\n**Q3 Weight value on $L_{sim}$ and $L_{vmr}$.**\nWe set the weights of $L_{vmr}$ and $L_{sim}$ to 1 and 4, respectively. Keeping the weight of $L_{vmr}$ unchanged, we organize the weight influence experiment of $L_{sim}$ on Charades-STA (VGG features), as shown in the following table:\n| $\\lambda_{L_{sim}}$ | R1@0.5 | R1@0.7 | MAP@0.5 | MAP@0.75| $\\text{MAP}_{avg}$ |  \n|:--- | :----: | ---:|  ---:|  ---:|  ---:|   \n|1|  50.21 | 27.42  | 58.33 | 28.02 | 30.17|\n|2|  51.23 | 27.95  | 59.55 | 28.83 | 31.19|\n|4| **51.94**  | **28.25**  | **59.86** | **29.11** | **31.66**|\n|8|  51.39 | 28.01  | 59.42 | 28.89 | 30.92|\n\nMore results will be added to the supplementary material.\n\n\n**Q4 the iteration round parameters on all datasets.**\nWe show the R1@0.7  results corresponding to different iteration steps (1, 2, 10, 50, 100) on the three datasets in the table below:\n\n| Dataset | Type | 1 | 2 | 10 | 50| 100| \n|:--- | :----: | ---:|  ---:|  ---:|  ---:|     ---:|   \n| Charades-STA |VGG,Glove | 26.12  |  27.93 | 28.21| **28.25**| 28.27|\n| QVHighlights | SF+C, C  |  37.47  |  39.42 | 39.59| **39.66**| 39.58|\n| TACoS | C3D,Glove  |  15.24  |  16.81 | 17.69| **17.83**| 17.97|\n\nMore results will be added to the supplementary material.\n\n"}, {"review_id": "eOWu4tANi2", "reviewer": "Reviewer_mu2A", "summary": "To deal with the problem of temporal location bias, the authors propose a diffusion-based video moment retrieval framework, MomentDiff. They introduce the diffusion process into temporal localization from a generative perspective, and gradually generate real span coordinates from coarse to fine. Compared to learnable queries, the random noise input to the model reduces the dependence on the location information of the dataset. Therefore, MomentDiff achieves better results on two \"anti-bias\" datasets with changing location distributions. Besides, MomentDiff consistently outperforms state-of-the-art methods on three public benchmarks.", "questions": "In image generation tasks, images generated by diffusion models are often diverse. With different random noise inputs, are the span coordinates generated by the MomentDiff model for the same video-text pair quite different, and is the model performance stable? If the result is relatively stable, what is the reason?", "limitations": "The paper argues that if the user enters words that violate the law, the model may have a potential negative impact. I think sensitive word filtering technology can effectively solve this problem.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "a) This work proposes a novel and effective diffusion framework on the video moment retrieval task and alleviates the important location bias problem. The paper is also well motivated and well written.\nb) To demonstrate the robustness of the model, they propose two anti-bias datasets, which seem to be one of the main contributions of the paper.  Promising experimental results.\nc) The paper presents promising experimental results. Authors will provide code and datasets.", "weaknesses": "While I don't see obvious weaknesses, there are a few minor suggestions, and additional questions that the author needs to answer carefully: \na) Please revise the notation and typos of the paper. For example, \\epsilon in Eq 6 is not clearly defined above, only \\epsilon_m.\nb) Please re-check the paper and correct errors on formatting, grammar, etc. For example, L154. The Eq (1) uses “Snj”, “Spj”, but L154 writes “Spj”, “Sni”.\nc) Some of the figures in Fig. 1 and Fig. 3 are so small that they are difficult to see even when zoomed in.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your appreciation of our paper, including the motivation, writing, and robustness of the model. We will carefully revise the paper according to the questions and suggestions raised by the reviewers.\n\n**Q1 typos and errors in formatting, grammar, and figures.**\n\nThank you so much for your constructive comments and review. \nWe will carefully fix these errors, including formulas, font sizes, and spelling mistakes.\n\n**Q2 is the model performance stable?**\n\nIn Figure 1 in our Supplementary Material, we show the performance of the model with multiple random seeds (seed= 2023, 2022, 2021, 2020, 2019) on the Charades-STA dataset.\nFor readability, we present it here in tabular form:\n\n|Type|R1@0.5|R1@0.7|MAP@0.5|MAP@0.75|$MAP_{avg}$|\n|:---|---:|---:|---:|---:|---:| \n|VGG, Glove|51.94 $\\pm$ 1.9|28.25 $\\pm$ 1.7|59.86 $\\pm$ 2.4|29.11 $\\pm$ 0.6|31.66 $\\pm$ 0.4|\n|C3D, Glove|53.79 $\\pm$ 2.1|30.18 $\\pm$ 0.7|59.32 $\\pm$ 1.6|29.85 $\\pm$ 0.4|31.89 $\\pm$ 0.4| \n|SF+C, C|55.57 $\\pm$ 0.9|32.42 $\\pm$ 1.8|61.07 $\\pm$ 2.6|32.51 $\\pm$ 1.5|32.85 $\\pm$ 0.9| \n\n\nThis shows that the model always converges and achieves stable results for different initializations. \n\nWe believe that the reason is related to the characteristics of the diffusion model itself and the loss constraints.\nSpecifically, during the training process of the model, the input is the noisy span, and we constrain the model to generate a real span.\nThe real span has fixed manual time annotation, which is obviously different from the image generation task.\nSo the model will not generate overly diverse results under the constraints of annotations and losses.\n\nIn addition, since the input noisy spans are random, the model needs to learn the ability to generate real spans from arbitrary spans.\nThis ability ensures that the model can perform well under different initializations to a certain extent.\n\n**Q3 recommendations to address potential negative impacts.**\n\nThank you for your very professional review. We will investigate recent projects on sensitive word filtering and add to the supplementary material.\n"}, {"review_id": "jZYWkGpqRi", "reviewer": "Reviewer_uAtb", "summary": "This paper tackles the video moment retrieval task from the generative perspective and proposes a diffusion-based localization model, named MomentDiff. It could sample random temporal segments as initial guesses and iteratively refine them to generate an accurate temporal boundary. Moreover, this paper proposes two“anti-bias” datasets with location distribution shifts to evaluate the influence of location biases. Experiments on three public datasets validate the effectiveness of the proposed approach.", "questions": "1. Could you please explain how to obtain the values of $Q_{\\hat{v}}$, $K_{\\hat{v}}$, and $V_{\\hat{v}}$ mentioned on page 4, line 146?\n2. Regarding the use of span embedding as the query in Intensity-aware attention instead of combining it with the textual query, could you please elaborate on the reasoning behind this decision? Additionally, it would be helpful to know if any experiments were conducted to validate this choice and provide justification.\n3. This paper aims to incorporate audio features and integrate multi-modal video information. Could you please explain the methodology used to integrate the multi-modal video information? Furthermore, it is important to elaborate on how the paper demonstrates that the performance improvements are not solely a result of introducing audio information.", "limitations": "no", "rating": 5, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. This paper addresses the cross-modal moment retrieval task using a diffusion-based model, which is interesting.\n2. This paper builds two datasets with location distribution shifts, which is valuable for this research community. \n3. Experiments on three datasets: Charades-STA, QVHighlights, and TACos, demonstrating the effectiveness of the proposed approach MomentDiff.\n", "weaknesses": "1. Despite the widespread use of datasets like TAcos, Charades-STA, and ActivityNet Captions, this paper chose not to conduct experiments using ActivityNet Captions.\n2. Previous studies [1][2] employed CharadesCD and ActivityNet-CD to examine the influence of location biases. Nevertheless, this paper made the decision not to directly employ these datasets. Why?\n[1] Towards Debiasing Temporal Sentence Grounding in Video\n[2] A Closer Look at Temporal Sentence Grounding in Videos: Datasets and Metrics\n3. To provide comprehensive evaluation, comparisons with other supervised, weakly supervised, and zero-shot moment retrieval methods are crucial. Examples of such methods include [3] DORi: Discovering Object Relationships for Moment Localization of a Natural Language Query in a Video, [4] Structured Multi-Level Interaction Network for Video Moment Localization via Language Query, [5] Multi-Modal Relational Graph for Cross-Modal Video Moment Retrieval, and [6] Language-free Training for Zero-shot Video Grounding.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1 why not organize experiments on ActivityNet Captions?**\nWe explore the VMR problem based on DETR. The datasets used by the representative methods (MomentDETR and UMT) are QVHighlights and Charades-STA, so we use the same datasets.\nBesides, the results of our method on ActivityNet Captions (C3D features) are shown below, where we record the 1 epoch training time and the inference time for all test samples in one A100 GPU:\n|Method|R1@0.3|R1@0.5|R1@0.7|${MAP}_{avg}$|Training time|Testing time|\n|:---|---:|---:|---:|---:|---:|---:|   \n|2DTAN [15]|59.92|44.63| 27.53|27.26|1.1h|523.74s|\n|MMN [16]|65.21|48.26|28.95|28.74|1.5h|662.12s|\n|MomentDETR [25]|61.87|43.19|25.74| 25.63|0.05h|52.72s|\n|**Ours**|62.79|46.52|28.43|28.19|0.06h|91.43s|\n\n1. Our model has been improved compared with baseline (MomentDETR). Compared to SOTAs, we still have competitive results.\n2. Compared with MMN, our method has **25 times** faster training time per epoch and **7.24 times** faster testing time.\n\nWe will add more results in the revised version.\n\n**Q2 why not choose Charades-CD and ActivityNet-CD to organize experiments?**\nThanks for your constructive suggestion. \nIn VMR task, the span center $c\\_{0}$ and length $w\\_{0}$ are two important parameters. \nSo we want to verify the model generalization from two perspectives of length and position (Charades-STA-Len and Charades-STA-Mom). \nThe dataset construction strategy is very simple, and the evaluation perspective is comprehensive.\n\nAs suggested, we use the same VGG features to organize OOD experiments on **Charades-CD**:\n|Method|R1@0.3|R1@0.5|R1@0.7|$MAP_{avg}$|  \n|:---|---:|---:|---:|---:|  \n|2DTAN [15]|49.71|28.95|12.78|12.60|\n|MMN [16]|55.91|34.56|15.84|15.73|\n|MomentDETR [25]| 57.34|41.18|19.31|18.95|\n|**Ours**|67.73|47.17|22.98| 22.76 |  \n\nIn the **ActivityNet-CD** dataset, we use the same C3D feature to organize OOD experiments:\n|Method|R1@0.3|R1@0.5|R1@0.7|$MAP_{avg}$|  \n|:---|---:|---:|---:|---:| \n|2DTAN [15]|40.04|22.07|10.29|12.77| \n|MMN [16]|44.13|24.69|12.22|15.06|\n|MomentDETR [25]|39.98|21.30|10.58|12.19|\n|**Ours**|45.54|26.96|13.69|16.38|\n\n\nConclusion:\n1. **On Charades-CD and ActivityNet-CD, we exceed baseline (MomentDETR) by a large margin.** \nAlthough MMN achieved SOTA results on ActivityNet in the answer for Q1,  MMN (R1@0.5: 24.69) is still lower than our model (R1@0.5: 26.96) on ActivityNet-CD.\n2. These results prove the robustness of the model in dealing with OOD scenarios.\nOur generative framework alleviates the location biases problem.\nWe will add **the above results and codes** in the revised version, which make our paper more convincing.\n\n**Q3 comparisons with other supervised, weakly supervised, and zero-shot methods.**\nVery good suggestion.\nWe will refer to and compare these works in the revised version.\nFor the fairness of the experiment, we use the same features for comparison.\n\nCharades-STA:\n|Method|Type|R1@0.5|R1@0.7|\n|:---|---:|---:|---:|\n| DORi[3] | VGG |  43.47  |  26.37  |  \n| Ours |VGG| 51.94  | 28.25  | \n| SMIN[4] |C3D|   50.32 |  28.95  |  \n| MMRG [5] |C3D|  44.25 | - | \n| Ours |C3D|  53.79 | 30.18 |  \n\nActivityNet:\n|Method|Type|R1@0.5|R1@0.7|\n|:---|---:|---:|---:|\n| ZSVG[6] |C3D|   32.59 |  15.42  |  \n| Ours |C3D|  46.52 |28.43 |\n\n**Q4 explain how to obtain $Q_{\\hat{v}}$, $K_{\\hat{v}}$ and $V_{\\hat{v}}.$**\nSorry for the confusion.\nWe get $Q_{\\hat{v}}$, $K_{\\hat{v}}$ and $V_{\\hat{v}}$ through three different linear projection:\n$Q_{\\hat{v}} = W_{q}\\hat{V}, K_{\\hat{v}} = W_{k}\\hat{V}, V_{\\hat{v}} = W_{v}\\hat{V}, $\nwhere $W_{q}$, $ W_{k}$ and $W_{v}$ are linear projection matrices, $\\hat{V}$ is the embedding output by the previous cross-attention layers.\nWe will describe this process clearly in the revised version.\n\n**Q5 the reason for using span embeddings as queries instead of combining it with the text.**\nThere are three reasons:\n1. The text has already interacted with the video in the Similarity-Aware Condition Generator, so continuing to add text information may be unnecessary.\n2. Usually the query of the decoder in DETR series work (such as MomentDETR) is learnable embeddings, not text features. \nLearnable queries introduce location bias inherent in the dataset, so we set the query to be data-independent random spans.\nBesides, our model is based on the diffusion model. The entire training process is the noise addition and denoising process of ground truth spans. Both the input query and output results should be consistent spans.\n3. We add text features to the query and find that the results on Charades-CD is reduced. We speculate that adding text features to the query may perturb the denoiser to perceive the noise intensity, and thus the results will decrease. The results on Charades-CD are as follows:\n|Method|R1@0.3|R1@0.5|R1@0.7|$MAP_{avg}$|\n|:---|---:|---:|---:|---:| \n|w/ text features|67.21|45.99|22.63|22.26|\n|Ours|67.73|47.17|22.98|22.76|\n\n**Q6 explain the method used to integrate multi-modal video information.**\nThe key to this paper is how to alleviate the location bias problem and iteratively generate accurate temporal spans.\nFor adding additional audio information, our purpose is only to prove that our method can expand more modalities, but this is not the focus of the paper.\n\nIn addition, the process of fusing audio information is very simple, we only concatenate the audio features and the input visual features (e.g., VGG features). \nAs shown in Table 1 of our main paper, the results are as follows:\n\n|Method|Type|R1@0.5|R1@0.7|$MAP_{avg}$|\n|:---|---:|---:|---:|---:| \n|UMT [26]|VGG+Audio|48.44|29.76|30.37|\n|MomentDiff|VGG|51.94|28.25|31.66|\n|MomentDiff|VGG+Audio|52.62|29.93|31.81|\n\nFrom the results, we can find:\n1. Without using audio, the model still can achieve good results, such as R1@0.5=51.94.\n2. Using the same settings (VGG+Audio), our model (R1@0.5=52.62) can exceed UMT (R1@0.5=48.44) by a large margin.\n\nThe above results and more results in the paper prove the effectiveness of the model itself."}]

Author rebuttal:
Thanks to all reviewers for their careful comments. 

We would like to thank all reviewers for their appreciation of our paper, including **"writing is easy to follow" (Reviewer oTs9), "the high-quality approach" (Reviewer KdBK), "a novel framework" (Reviewer 2CFW), "promising results" (Reviewer mu2A) and "valuable datasets" (Reviewer uAtb).**
Our contributions specifically come from three aspects:

1. we are the first to tackle video moment retrieval from a generative perspective, which  mitigates temporal location biases from datasets.
2. We propose a new framework, MomentDiff, which utilizes diffusion models to iteratively denoise random spans to the correct results.
3. We propose two “anti-bias” datasets with location distribution shifts to evaluate the influence of location biases. Extensive experiments demonstrate that MomentDiff is more efficient and transferable than state-of-the-art methods on three public datasets and two anti-bias datasets.

In addition, we mainly reply to the reviewers from three aspects: 
1. Clarify the motivation and the reasons for the module design. 
2. We prove the effectiveness of the model in a larger dataset and more OOD datasets. 
3. We show more ablation studies.

**Please refer to individual responses to each reviewer for specific details.**

Finally，**thanks again to the reviewers for their efforts, we have benefited a lot.**



Meta-review: This paper receives final review scores of 7/5/5/7/6.

The paper presents the first attempt to tackle video moment retrieval problem from a generative perspective. It also presents two anti-bias datasets for evaluating the influence of location bias. The reviewers appreciate the novelty of the work and the illustrative presentation which is easy to follow. The experimental results are also comprehensive and convincing.

All reviewers acknowledged that the responses effectively addressed their concerns, so they either kept their original rating or raised the rating.
Decision: Accept (poster)