Detecting Moments and Highlights in Videos via Natural Language Queries
Authors: Jie Lei
Tamara Lee Berg
Mohit Bansal
Venue: NeurIPS 2021 Poster (2021)
Keywords: query-based video moment retrieval
query-based video highlight detection

Abstract: Detecting customized moments and highlights from videos given natural language (NL) user queries is an important but under-studied topic. One of the challenges in pursuing this direction is the lack of annotated data. To address this issue, we present the Query-based Video Highlights (QVHighlights) dataset. It consists of over 10,000 YouTube videos, covering a wide range of topics, from everyday activities and travel in lifestyle vlog videos to social and political activities in news videos. Each video in the dataset is annotated with: (1) a human-written free-form NL query, (2) relevant moments in the video w.r.t. the query, and (3) five-point scale saliency scores for all query-relevant clips. This comprehensive annotation enables us to develop and evaluate systems that detect relevant moments as well as salient highlights for diverse, flexible user queries. We also present a strong baseline for this task, Moment-DETR, a transformer encoder-decoder model that views moment retrieval as a direct set prediction problem, taking extracted video and query representations as inputs and predicting moment coordinates and saliency scores end-to-end. While our model does not utilize any human prior, we show that it performs competitively when compared to well-engineered architectures. With weakly supervised pretraining using ASR captions, Moment-DETR substantially outperforms previous methods. Lastly, we present several ablations and visualizations of Moment-DETR. Data and code is publicly available at https://github.com/jayleicn/moment_detr.

QVHIGHLIGHTS: Detecting Moments and Highlights in Videos via Natural Language Queries
Jie Lei Tamara L. Berg Mohit Bansal
Department of Computer Science University of North Carolina Chapel Hill {jielei, tlberg, (bansal)ecc unc. edu
Abstract
Detecting customized moments and highlights from videos given natural language (NL) user queries is an important but under-studied topic. One of the challenges in pursuing this direction is the lack of annotated data. To address this issue, we present the Query-based Video Highlights (QVHIGHLIHHT)) dataset. It consists of over 10,000 YouTube videos, covering wide range of topics, from everyday activities and travel lifestyle vlog videos to social and political activities videos. Each video in the dataset is annotated with: (1) human-written free-form NL query, (2) relevant moments in the video w.r.t. the query, and (3) five-point scale saliency scores for all query-relevant clips. This comprehensive annotation enables us to develop and evaluate systems that detect relevant moments well salient highlights for diverse, flexible user queries. We also present a transformer encoder-decode model
strong baseline for this task, Moment-DETT, that views moment retrieval as a direct set prediction problem, taking extracted video and query representations and saliency scores end-to-end. While our model does not utilize any human prior, we show that it performs competitively when compared to well-engineered architectures. With weakly supervised pretraining using ASR captions, Moment- DETR substantially outperforms previous methods. Lastly, we present several ablations and visualizations of Moment-DETR. Data and code publicly available at https: /tt::/giiit githuu........ wtw.ttt..oooo..eeeeeeeeeettttttt wwtp///ayleinnnnonnn _eetr.
inputs and predicting moment coordinates
1 Introduction
Internet videos are growing at an unprecedented rate. Enabling users to efficiently search and browse these massive collections of videos essential for improving user experience online video platforms. While a good amount of work has been done in the area of natural language query based video search for complete videos (i.c., text-to- retrieval [41, 42, 17]), returning the whole video is not always desirable, since they be quite long (e.g., from few minutes to hours). Instead, users may want to locate precise moments within a video that are most relevant to their query or see highlights a glance so that they can skip to relevant portions of the video easily.
Many datasets [13, 7, 18, 16, 31] have been proposed for the first task of 'moment retrieval' localizing moments in video given a user query. However, most of the datasets are reported [4, 18] to have a strong temporal bias, where more moments appear at the beginning the videos than the end. Meanwhile, for each video-query pair, all of the datasets provide annotations with only single moment. In reality, there are often multiple moments, i.e., several disjoint moments a video, that related to given query. For the second task of 'highlight detection', many datasets [38, 12, 36, 8] are query-age aggost,,,,, where the detected highlights do not change for different input user queries. [22, 43] are the two existing datasets that collect highlights based on user queries. However, only
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

A shark swimming underwater.
Glass laying all over the street from broken windows beside other trash and debris in frunt sfore buildings.
Figure 1: QVHIGHLIGHTS examples. We show localized moments in dashed green boxes. The highlightness (or saliency) scores from 3 different annotators are shown under the frames as colored bars, with height and color intensity proportional the scores.
small set of frames or clips are annotated frames out of 331 seconds long videos in [22] or around 10 seconds clips out of 60 seconds video in [43]), limiting their ability to accurately learn and evaluate highlight detection methods. Lastly, although these two tasks of moment retrieval and highlight detection share many common characteristics (e.g., both require learning the similarity between user text query and video clips), they are typically studied separately, mostly due to the lack of annotations supporting both tasks in a single dataset.
To address these issues, we collect Q HIGHLLIGHHTS a unified benchmark dataset that supports query-based video moment retrieval and highlight detection. Based on over 10,000 YouTube videos covering a diverse range of topics (from everyday activities and travel in lifestyle vlog videos to social and political activities in news videos), we collect high-quality annotations for both tasks. Figure shows two examples from HIGGGHLIGHTS. For moment retrieval, we provide one or multiple disjoint moments for query in a video, enabling more realistic, accurate, and less-biased (see Section 3.2) evaluation moment retrieval methods. Within the annotated moments, we also provide five-point Likert-scale (from "Very Good' to 'Very Bad') saliencc/highlighhness score annotation for each 2-second clip. This comprehensive saliency annotation gives more space for designing and evaluating query-based video highlight detection methods.
Next, to present strong initial models for this task, we take inspiration from recent work such as DETR [3] for object detection, and propose Coment-DETR, an end-to-end transformer encoder- decoder architecture that views moment retrieval as a direct set prediction problem. With this method, we effectively eliminate the need for any manually-designel pre- peepprocsssing (e.g., proposal generation) or post-processing (e.g., non-maximum suppression) steps commonly seen moment retrieval methods. We further add saliency ranking objective on top of the encoder outputs for highlight detection. While Moment-DETR does not encode any human prior in its design, our experiments show that it is still competitive when compared to highle-engineered architectures. Furthermore, with additional weakky-supprriin pretraining from ASR captions, Moment-DETR substantially outperforms these strong methods. Lastly, also provide detailed ablations and visualizations to help understand the inner workings of Cooentt-ETTR
Overall, our contributions are 3-fold: (i) We collect the (THIGHLIGHTS dataset with over 10,000 videos, annotated with human-written natural language queries, relevant moments, and saliency scores. (ii) We propose Moment-DETR to serve as a strong baseline for our dataset. With weakly supervised pretraining, Moment-DETR substantially outperforms several baselines, on both our proposed QV QVIGGILIGHTS dataset and the moment retrieval dataset CharadesSTA [7]. (tii We present detailed dataset analyses, model ablations and visualizations. For ablations, we examined various design choices of Moment-DETT as well its pre-training strategy. We hope our work would inspire and encourage future work towards this important direction.
Related Work
Datasets and Tasks. Moment retrieval [13, 7, 18] requires localizing moments from a video given a natural language query. Various datasets [13, 7, 18, 16, 31] have been proposed or repurposed for

---

the task. However, as shown in [13, 4, 18], many of them have a strong temporal bias, where more moments are located at the beginning of the videos than the end. In Section 3.2 we show moments in QVHIGHLIGHTS distribute almost evenly over the videos. Meanwhile, while these datasets collect only single moment for each query-video pair, we collect one or more moments. Highlight detection is another important task in our dataset. Most existing datasets [38, 12, 36, 8] are query-agnostic, which do not provide customized highlights for a specific user query. [22, 43] are the two known datasets that collect highlights based on user queries. However, they only annotate small set of frames or clips, limiting their ability to accurately learn and evaluate highlight detection methods. In contrast, we provide comprehensive five-point Likert-scale aalieneyhighhlihhtesss score annotation for all clips that are relevant to the queries. Besides, although these two tasks share some common characteristics, they typically addressed separately using different benchmark datasets. In this work, we collect QVHIGHI.GHHT as a unified benchmark that supports both tasks. In Section 5.2 we also demonstrate that jointly detecting saliency is beneficial for retrieving moments.
Methods. There are a wide range of approaches developed for addressing the moment retrieval and highlight detection tasks. For highlight detection, prior methods [38, 22, 12, 21, 33] are typically ranking-based, where models are trained to give higher scores for highlight frames or clips, via a hinge loss, cross-entropy loss, or reinforcement approaches. For moment retrieval, there are work that try to score generated moment proposals [13, 35, 4, 48, 46, 40], predict moment start-end indices [9, 18, 20, 44, 47] regress moment coordinates [7]. However, most of them require preprocessing (e.g., proposal generation) or postprocessing step (e.g., non non-maximum suppression) that are hand-crafted, and are thus not end-to-end trainable. In this work, drawing inspiration from recent work on object detection [3, 15] and video action detection [27, 45], we propose Moment-DETR that views moment retrieval as direct set prediction problem. Moment-DEF DETe takes video and user query representations as inputs, and directly outputs moment coordinates and saliency scores end-to-end hence eliminating the need for any pre- or post-processing manually-designed human prior steps.
3 Dataset Collection and Analysis
Our LVGGHHLIGHTS dataset contains over 10,000 videos annotated with human written, free-form queries. Each query is associated with one or multiple variable-length moments in its corresponding video, and a comprehensive 5-point Likert-scale saliency annotation for each clip in the moments. In the following, we describe our data collection process and provide various data analyses.
3.1 Data Collection
Collecting videos. We would like collect set of videos that are less-edited and contain interesting and diverse content for user annotation. Following [6, 19], we start with user-created lifestyle vlog videos on YouTube. These are created by users from all over the world, showcasing various events and aspects of their life, from everyday activities, to travel and sightseeing, etc. These videos are captured via different devices (c.g., smartphones or GoPro) with different view angles (c.g.. first- person or third-person), posing important challenges to computer vision systems. To further increase the diversity of the dataset, we also consider news videos that have large portions of 'raw footage'. These videos tend to cover more serious and world event topics such as natural disasters and protests. To harvest these videos, we use list of queries, e.g., 'daily vlog', 'travel vlog'. 'news hurricane'. etc. We then download top videos that are 5-30 minutes long from YouTube's search results, keeping videos that are uploaded after 2016 for better visual quality, and filtering out videos with a view count under 100 or with very high dislike ratio. These raw videos are then segmented into 150-second short videos for annotation.
Collecting user queries and relevant moments. To collect free-form natural language queries and their associated moments in the videos, we create an annotation task on Amazon Mechanical Turk. In this task, we present workers with a video and ask them to watch the video and write a query in standard English depicting interesting activities in the video. Next, we present the same worker with a grid of 2-second long clips segmented from the video, and ask them to select all clips from the grid relevant to the query. The selection can be done very efficiently via click for selecting a single clip and click-and-drag for selecting consecutive clips. This 2-second clip annotation protocol allows for more precise annotation than using 5-second clip as [13]. Moreover, different from previous work [13, 7, 18, 16, 31] where only a single moment can be selected for query-vide query-video pair, users can
3

---

Table
Top unique verbs and nouns in queries, in each video category.
Category
#Queries
Top Unique Verbs
Top Unique Nouns
Daily Vlog
4,473
cook, apply, cut, clean swim, visit, order, travel report, gather, protest, discuss
dog, kitchen, baby, floor
Travel Vlog
4,694
beach, hotel, tour, plane
News
1,143
news, interview, weather, police
Percentage of Moments
004
12 14
18
20
12
24
26
28
2:30
0.1
0.2
0.4
0.5
0.5
0.7
035
0.9
1.2
Moment Length (soconds)
Normalized Moment Center
Figure 2: Distribution of moment lengths (left) and normalized (by video duration) center timestamps (right). The moments vary greatly in length, and they distribute almost evenly along the videos.
select multiple disjoint moments in our setup. To verify the quality of the moment annotation, we use a set of 600 query-video pairs, and collect 3 sets of moments for each query from different workers. We then calculate the Intersection-owee -over-Union (loU) between every pair of moments annotated for the same query, and take the average of the 3 IoU scores to check the inter-user agreement. We find that, for around 90% of queries, their moments have an average IoU score higher than 0.9, suggesting that the moments collected via our annotation process are of high inter-user agreement, thus high quality,
Annotating saliency scores. The relevant moment annotation in the previous step tells us which clips in the videos correspond to the user queries. Though all selected clips are relevant to the query, they may still vary greatly in their saliency, how representative they are for the query, or whether they would make good highlight. For example, we would expect a clip with a proper camera angle and lighting be better than clip with a lot of occlusions on the activities being queried, and therefore be a better highlight for the video. Thus, we create a second annotation task targeting at collecting the saliency scores for each relevant clip. We do not ask workers to select only a small set of clips as highlights [43] because many clips may look similar and be equally salient. Hence forcing people to pick only few clips from these similar clips can cause confusion and degrade annotation quality. Specifically, we present all the selected clips in the first task along with the queries to another set of workers. For each clip, the workers are required to rate them in a Likert-scale system' with five options, "Very Good', 'Good', 'Fair', 'Bad', 'Very Bad', As highlightness can be subjective, we collect ratings from 3 different workers and use all of them for evaluation.
Quality control. To ensure data quality, we only allow workers who have done more than 500 HITs and with an approval rate of 95% to participate in our annotation task. We also follow [18] to set up a qualification test. Our test contains seven multiple-choice questions (see supplementary file for an example), and workers have to correctly answer all questions in order to qualify for the task. In total, 543 workers took the test, with a pass rate of 48%. This qualification ensures high data quality as mentioned earlier this subsection, we observe high inter-user agreement moment annotations. For query and saliency annotation, we pay workers $0.25 and $0.18 per query, respectively. The average hourly pay around $1 1.00. The whole collection process took about months and cost approximately $16,000.
3.2 Data Analysis
In total, we collected 10,310 queries associated with 18,367 moments in 10,148 videos. The videos are from three main categories, daily vlog, travel vlog, and news events. In Table 1, we show the number of queries in each category and the top unique verbs and nouns in the queries. These top unique words reflect the major of activities occurring in the videos. For example, in daily and travel vlog videos, the top unique verbs mostly associated with daily activities such as "cook' and 'clean', while news videos, they are more about serious activities such as 'report', 'gather", 'protest'.
'https://en hatpp::/eewwiiiieddia.oo wtt.ediaa..rrg/iii/ikkee...t

---

Table 2: Comparison with existing moment retrieval (top) and highlight detection (bottom) datasets. Q=Query, MR=Momeett Retrieval, HD=Higghighht Detection.
Avg Query Len
Avg Len (sec) Momen//Vides
Avg. #Moments
Supported Tasks
Has Query
Dataset
Domain
#Queries/eVidens
per Query
MR
DiDeMo [13]
41.2K/
ANesCaptions [16]
Activity
72K,/ / 15K
14,8
36.2/ 117.6
CharadeSTTA [7]
Activity
16.1K 6,7K
7.2
30.6
TVR [18]
TV show
/218K
13.4
9.1/
YoghubeHighigghs [38]
Activity
-066K
-/143
Video2GIF [12]
Open
/ SOK
/332
BeautyThumh [36]
Open
LLIKK
-/ 166
ClicTThnughh [22]
Open
-/IK
-/331
ActivitYhuum Thumh [43]
Activity
8.7/60.7
QVHIGILIGITTS
Vlog / News
10.3K/ 10.2K
11.3
24.6/150
1.8
/
Table 2 shows a comparison between QVHIGHLIGHHS and existing moment retrieval and highlight detection datasets. QVHIGHLIGHTS can have multiple disjoint moments paired with a single query (on average 1.8 moments per query in video), while all the moment retrieval datasets have a single moment. This is a more realistic setup as relevant content to a query a video might be separated by irrelevant content. It also enables more accurate evaluation since the annotations are exhaustive and clean for a single video, i.e., all the relevant moments are properly selected and no irrelevant moments are selected. In Figure 2, show the distribution of moment lengths and normalized (by video duration) moment center timestamps. Our dataset has a rich variety of moments that vary greatly in length. Around 38% of the moments have length of equal or less than 10 seconds, while around 23% are more than 30 seconds. The moments are almost equally distributed across the video, with small peak in the middle (some moments span across the whole video), suggesting that our dataset suffers less from the temporal bias commonly seen in other moment retrieval datasets [13, 18] where moments tend to occur nearer the beginning of videos.
only
Most the highlight detection datasets [38, 12, 36] in Table focus on query-independent high- light detection while QVHIGHLIGHTS focuses query deerydde highlights detection. Click- Through [22] and ActivityThumbnaln Thumbnails [43] also have highlight annotations for queries, but their annotations not comprehensive: for video, ClickThrough only annotates 20 key frames and SetivityThuubnee restricts highlights to less than five clips. In contrast, we adopt a two-stage annotation process with comprehensive 5-scale saliency score for all relevant clips, making it more useful for developing effective models and more accurate for evaluating model performance.
Methods: Moment-DETR
Our goal is to simultancously localize moments and detect highlights in videos from natural language queries. Given a natural language query of T.g tokens, and a video e comprised of sequence of L clips, we aim to localize one v), as well as predicting clip-wise saliency scores S E Re (the highest scored clips are selected as highlights). Inspired by recent progress using transformers for object detection (DETR [3]), in this work we propose a strong baseline model for our QVHIGILLGIITS dataset, "Moment-DETR' an end-to-end transformer encoder-decoder architecture for joint moment retrieval and highlight detection. Moment-De DETR removes many hand-crafted components, e.g., proposal generation module and non- man-maximum suppression, commonly used in traditional methods [13, 35, 4, 48, 46, 40]. views moment localization as direct set prediction problem. Given set of learned moment queries, Moment- DETR models the global temporal relations of the clips in the videos and outputs moment span coordinates and saliency scores. In the following, we present Moment-DETR in detail.
moments {m,) (a moment is consecutive subset of clips in
4.1 Architecture
Figure shows the overall architecture of Moment-DETR. In the following, explain it in details.
Input representations. The input to the transformer encoder the concatenation of projected video and query text features. For video, we use SlowFast [5] and the video encoder (ViT-B/32) of CLIP [30] to extract features every seconds. We then normalize the two features and concatenate them at hidden dimension. The resulting video feature is denoted as E, E BR x2816 For query
5

---

sellency scoree
FFN
FFN
FFN
FFN
foreground
background
foreground
baskground
span
span
Transformer Encoder
Transformer Decoder
saliency
Moment Queries
unpacks his grocemes.
(Trainable embeddinge)
Cross Entropy Loss
Hinge
Figure 3: Moment- DETR model overview. The architecture simple, with a transformer encoder- decoder and three prediction heads for predicting saliency scores, ttee/bbcckgrroung scores and moment coordinates. For brevity, the video and text feature extractors are not shown in this figure.
text, we use the CLIP text encoder to extract token level features, Fg E 4yx122 Next, we use separate 2-layer perceptrons with layernorm [14] and dropout [14] to project the video and query features into shared embedding space size d. The projected features are concatenated [15] at length dimension as the input the transformer encoder, denoted as Einpue E ER L-L, Lgg
Transformer encoder-decoder. The video and query input sequence encoded using stack of T transformer encoder layers. Each encoder layer has the same architecture as in previous work [39, 3], with a multi-head self-attention layer and a feed forward network (FFN). Since the transformer architecture is permutation permutation-invariaan fixed positional encodings [28, 1] are added to the input of each attention layer, following [3]. The output of the encoder is Eene E IR Lxd The transformer decoder the same in [39, 3], with stack of T transformer decoder layers. Each decoder layer consists of a multi-head self-attention layer, a cross-attentiin layer (that allows interaction between the encoder outputs and the decoder inputs), and an FFN. The decoder input is set of N trainable positional embeddings size d, referred to as moment queries." These embeddings are added to the input to each attention layer in the encoder layers. The output of the decoder Edec R
Prediction heads. Given the encoder output Eanes we use a linear layer to predict saliency scores S E RR for the input video. Given the decoder output Edec. to predict the normalized moment center coordinate and width w.r.t. the input video. We also follow DETR [3] to use linear layer with softmax predict class labels. In DETR, this layer is trained with object class labels. In our task, since class labels are not available, for a predicted moment, we assign it a foreground label if it matches with ground truth, and background otherwise.
3-layer FFN with ReLU [11]
4.2 Matching and Loss Functions
Set prediction via bipartite matching. We denote J as the of N predictions from the moment queries, and {yr YN the set of ground truth moments with background Ø padding. Note that N is the number of moment queries and larger than the number of ground truth moments. Since the predictions and the ground truth do not have a one-to-one correspondence, in order to compute the loss, we need to first find an assignment between predictions and ground truth moments. We define the matching cost Cmatch between prediction and ground truth moment
Cmaach (yis (o()) - (o.3 +1 (cc Eooment (m, ma(ss))
where each ground truth can be viewed y/ (c,, m.). with e; the class label indicate foreground or background Ø and m, E [0, 1)2 vector that defines the normalized moment center coordinate and width w.r.t. input video; (a()) is the i-th element of the prediction under a permuta- tion a E G. Note that the background paddings in the ground truth are ignored in the matching cost. With this matching cost, follow [3, 37]. using the Hungarian algorithm find the optimal bipartite matching between the ground truth and predictions: & arg min Σ/ Cmatch (3is (o)))) Based on this optimal assignment o, in the following, we introduce our loss formulations.
Following [3], moment queries to refer to decoder positional embeddings. the text queries.

---

Moment localization loss. This loss Lmoment is used to measure the discrepancy between the prediction and ground truth moments. It consists of an L1 loss and a generalized IoU loss [32]:
mmmment (m;, mars)) = AmilIm; - ma) + Aioufiou (ms, (a))
(2)
where ALI. Aion E RR are hyperparameters balancing the two terms. The IoU loss Liou here computes 1D temporal loU instead of 2D box IoU as in [32, 3], but they share the same formulation.
Saliency loss. The saliency loss is computed via a hinge loss between two pairs of positive and negative clips. The first pair is a high score clip (with index /high) and low score clip (how) within the ground-truth moments.² The second pair consists of one clip (lmn) within and one clip (Cout) outside the ground-truth moments. This loss is calculated as (A E IR is the margin):
Lsaliency (S) (0, A + S(toww S(tigg)) + max(0, A + S(tout) - S(tw))
(3)
Overall loss. The final loss is defined as a linear combination of the losses introduced above:
L = dsaliency Cealiency (S) +
Aeta log (c) + [ (m,,, (4)
where Asaliency, Acts E are hyperparameters for saliency and fore/background classification loss. Following [3], we weight the log-probabilityy by factor of 10 for the background class Ø to account for class imbalance and apply classification and moment losses to every decoder layer.
4.3 Weakly-Supervised Pretraining via ASR
Moment-DETR is defined using an end-to-end transformer encoder-decoder architecture, eliminating the need for any human priors or hand-crafted components. Such a model typically requires a larger- scale dataset for training to unleash its true power, which would be prohibitively expensive to acquire with human labeling. Therefore, we additionally experiment with using captions from Automatic pretraining. Although very noisy,
Speech Recognition (ASR) on our videos for ASR captions have been shown improve performance for visual recognition and text-to-video retrieval [25, 24, 20]. Specifically, we download ASR captions from YouTube, and use these caption sentences as queries, training the model to predict their corresponding timestamps. In total, we harvest 236K caption-timestamp pairs associated with 5406 Irain videos. For pretraining. the model architecture and learning objectives are the same as in standard training, except that we remove the first term in the saliency loss (Fquation 3) since the saliency score annotation is not available.
5
Experiments and Results
5.1 Experimental Setup
Data and evaluation metrics. We split QVHIGHLIGHTS into 70% train, 15% val, and 15% test portions. To evaluate moment retrieval with multiple moments, we use mean average precision (mAP) with loU thresholds 0.5 and 0.75, as well as the average mAP over multiple loU thresholds [0.5: 0.05: 0.95], similar to action detection [2]. We also report standard metric Recall (R @1) used in single moment retrieval, where we define prediction to be positive if it has a high IoU (>= 0.7) with one of the ground truth moments. For highlight detection, we use mAP as the main metric. We also follow [22] to use HIT I to compute the hit ratio for the highest scored clip. Similar to [22], we define a clip as positive if it has score of 'Very Good'. Since we have ground truth saliency scores from 3 users, we evaluate performance against each then take the average.
Implementation details. Our model implemented in PyTorch [29]. We set the hidden size d=256, #layers in encoder/decoder 7=2, #moment queries N=10. We use dropout of 0. for transformer layers and 0.5 for input projection layers. We set the loss hyperparameters as A11=10, Aio==1, A41==4, A,=1, A=0.2. The model weights are initialized with Xavier init [10]. We use Adam [23] with an initial learning rate of le-4, weight decay Ic-4 to optimize the model parameters. The model is trained for 200 epochs with batch size 32. For pretraining, we use the same setup except that we train the model for 100 epochs with batch size 256. Both training/finetuning and pretraining are conducted on an RTX 2080Ti GPU, with training/finetunin
taking 12 hours and pretraining 2 days.
We average the saliency scores from 3 annotators and then choose pair of high and low score clips.
7

---

Table 3: Baseline Comparison on QVHIGHLIGHTS split. We highlight the best score in each column in bold, and the second best score with underline. XML+ denotes our improved XML [18] model. PT denotes weakly supervised pretraining with ASR captions. For Moment-DETT variants, we also report standard deviation of 5 runs with different random seeds.
Moment Retrieval
Highlight Delection
Method
RI
mAP
>= Very Good
@0.5
@0.7
@0.5
@0.75
avg
mAP
HIT@/
Beauty Thumb [36]
14.36
20,88
DVSE [22]
18.75
21.79
MCN [13]
11.41
2.72
24.94
8.22
10.67
CAL [4]
25.49
11.54
23.40
7.65
9.89
CLIP [30]
16.88
5.19
18.11
7.00
7.67
31.30
61.04
XML [18]
41.83
30.35
44.63
31.73
32.14
34.49
55.25
XML+
46.69
33.46
47.89
34.67
34.90
35.38
55.06
Moment-DErT DETT
52.89
33.02217
54.822117
29.40
30.73214
35.69=05
55.602216
Moment-DETR w/ PT
59.78+003
40.33+uu
60.51+
35.36++04
36.14+025
37.43-u2
60.17+07
5.2 Results and Analysis
Comparison with baselines. We compare Moment-DETR with various moment retrieval and high- light detection methods on the QVHIGHLIGHTS test split; results are shown in Table 3. For moment retrieval, we provide three baselines, two proposal-bas based methods MCN [13] and CAL [4], and a span prediction method XML [18]. For highlight detection, we provide two baselines, PeautyyThumb [36] based solely on frame quality, and DVSE [22] based on clip-query similarity. Since XML also outputs clip-wise similarity scores to the user query. we provide highlight detection results for this model as well. The original XML model has a smaller capacity than Moment-DETR, hence for a fair comparison, we increased its capacity by adding more layers and train it for the same number of epochs as Moment-DETR. Moreover, to leverage the saliency annotations in Q QVIGHHLIGHTS, we further added an auxiliary saliency loss to it (referred to as 'XML+'). These enhancements improve the original XML model by 2.76 average mAP. In addition, for both tasks, we also add CLIP [30] as baseline. Specifically, we compute clip- wise similarity scores by computing image-query scores where the image is the center frame of the clip. For highlight detection, we directly use these scores as prediction; for moment retrieval, we use TAG [49] progressively groups top-scored clips with the classical watershed algorithm [34].
Compared to the best baseline XML+, Moment-DETR performs competitively on moment retrieval, where it achieves significantly higher scores on a lower IoU threshold, i.e., R1 @0.5 and mAP@0.5 (>7% absolute improvement), but obtains lower scores on higher loU threshold, i.e., R1 @0.7 and mAP@0.75. We hypothesize that this is because the I.1 and generalized IoU losses give large penalties only to large mismatches (i.e., small IoU) between the predicted and ground truth moments. This property encourages Moment-DETR to focus more on predictions with small IoUs with the ground truth, while less those with reasonably large IoU (e.g., 0.5). This observation is the same as DETR [3] for object detection, where it shows a notable improvement over the baseline in APso, but lags behind in 73. For highlight detection, Moment-DEII performs similarily to XML+. As discussed in Section 4.3, Moment- DETR is designed without human priors or hand-crafted components, thus may require more training data to learn these priors from data. Therefore, also use ASR captions for weakly supervised pretraining. With pretraining, Moment-DETR greatly outperforms the baselines finding is that CLIP [30] gives the best highlight detection performance in terms of HIT@ 1, though its overall performance is much lower than Moment-DETR.
both tasks, showing the effectiveness of our approach." One surprising
Loss ablations. In Table 4, we show the impact of the losses by turning off one loss at a time. When turning off the saliency loss, observe significant performance drop for highlight detection, and surprisingly moment retrieval as well. We hypothesize that the moment span prediction losses (L1 and IoU) and classification (CLS) loss do not provide strong supervision for learning the similarity
*We also tried pretraining with XML+, and under careful tuning. the results are still worse than without pretraining, which might because XML+'s cross-entropp loss gives strong penalties to small mismatches. preventing from learning effectively from noisy (thus many small mismatches) data.
8

---

Table 4: Loss ablations on QVHIGHLIGHTS val split. All models are trained from seratch.
Moment Retrieval
Highlight Detection (>=Very Good)
gloU
Saliency
@00.5
RI @00.7
mAP avg
mAP
Hit@1
44.84
25.87
25.05
17.84
20.19
51.10
31.16
27.61
35.28
54.32
50.90
30.97
28.84
36.61
56.71
53.94
34.84
32.20
35.65
55.55
Figure 4: Visualization all moment span predictions for all the 1550 videos on QVHIGHLIGHTS val split, for all the 10 moment query slots in Moment-DETT decoder. x-axis denotes the normalized moment span center coordinates w.r.t. the videos, y-axis denotes the normalized moment span width (also indicated by color). We observe that each slot learns to predict moments in different temporal locations and different widths. For example, the first slot mostly predicts short moments near the beginning of the videos, while the second slot mostly predicts short moments near the end.
Table 5: Effect of pretraining data domain and size, results are on QVHIGHLIGHTS val split.
Moment Retricval
Highlight Detection
Pretraining Videos
RI 2.5
RI @0.7
mAP avg
mAP
HIT@
None
53.94
34.84
32.20
35.65
55.55
2.5K How To100M
54.58
36.45
32.82
36.27
56.45
HowTo100M
55.23
35.74
32.84
36.82
58.39
10k HowTo
56.06
38.39
34.16
35.97
56.84
5.4K QVHIGHLIGHTS
59.68
40.84
36.30
37.70
60.32
between the input text queries and their relevant clips, while the saliency loss gives a direct signal to learn such similarity. Under our framework, this also suggests that jointly detecting saliency is beneficial to relrieve moments. When turning off one of the span predictions losses (L1 or IoU), we see notable drop in moment retrieval performance while the highlight detection performance stays similar, showing that both losses are important for moment retrieval.
Moment query analysis. In Figure 4, we visualize moment span predictions for all the 1550 OVHIGHLGGHTS val videos, for the 10 moment query slots in Moment-DETR decoder. As shown in the figure, each slot learns to predict moments of different patterns, i.c., different temporal locations and different widths. For example, some slots learn to predict short moments near the beginning or end of the videos (e.g., the first two slots), while some slots learn to predict both short and long moments near the center (e.g., the third slot). Overall, most of the slots learn to predict short moments while only handful of them learn to predict long moments, possibly because there are more short moments HIGGGHLIGHTS than long moments (see our data analysis in Section 3.2).
Pretraining data domain and size. To better understand the role of pretraining, we examine the ef- feet of using data of different domain and sizes for pretraining. Specifically, we use HowTol00M [25] instructional videos of different sizes for pretraining Moment-DETR and then evaluate its finetuning dataset. The results shown in Table 5. We notice that out-of-
performance on QVHIGHLIGHHT domain pretraining on HowTo 100M videos also improves the model performance, even when only trained on 2.5K videos. When increasing the number videos, we see there steady performance gain for the moment retrieval task. While for highlight detection, the performance fluctuates, probably because of the pretraining task is not well aligned with the goal of detecting highlights. Compared to in-domain and out-of- videos, we notice that 5.4K in-domain QV QVHHGGH.IGHH videos greatly outperforms 10K HowTo 100M videos, demonstrating the importance of aligning the pretraining and the downstream task domain.

---

Table 6: Results on CharadesSTA [7] test split.
Method
R1@0.5
R1@0.7
CAL [4]
44,90
24.37
2D TAN [48]
39.70
23.31
VSINet [47]
47.31
30.19
IVG-DCL. [26]
50.24
32.88
Moment-DEi -DETR
53.63 55.65
31.37
Moment -ETTR w/ PT (on 10K HowTo videos)
34.17
Asian woman wearing Boston t-shirt her home talking.
Saliency scores
A family is playing baskethall together green court outside.
Saliency scores
Figure 5: Prediction visualization. Predictions are shown solid red boxes or lines, ground-trutt are indicated by dashed green lines. Top row shows correct prediction, bottom shows failure.
Generalization to other datasets. In Table 6, test Moment-DETR's performance moment retrieval dataset CharadesSTA [7]. Similar to our observations in Table 3, when comparing to SOTA methods, notice that Moment-DETR shows significant performance gain RI @0.5, while performs slightly worse on tighter metric, i.e., RI @0.7. Meanwhile, after pretrained on 10K HowTo 100M videos, Moment-DETR's performance is greatly improved, setting new state-of-the-art for the dataset. This shows Moment-DETR ' potential to adapt to different data and tasks.
Prediction visualization. Figure 5 (top) shows correct prediction from Moment-DETR. We can see that the model is able to correctly localize two disjoint moments relevant to the user query. Meanwhile, the saliency scores also align very well with the ground truth score curve (obtained by averaging the scores from 3 annotators). And not surprisingly, this saliency score curve also matches the moment predictions- where see higher scores for localized regions than those outside. Figure (bottom) shows a failure case, where it incorrectly localized a partially true moment (2nd frame) where the family is playing court but not playing basketball. More examples in Appendix.
Conclusion
We collect QVHIGHLIGHTS dataset for moment retrieval and highlight detection from natural language queries. This new dataset consists of over 10,000 diverse YouTube videos, each annotated with free-form query, relevant moment timestamps and clip-wise saliency scores. Detailed data analyses are provided comparing the collected data to previous works. We further propose Moment- DETR, encoder-decoder transformer that jointly perform moment retrieval and highlight detection. We show that this new model performs competitively with baseline methods. Additionally. it also learns effectively from noisy data. With weakly supervised pretraining using ASR captions, Moment- DETR substantially outperforms previous methods, setting a strong precedence for future work. Lastly, provide ablations and prediction visualizations of Moment-DETR.
Social Impact The predictions from our system reflect the distribution the collected dataset. These predictions can be inaccurate, and hence users should not completely rely on our predictions for making real-world decisions (similar to previous work on modeling video-based predictions). In addition, please license and usage of the data and code in the supplementary file.
10

---

Acknowledgements: We thank Linjie Li for the helpful discussion. This work is supported by NSF Award # #1562098. DARPA KAIROS Grant #FA8750-19-2-1000 19-2--004, ARO- RIP Award #WW1INNF1841 0336, DARPA MCS Grant N66001-19-224031, and Google Focused Research Award. The views contained in this article are those of the authors and not of the funding agency.
References
[1] Irwan Bello, Barret Zoph, Ashish Vaswani, Jonathon Shlens, and Quoe V Le. Attention augmented convolutional networks. In ICCV, 2019. 6
[2] Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. Activitynet: large-scale video benchmark for human activity understanding. In CVPR, 2015. 7
[3] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usumier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV, 2020. 3, 6, 7,
8
[4] Victor Escorcia, Mattia Soldan, Josef Sivie, Bernard Ghanem, and Bryan Russell. Tem- poral localization of moments in video collections with natural language. arXiv preprint arXiv:1907.12766 12763, 2019. 3, 8, 10
[5] Christoph Feichtenhofer. Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. In ICCV, 2019.
[6] David F Fouhey, Wei-cheng Kuo, Alexei A Efros, and Jitendra Malik. From lifestyle vlogs to everyday interactions. In CVPR, 2018. 3
[7] Jiyang Gao, Chen Sun, Zhenheng Yang, and Ram Nevatia. Tall: Temporal activity localization via language query. In ICCV, 2017. 1, 3,
[8) Ana Garcia del Molino and Michael Gygli. Phd-gifs: personalized highlight detection for automatic gif creation. In Proceedings of the 26th ACM international conference on Multimedia, pages 600-608, 2018. 3,
[9] Soham Ghosh, Anuva Agarwal, Zarana Parekh, and Alexander Hauptmann. Excl: Extractive clip localization using natural language descriptions. NAACI, 2019.
[10] Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In AISTATS, 2010. 7
[1]] Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Deep sparse rectifier neural networks. In AISTATS, 2011. 6
[12] Michael Gygli, Yale Song, and Liangliang Cao. Video2gif: Automatic generation of animated gifs from video. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1001-1009, 2016. 1, 3,5
[13] Lisa Anne Hendricks, Oliver Wang. Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan Russell. Localizing moments in video with natural language. ICCV, 2017. 1, 2, 5, 8
[14] Geoffrey E Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan R Salakhut- dinov, Improving neural networks by preventing co-adaptation of feature detectors. arXiv preprint arXiv: 7207 0580, 2012.
[15] Aishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel Synnaeve, Ishan Misra, and Nicolas Carion. Mdetr- modul--oddltted detection for end-to-end multi-modal understanding. In ICCV, 2021.
3,
[16] Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, and Juan Carlos Niebles. captioning events in videos. In ICCV, 2017, 2,
Dense-
[17] Jie Lei, Linjie Li, Luowei Zhou, Zhe Gan, Tamara I. Berg, Mohit Bansal, and Jingjing Liu, Less is more: Clipbert for video-and-lane lad-uagg learning via sparse sampling. In CVPR, 2021.

---

[18] Jie Lei, Licheng Yu, Tamara L Berg, and Mohit Bansal. Tvr: A large-scale dataset for video- subtitle moment retrieval. In ECCV, 2020. 2, 3, 5,
[19] Jie Lei, Licheng Yu, Tamara L Berg, and Mohit Bansal. What is more likely to happen next? video-and-language future event prediction. In EMNLP, 2020.
[20] Linjie Li, Yen-Chun Chen, Yu Cheng, Zhe Gan, Licheng Yu, and Jingjing Liu. Hero: Hier- archical encoder for video+ language onni-representation pre-trate trainiiinng.. In EMNLP, 2020. 3,
7
[21] Chunxi Liu, Qingming Huang, and Shuqiang Jiang. Query sensitive dynamic web video thumbnail generation. In ICIP, 3011. 3
[22] Wu Liu, Tao Mei, Yongdong Zhang, Cherry Che, and Jiebo Luo. Multi-task deep visual-semantic embedding for video thumbnail selection. CVPR, 2015. 5,
[23] Iyy Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019. 7
[24] Antoine Miech, Jean-Baptiste Alayrac, Lucas Smaira, Ivan Laptev, Josef Sivic, and Andrew Zisserman. End-to-end learning of visual representations from uncurated instructional videos. In CVPR, 2020.
[25] Antoine Miech, Dimitri Zhukov, Jean- CaantBaptstte Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic. Howtol00ml Learning text-video embedding by watching hundred million narrated video clips. In ICCV, 2019. 7, 9
[26] Guoshun Nan, Rui Qiao, Yao Xiao, Jun Liu, Sicong Leng, Hao Zhang, and Wei Lu. Interven- tional video grounding with dual contrastive learning. In CVPR, 2021. 10
[27] Megha Nawhal and Greg Mori. Activity graph transformer for temporal action localization. arXiv,, 2021.
[28] Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. Image transformer. In ICML. PMLR, 2018. 6
[29] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Vito,, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In NeurlPS, 2019. 7
[30] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. arXiv, 2021. 5,
[31] Michaela Regneri, Marcus Rohrbach, Dominikus Wetzel, Stefan Thater, Bernt Schiele, and Manfred Pinkal. Grounding action descriptions in videos. TACL, 2013. 1, 3
[32] Hamid Rezatofighi, Nathan Tsoi, Jun Young Gwak, Amir Sadeghian, Ian Reid, and Silvio Savarese. Generalized intersection over union: A metric and a loss for bounding box regression. In CVPR, 2019. 7
[33] Mrigank Rochan, Mahesh Kumar Krishna Reddy, Linwei Ye, and Yang Wang. Adaptive video highlight detection by learning from user history. In ECCV, 2020. 3
[34] Jos BTM Roerdink and Arnold Meijster. The watershed transform: Definitions, algorithms and parallelization strategies. Fundamenta informaticae, 2000. 8
[35] Dian Shao, Yu Xiong, Yue Zhao, Qingqiu Huang. Yu Qiao, and Dahua Lin. Find and focus: Retrieve and localize video events with natural language queries. In ECCV, 2018. 3,5
[36] Yale Song, Miriam Redi, Jordi Vallmitjana, and Alejandro Jaimes. To click or not to click: Automatic selection of beautiful thumbnails from videos. In CIKM, 2016.
1, 8
12

---

[37] Russell Stewart, Mykhaylo Andriluka, and Andrew Y Ng. End-to-end people detection in crowded scenes. In CVPR, 2016. 6
[38] Min Sun, Ali Farhadi, and Steve Seitz. Ranking domain-specific highlights by analyzing edited videos. In European conference on computer vision, pages 787-802. Springer, 2014. 1, 3,
[39] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurlPS, 2017.
[40] Huijuan Xu, Kun He, Bryan A Plummer, Leonid Sigal, Stan Sclaroff, and Kate Saenko. Multilevel language and vision integration for text-to-clip retrieval. In AAAI, 2019. 3, 5
[41] Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: large video description dataset for bridging video and language. In CVPR, 2016.
[42] Youngjae Yu, Jongseok Kim, and Gunhee Kim. A joint sequence fusion model for video question answering and retrieval. ECCV, 2018.
[43] Yitian Yuan, Lin Ma, and Wenwu Zhu, Sentence specified dynamic video thumbnail generation. In ACM MM, 2019. 1, 3,
[44] Bowen Zhang, Hexiang Hu, Joonseok Lee, Ming Zhao, Sheide Chammas, Vihan Jain, Eugene Ic, and Fei Sha. A hierarchical multi-modal encoder for moment localization in video corpus. arXiv preprint arXiv:2011.088 1.09046, 2020. 3
[45] Chuhan Zhang, Ankush Gupta, and Andrew Zisserman. Temporal query networks for fine- grained video understanding. In CVPR, 2021. 3
[46] Da Zhang, Xiyang Dai, Xin Wang, Yuan-Fang Wang, and Larry S Davis. Man: Moment alignment network for natural language moment retrieval via iterative graph adjustment. In CVPR, 2019. 3.5
[47] Hao Zhang, Aixin Sun, Wei Jing, and Joey Tianyi Zhou. Span-based localizing network for natural language video localization. In ACL, 2020. 3, 10
[48] Songyang Zhang, Houwen Peng, Jianlong Fu, and Jiebo Luo. Learning 2d temporal adjacent networks for moment localization with natural language. In AAAI, 2020. 3, 5, 10
[49] Yue Zhao, Yuanjun Xiong, Limin Wang. Zhirong Wu, Xiaoou Tang, and Dahua Lin, Temporal action detection with structured segment networks. In ICCV, 2017. 8
13


Reviews:
[{"review_id": "aoT6ot4EuTk", "reviewer": "Reviewer_zZwJ", "summary": "This paper collects the QVHIGHLIGHTS, a unified benchmark dataset for text-based moment retrieval and highlight detection in videos. It contains over 1000 Youtube videos with a) human-written free-form queries and their relevant moments, and b) five-point scale saliency scores for all query-relevant clips. Different from existing text-based moment retrieval datasets, QVHIGHLIGHTS has a smaller temporal bias on moments and allows a text query to match multiple disjoint moments. Compared to other highlight detection datasets, it presents temporally dense annotations dependent on queries. \nAs a baseline, they propose a Moment-DETR model which can do the two tasks jointly.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n1. The dataset unifies two tasks based on text queries -- moment retrieval and highlight detection, and show how joint training can affect the performance of the baseline model on these two tasks respectively.\n2.The collected dataset overcomes some of the limitations in other moment retrieval datasets by having less temporal bias and allowing a text query to match multiple disjoint moments.\n3. The dataset accounts for the subjectiveness in highlight detection and provide dense annotations.\n4. Good quality control through the qualification test and the verification between multiple workers.\n\n\nWeakness:\n1. Line 259-260 'we define a prediction to be positive if it has a high IoU with\none of the ground truth moments'.  It is not very clear here whether 'one of the ground truth moments' means one from 3 annotators or one from disjoint moments. \n2. For highlight detection: Line 262-263, 'we have ground-truth saliency scores from 3 users, we evaluate performance against each then take the average'. Does it mean the model's performance is dependent on the inter-user agreement?\n3. Missing baseline experiments/ablations: \n - The results from using only CLIP and Slow-Fast networks as visual encoder, as in how important the temporal information is in the two tasks.\n - Performance on moment retrieval after the model is pre-trained via ASR, to see how much improvement has supervised training brought.\n - Evaluation of SoTA models on moment retrieval which have official implementations online: 2D-TAN[1], DRN[2], CBP[3]\n4. Performance of the baseline model:\n - Since one advantage of the dataset is having queries matching disjoint moments, how well can the baseline model spotting all the disjoint moments given a query, is there a metric to evaluate it?\n - Table 1 shows the dataset has a wide range of granularity in the verbs (e.g, cut and report) and nouns(dog and weather), does the performance of the baseline model varies by the granularity?\n5. While the highlights are labelled on a five-point scale, is that only the 'very good' ones are used in training and evaluation?\n6. Related work: [4] uses a DETR-like architecture for temporal detection, and treat the task as a set-prediction task.\n\n[1] Learning 2D Temporal Localization Networks for Moment Localization with Natural Language. Zhang et al. AAAI 2020.\n[2] Local-Global Video-Text Interactions for Temporal Grounding. Mun et al. CVPR 2020.\n[3] Temporally Grounding Language Queries in Videos by Contextual Boundary-aware Prediction. Wang et al. AAAI 2020\n[4] Activity Graph Transformer for Temporal Action Localization. Nawhal et al. CVPR 2021.\n\nSummary:\n\nThe QVHIGHLIGHTS dataset addresses some limitations in current moment retrieval/highlight detection datasets. It is presented in a clear way and will be beneficial to future research in these areas. However, some detailed analysis on the dataset is missing, and more baseline experiments are needed to verify the design choice in the baseline architecture. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"ethical_concerns": "No", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6 hours"}, "extra_text": {}}, {"review_id": "SlMc6R9f6L", "reviewer": "Reviewer_sT4H", "summary": "In this paper, the authors focus on moment retrieval and highlight detection tasks from videos given text queries. They present the QVHIGHLIGHTS dataset which includes 10,000 videos collected from Youtube. Each video is annotated with a query, time span of a moment, and saliency scores.\nIn addition, the authors propose an Encode-Decoder Transformer model (Moment-DETR) that predicts moments and scores in an end-to-end manner. \nThe Moment-DETR model provides superior performance in comparison to previous methods on the QVHIGHLIGHTS dataset for moment temporal localization and highlight detection tasks.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- line 114: During data collection, how do you segment raw videos into short clips? Is it random or based on specific criteria?\n- In table 4, can you explain why removing L1 or gIoU losses decrease the highlight detection performance?\n- Can TVR and ANetCaptions datasets help in pre-training the MomentDetr model? (removing HD related losses)\n- Although the Moment-DETR model performs well on QVHIGHLIGHTS dataset, there is no evidence that it will generalize to other datasets. For instance, how will it perform on the TVR dataset in comparison to SOTA?\n- Authors extract video and text features using the CLIP model. Is there a way to use the nearest neighbor approach to evaluate moment retrieval (distance between text feature and clip features)? how well does it perform in comparison to Moment-DETR?\n\n\nTypos:\n- line 79: lsimiting -> limiting", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors described the technical limitations but I couldn't find any information regarding the negative societal impact of their work.", "review_text": "", "extra_scores": {"ethical_concerns": "There are no ethical issues with this paper.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "HafyMJiGgOw", "reviewer": "Reviewer_zVQi", "summary": "This paper introduces a new dataset using language queries to localize moments in videos. The paper also introduces a DETR-like baseline for this task. Several existing models and the baseline are run on the proposed dataset. I'm concerned the contributions will not be significant, the dataset is small (10k videos) and the baseline is an adaptation of existing methods to videos.", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall the paper is fairly well written, and easy to understand. The task of localizing moments in videos from language queries is interesting, important and would be quite valuable to solve. The code and dataset annotations are provided, which should help in reproducing and extending the work.\n\nHowever, the paper has a few weakness currently:\n\n(1) Throughout the paper, it is mentioned that the dataset is diverse, but I do not see any metrics of diversity of the dataset. It seems misleading to claim diversity without providing any support for it. Also, the paper does not have any mention of ethical concerns or societal impact. Given that such a dataset was created, especially using personal VLOG style videos, this seems like it should have some discussion.\n\n(2) The dataset is also fairly small by current standards, 10,000 videos is not many.\n\n(3) The proposed baseline approach is fairly straightforward adaptation of existing approaches, like DETR to videos. There is some value in seeing it work and providing a good baseline for the new dataset. It would also be good to compare the proposed baseline to previous works on other, existing datasets to help make comparisons.\n\nIn the checklist:\n\"Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [Yes] See Section 3.1.\"\n\nI do not see any estimated payments, wage, or time spent on the task in Section 3.1 or the paper or supplementary materials.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "There is some discussion on the limitations of the proposed pretraining, but otherwise there is no discussion on the limitations of the approach or dataset and no meaningful discussions on the societal impacts of the dataset.", "review_text": "", "extra_scores": {"needs_ethics_review": "Yes", "ethics_review_area": ["Inadequate Data and Algorithm Evaluation", "Inappropriate Potential Applications & Impact  (e.g., human rights concerns)"], "time_spent_reviewing": "2"}, "extra_text": {"ethical_concerns": "The paper does raise a few potential ethical concerns, regarding the dataset. The dataset consists of \"over 10,000 YouTube videos, covering a wide range of topics, from everyday activities and travel in lifestyle vlog videos to social and political activities in news videos\"\n\nThis is concerning because it can provide annotations that could be used to violate privacy (e.g., using personal vlog videos) or enabling other agencies to use this data to build models to understand vlog-style videos for undesirable purposes. Also having societal and political activities (e.g., protests as mentioned in the paper) in the data could be used for undesirable surveillance purposes.  Given that there are protests, there could also be violent imagery in this dataset (not mentioned if this has been removed or not).\n"}}, {"review_id": "CPYUigiH6v", "reviewer": "Reviewer_uoxL", "summary": "This paper introduces a new dataset for video moment retrieval and highlight detection named QVHighlights. The dataset contains around 10K YouTube videos. Each video is annotated with a free form text query and (i) the time intervals in the video corresponding to the query (multiple intervals can match the query) and (ii) for each annotated time interval a *saliency* scores (from 1 (Very bad) to 5 (very good)) describing how relevant are the individual frames within an annotated clip are to the query. \n\nIn addition to the dataset, the authors introduce a method named MomentDETR inspired from DETR in order to perform moment retrieval in their introduced dataset. The method performs better than existing techniques.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strengths \n\n- The paper is clearly written\n- The dataset is going to be useful to the community as it seems it is quite high quality (based on the description and the rater agreement). \n- I believe that the introduced method MomentDETR is a good extension to the DETR approach for moment retrieval in videos. \n- The pre-training strategy on weakly supervised data with ASR is interesting and leads to a substantial improvement.\n\n\n## Weaknesses\n\nOverall the paper is good but there a few points that could improve it further:\n\n- **Weakly supervised pretraining**: The weakly supervised pretraining seems to bring a substantial gain in performance (Table 3). However from what I understood, only a small dataset is used for this pretraining (only 5.4K videos). Have you thought about using larger datasets for this pretraining? How would that impact the performance of the method? One could for example use the HowTo100M dataset for that [22]. I believe that a scale study would be very valuable there even if not scaling to the full HowTo100M dataset (e.g. providing the performance with only 2.5K videos and 10K videos to see the impact of increasing the size of the pretraining). The VLOGs dataset [a] could also be used for that purpose.\n\n- **MomentDETR evaluation**: MomentDETR is an interesting method in itself. For that reason, I think it would be worthwhile to run MomentDETR on other moment retrieval benchmarks to show it is competitive on known benchmarks. I think one of the main issue of the paper is that it introduces a new dataset and a new method so it is hard to convince the reader that the method is indeed better than pas work such as [30,19,12,4,16] as we cannot compare direclty to some numbers provided in these papers. I do trust the authors for having run these baselines properly but I think that running the method on other datasets would increase the strength of the paper.\n\n- **Precision about the saliency loss**: \n  - How are $t_{high}$ and $t_{low}$ defined based on the scores from 1 to 5?  \n  - How important is the term using low/high versus the term using out/in. Could you provide an ablation about this in Table 4 (keeping all other elements i.e. L1/gIoU and CLS)? If the term low/high is indeed important that would justify more the need for annotating individual frames which is one of the specificity of the QVHighLights dataset.\n\n- **Highlight and saliency score**: Some frames within positive annotated segments are annotated as `Very bad` (and `Bad`) for the given query. This seems to contradict that these frames are indeed positive for that query. In that case why not removing these frames all together from the annotation and create more moments? It would also be useful to provide the score distribution (% of frames that receive a score of 1,2,3,4,5...). \n\n- Minor typos:\n  - L79: lsmiting -> limiting\n  - L100: sailiency -> saliency\n  - L326: detetcion -> detection\n\n- Missing related work:\n\n  - [a] From lifestyle vlogs to everyday interactions. Fouhey et al. CVPR18\n\n## Overall assessment \n\nOverall the paper makes good contributions. However I believe that these contributions could be strengthen following the advices provided in the weaknesses section. For now I lean towards acceptance and I am ready to increase my score if some/all concerns are addressed during the rebuttal.\n\n\n## Post rebuttal\n\nAfter reading other reviews and the author's response I am happy to support acceptance of this paper. I really encourage the authors to include all the new ablations and set of experiments requested by the reviewers as they will make for a stronger paper, which in turn will be more influential in the community. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have not commented on the limitations and societal impact of their work.\n\nSince the authors introduce a new dataset, I would recommend the authors to fill a datasheet for it (see [Datasheet for dataset, Gebru et al.](https://arxiv.org/abs/1803.09010))", "review_text": "", "extra_scores": {"ethical_concerns": "No ethical concerns apart from the datasheet request above.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3.5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors responded fully to the reviwers' concerns (new training, new baselines ...) and constructively to the ethics reviews. Following the rebuttal, all reviewers recommend acceptance.

Of course, the authors should update the paper and dataset as stated.

Congratulations!
Decision: Accept (Poster)