Detecting Moments and Highlights in Videos via Natural Language Queries

1 Introduction . Internet videos are growing at an unprecedented rate . Enabling users to efficiently search and browse these massive collections of videos is essential for improving user experience of online video platforms . While a good amount of work has been done in the area of natural language query based video search for complete videos ( i.e. , text-to-video retrieval [ 41 , 42 , 17 ] ) , returning the whole video is not always desirable , since they can be quite long ( e.g. , from few minutes to hours ) . Instead , users may want to locate precise moments within a video that are most relevant to their query or see highlights at a glance so that they can skip to relevant portions of the video easily . Many datasets [ 13 , 7 , 18 , 16 , 31 ] have been proposed for the first task of ‘ moment retrieval ’ – localizing moments in a video given a user query . However , most of the datasets are reported [ 4 , 18 ] to have a strong temporal bias , where more moments appear at the beginning of the videos than at the end . Meanwhile , for each video-query pair , all of the datasets provide annotations with only a single moment . In reality , there are often multiple moments , i.e. , several disjoint moments in a video , that are related to a given query . For the second task of ‘ highlight detection ’ , many datasets [ 38 , 12 , 36 , 8 ] are query-agnostic , where the detected highlights do not change for different input user queries . [ 22 , 43 ] are the two existing datasets that collect highlights based on user queries . However , only 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Glass is laying all over the street from broken windows beside other trash and debris in front of store buildings . A shark is swimming underwater . a small set of frames or clips are annotated ( 20 frames out of 331 seconds long videos in [ 22 ] or around 10 seconds clips out of 60 seconds video in [ 43 ] ) , limiting their ability to accurately learn and evaluate highlight detection methods . Lastly , although these two tasks of moment retrieval and highlight detection share many common characteristics ( e.g. , both require learning the similarity between user text query and video clips ) , they are typically studied separately , mostly due to the lack of annotations supporting both tasks in a single dataset . To address these issues , we collect QVHIGHLIGHTS , a unified benchmark dataset that supports query-based video moment retrieval and highlight detection . Based on over 10,000 YouTube videos covering a diverse range of topics ( from everyday activities and travel in lifestyle vlog videos to social and political activities in news videos ) , we collect high-quality annotations for both tasks . Figure 1 shows two examples from QVHIGHLIGHTS . For moment retrieval , we provide one or multiple disjoint moments for a query in a video , enabling a more realistic , accurate , and less-biased ( see Section 3.2 ) evaluation of moment retrieval methods . Within the annotated moments , we also provide a five-point Likert-scale ( from ‘ Very Good ’ to ‘ Very Bad ’ ) saliency/highlightness score annotation for each 2-second clip . This comprehensive saliency annotation gives more space for designing and evaluating query-based video highlight detection methods . Next , to present strong initial models for this task , we take inspiration from recent work such as DETR [ 3 ] for object detection , and propose Moment-DETR , an end-to-end transformer encoderdecoder architecture that views moment retrieval as a direct set prediction problem . With this method , we effectively eliminate the need for any manually-designed pre-processing ( e.g. , proposal generation ) or post-processing ( e.g. , non-maximum suppression ) steps commonly seen in moment retrieval methods . We further add a saliency ranking objective on top of the encoder outputs for highlight detection . While Moment-DETR does not encode any human prior in its design , our experiments show that it is still competitive when compared to highly-engineered architectures . Furthermore , with additional weakly-supervised pretraining from ASR captions , Moment-DETR substantially outperforms these strong methods . Lastly , we also provide detailed ablations and visualizations to help understand the inner workings of Moment-DETR . Overall , our contributions are 3-fold : ( i ) We collect the QVHIGHLIGHTS dataset with over 10,000 videos , annotated with human-written natural language queries , relevant moments , and saliency scores . ( ii ) We propose Moment-DETR to serve as a strong baseline for our dataset . With weakly supervised pretraining , Moment-DETR substantially outperforms several baselines , on both our proposed QVHIGHLIGHTS dataset and the moment retrieval dataset CharadesSTA [ 7 ] . ( iii ) We present detailed dataset analyses , model ablations and visualizations . For ablations , we examined various design choices of Moment-DETR as well as its pre-training strategy . We hope our work would inspire and encourage future work towards this important direction . 2 Related Work . Datasets and Tasks . Moment retrieval [ 13 , 7 , 18 ] requires localizing moments from a video given a natural language query . Various datasets [ 13 , 7 , 18 , 16 , 31 ] have been proposed or repurposed for the task . However , as shown in [ 13 , 4 , 18 ] , many of them have a strong temporal bias , where more moments are located at the beginning of the videos than the end . In Section 3.2 we show moments in QVHIGHLIGHTS distribute almost evenly over the videos . Meanwhile , while these datasets collect only a single moment for each query-video pair , we collect one or more moments . Highlight detection is another important task in our dataset . Most existing datasets [ 38 , 12 , 36 , 8 ] are query-agnostic , which do not provide customized highlights for a specific user query . [ 22 , 43 ] are the two known datasets that collect highlights based on user queries . However , they only annotate a small set of frames or clips , limiting their ability to accurately learn and evaluate highlight detection methods . In contrast , we provide a comprehensive five-point Likert-scale saliency/highlightness score annotation for all clips that are relevant to the queries . Besides , although these two tasks share some common characteristics , they are typically addressed separately using different benchmark datasets . In this work , we collect QVHIGHLIGHTS as a unified benchmark that supports both tasks . In Section 5.2 we also demonstrate that jointly detecting saliency is beneficial for retrieving moments . Methods . There are a wide range of approaches developed for addressing the moment retrieval and highlight detection tasks . For highlight detection , prior methods [ 38 , 22 , 12 , 21 , 33 ] are typically ranking-based , where models are trained to give higher scores for highlight frames or clips , via a hinge loss , cross-entropy loss , or reinforcement approaches . For moment retrieval , there are work that try to score generated moment proposals [ 13 , 35 , 4 , 48 , 46 , 40 ] , predict moment start-end indices [ 9 , 18 , 20 , 44 , 47 ] or regress moment coordinates [ 7 ] . However , most of them require a preprocessing ( e.g. , proposal generation ) or postprocessing step ( e.g. , non-maximum suppression ) that are hand-crafted , and are thus not end-to-end trainable . In this work , drawing inspiration from recent work on object detection [ 3 , 15 ] and video action detection [ 27 , 45 ] , we propose Moment-DETR that views moment retrieval as a direct set prediction problem . Moment-DETR takes video and user query representations as inputs , and directly outputs moment coordinates and saliency scores end-to-end , hence eliminating the need for any pre- or post-processing manually-designed human prior steps . 3 Dataset Collection and Analysis . Our QVHIGHLIGHTS dataset contains over 10,000 videos annotated with human written , free-form queries . Each query is associated with one or multiple variable-length moments in its corresponding video , and a comprehensive 5-point Likert-scale saliency annotation for each clip in the moments . In the following , we describe our data collection process and provide various data analyses . 3.1 Data Collection . Collecting videos . We would like to collect a set of videos that are less-edited and contain interesting and diverse content for user annotation . Following [ 6 , 19 ] , we start with user-created lifestyle vlog videos on YouTube . These are created by users from all over the world , showcasing various events and aspects of their life , from everyday activities , to travel and sightseeing , etc . These videos are captured via different devices ( e.g. , smartphones or GoPro ) with different view angles ( e.g. , firstperson or third-person ) , posing important challenges to computer vision systems . To further increase the diversity of the dataset , we also consider news videos that have large portions of ‘ raw footage ’ . These videos tend to cover more serious and world event topics such as natural disasters and protests . To harvest these videos , we use a list of queries , e.g. , ‘ daily vlog ’ , ‘ travel vlog ’ , ‘ news hurricane ’ , etc . We then download top videos that are 5-30 minutes long from YouTube ’ s search results , keeping videos that are uploaded after 2016 for better visual quality , and filtering out videos with a view count under 100 or with a very high dislike ratio . These raw videos are then segmented into 150-second short videos for annotation . Collecting user queries and relevant moments . To collect free-form natural language queries and their associated moments in the videos , we create an annotation task on Amazon Mechanical Turk . In this task , we present workers with a video and ask them to watch the video and write a query in standard English depicting interesting activities in the video . Next , we present the same worker with a grid of 2-second long clips segmented from the video , and ask them to select all clips from the grid relevant to the query . The selection can be done very efficiently via click for selecting a single clip and click-and-drag for selecting consecutive clips . This 2-second clip annotation protocol allows for more precise annotation than using 5-second clip as in [ 13 ] . Moreover , different from previous work [ 13 , 7 , 18 , 16 , 31 ] where only a single moment can be selected for a query-video pair , users can select multiple disjoint moments in our setup . To verify the quality of the moment annotation , we use a set of 600 query-video pairs , and collect 3 sets of moments for each query from different workers . We then calculate the Intersection-over-Union ( IoU ) between every pair of moments annotated for the same query , and take the average of the 3 IoU scores to check the inter-user agreement . We find that , for around 90 % of queries , their moments have an average IoU score higher than 0.9 , suggesting that the moments collected via our annotation process are of high inter-user agreement , thus high quality . Annotating saliency scores . The relevant moment annotation in the previous step tells us which clips in the videos correspond to the user queries . Though all selected clips are relevant to the query , they may still vary greatly in their saliency , how representative they are for the query , or whether they would make a good highlight . For example , we would expect a clip with a proper camera angle and lighting to be better than a clip with a lot of occlusions on the activities being queried , and therefore be a better highlight for the video . Thus , we create a second annotation task targeting at collecting the saliency scores for each relevant clip . We do not ask workers to select only a small set of clips as highlights [ 43 ] because many clips may look similar and be equally salient . Hence forcing people to pick only a few clips from these similar clips can cause confusion and degrade annotation quality . Specifically , we present all the selected clips in the first task along with the queries to another set of workers . For each clip , the workers are required to rate them in a Likert-scale system1 with five options , ‘ Very Good ’ , ‘ Good ’ , ‘ Fair ’ , ‘ Bad ’ , ‘ Very Bad ’ . As highlightness can be subjective , we collect ratings from 3 different workers and use all of them for evaluation . Quality control . To ensure data quality , we only allow workers who have done more than 500 HITs and with an approval rate of 95 % to participate in our annotation task . We also follow [ 18 ] to set up a qualification test . Our test contains seven multiple-choice questions ( see supplementary file for an example ) , and workers have to correctly answer all questions in order to qualify for the task . In total , 543 workers took the test , with a pass rate of 48 % . This qualification ensures high data quality – as mentioned earlier in this subsection , we observe high inter-user agreement of moment annotations . For query and saliency annotation , we pay workers $ 0.25 and $ 0.18 per query , respectively . The average hourly pay is around $ 11.00 . The whole collection process took about 3 months and cost approximately $ 16,000 .

Summary: In this paper, the authors focus on moment retrieval and highlight detection tasks from videos given text queries. They present the QVHIGHLIGHTS dataset which includes 10,000 videos collected from Youtube. Each video is annotated with a query, time span of a moment, and saliency scores. In addition, the authors propose an Encode-Decoder Transformer model (Moment-DETR) that predicts moments and scores in an end-to-end manner.  The Moment-DETR model provides superior performance in comparison to previous methods on the QVHIGHLIGHTS dataset for moment temporal localization and highlight detection tasks.