Detecting Moments and Highlights in Videos via Natural Language Queries

1 Introduction . Internet videos are growing at an unprecedented rate . Enabling users to efficiently search and browse these massive collections of videos is essential for improving user experience of online video platforms . While a good amount of work has been done in the area of natural language query based video search for complete videos ( i.e. , text-to-video retrieval [ 41 , 42 , 17 ] ) , returning the whole video is not always desirable , since they can be quite long ( e.g. , from few minutes to hours ) . Instead , users may want to locate precise moments within a video that are most relevant to their query or see highlights at a glance so that they can skip to relevant portions of the video easily . Many datasets [ 13 , 7 , 18 , 16 , 31 ] have been proposed for the first task of ‘ moment retrieval ’ – localizing moments in a video given a user query . However , most of the datasets are reported [ 4 , 18 ] to have a strong temporal bias , where more moments appear at the beginning of the videos than at the end . Meanwhile , for each video-query pair , all of the datasets provide annotations with only a single moment . In reality , there are often multiple moments , i.e. , several disjoint moments in a video , that are related to a given query . For the second task of ‘ highlight detection ’ , many datasets [ 38 , 12 , 36 , 8 ] are query-agnostic , where the detected highlights do not change for different input user queries . [ 22 , 43 ] are the two existing datasets that collect highlights based on user queries . However , only 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Glass is laying all over the street from broken windows beside other trash and debris in front of store buildings . A shark is swimming underwater . a small set of frames or clips are annotated ( 20 frames out of 331 seconds long videos in [ 22 ] or around 10 seconds clips out of 60 seconds video in [ 43 ] ) , limiting their ability to accurately learn and evaluate highlight detection methods . Lastly , although these two tasks of moment retrieval and highlight detection share many common characteristics ( e.g. , both require learning the similarity between user text query and video clips ) , they are typically studied separately , mostly due to the lack of annotations supporting both tasks in a single dataset . To address these issues , we collect QVHIGHLIGHTS , a unified benchmark dataset that supports query-based video moment retrieval and highlight detection . Based on over 10,000 YouTube videos covering a diverse range of topics ( from everyday activities and travel in lifestyle vlog videos to social and political activities in news videos ) , we collect high-quality annotations for both tasks . Figure 1 shows two examples from QVHIGHLIGHTS . For moment retrieval , we provide one or multiple disjoint moments for a query in a video , enabling a more realistic , accurate , and less-biased ( see Section 3.2 ) evaluation of moment retrieval methods . Within the annotated moments , we also provide a five-point Likert-scale ( from ‘ Very Good ’ to ‘ Very Bad ’ ) saliency/highlightness score annotation for each 2-second clip . This comprehensive saliency annotation gives more space for designing and evaluating query-based video highlight detection methods . Next , to present strong initial models for this task , we take inspiration from recent work such as DETR [ 3 ] for object detection , and propose Moment-DETR , an end-to-end transformer encoderdecoder architecture that views moment retrieval as a direct set prediction problem . With this method , we effectively eliminate the need for any manually-designed pre-processing ( e.g. , proposal generation ) or post-processing ( e.g. , non-maximum suppression ) steps commonly seen in moment retrieval methods . We further add a saliency ranking objective on top of the encoder outputs for highlight detection . While Moment-DETR does not encode any human prior in its design , our experiments show that it is still competitive when compared to highly-engineered architectures . Furthermore , with additional weakly-supervised pretraining from ASR captions , Moment-DETR substantially outperforms these strong methods . Lastly , we also provide detailed ablations and visualizations to help understand the inner workings of Moment-DETR . Overall , our contributions are 3-fold : ( i ) We collect the QVHIGHLIGHTS dataset with over 10,000 videos , annotated with human-written natural language queries , relevant moments , and saliency scores . ( ii ) We propose Moment-DETR to serve as a strong baseline for our dataset . With weakly supervised pretraining , Moment-DETR substantially outperforms several baselines , on both our proposed QVHIGHLIGHTS dataset and the moment retrieval dataset CharadesSTA [ 7 ] . ( iii ) We present detailed dataset analyses , model ablations and visualizations . For ablations , we examined various design choices of Moment-DETR as well as its pre-training strategy . We hope our work would inspire and encourage future work towards this important direction . 2 Related Work . Datasets and Tasks . Moment retrieval [ 13 , 7 , 18 ] requires localizing moments from a video given a natural language query . Various datasets [ 13 , 7 , 18 , 16 , 31 ] have been proposed or repurposed for the task . However , as shown in [ 13 , 4 , 18 ] , many of them have a strong temporal bias , where more moments are located at the beginning of the videos than the end . In Section 3.2 we show moments in QVHIGHLIGHTS distribute almost evenly over the videos . Meanwhile , while these datasets collect only a single moment for each query-video pair , we collect one or more moments . Highlight detection is another important task in our dataset . Most existing datasets [ 38 , 12 , 36 , 8 ] are query-agnostic , which do not provide customized highlights for a specific user query . [ 22 , 43 ] are the two known datasets that collect highlights based on user queries . However , they only annotate a small set of frames or clips , limiting their ability to accurately learn and evaluate highlight detection methods . In contrast , we provide a comprehensive five-point Likert-scale saliency/highlightness score annotation for all clips that are relevant to the queries . Besides , although these two tasks share some common characteristics , they are typically addressed separately using different benchmark datasets . In this work , we collect QVHIGHLIGHTS as a unified benchmark that supports both tasks . In Section 5.2 we also demonstrate that jointly detecting saliency is beneficial for retrieving moments . Methods . There are a wide range of approaches developed for addressing the moment retrieval and highlight detection tasks . For highlight detection , prior methods [ 38 , 22 , 12 , 21 , 33 ] are typically ranking-based , where models are trained to give higher scores for highlight frames or clips , via a hinge loss , cross-entropy loss , or reinforcement approaches . For moment retrieval , there are work that try to score generated moment proposals [ 13 , 35 , 4 , 48 , 46 , 40 ] , predict moment start-end indices [ 9 , 18 , 20 , 44 , 47 ] or regress moment coordinates [ 7 ] . However , most of them require a preprocessing ( e.g. , proposal generation ) or postprocessing step ( e.g. , non-maximum suppression ) that are hand-crafted , and are thus not end-to-end trainable . In this work , drawing inspiration from recent work on object detection [ 3 , 15 ] and video action detection [ 27 , 45 ] , we propose Moment-DETR that views moment retrieval as a direct set prediction problem . Moment-DETR takes video and user query representations as inputs , and directly outputs moment coordinates and saliency scores end-to-end , hence eliminating the need for any pre- or post-processing manually-designed human prior steps . 3 Dataset Collection and Analysis . Our QVHIGHLIGHTS dataset contains over 10,000 videos annotated with human written , free-form queries . Each query is associated with one or multiple variable-length moments in its corresponding video , and a comprehensive 5-point Likert-scale saliency annotation for each clip in the moments . In the following , we describe our data collection process and provide various data analyses . 3.1 Data Collection . Collecting videos . We would like to collect a set of videos that are less-edited and contain interesting and diverse content for user annotation . Following [ 6 , 19 ] , we start with user-created lifestyle vlog videos on YouTube . These are created by users from all over the world , showcasing various events and aspects of their life , from everyday activities , to travel and sightseeing , etc . These videos are captured via different devices ( e.g. , smartphones or GoPro ) with different view angles ( e.g. , firstperson or third-person ) , posing important challenges to computer vision systems . To further increase the diversity of the dataset , we also consider news videos that have large portions of ‘ raw footage ’ . These videos tend to cover more serious and world event topics such as natural disasters and protests . To harvest these videos , we use a list of queries , e.g. , ‘ daily vlog ’ , ‘ travel vlog ’ , ‘ news hurricane ’ , etc . We then download top videos that are 5-30 minutes long from YouTube ’ s search results , keeping videos that are uploaded after 2016 for better visual quality , and filtering out videos with a view count under 100 or with a very high dislike ratio . These raw videos are then segmented into 150-second short videos for annotation . Collecting user queries and relevant moments . To collect free-form natural language queries and their associated moments in the videos , we create an annotation task on Amazon Mechanical Turk . In this task , we present workers with a video and ask them to watch the video and write a query in standard English depicting interesting activities in the video . Next , we present the same worker with a grid of 2-second long clips segmented from the video , and ask them to select all clips from the grid relevant to the query . The selection can be done very efficiently via click for selecting a single clip and click-and-drag for selecting consecutive clips . This 2-second clip annotation protocol allows for more precise annotation than using 5-second clip as in [ 13 ] . Moreover , different from previous work [ 13 , 7 , 18 , 16 , 31 ] where only a single moment can be selected for a query-video pair , users can select multiple disjoint moments in our setup . To verify the quality of the moment annotation , we use a set of 600 query-video pairs , and collect 3 sets of moments for each query from different workers . We then calculate the Intersection-over-Union ( IoU ) between every pair of moments annotated for the same query , and take the average of the 3 IoU scores to check the inter-user agreement . We find that , for around 90 % of queries , their moments have an average IoU score higher than 0.9 , suggesting that the moments collected via our annotation process are of high inter-user agreement , thus high quality . Annotating saliency scores . The relevant moment annotation in the previous step tells us which clips in the videos correspond to the user queries . Though all selected clips are relevant to the query , they may still vary greatly in their saliency , how representative they are for the query , or whether they would make a good highlight . For example , we would expect a clip with a proper camera angle and lighting to be better than a clip with a lot of occlusions on the activities being queried , and therefore be a better highlight for the video . Thus , we create a second annotation task targeting at collecting the saliency scores for each relevant clip . We do not ask workers to select only a small set of clips as highlights [ 43 ] because many clips may look similar and be equally salient . Hence forcing people to pick only a few clips from these similar clips can cause confusion and degrade annotation quality . Specifically , we present all the selected clips in the first task along with the queries to another set of workers . For each clip , the workers are required to rate them in a Likert-scale system1 with five options , ‘ Very Good ’ , ‘ Good ’ , ‘ Fair ’ , ‘ Bad ’ , ‘ Very Bad ’ . As highlightness can be subjective , we collect ratings from 3 different workers and use all of them for evaluation . Quality control . To ensure data quality , we only allow workers who have done more than 500 HITs and with an approval rate of 95 % to participate in our annotation task . We also follow [ 18 ] to set up a qualification test . Our test contains seven multiple-choice questions ( see supplementary file for an example ) , and workers have to correctly answer all questions in order to qualify for the task . In total , 543 workers took the test , with a pass rate of 48 % . This qualification ensures high data quality – as mentioned earlier in this subsection , we observe high inter-user agreement of moment annotations . For query and saliency annotation , we pay workers $ 0.25 and $ 0.18 per query , respectively . The average hourly pay is around $ 11.00 . The whole collection process took about 3 months and cost approximately $ 16,000 .

Summary: This paper collects the QVHIGHLIGHTS, a unified benchmark dataset for text-based moment retrieval and highlight detection in videos. It contains over 1000 Youtube videos with a) human-written free-form queries and their relevant moments, and b) five-point scale saliency scores for all query-relevant clips. Different from existing text-based moment retrieval datasets, QVHIGHLIGHTS has a smaller temporal bias on moments and allows a text query to match multiple disjoint moments. Compared to other highlight detection datasets, it presents temporally dense annotations dependent on queries.  As a baseline, they propose a Moment-DETR model which can do the two tasks jointly.