CLIP-It! Language-Guided Video Summarization

1 Introduction . An effective video summary captures the essence of the video and provides a quick overview as an alternative to viewing the whole video ; it should be succinct yet representative of the entire video . Summarizing videos has many use cases - for example , viewers on YouTube may want to watch a 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . short summary of the video to assess its relevance . While a generic summary is useful for capturing the important scenes in a video , it is even more practical if the summary can be customized by the user . As seen in Fig . 1 , users should be able to indicate the concepts of the video they would like to see in the summary using natural language queries . Generic video summarization datasets such as TVSum [ 36 ] and SumMe [ 7 ] provide ground-truth annotations in the form of frame or shot-level importance scores specified by multiple annotators . Several learning-based methods reduce the task to a frame-wise score prediction problem . Sequence labeling methods [ 5 , 42 , 43 , 44 , 18 ] model variable-range dependencies between frames but fail to capture relationships across all frames simultaneously . While attention [ 3 ] and graph based [ 26 ] methods address this partly , they disregard ordering of the input frames which is useful when predicting scores . Moreover , these methods use only visual cues to produce the summaries and can not be customized with a natural language input . Another line of work , query-focused video summarization [ 32 ] , allows the users to customize the summary by specifying a query containing two concepts ( eg. , food and drinks ) . However , in their work the query can only be chosen from a fixed set of predefined concepts which limits the flexibility for the user . It is important to note that efforts in generic and query-focused video summarization have so far been disjoint , with no single method for both . Our key innovation is to unify these tasks in one language-guided framework . We introduce CLIP-It , a multimodal summarization model which takes two inputs , a video and a natural language text , and synthesizes a summary video conditioned on the text . In case of generic video summarization , the natural language text is a video description obtained using an off-the-shelf dense video captioning method . Alternatively , in the case of query-focused video summarization , the language input is a user-defined query . Unlike existing generic methods which only use visual cues , we show that adding language as an input leads to the “ discovery ” of relevant concepts and actions resulting in better summaries . Our method uses a Transformer with positional encoding , which can attend to all frames at once ( unlike LSTM based methods ) and also keep track of the ordering of frames ( unlike graph based methods ) . In contrast to existing query-focused methods [ 32 ] which only allow keyword based queries , we aim to enable open-ended natural language queries for users to customize video summaries . For example , as seen in Fig . 1 , using our method users can specify long and descriptive queries such as , “ All water bodies such as lakes , rivers , and waterfalls ” which is not possible with previous methods . Given an input video , CLIP-It generates a video summary guided by either a user-defined natural language query or a system generated description . It uses a Language-Guided Attention head to compute a joint representation of the image and language embeddings , and a Frame-Scoring Transformer to assign scores to individual frames in the video using the fused representations . Following [ 43 , 42 ] , the summary video is constructed from high scoring frames by converting framelevel scores to shot-level scores and using knapsack algorithm to fit the maximum number of high scoring shots in a timed window . Fig . 1 shows an output from our method . Given an hour long video of a national park tour , we generate a 2 minute generic summary comprising of all the important scenes in the video . Given the two language queries , our method picks the matching keyframes in both cases . We can train CLIP-It without ground-truth supervision by leveraging the reconstruction and diversity losses [ 9 , 31 ] . For generic video summarization , we evaluate our approach on the standard benchmarks , TVSum and SumMe . We achieve performance improvement on F1 score of nearly 3 % in the supervised setting and 4 % in the unsupervised setting on both datasets . We show large gains ( 5 % ) in the Transfer setting , where CLIP-It is trained and evaluated on disjoint sets of data . For the queryfocused scenario we evaluate on the QFVS dataset [ 33 ] , where we also achieve state-of-the-art results . To summarize our contributions , we introduce CLIP-It , a language-guided model that unifies generic and query-focused video summarization . Our approach uses language conditioning in the form of off-the-shelf video descriptions ( for generic summarization ) or user-defined natural language queries ( for query-focused summarization ) . We show that the Transformer design enables effective contextualization across frames , benefiting our tasks . We also demonstrate the impact of language guidance on generic summarization . Finally , we establish the new state-of-the-art on both generic and query-focused datasets in supervised and unsupervised settings . 2 Related Work . Generic Video Summarization . A video summary is a short synopsis created by stitching together important clips from the original video . Early works [ 21 , 35 , 34 ] referred to it as Video Skimming or Dynamic Video Summarization and used hand-designed features to generate summaries . Likewise , non-parametric unsupervised methods [ 13 , 16 , 23 , 19 , 20 , 28 ] used various heuristics to represent the importance of frames . Introduction of benchmark datasets such as TVSum [ 36 ] and SumMe [ 7 ] provided relevance scores for frames in videos annotated by users , resulting in multiple human generated summaries . This enabled automatic evaluation of video summarization techniques and has lead to the development of many supervised learning based methods [ 6 , 8 , 22 , 26 , 31 , 30 , 41 , 42 , 43 , 44 , 45 ] . These approaches capture high-level semantic information and outperform the heuristic unsupervised methods . Fully convolutional sequence networks [ 31 ] treat video summarization as a binary label prediction problem . Determinantal point processes [ 15 ] and LSTM [ 10 ] based approaches [ 5 , 18 , 42 , 43 , 44 ] model variable-range dependencies between frames . However , these are sequential and fail to capture relationships across all frames simultaneously . Attention [ 3 ] and graph based methods [ 26 ] address this issue by modeling relationships across all frames , but they disregard the ordering of frames in a video , which is also useful for summarization . Our method uses a Transformer [ 38 ] with positional encoding , which allows for joint attention across all frames while maintaining an ordering of the input sequence of frames . Some of the above works have supervised and unsupervised variants with modifications to the objective function . Specifically , for the unsupervised variant , they use reconstruction and diversity losses which do not require ground truth . We follow prior work in terms of using the same loss functions . Other notable unsupervised approaches include an adversarial LSTM based method [ 22 ] , a generative adversarial network to learn from unpaired data [ 30 ] , and a cycle consistent learning objective [ 41 ] . Video-Text Summarization . Plummer et al . [ 27 ] use image-language embeddings for generating video summaries but evaluate their approach in the text domain , and not on the generic video summarization benchmarks . Furthermore , they require text to be provided as input and don ’ t have a mechanism to generate captions if absent . On the other hand , our method works well with off-theshelf captions and we evaluate on both generic and query-focused benchmark datasets . Chen et al . [ 1 ] jointly train a text and video summarization network but rely on ground-truth text summaries . Query-Focused Video Summarization . Oftentimes users browsing videos on YouTube are looking for something specific so a generic summary might not suffice . In this case , there should be an option to customize the generated summary using a natural language query . Sharghi et al . [ 32 ] introduce the Query-Focused Video Summarization ( QFVS ) dataset for UT Egocentric [ 16 ] videos containing user-defined video summaries for a set of pre-defined concepts . Sharghi et al . [ 33 ] propose a memory network to attend over different video frames and shots with the user query as input . However , this recurrent attention mechanism precludes parallelization and limits modeling long-range dependencies , which is overcome by our Transformer architecture . Moreover , their method only works with the pre-defined set of keyword based queries in QFVS dataset . Since we use CLIP [ 29 ] to encode the language input and train our method on dense video descriptions , this allows users to define freeform queries at test time ( as seen in Fig . 1 ) . Other works [ 12 , 39 ] similarly condition the summary generation on keyword based queries but haven ’ t released their data . Transformers . Transformers [ 38 ] were introduced for neural machine translation and have since been applied to video-language tasks such as video-retrieval [ 4 ] and video captioning [ 17 , 48 ] . In this work we adapt transformers for video summarization . We modify self-attention [ 38 ] to a LanguageGuided Attention block that accepts inputs from two modalities . Additionally , our method also relies on CLIP [ 29 ] for extracting image and text features and a Bi-Modal Transformer [ 11 ] for dense video caption generation , both of which also have transformer backbone . 3 CLIP-It : Language-Guided Video Summarization . CLIP-It is a unified language-guided framework for both generic and query-focused video summarization . It takes an input video and a user-defined query or a system generated dense video caption and constructs a summary by selecting key frames from the video . First , we explain the intuition behind our approach . In the case of query-focused summarization , clearly , it is necessary to model the user query as input in order to produce an appropriate summary . In the case of generic video summarization no user query is available ; nonetheless , we show here that we can leverage the semantic information contained in associated natural language descriptions to guide the summarization process . Assuming we had a generic description that accompanies a video ( e.g. , A person is walking a dog . The person throws a ball . The dog runs after it . ) , we could leverage its semantic embedding to match it to the most relevant portions of the video . Such a description could be obtained automatically by generating dense video captions [ 11 ] . We first present an overview of our approach , CLIP-It , followed by a detailed description of the individual components . Overview . Our approach , CLIP-It , is outlined in Fig 2 . Given a video , we extract N frames denoted by Fi , i ∈ [ 1 , . . . N ] . We formulate the video summarization task as a per-frame binary classification problem . We embed the frames using a pretrained network fimg . If a query is provided ( in the form of a natural language string ) , we embed the query using a pretrained network ftxt . Alternatively , as seen in the figure , we use an off-the-shelf video captioning model to generate a dense video caption with M sentences denoted by Cj , j ∈ [ 1 , . . .M ] and embed each sentence using the pretrained network ftxt . Next , we compute language attended image embeddings I∗ using learned Language-Guided Multi-head Attention f∗img_txt . Finally , we train a Frame-Scoring Transformer which assigns scores to each frame in the video ( green indicating a high score and red indicating a low score ) . To construct the video summary during inference , we convert frame-level scores to shot-level scores and finally , use 0/1 knapsack algorithm to pick the key shots [ 43 ] . In the following , we describe the Language-Guided Attention and the Frame-Selection Transformer modules , followed by discussing the visual and language encoders . Language-Guided Attention . We use Language-Guided Multi-head Attention to efficiently fuse information across the video and language modalities and to infer long-term dependencies across both . Using a single attention head does not suffice as our goal is to allow all captions to attend to all frames in the video . We modify the Multi-Head Attention described in Vaswani et al . [ 38 ] to take in inputs from both modalities . We set Query Q , Key K , and Value V as follows , Q = fimg ( Fi ) , where i ∈ [ 1 , . . . N ] , K , V = ftxt ( Cj ) , where j ∈ [ 1 , . . .M ] , Language - Guided Attn . ( Q , K , V ) = Concat ( head1 , ... , headh ) WO , where headi = Attention ( QW Q i , KW K i , V W V i ) and Attention ( Q , K , V ) = softmax ( QKT√ dk ) V WQi , W K i , and W V i are learned parameter matrices and dk is the dimensions of K. The output of the Language-Guided Multi-Head Attention are the attended image embeddings , denoted as F ′ i . Frame-Scoring Transformer . Finally , it is also important to ensure that we do not include redundant information , e.g. , several key shots from the same event . To better model interactions across frames and contextualize them w.r.t . each other , we add a Frame-Scoring Transformer that takes image-text representations as input and outputs one score per frame . Based on the Transformer model [ 38 ] , this module assigns relevance scores to the attended image embeddings F ′ i . We feed F ′ i to the bottom of both the encoder and decoder stacks . Similar to [ 38 ] , we use positional encoding to insert information about the relative positions of the tokens in the sequence . We add positional encodings to the input embeddings at the bottom of the encoder and decoder stacks . Image Encoding . We encode the image using a pre-trained network fimg . We experiment with the following networks : GoogleNet [ 37 ] ( for a fair comparison to prior work ) , ResNet [ 8 ] , and CLIP [ 29 ] . Text Encoding . We encode the user-defined query or the system generated dense caption using a pre-trained network ftxt . In this case we tried the CLIP ( ViT and RN101 ) model . In case of generic video summarization , we generate dense video captions for the whole video in order to condition on language and incorporate added semantics into our video summarization pipeline . We use the Bi-Modal Transformer [ 11 ] dense video captioning model that generates a multi-sentence description given a video with audio . Since there are multiple sentences in the dense video caption , we first embed each sentence of the caption using the text encoder ftxt described above . They are then concatenated and fused using a multi-layer perceptron ( MLP ) .

Summary: This paper unified the generic video summarization method and the query-based video summarization method. Specifically, the paper proposed a language guided video summarzation approach that takes both video frames and text as input and summarizes the video frames under the influence of the text. For generic video summarization, the model relies on a video captioning method to provide the text guidance. The proposed approach achieves superior performance on both generic video summarization datasets and the query-based video summarization datasets.