CLIP-It! Language-Guided Video Summarization

1 Introduction . An effective video summary captures the essence of the video and provides a quick overview as an alternative to viewing the whole video ; it should be succinct yet representative of the entire video . Summarizing videos has many use cases - for example , viewers on YouTube may want to watch a 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . short summary of the video to assess its relevance . While a generic summary is useful for capturing the important scenes in a video , it is even more practical if the summary can be customized by the user . As seen in Fig . 1 , users should be able to indicate the concepts of the video they would like to see in the summary using natural language queries . Generic video summarization datasets such as TVSum [ 36 ] and SumMe [ 7 ] provide ground-truth annotations in the form of frame or shot-level importance scores specified by multiple annotators . Several learning-based methods reduce the task to a frame-wise score prediction problem . Sequence labeling methods [ 5 , 42 , 43 , 44 , 18 ] model variable-range dependencies between frames but fail to capture relationships across all frames simultaneously . While attention [ 3 ] and graph based [ 26 ] methods address this partly , they disregard ordering of the input frames which is useful when predicting scores . Moreover , these methods use only visual cues to produce the summaries and can not be customized with a natural language input . Another line of work , query-focused video summarization [ 32 ] , allows the users to customize the summary by specifying a query containing two concepts ( eg. , food and drinks ) . However , in their work the query can only be chosen from a fixed set of predefined concepts which limits the flexibility for the user . It is important to note that efforts in generic and query-focused video summarization have so far been disjoint , with no single method for both . Our key innovation is to unify these tasks in one language-guided framework . We introduce CLIP-It , a multimodal summarization model which takes two inputs , a video and a natural language text , and synthesizes a summary video conditioned on the text . In case of generic video summarization , the natural language text is a video description obtained using an off-the-shelf dense video captioning method . Alternatively , in the case of query-focused video summarization , the language input is a user-defined query . Unlike existing generic methods which only use visual cues , we show that adding language as an input leads to the “ discovery ” of relevant concepts and actions resulting in better summaries . Our method uses a Transformer with positional encoding , which can attend to all frames at once ( unlike LSTM based methods ) and also keep track of the ordering of frames ( unlike graph based methods ) . In contrast to existing query-focused methods [ 32 ] which only allow keyword based queries , we aim to enable open-ended natural language queries for users to customize video summaries . For example , as seen in Fig . 1 , using our method users can specify long and descriptive queries such as , “ All water bodies such as lakes , rivers , and waterfalls ” which is not possible with previous methods . Given an input video , CLIP-It generates a video summary guided by either a user-defined natural language query or a system generated description . It uses a Language-Guided Attention head to compute a joint representation of the image and language embeddings , and a Frame-Scoring Transformer to assign scores to individual frames in the video using the fused representations . Following [ 43 , 42 ] , the summary video is constructed from high scoring frames by converting framelevel scores to shot-level scores and using knapsack algorithm to fit the maximum number of high scoring shots in a timed window . Fig . 1 shows an output from our method . Given an hour long video of a national park tour , we generate a 2 minute generic summary comprising of all the important scenes in the video . Given the two language queries , our method picks the matching keyframes in both cases . We can train CLIP-It without ground-truth supervision by leveraging the reconstruction and diversity losses [ 9 , 31 ] . For generic video summarization , we evaluate our approach on the standard benchmarks , TVSum and SumMe . We achieve performance improvement on F1 score of nearly 3 % in the supervised setting and 4 % in the unsupervised setting on both datasets . We show large gains ( 5 % ) in the Transfer setting , where CLIP-It is trained and evaluated on disjoint sets of data . For the queryfocused scenario we evaluate on the QFVS dataset [ 33 ] , where we also achieve state-of-the-art results . To summarize our contributions , we introduce CLIP-It , a language-guided model that unifies generic and query-focused video summarization . Our approach uses language conditioning in the form of off-the-shelf video descriptions ( for generic summarization ) or user-defined natural language queries ( for query-focused summarization ) . We show that the Transformer design enables effective contextualization across frames , benefiting our tasks . We also demonstrate the impact of language guidance on generic summarization . Finally , we establish the new state-of-the-art on both generic and query-focused datasets in supervised and unsupervised settings . 2 Related Work . Generic Video Summarization . A video summary is a short synopsis created by stitching together important clips from the original video . Early works [ 21 , 35 , 34 ] referred to it as Video Skimming or Dynamic Video Summarization and used hand-designed features to generate summaries . Likewise , non-parametric unsupervised methods [ 13 , 16 , 23 , 19 , 20 , 28 ] used various heuristics to represent the importance of frames . Introduction of benchmark datasets such as TVSum [ 36 ] and SumMe [ 7 ] provided relevance scores for frames in videos annotated by users , resulting in multiple human generated summaries . This enabled automatic evaluation of video summarization techniques and has lead to the development of many supervised learning based methods [ 6 , 8 , 22 , 26 , 31 , 30 , 41 , 42 , 43 , 44 , 45 ] . These approaches capture high-level semantic information and outperform the heuristic unsupervised methods . Fully convolutional sequence networks [ 31 ] treat video summarization as a binary label prediction problem . Determinantal point processes [ 15 ] and LSTM [ 10 ] based approaches [ 5 , 18 , 42 , 43 , 44 ] model variable-range dependencies between frames . However , these are sequential and fail to capture relationships across all frames simultaneously . Attention [ 3 ] and graph based methods [ 26 ] address this issue by modeling relationships across all frames , but they disregard the ordering of frames in a video , which is also useful for summarization . Our method uses a Transformer [ 38 ] with positional encoding , which allows for joint attention across all frames while maintaining an ordering of the input sequence of frames . Some of the above works have supervised and unsupervised variants with modifications to the objective function . Specifically , for the unsupervised variant , they use reconstruction and diversity losses which do not require ground truth . We follow prior work in terms of using the same loss functions . Other notable unsupervised approaches include an adversarial LSTM based method [ 22 ] , a generative adversarial network to learn from unpaired data [ 30 ] , and a cycle consistent learning objective [ 41 ] . Video-Text Summarization . Plummer et al . [ 27 ] use image-language embeddings for generating video summaries but evaluate their approach in the text domain , and not on the generic video summarization benchmarks . Furthermore , they require text to be provided as input and don ’ t have a mechanism to generate captions if absent . On the other hand , our method works well with off-theshelf captions and we evaluate on both generic and query-focused benchmark datasets . Chen et al . [ 1 ] jointly train a text and video summarization network but rely on ground-truth text summaries . Query-Focused Video Summarization . Oftentimes users browsing videos on YouTube are looking for something specific so a generic summary might not suffice . In this case , there should be an option to customize the generated summary using a natural language query . Sharghi et al . [ 32 ] introduce the Query-Focused Video Summarization ( QFVS ) dataset for UT Egocentric [ 16 ] videos containing user-defined video summaries for a set of pre-defined concepts . Sharghi et al . [ 33 ] propose a memory network to attend over different video frames and shots with the user query as input . However , this recurrent attention mechanism precludes parallelization and limits modeling long-range dependencies , which is overcome by our Transformer architecture . Moreover , their method only works with the pre-defined set of keyword based queries in QFVS dataset . Since we use CLIP [ 29 ] to encode the language input and train our method on dense video descriptions , this allows users to define freeform queries at test time ( as seen in Fig . 1 ) . Other works [ 12 , 39 ] similarly condition the summary generation on keyword based queries but haven ’ t released their data . Transformers . Transformers [ 38 ] were introduced for neural machine translation and have since been applied to video-language tasks such as video-retrieval [ 4 ] and video captioning [ 17 , 48 ] . In this work we adapt transformers for video summarization . We modify self-attention [ 38 ] to a LanguageGuided Attention block that accepts inputs from two modalities . Additionally , our method also relies on CLIP [ 29 ] for extracting image and text features and a Bi-Modal Transformer [ 11 ] for dense video caption generation , both of which also have transformer backbone . 3 CLIP-It : Language-Guided Video Summarization . CLIP-It is a unified language-guided framework for both generic and query-focused video summarization . It takes an input video and a user-defined query or a system generated dense video caption and constructs a summary by selecting key frames from the video . First , we explain the intuition behind our approach . In the case of query-focused summarization , clearly , it is necessary to model the user query as input in order to produce an appropriate summary . In the case of generic video summarization no user query is available ; nonetheless , we show here that we can leverage the semantic information contained in associated natural language descriptions to guide the summarization process . Assuming we had a generic description that accompanies a video ( e.g. , A person is walking a dog . The person throws a ball . The dog runs after it . ) , we could leverage its semantic embedding to match it to the most relevant portions of the video . Such a description could be obtained automatically by generating dense video captions [ 11 ] . We first present an overview of our approach , CLIP-It , followed by a detailed description of the individual components . Overview . Our approach , CLIP-It , is outlined in Fig 2 . Given a video , we extract N frames denoted by Fi , i ∈ [ 1 , . . . N ] . We formulate the video summarization task as a per-frame binary classification problem . We embed the frames using a pretrained network fimg . If a query is provided ( in the form of a natural language string ) , we embed the query using a pretrained network ftxt . Alternatively , as seen in the figure , we use an off-the-shelf video captioning model to generate a dense video caption with M sentences denoted by Cj , j ∈ [ 1 , . . .M ] and embed each sentence using the pretrained network ftxt . Next , we compute language attended image embeddings I∗ using learned Language-Guided Multi-head Attention f∗img_txt . Finally , we train a Frame-Scoring Transformer which assigns scores to each frame in the video ( green indicating a high score and red indicating a low score ) . To construct the video summary during inference , we convert frame-level scores to shot-level scores and finally , use 0/1 knapsack algorithm to pick the key shots [ 43 ] . In the following , we describe the Language-Guided Attention and the Frame-Selection Transformer modules , followed by discussing the visual and language encoders . Language-Guided Attention . We use Language-Guided Multi-head Attention to efficiently fuse information across the video and language modalities and to infer long-term dependencies across both . Using a single attention head does not suffice as our goal is to allow all captions to attend to all frames in the video . We modify the Multi-Head Attention described in Vaswani et al . [ 38 ] to take in inputs from both modalities . We set Query Q , Key K , and Value V as follows , Q = fimg ( Fi ) , where i ∈ [ 1 , . . . N ] , K , V = ftxt ( Cj ) , where j ∈ [ 1 , . . .M ] , Language - Guided Attn . ( Q , K , V ) = Concat ( head1 , ... , headh ) WO , where headi = Attention ( QW Q i , KW K i , V W V i ) and Attention ( Q , K , V ) = softmax ( QKT√ dk ) V WQi , W K i , and W V i are learned parameter matrices and dk is the dimensions of K. The output of the Language-Guided Multi-Head Attention are the attended image embeddings , denoted as F ′ i . Frame-Scoring Transformer . Finally , it is also important to ensure that we do not include redundant information , e.g. , several key shots from the same event . To better model interactions across frames and contextualize them w.r.t . each other , we add a Frame-Scoring Transformer that takes image-text representations as input and outputs one score per frame . Based on the Transformer model [ 38 ] , this module assigns relevance scores to the attended image embeddings F ′ i . We feed F ′ i to the bottom of both the encoder and decoder stacks . Similar to [ 38 ] , we use positional encoding to insert information about the relative positions of the tokens in the sequence . We add positional encodings to the input embeddings at the bottom of the encoder and decoder stacks . Image Encoding . We encode the image using a pre-trained network fimg . We experiment with the following networks : GoogleNet [ 37 ] ( for a fair comparison to prior work ) , ResNet [ 8 ] , and CLIP [ 29 ] . Text Encoding . We encode the user-defined query or the system generated dense caption using a pre-trained network ftxt . In this case we tried the CLIP ( ViT and RN101 ) model . In case of generic video summarization , we generate dense video captions for the whole video in order to condition on language and incorporate added semantics into our video summarization pipeline . We use the Bi-Modal Transformer [ 11 ] dense video captioning model that generates a multi-sentence description given a video with audio . Since there are multiple sentences in the dense video caption , we first embed each sentence of the caption using the text encoder ftxt described above . They are then concatenated and fused using a multi-layer perceptron ( MLP ) .

Summary: The paper describes a video summarization model that is trained to use either user provided query or automatically generated dense  captioning to select key-frames as summaries.  At the core of the approach is an attention mechanism that computes similarity between frame embeddings and text embeddings. Subsequently, each frame representation is replaced by a weighted-average (softmaxed) of the language embeddings; key and value embeddings are from the text model while query embeddings are from the image model. Finally, a score is assigned to each frame via a standard transformer based model. A post-processing step converts the key frame scores to key-shot level summaries such that only 15% of the videos are selected. The overall model is trained with a weighted Binary cross-entropy loss.