CLIP-It! Language-Guided Video Summarization
Authors: Medhini Narasimhan
Anna Rohrbach
Trevor Darrell
Venue: NeurIPS 2021 Poster (2021)
Keywords: video summarization
language-guided video summarization
query-focused video summarization
multimodal video summarization

Abstract: A generic video summary is an abridged version of a video that conveys the whole story and features the most important scenes. Yet the importance of scenes in a video is often subjective, and users should have the option of customizing the summary by using natural language to specify what is important to them. Further, existing models for fully automatic generic summarization have not exploited available language models, which can serve as an effective prior for saliency. This work introduces CLIP-It, a single framework for addressing both generic and query-focused video summarization, typically approached separately in the literature. We propose a language-guided multimodal transformer that learns to score frames in a video based on their importance relative to one another and their correlation with a user-defined query (for query-focused summarization) or an automatically generated dense video caption (for generic video summarization). Our model can be extended to the unsupervised setting by training without ground-truth supervision. We outperform baselines and prior work by a significant margin on both standard video summarization datasets (TVSum and SumMe) and a query-focused video summarization dataset (QFVS). Particularly, we achieve large improvements in the transfer setting, attesting to our method's strong generalization capabilities.

CLIP-It!
Language-Guided Video Summarization
Medhini Narasimhan
Anna Rohrbach Trevor Darrell
University of California, Berkeley
{medhini, anna. rohrbcch, beevordareellJbbzreeleyy https://medhini gi thub. it/iio//lllo io/clipiit
Generic Summary
Query: All the scenes containing restaurants and shopping centers
Day Tour
Query: All water bodies such lakes, rivers, and waterfalls
Figure 1: We introduce CLIP-I,, language-guun -uudedd multimodal transformer for generic and query-focused video summarization. figure shows results from our method. Given a day-long video of national park tour, generic summary (top) is video with relevant and diverse keyframes. When using the query "All the scenes containing restaurants and shopping centers", the generated soccssf summary includes all the matching seenes. Similarly, the query "All water bodics such lakes, rivers, and waterfalls", yields short summary containing all the bodies present in the vidco.
Abstract
A generic video summary is an abridged version of video that conveys the whole story and features the most important scenes. Yet the importance of scenes in a video is often subjective, and users should have the option of customizing the summary by using natural language to specify what is important them. Further, existing models for fully automatic generic summarization have not exploited available language models, which can serve as effective prior for saliency. This work introduces CL,P-It,a a single framework for addressing both generic and query- focused video summarization, typically approached separately in the literature. We geguugegggidddd propose a language- guided multimodal transformer that learns to score frames video based on their importance relative to one another and their correlation with user-defined query (for query-focused summarization) or an automatically generated dense video caption (for generic video summarization), Our model can be extended to the unsupervised setting by training without ground-truth supervision. We outperform baselines and prior work by significant margin on both standard video summarization datasets (TVSum and SumMe) and query-focused video summarization dataset (QFVS). Particularly, we achieve large improvements in the transfer setting, attesting to our method's 's strong generalization capabilities.
Introduction
effective video summary captures the essence the video and provides quick overview as an alternative to viewing the whole video; it should be succinct yet representative of the entire video. Summarizing videos has many use cases for example, viewers on YouTube may want watch
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

short summary of the video to assess its relevance. While generic summary is useful for capturing the important scenes in a video, is even more practical if the summary can be customized by the user. As seen in Fig. 1, users should be able to indicate the concepts of the video they would like to see in the summary using natural language queries.
Generic video summarization datasets such as TVSum [36] and SumMe [7] provide ground-truth annotations in the form of frame shot-level importance scores specified by multiple annotators. Several learning-based methods reduce the task to a frame-wise score prediction problem. Sequence labeling methods [5, 42, 43, 44, 18] model variable-range dependencies between frames but fail to capture relationships across all frames simultaneously. While attention [3] and graph based [26] methods address this partly, they disregard ordering of the input frames which is useful when predicting scores. Moreover, these methods use only visual cues to produce the summaries and cannot be customized with a natural language input. Another line of work, query-focused video summarization [32], allows the users to customize the summary by specifying query containing two concepts (eg., food and drinks). However, in their work the query can only be chosen from a fixed set of predefined concepts which limits the flexibility for the user.
It important note that efforts in generic and query-focused video summarization have so far been disjoint, with no single method for both. Our key innovation is to unify these tasks in one taaguage-guide framework. We introduce CLIP-It, a multimodal summarization model which takes two inputs, a video and natural language text, and synthesizes a summary video conditioned on the text. In case of generic video summarization, the natural language text is video description obtained using an off-the-shelf dense video captioning method. Alternatively, in the case of query-focused video summarization, the language input is user-defined query. Unlike existing generic methods which only use visual cues, we show that adding language as an input leads to the "discovery" of relevant concepts and actions resulting in better summaries. Our method uses a Transformer with positional encoding, which can attend to all frames at once (unlike LSTM based methods) and also keep track of the ordering of frames (unlike graph based methods). In contrast to existing query-focused methods [32] which only allow keyword based queries, we aim to enable open-ended natural language queries for users to customize video summaries. For example, as seen in Fig. 1, using method users can specify long and descriptive queries such as, "All water bodies such as lakes, rivers, and waterfalls" which is not possible with previous methods.
Given an input video, CLIP-It generates a video summary guided by either a user-defined natural language query or a system generated description. It uses a Language-Guie Guided Attention head to compute joint representation of the image and language embeddings, and Frame-Scoring Transformer to assign scores to individual frames in the video using the fused representations. Following [43, 42], the summary video is constructed from high scoring frames by converting frame- level scores to shot- scores and using knapsack algorithm to fit the maximum number of high scoring shots in timed window. Fig. 1 shows output from our method. Given an hour long video of a national park tour, we generate a 2 minute generic summary comprising of all the important scenes in the video. Given the two language queries, our method picks the matching keyframes in both cases. We can train CLIP-It without ground-truth supervision by leveraging the reconstruction and diversity losses [9, 31]. For generic video summarization, we evaluate our approach benchmarks, TVSum and SumMe. We achieve performance improvement on F1 score of nearly 3% in the supervised setting and 4% in the unsupervised setting on both datasets. We show large gains (5%) in the Transfer setting, where CLIP-It is trained and evaluated on disjoint sets of data. For the query- focused scenario we evaluate on the QFVS dataset [33], where we also achieve state-of-the-ar results.
the standard
To summarize our contributions, we introduce CLIP-It, a language- model that unifies generic and focuusf video summarization. Our approach uses language conditioning in the form of off-the-shelf video descriptions (for generic summarization) or user-defined natural language queries (for query-focused summarization). We show that the Transformer design enables effective contextualization across frames, benefiting our tasks. We also demonstrate the impact of language guidance on generic summarization. Finally, we establish the new state-of-the-an -ftthe-arttttttt both generic and query-focused datasets in supervised and unsupervised settings.
2 Related Work
Generic Video Summarization. A video summary is short synopsis created by stitching together important clips from the original video. Early works [21, 35, 34] referred to as Video Skimming or Dynamic Video Summarization and used hand designed features to generate summaries. Likewise,
2

---

non aa--aaamtttic unsupervised methods [13, 16, 23, 19. 20, 28] used various heuristics to represent the importance of frames. Introduction of benchmark datasets such as TVSum [36] and SumMe [7] provided relevance scores for frames in videos annotated by users, resulting in multiple human generated summaries. This enabled automatic evaluation of video summarization techniques and has lead to the development of many supervised learning based methods [6, 8, 22, 26, 31, 30, 41, 42, 43, 44,44,4 45J. These approaches capture high-level semantic information and outperform the heuristic unsupervised methods. Fully convolutional sequence networks [31] treat video summarization as a binary label prediction problem. Determinantal point processes [15] and LSTM [10] based approaches [5, 18, 42, 43, 44] model variable-range raage dependencies between frames. However, these are sequential and fail to capture relationships across all frames simultaneously. Attention [3] and graph based methods [26] address this issue by modeling relationships across all frames, but they disregard the ordering of frames in a video, which is also useful for summarization. Our method uses Transformer [38] with positional encoding, which allows for joint attention across all frames while maintaining an ordering of the input sequence of frames.
Some of the above works have supervised and unsupervised variants with modifications to the objec- tive function. Specifically, for the unsupervised variant, they use reconstruction and diversity losses which do not require ground truth. We follow prior work in terms of using the same loss functions. Other notable unsupervised approaches include an adversarial LSTM based method [22]. a generative adversarial network to learn from unpaired data [30], and cycle consistent learning objective [41].
Video-Text Summarization. Plummer et al. [27] use image- image-language embeddings for generating video summaries but evaluate their approach in the text domain, and not on the generic video summarization benchmarks. Furthermore, they require text to be provided as input and don't have a mechanism to generate captions if absent. On the other hand, our method works well with off- the- shelf captions and we evaluate on both generic and query-focuse benchmark datasets. Chen et al. [1] jointly train a text and video summarization network but rely on ground-truth text summaries.
Query-Focuseds Video Summarization. Oftentimes users browsing videos on YouTube are looking for something specific so generic summary might not suffice. In this case, there should be an option to customize the generated summary using a natural language query. Sharghi et al. [32] introduce the Query-Focused Video Summarization (QFVS) dataset for UT Egocentric [16] videos containing user-defined video summaries for a set of pre-defined concepts. Sharghi et al. [33] propose memory network to attend over different video frames and shots with the user query as input. However, this recurrent attention mechanism precludes parallelization and limits modeling long-range dependencies, which is overcome by our Transformer architecture. Moreover, their method only works with the pre-defined set of keyword based queries in QFVS dataset. Since we use CLIP [29] to encode the language input and train our method on dense video descriptions, this allows users to define freeform queries at test time (as seen in Fig. 1). Other works [12, 39] similarly condition the summary generation keyword based queries but haven't 't released their data.
Transformers. Transformers [38] were introduced for neural machine translation and have since been applied to video-language tasks such as video-retrieval [4] and video captioning [17, 48). In this work we adapt transformers for video summarization. We modify self-attention [38] to Language- Guided Attention block that accepts inputs from two modalities. Additionally, our method also relies on CLIP [29] for extracting image and text features and a Bi-Modal Transformer [11] for dense video caption generation, both of which also have transformer backbone.
3 CLIP-It: Language-Guidee Video Summarization
CLIP-It is a unified mongage--uidee framework for both generic and query-focused video summa- rization. It takes an input video and a user-defined query or a system generated dense video caption and constructs a summary by selecting key frames from the video. First, we explain the intuition behind our approach. In the case of query-focused summarization, clearly, it is necessary to model the user query as input in order to produce an appropriate summary, In the case of generic video sum- marization no user query is available; nonetheless, we show here that we can leverage the semantic information contained in associated natural language descriptions to guide the summarization process. Assuming we had a generic description that accompanies a video (e.g.. A person is walking dog. The person throws a ball. The dog runs after it.), we could leverage its semantic embedding to match il to the most relevant portions of the video. Such a description could be obtained automatically by generating dense video captions [[]]. We first present an overview of our approach, CLIPP-It followed by detailed description of the individual components.
3

---

Frame Features
Attended Features
Frame-wise Scores
Video Summary
Video Frames
B
Language Guided Attention
Guided
B
Frame Frame-Scoring Transformer
D
Krapsack
B
K
Captions Query Features
MIP
0000000
OR
Caption
A man malls into shop. The
Generator
Food
Dense Video Caption
Query
Figure 2: Overview of CL.P-It. Given input video, CLIP-It generates summary conditioned either user-defined natural language query or an automatically generated dense video caption. The Guigge Attention head fuses the image and language emheddings and the Frame-Scoring Transformer jointly attends to all frames to predict their relevance scores. During inference, the video summary is constructed by converting frame scores to shot scores and using Knapsack algorithm to select high scoring shots.
Overview. Our approach, CLIP-It, is outlined in Fig 2. Given a video, we extract N frames denoted by F,, i [J. We formulate the video summarization task as a per-frame binary classification problem. We embed the frames using a pretrained network fimg- If query is provided (in the form of a natural language string), we embed the query using a pretrained network ftat. Alternatively, seen in the figure, caption with M sentences denoted by Cj. j E [1,...M and embed each sentence using the pretrained network ftat- Next, compute language attended image embeddings I" using learned Sanguage-Guided Multi-head Attention fing tzt- Finally, we train a Frame-Scoring Transformed which assigns scores to each frame in the video (green indicating high score and red indicating a low score). To construct the video summary during inference, we convert frame-level scores to shot-level scores and finally, use 0/1 knapsack algorithm pick the key shots [43]. In the following. Attention and the Trame-Selection Transformer modules, followed
off-the-shee video captioning model generate dense video
fing_at-
we describe the Language-Guidee by discussing the visual and language encoders.
Lundue Attention. We use Language- Guided Multi-head Attention to efficiently fuse information across the video and language modalities and infer long-term dependencies across both. Using a single attention head does not suffice as our goal is to allow all captions to attend to all frames in the video. We modify the Multi-Head Attention described in Vaswani et al. [38] to take in inputs from both modalities. We set Query Q, Key K, and Value V as follows,
Gugidee
Q Sing (F)), where E [1, [1,...N]
N].
K, V fine (C₃), where j E [1....M.
Language Guided Attn.(Q, V) Concat .... head.) W
where head, = Attention(QWe) kW: K VW.)
and Attention(Q,, K, V) softmax QKT
Vil V
W.O. W.K. and W.Y. are learned parameter matrices and dę the dimensions of K. The output of the Language-Guidee Multi-Head Attention are the attended image embeddings, denoted as F/.
Frame-Scoring Transformer. Finally, it is also important to ensure that we do not include redundant information, e-g., several key shots from the same event. To better model interactions across frames and contextualize them w.r.t. each other, we add Frame-Scoring Transformer that takes image-text representations as input and outputs one score per frame. Based on the Transformer model [38], this module assigns relevance scores to the attended image embeddings F'. We feed F to the bottom of both the encoder and decoder stacks. Similar to [38], positional encoding to insert information

---

about the relative positions of the tokens in the sequence. We add positional encodings to the input embeddings at the bottom of the encoder and decoder stacks.
Image Encoding. We encode the image using a pre-trained network fang. We experiment with the following networks: GoogleNet [37] (for fair comparison to prior work), ResNet [8], and CLIP [29].
Text Encoding. We encode the user-defined query or the system generated dense caption using a pre-trained network ftat. In this case we tried the CLIP (ViT and RN1OI) model. In case of generic video summarization, we generate dense video captions for the whole video order to condition on language and incorporate added semanties into video summarization pipeline. We use the Bi-Modal Transformer [11] dense video captioning model that generates multi-sentence description given video with audio. Since there multiple sentences in the dense video caption, first embed each sentence of the caption using the text encoder Stxt described above. They are then concatenated and fused using multi-layer perceptron (MIP)).
3.1 Learning
We employ 3 loss functions (classification, diversity, and reconstruction) to train our model. The supervised setting uses all 3 and the unsupervised setting uses only diversity and reconstruction losses.
Classification loss. We use weighted binary cross entropy loss for classifying each frame:
1
L.
Σ w [m]]og(<;)] + w0)((1 x log((1 - .z)). N
where zi is the ground -ruth label of the i-th frame and N is the total number of frames in the video. w* is the weight assigned to the class 2f, which is set to #keyframes frames if 2 is keyframe and 1 #keyframes x a background frame.
For the purpose of training without any supervision, we employ two additional losses that enforce diversity the selected keyframes. We first select keyframes X based on the scores assigned by the Frame-Scoring Transformer. We pass the output features of the transformer model for the selected keyframes through a decoder network consisting of convolution layers obtain reconstructed feature vectors for the selected keyframes such that each keyframe feature vector is of the same dimension as its corresponding input frame-level feature vector.
Reconstruction Loss. The reconstruction loss L, is defined as the mean squared error between the reconstructed features and the original features corresponding to the selected keyframes, such that:
C,
Σ IEX
| - -
X
where ý denotes the reconstructed features.
Diversity Loss. We employ a repelling regularizer [45] enforce diversity among selected keyframes. Similar to [31, 30J, we compute the diversity loss, Ca. as the pairwise cosine similarity between the selected keyframes:
1
Σ
y:
La = x(X iEX je
(3)
15
11
where y, and y, denote the reconstructed feature vectors of the i-th and j-th node. The final loss function for supervised learning is then,
Csup = a C. + 3 La A.
(4)
where a, B, and control the trade-off between the three loss functions. We modify the loss function to extend CLIP-It It to the unsupervised video summarization setting. We omit L. since the groundtruth summary cannot be used for supervision and represent the final loss function for unsupervised learning as:
Lunsup = B L. + Ly,,
(5)
where 3 and À are balancing parameters to control the trade-off between the two terms. We include implementation details of our method in the Supp.
5

---

Table 1: Supervised. Comparing F1 Scores of methods with supervised baselines the SumMe [7] and
TVSum [36] datasets using Standard, Augment, and Transfer configurations.
SumMe
TVSum
Method
Standard AugmentTransfer Standard AugmentTransfer
Zhang al. (SumTransfer) [42]
40.9
41.3
38.5
Zhang et al. (LSTM) [43]
38.6
42.9
41.8
54.7
59.6
58.7
Mahasseni al. (SUM-GAN [22]
41.7
43.6
56.3
61.2
Rochan et al. (SUM-FCN) [3]]
47.5
51.1
44.1
56.8
59.2
58.2
Rochan el al. (SUM-Dees -eMe-eaaaaa [3]]
48.8
50.2
45.0
58.4
59.1
57.4
Zhou et al. [47]
42.1
43.9
42.6
58.1
59.8
58.9
Zhang et al. [44]
44.9
63.9
Fajtl al. [3]
49.7
51.1
61,4
62.4
Rochan et al. [30]
48.0
41.6
56.1
55.7
Chen al. (V2TS) [1]
62.1
He et al. [9]
47.2
59.4
Park al. (SumGraph) [26]
51.4
$2.9
48.7
63.9
65.8
60.5
LoT
38.5
42.4
40.7
53.9
59.6
58.6
ResNet+hi-LSTM
39.4
44.0
42.6
55.0
61.0
59.9
CLIP-Image+hi-LLSTM
41.1
45.9
44.9
56.8
63.7
61.6
ClIP-Image+VVdde Caption+bi-L.TTM
41.2
45.5
57.1
62.4
GoogleNet+Transformee
51.6
53.5
49.4
64.2
66.3
61.3
ResNet+Transfomeer
52.8
54.9
50.3
65.0
67.5
62.8
LLIP-Image-Transtiee
53.5
55.3
51.0
65.5
68.1
63.4
CLIP-It: CLIP-Image+ Ir---mmaeVViioo Caption+Transforme
54.2
56.4
51.9
66.3
69.0
65.5
4
Experiments
In this section, describe the experimental setup and evaluation of our method on two tasks: generic video summarization and query-focused video summarization.
4.1 Generic Video Summarization
Generic video summarization involves generating single ceneral-purposs. summary to describe the input video. Note that while prior works only use visual cues from the video, our method also allows for video captions do not use any language cues and only the visual features used in earlier works [43].
input feature. For fair comparison, we include ablations of our method that
Datasets. We evaluate our approach on two standard video summarization datasets (TVSum [36] and SumMe [7]) and on generic summaries UT Egocentric videos [16] provided by the QFVS dataset [33]. TVSum [36] consists of 50 videos pertaining to 10 categories (how to videos, news, documentary, etc) with videos from each category, typically 1-5 minutes in length. SumMe [7] consists of 25 videos capturing multiple events such cooking and sports, and the lengths of the videos vary from to minutes. In addition to training on each dataset independently. we follow prior work and augment training data with 39 vidcos from the YouTube dataset [2] and 50 videos from the Open Video Project (OVP) dataset [24]. YouTube dataset consists of news, sports and cartoon videos. OVP dataset consists of multiple different genres including documentary videos. These datasets are diverse nature and come with different types annotations, frame-level scores for TVSum and shot-level scores for SumMe. They integrated create the ground-truth using the procedure in [43]. The UT Egocentric dataset consists of videos captured from head-mounted natural, uncontrolled setting and contains a diverse set of events. The QFVS dataset [33] provides ground-truth generic summaries for these 4 videos. The summaries were constructed by dividing the video into shots and asking users to select the relevant shots. The final ground-truth is an average of annotations from all users.
cameras. Each video is about 3-5 hours long, captured
Note. All the datasets YouTube [2], Open Video Project (OVP) dataset [24], TVSum [36],
SumMe [7], and QFVS [32] were collected by the creators (cited) and consent for any person- ally identifiable information (PII) was ascertained by the authors where necessary.
Data configuration for TVSum and SumMe. Following previous works [43, 42, 31, 26], we evaluate our approach in three different data settings: Standard, Augment, and Transfer. In the Standard setting, the training and test splits are from same dataset (i.e. either T'S or SumMe). For SumMe we use available splits, and for TVSum randomly select 20% of the videos for testing

---

Table 2: Unsupervised. Comparing F1 Scores of methods with unsupervised baselines the SumMe [7]
TVSum [36] datasets using Standard, Augment, and Transfer data configurations.
SumMe
TVSum
Method
Standard AugmentTransfer Standard AugmentTransfer
Mahasseni al. [22]
39.1
43.4
51.7
59.5
Yuan et al. [41]
41.9
57.6
Rochan et al. (SUM-FCN amsup) [31]
41.5
39.5
52.7
Rochan et al. [30]
47.5
41.6
55.6
55.7
He al. [9]
46.0
47.0
44.5
58.5
58.9
57.8
Park et al. (SumGraph) [26]
49.8
52.1
47.0
59.3
61.2
57.6
GoogleNet+bi-LSTM
33.1
38.0
36.5
47.7
54.9
52.3
ResNet+bi-LSTM
34.5
40.1
39.6
51.0
56.2
53.8
CLIP-Image+bi-LST
35.7
41.0
41.4
52.8
58.7
56.0
CLIP-Imae++Videe Video Caption+bi-LSTM
36.9
42.4
42.5
53.5
59.4
57.6
TrnnsoNe
50.0
52.7
47.6
59.9
62.1
58.4
DesNet+Transformss
50.8
53.9
49.3
61.1
63.0
59.9
CLIP- CLIP-Image+ Mee--maaee--aee Transforrre
51.2
53.6
49.2
61.9
64.0
60.6
CLIP-It: ILIP-Image+ Caption+ Tranonno--Tannnmmerrr
52.5
54.7
50.0
63.0
65.7
62.8
Table 3: Comparing F1 Scores for generic video summarization the QFVS dataset.
Supervised
Vid
Vid 2
Vid
Vid
Avg
SubMod [6]
49.51
51.03
64.52
35.82
50.22
QFVS [33]
62.66
46.11
58.85
33.50
50.29
CLIP-Image + bi-LSTM
65.43
56.55
68.63
40.06
57.67
ResNet Transformer
66.97
58.32
70.10
43.31
59.67
CLIP-Image + Transformer
70.8
61.67
72.43
47.48
63.11
CLIP-It (Gen. Caption)
74.13
63.44
75,86
50.23
65.92
CLIP-It (GT Caption)
84.98
71.26
82.55
61.46
75.06
Unsupervised
Vid
Vid 2
Vid 3
Vid
Avg
Quasi [46]
53.06
53.80
49.91
22.31
44.77
CLIP- CLIP-Image + Transformer
65.44
57.21
65.10
41.63
57.35
CLIP-It (Gen. Caption)
67.02
59.48
66.43
44.19
59.28
CLIP-It (GT Caption)
73.90
66.83
75.44
52.31
67.12
and construct different splits and report an average result on all 5 splits. For the Augment setting, the training set from one dataset (e.g.. TVSum) is combined with all the data from the remaining three datasets (e.g., SumMe, OVP, and YouTube). This setting yields the best performing models due to the additional training data. The Transfer setting the most challenging of the three. It involves training a model on three datasets and evaluating on the fourth unseen dataset.
Quantitative Results. We compare our method and its ablations supervised video summarization baselines in Tab. We report Fl scores on the TVSum and SumMe datasets for all three data settings. Our full method, CLIP-It (CLIP-Image+ Video Caption+Transsfrmeen outperforms sfate-of -hr by large margin on all three settings. Particularly, in the Transfer setting we outperform the previous of-e-of-thhe-rr SumGraph [26] by 5% on TVSum and 3% on SumMe, indicating that our model is better than the baselines in generalizing to out-of difdtriitittiinn data.
described in Sec.
To prove the effectiveness of each component of model, include comparisons to different Transformer, we ablate the Language-Guidee Attention module and
ablations. In CLIP-Image+ Image+Transffrmeer directly pass the Imaee-maae features input the transformer. As seen, the performance drops by 2% indicating that conditioning on CLIP language embeddings leads to better summaries. Sub- stituting the Prame-Scoring Transformer with Bidirectional LSTM in CLIP-Image+bi-LSTM and CLiP-Image+ Video Saption+bb-LLSTE again results in performance drop, thus highlighting the need for the Transformer module in our model. For fair comparison with the baselines. in GoogleNet+ Transformer we use the same GoogleNet features provided by Zhang et al. [43] and used by all the other baselines and do not include language features. This method still outperforms Sum-
7

---

Input Video (TVSum Video 16)
Luck Jat
Summary
Image+ Transformer
(Pred. Summary w/o Captions)
stanciing kitchen with food the table. She curtingg the food on table. She then the food together togecher adds purting putting food the mixing the food
(CLIP Image+Video Caption +Transformer)
(Pred. Summary Captions)
Figure 3: Comparison of ground-trute summary to results from CLIP- -all-Immae+Tie Transformer and the full CLIP-It model (CLIP-Image- Video Caption+ Transformer). The input is recipe video. Without captions, the model assigns high scores certain irrelevant frames such scenes of the talking eating which hurts the precision. With captions, the cross-attention mechanism ensures that frames with important actions and objects are assigned high scores.
True Positives
Predicted Scores
manseneni
Apreh
GT Scores
Am
True Negatives
Qualitative result comparing the generic summary from CLIP-It with the ground-truid summary. The plots showing predicted and ground-truth frame-level scores similar, indicating that frames that given high score ground were also assigned high scores by our model.
Figure 4:
Graph [26]. Replacing the GoogleNet features with ResNet features results in small performance improvement but CLIP- Image features prove to be most effective.
Tab. 2 compares Fl scores in the unsupervised setting to other unsupervised baselines. CLIP-It (CLIP-Image+Vides Caption+Transformeet outperforms the best performing baseline on all settings on both datasets. We again observe large performance improvements in the Transfer setting and notice that overall our unsupervised method performs almost well as our supervised counterpart. In ELIP-Image+ ToaangeeTaasffrmme we ablate the language component which causes drop in performance, thus proving that language cues are useful the unsupervised setting as well. Other ablations yield similar results reiterating the need for each component of our method. We also follow Otani et al. [25] and report results on rank based metrics, Kendall's r [14] and Spearman's p [49] correlation coefficients in the Supp.

---

Table 4: Results on UT Egocentric dataset [16]
Method
F-Measure
Recall
Supervised
Submod-V+Both al. [27]
34.15
31.59
CLIP Image + Transformer
41.58
39.96
CLIP-It: CLIP Image Gen. Video Caption + Transformer
44.70
43.28
CLIP-It: CLIP Image + GT' Video Caption + Transformer
52.10
50.76
Unsupervised
CLIP Image + Transformer
39.22
37.46
CLIP-It: CLIP Image + Gen. Video Caption + Transformer
42.10
40.65
CLIP-It: CLIP Image GT Video Caption + Transformer
49.98
47.91
Table 5: Results on TV Episodes dataset [16]
Method
F-Measure
Recall
Supervised
Submod-V+ Rep. et al. [27] CLIP Image + Transformer
40.90
37.02
47.82
46.02
CLIP-It: CLIP Image + GT Video Caption + Transformer
55.34
53.90
Unsupervised
CLIP Image + Transformer
45.77
44.01
CLIP-It: CLIP Image + GT Video Caption + Transformer
53.42
52.50
Tab. shows Fl scores for the generic setting on the QFVS Dataset [33]. Following [33], we run four rounds of experiments leaving out one video for testing and one for validation, while keeping the remaining two for training. Our method, CLIP-It (Gen. Captions) outperforms both supervised and unsupervised baselines by a large margin on all four videos, and particularly Video 4, which happens to be the most difficult for all methods. Adding captions helps significantly improve (2%) the summaries and outperforms the CLIP Image Transformer baseline. To see how well model would perform we had perfect captions, we also show results by using the ground-truth captions obtained from VideoSet [40]. Replacing Image features with ResNet features causes drop in performance. Likewise, replacing the Transformer with a bi-LSTM also hurts performance.
We include results of our method (1) without captions (2) using generated captions (3) using the ground truth captions provided by VideoSet [40] for UT Egocentric and TV Episodes datasets in Tables 4 and 5. As ground truth, we obtain 15 summaries for each video using the same greedy n-gram matching and ordered subshot selection procedures as previous work [27]. We follow the same procedure in prior work [40, 27] for creating and evaluating text summaries from video summaries. Our method outperforms [27] both the supervised and unsupervised settings on both datasets.
Qualitative Results. We highlight the need use language, specifically dense video captions, for constructing generic video summaries through a qualitative example in Fig. 3. The input is video of a woman demonstrating how make a chicken sandwich. The ground truth summary shows the scores computed by averaging the annotations from all users as in [43] and the keyframes that
Table 6: Comparing Fl Scores different methods on QFVS dataset.
Vid
Vid 2
Vid 3
Vid 4
Avg
SeqDPP [5]
36.59
43.67
25.26
18.15
30.92
SH-DPP [32]
35.67
42.74
36.51
18.62
33.38
QFVS [33]
48.68
41.66
56.47
29.96
44.19
CLIP-Image + Query bi-LSTM
54.47
48.59
62.81
38.64
51.13
ResNet Query + Transformer
55.19
51.03
64.26
39.47
52,49
CLIP-It: CLIP-Image + Query Transformer
57.13
53.60
66.08
41.41
54.55

---

Input Video
Summary
Query: Book and Chair
Summary
Query: Sun and Tree
Figure Result of our method on the QFVS dataset. The first row shows some frames from hour long input video. Given the query "book and chair", Summary shows some frames selected by our method. Summary 2 shows frames for the query "Sun and Tree".
received high scores. Next we show the result from the baseline CLIP-Image+ Transformer, which only visual features and no language input. The predicted scores show that the high scoring frames in the ground truth also receive a high score by our baseline, however a lot of irrelevant frames end up receiving high score too. Thus, when the model finally picks the keyframes it ends up selecting frames where the person is talking or eating the sandwich (shown red) which do not correspond to the key steps in the video. Adding language guidance via generated captions helps address this problem. The last row shows the captions generated by BMT [11]. The results shown are from our full CLIP-It model. The predicted scores are more similar ground truth scores and the highest scoring keyframes are the same as the ground truth.
Fig. 4 shows compares summaries generated by our method to the ground truth summary cooking video from the SumMe dataset. Predicted scores are the frame-wist wise scores predicted by CLIP-It and GT Scores are the scores computed from annotations [43]. The top row (True Positives) shows high scoring keyframes which are chosen by both our method and the ground truth, and the green arrows point to the assigned scores. As see, they are clear, distinct and represent key actions in the recipe. The bottom row (True Negatives) shows frames which are assigned a low score (shown by the red arrows) and are not part of the final summaries (both GT and predicted). E.g., the first frame is irrelevant and corresponds to a segment between key steps, while the second frame has poor lighting and its hard tell what being done.
4.2 Query-Focused Video Summarization
In Query-Focused Video Summarization. the summarization process is conditioned input user query, thus, multiple summaries can be obtained for the same video using different input queries.
Dataset. We evaluate our method on the QFVS dataset based on the UT Egocentric videos described earlier. The dataset consists of 46 queries for each of the four videos and user-annotated video summaries for each query.
Quantitative Results. In Tab. 6, we compare Fl scores of our method to 3 baselines, SH-DPP [32], DeP [5], and QFVS [33]. Following [33], we run four rounds of experiments leaving out one video for testing and one for validation, while keeping the remaining two for training. Our full model achieves an avg Fl score of 54.55% outperforming the best baseline (44.19%) 19%) by 10%. We would like to point out that our method uses more recent image features compared to the baselines. At the same time, when switching from CLIP to ResNet image embedding, we still improve significantly over the baselines. We expect the improvement to also hold with weaker features (e.g., as demonstrated with GoogleNet results in Tab. 1, 2).
Qualitative Results. Fig. 5 shows result on the QFVS dataset for UT Egocentric videos. The input is an egocentric video shot from a head-mounted camera and spans 4 hours. It consists of multiple events from a person's day, such as reading books, working on the laptop, walking the streets, and so on. Summary and 2 show results from our CLIP-It It method when the input query is "Book and chair" and "Sun and tree" respectively. The frames shown are frames assigned high-scores by our method. As seen, given the same input video, different queries yield different summaries.
10

---

5 Discussion and Broader Impacts
We introduced CLIP-It. unified language-guin gugged framework for generic and query focused video summarization. Video summarization is a relevant problem with many use-cases, and our approach provides greater flexibiliiyy to the users, allowing them to guide summarization with open-ended natural language queries. We envision potential positive impact from improved user experience if adopted on video platforms such as YouTube. We rely
off-the-shelf video captioning model [11] and a large-scale Iision-language model (CLIP (29]) which may have encoded some inappropriate biases that could propagate to our model. Our visual inspection of the obtained summarization results did not raise any apparent concerns. However, practitioners who wish to use our approach should be mindful of the sources of bias we have outlined above depending on the specific use case they are addressing.
Acknowledgements. We thank Arun Mallya for very helpful discussions and feedback. We' also like to thank Huijuan Xu for feedback on the draft. This work was supported in part by DoD including DARPA's XAI, LwLL, and SemaFor programs, as well as BAIR's industrial alliance programs.
References
[1] Chen, B.C., Chen, Y.Y., Chen, F: Video to text summary: Joint video summarization and captioning with recurrent neural networks. British Machine Vision Conference (BMVC) (2017) 3,
[2] De Avila, S.E.F., Lopes, A.P.B., da Luz Jr, A., de Albuquerque Araújo, A.: Vsumm: A mechanism designed
to produce static vidco summaries and a novel evaluation method. Patt. Rec. Letters (2011) 6
[3] Fajtl, J.. Sokeh, H.S., Argyriou, V., Monekosso, D., Remagnino, P.: Summarizing vidcos with attention.
Asian Conference on Computer Vision (ACCV) (2018) 2, 6,
[4] Gabeur, V., Sun, C., Alahari, K., Schmid, C.: Multi-moda transformer for video retrieval. European Conference Computer Vision (ECCV) (2020) 3
[5] Gong, B., Chao, W.L., Grauman, K., Sha, F: Diverse sequential subset selection for supervised video
summarization. Advances in Neural Information Processing Systems (NeurIPS) (2014) 2, 9, 10
[6] Gygli, M., Grabner, H., Gool, L. V.: Video summarization by learning submodular mixtures of objectives. IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2015) 3, 7
Gygli, M., Grabner, H., Riemenschneider, H., Gool, L.V. Creating summaries from user videos. European
Conference on Computer Vision (ECCV) (2014) 6,7
[8] He, K. Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2016) 3,5
[9] He, X., Hua, Y., Song, T., Zhang, Z., Xue, Z., Ma, R., Robertson, N., Guan, H.: Unsupervised video summarization with attentive conditional generative adversarial networks. ACM international conference on Multimedia (2019) 6,
[10] Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural computation (1997) 3
[11] lashin, V., Rahtu, E.: better use of audio-visual cues: Dense video captioning with bi-modal transformer. British Machine Vision Conference (BMVC) (2020) 3, 5, 10, 11
[12] Kanchira, A., Gool, L.V., Ushiku, Y., Harada, T:: Viewpoint- aware video summarization. IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2018)
[13] Kang, H.W., Matsushita, Y., Tang, X., Chen, X.Q.: Space-time video montage. IEEE Conference Computer Vision and Pattern Recognition (CVPR) (2006) 3
[14] Kendall, M.G.: The treatment of ties in ranking problems. Biometrika 33(3), 239-251 (1945) 8
[15] Lavancier, F., Møller, Rubak, E.: Determinantal point process models and statistical inference. Journal of the Royal Statistical Society: Series B: Statistical Methodology (2015) 3
[16] Lee, Y.J., Ghosh, J., Grauman, K.: Discovering important people and objects for egocentric video
summarization. IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2012) 3, 6, 9
[17] Lei, J., Wang, I.,, Shen, Y., Yu, D., Berg, T., Bansal, M.: Mart: Memory-augmentee recurrent transformer for coherent video paragraph captioning. Association for Computational Linguistics (2020)
[18] Li, Y., Wang, L., Yang, T., Gong, B.: How local is the local diversity? reinforcing sequential determinantal point processes with dynamic ground sets for supervised video summarization. European Conference on Computer Vision (ECCV) (2018) 2,3
[19] Liu, T., Kender, J.R.: Optimization algorithms for the selection of key frame sequences of variable length. European Conference Computer Vision (ECCV) (2002) 3

---

[20] Lu, Z., Grauman, K.: Story-driven summarization for egocentric video. ICEE Conference Computer Vision and Pattern Recognition (CVPR) (2013) 3
[21] Ma, Y.F., Zhang, H.J.: A model of motion attention for video skimming. International Conference on Image Processing (ICTP) (2002) 2
[22]
Mahasseni, B., Lam, M., Todorovic, S.: Unsupervised video summarization with adversarial Istm networks.
TEFE Conference on Computer Vision and Pattern Recognition (CVPR) (2017) 3, 6,
[23] Ngo, C.W., Ma, Y.F., Zhang, H.J.: Automatic video summarization by graph modeling. IEEE International Conference on Computer Vision (ICCV) (2003) 3
[24] Open vidco project. https: hpw.n video.ooorr..
[25] Otani, M., Nakashima, Y. Rahtu, E.. Heikkilä, J.: Rethinking the evaluation of video summaries. IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2019) 8
[26] Park, J., Lee. J., Kim, IJ.,, Sohn, K.: Sumgraph: Video summarization via recursive graph modeling. European Conference on Computer Vision (ECCV) (2020) 2, 3. 7,
[27] Plummer, B.A., Brown, M., Lazebnik, S.: Enhancing video summarization via vision-language embedding.
IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017) 3,9
[28] Potapov, D., Douze, M., Harchaoui, 7., Schmid, C.: Category-speciffe video summarization. European Conference Computer Vision (ECCV) (2014) 3
[29] Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., al.: Learning transferable visual models from natural language supervision. arXiv preprint
arXiv:2103.0000 (2021)3 3, 5,
[30] Rochan, M. Wang, Y.: Video summarization by learning from unpaired data. IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2019) 3, 5, 6,
[31] Rochan, M., Ye, I.., Wang, Y.: Video summarization using fully convolutional sequence networks. European
Conference on Computer Vision (ECCV) (2018) 2, 3, 5, 6,
[32] Sharghi, A., Gong, B., Shah, M.: Query-focused extractive video summarization. European Conference on Computer Vision (ECCV) (2016) 2, 3, 6, 9, 10
[33] Sharghi, A., Laurel, J.S., Gong, B.: Query-focusee video summarization: Dataset, evaluation, and memory network based approach. IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017) 2. 3, 6, 7, 9, 10
[34] Smith, M.A., Kanade, T.: Video skimming for quick browsing based on audio and image characterization.
School of Computer Science, Carnegie Mellon University (1995) 2
[35] Smith, M.A., Kanade, T.: Video skimming and characterization through the combination of image and language understanding techniques. IEEE Conference Computer Vision and Pattern Recognition (CVPR) (1997) 2
[36] Song, Y., Vallmitjana, J., Stent, A., Jaimes, A.: Tysum: Summarizing web videos using titles. IEEE Conference Computer Vision and Pattern Recognition (CVPR) (2015) 3.
[37] Szegedy, C., Liu, W.. Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., Rabinovich, A.: Going deeper with convolutions. JEEE Conference Computer Vision and Pattern Recognition (CVPR) (2015) 5
[38] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, ł.,, Polosukhin, I.: Attention is all you need, Advances in Neural Information Processing Systems (NeurIPS) (2017) 3, 3
[39] Wei, H., Ni, B., Yan, Y., Yu, H., Yang, X., Yao, C.: Video summarization via semantic attended networks.
The Association for the Advancement of Artificial Intelligence Conference (AAAI) (2018) 3
[40] Yeung, S. Fathi, A., Fei-Fei, L.: Videoset: Vidco summary evaluation through text. arXiv preprint #rXiv:1406.5822 (2014)
[41] Yuan, L., Tay. F.E., Li, P., Zhou, L., Feng, J.: Cycle- Cycle-consistent adversarial Istm networks for unsupervised video summarization. The Association for the Advancement of Artificial Intelligence Conference (AAAT) (2019) 3,7
[42] Zhang, K. Chao, W.L., Sha, F,, Grauman, K.: Summary transfer: Examplar-based subset selection for video summarization. IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2016) 2, 3,
[43] Zhang. K., Chao, W.L., Sha, F. Grauman, K.: Video summarization with long short-term memory. European Conference on Computer Vision (ECCV) (2016) 3, 4. 6, 9,
[44] Zhang, K., Grauman, K., Sha, F: Retrospective encoders for video summarization. European Conference Computer Vision (ECCV) (2018) 3,
12

---

[45] Zhao, B., Li. Lu, X.: Hierarchical recurrent neural network for video summarization, ACM international conference Multimedia (2017) 3,5
[46] Zhao, B., Xing, E.P: Quasi real-time summarization for consumer videos. IEEE Conference on Computer
Vision and Pattern Recognition (CVPR) (2014) 7
[47] Zhou, K.. Qiao, Y.. Xiang. Deep reinforcement learning unsupervised video summarization with diverrsity-eprreentttivvee Conference (AAAI) (2018)
reward. The Association for the Advancement Artificial Intelligence
[48] Zhou, L., Zhou, Y.. Corso, J.J., Socher, Xiong, C.: End-to-end dense vidco captioning with masked transformer. IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2018)
[49] Zwillinger, D., Kokoska, S.; Cre standard probability and statistics tables and formulae. CRC Press (1999)
13


Reviews:
[{"review_id": "tR54RTg3j0v", "reviewer": "Reviewer_Uj81", "summary": "The paper describes a video summarization model that is trained to use either user provided query or automatically generated dense  captioning to select key-frames as summaries.\n\nAt the core of the approach is an attention mechanism that computes similarity between frame embeddings and text embeddings. Subsequently, each frame representation is replaced by a weighted-average (softmaxed) of the language embeddings; key and value embeddings are from the text model while query embeddings are from the image model. Finally, a score is assigned to each frame via a standard transformer based model. A post-processing step converts the key frame scores to key-shot level summaries such that only 15% of the videos are selected. The overall model is trained with a weighted Binary cross-entropy loss.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": " \nStrength: \n - The idea of guiding visual summarization with language models is very nice. Intuitively, one would expect the caption/query of a video to represent the summary of visual concepts in it. I encourage the authors to explore the idea further.\n- The paper is clearly written and well organized.\n- The overall architecture (Clip Image + visual captioning + score transformers) is better than earlier approaches.\n\nWeakness: \n- Some important details about the model are not discussed (see main questions below)\n- The motivation for guiding video summarization with language model, as far as my understanding goes, is to train systems that can learn visual representation in correlation with high-level concepts encoded in text-embeddings. The paper did not show, empirically, that the introduced attention led to better visual representation learning. If this is shown it would make the paper a strong submission.\n  - I recommend to design an experiment where the system is trained with language-guided attention, and evaluated without generating captions or the attention mechanism. If such a system proves to be better, then it would be a strong validation for the very nice motivation.\n\nMain questions:\n- How are N frames selected? The sampling method is not discussed, however, the sampling could have huge difference which might lead to significant misalignment with respect to a query.\n- In the case of the generated captions, are the videos subsampled for the caption generation? if so is the sampling for caption generation and visual embedding the same?\n- The paper discusses using global attention (i.e., computing similarity between each frame and textual embedding). Does this not lead to a high-entropy distribution? I am referring to the distribution defined by the softmax of the similarity scores. This can become particular problematic for captions/queries that are very long.\n- Do you think a (dense) video captioning model (like the one used here) along with the scoring transformer would suffice to achieve the results? In other words, to what extent does the language-guided attention architecture plays a role in getting the results?\n  - Table 1. Shows the important components of the model, with respect to performance, are the quality of the embeddings and the transformer architecture. For instance, in case of [Clip Image + video caption + bi-LSTM] the approach is comparable or worse, in most cases, than related works. \n- Can you say something about choosing the key and value embeddings to be from the text model rather than the video? Is this to avoid the the high-entropy problem in long sequences? ( see Question 3)\n\nSignificance: \n- The work address an important problem in a research application that is early on its development. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Concerns are addressed.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4-5"}, "extra_text": {}}, {"review_id": "KIUrWIpt3uR", "reviewer": "Reviewer_tjo5", "summary": "This paper proposes a new and uniform method for generic and query-focused video summarization. In addition to typical video input, the method also takes caption or user query as input, and thus learns a language-guided representation to help video summarization. ", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength: \n\n- Applying transformers to model different modalities and improve video summarization.\n- The proposed method outperforms state-of-the-art generic summarization methods on both benchmark dataset by 5% and 3%, respectively.\n\nWeakness: \n\n- The reconstruction loss: training the model to reconstruct keyframes is not very intuitive to the reviewer.  Any frame could be leant to reconstructed from the transformers, which does not help summarization, in particular for the unsupervised setting(s).\n- The diversity loss only considers pairwise similarities among keyframes. This may be insufficient : as the major event may take a few frames to summarize, they are more similar to each other than a random frame (noise).\n- There are manually annotated captions as ground truth for UT Egocentric dataset as shown in [A].\n    - It would be more insightful to check the ground truth captions for video summarization.\n    - [26] in this case could also be fair comparison.\n- [26] can also use captions as input, the reviewer would encourage the authors to compare and report the results.\n- The clarification of the paper could be improved:\n    - Implementation: Are the weights on each loss summed up to 1? This is not claimed in the paper and chosen by heuristics.\n    - Experiments: is the summary taking at most 15% of the whole video? This is very important for fair comparison. Also TVSum has a commonly used train/test split, as shown in [1, 2, 9, 25, 29, 42].\n    - Typo: line 43, In case of generic; line 61, comprising of all; line 84, has lead to; line 258 Fig. 4 shows compares summaries; line 266, its hard to tell;\n\nAdditional comments: \n\nThere are many objects and actions in a video, but most can be  not relevant to the major activities  and considered as 'background'. It would be great to visualize the attention weights among video frames to captions/queries. \n\n[A] Videoset: Video summary evaluation through text. Serena Yeung, Alireza Fathi, Li Fei-Fei. ArXiv 2014.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes. The authors discussed the impacts of the proposed method in Sec. 5. ", "review_text": "", "extra_scores": {"ethical_concerns": "No", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4.5 hrs"}, "extra_text": {}}, {"review_id": "CE2jZz2vROk", "reviewer": "Reviewer_X5gK", "summary": "This paper unified the generic video summarization method and the query-based video summarization method. Specifically, the paper proposed a language guided video summarzation approach that takes both video frames and text as input and summarizes the video frames under the influence of the text. For generic video summarization, the model relies on a video captioning method to provide the text guidance. The proposed approach achieves superior performance on both generic video summarization datasets and the query-based video summarization datasets.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Method:\n\n1. I have concerns with the generic video summarzation part. The proposed approach relies on video captioning method to provide a text that describe the event in the video. This seems pretty straightforward conceptually. However, in practical, it relies on a very good video captioning system. \n\n  1.a The videos in summarzation datasets are very long. I wonder how many sentences are generated by the video captioning method? How to generate a caption for a day-long video? Did you sub-sample the frames?\n\n  1.b The genres of videos in the summarzation dataset might be differ to the videos for training the video captioning model. How to handle the domain gap?\n\n  1.c It seems the main reasons for the improvement in Table 1 and Table 2 are 1. Using Transformer. 2. Using better image feature (CLIP). The improvement brought by the caption is very limited (0.1 in Table1 under the setting of CLIP-Image + bi-LSTM. 0.7 in Table 1 under the setting of CLIP-Image + Transformer). The caption part only brought marginally improvement. It seems suggest the Caption is not useful? \n\n  1.d If the caption is helpful for video summarization.  I wonder what property would the caption need to improve the performance? Does the caption need to be concise or complicated? There are multiple off-the-shelf video captioning system. Why would the paper use the approach proposed by Iashin et. al?\n\n  1.e It would be better to show the example of captions in SumMe and TVSum.\n\n---\n\nRating:\n\nMy main concerns are with the captioning part and the ablation study in Table 1. Conceptually I think the proposed approach makes sense and the direction is very interesting. However, I have doubt on whether the experiments support the claim. I would happy to raise the score to 7 if the concerns are answered.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This submission received the following final ratings: 6, 6, 7.

Reviewer X5gK had initially expressed concerns about the effectiveness of the approach and the value of captions for summarization. The responses on these points provided by the authors convinced the Reviewer to raise the rating to 7.

Reviewer Uj81 gives a rating of 6, but recommends adding details about the sampling of the generated captions to the paper.

Reviewer tjo5 appreciates the clarifications given in the author response and confirms the original rating.

The ACs agree with the recommendation of acceptance.
Decision: Accept (Poster)