Generic Video Summarization. A video summary is a short synopsis created by stitching together
important clips from the original video. Early works [21, 35, 34] referred to it as Video Skimming or
Dynamic Video Summarization and used hand-designed features to generate summaries. Likewise,non-parametric unsupervised methods [13, 16, 23, 19, 20, 28] used various heuristics to represent
the importance of frames. Introduction of benchmark datasets such as TVSum [36] and SumMe [7]
provided relevance scores for frames in videos annotated by users, resulting in multiple human
generated summaries. This enabled automatic evaluation of video summarization techniques and
has lead to the development of many supervised learning based methods [6, 8, 22, 26, 31, 30, 41, 42,
43, 44, 45]. These approaches capture high-level semantic information and outperform the heuristic
unsupervised methods. Fully convolutional sequence networks [31] treat video summarization
as a binary label prediction problem. Determinantal point processes [15] and LSTM [10] based
approaches [5, 18, 42, 43, 44] model variable-range dependencies between frames. However, these
are sequential and fail to capture relationships across all frames simultaneously. Attention [3] and
graph based methods [26] address this issue by modeling relationships across all frames, but they
disregard the ordering of frames in a video, which is also useful for summarization. Our method uses
a Transformer [38] with positional encoding, which allows for joint attention across all frames while
maintaining an ordering of the input sequence of frames.
Some of the above works have supervised and unsupervised variants with modifications to the objec-
tive function. Specifically, for the unsupervised variant, they use reconstruction and diversity losses
which do not require ground truth. We follow prior work in terms of using the same loss functions.
Other notable unsupervised approaches include an adversarial LSTM based method [22], a generative
adversarial network to learn from unpaired data [30], and a cycle consistent learning objective [41].
Video-Text Summarization. Plummer et al. [27] use image-language embeddings for generating
video summaries but evaluate their approach in the text domain, and not on the generic video
summarization benchmarks. Furthermore, they require text to be provided as input and don’t have a
mechanism to generate captions if absent. On the other hand, our method works well with off-the-
shelf captions and we evaluate on both generic and query-focused benchmark datasets. Chen et al. [1]
jointly train a text and video summarization network but rely on ground-truth text summaries.
Query-Focused Video Summarization. Oftentimes users browsing videos on YouTube are looking
for something specific so a generic summary might not suffice. In this case, there should be an option
to customize the generated summary using a natural language query. Sharghi et al. [32] introduce
the Query-Focused Video Summarization (QFVS) dataset for UT Egocentric [16] videos containing
user-defined video summaries for a set of pre-defined concepts. Sharghi et al. [33] propose a memory
network to attend over different video frames and shots with the user query as input. However, this
recurrent attention mechanism precludes parallelization and limits modeling long-range dependencies,
which is overcome by our Transformer architecture. Moreover, their method only works with the
pre-defined set of keyword based queries in QFVS dataset. Since we use CLIP [29] to encode
the language input and train our method on dense video descriptions, this allows users to define
freeform queries at test time (as seen in Fig. 1). Other works [12, 39] similarly condition the summary
generation on keyword based queries but haven’t released their data.
Transformers. Transformers [38] were introduced for neural machine translation and have since
been applied to video-language tasks such as video-retrieval [4] and video captioning [17, 48]. In this
work we adapt transformers for video summarization. We modify self-attention [38] to a Language-
Guided Attention block that accepts inputs from two modalities. Additionally, our method also relies
on CLIP [29] for extracting image and text features and a Bi-Modal Transformer [11] for dense video
caption generation, both of which also have transformer backbone.لخص هذا بالعاميه بطريقه منفصله