# Multimodal Summarization of User-Generated Videos
## 2 Related Work
### Approaches for Video Summarization
VISCOM was tested on a large amount of videos from a variety of categories, in order to make the aforementioned video summarization model robust. In [22] authors focused on the importance of the video summary on tasks such as video search, retrieval and so forth.. In relation to the approaches based on recurrent neural networks, they tested a fully convolutional sequence neural network on semantic segmentation as the solution of the sequence labeling problem for the video summarization task. A deep video feature extraction process was proposed in [23], aiming to find the most informative parts of the video which are required so as to analyze video content. They included various levels of content to train their deep feature extraction technique. Their deep neural network also combined the description of the video, in order to extract the video features and then constructed the video summary by applying clustering based techniques also mentioned by the authors in [24]. The evaluation followed on their work is based on their own video summaries constructed by humans. The main goal of [24] was to remove redundant frames of an input video by clustering informative frames, which appeared to be the most effective way to construct a static video summary, built from all cluster centers. The frame representation that has been used within the clustering process was based on the Bag-of-Visual Words model. KVS is a novel video summarization approach, proposed in [25], specified from the video category provided, mainly from the title or the description of the video. A temporal segmentation is initially applied on a given video; its result is used as input on the KVS supervised algorithm, in order to build a higher quality video summaries compared to the unsupervised blind video category approaches. [PERSON] et al. [26] proposed an approach for keyframe extraction and video skimming that was based on a user attention model. To build a motion model, they extracted video,audio, and linguistic features and built an attention model based on the motion vector field. They created three types of maps based on intensity, spatial and temporal coherence which were then fused to form a saliency map. They also incorporated a static model to select salient background regions and extracted faces as well as camera attention features and finally, the created audio, speech and music models. The aforementioned attention components were linearly fused to create an \"attention\" curve. Local maxima of this curve within shots were used for keyframe extraction, while skim segments were selected using several criteria. [PERSON] et al. [27] trained a deep adversarial LSTM network consisting of a \"summarizer\" and a \"discriminator\" so as to minimize distance between ground truth videos and their summarizations, based on deep features extracted by a CNN. More specifically, the former consists of a selector and an encoder that selects interesting frames from the input video and encode them to a deep feature. The latter is a decoder that classifies a given frame as \"original\" or \"summary\". The deep neural network proposed here tries to fool the discriminator by providing the video summary as the original input video, assuming that both representations are the same. We should note that all methods and techniques presented in this section are quite significant for creating video summaries, with some of them being the current state-of-the-art. However, most of them do not consider both visual and aural information. Adding that none of the aforementioned works is applied on user-generated videos, our work, which concentrates at the combination of information from the different modalities extracted from a user-generated video stream, can address this need.