# Multimodal Summarization of User-Generated Videos
## 3 Multimodal Video Summarization
### Feature Extraction
#### 3.2.2 Video  
Apart from extracting auditory features from the sound signal of each video, we have adopted a wide range of visual features to describe the content of the visual information, as this modality is expected to be of major importance in the summarization procedure. For extracting these visual features, the multimodal_movie_analysis library ([[https://github.com/tyiannak/multimodal_movie_analysis](https://github.com/tyiannak/multimodal_movie_analysis)]([https://github.com/tyiannak/multimodal_movie_analysis](https://github.com/tyiannak/multimodal_movie_analysis))) has been used to extract features representing visual characteristics of a video. In particular, every 0.2 s, the following 88 visual features are extracted from the corresponding frame:  
* Color--related features (45 features):
* 8-bin histogram of the red values
* 8-bin histogram of the green values
* 8-bin histogram of the blue values
* 8-bin histogram of the grayscale values
* 5-bin histogram of the max-by-mean-ratio for each RGB triplet
* 8-bin histogram of the saturation values
* Average absolute difference between two successive frames in grey scale (1 feature)
* Facial features (2 features): The Viola-Jones [34] OpenCV implementation is used to detect frontal faces and the following features are extracted per frame:
* number of faces detected
* average ratio of the faces' bounding boxes areas divided by the total area of the frame
* Optical-flow related features (3 features): The optical flow is estimated using the Lucas-Kanade method [35] and the following 3 features are extracted:
* average magnitude of the flow vectors
* standard deviation of the angles of the flow vectors
* a hand-crafted feature that measures the possibility that there is a camera tilt movement--this is achieved by measuring a ratio of the magnitude of the flow vectors by the deviation of the angles of the flow vectors.
* Current shot duration (1 feature): a basic shot detection method is implemented in this library. The length of the shot (in seconds) in which each frame belongs to, is used as a feature.
* Object-related features (36 features): We use the Single Shot Multibox Detector [36] method for detecting 12 categories of objects. For each frame, as soon as the object(s) of each category are detected, three statistics are extracted: number of objects detected, average detection confidence and average ratio of the objects' area to the area of the frame. So in total, 3 \(\times\) 12 = 36 object-related features are extracted. The 12 object categories we detect are the following: person, vehicle, outdoor, animal, accessory, sports, kitchen, food, furniture, electronic, appliance and indoor.  
The aforementioned features provide a wide range of low (simple color aggregates), mid (optical flows) and high (existence of objects and faces) representation levels. The rationale behind the selection of this wide range of types of features lies in the fact that our goal is to cover every type of information that may possibly be correlated to the visual \"informativeness\" of the video. In other words, part of this work is to discover which types of visual information is mostly associated with what makes (or does not make) an informative video part, that is, which visual cues make a visual segment interesting.  
The dataset presented in this work has been annotated at a resolution of 1-s segments, and thus a way to represent these intervals in the feature space is needed, since the visual features are produced in a 0.2 s step. For that reason, the mean value across 5 subsequent feature vectors was calculated for each feature. In this way, the representation of every 1-s segment means a straightforward alignment to the respective audio features.