# Multimodal Summarization of User-Generated Videos
## 3 Multimodal Video Summarization
### Feature Extraction
#### 3.2.1 Audio  
Low-level hand-crafted features have been shown that they may capture both perceived information, as well as the harmonic information of sound signals and have been  
Figure 1: Conceptual diagram of the feature extraction process for both audio and visual modalities.  
widely used in several application domains. Regarding the low-level description of the whole event, it has been shown that feature vectors constructed using statistics, such as mean and standard deviation of features can be efficiently used in event recognition-related tasks [32].  
Therefore, for each audio clip, extracted from the respective video file using ffmpeg, we calculate segment-level audio features, using the pyAudioAnalysis library ([[https://github.com/tyiannak/pyaudioanalysis](https://github.com/tyiannak/pyaudioanalysis)]([https://github.com/tyiannak/pyaudioanalysis](https://github.com/tyiannak/pyaudioanalysis))) [33]. According to this procedure, audio feature extraction is firstly carried out at a short-term basis. At a second level, segment-level feature statistics are computed and compose the final segment representation. In particular, the audio signal is divided into segment-level windows (either overlapping or non-overlapping) and for each segment a short-term processing is taken place, according to which 68 short-term features are computed (34 features and 34 deltas) for each short-term window. Short-term windows usually vary from 10 to 200 ms, while segment windows can be from 0.5 s to several seconds, depending on what is considered a homogeneous segment in the individual application domain. The short-term features extracted by the particular library are of three categories: time-domain, frequency domain and cepstral domain. The adopted short-term features are implemented in the pyAudioAnalysis library [33] and are shown in Table 1.  
According to the aforementioned procedure, for each audio segment a sequence of \(68-D\) feature vectors is extracted for each short-term window. These vectors are used to compute segment-level statistics, as the final segment representation: for each segment (that contains several short-term windows corresponding to several \(68-D\) short-term feature vectors), two segment-level statistics are extracted, namely the mean and standard deviation. Therefore, in total, \(2\times 68=136\) audio statistics are used to represent each audio segment. In this paper we propose using a short-term window size and step of 100 ms, while a non-overlapping segment window of 1 s has been adopted.  
\begin{table}
\begin{tabular}{p{56.9 pt}|p{113.8 pt}|p{113.8 pt}} \hline
**Index** & **Name** & **Description** \\ \hline
1 & Zero Crossing Rate & Rate of sign-changes of the frame \\
2 & Energy & Sum of squares of the signal values, normalized by frame length \\
3 & Entropy of Energy & Entropy of sub-frames’ normalized energies. \\  & & A measure of abrupt changes \\
4 & Spectral Centroid & Spectrum’s center of gravity \\
5 & Spectral Spread & Spectrum’s second central moment of the spectrum \\
6 & Spectral Entropy & Entropy of the normalized spectral energies for a set of sub-frames \\
7 & Spectral Flux & Squared difference between the normalized magnitudes of the spectra of the two successive frames \\
8 & Spectral Rolloff & The frequency below which 90\% of the magnitude distribution of the spectrum is concentrated. \\
9–21 & MFCCs & Mel Frequency Cepstral Coefficients: a cepstral representation with mel-scaled frequency bands \\
22–33 & Chroma Vector & A 12-element representation of the spectral energy in 12 equal-tempered pitch classes of western-type music \\
34 & Chroma Deviation & Standard deviation of the 12 chroma coefficients. \\ \hline \end{tabular}
\end{table}
Table 1: Adopted short-term audio features.