# Multimodal Summarization of User-Generated Videos
## 6 Conclusions & Future Work  
In this work, we have presented an approach for video skimming that effectively used both audio and visual modalities and has been applied to user-generated videos. We trained binary classifiers that learnt to discriminate between \"important\" (informative) segments, that is, those that should be a part of the produced summary and \"non-important\" ones, that is, those that should be discarded. A novel training and validation set has been created by human annotators for every 1-s part of each video. The dataset contains user-generated videos, collected from YouTube, and have been recorded using either action cameras or smartphones. We used several annotators and filtered ambiguous or insufficient annotations, while we also measured inter-annotator agreement. The dataset has been made available for future use and comparison. We evaluated our approach using six classifiers trained on audio, video and fused features. Our experimental results indicated that both audio and visual features are important for classification.  
As future work, it would be quite interesting to train the proposed method on a robust training data set with a large number of videos from categories annotated from an extensive number of users. An exploration in further modalities, such as text, may accompany a video in places likes the description, title, subtitles or even in the comments. The possible absence of such a huge data set would be interesting to study and start as a data collection process in our methodology. Extension to the machine learning models and deep learning algorithms, with different pre-processing techniques, such as time-series models, can be tested, or a deep neural network can be used as a video feature extractor. Lastly, A/B testing by measuring the final result of the method based on the point of view of users who did not participate in the annotation process, could be explored as a totally different and novel evaluation method, evaluating the videos at the summary level.  
Conceptualization, T.G. and E.S.; methodology, T.G., E.S., T.P. and P.K.; software, T.P. and P.K.; validation, T.G. and E.S.; data curation, T.P. and P.K.; writing--original draft preparation, T.P., P.K., T.G. and E.S.; writing--review and editing, T.P., P.K., T.G. and E.S.; funding acquisition, E.S. All authors have read and agreed to the published version of the manuscript.  
This research has been co-financed by the European Union and Greek national funds through the Operational Program Competitiveness, Entrepreneurship and Innovation, under the call RESEARCH--CREATE--INNOVATE (project code: 1 EDK-02070).  
The authors would like to thank all anonymous annotators, for their participation in the video annotation process.  
The authors declare no conflict of interest.  
The following abbreviations are used in this manuscript:  
\begin{tabular}{l l} UAVs & Unnamed Aired Vehicles \\ CNN & Convolutional Neural Network \\ LSTM & Long Short Term Memory \\ GAN & Generative Adversarial Network \\ MFCCs & Mel Frequency Cepstral Coefficients \\ RGB & Red Green Blue \\ VAT & Video Annotator Tool \\ Log Reg & Logistic Regression \\ KNN & k-Nearest Neighbors \\ XGBoost & eXtreme Gradient Boosting \\ FNN & Fully Connected Neural Network \\ ROC & Receiver Operating Characteristic \\ AUC & Area Under Curve \\ RFE & Recursive Feature Elimination \\ \end{tabular}