# Multimodal Summarization of User-Generated Videos
## 5 Results
### Results  
Table 4 shows AUC and F1 for six different classification methods and for the three modalities (audio, visual and fusion). Random Forest seems to be the best choice as it achieves the best AUC score and one of the best F1 scores. However, AUC is more important in our case as it incorporates the ability of the classifier to function at different operation points, that is, different probabilistic thresholds. It is also important to note that the classifiers based on the visual modality are always at least 4% relatively better than the audio-based classifiers, while the relative improvement is almost 3% better in fusion \begin{table}
\begin{tabular}{c c c} \hline \hline
**Subset** & **Total Videos** & **Total Samples** \\ \hline Training Dataset & 268 & 127,972 \\ Test Dataset & 68 & 31,113 \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Training-test samples. compared to the visual modality. ROC curves for top three classifiers are illustrated in Figure 6.  
Table 5 presents the final Precision, Recall, F1 and Accuracy for the best classification method (Random Forest), when also using the post-processing approach described in Section 3.4, for different values of the \(N_{1}\) and \(N_{2}\) parameters of the filtering process. It can be seen that for \(N_{1}=3\) and \(N_{2}=5\), F1 is relatively boosted by almost 4%. In terms of the positive class recall and precision rates: the first is increased and the latter is decreased as expected, since the filtering process removes positive predictions that do not match the aforementioned criteria. Overall, the 63% F1 macro-averaged score achieved by the method is reasonable, since the human performance on that metric, as measured from the agreement between the annotators, is 72.8%. \begin{table}
\begin{tabular}{|c|c|c|c|c|c|c|} \hline \multirow{2}{*}{**Classifier**} & \multicolumn{3}{c|}{**ROC AUC**} & \multicolumn{3}{c|}{**F1 macro averaged**} \\ \cline{2-7}  & **Audio** & **Visual** & **Fused** & **Audio** & **Visual** & **Fused** \\ \hline Random & \multicolumn{3}{c|}{49.7\%} & \multicolumn{3}{c|}{47.6\%} \\ \hline Naive Bayes & 59.5\% & 64\% & 63.4\% & 51.7\% & 48.3\% & 51.6\% \\ \hline KNN & 59.3\% & 60.7\% & 62.6\% & 54.6\% & 56.3\% & 57.7\% \\ \hline Log Reg & 62.8\% & 67.2\% & 67.4\% & 41.4\% & 44.6\% & 49.4\% \\ \hline Decision Tree & 60.6\% & 66.3\% & 66.5\% & 41.8\% & 45.6\% & 45.6\% \\ \hline Random Forest & 66.7\% & **69.8\%** & **71.8\%** & 57.8\% & 60.4\% & 60.6\% \\ \hline XGBOOST & 65.3\% & 66.8\% & 69.6\% & 59.8\% & 60.4\% & 62.3\% \\ \hline FNN & **67.45\%** & 68.6\% & 70.14\% & **62.12\%** & **64.4\%** & **66.37\%** \\ \hline \end{tabular}
\end{table} 
Table 4: Segment Level Metrics. Best performance per modality/metric is indicated in bold. Figure 6: Receiver Operating Characteristic (ROC) curve of the Random Forest classifier (**left**), XGBoost classifier (**middle**), Fully Connected classifier (**right**). \begin{table}
\begin{tabular}{|c|c|c|c|c|} \hline \multicolumn{1}{|c|}{**Thresholds**} & \multirow{2}{*}{**Precision**} & \multirow{2}{*}{**Recall**} & \multirow{2}{*}{**f1 Macro**} & \multirow{2}{*}{**Accuracy**} \\ \cline{2-3} \cline{5-6}
**(med (N\({}_{1}\))-hard (\(N_{2}\)))** & & & & \\ \hline no & 42.2\% & 69.9\% & 60.6\% & 62\% \\ \hline
3-3 & 43.7\% & 69.7\% & 62\% & 63.8\% \\ \hline
3-5 & **44.9\%** & **66.9\%** & **63\%** & **65.3\%** \\ \hline
5-3 & 43.4\% & 70.8\% & 61.8\% & 63.4\% \\ \hline
5-5 & 44.2\% & 69.9\% & 62.5\% & 64.3\% \\ \hline \end{tabular}
\end{table} 
Table 5: Random Forest performance metrics for different parameters of the post-processing technique described in Section 3.4. Best result is indicated in bold. In addition to the methodology for distilling the summary of a video, emphasis was placed on recognizing the features that contributed the most to the final result of the random forest classifier.