# Multimodal Summarization of User-Generated Videos
## 4 Dataset Compilation
### Annotation Data Aggregation  
Once the annotation process was completed, the individual video summaries had to be combined, resulting in an acceptable, final ground truth summary that aggregates the opinions of all the users who watched and annotated the specific video. This aggregation process is rather important for constructing a robust ground truth, since it will be used to evaluate and train the proposed supervised pipeline.  
The resulting video summaries that were annotated, as mentioned in the previous section, did not necessarily have the same number of annotators per video, making the construction of a robust dataset more difficult. For that reason, the original dataset has been reduced by deleting the videos that have been annotated by less than three annotators.  
Figure 4: Distribution of the number of users annotated each video. Only 73 videos have been annotated by only 2 or less humans.  
Figure 3: The annotating page of the web application.  
During that process, 61 videos have been excluded from the dataset as not having sufficient annotation data. For 12 videos the aggregated annotation resulted in no informative segments at all, therefore these videos were also removed from the dataset. This procedure led to the final dataset comprising of 336 videos that have been annotated by at least three different annotators. For these videos that consist the final dataset, the respective ground truth was generated by a simple majority-based aggregation rule. In particular, each segment of 1-s was considered to be included in the summary (i.e., characterized as \"informative\") if at least by 60% of the annotators agreed on that decision.  
Figure 5 illustrates the aforementioned process: in these examples, five annotators have provided their opinions about the non-interesting and informative areas of each video. All annotations are first translated into arrays of binary annotations, corresponding to each 1-s segment. Then, for each segment we extract the aggregated possibility that this segment can be characterized as \"informative\", based on the five annotations. We then threshold and accept as final ground truth, the segments for which the respective threshold is greater or equal than 0.6. Note that the aggregated agreement is computed as the average agreement between each individual annotation and the aggregated (final) ground truth. In this particular example this is the average of \([0.9,0.8,0.9,0.8,0.8]\).  
The final dataset (both raw data and respective annotations) ([[https://drive.google.com/drive/folders/1-nBp2](https://drive.google.com/drive/folders/1-nBp2) zJKXsUe2 xa9 DkonNdZ6 frwWkMp?usp=sharing]([https://drive.google.com/drive/folders/1-nBp2](https://drive.google.com/drive/folders/1-nBp2) zJKXsUe2 xa9 DkonNdZ6 frwWkMp?usp=sharing)), along with the respective tools for the entire aggregation process are publicly available ([[https://github.com/theopsall/Video-Summarization](https://github.com/theopsall/Video-Summarization)]([https://github.com/theopsall/Video-Summarization](https://github.com/theopsall/Video-Summarization))). Some statistics for our dataset are presented in Table 2.  
We calculated the average agreement as described above, but as a macro averaged F1 metric to have a direct comparison with the automatically generated summarizations. The result was found to be equal to 72.8%.  
\begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Dataset** & **Total Videos** & **Total Duration** & **Av. Dur.** & **Min Dur.** & **Max Dur.** \\ \hline Raw Dataset & 409 & \(\sim\)56.3 h & \(\sim\)8.25 m & 15 s & 15 m \\ Final Dataset & 336 & \(\sim\)44.2 h & \(\sim\)8 m & 15 s & \(\sim\)15 m \\ \hline \hline \end{tabular}
\end{table}
Table 2: Video dataset statistics.  
Figure 5: Example of aggregating annotation decisions from five different human annotators.