SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Motivation. To demonstrate that the Most replayed stats can supervise video highlight detection and summarization models, we conduct cross-dataset evaluation experiments. Concretely, we compare the performance of several representative highlight detection and summarization models trained on our dataset vs. those trained on existing small benchmarks, tested on the existing ones. If the Most replayed labels provide adequate frame importance ground truth to train and validate these models, they would perform well on existing benchmarks through zero-shot or fine-tuned transfer learning.
Experimental Settings. We use the Inception-v3 [63] features reduced to 1024D by PCA, following YouTube-8M [1]. For a fair comparison, the same set of features are extracted for TVSum [59], SumMe [23], and YouTube highlight [61], allowing seamless transfer learning across datasets. We choose the current state-of-the-art models, PGL-SUM [4] and VASNet [16] for video summarization and SL-Module [70] for video highlight detection. We exclude videos with artificial scenes (e.g., video games, cartoons, and animations) for this experiment, since the target datasets we transfer to contain only realistic videos, leaving 22,743 videos for training. For highlight detection, we set-aside  $20\%$  (TVSum) and  $30\%$  (YouTube highlight) of the videos within each category for testing, following previous works [70, 61]. For summarization (TVSum and SumMe), we use 5-fold cross-validation ( $20\%$  videos used for testing at each trial) following existing works [74, 4, 16]. From the 20 labels provided in TVSum, we take the segments with top  $\rho = 50\%$  average scores as ground truth following Hong et al. [27], and measure top-5 MAP. YouTube highlight does not need this processing, since it provides a single ground truth label (with different  $\rho$ ) for each video. We repeat all experiments with 5 random seeds.
Results and Discussion. Tab. 3 shows the cross-dataset transfer performance on video highlight detection benchmarks, TVSum [59] and YouTube highlight [61]. On average, the models pretrained on Mr. HiSum then fine-tuned on the target perform the best, as expected. Interestingly, however, the zeroshot often outperforms the fine-tuned on some categories. This is because Mr. HiSum contains a large number of samples in those categories, e.g., 'Bike tricks', 'Dog show', 'dog', or 'skating'. In this case, fine-tuning on a small target dataset unlearns some features learned from Mr. HiSum, slightly lowering the performance. The fine-tuned still outperforms the target, indicating that the learned features from pre-training still play some role after fine-tuning.
Datasets	Model
Training	Test	PGL-SUM	VASNet
TVSum	TVSum	55.5±0.7	54.2±0.9
Mr. HiSum	TVSum	57.1±0.7	57.1±1.0
SumMe	SumMe	41.7±3.2	41.9±3.3
Mr. HiSum	SumMe	42.3±2.1	42.6±1.3
Table 4: Video summarization performance on Cross-dataset transfer.
Agreement	A	D	Count	Percentage
Agree	6	0	59	23.6%	64.8%
5	1	61	24.4%
4	2	42	16.8%
Neutral	3	3	39	15.6%	15.6%
Disagree	2	4	25	10.0%	19.6%
1	5	12	4.8%
0	6	12	4.8%
Table 5: User Study Results. A:D is the number of raters whose answer is same:opposite to our label.
On categories with a relatively small number of videos in Mr. HiSum such as 'gymnastics', 'skiing', 'surfing', 'Beekeeping', or 'Making sandwich', zero-shot underperforms compared to target. Once fine-tuned, however, the performance gets significantly better, sometimes outperforming target or being comparable. This implies that pre-training on large-scale data is beneficial in most cases, unless we target a very specific topic that is not well-represented in the large dataset.
Tab. 4 compares the video summarization performance on TVSum [59] and SumMe [23], when the models are trained on our dataset vs. the corresponding training sets. The results demonstrate that for all models, training on our Mr. HiSum dataset exceed the ordinary F1-scores both on TVSum and SumMe.
From these experiments, we conclude that the Most replayed statistics does effectively supervise highlight detection and summarization models, indicating that Mr. HiSum provides reliable ground truth labels on these tasks.
5.2 User Study
Experimental Setup. To verify that our label is a reasonable proxy highlight score, we conduct the following user study. We select 25 videos from our dataset and choose 10 5~10-second-long clips in each video, such that 5 of them are highlights and the other 5 are not. We recruit 30 annotators in total and each of them has been assigned with 5 videos. They are instructed to watch each video fully, and then are given 10 candidate clips per video to rate. They are asked to choose 5 most interesting (engaging) ones among them. Each clip is marked either it is a highlight or not by 6 annotators. If the majority vote is the same as the Most replayed label in our dataset, the clip is scored 'Agree'. If the majority vote is the opposite, it is scored 'Disagree'. If the votes are 3:3, it is considered 'Neutral'.
Results and Discussion. Tab. 5 shows that the labels for  $64.8\%$  of the clips are in accordance with the annotators, while only  $19.6\%$  are disagreed, leaving  $15.6\%$  as the gray area. From the significantly larger portion of the 'Agree' than 'Disagree', we conclude that the Most replayed labels in our dataset reasonably reflect the general human perception of video highlights.
5.3 Benchmarks on Mr. HiSum Dataset
We provide performance metrics of several representative video highlight detection and summarization models on our dataset for reference. We report two metrics: F1 score (widely-used for summarization) and Mean Average Precision (MAP; widely-used for highlight detection). We set aside 2,000 videos for testing in this experiment.
Model	F1	MAPρ=50%	MAPρ=15%
PGL-SUM [4]	55.89±0.04	61.60±0.14	27.45±0.15
VASNet [16]	55.26±0.05	58.69±0.30	25.28±0.40
SL-module [70]	55.31±0.09	58.63±0.13	24.95±0.13
DSNet [78]	50.78±0.16	57.31±0.18	24.35±0.34
iPTNet [30]	50.53±0.16	55.53±0.25	22.74±0.13
Table 6: Benchmark on Mr. HiSum dataset
Figure 3: F1-Scores with various validation set sizes  $|\mathcal{V}|$  (Shaded area indicates  $\pm 1\sigma$  from the mean.)
Tab. 6 reports the performance on our test set. PGL-SUM shows the best performance on all metrics. It is noteworthy that the standard deviations on the metrics are relatively lower when evaluated on Mr. HiSum (Tab. 6), compared to the case on SumMe, TVSum or YouTube highlight reported in Tab. 3-4. This reconfirms that evaluating on a sufficient number (2,000) of videos allows a significantly more robust evaluation and avoids performance fluctuations due to random seeds.
5.4 Analysis on the Desired Validation and Evaluation Set Size
It is now clear that tens of videos are insufficient to cover the vast video space. Then, how many videos do we need for robust training and evaluation? We aim to answer this question as a reference for future dataset creators. We admit that it is not just a matter of quantity but diversity of those samples is also important. Since YouTube-8M, where we sample from, is a carefully designed dataset to cover wide range of 3,862 topics with at least 100 videos per category, our analysis provides a more optimistic bound for a minimum number of videos required to achieve certain level of confidence, where the video pool is relatively well-representative of the real world.
We vary the size of validation set  $|\mathcal{V}| = \{25, 50, 400, 1000, 1500, 2000\}$  to see the robustness of validation performance throughout the training. We trace the validation and test F1-scores throughout training the PGL-SUM model on 27,892 videos from Mr. HiSum up to 200 epochs. Fig. 3 illustrates notably unstable validation and test scores with large fluctuation when  $|\mathcal{V}|$  is small. The curves get smoother with larger  $|\mathcal{V}|$ .
5.5 Category-specific Analysis
Motivation. In general, videos in a particular topic tend to have their own way of storytelling or commonly-used structures. Such a particular tendency would make the summarization method more specialized to the category, and thus a category-specific summarization would be simpler than its general counterpart. We test a hypothesis that each category poses different levels of complexity in summarization, requiring different scale of data samples to achieve certain level of accuracy.
Figure 4: Summarization performance of different video categories with varied training set size
Experimental Settings. We focus on 3 video categories: Food, Vehicle, and Video Game. We filter videos based on class labels from our dataset, resulting in 1,456 Food videos ('food', 'cooking', and 'recipe' classes), 2,497 Vehicle videos ('vehicle' and 'car' classes), and 3,785 Video Game videos ('game' and 'videogame' classes). We use the PGL-SUM model, and set aside 200 videos for validation and test, respectively, from each category. Among the remaining videos, we vary the training set size to \{25, 50, 400, 1000, 1500, 2000\} and measure F1.
Results and Discussion. As demonstrated in Fig. 4, the Food category shows significant performance boost with the increased training set size, while the Vehicle and Video Game categories tend to show slower improvement. Overall, Food categories require far less number of videos to achieve higher performance than the other two. We conjecture that the Food videos tend to have a similar format of demonstrating recipes step by step, and thus these videos can be summarized relatively simply. On the other hand, Vehicle and Video Game tend to have more diverse contents, making summarization harder. To conclude, the difficulty of the video summarization task varies by categories. We leave further analysis with more categories as an interesting future work.
To address the data sparsity problem inherent in video highlight detection and summarization, we create Mr. HiSum, which consists of 31,892 videos and labels from the watching behaviors aggregated over  $50,000+$  viewers per video. From robustness analysis, we conclude that a validation set size of 1,000 or larger is necessary to obtain robust evaluation, disqualifying most existing datasets for these tasks. We also discover that the difficulty of the video summarization task significantly varies by video categories. With the dataset scarcity issue resolved with our dataset, we look forward to seeing further development of new models in the research community.
Our dataset also has some limitations. While the definition of replay is intuitive to understand, it is unclear how to count a replay if a segment is rewatched only partially. Unfortunately, YouTube does not provide the exact way of counting replays, so our Mr. HiSum labels may not be precise. However, we argue that this does not bring a significantly different distribution in our case, since the length of each segment is quite short. Only the adjacent segments will be affected depending on how to treat partial rewatches, while the entire distribution will remain roughly the same.
In addition, the Most replayed stats may be affected by self-fulfilling bias. That is, users might replay certain parts of the video just because the UI indicates that the parts have been rewatched by many others already. Our labels may not be completely free from such a bias. Actually, the first 50,000 viewers are not affected at all, as the Most replayed UI is shown after at least 50,000 views. After then, however, once the Most replayed distribution appears on the UI, it may be hard to assert that users were not influenced by it. Unfortunately, it is hard to measure or remove the impact of this bias, since YouTube does not provide the stats separately for the first 50,000 viewers. Considering that this is based on explicit replays, not just watching, we believe the impact of such a bias would not be significant.
We sincerely thank Kwanseok Kim (Seoul National University) for insightful discussion. This work was supported by the New Faculty Startup Fund from Seoul National Univer- sity and by National Research Foundation (NRF) grant (No. 2021H1D3A2A03038607/50%, 2022R1C1C1010627/20%, RS- 2023-00222663/10%) and Institute of Information & communications Technology Planning & Evaluation (IITP) grant (No. 2022-0-00264/20%) funded by the government of Korea.
[1] S. Abu-El-Haija, N. Kothari, J. Lee, P. Natsev, G. Toderici, B. Varadarajan, and S. Vijayanarasimhan. Youtube-8M: A large-scale video classification benchmark. arXiv:1609.08675, 2016.
[2] E. Apostolidis, E. Adamantidou, A. Metsai, V. Mezaris, and I. Patras. Unsupervised video summarization via attention-driven adversarial learning. In Proc. of the International Conference on multimedia modeling, 2020.
[3] E. Apostolidis, E. Adamantidou, A. I. Metsai, V. Mezaris, and I. Patras. AC-SUM-GAN: Connecting actor-critic and generative adversarial networks for unsupervised video summarization. IEEE Transactions on Circuits and Systems for Video Technology, 31(8):3278-3292, 2020.
[4] E. Apostolidis, G. Balaouras, V. Mezaris, and I. Patras. Combining global and local attention with positional encoding for video summarization. In Proc. of the IEEE International Symposium on Multimedia (ISM), 2021.
[5] E. Apostolidis, G. Balaouras, V. Mezaris, and I. Patras. Summarizing videos using concentrated attention and considering the uniqueness and diversity of the video frames. In Proc. of the International Conference on Multimedia Retrieval, 2022.
[6] N. Babaguchi, Y. Kawai, T. Ogura, and T. Kitahashi. Personalized abstraction of broadcasted american football video by highlight selection. IEEE Transactions on Multimedia, 6(4):575-586, 2004.
[7] T. Badamdorj, M. Rochan, Y. Wang, and L. Cheng. Joint visual and audio learning for video highlight detection. In ICCV, 2021.
[8] T. Badamdorj, M. Rochan, Y. Wang, and L. Cheng. Contrastive learning for unsupervised video highlight detection. In CVPR, 2022.
[9] J. Carreira and A. Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In CVPR, 2017.
[10] F. Chen and C. De Vleeschouwer. Formulating team-sport video summarization as a resource allocation problem. IEEE Transactions on Circuits and Systems for Video Technology, 21(2):193-205, 2011.
[11] K. Cho, B. Van Merrienboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y. Bengio. Learning phrase representations using rnnc encoder-decoder for statistical machine translation. arXiv:1406.1078, 2014.
[12] W. Chu, Y. Song, and A. Jaimes. Video co-summarization: Video summarization by visual co-occurrence. In CVPR, 2015.
[13] S. E. F. De Avila, A. P. B. Lopes, A. da Luz Jr, and A. de Albuquerque Araujo. VSUMM: A mechanism designed to produce static video summaries and a novel evaluation method. Pattern recognition letters, 32(1):56-68, 2011.
[14] J. Deng, W. Dong, R. Socher, L. Li, K. Li, and L. FeiFei. ImageNet: A large-scale hierarchical image database. In CVPR, 2009.
[15] B. G. Fabian Caba Heilbron, Victor Escorcia and J. C. Niebles. ActivityNet: A large-scale video benchmark for human activity understanding. In CVPR, 2015.
[16] J. Fajtl, H. S. Sokeh, V. Argyriou, D. Monekosso, and P. Remagnino. Summarizing videos with attention. In ACCV, 2019.
[17] M. Fleischman, B. Roy, and D. Roy. Temporal feature induction for baseball highlight classification. In ACM MM, 2007.
[18] T. Fukusato, T. Hirai, S. Kawamura, and S. Morishima. Computational cartoonist: A comic-style video summarization system for anime films. In International Conference on MultiMedia Modeling, 2016.
[19] J. A. Ghauri, S. Hakimov, and R. Ewerth. Supervised video summarization via multiple feature sets with parallel attention. In IEEE International Conference on Multimedia and Expo (ICME), 2021.
[20] Google Support. Youtube video player updates: Most replayed video chapters, single loop & more. https://support.google.com/youtube/thread/164099151, 2023.
[21] R. Goyal, S. Ebrahimi Kahou, V. Michalski, J. Materzynska, S. Westphal, H. Kim, V. Haenel, I. Fruend, P. Yianilos, M. MuellerFreitag, et al. The "something something" video database for learning and evaluating visual common sense. In ICCV, 2017.
[22] A. Graves, A. Mohamed, and G. Hinton. Speech recognition with deep recurrent neural networks. In Proc. of the IEEE international conference on acoustics, speech and signal processing (ICASSP), 2013.
[23] M. Gygli, H. Grabner, H. Riemenschneider, and L. Van Gool. Creating summaries from user videos. In ECCV, 2014.
[24] M. Gygli, Y. Song, and L. Cao. Video2GIF: Automatic generation of animated gifs from video. In CVPR, 2016.
[25] B. He, J. Wang, J. Qiu, T. Bui, A. Shrivastava, and Z. Wang. Align and attend: Multimodal summarization with dual contrastive losses. In CVPR, 2023.
[26] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In CVPR, 2016.
[27] F.-T. Hong, X. Huang, W. Li, and W. Zheng. Mini-Net: Multiple instance ranking network for video highlight detection. In ECCV, 2020.
[28] T. Hussain, K. Muhammad, A. Ullah, Z. Cao, S. W. Baik, and V. H. C. de Albuquerque. Cloud-assisted multiview video summarization using cnn and bidirectional LSTM. Trans. on Industrial Informatics, 16(1):77-86, 2019.
[29] S. J. Hwang, J. Lee, B. Varadarajan, A. Gordon, Z. Xu, and A. Natsev. Large-scale training framework for video annotation. In KDD, 2019.
[30] H. Jiang and Y. Mu. Joint video summarization and moment localization by cross-task sample transfer. In CVPR, 2022.
[31] Y. Jiao, Z. Li, S. Huang, X. Yang, B. Liu, and T. Zhang. Three-dimensional attention-based deep ranking model for video highlight detection. IEEE Trans. Multim., 20(10):2693-2705, 2018.
[32] Y. Jung, D. Cho, D. Kim, S. Woo, and I. S. Kweon. Discriminative feature learning for unsupervised video summarization. In AAAI, 2019.
[33] A. Karpathy, G. Toderici, S. Shetty, T. Leung, R. Sukthankar, and L. FeiFei. Large-scale video classification with convolutional neural networks. In CVPR, 2014.
[34] V. Kaushal, S. Kothawade, A. Tomar, R. Iyer, and G. Ramakrishnan. How good is a video summary? A new benchmarking dataset and evaluation framework towards realistic video summarization. arXiv:2101.10514, 2021.
[35] V. Kaushal, S. Subramanian, S. Kothawade, R. Iyer, and G. Ramakrishnan. A framework towards domain specific video summarization. In WACV. IEEE, 2019.
[36] W. Kay, J. Carreira, K. Simonyan, B. Zhang, C. Hillier, S. Vijayanarasimhan, F. Viola, T. Green, T. Back, P. Natev, et al. The kinetics human action video dataset. arXiv:1705.06950, 2017.
[37] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
[38] A. Krizhevsky, I. Sutskever, and G. E. Hinton. ImageNet classification with deep convolutional neural networks. Communications of the ACM, 60(6):84-90, 2017.
[39] H. Lee, J. Lee, J. Y. Ng, and P. Natsev. Large scale video representation learning via relational graph clustering. In CVPR, 2020.
[40] J. Lee and S. Abu-El-Haija. Large-scale content-only video recommendation. In ICCV, 2017.
[41] J. Lee, S. Abu-El-Haija, B. Varadarajan, and A. Natev. Collaborative deep metric learning for video understanding. In KDD, 2018.
[42] J. Lee, W. Reade, R. Sukthankar, G. Toderici, et al. The 2nd youtube-8M large-scale video understanding challenge. In ECCV, 2018.
[43] Y. J. Lee, J. Ghosh, and K. Grauman. Discovering important people and objects for egocentric video summarization. In CVPR, 2012.
[44] Z. Lei, C. Zhang, Q. Zhang, and G. Qiu. FrameRank: a text processing approach to video summarization. In Proc. of the IEEE International Conference on Multimedia and Expo (ICME), 2019.
[45] Y. Liu, S. Li, Y. Wu, C. Chen, Y. Shan, and X. Qie. UMT: Unified multi-modal transformers for joint video moment retrieval and highlight detection. In CVPR, 2022.
[46] B. Mahasseni, M. Lam, and S. Todorovic. Unsupervised video summarization with adversarial LSTM networks. In CVPR, 2017.
[47] A. Miech, D. Zhukov, J.-B. Alayrac, M. Tapaswi, I. Laptev, and J. Sivic. HowTo100M: Learning a text-video embedding by watching hundred million narrated video clips. In ICCV, 2019.
[48] J. Mun, M. Shin, G. Han, S. Lee, S. Ha, J. Lee, and E. Kim. BaSSL: Boundary-aware self-supervised learning for video scene segmentation. In ACCV, 2022.
[49] M. Narasimhan, A. Nagrani, C. Sun, M. Rubinstein, T. Darrell, A. Rohrbach, and C. Schmid. TL; DW? summarizing instructional videos with task relevance and cross-modal saliency. In ECCV, 2022.
[50] M. Narasimhan, A. Rohrbach, and T. Darrell. Clip-it! language-guided video summarization. NeurIPS, 34, 2021.
[51] M. Otani, Y. Song, Y. Wang, et al. Video summarization overview. Foundations and Trends® in Computer Graphics and Vision, 13(4):284-335, 2022.
[52] R. Panda, A. Das, Z. Wu, J. Ernst, and A. K. RoyChowdhury. Weakly supervised summarization of web videos. In ICCV, 2017.
[53] R. Panda and A. K. RoyChowdhury. Collaborative summarization of topic-related videos. In CVPR, 2017.
[54] Z. Pang, Y. Nakashima, M. Otani, and H. Nagahara. Contrastive losses are natural criteria for unsupervised video summarization. In WACV, 2023.
[55] D. Potapov, M. Douze, Z. Harchaoui, and C. Schmid. Category-specific video summarization. In ECCV, 2014.