SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 3: Video chapter generation (global metrics) on VidChapters-7M test set. Here, finetuned refers to finetuning on the VidChapters-7M train set, and speech refers to transcribed speech (ASR).
Method	Modalities	Pretraining Data	Finetuned	R@5s	R@3s	R@0.5	R@0.7	P@5s	P@3s	P@0.5	P@0.7
Text tiling [31]	Speech	∅	×	9.4	5.8	23.6	8.9	12.6	7.9	26.0	8.8
Shot detect [90]	Visual	∅	×	31.2	27.4	24.9	12.5	33.2	29.7	18.0	8.7
Vid2Seq [111]	Speech+Visual	C4 + HowTo100M	×	10.7	9.5	5.8	0.2	23.3	18.5	1.9	0.8
PDVC [99]	Visual	∅	✓	21.1	17.8	31.2	22.5	45.3	40.2	47.2	26.9
Vid2Seq [111]	Speech	C4	✓	37.8	29.5	44.6	26.1	29.0	23.0	38.0	23.4
Vid2Seq [111]	Speech	C4 + HowTo100M	✓	36.7	28.9	46.5	27.2	29.5	23.3	40.4	24.8
Vid2Seq [111]	Visual	C4	✓	35.3	26.4	23.6	8.7	17.9	13.6	17.2	7.1
Vid2Seq [111]	Visual	C4 + HowTo100M	✓	33.5	25.0	33.0	14.5	19.5	14.7	26.2	12.5
Vid2Seq [111]	Speech+Visual	C4	✓	36.3	28.6	45.8	26.9	29.9	23.8	40.9	24.9
Vid2Seq [111]	Speech+Visual	C4 + HowTo100M	✓	36.4	28.5	48.2	28.5	30.3	24.0	43.1	26.4
Table 4: Video chapter generation (segmentation metrics) on VidChapters-7M test set.
Models. For the video chapter segmentation subtask, we evaluate two zero-shot approaches (i.e., that are not trained on VidChapters-7M): speech text tiling [31], which detects subtopic shifts based on the analysis of lexical co-occurrence patterns, and a visual scene change detection algorithm [90] based on the sum of absolute differences. To derive zero-shot baselines for the full video chapter generation task, we combine text tiling and shot detection with various alternatives that can generate text given text or visual input: a random baseline that predicts a random speech sentence spoken inside the predicted boundaries, LLaMA-7B [91] (prompted to summarize the speech transcript spoken inside the predicted boundaries) and BLIP-2 [50] (prompted to describe the middle video frame of the predicted segment). Finally, we also train and evaluate two state-of-the-art end-to-end dense video captioning models on VidChapters-7M: PDVC [99] which consists of a visual-only DETR-style [11] architecture and Vid2Seq [111] which is a multi-modal sequence-to-sequence model pretrained on the C4 text corpus [72] and on narrated videos with ASR (e.g., YT-Temporal-1B [115]). For Vid2Seq, we also report zero-shot results after pretraining on narrated videos without finetuning on VidChapters-7M.
Implementation details. We use the text tiling implementation from the NLTK library [9] which tokenizes the text into pseudosentences of size 50. We use the shot detection software from the FFMPEG library [90] with a confidence threshold of 0.7. For BLIP-2, we use the 3.4B-parameter variant with FLAN-T5-XL [104] and CLIP ViT-L/14 [20, 70]. We reimplement Vid2Seq [111] (originally released in Jax) in PyTorch, use T5-Base pretrained on C4 [72] for initialization and pretrain Vid2Seq on HowTo100M [62]. More details are included in the Supplementary Material.
Results. We report the results for video chapter generation using global metrics and localization-only metrics in Tables 3 and 4, respectively. We observe that models trained on VidChapters-7M outperform zero-shot baselines, demonstrating the effectiveness of training on VidChapters-7M. In particular, PDVC [99] has the best precision and Vid2Seq [111] achieves the best results in terms of overall generation and recall. We also find that Vid2Seq's speech-only mode outperforms its visual-only mode and that using both speech and visual inputs leads to the best performance. This demonstrates that video chapter generation is a multi-modal task. Finally, we observe that pretraining using ASR in narrated videos from HowTo100M [62] improves the video chapter generation performance of the Vid2Seq model. Specifically, pretraining on HowTo100M is more beneficial for vision-aware models than for the speech-only model.
Method	Modalities	Pretraining Data	Finetuned	B1	B2	B3	B4	C	M	RL
Random	Speech	∅	✗	2.4	1.3	0.9	0.7	10.4	2.2	4.4
LLaMA [91]	Speech	Text mixture	✗	0.0	0.0	0.0	0.0	0.0	0.1	0.2
BLIP-2 [50]	Visual	129M image-texts	✗	3.1	1.5	0.9	0.7	12.4	2.2	4.5
Vid2Seq [111]	Speech+Visual	C4 + HowTo100M	✗	2.0	1.2	0.9	0.6	0.9	0.3	0.6
Vid2Seq [111]	Speech	C4 + HowTo100M	✓	21.0	15.5	12.1	10.0	105.3	11.5	24.5
Vid2Seq [111]	Visual	C4 + HowTo100M	✓	10.1	5.6	3.5	2.4	47.1	5.1	14.7
Vid2Seq [111]	Speech+Visual	C4	✓	21.6	15.7	12.3	10.0	110.8	11.5	26.0
Vid2Seq [111]	Speech+Visual	C4 + HowTo100M	✓	23.5	17.2	13.4	11.0	120.5	12.6	28.3
Table 5: Chapter title generation given ground-truth boundaries on VidChapters-7M test set.
Method	Modalities	Pretraining Data	Finetuned	R@10s	R@5s	R@3s	R@1s	R@0.3	R@0.5	R@0.7	R@0.9
Random	Speech	∅	✗	3.1	1.8	1.2	0.6	0.7	0.3	0.1	0.0
BERT [19]	Speech	BookCorpus + Wikipedia	✗	9.0	6.8	5.4	2.9	0.6	0.3	0.1	0.0
CLIP [70]	Visual	400M image-texts	✗	8.1	5.2	3.7	1.4	10.7	5.2	2.3	0.5
Moment-DETR [44]	Visual	5.4K narrated videos [44]	✗	3.2	1.6	1.1	0.5	11.3	3.6	0.8	0.1
Moment-DETR [44]	Visual	∅	✓	21.8	15.5	12.4	8.3	37.4	27.3	17.6	6.4
Table 6: Video chapter grounding on VidChapters-7M test set.
Qualitative examples. See the Supplementary Material.
4.2 Video chapter generation given ground-truth boundaries
In this Section, we study the task of generating chapter titles provided correct temporal boundaries of video chapters. This task is a simplification of the previously studied task where we assume perfect temporal segmentation. We adopt the same models and implementation details as previously introduced in Section 4.1.
Results. We report results for video chapter generation given ground-truth boundaries in Table 5. Similar to the full video chapter generation task, we observe that solving the task without training on VidChapters-7M is hard. Indeed, LLaMA [91] struggles to summarize the speech content into a chapter title and underperforms the random baseline. Furthermore, BLIP-2 [50] slightly improves over the random baseline. In addition, Vid2Seq [111] in zero-shot mode underperforms the random baseline due to the large domain gap between ASR and chapter titles (see Section 3.3). In comparison, the performance of models trained on VidChapters-7M is significantly higher. Moreover, Vid2Seq's speech-only mode outperforms its visual-only mode, and using both speech and visual inputs is beneficial, confirming the benefit of multi-modal reasoning for the task of generating chapter titles. Finally, pretraining on narrated videos from HowTo100M [62] improves the performance of the Vid2Seq model on VidChapters-7M.
4.3 Video chapter grounding
In this Section, we study the task of video chapter grounding that requires a model to temporally localize a chapter start time (or start-end window) given an annotated chapter title (query). Hence, compared to the video chapter generation task, we here assume chapter titles to be given and focus on the temporal chapter localization only.
Models. We evaluate three zero-shot alternatives: a random baseline that randomly picks the timestamps of a speech sentence in the video, a BERT [19] baseline that picks the timestamps of the speech sentence that has the closest text embedding with the queried chapter title, and a CLIP [70] baseline picking the frames where the query-frame similarity score drops from the highest scoring frame by a certain threshold  $\epsilon$ . We also train and evaluate on VidChapters-7M a state-of-the-art end-to-end video grounding model: Moment-DETR [44] which is designed for moment retrieval based on visual inputs. Furthermore, we report zero-shot performance of Moment-DETR obtained with the model checkpoint from Lei et al. [44] pretrained on 5.4K narrated videos with ASR from the QVHighlights dataset [44].
Implementation details. We use the [CLS] token sequence embedding for the BERT baseline and a threshold of  $\epsilon = 0.05$  for the CLIP baseline. More details are provided in the Supplementary Material.
Method	Modalities	Pretraining Data	YouCook2 (val)	ViTT (test)
S	C	M	R	P	S	C	M	R	P
PDVC [99]	V	∅	4.4	22.7	4.7	—	—	—	—	—	—	—
E2ESG [127]	T+V	C4 + WikiHow	—	25.0	3.5	20.7	20.6	—	25.0	8.1	32.2	32.1
Vid2Seq [111]	T+V	C4 + HTM	8.3	48.3	9.5	27.1	27.0	—	—	—	—	—
Vid2Seq [111]	T+V	C4 + YT-Temporal-1B	7.9	47.1	9.3	27.9	27.8	13.5	43.5	8.5	42.6	46.2
PDVC†	V	∅	4.8	28.8	5.8	22.6	33.1	9.4	40.6	16.5	19.2	37.4
PDVC†	V	VC (Chap.)	5.9	34.7	7.5	28.8	36.4	10.1	41.5	16.1	21.3	37.2
Vid2Seq†	T+V	C4 + HTM	8.6	53.2	10.5	29.2	26.2	14.1	44.8	8.7	43.8	44.5
Vid2Seq†	T+V	C4 + VC (ASR+Chap.)	9.8	62.9	11.7	32.5	30.1	15.1	50.9	9.6	45.1	46.7
Vid2Seq†	T+V	C4 + HTM + VC (ASR)	8.4	50.1	10.3	29.7	26.3	14.3	45.6	8.8	43.7	44.9
Vid2Seq†	T+V	C4 + HTM + 1% of VC (ASR+Chap)	8.8	52.7	10.4	29.3	27.6	13.5	41.6	8.2	44.7	42.1
Vid2Seq†	T+V	C4 + HTM + 10% of VC (ASR+Chap.)	9.9	63.9	12.1	32.4	31.4	14.5	47.4	9.2	45.3	45.9
Vid2Seq†	T+V	C4 + HTM + VC (ASR+Chap.)	10.3	67.2	12.3	34.0	31.2	15.0	50.0	9.5	45.5	46.9
Table 7: Comparison with the state of the art on the YouCook2 and ViTT dense video captioning benchmarks. T: Transcribed speech, V: Visual, HTM: HowTo100M [62], VC: VidChapters-7M, Chap.: Chapters. † denote results of our experiments.
Method	Modalities	Pretraining Data	YouCook2 (val)	ViTT (test)
S	C	M	R	P	S	C	M	R	P
Text tiling [31] + Random	T	∅	0.3	0.9	0.3	3.8	6.6	0.3	0.6	0.6	11.6	24.4
Text tiling [31] + LLaMA [91]	T	Text mixture	0.2	0.6	0.2	3.8	6.6	0.2	0.6	0.5	11.6	24.4
Shot detect [90] + BLIP-2 [50]	V	129M image-texts	0.6	1.0	0.5	8.9	5.5	0.2	0.1	0.2	3.1	13.7
Vid2Seq [111]	V	C4 + VC (ASR)	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.2	0.8
Vid2Seq [111]	V	C4 + VC (Chap.)	0.7	1.1	0.5	21.3	8.6	1.5	1.9	0.6	18.9	10.4
Vid2Seq [111]	T+V	C4 + HTM	0.0	0.1	0.0	0.5	0.6	0.0	0.0	0.0	0.5	1.0
Vid2Seq [111]	T+V	C4 + VC (ASR)	0.1	0.1	0.0	1.1	0.9	0.0	0.0	0.0	0.7	0.6
Vid2Seq [111]	T+V	C4 + VC (Chap.)	0.1	0.2	0.1	0.7	1.4	0.7	1.1	0.3	14.3	12.8
Vid2Seq [111]	T+V	C4 + VC (ASR+Chap.)	3.2	10.2	2.9	20.6	19.7	9.1	30.2	6.7	33.8	40.8
Vid2Seq [111]	T+V	C4 + HTM + VC (ASR)	0.0	0.1	0.0	1.2	0.9	0.0	0.0	0.0	0.8	0.7
Vid2Seq [111]	T+V	C4 + HTM + 1% of VC (ASR+Chap.)	2.7	7.2	2.1	18.1	17.3	5.5	15.5	4.3	31.3	37.1
Vid2Seq [111]	T+V	C4 + HTM + 10% of VC (ASR+Chap.)	3.2	11.5	3.0	19.4	19.2	6.4	21.6	5.3	31.0	38.2
Vid2Seq [111]	T+V	C4 + HTM + VC (ASR+Chap.)	3.9	13.3	3.4	22.3	20.1	9.0	28.0	6.5	33.7	40.1
Table 8: Zero-shot dense video captioning on the YouCook2 and ViTT benchmarks. T: Transcribed speech, V: Visual, HTM: HowTo100M [62], VC: VidChapters-7M, Chap.: Chapters.
Results. We report results for the video chapter grounding task in Table 6. We first observe that the simple zero-shot baselines based on ASR can decently find start times, but struggle to predict start-end windows due to the important domain gap between ASR and video chapters (see Section 3.3). The CLIP [70] baseline slightly underperforms the BERT baseline [19] at retrieving start times, but is much better at finding start-end windows. Furthermore, the Moment-DETR model [44] trained on VidChapters-7M outperforms the zero-shot baselines for both localization of start times and start-end windows, which further demonstrates the effectiveness of training on VidChapters-7M. Finally, we note that Moment-DETR cannot handle speech inputs, but hope that our results showing the benefit of this modality on other tasks in VidChapters-7M will foster research in the localization of language queries in untrimmed videos using multi-modal inputs (vision and speech transcripts).
4.4 Transfer learning on dense video captioning
In this Section, we investigate the pretraining of video-language models on our new VidChapters-7M. To this end, we adopt video chapter generation models trained on VidChapters-7M (see Section 4.1) to the tasks of dense video captioning with or without finetuning.
Datasets. We use two dense video captioning datasets. YouCook2 [124] has 2K untrimmed videos of cooking procedures. On average, each video lasts 320s and is annotated with 7.7 temporally-localized sentences. ViTT [35] was created to better reflect the distribution of instructional videos in the wild compared to YouCook2, and consists of 8K untrimmed instructional videos. On average, each video lasts 250s and is annotated with 7.1 temporally-localized short tags. For both datasets, we extract speech transcripts and visual features as described in Section 3.2, and follow the standard splits for training, validation and testing. Note that we only use videos available on YouTube at the time of the work, resulting in 10 to  $20\%$  less videos than in the original datasets.
Implementation details. See Section 4.1 and the Supplementary Material.
Results after finetuning. In Table 7, we show that pretraining for video chapter generation on VidChapters-7M greatly improves the downstream dense video captioning performance compared to training from scratch or pretraining only with ASR data as done in previous work [111]. We also find that pretraining both on HowTo100M [62] and VidChapters-7M results in the best overall performance. In particular, the Vid2Seq model pretrained on both HowTo100M and VidChapters-7M largely improves the state of the art on both the YouCook2 and ViTT benchmarks. In detail, on the YouCook2 benchmark, in the setting with C4 + HowTo100M pretraining, we observe that a boost of about 4.9 points in CIDEr is obtained with our reimplementation of Vid2Seq, and that 14.0 additional points in CIDEr are obtained by pretraining on VidChapters-7M. Finally, we report the results of the Vid2Seq model after pretraining on different fractions of VidChapters-7M for a fixed number of iterations. We construct these subsets such that larger subsets include the smaller ones. These results suggest that the scale of the chapter dataset is an important factor in the downstream dense video captioning performance. We conclude that VidChapters-7M opens a promising avenue for multi-modal pretraining. We further show qualitative examples of dense video captioning in the Supplementary Material.
Zero-shot dense video captioning. In Table 8, we report results obtained by directly applying video chapter generation models trained on VidChapters-7M for dense video captioning without finetuning for this task. As far as we know, our work is the first to explore this challenging zero-shot setting where no manual annotation of dense video captions is used for training. The Vid2Seq model trained only using ASR data underperforms the random baseline, due to the large domain difference between speech transcripts and dense captions [111]. In the visual-only setting, the variant trained on chapter annotations is better than the variant trained on ASR annotations. In the visual+speech settings, only using chapter annotations does not perform well, as training only on chapters (i.e., without speech) does not enable the model to learn how to use the input speech modality at inference. However, using both ASR and chapter annotations results in a largely better zero-shot dense video captioning performance and outperforms all baselines not trained on VidChapters-7M, demonstrating the complementary nature of the ASR and chapters annotations. Finally, we also observe the benefits of increasing the size of the pretraining dataset of chapters in this setting.
5 Conclusion, Limitations, and Societal Impacts
In this work, we presented VidChapters-7M, a large-scale dataset of user-chaptered videos. Furthermore, we evaluated a variety of baselines on the tasks of video chapter generation with and without ground-truth boundaries and video chapter grounding. Finally, we investigated the potential of VidChapters-7M for pretraining video-language models and demonstrated improved performance on the dense video captioning tasks. VidChapters-7M thus provides a new resource to the research community that can be used both as a benchmark for the video chapter generation tasks and as a powerful means for pretraining generic video-language models.
Limitations. As it is derived from YT-Temporal-180M [114], VidChapters-7M inherits the biases in the distribution of video categories reflected in this dataset.
Societal Impacts. The development of video chapter generation models might facilitate potentially harmful downstream applications, e.g., video surveillance. Moreover, models trained on VidChapters-7M might reflect biases present in videos from YouTube. It is important to keep this in mind when deploying, analysing and building upon these models.
This work was granted access to the HPC resources of IDRIS under the allocation 2023-A0131011670 made by GENCI. The work was funded by Antoine Yang's Google PhD fellowship, the French government under management of Agence Nationale de la Recherche as part of the "Investissements d'avirin" program, reference ANR-19-P3IA-0001 (PRAIRIE 3IA Institute), the Louis Vuitton ENS Chair on Artificial Intelligence, the European Regional Development Fund under project IMPACT (reg. no. CZ.02.1.01/0.0/0.0/15 003/0000468). We thank Jack Hessel and Rémi Lacroix for helping with collecting the dataset, and Antoine Miech for interesting discussions.
[111] Antoine Yang, Arsha Nagrani, Paul Hongsuck Seo, Antoine Miech, Jordi Pont-Tuset, Ivan Laptev, Josef Sivic, and Cordelia Schmid. Vid2Seq: Large-scale pretraining of a visual language model for dense video captioning. In CVPR, 2023. 2, 7, 8, 9, 10
[112] Fei Yu, Jiji Tang, Weichong Yin, Yu Sun, Hao Tian, Hua Wu, and Haifeng Wang. ERNIE-ViL: Knowledge enhanced vision-language representations through scene graph. In AAAI, 2020. 3
[113] Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, et al. Florence: A new foundation model for computer vision. arXiv preprint arXiv:2111.11432, 2021. 3
[114] Rowan Zellers, Ximing Lu, Jack Hessel, Youngjae Yu, Jae Sung Park, Jize Cao, Ali Farhadi, and Yejin Choi. MERLOT: Multimodal neural script knowledge models. In NeurIPS, 2021. 3, 4, 10
[115] Rowan Zellers, Jiasen Lu, Ximing Lu, Youngjae Yu, Yanpeng Zhao, Mohammadreza Salehi, Aditya Kusupati, Jack Hessel, Ali Farhadi, and Yejin Choi. MERLOT Reserve: Neural script knowledge through vision and language and sound. In CVPR, 2022. 2, 3, 7
[116] Kuo-Hao Zeng, Tseng-Hung Chen, Juan Carlos Niebles, and Min Sun. Title generation for user generated videos. In ECCV, 2016. 3
[117] Runhao Zeng, Wenbing Huang, Mingkui Tan, Yu Rong, Peilin Zhao, Junzhou Huang, and Chuang Gan. Graph convolutional networks for temporal action localization. In CVPR, 2019. 3
[118] Chenlin Zhang, Jianxin Wu, and Yin Li. ActionFormer: Localizing moments of actions with transformers. In ECCV, 2022.3
[119] Hao Zhang, Aixin Sun, Wei Jing, and Joey Tianyi Zhou. Span-based localizing network for natural language video localization. In ACL, 2020. 3
[120] Shengyu Zhang, Ziqi Tan, Zhou Zhao, Jin Yu, Kun Kuang, Tan Jiang, Jingren Zhou, Hongxia Yang, and Fei Wu. Comprehensive information integration modeling framework for video titling. In ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 2020. 3
[121] Songyang Zhang, Houwen Peng, Jianlong Fu, and Jiebo Luo. Learning 2d temporal adjacent networks for moment localization with natural language. In AAAI, 2020. 3
[122] Ziqi Zhang, Yaya Shi, Chunfeng Yuan, Bing Li, Peijin Wang, Weiming Hu, and Zheng-Jun Zha. Object relational graph with teacher-recommended learning for video captioning. In CVPR, 2020. 3
[123] Yue Zhao, Ishan Misra, Philipp Krahenbuhl, and Rohit Girdhar. Learning video representations from large language models. In CVPR, 2023. 3
[124] Luowei Zhou, Xu Chenliang, and Jason J. Corso. Towards automatic learning of procedures from web instructional videos. In AAAI, 2018. 2, 3, 9
[125] Luowei Zhou, Yingbo Zhou, Jason J Corso, Richard Socher, and Caiming Xiong. End-to-end dense video captioning with masked transformer. In CVPR, 2018. 3
[126] Luowei Zhou, Hamid Palangi, Lei Zhang, Houdong Hu, Jason J Corso, and Jianfeng Gao. Unified vision-language pre-training for image captioning and VQA. In AAAI, 2020. 3
[127] Wanrong Zhu, Bo Pang, Ashish Thapliyal, William Yang Wang, and Radu Soricut. End-to-end dense video captioning as sequence generation. In COLING, 2022. 9