MAV: A Multimodal, Multigenre, and Multipurpose Audio-Visual Academic Lecture Dataset (2024)

Abstract: Publishing open-source academic video recordings is an emergent and prevalent approach to sharing knowledge online. Such videos carry rich multimodal information including speech, the facial and body movements of the speakers, as well as the texts and pictures in the slides and possibly even the papers. Although multiple academic video datasets have been constructed and released, few of them support both multimodal content recognition and understanding tasks, which is partially due to the lack of high-quality human annotations. In this paper, we propose a novel multimodal, multigenre, and multipurpose audio-visual academic lecture dataset (MAV), which has almost 367 hours of videos from five sources covering computer science, mathematics, and medical and biology topics. With high-quality human annotations of the slide text and spoken words, in particular high-valued name entities, the dataset can be used for multiple audio-visual recognition and understanding tasks. Evaluations performed on contextual speech recognition, speech synthesis, and slide and script generation tasks demonstrate that the diversity of MAV makes it a challenging dataset1.

heading: 1 Introduction
text: The rapid progress of technology has brought numerous academic presentations and talks available on the web with open access (Lev et al., 2019; Atri et al., 2021; Lee et al., 2022, 2023). These resources are particularly helpful to researchers as they contain rich specialised knowledge in auditory and visual modalities. With the development of neural-based AI systems, it is desired that AI could comprehend and process these multimodal information resources to assist scientists in accelerating their research. Abilities, including transcribing and
B: Corresponding author. 1Project website: https://jack-zc8.github.io/
M3AV-dataset-page
generating speech for a presentation with the aid of slides, as well as presentation slide or script generation, are particularly useful for researchers to conduct investigations and prepare presentations.
As a natural data source containing multimodal content and knowledge, academic lectures have been explored to construct a series of datasets. Some datasets focus on the evaluation of the model’s ability to recognize multimodal content (Dutta et al., 2018; Wang et al., 2023), others focus on promoting models to understand multimodal knowledge in academic videos (Lev et al., 2019; Li et al., 2019; Atri et al., 2021; Lee et al., 2022, 2023). However, none of them has yet simultaneously paid attention to the model’s ability to recognize multimodal content and understand rich academic knowledge, which is partly due to their respective lack of adequate manual labelling. The homogeneity of holding two characteristics is critical to the implementation of end-to-end academic recognition and understanding systems.
To fill the gap, we release the Multimodal, Multigenre, and Multipurpose Audio-Visual Academic Lecture Dataset (M3AV) as shown in Figure 1. The dataset contains selected academic lectures and presentation videos from multiple fields, including computer science, biomedical science, and mathematics. Each video is endowed with highly qualified speech transcriptions and OCR labels for both printed and handwritten characters, including complex mathematical formulae. In addition, academic papers for some videos are also provided to supplement knowledge more extensively.
Together with the dataset, three benchmark tasks are also proposed that reflect the perception and understanding of the multimodal information in those videos. This includes automatic speech recognition (ASR) with a particular focus on contextual ASR (CASR), spontaneous text-to-speech (TTS) synthesis, and slide and script generation (SSG). Representative baseline models are used to bench-
9041
mark the performance for each task. Through our experiments, it is observed that the existing models showcase limited performance in perceiving and understanding the multimodal content in this dataset, and the rich academic knowledge is not utilized effectively.

heading: 2 Related Work
text: The datasets based on academic lectures mainly fall into two categories: measuring the model’s ability to recognize multimodal content and evaluating its capacity to capture academic knowledge. In Table 1, we list the data types and dataset tasks of the relevant academic lecture-based datasets.
Multimodal Content Recognition. LectureVideoDB (Dutta et al., 2018) exploits frames from lecture videos to test the model’s ability to perform handwritten and scene text recognition. SlideSpeech (Wang et al., 2023) enriches the audiovisual corpus with synchronized slides to provide additional textual information. It is worth noting that M3AV dataset provides both OCR data that is fully manually labelled and speech data that combines multiple ASR system outputs and manual labelling, which is completely qualified to cover tasks in the aforementioned datasets.
Academic Knowledge Understanding. LectureBank (Li et al., 2019) manually annotates the prerequisite relationships between course concepts to carry out prerequisite chain learning. GoogleI/O (Chen et al., 2014) and LaRochelle (Nguyen et al., 2014) both study videolevel retrieval using presentation or lecture videos. AVIATE (Atri et al., 2021) adopts multimodal information from academic presentation videos to conduct abstract summarization. Similarly, TalkSumm (Lev et al., 2019) uses a scalable annotation method to construct large-scale paper summaries from corresponding video transcription. LPM (Lee et al., 2022, 2023) introduces crossmodal retrieval and generation tasks around its aligned slides and spoken language. Our M3AV dataset contains annotated video data for both slides and speeches, which implies little knowledge loss. The addition of paper documentation information also greatly complements the knowledge information. Furthermore, a challenging Slide and Script Generation task is developed to perform the inter-transformation of two kinds of academic materials.
As illustrated in Table 1, our dataset contains the most complete and human-annotated resources of slide, speech, and paper, thus supporting not only the recognition of multimodal content but also the comprehension of high-level academic knowl-
edge. At the same time, the size of our dataset is also relatively rich while accessible as shown in Table 2.
3 Summary of M3AV Dataset

heading: 3.1 Metadata
text: We treat each video as a unit and rename each video in the form of "{source}-{id}". For the raw data, we provide the YouTube URL of each video for users to download.
Speech Annotation. The speech annotations are saved in JSON files. Each unit contains the start and end timestamps, the transcribed text in the spoken and written form, and the corresponding word-level timestamps.
OCR Annotation. In OCR annotations, the coordinates and manually corrected text (formulae are in LaTeX format) of each block are provided in JSON files. Results of merging (detailed in Appendix A.1) are also provided.
Dataset Split. The division of the sets is determined by the speech data, as the duration can measure the amount of information consistently. The video durations are illustrated in Table 3.

heading: 3.2 Characteristics of Speech Data
text: Variety of Academic Fields. As shown in Figure 2(a), M3AV contains academic presentations and talk videos from three different fields including computer science, biomedical science, and mathematics. The Conference on Human Factors in Computing Systems (CHI) 2021 Presentations and ACM International Conference on Ubiquitous Computing (Ubi) 2020 Presentations focus on research in computer science. The National Institutes of Health Director’s Wednesday Afternoon Lectures (NIH) presents weekly science talks given by top biomedical scientists. The Introduction to the Principles and Practice of Clinical Research (IPP) teaches how to conduct clinical research effectively and safely. Oxford Mathematics (MLS) contains mathematics lectures. The total number of videos is 1113, with 366.9 hours of speech.
Abundance of Rare Words. The size of our spoken form word table is 47865, while the word frequency within 1k reaches 47483 words (99.20%) as shown in Figure 2(e). It effectively represents the typical scenario met in academic presentations where new terminologies constantly appear and are crucial to the overall understanding.
Quality of Transcription. M3AV is labelled by combining multiple high-performing ASR systems together with manual labels. As human often fails to label rare terminologies, the assistance of a highperforming ASR is indispensable. The multimodal labelling pipeline is described in Section 4.2.

heading: 3.3 Characteristics of Visual Data
text: Suffient OCR Text Labelling. The total number of slides has reached 24956 as illustrated in Figure 2(b). The average number of words per slide is 40.96 as shown in Figure 2(c). We can observe that there is a sufficient number of slides and the density of the words is high, which indicates the richness of the textual information in our slides. The duration of every slide segment is 50.67 as indicated in Figure 2(d).
Complex Handwriting and Formulae. Significant numbers of pages and blocks in Table 4 show the enrichment of our complex texts. Furthermore, handwriting and formula texts which both contain lots of presentation content and academic knowledge have already been processed into standard text or LaTeX format and thus can be directly utilised by large language models.

heading: 4 Data Creation Pipeline

heading: 4.1 Data Collection
text: Videos of open-source conferences and lectures are collected from YouTube, together with descriptions and subtitles. Papers for computer science videos are downloaded and subsequently parsed
using science-parse2.

heading: 4.2 Speech Transcription
text: The diagram of creating speech transcription is shown in Figure 3, comprising four steps.

heading: 4.2.1 Step 1: Candidate Generation
text: The downloaded subtitles are not high-quality as expected by our observation. Therefore, we introduce expert ASR systems following Zhang et al. (2022). In particular, we select Microsoft STT3,
2https://github.com/allenai/science-parse 3https://azure.microsoft.com/en-us/products/
ai-services/speech-to-text
and Whisper-large-v2 (Radford et al., 2023)4. They both achieve a WER of less than 10% or even 5% on multiple test sets indicated in the SpeechColab ASR Benchmark (EN)5 in Oct. 2022.
The Microsoft transcriptions in written and spoken form6 are denoted as Mw and Ms respectively, and the Whisper transcriptions result in written form are denoted as Ww (only the written form exists in the Whisper result).

heading: 4.2.2 Step 2: Audio Segmentation
text: Most ASR models expect that the input audio comes in the form of a relatively short duration, usually up to a few seconds in length. Therefore, we conduct re-segmentation using the word-level timestamps in Mw. The split is allowed to be in silence and punctuation. And the duration of each segment is kept to 10 seconds or less. Detailed rules can be found in Appendix A.1.
4https://github.com/openai/whisper 5https://github.com/SpeechColab/Leaderboard 6“Spoken form” refers to the representation of language as it is actually spoken or pronounced, while “written form” refers to the representation of language in its written or textual format.

heading: 4.2.3 Step 3: Dataset Partition
text: We divide all the videos into training, development, and test sets, with a ratio of roughly 8:1:1 for total duration. We also avoid a speaker appearing in different sets, which is identified by the introductory information and description text.

heading: 4.2.4 Step 4: Candidate Combination
text: For the training set, we directly adopt the Microsoft transcription Mw and Ms, because it shows high quality verified by checking against manual labelling on a subset and has both spoken and written form. For the development and test set, we design the candidate combination process shown in Figure 4.
In Case I where Mw is "approximately equal"7 to Ww, Ms and Mw will be directly adopted. In Case II where Mw is not "approximately equal" to Ww, we design the following steps: 1. We use the misalignment to mask the words
in Ms as [???]. 2. Annotators are assigned to fill in the masks.
The result, Mas can be regarded as a type of data augmentation. 3. Annotators are instructed to combine Ms and Mas to provide the final corrected results M c s . The postprocessing of M cs is detailed in Appendix A.3. The manual labelling process is detailed in Appendix A.4. The comparison with manual labelling on a subset for the training set and the manual processing for the dev/test set ensure the accuracy and consistency of the speech annotation.

heading: 4.3 Slide Annotation
text: The diagram of creating slide annotation is shown in Figure 5, comprising four steps.

heading: 4.3.1 Step 1: Extensive Pre-Annotation
text: We conduct frame sampling for the video at 1 frame per second and use PaddleOCR8 for large-scale text detection and recognition. PaddleOCR is selected for its completeness and high accuracy of the recognition results with little sticking between words.

heading: 4.3.2 Step 2: Slide Deduplication
text: Since the frames are repetitive, we first perform automatic deduplication. Specifically, we remove uninformative frames and blocks and filter out frames with too few or too many blocks. Next, we compare
7It forgives misalignments due to pauses and common words, as they are mainly short-pronounced, and Mw and Ms have high confidence in them.
8https://github.com/PaddlePaddle/PaddleOCR
the OCR contents of neighbouring frames and only keep the latter one when the content of the former is entirely included. Detailed criteria are shown in Appendix A.2. After completing the automatic deduplication, manual deduplication is performed to retain the last frame in each segment.

heading: 4.3.3 Step 3: Manual Correction
text: For frames with simple text, corrections are directly executed by experienced researchers for the location and content of the OCR results. Note that blocks of presentation-irrelevant text such as URLs and email addresses are omitted. For frames with complex handwriting or formulae, we pre-label them using the Mathpix OCR API9. Then, the annotator performs careful corrections for the OCR result due to their complexity. The formulae are labelled in the format of in-line LaTeX. The details of manual labelling are shown in Appendix A.4. To control the quality of labelling, we sampled the results of 9 annotators individually and checked whether 120 of the annotated images had a high number of missing or incorrect items, with no more than 10 wrong items.

heading: 4.3.4 Step 4: Rule-Based Merging
text: To provide comprehensive semantic information, we devise three rules to perform merging10:
• Similar Height:
max(ha0, h b 0)−min(ha0, hb0)
≤ a×max(ha0, hb0) (1)
• High Overlap in the Horizontal Direction:
min(ax2 , b x 1)−max(ax3 , bx0)
≥ b×min(bx1 − bx0 , ax2 − ax3) (2)
9https://mathpix.com/ocr 10As demonstrated in the top part of Figure 1, the recognition results of the general OCR model are in rows. It is required to merge these lines into paragraphs to provide complete semantic information.
• Proximity of the Upper and Lower:
by0 − ay3 ≤ c×min(ha0, hb0) (3) where the notations are shown as Figure 5. We set a = 0.8, b = 0.8, and c = 0.6.

heading: 5 Benchmarks and Experiments

heading: 5.1 ASR and CASR Task

heading: 5.1.1 Task Description
text: E2E approaches have gained substantial attention for ASR, which enable direct mapping from raw speech audio to its transcription. However, they often struggle with the accurate recognition of rare words that carry important information for understanding. CASR aims to improve the model’s recognition performance for these rare words by incorporating them into the biasing list.

heading: 5.1.2 Benchmark Systems
text: We adopt two widely-used benchmark systems for ASR and CASR, namely Attention-based EncoderDecoder (AED) (Chan et al., 2016) and RNNTransducer (RNN-T) (Graves, 2012). Both systems employ a Conformer (Gulati et al., 2020) encoder. The AED system is jointly trained and decoded with Connectionist Temporal Classification (CTC) (Graves et al., 2006; Watanabe et al., 2017).
For ASR, we also introduce a variety of state-of-the-art pretrained ASR models. Note that the results of Whisper-v3 are for reference only, as the Whisper-series model is involved in dataset construction. The performance of the other open-source ASR models can be fairly evaluated, given that our videos are properly transcribed for the first time, avoiding the issue of training data contamination. These pretrained models are selected from the SpeechColab ASR Benchmark (EN)11 and Hugging Face
11https://github.com/SpeechColab/Leaderboard
Community12 which include Whisper-v3 (Radford et al., 2023), Wav2Vec2+CTC (Grosman, 2021), Conformer+Transducer13, Conformer+CTC14 and SeamlessM4T-v2 (Barrault et al., 2023)15.
For CASR, the Tree-Constrained Pointer Generator (TCPGen) (Sun et al., 2023b) with GNN tree encodings (Sun et al., 2023a) is used which employs a combination of symbolic prefix-tree search and a neural pointer generator for contextual biasing. More details are illustrated in Appendix B.1 and Appendix B.2.

heading: 5.1.3 Experimental Results
text: In addition to the character error rate (CER) and word error rate (WER) for ASR, we use biasing WER (BWER) for CASR. BWER is defined as the total rare word errors divided by the total number of rare words, including insertions of rare words. Table 5 shows the experimental results. The AED and RNN-T systems achieve around 10.0% WER on both the Dev and Test sets. However, such endto-end models suffer from rare word recognition as reflected by the BWER where a more than two times increase in the error rate is observed comparing BWER to the WER.
When 1000 distractors16 are randomly selected
12https://huggingface.co/models?pipeline_tag= automatic-speech-recognition&sort=trending
13https://catalog.ngc.nvidia.com/orgs/nvidia/ teams/nemo/models/stt_en_conformer_transducer_ xlarge
14https://catalog.ngc.nvidia.com/orgs/nvidia/ teams/nemo/models/stt_en_conformer_ctc_xlarge
15https://huggingface.co/facebook/ seamless-m4t-v2-large
16We select a certain number of significant words, which are the words in the bias list we aim to recognize accurately. When decoding, a particular utterance only includes a few of these words, while the rest will appear as distractors. The use
from all rare words and are added to the biasing list, significant improvements in rare word recognition can be observed. When using OCR information together with rare words from the same topic to form the biasing list of each utterance during decoding, the recognition performance slightly deteriorates. This is because strongly related words are included in the biasing list, which may cause more confusion since words with related meanings often have similar pronunciations and word forms. When only OCR words are used, we achieve the best performance of the CASR task. In conclusion, by using TCPGen utilizing the OCR information, we achieve a relative BWER decrease of 37.8% and 34.2% on dev and test sets respectively.

heading: 5.2 Spontaneous TTS Task

heading: 5.2.1 Task Description
text: Spontaneous Text-to-Speech (TTS) (Guo et al., 2021; Yan et al., 2021) focuses on generating speech with a natural and conversational session. Most TTS systems use clean speech. This form makes it easier to extract pronunciation features, while it places higher requirements on the quality of the training audio and often leads to a discernible gap between the synthesized output and authentic speech. We are devoted to using real speech data in M3AV for the TTS system to produce speech that aligns more closely with natural conversational patterns, as it is spontaneous and has regulated pronunciations.

heading: 5.2.2 Benchmark Systems
text: Multi-codebook vector quantized TTS (MQTTS) (Chen et al., 2023) is introduced as our spontaneous
of distractors follows Le et al.; Sun et al..
TTS baseline. It employs a two-stage training architecture. In the first stage, a quantizer is trained to discretize the raw waveform into a sequence of codes, minimizing quantization and reconstruction losses. A discriminator guides reconstruction via adversarial learning. In the second stage, the fixed quantizer is combined with a Transformer that performs autoregressive generation on the code sequence, conditioned on a speaker embedding for multi-speaker capability. More details can be found in Appendix B.3.

heading: 5.2.3 Experimental Results
text: For the objective metrics, we adopt the F0 Frame Error (FFE) (Chu and Alwan, 2009) and MelCepstral Distortion (MCD) (Kubichek, 1993). For the subjective metrics, the Mean Opinion Score (MOS) test on a scale of 1 to 5 is performed, which is detailed in Appendix A.4. We randomly choose 20 samples and generate speech segments with the MQTTS model and well-known pre-trained models, including Bark17 and SpeechT5 (Ao et al., 2022)18. Table 6 shows the experimental results, in which the MQTTS model shows the best performance within all the evaluation metrics. It indicates that the real speech in our dataset can drive AI systems to simulate more natural speech.

heading: 5.3 Slide and Script Generation Task

heading: 5.3.1 Task Description
text: We define the task formally as follows. Let I , Ti and Ts be the slide picture, slide text and the speech transcription of every segment, respectively. For the “Slide→Script” task, we need to generate Ts based on I . Note that Ti can be viewed as the intermediate result generated from I by the OCR model. For the “Script→Slide” task, we need to generate Ti based on Ts. The Slide and Script Generation (SSG) Task is designed to promote AI models to understand and reconstruct advanced academic knowledge, thus assisting researchers in
17https://huggingface.co/suno/bark 18https://huggingface.co/microsoft/speecht5_tts
creating slides and presentations. The selected paper sentences are provided for retrieval-augmented generation (RAG), supplementing rich knowledge. Such models with academic comprehension can facilitate researchers to handle frequently updated academic materials to acquire knowledge and innovation efficiently.

heading: 5.3.2 Benchmark Systems
text: Having witnessed the excellent performance of large language models (LLMs) and large multimodal models (LMMs) nowadays, we explore their capabilities there. For the "Slide→Script" task, we adopt multimodal cascading pipelines and integrated systems. The cascading pipeline consists of an OCR model to extract the text (we directly adopt the OCR annotations) and a language model to generate the answer. Specifically, the cascading pipelines include fine-tuned LLaMA-2 (Touvron et al., 2023) and GPT-4 (OpenAI, 2023a) using slide OCR text and the integrated systems include fine-tuned InstructBLIP (Dai et al., 2023) and GPT4V (OpenAI, 2023b,c). For the "Script→Slide" task, we adopt fine-tuned LLaMA-2 and GPT-4. For the selection of related paper sentences, We calculate the similarity between the script text and each paper sentence and filter from them. More details can be found in Appendix B.4.

heading: 5.3.3 Experimental Results
text: ROUGE (Lin, 2004), BERTScore (Zhang et al., 2019), and BARTScore (Yuan et al., 2021) results are reported in Table 7 and Table 8. We also perform manual scoring, where each sample is scored on a scale from 1 (worst) to 5 (best) by 3 annotators on multiple dimensions, including coherence, consistency, fluency and relevance (Fabbri et al., 2021). We show their average values, while the detailed values are in Appendix B.5. Scoring is performed on a subset of size 100, detailed in Appendix A.4.
We get significant observations: 1) The opensource models (LLaMA-2, InstructBLIP) show a limited performance improvement when raised from 7B to 13B. Their performances are far from the closed-source models (GPT-4 and GPT-4V). We believe that high-quality pre-training data, e.g., informative corpus and visual QA data which encapsulate multimodal information, is required to enhance their SSG performances beyond just boosting the model size. 2) The latest LMM (GPT4V) has already exceeded the cascaded pipeline composed of unimodal expert models. It sug-
gests that the LMM not only maintains the ability to process textual information but also possesses multi-sensory capabilities, such as the perception and recognition of the slides (Yang et al., 2023). 3) RAG substantially enhances the generation, as shown in Table 8. Qualitative analyses in Appendix C provide a more fine-grained view.
As for future development, we are urged to strengthen the open-source model’s ability to understand high-level knowledge and perceive multimodal information. They can be achieved by increasing the model size and supplying high-quality data. The issue that the open-source multimodal models struggle with recognising text in images can be mitigated by increasing the OCR input. Furthermore, how to effectively bring in external knowledge is an essential topic.

heading: 6 Conclusion
text: We release the Multimodal, Multigenre, and Multipurpose Audio-Visual Dataset with Academic Lectures (M3AV) covering a range of academic fields. This dataset contains manually annotated speech transcriptions, slide text, and additional extracted papers, providing a basis for evaluating AI models for recognizing multimodal content and understanding academic knowledge. We detail the cre-
ation pipeline and conduct various analyses of the dataset. Furthermore, we build benchmarks and conduct experiments around the dataset. We find there is still large room for existing models to improve perceptions and understanding of academic lecture videos.
Limitations
There are some limitations in our work. First, there may exist biases in the dataset, such as the types of presentations and the demographics of the speakers. It is hard to completely avoid biases, although we have incorporated a diverse range of data sources. The users of the dataset should be aware of the impact brought to the AI models by these biases.
Secondly, the data we collected has limitations in terms of academic domains and multilingualism. We will consider adding videos from more research directions, such as humanities disciplines (e.g. economics, law, and sociology) to an extended dataset in future work. And we also plan to collect more data in other languages to prompt multilingual academic research. Nevertheless, we believe our current collection would already be useful for the community to develop some initial approaches.
Thirdly, we have some deficiencies in mining and labelling for visual information of the dataset.
The illustrations in the video frames are not extracted separately like Lee et al. (2022, 2023). And, the talking heads of the speakers are not picked up, which can be utilised to research the generation of virtual presenters. In summary, further exploitation of the visual elements in the slides would be a future extension of the dataset.
Finally, the benchmark systems for the Slide Generation task do not involve the generation of visual pictures, instead, we only get the generated slides in the form of text. The generation of slide images requires complex multimodal models, and the corresponding evaluation metrics require more sophisticated designs.
Ethical Consideration
This work presents a dataset based on academic speeches with manual annotation in terms of slides and speech. The presentations and talks included in the dataset, are open accessed on the YouTube website. The copyright remains with the original owners of the videos. Our dataset is provided under a Creative Commons Attribution-NonCommercialShareAlike 4.0 International License. The annual annotation is done by the team’s researchers and workers on the Amazon Mechanical Turk. All annotators received clear annotation rules before the annotation and fair payments upon completion. No personal information is requested from any of the annotators.

heading: Acknowledgements
text: This work is supported by National Key R&D Program of China (No. 2022ZD0162101), National Natural Science Foundation of China (No. 62106140) and STCSM (No. 21511101100, No. 22DZ2229005).

heading: A Creation Pipeline Details
text: A.1 Audio Segmentation Rules
We establish the following rules of segmentation:
1. Split is allowed when the silence exceeds 0.2 seconds or there is sentence break punctuation (".", "!", "?"). 2. If the accumulated "small segments" obtained are greater than or equal to 8 seconds or the accumulation is greater than or equal to 10 seconds after the next "small segment" is added, then the accumulated segments will form a new segment. 3. If the silence exceeds 5 seconds, then this silence is discarded and a new segment is formed from the accumulated "small segments". The first rule aims to preserve boundary words as well as the semantics of sentences inspired by LibriSpeech (Panayotov et al., 2015) and GigaSpeech (Chen et al., 2021). The second rule controls the duration of each segment, while the third rule avoids too long silence within a segment.
A.2 Auto Slide Deduplication Details
1. For the frames that contain some OCR block representing the video cover Bcover, we delete these images directly. And we remove fixed appearing logo blocks Bexcluded, such as "NIH". 2. Then, we set a threshold minBalpha to filter blocks with too few letters. For the filtering process for images, we set the thresholds minIB and max I B to filter images with too few
blocks and too many blocks, respectively. 3. After the above filtering, we merge all the
blocks of each image into long text by their coordinates. 4. Initially, we assign the first image to segment 0. Next, we will iterate through each image chronologically: if the error rates (ER) between the current image (segment i) content and all past image contents are all greater than the threshold minER, then it is considered as a new image and assigned to segment i + 1, otherwise, it is not and kept as segment i. 5. Eventually, the last image in each segment will be the result of the deduplication since they are not repeating each other and contain the most text of the images in their segment.
We would like to emphasize that the thresholds mentioned above depend on the characteristics of each source of videos, such as the clarity of the picture, the density of the text, and the presence of handwriting and formulae. Moreover, we adopt a modified Word Error Rate (WER) or Char Error Rate (CER) in the calculation of ER mainly considering the incremental content of slides. They are formulated as follows:
WERmod = S +D + 0× I H + S +D
(4)
CERmod = S +D + 0.1× I
H + S +D (5)
where S, D, I , and H represent the number of substitutions, deletions, insertions, and hits, respectively. The choice of WER or CER also depends on the aforementioned characteristics of the videos.
A.3 Postprocessing for Manual Combination Results
To preserve the consistency of the data after obtaining M cs shown in Figure 4, we perform inverse text normalization19, add punctuations, and capitalize words using tools20. Finally, we obtain the wordlevel timestamps of M cs using Montreal Forced Aligner (McAuliffe et al., 2017)21.
A.4 Manual Annotation Details
The master-granted annotators are assigned to complete manual filling and combination in creating speech transcription. In these two tasks, each worker receives 0.06 USD for each finished sample, respectively. The annotation interfaces shown in Figure 12 and Figure 13 describe the rules as well as the rejection cases of manual filling and combination in detail, respectively. The failed samples are passed to other workers for annotation.
The experienced researchers in our team are assigned to annotate the OCR data, and the annotation of each sample includes slide deduplication and OCR content correction. Each worker received the appropriate remuneration. The interface for OCR content correction is shown in Figure 14,
19https://docs.nvidia.com/deeplearning/ nemo/user-guide/docs/en/main/nlp/text_ normalization/wfst/wfst_text_normalization.html# inverse-text-normalization
20https://docs.nvidia.com/deeplearning/nemo/ user-guide/docs/en/main/nlp/punctuation_and_ capitalization.html
21https://github.com/MontrealCorpusTools/ Montreal-Forced-Aligner
where PPOCRLabel22 is adopted. To control the quality of labelling, we sampled the results of 9 annotators individually and checked whether 120 of the annotated images had a high number of missing or incorrect items, with no more than 10 wrong items.
The master-granted annotators are assigned to complete the MOS scoring in the TTS task evaluation. A batch of 6 TTS results from the same audio clip (with the addition of a trap sample used to detect attentiveness). Each worker receives 0.2 USD for each completed batch respectively. Similarly, the annotation interface shown in Figure 15 details the rules as well as the rejection cases, which is inspired by Choi et al. (2023). The failed samples will be passed to other workers for annotation.
As for the manual scoring of SSG tasks, we posted the annotation task on Amazon Mechanical Turk initially but found that the quality was unsatisfactory due to the complexity of the scripts and speech texts. Therefore, three experienced researchers were assigned to carry out the annotation task. Specifically, annotators need to rate the generated slides/scripts on multiple dimensions, including coherence, consistency, fluency and relevance, as referenced in SummEval (Fabbri et al., 2021). Each sample is scored on a scale from 1 (worst) to 5 (best) by 3 annotators. Note that we sampled 100 items due to cost and multiple models. The annotation interfaces are shown in Figure 16 and Figure 17.

heading: B Experiment Details
text: B.1 ASR Benchmark Details
We construct the Attention-based Encoder-Decoder (AED) with the help of the well-known ESPnet (Watanabe et al., 2018) toolkit. We use Conformer (Gulati et al., 2020) as the encoder, which consists of 12 blocks, with hidden linear units dff = 2048 and attention output size datt = 512. Attention head H is set to 8 and the front CNN kernel size is 31. As for the decoder, we use the Transformer with 6 blocks (dff = 2048,H = 8). We use BPE as our recognition unit, which consists of 1k tokens obtained by SentencePiece (Kudo and Richardson, 2018). In addition to the attention training objectives, We use a certain degree of CTC loss function. CTC weight λ is 0.3 during training, while 0.2 for infering. SpecAugment (Park et al.,
22https://github.com/PaddlePaddle/PaddleOCR/ tree/release/2.7/PPOCRLabel
2019) is applied with time mask width T = 40 and frequency mask with F = 30. Our model is trained on four 24GB 3090 RTX GPUs for 60 epochs. The top 10 checkpoints after one epoch training are preserved for model averaging.
For the RNN-Transducer (RNN-T), we also adopt Conformer as the encoder, with an intermediate joint network of 320 dimensions. The Conformer has 15 blocks, 1024 hidden linear units, 4 attention heads, and 256 attention output dimensions. After that, a single layer of LSTM is performed.
B.2 CASR Benchmark Details Our CASR task is performed under two paradigms. One paradigm is to form the biasing list for an utterance using target biasing words together with distractors that are randomly selected from all rare words, as a simulation of the real-world scenario (Aleksic et al., 2015; Zhao et al., 2019; Pundak et al., 2018; Le et al., 2021b). Our experiment uses 1000 distractors to form the biasing list for each utterance, and rare words are defined by removing the top 5k most frequent words from the vocabulary. Another paradigm is audio-visual CASR (Sun et al., 2022) which extracts the biasing list from visual information, such as OCR annotation.
As for TCPGen, we use a 6-layer GCN (Kipf and Welling, 2017) for future information representation. Following the method proposed in (Le et al., 2021a), the biasing lists are extracted by identifying words in the speech transcription of each utterance that also appears in the total rare words list, followed by the addition of plenty of distractors to simulate a real-life setting. The length of the biasing list is set to 500 during training and 1000 during inferencing. Model training lasts for 70 epochs, and the best 10 of the checkpoints are saved for model averaging. Our models are trained on 24GB 3090 RTX GPUs.
Before model training, the rare word selection is first performed. Our speech corpus includes approximately 43k different words in total. We conduct a word frequency analysis on the training set, identifying words with a frequency ranking exceeding 5k as members of the total rare word set. The number of our rare word list is about 38k.
B.3 Spontaneous TTS Benchmark Details We followed the settings recommended by the architecture authors, and their experiments were trained on a subset of GigaSpeech (Chen et al., 2021), which is a speech corpus containing tran-
scribed audio from audiobooks, Podcasts, and YouTube. We use the training set of the M3AV for MQTTS model training and arbitrarily extract a sufficient number of samples from the Dev and Test Set for speech synthesis to verify the effectiveness of our model. We adopt HiFi-GAN (Kong et al., 2020) as the backbone architecture for the quantizer and discriminator, which we trained for 600k steps. After that, a Transformer is trained to autoregressively generate the code representation based on the past code sequence. The last checkpoints from both steps are preserved. It is worth noting that MQTTS training may take a long time. Our models are trained on 24GB 3090 RTX GPUs.
B.4 Slide and Script Generation Benchmark Details
For the selection of related paper sentences, we adopt the DistilBERT (Sanh et al., 2019) model in the repository of sentence-transformers (Reimers and Gurevych, 2019) 23 to calculate the similarity between the script text and each paper sentence. Then we select the top 3 candidates with similarity greater than 0.5 as the selection results. In the training phase of benchmark systems, we use LoRA (Hu et al., 2022) to fine-tune LLaMA2 (Touvron et al., 2023). The optimizer we use is AdamW (Loshchilov and Hutter, 2018). The learning rate curve is 1 epoch of warm-up and then drops with a peak learning rate of 2× 10−4. The total number of training epochs is 3. Our models are trained on 24GB 3090 GPUs and 48GB A6000 GPUs. Fine-tuned InstructBLIP (Dai et al., 2023) is another baseline. Specifically, we fine-tune the officially released checkpoint for 20 epochs with the learning rate increasing linearly from 1× 10−7 to 5 × 10−5 during the first 1000 steps and dropping to 0 as a cosine function. The model is trained on 80GB A100 GPUs with AdamW (Loshchilov and Hutter, 2018) as the optimizer. The best checkpoints are chosen according to the performance of the development set. As for the generation settings, we use nucleus sampling (Holtzman et al., 2020) with a topp of 0.9. The maximum number of tokens generated is 300. These generation settings are consistent across LLaMA-2, InstructBLIP, GPT-4 and GPT-4V.
23https://huggingface.co/sentence-transformers/ msmarco-distilbert-base-v4
B.5 Detailed Manual Scoring Results on SSG Tasks
We show the detailed scoring results in Table 9 and Table 10, which include the coherence (Coh.), consistency (Cons.), fluency (Flu.), relevance (Rel.) and average value (Avg.) between the hypothesis and its reference text.

heading: C Qualitative Analyses of SSG Benchmark Systems
text: The comparisons of the benchmark systems in “Slide→Script” are shown in Figure 6 and Figure 7. LLaMA-2 and InstructBLIP below refer to models of size 7B. We can find that the scripts generated by GPT-4+OCR and GPT-4V are highly qualified and roughly match the references. The results of LLaMA-2+OCR are acceptable, although slightly weaker than the previous two. All three can develop detailed descriptions around topics in slides, such as the “NIH Outreach Toolkit” in Figure 6 and “Aging and Migration” in Figure 7. InstructBLIP, on the other hand, fails to find the right theme, as shown in Figure 6 and Figure 7, suggesting that today’s open-source LMM models struggle in handling text in slides.
The comparisons of the benchmark systems in “Script→Slide” are shown in Figure 9 and Figure 10. The slides generated by GPT-4 are closer to references than those generated by LLaMA-2, and the contents are more accurate and condensed. For example, GPT-4 presents “Drosophila ToLL” in Figure 9 and “Pipeline” in Figure 10, while
LLaMA-2 does not. We can conclude that GPT-4 has a better understanding of complex academic knowledge than LLaMA-2.
Figure 8 and Figure 11 show the improvement in the quality of the generated scripts and slides brought by external knowledge. The addition of external knowledge enables AI models to produce necessary details, such as “WiFi CSI” in Figure 8 and “13 long-term users” in Figure 11. It shows that the introduction of external knowledge leads to more comprehensive information captured by AI models.

heading: Future Work
text: As for TCPGen, we use a 6-layer GCN (Kipf and Welling, 2017) for future information representation. Following the method proposed in (Le et al. , 2021a), the biasing lists are extracted by identifying words in the speech transcription of each utterance that also appears in the total rare words list, followed by the addition of plenty of distractors to simulate a real-life setting. The length of the biasing list is set to 500 during training and 1000 during inferencing. Model training lasts for 70 epochs, and the best 10 of the checkpoints are saved for model averaging. Our models are trained on 24GB 3090 RTX GPUs. Before model training, the rare word selection is first performed. Our speech corpus includes approximately 43k different words in total. We conduct a word frequency analysis on the training set, identifying words with a frequency ranking exceeding 5k as members of the total rare word set. The number of our rare word list is about 38k. B. 3 Spontaneous TTS Benchmark Details We followed the settings recommended by the architecture authors, and their experiments were trained on a subset of GigaSpeech (Chen et al. , 2021), which is a speech corpus containing tran-
scribed audio from audiobooks, Podcasts, and YouTube. We use the training set of the M3AV for MQTTS model training and arbitrarily extract a sufficient number of samples from the Dev and Test Set for speech synthesis to verify the effectiveness of our model. We adopt HiFi-GAN (Kong et al. , 2020) as the backbone architecture for the quantizer and discriminator, which we trained for 600k steps. After that, a Transformer is trained to autoregressively generate the code representation based on the past code sequence. The last checkpoints from both steps are preserved. It is worth noting that MQTTS training may take a long time. Our models are trained on 24GB 3090 RTX GPUs. B. 4 Slide and Script Generation Benchmark Details
For the selection of related paper sentences, we adopt the DistilBERT (Sanh et al. , 2019) model in the repository of sentence-transformers (Reimers and Gurevych, 2019) 23 to calculate the similarity between the script text and each paper sentence. Then we select the top 3 candidates with similarity greater than 0. 5 as the selection results. In the training phase of benchmark systems, we use LoRA (Hu et al. , 2022) to fine-tune LLaMA2 (Touvron et al. , 2023). The optimizer we use is AdamW (Loshchilov and Hutter, 2018). The learning rate curve is 1 epoch of warm-up and then drops with a peak learning rate of 2× 10−4. The total number of training epochs is 3. Our models are trained on 24GB 3090 GPUs and 48GB A6000 GPUs. Fine-tuned InstructBLIP (Dai et al. , 2023) is another baseline. Specifically, we fine-tune the officially released checkpoint for 20 epochs with the learning rate increasing linearly from 1× 10−7 to 5 × 10−5 during the first 1000 steps and dropping to 0 as a cosine function. The model is trained on 80GB A100 GPUs with AdamW (Loshchilov and Hutter, 2018) as the optimizer. The best checkpoints are chosen according to the performance of the development set. As for the generation settings, we use nucleus sampling (Holtzman et al. , 2020) with a topp of 0. 9. The maximum number of tokens generated is 300. These generation settings are consistent across LLaMA-2, InstructBLIP, GPT-4 and GPT-4V. 23https://huggingface. co/sentence-transformers/ msmarco-distilbert-base-v4
B. 5 Detailed Manual Scoring Results on SSG Tasks
We show the detailed scoring results in Table 9 and Table 10, which include the coherence (Coh. ), consistency (Cons. ), fluency (Flu. ), relevance (Rel. ) and average value (Avg. ) between the hypothesis and its reference text