SlideAVSR: A Dataset of Paper Explanation Videos for Audio-Visual Speech Recognition (2024)

Abstract: Audio-visual speech recognition (AVSR) is a multimodal extension of automatic speech recognition (ASR), using video as a complement to audio. In AVSR, considerable efforts have been directed at datasets for facial features such as lip-readings, while they often fall short in evaluating the image comprehension capabilities in broader contexts. In this paper, we construct SlideAVSR, an AVSR dataset using scientific paper explanation videos. SlideAVSR provides a new benchmark where models transcribe speech utterances with texts on the slides on the presentation recordings. As technical terminologies that are frequent in paper explanations are notoriously challenging to transcribe without reference texts, our SlideAVSR dataset spotlights a new aspect of AVSR problems. As a simple yet effective baseline, we propose DocWhisper, an AVSR model that can refer to textual information from slides, and confirm its effectiveness on SlideAVSR.

heading: 1 Introduction
text: Research on multimodal models capable of handling multiple types of data, such as language, images, videos, and audio simultaneously, has garnered significant attention. An example is audiovisual speech recognition (AVSR), a multimodal extension of automatic speech recognition (ASR), using video as a complement to audio. Most previous studies in AVSR have been conducted with the aim of improving accuracy on lip reading datasets (Afouras et al., 2018a,b). While models built in these studies (Shi et al., 2022; Pan et al., 2022; Haliassos et al., 2023) demonstrate high performance on lip reading data, their applicability to other types of videos remains limited.
In this paper, we aim to evaluate the image comprehension capabilities of AVSR models across a broader spectrum of visual contents than facial features. To achieve this, we construct SlideAVSR, an AVSR dataset that contains various technical
terms that are notoriously challenging to transcribe without referring to textual information on slides. Specifically, we collect scientific paper explanation videos from YouTube, apply data refinement procedures with several custom filters, and perform data partitioning considering the speakers’ accents.
Furthermore, we propose DocWhisper, a simple yet effective AVSR baseline that can efficiently refer to the content of slides using optical character recognition (OCR). In experiments utilizing SlideAVSR, DocWhisper demonstrated a performance improvement of up to 14.3% compared to Whisper (Radford et al., 2022), which relies solely on audio input. Additionally, to address the long-tail problem in OCR results, we introduce FQ Ranker, which calculates word ranks based on the frequency of word occurrences, and we evaluate its effectiveness integrated with DocWhisper.

heading: 2 Related Work
text: Compared to the efforts that have been made on lip reading datasets (Chung et al., 2017; Chung and Zisserman, 2017a,b; Afouras et al., 2018a,b; Shillingford et al., 2019), AVSR datasets in other types of videos remain scarce. VisSpeech (Gabeur et al., 2022) is constructed from a subset of the instructional video dataset HowTo100M (Miech et al., 2019) where the visual stream and speech audio are semantically related. The audiovisual diarization benchmark in the Ego4D challenge (Jain et al., 2023) consists of 585 egocentric video clips. In terms of utilizing textual information extracted from videos, SlideSpeech (Wang et al., 2023) builds an AVSR dataset on online conference videos enriched with slides. However, the filtering process of SlideSpeech relies heavily on human annotators. In this work, we exclude videos and utterances that do not correspond to slides by utilizing a vision language model. This approach reduces annotation costs and enhances the purity and quality of our dataset within the slide domain.
129
In the context of extending Whisper to an AVSR model, Peng et al. (2023) employed CLIP (Radford et al., 2021) to transform the input visual stream into word sequences, which were then utilized as prompts for Whisper. They reported that this approach enhances the zero-shot performance on VisSpeech. In this study, we employ OCR to create prompts and implement fine-tuning to improve performance rather than using zero-shot prompting.

heading: 3 SlideAVSR: Dataset Construction
text: In this study, we construct SlideAVSR, an AVSR dataset based on scientific paper explanation videos incorporating various technical terms, making accurate transcription difficult without referring to the slides. Based on JTubeSpeech (Takamichi et al., 2021), a framework for building audio corpora from YouTube videos, we implement several custom filters to target videos, thereby applying highprecision data refinement. This section describes the construction flow of SlideAVSR. Figure 1 illustrates the flow.

heading: 3.1 Data Collection
text: Creating search queries. We first collect videos with search queries that are related to top conferences in the field of artificial intelligence. We create queries in the format {Conference} {Year} {Form} . The list of target conferences is provided in Appendix A. Considering the increased prevalence of online conferences since COVID-19, we focus on the years 2020 to 2023. The forms include “paper”, “workshop”, and “talk”. An example search query is “ACL 2023 paper”.
Obtaining videos with subtitles. Using the search queries, we retrieve video IDs with subtitles and download them.1 To ensure data quality, only videos with manual subtitles are considered. Additionally, we set the following criteria:
• Duration between 5 and 20 minutes (videos that are too short or too long are less likely to be paper explanation videos). • Video format: MP4, 720P, H264. • Audio format: single-channel, 16bit, 16kHz.
A total of 636 videos were downloaded.

heading: 3.2 Filtering
text: We curate several filters to remove videos that are not paper explanations or do not include slides.
1https://github.com/yt-dlp/yt-dlp
ChatGPT filter. We provide the videos’ description for ChatGPT2 to confirm the following:
• This video is an explanation of a paper. • The description is written in English.
We perform three times of generation, and if “Yes” is outputted at least once, we adopt the video; otherwise, we discard it. We show the details of the model and prompt in Appendix B. A total of 342 videos were excluded, leaving 294 videos for subsequent processes.
BLIP-2 filter for videos. We capture screenshots at the beginning, end, and three quartile points in the timeline for each video, and then present these screenshots to the vision language model BLIP2 (Li et al., 2023) to verify the following:
• This image is a screenshot, not a photo. • This image is a part of slides.
We perform generation for each screenshot, and if “Yes” is outputted at least once, we adopt the video; otherwise, we discard it. We show the details of the model and prompt in Appendix B. A total of 6 videos were excluded, leaving 288 videos for subsequent processes.
Manual filter. We conduct manual checks to remove inappropriate videos that are not excluded by the automatic filters, including:
• Videos rarely showing slides. • Videos unrelated to paper explanations, such
as conference openings.
A total of 38 videos were excluded, leaving 250 videos for subsequent processes.

heading: 3.3 Cleansing
text: We implement audio-subtitle alignment, exclude utterances that do not correspond to slides, and merge short utterances for data cleansing.
2https://openai.com/product
CTC alignment. Due to the inaccuracy in the timing of subtitles, we implement audio-subtitle alignment and scoring using CTC segmentation (Kürzinger et al., 2020). We set the threshold to -7 and exclude utterances with lower scores. The details of the model are shown in Appendix B. Approximately 2.5% of utterances were excluded through this process.
BLIP-2 filter for utterances. We capture screenshots at the midpoint of each utterance, followed by filtering using BLIP-2. Three generations are conducted for each screenshot, and if “Yes” is outputted at least once, we adopt the utterance; otherwise, we discard it. The employed prompt is identical to the BLIP-2 filter in Section 3.2. Approximately 1.0% of utterances were excluded through this process.
Merging utterances. Subtitles created by video authors occasionally exhibit unnatural segmentation, resulting in exceedingly brief spans. Utilizing the audio segments obtained through CTC segmentation, we implement a merging process, combining two consecutive utterances into a single entity if the end time of the preceding utterance aligns with the start time of the subsequent one and their cumulative duration does not exceed 15 seconds. This procedure significantly enhanced Whisper’s ASR performance by approximately 20%.

heading: 3.4 Data Partitioning
text: Previous studies (Meyer et al., 2020; Javed et al., 2023; DiChristofano et al., 2023) have suggested that the performance of ASR systems significantly varies depending on the speaker’s accent3. Based on the hypothesis that visual information contributes to the recognition of challenging accents, we ask native English speakers to classify the speakers’ accents in SlideAVSR and perform dataset partitioning. We partition the dataset into Train, Dev, and TestA, reserving a smaller yet significant TestB subset for South Asian English (SAE) accents. During partitioning, we have ensured that the same speaker did not belong to multiple partitions. Additionally, 5 videos with machine-generated audio were manually excluded by the annotators.
Through the construction flow, we produced an AVSR dataset of around 36 hours from 245 videos. We show the statistics of the dataset in Table 1.
3The term “accent” in this paper refers to comprehensive prosodic information, including accent, intonation, tone, etc.

heading: 4 Experiments

heading: 4.1 Approaches
text: DocWhisper processes the input video stream through an OCR module, extracting textual information into word sequences, which are then provided to Whisper as prompts for fine-tuning and inference. While Peng et al. (2023) employed prompts derived from CLIP in zero-shot learning, our preliminary experiments did not reveal a performance improvement in zero-shot learning on SlideAVSR. Given that Whisper’s pretraining (Radford et al., 2022) did not use prompts, we speculate that Whisper loses robustness when it faces diverse prompts.
We show the frequency distribution of the number of words in OCR results in Figure 2. The distribution is long-tail, which means that only 70% of the samples can be covered even if we include 100 words in the prompts4. To address this issue, we propose FQ Ranker, which calculates word ranks based on the frequency of word occurrences. Given the demonstrated high correlation between word frequency and familiarity as shown in previous studies (Coltheart, 1981; TanakaIshii, 2021), increasing the rank of less frequent and more challenging words is expected to enhance the information content of prompts.
4Whisper typically assigns a maximum length of 224 to prompts, making inputs with over 100 words challenging.

heading: 4.2 Implement Details
text: We used Whisper large-v35 as a base model and Word Error Rate (WER) for evaluation. In the case of DocWhisper, we captured screenshots at the midpoint of each utterance, fed them into the OCR module, and used the recognized text as the prompts to Whisper. In this case, multiple utterances might share the same slide. We use Google Cloud Vision API6 for OCR. The prompts were presented to the model as word sequences, such as “word 1, word 2, ..., word n”. FQ Ranker utilized word frequency counts obtained from the English Wikipedia and sorted the OCR results in ascending order based on word frequency. We conducted experiments with different maximum word counts for prompts (K ∈ {25, 50, 75, 100}) and with or without FQ Ranker. More implementation details are provided in Appendix C.

heading: 4.3 Results
text: We show the results of quantitative evaluations for Whisper and DocWhisper in Table 3. In both models, the scores of the TestB set, consisting of videos with SAE accents, were inferior to the scores of the TestA set, indicating that Whisper struggles with rare accents. With fine-tuning, Whisper demonstrated a 1.9% improvement on the TestA set. However, no notable improvement was observed for the
5https://huggingface.co/openai/whisper-large-v3 6https://cloud.google.com/vision
TestB set. Despite the presence of videos with SAE accents in the training data, their limited quantity was deemed insufficient to address the challenges posed by difficult accents.
Compared to the fine-tuned Whisper, DocWhisper exhibited a maximum improvement of 14.3% on TestA and 11% on TestB. We gather that referring to textual information on slides can significantly improve speech recognition performance on SlideAVSR. We also found that as the maximum word count of prompts increased, the performance improved, indicating that maximizing information content contributes to performance enhancement.
FQ Ranker improved the scores on TestB when the maximum word count of prompts was set to 25; however, this advantage was reversed when the maximum word count exceeded 50. Details provided in Section 4.4 indicate that transcriptions corrected by DocWhisper do not exclusively consist of technical terms, which suggests the potential for misinterpretation even in words with high familiarity. We also speculate that sorting words based on word frequency disrupts the ordered contextual information, thus increasing the difficulty of Whisper’s decoder, which is a language model, to refer to the textual information on the slides.

heading: 4.4 Analysis of Specific Examples
text: Among Whisper’s errors (deletions, substitutions, and insertions), DocWhisper corrected substitution errors the most. To delve into the details, we collected 100 instances that are substitution errors in Whisper but correct in DocWhisper and categorized them into four groups: technical term, inflection, mishearing, and name. While the anticipated large proportion (41%) of technical terms was observed, noteworthy percentages were also found for inflection (28%) and mishearing (24%). Many words with high familiarity could result in lower ranks when sorting based on word frequency, potentially causing a decline in the performance of FQ Ranker. We show the error types and specific examples in Table 2 and more details in Appendix D.

heading: 5 Conclusion and Future Work
text: We constructed an AVSR dataset, SlideAVSR, by utilizing paper explanation videos. We proposed DocWhisper, which leverages OCR to refer to slide content. We verified the effectiveness of DocWhisper on SlideAVSR and conducted a detailed analysis. Additionally, we introduced FQ Ranker, which calculates word ranks based on word frequency, and evaluated its performance on DocWhisper.
In the future, we plan to continually refine OCRbased methods and aim to construct an end-to-end AVSR model that is not dependent on OCR. Furthermore, we intend to build a benchmark that allows a comprehensive evaluation of the image comprehension capabilities of AVSR models by incorporating diverse types of videos, such as sports commentary, gaming commentary, cooking videos, and more. Ultimately, we aim to construct a foundation model for AVSR that exhibits high performance across diverse video inputs.
Limitations
In comparison to mainstream AVSR datasets, SlideAVSR exhibits a notably limited number of videos and speakers. This may lead to data imbalance and create obstacles to the model’s training process. In addition, due to our focused collection of scientific paper explanation videos related to artificial intelligence, imbalances may have emerged in terms of speaker nationality, age, and gender.
Compared to SlideSpeech (Wang et al., 2023), we introduced a vision-language model to filter videos and utterances that do not correspond to the slides, thereby reducing annotation costs and improving the quality of the dataset. However, our dataset construction process still relies on manual annotation. Fully automating this process will be a major challenge for the future.
In Section 3.4, we attempted to classify speakers’ accents by collaborating with native English speakers. However, the task of assigning precise labels to every video was impeded by the complexity of distinguishing certain speakers’ accents. As a result, we selectively picked out videos with South Asian English accents, leaving the remainder unlabeled. Ideally, each data split should exhibit a comparable distribution of accents, but this was unattainable due to the aforementioned challenges.
Ethical Considerations
In adherence to the terms of use and copyright policies governing the YouTube platform, we collected data exclusively from publicly available videos. We acknowledge the potential presence of sensitive information in our dataset, such as personal names and portraits. To prioritize privacy and responsible data sharing, we plan to release OCR results and public video URLs instead of raw video files. Furthermore, the release of our dataset will be strictly limited to research purposes.

heading: A The list of target conferences used in data collection
text: We show our target conferences in Table 4.

heading: B Models and prompts used in data filtering and cleansing
text: We introduce the details of the models and prompts employed in the ChatGPT filter, BLIP-2 filter, and CTC alignment as described in Section 3.2 and 3.3.
ChatGPT filter. We used gpt-3.5-turbo. The prompt we used is shown in Table 5.
BLIP-2 filter. We used blip2-flan-t5-xl7. The prompt we used is shown in Table 6.
CTC alignment. We used kamo-naoyuki_wsj8 and ESPnet implemenations9.
7https://huggingface.co/Salesforce/blip2-flan-t5-xl 8https://huggingface.co/espnet/kamo-naoyuki_wsj 9https://github.com/espnet/espnet
C Implement details
We fine-tuned both Whisper and DocWhisper using AdamW (Loshchilov and Hutter, 2019) with a learning rate of 2e-5, and we linearly warmed up the learning rate over 1,000 steps. The batch size was set to 16. Training was conducted for 10 epochs, and the checkpoint with the best performance on the Dev set was used for evaluation. Additionally, training was performed with three different seed values, and the average was computed. We performed text normalization10 for evaluation. All experiments were conducted on a single NVIDIA A100 (40G) GPU.

heading: D Specific examples
text: The corresponding screenshots to Table 2 are shown below, and the parts referred to in the correction are circled in red.
All the variations from the same lexical element, such as plural nouns, conjugated verbs, and thirdperson singular verbs, were classified as inflection. If the label and prediction are not from the same lexical element, we classified the error as technical terms, mishearing, and names, respectively.
https://www.youtube.com/watch?v=eepUV9NJxFs
https://www.youtube.com/watch?v=dvUutyo72R4
10https://github.com/openai/whisper
https://www.youtube.com/watch?v=0VGKPmomrR8
https://www.youtube.com/watch?v=CQBdQz1bmls

heading: Future Work
text: In the future, we plan to continually refine OCRbased methods and aim to construct an end-to-end AVSR model that is not dependent on OCR. Furthermore, we intend to build a benchmark that allows a comprehensive evaluation of the image comprehension capabilities of AVSR models by incorporating diverse types of videos, such as sports commentary, gaming commentary, cooking videos, and more. Ultimately, we aim to construct a foundation model for AVSR that exhibits high performance across diverse video inputs. Limitations
In comparison to mainstream AVSR datasets, SlideAVSR exhibits a notably limited number of videos and speakers. This may lead to data imbalance and create obstacles to the model’s training process. In addition, due to our focused collection of scientific paper explanation videos related to artificial intelligence, imbalances may have emerged in terms of speaker nationality, age, and gender. Compared to SlideSpeech (Wang et al. , 2023), we introduced a vision-language model to filter videos and utterances that do not correspond to the slides, thereby reducing annotation costs and improving the quality of the dataset. However, our dataset construction process still relies on manual annotation. Fully automating this process will be a major challenge for the future. In Section 3. 4, we attempted to classify speakers’ accents by collaborating with native English speakers. However, the task of assigning precise labels to every video was impeded by the complexity of distinguishing certain speakers’ accents. As a result, we selectively picked out videos with South Asian English accents, leaving the remainder unlabeled. Ideally, each data split should exhibit a comparable distribution of accents, but this was unattainable due to the aforementioned challenges. Ethical Considerations
In adherence to the terms of use and copyright policies governing the YouTube platform, we collected data exclusively from publicly available videos. We acknowledge the potential presence of sensitive information in our dataset, such as personal names and portraits. To prioritize privacy and responsible data sharing, we plan to release OCR results and public video URLs instead of raw video files. Furthermore, the release of our dataset will be strictly limited to research purposes