SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Limitations. There are several limitations associated with MAViL. Firstly, the scale of the data poses a limitation. The AudioSet [28] dataset used by MAViL, with two million samples, is approximately two orders of magnitude smaller than the text corpora used in recent language models [20, 54, 9]. It is also an order smaller than image corpora like ImageNet-21K used by MBT [64].
Another limitation pertains to the duration of each audio sample. The 10-second recording in AudioSet are relatively short, which can hinder the proper learning of distant temporal dependencies in audio and video. This limitation restricts the potential applicability of MAViL to tasks that require modeling longer audio sequences, such as automatic speech recognition (ASR). Regarding video modeling, due to GPU memory constraints and choice of video footprints, MAViL only models 4-second video segments. This limitation makes it challenging to effectively model long video sequences. Additionally, the presence of low-quality videos and misaligned audio-video pairs in AudioSet may adversely affects pre-training.
Potential Societal Impacts. The datasets used in this paper, including AudioSet and other end task datasets, were properly licensed and publicly available at the time of data collection. It is important to note that some of the data may have been removed by YouTube or the dataset uploaders. Most of the data in these datasets are licensed under the Creative Commons BY-NC-ND 4.0 license or the Creative Commons 3.0 International License.
To investigate the bias in AudioSet, we selected 200 videos containing speech. In these videos, we did not observe any visual bias in the sampled speakers, which encompassed a wide range of ages, races, and genders. However, it is possible that there may be biases in the distribution of population and ethnicity within AudioSet. It is important to exercise caution and be aware of the potential unintended gender, racial, and societal biases present in AudioSet, which serves as the pre-training data for MAViL.
Given that AudioSet consists of a vast collection YouTube videos, there is a potential risk that MAViL could learn to reconstruct sensitive personal information, which could then be exploited for malicious purposes, including the creation of audio deepfakes [81, 16]. To address this concern, the released MAViL would be discriminative models, specifically the audio and video encoders, rather than generative models such as decoders. This shift aims to mitigate the potential risks associated with generating synthetic content that could be misused.