WhisBERT: Multimodal Text-Audio Language Modeling on 100M Words
Authors: name: Lukas Wolf, name: Klemen Kotar, name: Greta Tuckute, name: Eghbal Hosseini, name: Tamar Regev, name: Ethan Wilcox, name: Alex Warstadt

Training on multiple modalities of input can augment the capabilities of a
language model. Here, we ask whether such a training regime can improve the
quality and efficiency of these systems as well. We focus on text--audio and
introduce Whisbert, which is inspired by the text--image approach of FLAVA
singh_flava_2022. In accordance with Babylm warstadt2023papers
guidelines, we pretrain Whisbert on a dataset comprising only 100 million words
plus their corresponding speech from the word-aligned version of the People's
Speech dataset galvez_peoples_2021. To assess the impact of
multimodality, we compare versions of the model that are trained on text only
and on both audio and text simultaneously. We find that while Whisbert is able
to perform well on multimodal masked modeling and surpasses the Babylm
baselines in most benchmark tasks, it struggles to optimize its complex
objective and outperform its text-only Whisbert baseline.