Zero-AVSR: Zero-Shot Audio-Visual Speech Recognition with LLMs by
  Learning Language-Agnostic Speech Representations
Authors: name: Jeong Hun Yeo, name: Minsu Kim, name: Chae Won Kim, name: Stavros Petridis, name: Yong Man Ro

We explore a novel zero-shot Audio-Visual Speech Recognition (AVSR)
framework, dubbed Zero-AVSR, which enables speech recognition in target
languages without requiring any audio-visual speech data in those languages.
Specifically, we introduce the Audio-Visual Speech Romanizer (AV-Romanizer),
which learns language-agnostic speech representations by predicting Roman text.
Then, by leveraging the strong multilingual modeling capabilities of Large
Language Models (LLMs), we propose converting the predicted Roman text into
language-specific graphemes, forming the proposed Cascaded Zero-AVSR. Taking it
a step further, we explore a unified Zero-AVSR approach by directly integrating
the audio-visual speech representations encoded by the AV-Romanizer into the
LLM. This is achieved through finetuning the adapter and the LLM using our
proposed multi-task learning scheme. To capture the wide spectrum of phonetic
and linguistic diversity, we also introduce a Multilingual Audio-Visual
Romanized Corpus (MARC) consisting of 2,916 hours of audio-visual speech data
across 82 languages, along with transcriptions in both language-specific
graphemes and Roman text. Extensive analysis and experiments confirm that the
proposed Zero-AVSR framework has the potential to expand language support
beyond the languages seen during the training of the AV-Romanizer.