Adaptive Audio-Visual Speech Recognition via Matryoshka-Based Multimodal
  LLMs
Authors: name: Umberto Cappellazzo, name: Minsu Kim, name: Stavros Petridis

Audio-Visual Speech Recognition (AVSR) leverages both audio and visual
modalities to enhance speech recognition robustness, particularly in noisy
environments. Recent advancements in Large Language Models (LLMs) have
demonstrated their effectiveness in speech recognition, including AVSR.
However, due to the significant length of speech representations, direct
integration with LLMs imposes substantial computational costs. Prior approaches
address this by compressing speech representations before feeding them into
LLMs. However, higher compression ratios often lead to performance degradation,
necessitating a trade-off between computational efficiency and recognition
accuracy. To address this challenge, we propose Llama-MTSK, the first
Matryoshka-based Multimodal LLM for AVSR, which enables flexible adaptation of
the audio-visual token allocation based on specific computational constraints
while preserving high performance. Our approach, inspired by Matryoshka
Representation Learning, encodes audio-visual representations at multiple
granularities within a single model, eliminating the need to train separate
models for different compression levels. Moreover, to efficiently fine-tune the
LLM, we introduce three LoRA-based Matryoshka strategies using global and
scale-specific LoRA modules. Extensive evaluations on the two largest AVSR
datasets demonstrate that Llama-MTSK achieves state-of-the-art results,
matching or surpassing models trained independently at fixed compression
levels.