MoME: Mixture of Matryoshka Experts for Audio-Visual Speech Recognition
Authors: name: Umberto Cappellazzo, name: Minsu Kim, name: Pingchuan Ma, name: Honglie Chen, name: Xubo Liu, name: Stavros Petridis, name: Maja Pantic
Organization: Imperial College London

Large language models (LLMs) have recently shown strong potential in
audio-visual speech recognition (AVSR), but their high computational demands
and sensitivity to token granularity limit their practicality in
resource-constrained settings. Token compression methods can reduce inference
cost, but they require fixing a compression rate in advance and produce a
single fixed-length output, offering no flexibility to balance information
density and efficiency at inference time. Matryoshka representation learning
(MRL) addresses this by enabling a single model to operate across multiple
token granularities, allowing compression rates to be adjusted dynamically.
However, current MRL-based methods treat each scale independently during
training, limiting cross-scale generalization, robustness at high compression,
and interpretability. To overcome these limitations, we propose MoME (Mixture
of Matryoshka Experts), a novel framework that integrates sparse
Mixture-of-Experts (MoE) into MRL-based LLMs for AVSR. MoME augments a frozen
LLM with top-k routed and shared experts, allowing dynamic capacity allocation
across scales and modalities. A shared router promotes consistent expert
activation across granularities, enabling compressed sequences to benefit from
representations learned at lower compression. Experiments on LRS2 and LRS3
demonstrate that MoME achieves state-of-the-art performance across AVSR, ASR,
and VSR tasks, while requiring significantly fewer parameters and maintaining
robustness under noise. MoME unifies the adaptability of MRL with the
efficiency of MoE, offering a scalable and interpretable solution for
resource-aware speech recognition.