Title: MoME: Mixture of Matryoshka Experts for Audio-Visual Speech Recognition
Venue: NeurIPS 2025
Accepted: yes
Keywords: Mixture of Experts

Matryoshka Representation Learning

Audio-Visual Speech Recognition
Abstract: Large language models (LLMs) have recently shown strong potential in audio-visual speech recognition (AVSR), but their high computational demands and sensitivity to token granularity limit their practicality in resource-constrained settings. Token compression methods can reduce inference cost, but they require fixing a compression rate in advance and produce a single fixed-length output, offering no flexibility to balance information density and efficiency at inference time. Matryoshka representation learning (MRL) addresses this by enabling a single model to operate across multiple token granularities, allowing compression rates to be adjusted dynamically. However, current MRL-based methods treat each scale independently during training, limiting cross-scale generalization, robustness at high compression, and interpretability. To overcome these limitations, we propose MoME (Mixture of Matryoshka Experts), a novel framework that integrates sparse Mixture-of-Experts (MoE) into MRL-based LLMs for AVSR. MoME augments a frozen LLM with top-k routed and shared experts, allowing dynamic capacity allocation across scales and modalities. A shared router promotes consistent expert activation across granularities, enabling compressed sequences to benefit from representations learned at lower compression. Experiments on LRS2 and LRS3 demonstrate that MoME achieves state-of-the-art performance across AVSR, ASR, and VSR tasks, while requiring significantly fewer parameters and maintaining robustness under noise. MoME unifies the adaptability of MRL with the efficiency of MoE, offering a scalable and interpretable solution for resource-aware speech recognition.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper is about a particular type of mixture of experts in parallel to frozen LLM layers as part of an audio-visual (AV) automatic speech regognition system (ASR). It builds upon the successful Matryoshka representation learning concept and extends it to a mixture of experts, allowing to adjust MoE experts of various token granularities to be dynamically routed. Both shared and routed experts are used. The overall goal is an efficient LLM-based AV ASR approach that performs strong under a minimal extra use of parameters.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\nThe paper is very well written and organized. Issues with state of the art are nicely analyzed and thereby the own contribution is very well motivated. The novel presented concepts are clearly understandable, math is correct, and experimental validation is convincing w.r.t. the number of active parameters. \n\nWeaknesses\n\nThere is not much to say here, only some minor points:\n\n- As the claims are efficient AV ASR solutions, I would have expected not only to see the number of active parameters reported as proof of efficiency, but also some computational complexity on some H/W platform. Rather than FLOPs/sec that could be tokens/sec or similar.\n-  Figure 2 does not use its space efficiently. Fonts could/should be enlarged.\n- l. 202: You may want to add K<N_r (instead of K only)\n- l.210: a blanc too much\n- l.222: Why do you use a load-balancing loss? The MoE used in DeepSeek V3 come with the explicitly claimed asset, that *not* training the load balance gives better performance... Idea for future research or did I misunderstand s.th.?\n- Tables 2 and 3 captions miss to state that WER (5) is shown.\n- Table 2 caption: Replace \"noise levels\" by \"conditions\", as SNR does not represent a noise level.\n- Table 3: What is \"expert size\"? Does it refer to one of the math symbols introdiced before? Use it here!\n- Limitations are missing in the main paper. Please check the guidelines, but that should have been part of it?\n\nQUESTIONS:\nOn line 222: Why do you use a load-balancing loss? The MoE used in DeepSeek V3 come with the explicitly claimed asset, that *not* training the load balance gives better performance... Idea for future research or did I misunderstand s.th.?\n\nOn Table 3: What is \"expert size\"? Does it refer to one of the math symbols introduced before? Use it here!\n\nLIMITATIONS:\nLimitations are described in the Appendix only, not in the main paper. Isn't that against the guideline?", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-06-22"}

comments: SUMMARY:
This paper applies Mixture of Matryoshka Experts to audio-visual speech recognition. It achieves strong performance on LRS2 and LRS3 on AVSR, ASR, and VSR tasks.

STRENGTHS AND WEAKNESSES:
Strengths: Detailed analysis on compression rate (and combinations) v.s. performance, expert load distribution

Weakeness:

1. The core idea—replacing LoRA (used in prior work like [22] Llama-MTSK) with a sparse Mixture-of-Experts (MoE) in a Matryoshka framework—is incremental and lacks sufficient conceptual novelty. The architecture is essentially a mechanical substitution: swapping out LoRA adapters for top-k routed MoE blocks, while keeping most of the structure (Matryoshka-style multi-scale token compression, shared LLM backbone) intact.

2. Having a few always-on shared experts is not novel, as cited papers such as DeepseekMoE has already used this idea. Applying this idea on audio-visual speech recognition has a small impact for the research community

3. The authors claim SotA performance on LRS2 and LRS3, but in table 1, they only compared their models with two other Llama-based models. As far as I know, they numbers are far from real SotA, or even older papers such as Whisper‑Flamingo (https://arxiv.org/abs/2406.10082)

4. The comparison with Llama-AVSR and Llama-MTSK doesn't seem fair. Because they are trained for each scale combination separately, while proposed model is trained on all scales, and therefore the proposed model consumes a lot more compute.

QUESTIONS:
I recommend submitting this work to speech or CV specific conferences

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-06-30

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a framework called MoME (Mixture of Matryoshka Experts) for AVSR, which combines sparse MoE with MRL to dynamically adjust token granularities and allocate computing resources. They propose routed experts controlled by a top-k router plus shared experts. This allows implicit alignment and knowledge transfer across different compression levels, showing lower WER for ASR and AVSR.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The MoME model can adjust token compression according to hardware or scenario requirements, balancing computational efficiency without retraining for each level.\n- The shared router and expert allow high-compression inputs to reuse fine-grained features learned at lower compression levels.\n\nWeakness:\n\n- The core idea is integrating MoE into MRL without introducing fundamentally new mechanisms. Given existing examples like LLAMA4, DeepSeekMoE, and Llama-MTSK, what distinguishes MoME’s contribution beyond merging sparse experts with multi-granularity tokens?\n- This work claims it is improved for AVSR. However, the results of MoME-23/4-MHSA AVSR compression (4, 2), (4, 5), and ASR compression 4 are very close (1.7, 1.8, and <2.0 %). Is MoME really better at visual understanding, or is it simply boosting ASR? Is this framework better for ASR instead of AVSR?\n- Additionally, the standalone VSR WER ranges from approximately 36 % to 42 % across compression rates, which is still far from practical deployment. Moreover, MoME’s VSR curve tracks the Llama-MTSK SS baseline within a narrow margin (around 1 percentage point) in many compression rates. Are these gains statistically significant?\n\nQUESTIONS:\n1. What impact do the number of shared experts and the choice of top-k for routed experts have on final WER, and is there an automated strategy to adjust these?\n2. If applying MoME to other multimodal tasks (e.g., image–text retrieval), what architectural or training modifications would be needed?\n3. Is it possible at inference time to adaptively switch granularities based on input content (e.g., speaking rate or noise level) rather than using preconfigured settings?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces MoME (Mixture of Matryoshka Experts), a novel framework that combines Mixture-of-Experts (MoE) with Matryoshka Representation Learning (MRL) for audio-visual speech recognition (AVSR). MoME employs top-k routed and shared experts within a bottleneck design, enabling efficient expert sharing across token granularities and modalities. MoME achieves state-of-the-art results on AVSR, ASR, and VSR tasks with fewer active parameters and remains robust under high compression and noise. Visualization analyses further reveal MoME’s cross-scale consistency in expert utilization.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n1. The paper is well-written and clearly organized.\n2. The proposed MoME framework is supported by extensive experiments across multiple tasks (AVSR, ASR, VSR) and datasets (LRS2, LRS3), and consistently outperforms or matches strong baselines across various compression settings. They also show robustness under varying levels of acoustic noise.\n3. This work includes thorough ablation studies on different insertion points of the MoME module within the LLM architecture, varying the number of routed and shared experts, providing insights into optimal integration strategies.\n4. Visualization of cross-modality correlations and expert activations provides insights into MoME’s behaviours, highlighting its ability to align representations across modalities and scales.\n\n**Weaknesses**\n1. The appendix includes a TFLOP analysis of MoME under different compression rates, but it does not compare against other baselines (e.g., LLaMA-AVSR, LLaMA-MTSK) regarding inference cost. The effect of varying the number of experts on computational efficiency is not explored.\n2. The effectiveness of the router mechanism is not fully explored, as there is no analysis on different values of k in top-k routing. Such an analysis would strengthen claims about routing efficiency and expert selection.\n3. Figure 1 uses training hours instead of FLOPs, limiting hardware-agnostic comparison.\n\nQUESTIONS:\n1. I suggest including a comparison of MoME’s inference cost against other baselines, and providing an ablation that shows how the cost changes with different numbers of experts.\n2. Could you provide results for different values of k in the top-k routing, while keeping the number of routed experts fixed? It would be interesting to see how performance changes, particularly when all routed experts are selected.\n3. Using training FLOPs instead of training hours could improve the reproducibility and hardware independence of the comparisons.\n4. Some tables reuse model names for different bottleneck settings, which may confuse readers.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-19"}