MoME: Mixture of Matryoshka Experts for Audio-Visual Speech Recognition
Authors: Umberto Cappellazzo
Minsu Kim
Pingchuan Ma
Honglie Chen
Xubo Liu
Stavros Petridis
Maja Pantic
Venue: NeurIPS 2025 poster ()
Keywords: Mixture of Experts
Matryoshka Representation Learning
Audio-Visual Speech Recognition
Primary area: applications

Abstract: Large language models (LLMs) have recently shown strong potential in audio-visual speech recognition (AVSR), but their high computational demands and sensitivity to token granularity limit their practicality in resource-constrained settings. Token compression methods can reduce inference cost, but they require fixing a compression rate in advance and produce a single fixed-length output, offering no flexibility to balance information density and efficiency at inference time. Matryoshka representation learning (MRL) addresses this by enabling a single model to operate across multiple token granularities, allowing compression rates to be adjusted dynamically. However, current MRL-based methods treat each scale independently during training, limiting cross-scale generalization, robustness at high compression, and interpretability. To overcome these limitations, we propose MoME (Mixture of Matryoshka Experts), a novel framework that integrates sparse Mixture-of-Experts (MoE) into MRL-based LLMs for AVSR. MoME augments a frozen LLM with top-k routed and shared experts, allowing dynamic capacity allocation across scales and modalities. A shared router promotes consistent expert activation across granularities, enabling compressed sequences to benefit from representations learned at lower compression. Experiments on LRS2 and LRS3 demonstrate that MoME achieves state-of-the-art performance across AVSR, ASR, and VSR tasks, while requiring significantly fewer parameters and maintaining robustness under noise. MoME unifies the adaptability of MRL with the efficiency of MoE, offering a scalable and interpretable solution for resource-aware speech recognition.

MoME: Mixture of Matryoshka Experts for Audio Visual Speech Recognition
Umberto Cappellazzo Imperial College London
Minsu Kim Meta Al
Pingchuan Ma Meta AI
Honglie Chen Meta Al
Xubo Liu Meta Al
Stavros Petridis
Maja Pantic
Imperial College London NatWest AI Research
Imperial College London NatWest Al Research
Abstract
Large language models (LLMs) have recently shown strong potential in audio- visual speech recognition (AVSR), but their high computational demands and sensitivity to token granularity limit their practicality in resource-come coourre-constraieed settings. Token compression methods can reduce inference cost, but they require fixing compression rate in advance and produce single fixed-length output, offering no flexibility to balance information density and efficiency inference time. Matryoshka representation learning (MRL) addresses this by enabling a single model to operate across multiple token granularities, allowing compression rates be adjusted dynamically. However, current MRL-based methods treat each scale independently during training, limiting cross-scale generalization, robustness at high compression, and interpretability. To overcome these limitations, we propose MoME (Mixture Matryoshka Experts), novel framework that integrates sparse Mixture-of Ef-erts (MoE) into MRL-based LLMs for AVSR. MoME augments a frozen LLM with top-k routed and shared experts, allowing dynamic capacity allocation across scales and modalities. A shared router promotes consistent expert activation across granularities, enabling compressed sequences to benefit from representations learned lower compression. Experiments LRS2 and LRS3 demonstrate that MoME achieves state-of-the-arr performance across AVSR, ASR, and VSR tasks, while requiring significantly fewer parameters and maintaining robustness under noise. MoME unifies the adaptability of MRL with the efficiency of MoE, offering scalable and interpretable solution for resource-aware speech recognition.
1 Introduction
To overcome the inherent limitations of Auditory Speech Recognition (ASR) 22 which typically suffers from degraded performance in acoustically challenging environments (e.g.. crowded areas subways), Audio- Visual Speech Recognition (AVSR) [4] 6 has attracted substantial attention in the speech processing community. By incorporating additional visual modality, such lip movements, that remains unaffected by acoustic noise, AVSR seeks to enhance the robustness and accuracy of speech recognition systems under noisy conditions.
Recent advancements in Multimodal Large Language Models (MLLMs) have shown that integrating modalities such vision and speech significantly broadens the scope and capability of LLMs, achieving state-of-the-ar performance across a range of tasks 7 g 10 II 12 B
[5| 16]. Following this trend, recent literature has explored the incorporation of LLMs into ASR and AVSR tasks and achieved promising performance [17
14
22 However, a critical challenge
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

with these models is their token hunger: they tend to perform better when given fine- ime-grained, dense token representations of the imput. This is particularly problematic in the audio-visual speech domain, where inputs are long in duration and have a higher temporal resolution compared to text, resulting in a much larger number of input tokens than text-only settings 23 To address the computational burden, it is common practice to reduce the number of tokens before feeding them to the LLM. Several strategies have been proposed for this purpose, including token concatenation
modules like query transformers along the hidden dimension [4 24 25 26). resampling-basee
[27 28 29 30 MI average pooling (2 62 63] CTC-based compression [34], and compression via
speech-text alignment (31. While effective, these methods suffer from key limitation:: they require to specify a fixed token compression rate in advance, thus producing a single fixed-length token sequence, which may not be optimal for all inputs deployment scenarios and prevents dynamic
control over the trade-off between information fidelity and computational efficiency.
To mitigate these challenges, Matryoshka- based L.MSs have emerged as a flexible and ef- ficient framework for elastic inference, initially
MoME- WW- (15-2)
applied to vision-language tasks [32 [6], and
Troiiing Dita Hours
Liane WTSE-5 45
more recently extended to AVSR 23 Inspired
WER (3)
by the Matryoshka Representation Learn- ing (MRL) principle (37]. these approaches train models across multiple token gramurali- ties, allowing the number of audio and visual input tokens to he dynamically adjusted at in- ference time, depending on resource availability or task-specific constraints. However, current Matryoshka models rely on uniform, monolithic representations at each scale and treat each res- olution independently during training. This lack of inter-scale interaction forces the model to compromise between generality and specializa- tion, often yielding suboptimal performance at higher rates, where less input information is pre- served. Moreover, a detailed analysis of the in- teraction between audio and visual tokens across different scales remains largely unexplored, limiting the interpretability and adaptability of these models.
USR
MoMe- 22/4 -WHBA (4-1)
2000
Wh Raminng
100
150
100
20 400 200
600
Activated Paramatirrs [M)
Figure 1: Comparison of MoME with SOTA meth- ods in terms of WER, number of activated param- eters, and training data hours on LRS3 dataset. MoME achieves performance parity with or out- performs recent AVSR models while training lesser amount of hours, activating fewer parame- ters and catering to user's wer's resource constraints with a single set of model weights.
To address these limitations, we propose Mixture of Matryoshka Experts (MoME), a novel module that integrates sparse Mixture-of-Expe (MoE) into the Matryoshka framework for AVSR. MoME introduces top-k routed experts into each layer of frozen pretrained LLM, allowing the model to dynamically allocate computational capacity across token granularities and modalities. Each MoME module includes shared router that processes both audio and visual tokens and selects specialized experts per token. Moreover, inspired by recent models like DeepSeekMoE (38] and Llama 4 [39]. we also incorporate one or two shared experts to capture global, cross-modal, and scale-invariant knowledge that is useful across all scales. A key feature of MoME is that, during inference, the router consistently activates similar expert subsets across scales. This implicit alignment enables sequences at higher compression rates to leverage expert pathways shaped by richer, lower- compression sequences, promoting knowledge transfer and improving representation quality at all scales. When combined with the shared experts, this design enhances both specialization and generalization, leading to stronger performance across wide range of inference-tin -ieeeneeeemeeeeeee budgets. To the best of our knowledge, this is the first work to propose a unified framework that encompasses both MoE and MRL paradigms.
MoME consistently outperforms prior Matryoshka- based methods as well as independently trained models across AVSR, ASR, and Visual Speech Recognition (VSR) tasks on LRS2 [10] and LRS3 [1] benchmarks. Thanks to its sparsely activated architecture, MoME requires significantly fewer parameters during inference than competing baselines. In addition, MoME offers architectural flexibility, as it can be seamlessly inserted at multiple points within the LLM, as well as strong robustness in noisy scenarios, outperforming alternative methods. Finally, we conduct extensive ablations on the key components of MOME, such as the number of routed and shared experts. We
2

---

further provide visualization analyses that shed light on how audio and visual tokens interact across scales, offering new insights into cross- modal and cross-scale alignment. Altogether, MoME unifies the scale-aware adaptability of MRI. with the efficiency and modularity of MoE, resulting in single model capable of resource-aware inference with strong interpretability.
2
Related Work
2.1 Mixture of Experts
Mixture-of-Expere Experts (MoE) models have recently demonstrated significant advancements
42
38
44 45 46 47 The key idea is to replace standard Ferd-Forward Network (FFN) layer with an MoE layer, which comprises multiple FFN experts and a learnable (usually sparse) routing mechanism. By selectively activating only a subset of experts for each input, MoE architectures enhance performance while preserving computational efficiency. Recent works have focused on making MoE models more efficient and stable via co-upelying 48
50] sharing parameters across layers
[1 52) and
adjusting the number of experts and those to activate dynamically (53. While most of the works focus on MoEs as a pretraining strategy, they can also be used in parameter-efficient finetuning by keeping the Transformer backbone frozen and only update a small number of parameters [13 54, (55 56].
Finally, in the audio-visual domain, recent works have demonstrated that MoEs can efficiently scale
model capacity and optimally process audio and visual data [57 58
2.2
Matryoshka Representation Learning
The challenge of developing flexible representations adaptable a spectrum of downstream tasks with varying computational demands has been addressed by Matryoshka Representation Learning (MRL) This method encodes information at different granularities within unified, high- digendimnsiooall feature vector produced by neural network. Recently, the MRL paradigm has been used in vision-language and audio-visual L.MSs to learn a hierarchy of representation granularities at enabling adaptive deployment per computational constraints.
B3 60 61
63
the token sequence level
36 32 23
2.3 Audio-Vismn Speech Recognition
Early deep learning approaches to AVSR focused on designing modality-specific encoders and
fusion strategies for integrating audio and visual inputs [65] 66 67 68). The introduction of Trans-
formers [69] led to substantial performance gains [70, 71 72 which spurred further research into
multimodal modeling, including self-supervised learning 73 74 75 76). knowledge distillation
from ASR to AVSR (8). and leveraging cross-modal complementarity
779 80
81).
have explored integrating audio and audio-visual speech recognition capabilities into pre-trained LLMs. In LLLM- AVSR, a key challenge is the computational cost of processing long speech and visual sequences. Llama-AVSR [14] addressed this by compressing audio-visual features at fixed rates, demonstrating reduced compute demands with limited performance degradation. More recently, Llama-MTSK 2 uses MRL 36
With the rise of LLMs, recent works 20
28
19
15
22
16 21
to enable elastic inference, allowing a single model to process multiple token granularities at test time. Concurrently, MMS-
29 proposed an adaptive compression strategy based on speaking rate using a Q-Former [82]. However, both Llama-AV AVSR and MMS- require retraining for each specific token rate to match the desired GPU memory usages, limiting deployment flexibility. This hinders the practicality of using LLM--bsed AVSR models, which usually require a GPU with a large memory capacity. Inspired by the wauuullrtty-ddaatii processing capabilities of Llama- MTSK, we propose MoME, which combines MRL [37] with sparse MoE layers inserted into a frozen LLM. Unlike prior methods, MoME facilitates knowledge transfer by reusing expert activations across token granularities and incorporating shared experts that encode scale-invariant information. This design significantly mitigates performance drops at high compression rates, offering single, resource-adaptive model that maintains strong performance across wide range of compute budgets.
3

---

Legend
They put opero knng
Text Tokers
Audio Tokens
Shared Expert
Bouttel Experts
Visual Tokend
MHSSA
Audio-Visual Tokens
FPN
CMMME
(ote) (16,5)
MHSA
(32 tokens) Audio-Visual
Large Langange Model
K.p-E Router
(4,V) rates
(100 tokens)
Natryshke
Trozer/Trcinable
Projector
Projector
and saaoch to fest.
Audio
Video
Encoder
Encoder
Figure 2: Overview of our proposed MoME module. We start by producing audio-visual tokens at different scales via modality pre-trained encoders and projectors. Each Matryoshka sequence goes through MoME, which can be placed parallel to multiple modules within each L.M layer (parallel to the MHSA module in the Figure). Each MoME module comprises top-k router, which sparsely activates subset of routed experts, and pool of shared experts to capture scale-invariant knowledge.
Methodology
3.1
Audio-Visien Matryoshka Token Sequences
Given an audio waveform and its corresponding lip movements video v, we process them using pre-trained audio encoder (e.g., Whisper (83) and pre-trained video encoder (e.g., AV-HuBERT 73] to yield audio and visual token sequences, Z and Z", respectively.
When processing audio-visual tokens with an LLM, reducing token granularity is essential for lowering computational cost and improving inference efficiency. This is particularly important in AVSR, where both modalities exhibit temporal continuity, resulting in redundant information. However, most token compression methods require fixed compression limiting the ability to dynamically balance recognition performance and resource constraints. While finer-grained tokens improve recognition accuracy, they substantially increase inference cost due to the quadratic complexity of Transformers, reducing their practicality in constrained settings.
be set in advance,
To address this, adopt the MRL framework flexibly controlled at inference time based on specific requirements. We generate token sequences varying granularities by applying different compression rates the audio and visual streams, enabling the model to learn to operate across a spectrum of coarse-to-fine resolutions within a single architecture. This approach has proven successful in both assion-language and audio-visual MLLMs 36 23 32).
32
which allows audio-visual granularity to be
Concretely, following 23), in training apply audio compression rates {a1, a2 0g to obtain audio token sequences at multiple scales (Z "e..... ................................ Z and L video compression rates (U1,
to obtain IZ 2 Z Z. Each sequence passed through modality-specific projection layers to match the LLM's embedding dimension. We then generate G audio-visual token sequences by concatenating all audio-video rate combinations along the temporal axis. Finally, textual tokens (i.e., instructions and ground truth) are appended to each sequence. We denote a sequence with audio rate a; and video rate Uy Z
3.2 Mixture of Matryoshka Experts (MoME) Module
Even though MRL enables elastic inference by supporting multiple token granularities, some per- formance degradation at lower token scales is inevitable due to the information loss introduced by compression (23) To mitigate this, we leverage the fact that all coarse-to-fine token sequences are accessible during MRL training. Our proposed solution, Mixture of Matryoshka Experts (MoME),

---

introduces a set of routed and shared experts trained jointly across token granularities, Each expert is responsible for modeling specific phonetic patterns and is exposed to both high- and low-resolution representations of the same utterance during training. This design allows experts to learn fine-grained (high-resolution) features that can later be reused when processing compressed (low-resolution) tokens at inference time. By aligning expert training across scales, MoME promotes cross-granularity knowledge transfer and improves the robustness of the model under aggressive compression.
The MoME module adapts the standard Mixture-of-Experts (MoE) layer to the Matryoshka framework through the following key design modifications. (1) In typical MoE-based LLMs, each FFN layer is replaced with an MoE layer and trained over large-scale data [84 44
85
86
43
42
46 47).
In contrast, we insert MoME modules in parallel with the existing LLM layers, allowing efficient
fine-tuning of a frozen pretrained backbone. To minimize the number of active parameters during inference, we design the experts with a small bottleneck dimension, pushing down to 1 in the extreme case. This aligns with the shallow brain hypothesis [7], which posits that cognition involves not only deep sequential processing (analogous to LLMMs) but also many parallel shallow modules (as realized by MoME experts). Furthermore, reducing the bottleneck dimension allows us to activate
more experts per token, akin to the fine-grained expert segmentation proposed in [38]. (2) We further include one or more shared experts, which are always active regardless of input, to capture global and cross- scale knowledge. This guarantees that even highly compressed inputs benefit from stable, general representations, following insights from prior work [38. (3) Crucially, both the experts and router in each MoME module are shared across all Matryoshka sequences. This design encourages the router to activate similar subsets of routed experts across different granularities, creating implicit alignment. As a result, lower-resolution sequences can leverage the representational capacity learned from higher-resolution inputs, promoting robustness and knowledge transfer across scales.
Each MoME module consists of N, routed experts, which are sparsely activated based on the router's scores, and N₂ shared experts, which process deterministically each input token. Each expert follows a bottleneck structure similar to adapter and LoRA experts [88 54
89 90, 13 [I], where the first
linear layer serves as downsampler, followed by a non-linear activation (e.g.. GELU), and uprojected back to the LLM hidden size. Then, the output of the MoME module can be expressed as:
N.
N.+N, Σ
(HME(H)))
Σ
E
H
B
(1)
E Top-k(fsp No + +1 +
8n 0,
(2)
otherwise,
S Softmax, (H, WT).
(3)
where H is the input to the MoME block corresponding to the ij-th Matryoshka sequence and I-th LLM layer, and it can be the output of the Multi-Head Self-Attention (MHSA) module if it is placed parallel to the FFN module, otherwise it is the input of the LLM layer after the layer normalization, (-) is the n-th expert, gn represents the gate value for n-th expert, S,, denotes the token-to-ere to-eerrrt affinity scores, Top-k(-, K) denotes the set comprising the K highest affinity scores among those calculated for the all N, routed experts, and W represents the router's weights (in our case linear layer) corresponding to the n-th expert in the I-th layer. Note that gn is sparse, indicating that only K out of N, gate values non-zero, with K < N,. This sparsity property ensures computational efficiency within a MoME module, i.e., each token will be assigned to and computed via only K routed experts.
The proposed MoME module can be placed in parallel to three locations within a Transformer layer of the LLM: 1) the MHSA module, 2) the FFN, and 3) the whole Transformer layer (see section[Binn in the Appendix for the visualization of the three variants). In the following, we consider inserting MoME in parallel with the FFN module, and the extension to the other two cases can be straightforwardly achieved. Thus, with the MoME insertion, the I-th Transformer layer of the LLM for the sequence Z is:
H = MZSA + ZE_1 z (H) + MoME(H +H
4
where H. is the hidden state after the 1-th attention module, , and Z is the output hidden state after the 1-th Transformer layer. For brevity, we omit the layer normalization in the above formulations.
5

---

Our model is trained by averaging the auto-regressive next token prediction loss for each audio-visual scale ij for each imput data. The LLM predicts the response Y {y+} conditioned on the multimodal input tokens, where S represents the number of tokens of the ground truth transcription to be generated. Accordingly, for each Matryoshka audio-visual representation Z the probability of the target Y is computed by p ( = IT== pe (v. Z yea), where y< is the generated output sequence up to token over all the audio-visual token scales: LLM
and 0 is the trainable parameters. The final objective is the average 61 Ei= و== p( ( Cij- where Cij is parameter that controls the importance of each audio-visus sequence, and is set to 1 following previous works [32 36 23]. We experiment with different weights in Section Cf the Appendix. In addition to this, to avoid the risk of router collapse, where the majority of tokens are assigned to only a few experts, we employ a load balancing loss La (45) for each audio-visual sequence Z To compute it, we multiply the fraction of tokens Ln routed one expert E,, with the total routing probability P,, allocated to En for one batch and sum it across the number of routed experts N, as
follows: LB N,- Σ N,,,
N fn Pn. This loss is scaled by a factor 0.01 following 45 42
4 Experiments
4.1 Experiment Settings
Datasets. We train and evaluate MoME on LRS2 [40] and LRS3 [4] datasets. LRS2 includes 225 hours of video clips. LRS3 contains 433 hours of transcribed English video clips.
Tasks. We study the AVSR task for our main experiments on LRS2 and LRS3, and we also report the results for the ASR and VSR tasks on LRS3.
Model Details. We use AV-HuBERT Large 773 the visual encoder and Whisper (small and medium versions) 83 as the audio encoder. The projectors consist of two linear layers with ReLU activation in between. For the LLM backbone, we adopt three variants from the Llama 3 family [92] Llama BB2 Llama 3.2-3B, and Llama 3.1- Each MoME module comprises a router, implemented as a linear layer, which activates sparse subset of routed experts per input token using top-k selection, along with one or two shared experts that are always active. We also investigate different insertion points for MoME within the L.M architecture, resulting in three variants: 1) parallel insertion to the FFN module, 2) parallel insertion to the MHSA module, and 3) parallel insertion to the entire LLM layer. For example, MoME-23/4-MHSA. denotes a configuration where MoME is inserted parallel to the MHSA module, using 23 routed experts with 4 active per token (i.e., top-4). For the main experiments, we primarily use a single routed expert. Further training and preprocessing details are provided in the Appendix, Section
Audio Granularities. For fair comparison with previous methods, apply the same compression rates as in (23]. The compression rates are chosen in a coarse-to-fine manner, in order encompass range of tokens sequences that trade-off between efficiency and performance at inference. For ASR, the compression rates are (4, 8, 12, 16, 20). For VSR, we apply (1, 2, 3, 4, 5). For AVSR, we apply audio rates in (4, 16) and video rates in (2, 5), leading to audio-visual configurations. For the main experiments, we compress the tokens via average pooling.
Baselines. As shown in Table 2. compare our proposed MoME with multiple methods. Fixed-rate methods: 1) Llama-AVSR 14 is the first AVSR LLM-based model, achieving sota results on LRS3. This approach is trained on each audio-vian visual scale separately, leading to many models as the number of scales. 2) Similar to this, we also include the baseline MoME-23/44 23/4-MHSA--L which is trained independently
each fixed scale. Both approaches do not provide elastic inference within single model. Matryoshka methods: we include a recent work, Llama-MTSK 223, which is the adaptation of Llama-AVSR to the Matryoshka setting. Llama-MTSK supports three configurations: 1) MS uses a single Multi-Scale LoRA module for all scales, 2) SS uses a single Scale-Specific LoRA module for each scale, and 3) MSS combines SS and MS together. More baselines are used in the bubble plot [I (see Section 4.2
4.2 Main Results
AVSR Results (1). Table summarizze the AVSR performance of our proposed MoME variants FFN, MHSA, and LAYER. For fair comparison with prior work, use Whisper-small as the audio

---

Table 1: Main results for AVSR on LRS2 and LRS3 in terms of WER (%) across different (audio-
visual) compression rates (e.g., (4,2)). 8 Llama-AVSR [14] and MoME-233 23/4-MHSA-1 involves training separate models for each pair of audio-visual rates. The active parameter count does not
include the projector parameters since they are the same for all the reported methods (around 8. M).
Active
LRS2 Dataset
Active
LRS3 Dataset
Method
Params
(4
2)
(4,5)
(16,2)
(16,5)
Params
(4
(4,5)
(16,2) (16,5)
Llama-AVSR [4]
27.5M
4.1
4.5
5.3
8.1
6.8M
2.4
2.8
3.3
4.1
LJama-MTSK MS
23
27.5M
4.8
5.9
6.4
8.9
8.iM
2.6
2.7
3.7
4.1
Llama-MTSK S5
22
27.5M
3.4
4.7
4.8
6.4
8.1M
2.3
2.2
3.3
3.6
Llama-MTSK MSS
[23
55.0M
3.6
4.8
6.1
9.0
13.6M
2.4
2.4
3.2
3.5
NOME-23/4-MIISA.
12.7M
3.2
3.1
4.9
5.3
3.5M
2.1
1.9
3.3
3.7
MoME-23/4-FFR
12.7M
3.2
3.1
4.5
4.6
3.5M
2.1
2.2
4.0
4.0
12.7M
2.9
3.0
4.2
4.3
3.5M
1.8
1.7
2.9
2.9
MoME-23/4-LAYER
12.7M
2.7
2.7
4.2
4.2
3.5M
1.5
1.8
3.1
3.2
MoME-23/4
2.3M
3.0
3.2
4.3
4.7
0.9M
2.0
2.2
3.2
3.7
ASR. Stocking
V.R. Aug Pooling
VSR, Storking
MTNE
UNSA
5:-
A033
Usmo AV33
50.0
Liato ATTT
Mole LAVER
2214
MiNE 73M LAVEk
Neatt 22
WER
42.3
1--
150-
57.5
10
30
20
Compr ession Rote
Compreeden Rote
Comprestion Rate
Compression Rote
Figure 3: MoME-23/4 results for VSR and ASR tasks on the LRS3 dataset.
encoder, and Llama 3.2-IB and Llama 3.2-3B as the LLMs for LRS3 and LRS2, respectively. (1) In the fixed-rate setting, MHSS- outperforms Alama-AVSR by a significant margin across all compression rates, highlighting the superiority of MoME over LoRA as a parameter- efficient adaptation method. (2) All three MoME variants consistently outperform Llama-MTSK, with notably lower WER degradation at harsher compression rates. Among the three configurations, inserting MoME parallel to the entire LLM layer or to the MHSA module yields the best performance, particularly the former higher granularities (e.g., token rates (4,2) and (4,5)). Furthermore, MoME activates consistently fewer parameters thanks to the sparse activation of the experts. (3) MoME also outperforms models trained independently at each vndio-vissall scale, demonstrating not only its support for elastic inference within a single model but also its strong competitive accuracy. (4) To further minimize inference cost, we reduce the expert bottleneck dimension to 1, resulting in only 2.3M and 0.9M active MoME parameters for LRS2 and LRS3, respectively. Even under this extremely parameter-efficient setting, MoME-23/4-LAYER exhibits minimal performance degradation, confirming MoME's s robustness and scalability.
AVSR Results (2). Figure I presents a com- parison between MoME and recent state-of- the-art methods on the LRS3 benchmark us- ing a bubble chart. Baselines include Uni 193, USR (94), Whisper-Flandne Flamingo 777, Llama-
Table 2: AVSR WER results on LRS2 under differ- ent acoustic noise levels.
SNR (dB)
Method
7.5
5
2.5
0
-5
AVSR 14), Llama-MTSK [23] Auto-AVSR
Llama-AVSR
5,6
7.1
10.6
11.8
41.8
778. AV-HuBERT (73), and MMS-Llama (29). Following [14 23), we use Llama 3.B as
Llama-MTSK MS
23
6,2
8.0
13.0
12.4
44.9
MoME-2V4LLYER
4.8
6.4
9.6
9.6
32.6
LLM and Whisper-medium
The results show
that MoME (evaluated at A-V token rates of (4.5) and (16,2)) achieves competitive WER results while activating significantly fewer parameters and requiring fewer training data hours, all under one suite of weights. More extensive results can be found in Section of the Appendix.
ASR and VSR Results. To validate the effectiveness of MoME unimodal settings, we evaluate its performance on the ASR and VSR tasks. Following (2 we apply average pooling and "stacking" compression, where the latter compresses input by concatenating consecutive tokens along the hidden dimension. As shown in Figure 3 our MoME methods remain effective also when dealing with
7

---

Table 3: Ablation on the optimal number of shared and routed experts of MoME-MHSA on LRS2.
Routed Shared
Expert
Compression Rates (A,V)
Top-k
Experts Expert
Size
(4,2)
(4,5)
(16, 2)
(16,5)
0
48
/
3.4
3.4
4.9
5.1
4
0
24
2
3.3
3.3
4.8
5.0
4
I
24
2
3.2
3.2
4.4
4.7
7
I
24
4.5
15
1
24
3
2.9
3.0
4.3
4.5
23
4.2
4.3
23
4.1
23
3
12
4
2.9
3.0
4.2
4.6
1.0
025
220
Audio r6
Audio rote
Aude, rote
Figure 4: Intra-modality and gross-modality correlation matrices for MoME-15/3 trained on LRS2. We study the sentence: "it's 's a long way from home".
tokens coming from single modality. In particular, observe consistent gains across all scales for the FFN and LAYER variants for the VSR task.
4.3 Ablation Studies
Noisy Setting. To evaluate the robustness of MoME in noisy conditions, we perform inference with varying levels of babble noise from the NOISEX dataset [95]. shown in Table our MoME-23/4. LAYER configuration exhibits greater resilience to noise compared to prior methods, with particularly stronger gains highly degraded scenarios.
Optimal Number of Shared/Routed Experts. In Table we analyze the impact of varying the number of routed and shared experts in the MoME-MHSA configuration. Following the approach in 38), increasing the number of top-k activated routed experts requires proportionally reducing the bottleneck size of each expert maintain the same number of active parameters during inference. We derive the following insights: (1) The use of shared expert is beneficial. When comparing the second and third rows in the table (with and without shared experts, respectively), we observe a consistent reduction in WER across all compression rates when shared expert is included. This supports the hypothesis that shared experts help capture scale-invarian knowledge. For the configuration with 23 routed experts, adding second shared expert yields slight improvements at some rates but results in degradation the (16,5) token rate, along with an increase active parameters. Therefore, we use single shared expert in all main experiments. (2) Regarding routed experts, performance generally improves as more routed experts are activated, suggesting that increased expert diversity enhances the model's representational capacity, even when the total parameter budget remains fixed.
Optimal Number of Activated Experts (k). To analyze the impact of top-k, we conduct ablation configuration with single shared expert on the LRS2 AVSR task. The number of activated experts (i.e., k) varies while keeping the number of routed experts fixed at 8. Table 4 shows that increasing k yields moderate gains, especially at higher compression rates. However, these improvements come at the expense of increased computation, which contradicts our design objective of Sparse MoF. Therefore, we conclude that using small k, typically 2, offers a practical trade-off between efficiency performance. This choice preserves sparsity while retaining most of the accuracy benefits of broader expert activation.
experiments using the

---

M
Exper Aahrarinn
Separt
MoME Leper Teane
MoME Loper Incex
Exert Load Distribttion AMER, (16.2)
Export Lead Distriutton, aesse,ne (16.5)
Epert Activote aativoree (h)
Syart
Doant
Dyerts
Baart
MAME Lepar Tains
Wo LayerIncee Incex
Figure 5: MoME-15/3-MHSA expert activation analysis across multiple scales and layers on LRS2.
4.4 Visualization Analysis
We analyze how MoME processes audio-visual speech tokens at different granularities through two complementary visualizations: expert activation patterns scales and layers, and (2) similarity matrices between speech representations at varying granularities. Additional results can be found in the Appendix (see Section D).
Cross-Modal Matryoshka Token Analysis. We visualize the similarity matrix between token sequences at different granularities. Specifically, compare video tokens with compression rates of 2 and 5, as well as audio tokens at compression rates of 4 and 16 before being fed to the LLM. We present two types of similarity matrices: 1) intra-modality similarity matrices, where compare tokens of the same modality at different compression rates (e.g., the 21 audio tokens from the sequence with compression ratio of are compared the 5 audio tokens with a compression ratio of 16), and 2) cross-modality similarity matrices, where we compare tokens from one modality with those from the other. Figure reveals strong linear correlation between speech features across different granularities, as indicated by high cosine similarity values along the diagonal. On average, each token in sequences with higher compression ratio correlates mainly with approximately two three of the tokens coming from less compressed sequences. These findings suggest that when more tokens are available, they tend encode similar redundant information, which is later condensed into single token in sequences with higher compression rates. Moreover, they indicate that audio and video tokens different granularities tend to cluster together rather than being learned independently. behavior encouraged and reinforced by MoME, which promotes cross-scale alignment through shared expert routing and consistent expert activation across resolutions.
Expert Activation Analysis. Figure 5 illustrates the layer-wise expert activation distribution across different scales for the MoME-15/3-MHSS configuration on the LRS2 dataset. Across all settings, we observe a consistent pattern: the same subset of experts tends to be activated across different token granularities, demonstrating strong alignment routing behavior. This confirms the intended effect of using shared router and shared experts, which promotes scoss-scale consistency in expert utilization. At the same time, the set of activated experts varies significantly from layer to layer, ensuring that all experts are utilized across the network. This layer-wise diversity in routing strategies allows MoME to avoid redundancy and encourages specialization among experts, while still maintaining alignment across input scales. Together, these results validate MoME's ability to balance scale- invariant knowledge sharing with depth-wise winth-wise-ee expert diversity, which is crucial for achieving robust and efficient performance across wide range of compression levels.

---

Table 4: Ablation on the optimal top-k value.
Table 5: Inference time and token per second
(TPS) across different compression ratios.
Top-k
(4,2)
(4,5)
(16,2)
(16,5)
3.3
3.3
4.6
4.7
Ratio
Tokens
Inf. Time (s)
TPS
2
3.2
3.3
4.5
4.7
(1,1)
1673
12.75
7.76
3,2
3.2
4.5
4.6
3,2
3.2
4.5
4.5
(4,2)
560
8.04
12.90
3.1
3.2
4.4
4.5
(16,5)
184
6.74
14.17
4.5
Computation Cost Analysis
In Figure 6 we illustrate the benefits of MoME in terms of TFLOPs and inference costs. Compared to the uncompressed case (i.e., 1673 tokens), MoME enables elastic inference by allowing users to select compression rates based on their computational constraints. By increasing the audio-visudl compression rates, we reduce the number of tokens processed by the LLM, and thus the TFLOPs, by 8x when applying audio-visual compression rates of (16,5) (184 tokens). Despite this substantial reduction in TFLOPs, the resulting increase in WER remains modest.
In addition to this, we include the actual in- ference time and generated tokens per second (TPS) measured on NVIDIA 140 46GB GPU with MoME-15/3-MHSS for 23-second speech input, the results demon- strate a significant reduction in inference time compression ratios increase. For instance, compression ratio of (16,5) reduced the infer- ence time to 6.74 seconds for transcribing 23 seconds of speech, compared to 12.75 seconds in the no-compression case. This confirms that higher compression rates lead to faster inference, lower GPU memory usage, and reduced com- putational load, while still maintaining strong performance thanks MoME's expert routing mechanism.
26.5
shown Table
(6-2)
(3)
16
(13)
(4.2)
3
1
10
TFLOPs
Figure 6: Comparison of MoMME-23/4-HHH in terms of number of audio-visual processed tokens, achieved WER, and TFLOPS.
5
Conclusion
introduce MoME, a framework for audio-visual speech recognition that combines MRL with MoE to enable resource-aiee resource-adaptive inference across token granularities. By using shared router and shared experts, MoME promotes expert alignment across scales, allowing compressed sequences to benefit from richer representations and maintaining strong performance levels. MoME outperforms prior Matryoshka based and fixed-scale models on AVSR, ASR, and VSR tasks across LRS2 and LRS3, while using significantly fewer parameters and training data hours. It also supports extremely parameter-efficien fine-tuning and demonstrates robustness in noisy conditions. Ablation and visualization analyses further validate MoME's ability capture cross-scale and cross-modal interactions, making a scalable and efficient solution for LLM-based AVSR. Although developed for audio-visual speech processing, MoME expected be broadly applicable to other multimodal domains, such as vision-language tasks, with minimal adaptation.
at high compression
Acknowledgments
All data collection, processing, and experiments were conducted by Imperial College London.
References
[1] Dario Amodei, Sundaram Ananthanarayanan, Rishita Anubhai, Jingliang Bai, Eric Battenberg, Carl Case, Jared Casper, Bryan Catanzaro, Qiang Cheng, Guoliang Chen, et al. Deep speech 2: End-to-end speech
10

---

recognition in english and mandarin. In Internatioonal conference on machine learning, pages 173-182. PMLR, 2016.
[2] Shinji Watanabe, Takaaki Hori. Suyoun Kim, John R Hershey, and Tomoki Hayashi. Hybrid ctc/attention architecture for end-to-end speech recognition. IEEE Journal of Selected Topics in Signal Processing. 188):1240-1255 2017.
[3] Rohit Prabhavalkar, Takaaki Hori, Tara N Sainath, Ralf Schlüter, and Shinji Watanabe. End-to-end speech recognition: A survey. IEEF/ACM Transactions on Audio, Speech, and Language Processing, 32:325-351, 2023.
[4] Stéphane Dupont and Juergen Luettin, Audio-visual speech modeling for continuous speech recognition. IEEE transactions on multimedia, 2(3):141-151, 2000.
[5] Gerasimos Potamianos, Chalapathy Neti, Juergen Luettin, and Iain Matthews. Audio-visual automatic speech recognition: An overview. Issues in visual and audio-visual speech processing. 22:23, 2004.
[6] Stavros Petridis, Themas Stafylakis, Pingehuan Ma, Feipeng Cai, Georgios Tzimiropoulos, and Maja Pantic. End-to-end audiovisual speech recognition. In 2018 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 6552. IEEE, 2018
[7] Bin Lin. Zhenyu Tang, Yang Ye, Jiaxi Cui, Bin Zhu, Peng Jin, Jinfa Huang, Junwu Zhang, Yatian Pang. Munan Ning. et al. Moe-llava: Mixture of experts for large vision-language models. arXiv preprint 19447,, 2024.
[8] Shuai Bai. Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge, Sibo Song, Kai Dang. Peng Wang. Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report, arXiv preprint arXiv:2500. 13923, 2025.
[9] Min Shi, Fuxiao Liu, Shihao Wang. Shijia Liao, Subhashree Radhakrishnan, Yilin Zhao, De-An Huang, Hongxu Yin, Karan Sapra, Yaser Yacoob, et al. Fagle: Exploring the design space for multimodal llms with mixture of encoders. arXiv preprint arXiv:2408. 155988, 2024.
[10] Peter Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Adithya Jairam Vedagiri IYER, Sai Charitha Akula, Shusheng Yang. Jihan Yang, Manoj Middepogu, Ziteng Wang, et al. Cambrian-1: A fully open, vision-centric exploration of multimodal Ilms. Advances in Neural Information Processing Systems, 37:87310-885 1187310-87356,. 2024.
[11] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiw:2304.10592 2023.
[12] Dong Zhang, Shimin Li, Xin Zhang, Jun Zhan, Pengyu Wang. Yaqian Zhou, and Xipeng Qiu. Speechgpt: Empowering large language models with intrinsic cross-moda modal conversational abilitics. arXiv preprint 2rXiv:2305.11000 11000 2023.
[13] Jialin Wa, Xia Hu, Yaqing Wang, Bo Pang, and Radu Soricut. Omni-smola: Boosting generalist multimodal models with soft mixture of low-rank experts. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 14205-14215, 2024.
[14] Umberto Cappellazzo, Minsu Kim, Honglie Chen, Pingchuan Ma, Stavros Petridis, Daniele Falavigna, Alessio Brutti, and Maja Pantic. Large language models are strong audio-visual speech recognition learners. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2025.
[15] Yuchen Hu, Chen Chen, Chao-han Huck Yang, Ruizhe Li. Chao Zhang, Pin- Yu Chen, and Ensiong Chng. Large language models are efficient learners of noise-robust speech recognition. In International Conference on Learning Representations, 2024.
[16] Jeong Hun Yeo, Minsu Kim, Chae Won Kim, Stavros Petridis, and Yong Man Ro. Zero-avsr: Zero-shot audio-visual speech recognition with llms by learning language-agnostic speech representations. arXiv preprint Xiv: 06273,, 2025.
[17] Junteng Jia, Gil Keren, Wei Zhou, Egor Lakomkin, Xiaohui Zhang, Chunyang Wu, Frank Seide, Jay Mahadeokar, and Ozlem Kalinli. Efficient streaming Ilm for speech recognition. ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5. IEBE, 2025.
[18] Amit Kumar Singh Yadav, Gil Keren, Desh Raj. Wei Zhou, Junteng Jia, Ke Li, Ying Xu, Chunyang Wu, Jay Mahadeokar, and Ozlem Kalinli. -peech-n-llamaa Improving speech 1lms with multi- training. In ICASSP 2025-2023 2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pages 1-5. IEEE, 2025.
11

---

[19] CHEN CHEN, Ruizhe Li, Yuchen Hu. Sabato Marco Siniscalchi, Pin-Yu Chen, EngSiong Chng, and Chao-Han Huck Yang. It's never too late: Fusing acoustic information into large language models for automatic speech recognition. In International Conference on Learning Representations. 2024.
[20] Yunfei Chu, Jin Xu, Xiaohuan Zhou, Qian Yang, Shiliang Zhang, Zhijie Yan, Chang Zhou, and Jingren
Zhou. Qwen-audio: Advancing universal audio understanding via unified large-scale audio- ladoo-language models. arXiv preprint arXiv:2311.07919 2023.
[21] Umberto Cappellazzo, Minsu Kim, Stavros Petridis, Daniele Falavigna, and Alessio Brutti. Sealing and enhancing Ilm-based avsr: A sparse mixture of projectors approach. In Interspeech, 2025.
[22] Jeonghun Yeo, Seunghee Han, Minsu Kim, and Yong Man Ro. Where visual speech meets language: Vsp-Ilm framework for efficient and context-aware visual speech processing. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 11391-11406, 2024.
[23] Umberto Cappellazzo, Minsu Kim, and Stavros Petridis. Adaptive audio-visual speech recognition via matryoshka-based multimodal llms, arXiv preprint arXiv:2503.06362, 2025.
[24] Qingkai Fang, Shoutao Guo, Yan Zhou, Zhengrui Ma, Shaolei Zhang, and Yang Feng. Llama-omni: Seamless speech interaction with large language models. In arXiv preprint arXiv:2409. 066666 2025.
[25] Ziyang Ma, Guanrou Yang. Yifan Yang, Zhifu Gao, Jiaming Wang. Zhihao Du, Fan Yu, Qian Chen, Siqi Zheng, Shiliang Zhang, et al. embarrasiingl simple approach for Ilm with strong asr capacity, arXiv preprint XiX::2442.008444 2024.
[26] Yassir Fathullah, Chunyang Wu, Egor Lakomkin, Junteng Jia, Yuan Shangguan, Ke Li, Jinxi Guo, Wenhan Xiong, Jay Mahadeokar, Ozlem Kalinli, et al. Prompting large language models with speech recognition abilities. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pages 13351 13355. IEEE, 2024.
[27] Wenyi Yu, Changli Tang, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun Ma, and Chao Zhang. Connecting speech encoder and large language model for asr. In ICASSP 2024-2024 IEFE International Conference on Acousticy, Speech and Signal Processing (ICASSP), pages 12637-12641. IEEE, 2024.
[28] Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun Ma, and Chao Zhang. Salmonn: Towards generic hearing abilities for large language models. arXiv preprint arXiv:2310.. 13389, 2023.
[29] Jeong Hun Yco, Hycongscop Rha, Se Jin Park, and Yong Man Ro. Mms-llama: Efficient Ilm-bases based audio-visual speech recognition with minimal multimodal speech tokens. arXiv preprint 2rX1v:5503.11335 2025.
[30] Junbum Cha, Wooyoung Kang, Jonghwan Mun, and Byungseok Roh. Honeybee: Locality projector for multimodal 11m. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 13817-13827, 2024.
[31] Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al. mplug-owl: Modularization empowers large language models with multimodality. arXiv preprint 2rXiv22304.14178 2023.
[32] Mu Cai, Jianwei Yang, Jianfeng Gao, and Yong Jac Lec. Matryoshka multimodal models. In ICLR, 2025.
[33] Jindong Jiang, Xiuyu Li, Zhijian Liu, Muyang Li, Guo Chen, Zhiqi Li, De-An Huang, Guilin Liu, Zhiding Yu. Kurt Keutzer, et al. Token-efficieet long video understanding for multimodal 1lms. arXiv preprint arXiv:2503.0413 040300, 2025.
[34] Jian Wu, Yashesh Gaur, Zhuo Chen, Long Zhou, Yimeng Zhu, Tianrui Wang, Jinyu Li, Shujie Liu, Bo Ren, Linquan Liu, et al. On decoder-only architecture for speech-to-text and large language model integration. In 2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 1-8. IEEE, 2023.
[35] Weiting Tan, Hirofumi Inaguma, Ning Dong, Paden Tomascllo, and Xutai Ma. Ssr: Alignment-aware modality connector for speech language models. arXiv preprint arXiv:2410.000 2024.
[36] Wenbo Hu, Zi-Yi Dou, Liunian Li, Amita Kamath, Nanyun Peng. and Kai-Wei Chang. Matryoshka query transformer for large vision- lisiun-aanguage models. Advances in Neural Information Processing Sysiems, 1830066-50188 2024.
12

---

[37] Aditya Kusupati, Gantavya Bhatt, Aniket Rege. Matthew Wallingford, Aditya Sinha, Vivek Ramanujan. William Soward-Snydert Kaifeng Chen, Sham Kakade. Prateek Jain, al. Matryoshka representation learning. Advances in Neural Information Processing Systems, 35:30233-3029,, 2022.
[38] Damai Dai, Chengqi Deng. Chenggang Zhao, RX Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Yu Wu, et al. Deepseekmoe: Towards ultimate expert specialization in mixuureoofeexeets language models. arXiv preprint arXiv:2401.066 .66066, 2024.
[39] Meta AL The Ilama 4 herd: The beginning of a new era of natively multimodal ai innovation. Al at Meta blog, 2025.
[40] Joon Son Chung, Andrew Senior, Oriol Vinyals, and Andrew Zisserman. Lip reading sentences in the wild. In 2017 IEFE Conference Computer Vision and Pattern Recognition (CVPR), pages 34533343.. IERR, 2017.
[41] Triantafyllos Afouras, Joon Son Chung. and Andrew Zisserman. Lrs3-ted: large-scale dataset for visual speech recognition, arXiv preprint arXiv: 08040,,,, 2018.
[42] Niklas Muennighoff, Luca Soldaini, Dirk Grocneveld, Kyle Lo, Jacob Morrison, Sewon Min, Weijia Shi, Pete Walsh, Oyvind Tafjord, Nathan Lambert, et al. Olmoe: Open mixture-of-experts language models. arXiv preprint arXiv:2409.000 020000, 2024.
[43] Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401.0400 04088, 2024.
[44] William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1-39, 2022.
[45] Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoe Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv: 1001.06538, 2017.
[46] Jakub Krajewski, Jan Ludziejewski, Kamil Adamczewski, Maciej Pióro, Michal Krutul, Szymon Antoniak, Kamil Ciebiera, Krystian Król, Tomasz Odrzygóádž. Piotr Sankowski, et al. Scaling laws for fine- fine- grained mixture of experts. arXiv preprint arXiv:2402.0787 2024.
[47] Jan Ludzicjewski, Maciej Pióro, Jakub Krajewski, Macicj Stefamiak, Michat Krutul, Jan Malašnicki, Marek Cygan, Piotr Sankowski, Kamil Adamczewski, Piotr Miloš, et al. Joint moe scaling laws: Mixture of experts can be memory efficient. arXiv preprint 25X 2025.
[48] Yongqi Huang, Peng Ye, Chenyu Huang, Jianjian Cao. Lin Zhang. Baopu Li, Gang Yu, and Tao Chen. Ders: Towards extremely efficient upcycled mixture-of-experts models. In CVPR, 2025.
[49] Jiachen Li, Xinyao Wang, Sijie Zhu, Chia-Wer Wen Kuo, Lu Xu, Fan Chen, Jitesh Jain, Humphrey Shi, and Longyin Wen. Cumo: Scaling multimodal IIM with co-yplld mixture-of-experts, Advances in Neural Information Processing Systems, 77:13224-131244 2024.
[50] Ethan He, Abhinav Khattar, Ryan Prenger, Vijay Korthikanti, Zijie Yan, Tong Liu, Shiqing Fan, Ashwath Aithal, Mohammad Shocybi, and Bryan Catanzaro. Upcycling large language models into mixture of experts. arXiv preprint arXiv:2410.072:
2024.
[51] Róbert Csordás, Kazuki Irie, Jürgen Schmidhuber. Christopher Potts, and Christopher D Manning, Moeut: Mixture-of eeeeeet- universal transformers. In NeurIPS, 2024.
[52] Sangmin Bac, Yujin Kim, Reza Bayat, Sungnyun Kim, Jiyoun Ha, Tal Schuster, Adam Fisch, Hrayr Harutynyyan, Ziwei Ji, Aaron Courville, al. Mixtureeof-rrccursinss: Learning dynamic recursive depths for adaptive token-level computation. arXiv preprint arXlv:2507.10524, 2025.
[53] Yongxin Guo, Zhenglin Cheng, Xiaoying Tang, Zhaopeng Tu, and Tao Lin. Dynamic mixture of experts: An auto- auto-tuning approach for efficient transformer models. arXiv preprint arXiv:2405.1 140.2997.14497 2024.
[54] Ted Zadouri, Ahmet Üstün, Arash Ahmadian, Beyza Ermiş, Acyr Locatelli, and Sara Hooker. Pushing mixture of experts to the limit: Extremely parameter efficient moe for instruction tuning. arXiv preprint arXiv:2309.0S: 300.0044..00444 2023.
[55] Umberto Cappellazzo. Daniele Falavigna, and Alessio Brutti. Efficient fine-tuning of audio spectrogram transformers via soft mixture of adapters. In Interspeech, 2024.
13

---

[56] Shihan Dou, Enyu Zhou, Yan Liu, Songyang Gao, Jun Zhao. Wei Shen. Yuhao Zhou, Zhiheng Xi, Xiao Wang. Xiaoran Fan, et al. Loramoe: Alleviate world knowledge forgetting in large language models via moe-style plugin. arXiv preprint arXiv:2312.09979 2023.
[57] Yihan Wu, Yifan Peng. Yichen Lu, Xuankai Chang, Ruihua Song. and Shinji Watanahe. Rohust audiovisual speech recognition models with mixture-of-expers.. 2024 IEFE Spoken Language Technology Workshop (SL.T), pages 43-48. IEFF, 2024.
[58] Ying Cheng, Yang Li, Junjie He, and Rui Feng. Mixtures of experts for audio-visual learning, Advances in Newral Information Processing Systems, 37:219-244 243, 2024.
[59] Sungnyun Kim, Kangwook Jang, Sangmin Bac, Sungwoo Cho, and Se- Young Yun. Mohave: Mixture of hicrarchical audio-visual experts for robust speech recognition. arXiv preprint 2502 10447, 2025.
[60] Sncha Kudugunta, Aditya Kusupati, Tim Dettmers, Kaifeng Chen, Inderjit Dhillon, Yulia Tsvetkow, Hannanch Hajishirzi, Sham Kakade, Ali Farhadi. Prateck Jain, et al. Matformer: Nested transformer for elastic inference. arXiv preprint arXiv:2310.07707. 2023.
[61] Abhinav Shukla, Sai Vemprala, Aditya Kusupati, and Ashish Kapoor. Matmamba: matryoshka state space model. arXiv preprint arXiv:2410.067/6 2024.
[62] Vladimir Zaigrajew, Hubert Baniecki, and Przemyslaw Biecek. Interpreting clip with hierarchical sparse autoencoders. arXiv preprint arXiv:2502.20578 2025.
[63] Ali Hojjat, Janek Haberer, Soren Pirk, and Olaf Landsiedel. Thinkingvit: Matryoshka thinking vision transformer for elastic inference. arXiv preprint arXiv:2507.1088 2025.
[64] Bokai Lin, Zihao Zeng. Zipeng Xiao, Siqi Kou, Tianqi Hou, Xiaofeng Gao, Hao Zhang, and Zhijie Deng. Matryoshkakkv Adaptive kv compression via trainable orthogonal projection. In ICLR, 2025.
[65] Kuniaki Noda, Yuki Yamaguchi, Kazuhiro Nakadai, Hiroshi G Okuno, and Tetsuya Ogata. Audio-visual speech recognition using deep learning. Applied intelligence, 42:722-737, 2015.
[66] Youssef Mroueh, Etienne Marcheret, and Vaibhava Goel. Deep multimodal learning for audio-visual speech recognition. In 2015 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pages 2130-2134. TEEE, 2015.
[67] Hendrik Meutzner, Ning Ma, Robert Nickel, Christopher Schymura, and Dorothea Kolossa. Improving audio-visuu speech recognition using deep neural networks with dynamic stream reliability estimates. In 2017 IEEE international conference on acoustics, speech and signal processing (ICASSP). pages 5320-5324. IEEE, 2017.
[68] Stavros Petridis, Themos Stafylakis, Pingchuan Ma, Georgios Tzimiropoulos, and Maja Pantic. Audio- visual speech recognition with hybrid etc/attention architecture. In 2018 IEEE Spoken Language Technology Workshop (SLT), pages 513-520. IECE, 2018
[69] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances neural information processing systems. 30, 2017.
[70] Triantafyllos Afouras, Joon Son Chung, Andrew Senior, Oriol Vinyals, and Andrew Zisserman. Deep audio- visual speech recognition. IEEE transactions on pattern analysis and machine intelligence, 44( 44(12):8717 8727, 2018
[71] Pingchuan Ma, Stavros Petridis, and Maja Pantic. End-to-end audio-visual speech recognition with conformers. In ICASSP 2021-2021 IEFF International Conference Acoustics, Speech and Signal Processing (ICASSP), pages 7613-7617. TEBE, 2021.
[72] Dmitriy Serdyuk, Otavio Braga, and Olivier Siohan. Audio-visual speech recognition is worth 32 x 32 8 voxels. In 2021 IEEE automatic speech recognition and understanding workshop (ASRU), pages 796 802. IEEE, 2021.
[73] Bowen Shi, Wei-Ning Hsu, Kushal Lakhotia, and AbdeIrahman Mohamed. Learning audio-visual speech representation by masked multimodal cluster prediction. In International Conference on Learning Repre- sentations, 2022.
[74] Bowen Shi, Wei-Ning Hsu, and AbdeIrahman Mohamed. Robust self-supervised audio speech recognition. In Proc. Interspeech, 2022.
14

---

[75] Alexandros Haliassos. Pingchuan Ma, Rodrigo Mira, Stavros Petridis, and Maja Pantic. Jointly learning visual and auditory speech representations from raw data. In International Conference on Learning Representations, 2023.
[76] HyoJung Han, Mohamed Anwar, Juan Pino, Wei-Ning Hsu, Marine Carpuat, Bowen Shi, and Changhan Wang. Xlavs-r: Cross-lingual audio-visual speech representation learning for noise-robust speech percep- tion. In Proceedings of the 62nd Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pages 12896-1299 1091-1221.11111 2024.
[77] Andrew Rouditchenko, Yuan Gong, Samuel Thomas, Leonid Karlinsky, Hilde Kuchne, Rogerio Feris, and James Glass. Whisper-faminno: Integrating visual features into whisper for audio-visual speech recognition and translation. In Proc. Interspeech, pages 2420-2424, 2024.
[78] Pingchuan Ma, Alexandros Haliassos, Adriana Hrmmandzz-Lopez Honglic Chen, Stavros Petridis, and Maja Pantic. Auto-avsr: Audio-visual speech recognition with automatic labels. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5, IEEE, 2023.
[79] Joanna Hong. Minsu Kim, Daehun Yoo, and Yong Man Ro. Visual context-driven audio feature enhance- ment for robust end-to-end audio-visual speech recognition. In Proc. Interspeech, 2022.
[80] Joanna Hong, Minsu Kim, Jeongsoo Choi, and Yong Man Ro. Watch or listen: Robust audio-visual speech recognition with visual corruption modeling and reliability scoring. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18783-18794, 2023.
[81] Chen Chen, Yuchen Hu, Qiang Zhang, Heqing Zou, Beier Zhu, and Eng Siong Chng.. Leveraging modality- specific representations for audio-visual speech recognition via reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 12607-12615, 2023.
[82] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 21ip-2: Bootstrapping language image pre-tranning with frozen image encoders and large language models. In International conference on machine learning. pages 19730-19742. PMLR, 2023.
[83] Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. In International conference on machine learning. pages 28492-28518. PMLR, 2023.
[84] Nan Du, Yanping Huang, Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, al. Glam: Efficient scaling of language models with mixture- of-experts. International conference on machine learning, pages 5547-5569, PMLR, 2022.
[85] Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. arXiv preprint arXiv:2006. 16668, 2020.
[86] Sheng Shen, Le Hou, Yanqi Zhou, Nan Du, Shayne Longpre, Jason Wei, Hyung Won Chung, Barret Zoph, William Fedus, Xinyun Chen, et al. Lixture-of-cxperr meets instruction tuning: A winning combination for large language models. arXiv preprint arXiv:2305. 14705, 2023.
[87] Mototaka Suzuki. Cyriel MA Pennartz, and Jaan Aru. How deep is the brain? the shallow brain hypothesis. Nature Reviews Neuroscience, 24(12):778-79) 791, 2023.
[88] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang. Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In Internatioal Conference on Representation Learning, volume 1, page 3, 2022.
[89] Junxian He, Chunting Zhou, Xuezhe Ma. Taylor Berg-Kirkpatte and Graham Neubig. Towards a unified view of parameter-effiicnt transfer learning. arXiv preprint arXiv:21 10.04366, 2021.
[90] Xun Wu, Shaohan Huang, and Furu Wei. Mixture of lora experts. arXiv preprint arXiv:2404.33626 2024.
[91] Umberto Cappellazzo, Danicle Falavigna, Alessio Brutti, and Mirco Ravanelli. Parameter-efficient transfer learning of audio spectrogram transformers. In 2024 IEEE 34th International Workshop on Machine Learning for Signal Processing (MLSP), pages 1-6. IEEE, 2024.
[92] Abhimanyu Dubey, Abhinay Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad AI- A-Dallee Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The lama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024.
15

---

[93] Yuchen Hu, Ruizhe Li. Chen Chen, Chengwei Qin, Qiushi Zhu, and Eng Siong Chng. Hearing lips in noise: Universal viseme- viseme-phoneme mapping and transfer for robust audio-visual speech recognition. arXiv preprint arXiv:2306. 10563, 2023.
[94] Alexandros Haliassos, Rodrigo Mira, Honglie Chen, Zoe Landgraf. Stavros Petridis, and Maja Pantic. Unified speech recognition: single model for auditory, visual, and audiovisual imputs. arXiv preprint ar225 2024.
[95] Andrew Varga and Herman Steeneken. Assessment automatic speech recognition: noisex-92 database and an experiment to study the effect of additive noise on speech recognition systems. Speech communication, 12(3):247-255 251, 1993.
16

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The limitations of work are discussed in the Appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
17

---

Justification:
Our paper does not include theory assumptions or proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: The training and model details described in Section 4.1 and in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
18

---

Answer: [Yes]
Justification: The code will be publicly available upon acceptance and the training data is described in Section 4.1
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co
for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (ht cttpsscccclllliiicc hwwpoliccggiiie/////// wtwtieeee//oeeubiisssssssssse ssion for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The training and testing details are listed in Section 4.1 and in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification:
AVSR research typically does not test and report error bars experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
19

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide sufficient information on the computer resources in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http:////wuurri.s..oooooo///tttt
Answer: [Yes]
Justification:
We fully comply with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Please the Appendix for discussion about societal impacts of our work.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
20

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The creators and original owners the assets used in the paper are properly credited and explicitly mentioned with respect.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paper swi thcode. com/ coomdatasets. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
21

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
22

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs any important, original, or non-standard components.
Guidelines:
The answer NA means that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy https////eeerroo.cccoooooeeeeess for what should or should not be described.
23

---

N
MoME
FFN
+
1 MHSA
MoME
MHSA
MoME
MHSA
Large Longuage Model
Large
Language Model
Large Language Model
MHSA insertion
FFN insertion
Layer insertion
Figure 7: Overview of MoME's placement strategies. supports parallel insertion to: 1) MHSA module, 2) FFN module, and 3) whole LLM layer. Layer normalizations are omitted for simplicity. Extensive experiments with the three configurations can be found in the main paper and in the Appendix.
Appendix
A
Experiment Details
A.1
Data Licenses
LRS3
41
is licensed under CC BY-NC-ND 4.0. LRS2 40 allows for academic, non-commercial
research.
A.2
Pre-Processing
We follow
for the pre-processing of the datasets. For the video modality, we crop the mouth region of interests (ROIs) through bounding box of 96 x 96. Each frame is normalised by subtracting the mean and dividing by the standard deviation of the training Audio data only undergo z-normalisation per utterance.
78
78 23
A.3 Training Details
Following 14 78
we augment visual inputs through horizontal flipping, random cropping, and adaptive time masking, while for audio only apply adaptive time masking. For training, similar to (78.. we sample bubble noise from the NOISEX dataset (95] using uniform distribution from the range [-5, 0, 5, 10, 15, 20, oo] dB and add it to the clean speech signal. We define the textual prompts as in [14] "Transcribe (task_promp)
to text.", where pask_prompt E ("speech", "video", "speech and video"). We train our model for 10 epochs with the AdamW optimizer with cosine annealing scheduler and weight decay set to 0.1 using NVIDIA H200 GPUs. The learning rate is set to le-3 for ASR and AVSR tasks, and 5 for VSR.
A.4 Inference Details
For decoding, we use beam search with a beam width of 15 and temperature of 0.6.
B MoME's Insertion Variants
As discussed in Section 3.2, MoME offers high flexibility in that it can be inserted in multiple points within the LLM. Specifically, it supports three variants: (1) parallel insertion to the MHSA module, (2) parallel insertion to the FFN module, and (3) parallel insertion to the whole LLM layer. Figure illustrates these three configurations.
C Additional AVSR Experiments
In Figure of the main paper, we compare MoME-23/4-MHSS with several state-of-the-art methods on the LRS3 dataset. To complete analysis, we report in Table 6 the performance of multiple

---

To Table 6: AVSR results on LRS2 and LRS3 in terms of WER (%) across different (audio-visual compression rates (e.g., (4.2)). All methods use Whisper medium as audio encoder and Llama 3.1-8B
as L.L. & Llama-AVSR [1] involves training separate models for each pair of audio-visus rates.
Active
LRS2 Dataset
LRS3 Dataset
Method
Params
(4
(4,5)
(16,2)
(16,5)
(4,2)
(4,5)
(16,2)
(16,5) 5)
Llama-AVSR [
27.3M
2.4
2.2
2.9
3.3
0.9
0.9
1.6
2.1
Llama-MTSK MS
23
27.3M
2.1
2.3
2.9
3.2
1.0
1.1
1.5
1.6
Llama-MTSK SS
23
27.3M
2.4
2.1
2.9
2.9
0.9
1.0
1.7
1.8
Llama-MTSK MSS
23
54.5M
2.4
2.5
3.2
3.4
1.2
1.0
1.5
1.6
MoME-23/4-MHSA
14.1M
1.9
1.8
2.3
2.2
0.9
0.9
1.3
1.5
MoME-23/4LLAER
14.1M
1.9
1.9
2.0
2.2
0.9
0.9
1.4
1.4
MoME-23/4-MHSA
3.5M
2.1
2.0
2.8
3.0
1.0
1.1
1.5
2.0
MoME configurations across all four scales on both LRS2 and LRS3. We observe that the two configurations with 14. 1M parameters consistently outperform prior methods, showing significantly less degradation at lower scales. Furthermore, even when reducing the bottleneck dimension of each expert resulting in just 3.5M active parameters, MoME still achieves strong performance while minimizing parameter usage. These results further highlight MoME's superiority over existing methods and its robustness across scales.
C.1 Additional Ablations MoME
In Section 4.3, we ablate key components MoME, including the number of routed experts, the use of shared experts, and its robustness under noisy conditions. In this section, focus the role of 1) the shared router and 2) the Matryoshka weights applied to the token sequences.
MoME uses router shared across all granularities to activate similar subsets of experts, thereby promoting implicit alignment. To ablate this design choice, we introduce granularity-speciff router, assigning a separate router to each scale while keeping the experts shared. We refer to this variant as disjoint routers (DR). As shown Table 7 using separate routers leads to poorer performance, with greater WER degradation at higher compression rates. This is attributed to the loss of cross-scale alignment enabled by the shared router.
As discussed in Section 3.2, MoME is trained over all the audio-visual token scales:
LLM G-L ==
log (E)(((ZZ) cij.
The weight coefficients Gjj are set to I across all scales, following prior work 32 36
23
try to improve performance at lower scales, we experiment with non-uniform weights (NUW) by increasing the training loss contribution of lower scales, setting G.j = [1, 1.5, 2). This design emphasizes lower scales during training. However, as shown in Table the NUW variant does not improve performance at lower scales and consistently degrades results at higher scales such (4, 2) and (4, 5). Consequently, in our main experiments, we adopt uniform weighting by setting C. = for all scales.
D Additional Visualization Analyses
In Section 4.4, we provide several visual analyses to understand how MoME learns tokens at multiple scales. Here, include additional examples in this direction.
D.1 Additional Cross-Modal Matryoshka Token Analyses
We include additional correlation matrices in Figure Specifically, we analyze the sentence "ir's long way from home" from the LRS2 dataset using the MoME-157 configuration. Consistent with
25

---

Table Ablation on the use of 1) disjoint routers (DR) and non-uniform weights (NUW) on the LRS2 and LRS3 datasets for the
configuration.
Compression Rates (A,V)
Method
(4,2)
(4,5)
(16,2)
(16,5)
LRS2 Dataset
MoME-1553-LAYER
2.9
2.9
4.0
4.8
DR
2.9
2.9
4.7
5.0
NUW
4.0
3.7
4.0
4.9
LRS3 Dataset
MOME-I5/3-LAYER
1.8
1.9
3.0
3.4
DR
1.9
2.0
3.2
3.7
NUW
2.4
2.2
3.5
3.6
1233 18195
Figure 8: Additional intra-modality and cross-modality correlation matrices for MoMB-15/3 trained on LRS2. We study the sentence: "it's 's a long way from home".
previous results, we observe strong linear correlation between token sequences across different granularities and modalities (i.e., high cosine similarity values along the diagonal). Thanks to the use of shared router and shared experts, MoME encourages token sequences at lower granularities to attend to specific portion of sequences at higher granularities. This mechanism helps mitigate performance degradation at lower granularities.
D.2 Additional Expert Activation Analysis
Figure shows the layer-wise expert activation distribution across different scales for the MoME- FFN configuration the LRS2 dataset. Similar to the results in Section 4.4, MoME tends to activate the same experts given layer index across the four different scales. Additionally, MoME activates different pools of experts from layer to layer, indicating expert heterogeneity. These results suggest that MoME exhibits consistent behavior across different configurations (e.g., 15 vs. 23 experts, MHSA vs. FFN).
E Inference Cost Comparison
We compare MoME with Llama-AVSR 14 and Llama-MTSK 23 in terms of inference cost. Specifically, we compare the three methods in terms of inference time and number of generated tokens per second (TPS). We average over 4 speech inputs with an audio/visual compression rate of (4,2). As reported in Table 8 MoMEE-55/3NM 15/3M115// shows a modest increase in inference time (around 1.25x) and a corresponding reduction in tokens generated per second, primarily due to router overhead and expert dispatching latency, as expected in sparse MoE-based systems, while outperforming by significant margin the other two methods. However, the primary design goal of MoME is not faster inference, but rather: 1) Substantially reduced memory requirements via support for high compression rates with strong performance and within the same model, 2) Robustness under noise, and 3) Improved interpretability through expert specialization and shared routing. In conclusion, we believe that this moderate inference overhead is justified by the gains in scalability, generalization, and parameter efficiency, particularly for deployment in memory-constraine or
26

---

(4,20
Dowe Actisttonn
Egent
Sare
Expert wereen
Srark
X3
Layor 2rdex
Figure 9: MoMEE-2344FTT expert activation analysis across multiple scales and layers LRS2.
Table 8: Inference cost comparison between our proposed MoME, Llama-AVSR and Llama-MTSK in terms of inference time and generated tokens per second (TPS).
Method
Inference Time (s)
TPS
WER 1
Illma--VSSI 14
2.39
15.85
4.1
Llama-MTSK 23
2.51
15.09
3.6
MOMF-15/3-MHSA
2.98
12.98
2.9
multi-resolution scenarios. Nonetheless, optimizing router execution remains interesting avenue for further improving inference efficiency.
F MoME's Extension to Other Multimodal Tasks
MoME is designed as a parameter-efficient fine-tuning module that operates in parallel with frozen layers a pretrained LLM or encoder. To apply MoME to other multimodal tasks (e.g., image-text retrieval), two key conditions should be met: (1) the presence of pretrained encoder or LLM backbone that can remain frozen during fine-tuning, and (2) input modalities (e.g., image tokens) whose token granularity impacts performance and efficiency, allowing for compression.
Beyond these considerations, MoME can be integrated with minimal architectural changes. For instance, in image-ter text retrieval, one could use CLIP to generate image tokens and reduce their number via average pooling or other compression methods. A MoME layer could then be inserted in parallel with CLIP's transformer layers, enabling elastic retrieval based on resource constraints. Training would proceed by computing the task-specific loss (e.g., contrastive loss) across different token compression rates, similar to how we use multiple audio-visual granularities in AVSR.
G Limitations
While MoE-based models limit inference-ting -ime computation by activating only small subset of experts, they still require all experts to reside in memory, resulting in increased memory usage. Since MoME follows the MoE paradigm, inherits this memory overhead. However, because our experts are lightweight due to the bottleneck design, the overall memory footprint remains modest. Additionally, the inference cost of MoME is slightly higher than baselines like Llama-AVSR and Alama-MSTK. Finally, our paper focuses exclusively audio-visual models. However, due to
27

---

its flexible design, MoME can be readily integrated into other LLMs and applied to tasks beyond audio-visual learning, such as vision-langue -anguae modeling. Exploring its effectiveness in these broader domains is promising direction for future work.
H Societal Impacts
MoME utilizes pre-trained large language models (LLMs), which inherently carry the limitations of LLMs, such as the potential to generate inacourate or biased outputs. Therefore, we advise caution and recommend conducting thorough safety and fairness evaluations before deploying MoME in any downstream applications.
28


Reviews:
[{"review_id": "Hkpox25wqT", "reviewer": "Reviewer_SngT", "summary": "This paper introduces MoME (Mixture of Matryoshka Experts), a novel framework that combines Mixture-of-Experts (MoE) with Matryoshka Representation Learning (MRL) for audio-visual speech recognition (AVSR). MoME employs top-k routed and shared experts within a bottleneck design, enabling efficient expert sharing across token granularities and modalities. MoME achieves state-of-the-art results on AVSR, ASR, and VSR tasks with fewer active parameters and remains robust under high compression and noise. Visualization analyses further reveal MoME’s cross-scale consistency in expert utilization.", "questions": "1. I suggest including a comparison of MoME’s inference cost against other baselines, and providing an ablation that shows how the cost changes with different numbers of experts.\n2. Could you provide results for different values of k in the top-k routing, while keeping the number of routed experts fixed? It would be interesting to see how performance changes, particularly when all routed experts are selected.\n3. Using training FLOPs instead of training hours could improve the reproducibility and hardware independence of the comparisons.\n4. Some tables reuse model names for different bottleneck settings, which may confuse readers.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n1. The paper is well-written and clearly organized.\n2. The proposed MoME framework is supported by extensive experiments across multiple tasks (AVSR, ASR, VSR) and datasets (LRS2, LRS3), and consistently outperforms or matches strong baselines across various compression settings. They also show robustness under varying levels of acoustic noise.\n3. This work includes thorough ablation studies on different insertion points of the MoME module within the LLM architecture, varying the number of routed and shared experts, providing insights into optimal integration strategies.\n4. Visualization of cross-modality correlations and expert activations provides insights into MoME’s behaviours, highlighting its ability to align representations across modalities and scales.\n\n**Weaknesses**\n1. The appendix includes a TFLOP analysis of MoME under different compression rates, but it does not compare against other baselines (e.g., LLaMA-AVSR, LLaMA-MTSK) regarding inference cost. The effect of varying the number of experts on computational efficiency is not explored.\n2. The effectiveness of the router mechanism is not fully explored, as there is no analysis on different values of k in top-k routing. Such an analysis would strengthen claims about routing efficiency and expert selection.\n3. Figure 1 uses training hours instead of FLOPs, limiting hardware-agnostic comparison.", "was_revised": true, "final_justification": "After reading the rebuttal and considering the discussions, I am satisfied with the authors’ clarifications and additional experiments. The authors have addressed my main concerns in the rebuttal. The clarification regarding the claim about computational efficiency, the new experiments on top-k routing, and improvements to terminology were appreciated.", "rebuttal": "We thank the reviewer SngT for their time and constructive feedback, and for acknowledging the organization, extensive experiments, and ablation studies in our paper. We address all their mentioned concerns and questions one by one below. Each reviewer comment/question is formatted as a block quote.\n\n> *The appendix includes a TFLOP analysis of MoME under different compression rates, but it does not compare against other baselines (e.g., LLaMA-AVSR, LLaMA-MTSK) regarding inference cost. The effect of varying the number of experts on computational efficiency is not explored.I suggest including a comparison of MoME’s inference cost against other baselines, and providing an ablation that shows how the cost changes with different numbers of experts.*\n\nThank you for highlighting these important points. We included the TFLOP analysis to demonstrate the computational benefits of MoME in terms of reduced FLOPs and, consequently, lower memory usage. This enables users to choose among a range of audio-visual compression rates within a single model, depending on their resource constraints.\n\nRegarding a comparison between MoME and Llama-AVSR/Llama-MTSK in terms of inference cost, we provide such analysis below. We compare the three methods in terms of inference time and number of generated tokens per second (i.e., throughput). We average over 4 speech inputs with an audio/visual compression rate of 4/2. MoME shows a modest increase in inference time (~1.25x) and a corresponding reduction in throughput, primarily due to router overhead and expert dispatching latency, as expected in sparse MoE-based systems, while outperforming by a significant margin the other two methods. However, the primary design goal of MoME is not faster inference, but rather: **1)** Substantially reduced memory requirements via support for high compression rates with strong performance and within the same model **2)** Robustness under noise and **3)** Improved interpretability through expert specialization and shared routing. \n\n|Method | Inference Time (s) | Throughput (tok/s) | WER |\n|--| -- |-- |--| \n|Llama-AVSR |2.39 |15.85 | 4.1\n|Llama-MTSK MSS | 2.51| 15.09| 3.6\n|MoME-16/3-MHSA| 2.98| 12.98| 2.9\n\nWe believe that this modest inference overhead is justified by the gains in scalability, generalization, and parameter efficiency, particularly for deployment in memory-constrained or multi-resolution scenarios. Nonetheless, optimizing router execution remains an interesting avenue for further improving inference efficiency.\n\nAdditionally, as per your request, we compare MoME while varying the number of routed experts in the set {1, 4, 7, 15}. The corresponding results are reported below. As expected, increasing the number of experts leads to a slight increase in inference cost. However, this is consistently accompanied by improved performance, as shown in the Table below and Table 3 in the paper.\n\n|# of Experts | Inference Time (s) | Throughput (tok/s) | WER\n|--| -- |-- |--|\n|1 | 2.49| 15.26| 3.4\n|4 | 2.64| 14.39| 3.2\n|7| 2.81| 13.50| 3.0\n|15| 2.98| 12.98| 2.9|\n\nWe will add a dedicated section in the paper where we discuss and analyze inference costs as we have done above. We will also update the Limitations section to reflect the increase in inference costs incurred by MoME. \n\n> *The effectiveness of the router mechanism is not fully explored, as there is no analysis on different values of k in top-k routing. Such an analysis would strengthen claims about routing efficiency and expert selection. Could you provide results for different values of k in the top-k routing, while keeping the number of routed experts fixed? It would be interesting to see how performance changes, particularly when all routed experts are selected.*\n\nThank you for raising this point. We conducted additional experiments to analyze the impact of varying the value of *k* in top-*k* routing, using the MoME-8/k-MHSA configuration with one shared expert on the AVSR task (LRS2 dataset). The table below summarizes the results, where each row corresponds to a different number of activated routed experts out of the 8 available (i.e., varying *k* in top-*k*), and we compare across multiple compression rates.\n\nWe observe that increasing *k* leads to moderate performance improvements, particularly at higher compression rates. However, this comes at the cost of increased computation and reduced sparsity, which deviates from the core objective of sparse MoE design. In practice, increasing the number of active experts without adjusting the expert bottleneck size leads to higher resource usage and latency.\n\nTherefore, while selecting all routed experts (i.e., dense activation) does offer slight gains, we find that top-*k* values of 1 or 2 strike a good balance between efficiency and performance, maintaining the sparse, modular nature of MoME.\n\n| Top-k| (4,2) | (4,5) | (16,2) | (16,5) |\n|-- | -- | -- | -- | -- |\n| 1| 3.3 |3.3  |4.6  | 4.7 |\n| 2|  3.2|  3.3|  4.5|  4.7|\n| 4|  3.2|  3.2|  4.5|  4.6|\n| 6|  3.2|  3.2|  4.5|  4.5|\n| 8|  3.1|  3.2|  4.4|  4.5|\n\n> *Using training FLOPs instead of training hours could improve the reproducibility and hardware independence of the comparisons.*\n\nWe thank the reviewer for the valuable suggestion. We would like to clarify that in Figure 1, “training hours” refers to the **total amount of labeled and unlabeled speech/video data** used to train each method, not the **wall-clock time** or number of GPU hours required for training. This metric is commonly reported in AVSR literature as a way to quantify the **scale of supervision** available during training. For example, Auto-AVSR is trained on approximately **3448 hours** of speech data, taken from multiple datasets: LRS2, LRS3, AVSpeech, and VoxCeleb2. In contrast, MoME is trained only on LRS2 and LRS3, totaling **658 hours**. To avoid this confusion, we have updated “training hours” with “training data hours” in Figure 1 and in the paper.\n\nWhile we agree that reporting training FLOPs would enhance hardware independence and reproducibility, in the AVSR domain it is standard practice to report the number of (labeled/unlabeled) training data hours, and existing works do not typically provide training FLOP counts. As such, a direct comparison based on training FLOPs is unfortunately not feasible. To address the reviewer's concern, though, we provided the inference details to quantify computational efficiency across methods publicly available, which will be included in our potential future manuscript.\n\nWe hope our response clarifies your doubts and questions. We are more than happy to further discuss if you have any other questions or suggestions."}, {"review_id": "clgHgC5wbu", "reviewer": "Reviewer_XaQo", "summary": "This paper introduces a framework called MoME (Mixture of Matryoshka Experts) for AVSR, which combines sparse MoE with MRL to dynamically adjust token granularities and allocate computing resources. They propose routed experts controlled by a top-k router plus shared experts. This allows implicit alignment and knowledge transfer across different compression levels, showing lower WER for ASR and AVSR.", "questions": "1. What impact do the number of shared experts and the choice of top-k for routed experts have on final WER, and is there an automated strategy to adjust these?\n2. If applying MoME to other multimodal tasks (e.g., image–text retrieval), what architectural or training modifications would be needed?\n3. Is it possible at inference time to adaptively switch granularities based on input content (e.g., speaking rate or noise level) rather than using preconfigured settings?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- The MoME model can adjust token compression according to hardware or scenario requirements, balancing computational efficiency without retraining for each level.\n- The shared router and expert allow high-compression inputs to reuse fine-grained features learned at lower compression levels.\n\nWeakness:\n\n- The core idea is integrating MoE into MRL without introducing fundamentally new mechanisms. Given existing examples like LLAMA4, DeepSeekMoE, and Llama-MTSK, what distinguishes MoME’s contribution beyond merging sparse experts with multi-granularity tokens?\n- This work claims it is improved for AVSR. However, the results of MoME-23/4-MHSA AVSR compression (4, 2), (4, 5), and ASR compression 4 are very close (1.7, 1.8, and <2.0 %). Is MoME really better at visual understanding, or is it simply boosting ASR? Is this framework better for ASR instead of AVSR?\n- Additionally, the standalone VSR WER ranges from approximately 36 % to 42 % across compression rates, which is still far from practical deployment. Moreover, MoME’s VSR curve tracks the Llama-MTSK SS baseline within a narrow margin (around 1 percentage point) in many compression rates. Are these gains statistically significant?", "was_revised": true, "final_justification": "The author's response addressed my concerns, particularly with the new experiment on the optimized VSR setting. Therefore, I will increase my score to 4.", "rebuttal": "We sincerely thank Reviewer XaQo for taking the time to review our submission and providing thoughtful feedback. We provide our response to your questions and doubts below. Each reviewer comment/question is formatted as a block quote.\n\n> *The core idea is integrating MoE into MRL without introducing fundamentally new mechanisms. Given existing examples like LLAMA4, DeepSeekMoE, and Llama-MTSK, what distinguishes MoME’s contribution beyond merging sparse experts with multi-granularity tokens?*\n\nWhile MoME indeed brings together MoE and MRL, its contribution extends far beyond a simple combination of the two. We highlight the key novel components and architectural innovations that distinguish MoME: \n\n- **Shared Routing Across Scales**: MoME introduces a shared router module across all token granularities. This is not present in LLaMA4, DeepSeekMoE, or Llama-MTSK. The shared router learns to select consistent subsets of experts across compression scales, enabling implicit alignment between fine and coarse representations. This promotes knowledge transfer from richer inputs to more compressed ones, significantly boosting performance under high compression, something not achieved in prior MRL works like Llama-MTSK in AVSR and MMM and MQT in vision-language tasks. \n- **Cross-Scale Expert Sharing with Sparse Routing**: unlike Llama-MTSK, which trains separate LoRA adapters per scale (leading to scale-isolated representations), MoME routes tokens at all scales through the same pool of routed and shared experts, which are trained jointly. This enables cross-scale representation reuse and implicit specialization.\n- **Extremely Parameter-Efficient Fine-Tuning**: thanks to sparse routing and expert reuse, MoME requires significantly fewer parameters during inference and supports extremely parameter-efficient fine-tuning with strong performance (down to 0.9M active parameters)\n- **Interpretability through Expert Specialization**: MoME offers insights into expert behavior across layers and scales (see Fig. 3 and 4). The visual consistency in expert activation and cross-granularity representation clustering are novel findings in the MRL setting and show how MoME enables interpretable and scalable adaptation.\n- **Robustness to Noise**: MoME demonstrates superior performance in noisy conditions (Table 3). We hypothesize this is due to expert specialization to different input regimes, an emergent property of our routing design and joint scale training.\n- **To our knowledge**, this is the first work to propose MoE layers within a Matryoshka-trained, multimodal LLM, specifically in the context of AVSR. \n\n> *This work claims it is improved for AVSR. However, the results of MoME-23/4-MHSA AVSR compression (4, 2), (4, 5), and ASR compression 4 are very close (1.7, 1.8, and <2.0 %). Is MoME really better at visual understanding, or is it simply boosting ASR? Is this framework better for ASR instead of AVSR?*\n\nWe would like to clarify that the primary motivation behind AVSR is to enhance the noise robustness of ASR systems. Since ASR models are vulnerable to acoustic interference, AVSR incorporates visual information, such as lip movements, which is unaffected by noise. As noted by the reviewer, AVSR and ASR often perform similarly under clean conditions; however, in noisy environments, AVSR consistently outperforms ASR due to the complementary robustness provided by the visual modality. To confirm this on MoME, if we extend Table 2 in the main paper by including the results obtained by its ASR counterpart with different noise intensity [7.5, 5, 2.5, 0, -5] we obtain these WERs: [5.4, 9.8, 16.1, 25.4, 78.9]. \n\nSimilar trends have been reported in recent AVSR models such as Llama-AVSR, Auto-AVSR, and USR, where clean-condition WERs for AVSR and ASR are also closely aligned.\n\n> *Moreover, MoME’s VSR curve tracks the Llama-MTSK SS baseline within a narrow margin (around 1 percentage point) in many compression rates. Are these gains statistically significant?*\n\nWe thank the reviewer for mentioning the VSR performance. Upon reviewing our experiments, we discovered that the VSR evaluations were inadvertently conducted using hyperparameters optimized for ASR. We have since re-run the VSR experiments with the appropriate configuration and provide the updated results below in tabular format, as PDFs are not supported, which show that MoME can achieve even stronger performance.\n\nWe report results under both Average Pooling and Stacking compression across compression rates 1 and 5. Our updated findings show that the LAYER and FFN variants of MoME consistently outperform both Llama-AVSR and Llama-MTSK across all compression levels. While the MHSA configuration still improves over the baselines, it is relatively less competitive compared to the other two MoME variants. These corrected results confirm the effectiveness of MoME in the VSR setting.\n\n\"**Average Pooling**” Results\n\n|Method|1|2|3|4|5|\n|--|--|--|--|--|--|\n|Llama-AVSR|39.59|37.85|43.85|47.51|51.50|\n|Llama-MTSK SS|37.03|37.44|41.23|41.11|42.18|\n|MoME-23/4-MHSA|35.37|34.84|38.38|37.78|40.61|\n|MoME-23/4-LAYER|31.74|33.07|33.93|34.10|37.11|\n|MoME-23/4-FFN|29.12|30.00|32.29|33.22|34.47|\n\n“**Stacking**” Results\n\n|Method|1|2|3|4|5|\n|--|--|--|--|--|--|\n|Llama-AVSR |37.24|38.60|43.65|49.72|51.25|\n|Llama-MTSK SS|36.78|38.92|39.78|39.44|41.71|\n|MoME-23/4-MHSA|34.45|36.14|39.30|42.65|44.74|\n|MoME-23/4-LAYER|31.73|32.68|32.90|33.47|36.60|\n|MoME-23/4-FFN|31.15|32.91|33.08|36.04|35.63|\n\n> *What impact do the number of shared experts and the choice of top-k for routed experts have on final WER, and is there an automated strategy to adjust these?*\n\nAs shown in Table 3 in the paper (last two rows), increasing the number of shared experts from 1 to 2 results in marginal gains at some compression rates but leads to performance degradation at the highest compression rate of (16,5). To further validate this trend, we conducted an additional experiment with 3 shared experts and observed no significant improvements. These results suggest that a single shared expert is sufficient to capture scale-invariant knowledge in our setting. Our findings are consistent with recent MoE architectures such as DeepSeekMoE and Llama 4, which also adopt a single shared expert.\n\n|Routed Experts|Shared Expert(s)|Expert Size|Top-k|(4,2)|(4,5)|(16,2)|(16,5)|\n|--|--|--|--|--|--|--|--|\n|23|1|12|4|2.9|3.0|4.2|4.3| \n|23|2|12|4|2.8|3.0|4.1|4.7|\n|23|3|12|4|2.9|3.0|4.2|4.6|\n\nRegarding the impact of top-*k*, we conducted ablations using the MoME-8/k-MHSA configuration with a single shared expert on the LRS2 AVSR task. The number of activated experts (i.e., *k*) was varied while keeping the number of routed experts fixed at 8. \n\nOur results show that increasing *k* yields moderate gains, especially at higher compression rates. However, these improvements come at the expense of increased computation, which contradicts our design objective of **Sparse** MoE. Therefore, we conclude that using a small *k*, typically 1 or 2, offers a practical trade-off between efficiency and performance. This choice preserves sparsity while retaining most of the accuracy benefits of broader expert activation.\n\n|Top-k|(4,2)|(4,5)|(16,2)|(16,5)|\n|--|--|--|--|--|\n|1|3.3|3.3|4.6|4.7|\n|2|3.2|3.3|4.5|4.7|\n|4|3.2|3.2|4.5|4.6|\n|6|3.2|3.2|4.5|4.5|\n|8|3.1|3.2|4.4|4.5|\n\nFinally, regarding the possibility of automatically selecting the optimal value of k in top-k routing and determining the number of shared experts, we believe this is a promising avenue for future work. In particular, a recent ICLR 2025 paper by Yongxin Guo et al., \"Dynamic Mixture of Experts: An Auto-Tuning Approach for Efficient Transformer Models\",  proposes a mechanism to dynamically adjust *k* and even add/drop routed experts based on input requirements. Such techniques could be particularly beneficial in our setting, as higher compression rates may require more active parameters than lower ones.\n\n> *If applying MoME to other multimodal tasks (e.g., image–text retrieval), what architectural or training modifications would be needed?*\n\nMoME is designed as a parameter-efficient fine-tuning module that operates in parallel with frozen layers of a pretrained LLM or encoder. To apply MoME to other multimodal tasks such as image–text retrieval, two key conditions should be met: (1) the presence of a pretrained encoder or LLM backbone that can remain frozen during fine-tuning, and (2) input modalities (e.g., image tokens) whose token granularity impacts performance and efficiency, allowing for compression.\n\nBeyond these considerations, MoME can be integrated with minimal architectural changes. For instance, in image–text retrieval, one could use CLIP to generate image tokens and reduce their number via average pooling or other compression methods. A MoME layer could then be inserted in parallel with CLIP’s transformer layers, enabling elastic retrieval based on resource constraints. Training would proceed by computing the task-specific loss (e.g., contrastive loss) across different token compression rates, similar to how we use multiple audio-visual granularities in AVSR.\n\n> *Is it possible at inference time to adaptively switch granularities based on input content (e.g., speaking rate or noise level) rather than using preconfigured settings?*\n\nIn current Matryoshka-based LLMs, the compression rate is typically selected based on user-defined resource constraints (e.g., available memory or inference latency budget) rather than input content characteristics such as speaking rate or noise level.\n\nHowever, MoME introduces routed and shared experts, which are conditioned on token-level content. As a result, even when using a fixed compression rate, MoME can dynamically specialize its computation based on the nature of the input, e.g., activating different experts in response to noisy or fast-spoken segments. This ability likely contributes to MoME's superior robustness in noisy conditions (see Table 2)."}, {"review_id": "LnAEPw7Qp9", "reviewer": "Reviewer_CWEH", "summary": "This paper applies Mixture of Matryoshka Experts to audio-visual speech recognition. It achieves strong performance on LRS2 and LRS3 on AVSR, ASR, and VSR tasks.", "questions": "I recommend submitting this work to speech or CV specific conferences", "limitations": "Yes", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths: Detailed analysis on compression rate (and combinations) v.s. performance, expert load distribution\n\nWeakeness:\n\n1. The core idea—replacing LoRA (used in prior work like [22] Llama-MTSK) with a sparse Mixture-of-Experts (MoE) in a Matryoshka framework—is incremental and lacks sufficient conceptual novelty. The architecture is essentially a mechanical substitution: swapping out LoRA adapters for top-k routed MoE blocks, while keeping most of the structure (Matryoshka-style multi-scale token compression, shared LLM backbone) intact.\n\n2. Having a few always-on shared experts is not novel, as cited papers such as DeepseekMoE has already used this idea. Applying this idea on audio-visual speech recognition has a small impact for the research community\n\n3. The authors claim SotA performance on LRS2 and LRS3, but in table 1, they only compared their models with two other Llama-based models. As far as I know, they numbers are far from real SotA, or even older papers such as Whisper‑Flamingo (https://arxiv.org/abs/2406.10082)\n\n4. The comparison with Llama-AVSR and Llama-MTSK doesn't seem fair. Because they are trained for each scale combination separately, while proposed model is trained on all scales, and therefore the proposed model consumes a lot more compute.", "was_revised": true, "final_justification": "The rebuttal have addressed some of my concerns and therefore I’m raising my score", "rebuttal": "We thank reviewer CWEH for their review and relevant feedback. Please find below our answers to your concerns. Each reviewer comment/question is formatted as a block quote.\n\n> *The core idea—replacing LoRA (used in prior work like [22] Llama-MTSK) with a sparse Mixture-of-Experts (MoE) in a Matryoshka framework—is incremental and lacks sufficient conceptual novelty. The architecture is essentially a mechanical substitution: swapping out LoRA adapters for top-k routed MoE blocks, while keeping most of the structure (Matryoshka-style multi-scale token compression, shared LLM backbone) intact.*\n\nThank you for the thoughtful comment. While MoME builds on prior work such as Llama-MTSK, it introduces several key conceptual and architectural advances that go beyond a simple substitution of LoRA with MoE: **1) Scalability across granularities**: unlike Llama-MTSK, which trains separate LoRA adapters for each scale (leading to linear growth in parameters with the number of scale combinations), MoME uses a shared pool of routed and shared experts, making it more scalable and parameter-efficient. **2) Cross-scale knowledge transfer**: MoME's shared router and expert design encourages expert reuse across granularities, enabling sequences at high compression to benefit from representations shaped at lower compression. This addresses one of the core limitations of Matryoshka-style models, scale isolation, and is a novel architectural feature not present in prior work. **3) Improved interpretability and robustness**: by routing tokens based on their content and scale, MoME enables expert-level interpretability (e.g., activation patterns across layers/scales) and demonstrates strong robustness to noise, likely due to expert specialization, a property that LoRA-based models do not offer. **4) First unified MoE–MRL integration**: to the best of our knowledge, MoME is the first to integrate sparse Mixture-of-Experts into Matryoshka representation learning, showing how these two paradigms can be combined for adaptive, scale-aware inference.\n\nThese elements collectively demonstrate that MoME is more than a mechanical substitution. It introduces *a new design paradigm* that improves scalability, performance, and interpretability in Matryoshka-based multimodal models.\n\n> *Having a few always-on shared experts is not novel, as cited papers such as DeepseekMoE has already used this idea. Applying this idea on audio-visual speech recognition has a small impact for the research community*\n\nWe appreciate the reviewer’s observation. Please note that we do not claim the use of always-on shared experts as our main novelty, but rather emphasize MoME’s ability to flexibly adapt to available computational resources, while maintaining performance under high compression. While the concept of shared experts is not new (e.g., in DeepSeekMoE or Llama 4), MoME leverages this idea in a substantially different context, with important architectural and practical distinctions: **1) Parameter-efficient fine-tuning**: unlike DeepSeekMoE or Llama 4, which use MoE layers in place of the FFN and typically expand hidden dimensions (e.g., 4× width), MoME is designed for lightweight adaptation. Our experts are extremely lightweight, with bottleneck sizes reduced by up to 256× (down to 1), enabling fine-tuning of frozen LLMs with minimal added parameters. **2) Matryoshka-aware design**: the inclusion of shared experts in MoME is not generic but tailored to the Matryoshka framework. These shared experts capture *cross-scale, global knowledge*, ensuring robustness even at high compression rates. This is essential in our setting, where tokens across multiple granularities must coexist and benefit from shared representation pathways. **3) Integrated with cross-modal routing**: MoME combines shared experts with top-k routed experts and a shared router. This unified mechanism not only enables scale-adaptive inference, but also improves interpretability and performance across AVSR, ASR, and VSR tasks, demonstrating practical benefits under a single set of model weights.\n\nFinally, while our work focuses on AVSR, we believe the generality of the MoME design opens avenues for broader applications, including vision-language tasks, as we discussed in the Conclusion and Limitations sections.\n\n> *The authors claim SotA performance on LRS2 and LRS3, but in table 1, they only compared their models with two other Llama-based models. As far as I know, they numbers are far from real SotA, or even older papers such as Whisper‑Flamingo* (https://arxiv.org/abs/2406.10082)\n\nWe thank the reviewer for raising this point. To clarify: **Table 1** focuses on a controlled comparison between MoME and prior Matryoshka-based or parameter-efficient LLM-based AVSR methods (e.g., Llama-AVSR, Llama-MTSK) under fixed settings. \n\nThe comparison against broader SoTA AVSR models, including Whisper-Flamingo, Auto-AVSR, AV-HuBERT, USR, and others, is provided in **Figure 1**, which include WER results, training data hours and active parameters on the LRS3 benchmark. This broader comparison is discussed in **Section 4.2** (\"**AVSR Results (2)**\"). For example, MoME achieves SoTA results while providing elastic inference, activating consistently less parameters (e.g., around 14M compared to more than 600M of Whisper-Flamingo), and training on fewer hours of data (e.g., 658 hours compared to the 3448 hours of auto-avsr).\n\nWhisper-Flamingo is indeed included in this plot and discussion. We chose Figure 1 to present this comparison because it provides a more holistic view of **performance-efficiency trade-offs** rather than focusing solely on WER. We would be happy to add any other baselines the reviewer suggests or clarify this distinction further in the final version.\n\n> *The comparison with Llama-AVSR and Llama-MTSK doesn't seem fair. Because they are trained for each scale combination separately, while proposed model is trained on all scales, and therefore the proposed model consumes a lot more compute.*\n\nThank you for raising this important point. To clarify: Llama-AVSR is indeed trained separately for each scale, while Llama-MTSK and our proposed MoME are designed as “generalist” models that handle multiple scales within a single model. We believe having a single generalist model (like Llama-MTSK or MoME) that supports multiple scales is more practical and resource-efficient than training multiple specialist models (one per scale) like Llama-AVSR. This approach allows users to dynamically select scales based on available resources without needing to switch models. \n\nRegarding compute, MoME actually requires **less** computation than training multiple Llama-AVSR models. Assuming N scales, Llama-AVSR requires training N separate models, each needing one forward pass through both the audio and video encoders per sample, resulting in N total encoder passes. In contrast, MoME trains a single model with one forward pass through the encoders, followed by applying multiple compression rates internally. Therefore, MoME training process is even faster than training N separate Llama-AVSR models, other than obtaining superior performance across all the scales. Both approaches require N forward passes through the LLM due to the multiple scales, but the main difference in compute cost lies in the encoder passes, where MoME is more efficient.\n\nWe will include this analysis of training time and compute efficiency in the final revision to clarify this benefit. Please let us know if you would like further details or discussion.\n\n> *I recommend submitting this work to speech or CV specific conferences*\n\nThank you for your suggestion. We appreciate your perspective on submitting to speech- or CV-specific conferences. However, we believe NeurIPS is an appropriate venue for our work for multiple reasons. \n\nFirst, our paper addresses audio-visual speech recognition, which inherently involves multiple modalities—speech, vision, and text—and aligns well with the **Applications** category highlighted in the NeurIPS call for papers (e.g., **vision, language, speech and audio, Creative AI**), which is the very first topic of the list.\n\nSecond, the core methodologies we propose, including matryoshka representation learning and Mixture of Experts, fit within the **Deep Learning** topics emphasized by NeurIPS (e.g., **architectures, generative models, optimization for deep networks, foundation models, LLMs**).\n\nWe will include this analysis of training time and compute efficiency in the final revision to clarify this benefit. Please let us know if you would like further details or discussion.\n\nWe hope our response clarifies your doubts and questions. We are more than happy to further discuss if you have any other questions or suggestions."}, {"review_id": "ZNCGx6KTR2", "reviewer": "Reviewer_VnhM", "summary": "The paper is about a particular type of mixture of experts in parallel to frozen LLM layers as part of an audio-visual (AV) automatic speech regognition system (ASR). It builds upon the successful Matryoshka representation learning concept and extends it to a mixture of experts, allowing to adjust MoE experts of various token granularities to be dynamically routed. Both shared and routed experts are used. The overall goal is an efficient LLM-based AV ASR approach that performs strong under a minimal extra use of parameters.", "questions": "On line 222: Why do you use a load-balancing loss? The MoE used in DeepSeek V3 come with the explicitly claimed asset, that *not* training the load balance gives better performance... Idea for future research or did I misunderstand s.th.?\n\nOn Table 3: What is \"expert size\"? Does it refer to one of the math symbols introduced before? Use it here!", "limitations": "Limitations are described in the Appendix only, not in the main paper. Isn't that against the guideline?", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths\n\nThe paper is very well written and organized. Issues with state of the art are nicely analyzed and thereby the own contribution is very well motivated. The novel presented concepts are clearly understandable, math is correct, and experimental validation is convincing w.r.t. the number of active parameters. \n\nWeaknesses\n\nThere is not much to say here, only some minor points:\n\n- As the claims are efficient AV ASR solutions, I would have expected not only to see the number of active parameters reported as proof of efficiency, but also some computational complexity on some H/W platform. Rather than FLOPs/sec that could be tokens/sec or similar.\n-  Figure 2 does not use its space efficiently. Fonts could/should be enlarged.\n- l. 202: You may want to add K<N_r (instead of K only)\n- l.210: a blanc too much\n- l.222: Why do you use a load-balancing loss? The MoE used in DeepSeek V3 come with the explicitly claimed asset, that *not* training the load balance gives better performance... Idea for future research or did I misunderstand s.th.?\n- Tables 2 and 3 captions miss to state that WER (5) is shown.\n- Table 2 caption: Replace \"noise levels\" by \"conditions\", as SNR does not represent a noise level.\n- Table 3: What is \"expert size\"? Does it refer to one of the math symbols introdiced before? Use it here!\n- Limitations are missing in the main paper. Please check the guidelines, but that should have been part of it?", "was_revised": true, "final_justification": "The authors provided a very fair and solid \"author's final remarks\" text, where I have nothing to add. Also in my view, the raised issues of the three active reviewers were adequately anwered and I do not see any open major point to criticize.", "rebuttal": "We thank reviewer VnhM for their positive assessment towards our paper and for their suggestions/questions! We really appreciate your thorough analysis and several points you mention, which help us make our paper clearer. We would like to share our thoughts on these below. Each reviewer comment/question is formatted as a block quote.\n\n> *As the claims are efficient AV ASR solutions, I would have expected not only to see the number of active parameters reported as proof of efficiency, but also some computational complexity on some H/W platform. Rather than FLOPs/sec that could be tokens/sec or similar.*\n\nWe thank the reviewer for this important observation. We would like to clarify that our primary goal is to improve Matryoshka-based MLLMs by combining them with sparse Mixture-of-Experts (MoE), enabling flexible trade-offs between accuracy and efficiency at inference time under a single model. In our paper, we discuss efficiency from two complementary angles: **1) Training Efficiency (Parameter-efficient fine-tuning)**: MoME keeps the backbone LLM frozen and only trains lightweight adapter modules (the experts), dramatically reducing trainable parameters (this is detailed in Table 1). **2) Inference Efficiency (Token and compute reduction)**: MoME supports varying audio-visual compression rates, which reduces the number of tokens processed by the LLM, leading to lower compute cost and memory usage. Following prior work [7,8], we quantify this in Appendix D, where we show up to **8× TFLOP reduction** with aggressive compression (e.g., (16,5)). \n\nTo further support your point, we now include actual *inference time* and *token throughput* measured on an NVIDIA L40 46GB GPU with MoME 16/3-MHSA. For a 23-second speech input, the results demonstrate a significant reduction in inference time as compression ratios increase. For instance, a compression ratio of (16,5) reduced the inference time to 6.74 seconds for transcribing 23 seconds of speech, compared to 12.75 seconds in the no-compression case. This confirms that higher compression rates lead to *faster inference*, *lower GPU memory usage*, and *reduced computational load*, while still *maintaining strong performance* thanks to MoME’s expert routing mechanism. We will clarify this further in the paper. \n\n| (A,V) Ratio | # of AV Tokens | Inference Time (s) | Throughput (tok/s) |\n| -- | -- | -- | -- |\n|(1,1) | 1673 |  12.75| 7.76 |\n|(4,2) | 560 | 8.04 | 12.90 |\n|(16,5) | **184** | **6.74** | **14.17** |\n\n> *Why do you use a load-balancing loss? The MoE used in DeepSeek V3 come with the explicitly claimed asset, that not training the load balance gives better performance... Idea for future research or did I misunderstand s.th.?*\n\nThank you for this insightful question. The load-balancing loss is a widely used technique to prevent **router collapse**, a common issue in sparse MoE models where the router assigns most tokens to a small subset of experts, leaving others underutilized [1]. While DeepSeek V3 observed improved performance without using load balancing, our MoME design is primarily inspired by DeepSeekMoE [2], which does incorporate a load-balancing loss to ensure expert diversity (see Section 3.3 of [2]). This aligns with other recent MoE architectures [3–6], where load balancing is key to encouraging better expert utilization and learning dynamics.\n\n> *On Table 3: What is \"expert size\"? Does it refer to one of the math symbols introduced before? Use it here!*\n\nThank you for pointing this out. In Table 3, \"*expert size*\" refers to the **bottleneck dimension** of each expert. As mentioned in lines 301–303, we adjust this value so that all configurations maintain a similar total number of activated parameters. Specifically, when the number of active experts increases (i.e., higher values of top‑k), we reduce the expert size accordingly. We agree that the term \"expert size\" was not clearly defined, and we have revised the text in lines 301–303 to clarify this. We appreciate your careful reading.\n\n\n> *Limitations are described in the Appendix only, not in the main paper. Isn't that against the guideline?*\n\nThank you for your observation. We have read carefully the NeurIPS submission guidelines, and we confirmed that NeurIPS does not require a Limitations section to appear in the main paper. Therefore, we have included it in the Appendix, section F, in line with the formatting freedom allowed by the conference. We note that while ACL conferences explicitly require a Limitations section in the main submission, NeurIPS does not impose this requirement.\n\n*Following your suggestions*, we have: **1)** Enlarged the font in Figure 2 to make it more readable; **2)** Specified that we evaluate in terms of WER in the captions of Table 2 and 3; **3)** Replaced \"noise levels\" with \"conditions\" in Table 2 caption; **4)** Removed the extra blank space before the comma in line 210 (thank you for the catch!); **5)** Explicitly stated that K is smaller than Nr to achieve sparsity. \n\nWe hope our response clarifies your doubts and questions. We are more than happy to further discuss if you have any other questions or suggestions.\n\n\n\n\n\n**References**:\n\n[1] Shazeer et al., Outrageously large neural networks: The sparsely-gated mixture-of-experts layer, 2017.\n\n[2] Damai Dai et al., DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models, 2024.\n\n[3] Niklas Muennighoff et al., OLMoE: Open Mixture-of-Experts Language Models, ICLR 2025.\n\n[4] Jan Ludziejewski et al., Joint MoE Scaling Laws: Mixture of Experts Can Be Memory Efficient, ICML 2025.\n\n[5] Sheng Shen et al., Mixture-of-Experts Meets Instruction Tuning: A Winning Combination for Large Language Models, ICLR 2024.\n\n[6] Jiachen Li et al., CuMo: Scaling Multimodal LLM with Co-Upcycled Mixture-of-Experts, NeurIPS 2024.\n\n[7] Mu Cai et al., Matryoshka multimodal models, ICLR 2025.\n\n[8] Wenbo Hu et al., Matryoshka query transformer for large vision-language models, NeurIPS 2024."}]

Author rebuttal:

Meta-review: This paper presents a novel audio-visual (AV) automatic speech recognition approach that integrates a mixture of experts (MoE) in parallel with frozen LLM layers, building on the Matryoshka representation learning paradigm. By dynamically routing experts at various token granularities and combining shared and routed components, the method achieves strong performance with minimal additional parameters. Reviewers mostly agree that the work is clearly written, well-motivated, mathematically sound, and experimentally validated. Minor issues exist but they do not affect the technical contribution. Reviewers concern were mostly addressed effectively during the discussion period. I recommend acceptance.
Decision: Accept (poster)