MoHAVE: Mixture of Hierarchical Audio-Visual Experts for Robust Speech Recognition
Authors: Sungnyun Kim
Kangwook Jang
Sangmin Bae
Sungwoo Cho
Se-Young Yun
Venue: ICML 2025 poster (2025)
Keywords: audio-visual speech recognition
mixture-of-experts
multimodal
Primary area: applications->language_speech_and_dialog
TL;DR: We propose a scalable and robust AVSR framework that leverages MoE with hierarchical gating to adaptively utilize audio and visual expert groups.

Abstract: Audio-visual speech recognition (AVSR) has become critical for enhancing speech recognition in noisy environments by integrating both auditory and visual modalities. However, existing AVSR systems struggle to scale up without compromising computational efficiency. In this study, we introduce MoHAVE (Mixture of Hierarchical Audio-Visual Experts), a novel robust AVSR framework designed to address these scalability constraints. By leveraging a Mixture-of-Experts (MoE) architecture, MoHAVE activates modality-specific expert groups, ensuring dynamic adaptation to various audio-visual inputs with minimal computational overhead. Key contributions of MoHAVE include: (1) a sparse MoE framework that efficiently scales AVSR model capacity, (2) a hierarchical gating mechanism that dynamically utilizes the expert groups based on input context, enhancing adaptability and robustness, and (3) remarkable performance across robust AVSR benchmarks, including LRS3 and MuAViC transcription and translation tasks, setting a new standard for scalable speech recognition systems.

MoHAVE: Mixture of Hierarchical Audio- Visual Experts for Robust Speech Recognition
Sungnyun Kim
Kangwook Jang?
Sangmin Bae'
Sungwoo Cho Se-Young Yun
Abstract
Audio-visual speech recognition (AVSR) has be- come critical for enhancing speech recognition in
AVSR Performance (100-WER%)
model size
1B
noisy environments by integrating both auditory and visual modalities. However, existing AVSR systems struggle to scale up without compromis- ing computational efficiency. In this study, we introduce MoHAVE (Mixture of Hierarchical Audio-Visual Experts), a novel robust AVSR framework designed to address these scalability constraints. By leveraging Mixture-of- Mixture-of-Eppert. (MoE) architecture. MoHAVE activates modality- specific expert groups, ensuring dynamic adap- tation to various audio-visual inputs with mini- mal computational overhead. Key contributions of MoHAVE include: (1) a sparse MoE frame- work that efficiently scales AVSR model capacity, (2) a hierarchical gating mechanism that dynami- cally utilizes the expert groups based on input con- text, enhancing adaptability and robustness, and (3) remarkable performance across robust AVSR benchmarks, including LRS3 and MuAViC tran- scription and translation tasks, setting new stan- dard for scalable speech recognition systems.
MoHAVE-Large
X0.4 activations
MoHAVE-Base
AV-HuBERT
90
MoE experts:
MbE experts: 4)
AV-HuBERT-Bass
100
500 600
Activated Params (M)
Figure I. Comparison of AVSR models based standard Trans- formers (AV-HuBERT, Shi et al. 2022a), MoE, and MoHAVE, evaluated under babble noise. The MoE structure boosts the model capacity while maintaining the number of activations. MoHAVE-BARe (359M) achieves similar performance to AV- HuBERT-LARGE (477M) while activating only 189M parameters.
tages over audio-only automatic speech recognition (ASR) approaches. This multimodal approach indispensable in situations where auditory data alone is insufficient for reli- able recognition.
Despite significant advances, AVSR systems have not kept pace with advancements in model scalability as seen in ASR (Radford et al., 2023) or large language models (LLMs) (Ka- plan al., 2020; Clark al., 2022: Achiam al., 2023). Contemporary AVSR models, such AV-HuBERT (Shi et al., 2022a), AV-data2vec (Lian et al., 2023), and Auto- AVSR (Ma et al., 2023), generally employ fewer than 0.5B parameters, stark contrast to large-scale ASR models like Whisper (Radford et al., 2023) or Seamless (Barrault et al., 2023) which boasts up to 1.6B and 2.3B parameters, re- spectively. This disparity is not merely matter size but reflects a fundamental challenge in AVSR scalability: in- creasing the model size often disproportionately enhances audio semantic understanding without similarly improving visual processing capabilities (Dai et al., 2024; Kim et al., 2024). Moreover, the computational complexity and latency of larger models pose challenges for efficient deployment, especially scenarios where AVSR users often require
1. Introduction
Audio-visual speech recognition (AVSR) (Noda et al., 2015; Afouras et al., 2018a; Ma et al., 2021b; Shi et al., 2022a; Hsu & Shi, 2022; Hu et al., 2023b) has emerged as piv- otal technology in enhancing the robustness and accuracy of speech recognition systems, particularly noisy envi- ronments. By integrating auditory and visual modalities, AVSR leverages the complementary information from both speech signals and lip movements (Makino al., 2019; Ren et al., 2021; Chen et al., 2023), offering significant advan-
'KAIST AI, Republic of Korea "School of Electrical Engineer- ing, KAIST, Republic of Korea. Correspondence Se- Young Yun <yunscyoung kaist.ac.kr.
Proceedings of the nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

MoHAVE: Mixture of Hicrarchical Visual Experts for Robust Speech Recognition
rapid processing and low latency. These factors make the integration of larger, more computationally intensive models impractical for many real-world applications.
2. Related Work
2.1. Robustness of Audio-Visual Speech Recognition
The robustness of AVSR systems has significantly advanced by integrating auditory and visual cues to improve speech recieiltioneespeciallly in noisy environments. Conventional ASR methods have evolved from relying solely audio signals (Schneider et al., 2019; Gulati et al., 2020; Baevski et al., 2020; Hsu et al., 2021; Chen et al., 2022; Chiu et al., 2022; Radford et al., 2023) to incorporating visual data from speech videos (Makino et al., 2019). The multimodal AVSR methods (Pan et al., 2022: Shi et al., 2022a; Seo et al., 2023; Ma al., 2023) have enhanced robustness under audio-corrupted conditions, leveraging visual details like speaker's 's face or lip movements well as acoustic features of speech. These advancements have been driven by vari- ous approaches, including end-to-end learning frameworks (Dupont & Luettin, 2000; Ma et al., 2021b; Hong et al., 2022: Burchi & Timofte, 2023) and self-supervised pretrain- ing (Ma et al., 2021a; Qu et al., 2022: Sco et al., 2023; Zhu et al., 2023; Kim al., 2025), which focus on audio-visual aligmment and the joint training of modalities (Zhang et al., 2023; Lian et al., 2023; Haliassos et al., 2023; 2024).
To address the scalability challenges in AVSR systems, (MoE) architec- ture (Shazeer et al., 2017; Fedus et al., 2022) that activates only subset of parameters (i.e. experts) for efficient scaling of model capacity. Furthermore, recognizing the inherent bias in AVSR systems toward the audio modality, we find it essential to harness the full potential of both audio and video data. One approach is expert group specialization, also known as hard routing (Zhu et al., 2022; Li et al., 2023; Lee et al., 2025), which assigns specific roles to expert groups and manually activates them based on input types. While effective, this fixed activation strategy lacks adaptability, making it sub-optimal for AVSR where noise conditions and modality reliability vary. A more flexible routing mech- anism is needed to dynamically utilize expert groups.
we leverage a sparse
We thus propose a novel MoE framework, MoHAVE? (Mixture of Hicrarchical Audio- AusiolVisual Experts), which employs a hierarchical gating mechanism with two-layer routers. MoHAVE introduces an inter-modal router that makes decision utilizing audio and visual expert groups (64.1). This dynamic routing adapts to input characteristics, specifically trained by our novel load biasing loss ($4.2). MoHAVE achieves state-of-the-art performance ($5.3) on the noisy LRS3 benchmark (Afouras et al., 2018b) and in multilingual tasks (Anwar et al., 2023). Empirical analysis shows that MoHAVE adaptively adjusts token distribution based on input context ($5.4), e-g., visual expert group being more actively utilized under high auditory noise.
Furthermore, recent advancements in AVSR highlight the importance of visual understanding alongside audio (Dai et al., 2024; Kim et al., 2024). While initial research pri- marily targeted audio disturbances (Shi et al., 2022b; Hu et al., 2023c;d; Chen al., 2023), latest studies increas- ingly focus on the visual robustness to address challenges such as real-world audio-visual corruptions (Hong et al., 2023; Wang et al., 2024; Kim et al., 2025) or modality asyn- chrony (Zhang et al., 2024; Fu et al., 2024; Li et al., 2024a). These efforts remark a shift towards a more balanced use of audio and visual modalities. Yet, there has been lim- ited exploration in scaling model capacity or introducing innovative architectural designs, leaving room for further de- velopments in AVSR system that can meticulously balance audio and visual modalities.
As shown in Figure 1, MoHAVE capitalizes on its increased model capacity to significantly enhance performance while maintaining efficiency. Unlike simple MoE implementa- tions, which yield only modest gains over Transformers, our innovative expert group strategy unlocks substantial advancements in adaptability and robustness. Our main con- tributions are outlined as follows:
MoE architecture for scaling AVSR systems: We present MoHAVE, a framework that integrates a sparse MoE architecture to efficiently scale AVSR model capac- ity and optimally process audio and visual data. Hierarchical gating for adaptive expert utilization: MoHAVE features novel hierarchical gating mecha- nism that dynamically adjusts the usage of audio and visual expert groups based on input context, significantly improving adaptability and robustness.
2.2. MoE for Language, Vision, and Speech Models
Mfxture-of-Experts (MoE), first introduced by Jacobs et al. (1991), is a hybrid structure incorporating multiple sub- models, iee, experts, within a unified framework. The essence of sparsely-gated MoE (Shazeer et al., 2017; Lep- ikhin et al., 2021; Dai et al., 2022) lies in its routing mech- anism where a learned router activates only subset of experts for processing each token, significantly enhancing computational efficiency. Initially applied within LLMs us- ing Transformer blocks, this structure has enabled unprece- dented scalability (Fedus et al., 2022; Zoph et al., 2022; Jiang et al., 2024; Guo et al., 2025) and has been progres- sively adopted in multimodal models, especially in large vision- Lainuge models (LVLMs) (Mustafa al., 2022: Lin
Robust AVSR performance: Our model showcases sub- stantial improvements across robust AVSR benchmarks including multilingual tasks, delivering high accuracy while maintaining computational overhead.
MooHAVE's is pronounced as Mojave Desert.
2

---

MoHAAE:: Mixture of Hierarchical Audio-Visual Experts for Robust Speech Recognition
al., 2024; McKinzie et al., 2025). Among these multi- modal MoEs, Zhu et al. (2022); Shen al. (2023); Li al. (2023; 2024b) and Lee et al. (2025) share the similar phi- losophy to ours, assigning specific roles to each expert and decoupling them based on distinct modalities or tasks. These models design an expert to focus on specialized segments of input and enhance the targeted processing.
love
MoE
textge1
Decoder
X L
Add Norm
Transformer Block
MoE
Transformer Block
Exp
Exp
Exp E
f
Beyond its applications in LLMs and LVLMs, the MoE framework has also been applied for speech processing (You et al., 2021; 2022; Hu et al., 2023a; Wang et al., 2023), where it has shown remarkable effectiveness in multilingual and code-switching ASR tasks. In addition, MoE has been employed in oudio-visual models (Cheng et al., 2024; Wu et al., 2024), although they primarily focus on general video processing and not specifically on human speech videos. These approaches leverage MoE to model interactions be- tween audio and visual tokens without directly processing multimodal tokens. Our research advances the application of the MoE framework to AVSR by designing a modality- aware hierarchical gating mechanism, which categorizes ex- perts into audio and visual groups and effectively dispatches multimodal tokens each expert group. This tailored design enhances the adaptability in managing audio-visual speech inputs, which often vary in complexity due to diverse noise conditions.
<bos>
love
Router
W, Xx
Add Norm
Multimodal Encoder
Cross-Attention
Audio
Video
text:
Figure 2. Overview of sparsely-gated MoE for AVSR. A select subset of experts are activated for each token representation (rr))
where is the normalization of top-k probabilities. Note that each expert follows the same structure as feed-forward network (FFN) in a Transformer block. Figure 2 presents the overall MoE architecture and its token routing.
Load balancing. To mitigate the load imbalance issue commonly observed in the top-k expert selection strategy, load balancing loss has been implemented to encourage the balanced token load across all experts. Specifically, we use differentiable load balancing loss (Fedus et al., 2022):
3. Preliminaries
3.1. Sparsely-gated MoE
E Σ S.
In AVSR systems, the multimodal encoder processes a sequence of audio = [ai, ...... and video v [U1, ... data into combined audio-visual embeddings Enc(a, v). These embeddings are utilized by the decoder to predict subsequent text tokens, where the predicted to- ken is given by 1 = Dec(Enc(a, v), text.). Within the Transformer layer, x, is the intermediate representation of the token text, derived by cross-attending to the combined audio-visual embeddings from and (see Figure 2).
LB [3]
(3)
where fa denotes the frequency of expert i being selected as top-1, averaged over all tokens within a batch B,
1 Σ fo = T
Marg = =i}
(4)
FEB
and P; is the average assigned probability for expert i,
The integration of sparsely-gated MoE framework (Shazeer et al., 2017: Lepikhin et al., 2021) leverages experts 3 = {EE}} to scale model capacity. Each token representa- tion routed to a selected subset of these experts through a learned gating mechanism. Specifically, the routing func- tion [xx)) Wr, assigns weights for each token, and the weight for expert i is computed using a softmax function:
1 T
P
Σ pr(2)
(5)
with T' representing the total number of tokens.
An additional router z-loss (Zoph et al., 2022) is employed to stabilize the routing mechanism:
exp(hu(x))
(x( = Σ 1 exp(h)(x))
2
L =
Σ ZEB
log
exp(hi(a))
(6)
T
and the output is the aggregated result of computations from the top-k selected experts:
This sparse MoE structure ensures that token processing is efficiently managed across multiple experts, utilizing lower compute relative to its expansive capacity.
Σ (x),
y
xEl))((E)
3

---

MoHAAV:: Mixture of Hicrarchical Audio-Visual Experts for Robust Speech Recognition
(a) MoE Routing
(b) Hard Routing
Top-1
3
Load Balancing
Top-1 Exp*
1
3
Load Balancing
5
7
Top-1 Exp"
Top-2 Exp
6
5
Top-2 Exp
3
4
2
7
5
Top-2 Exp'
Exp 1
Exp 8
Audio Expert
E1
E2
E3
E4
E5
E6
E7
E8
Visual Expert
Router
Group
Router A
Router V
Group
X-Attn @ Norm
X-Attn ① Norm
X-Attn ① Norm
shared
Audio-visual
Audio embeddings
Video embeddings
embeddings
Text embeddings
Text embeddings
(c) MoHAVE: Hierarchically-gene Adaptive Routing
Top-1 Exp
2
4
(ii)
2
1
i
Load Blasing
5
6
5
Top-1 Exp"
Top-1 Exp" 7
8
Biasing
5
8
2
1
3 4
Top-1 Exp"
Audio Expert Group
E2
E8
Visual Expert
Router A
Router V
Group
Inter-modal Router
(i) Audio
X-Attn Norm
(ii) Ausio-visual
(iii) Video
Figure MoE-basee routing strategies for AVSR. (a) A conventional MoE approach where Icarned router selects the top-2 experts for each token, enforcing the balaneed expert load. (b) Experts explicitly divided into audio and visual groups, with manual activation based on the input modality. (c) MoHAVE introduces an inter-modal router that can dynamically assign weights to modality-specifee expert groups, followed intra-modal routers that select the top-1 expert within each group. The inter-modal router trained by the load biasing loss that guides the expert group specialization.
3.2. Expert Group Specialization
V within batch B, consisting of audio-only video-only sequences, respectively. A token representation x, E A in- dicates that the cross-attention module processes the input text; with Enc(a. a, -0-wweree the visual component is zeroed out-and and vice versa for x; E For these, we utilize two distinct intra-modal routing networks, W.A. and W.Y
To enhance expert management within the AVSR system, a hard routing technique can be used for expert group spe- cialization. This approach is inspired by several practices in visual- -sngua-a MoE models (Zhu et al., 2022; Li et al., 2023; Shen et al., 2023; Lee et al., 2025) where the role of experts strictly defined by the input modality, eliminating the need for trained router.
h AA(x) W.A for E A, 1. (x) W x for x V.
Hard routing. Our hard routing enforces modality- specific activation of expert groups: audio data activate only audio experts, and video data activate only visual experts. This segregation encourages the independent development of specialized expert groups. As suggested in V/T-MoE (Shen et al., 2023), once the group is activated, we use intra-modal router for the modality experts.
These routers calculate the weights p(A.V as in Eq. (I) within their respective expert group, either {E^} for audio or (EV) for visual. The output for each token is then
Eupp(( (x) ifrEA. Eupk(Err DY (x) if
(8)
Figure 3(b) visualizes the hard routing mechanism with au- dio and visual expert groups. During training, audio or video sequence randomly dropped, leading to subsets A and
For audio-visual sequences, outputs from both groups are averaged, with the top-(k/2) experts from each group con- tributing to ensure balanced processing.

---

MoHIAVE: Mixture of Hicrarchical Visuol Experts for Robust Speech Recognition
4. MoHAVE: Mixture of Hierarchical Audio-Visual Experts
Table 1. Computational cost of AV-HuBERT and MoHAVE in
FLOPs, along with their sizes (activated and total parameters).
Despite the benefits of hard routing in specializing expert groups according to decoupled input modality, it lacks the flexibility to autonomously determine the group utilization. In practice, the optimal balance between audio and visual groups varies depending on ambient conditions such as noise type and intensity (more detailed in Figure 4(b)). To address this limitation and enhance the model's adaptability, we introduce an adaptive routing mechanism with hierar- chical gating (Jordan & Jacobs, 1994), providing more dynamic approach to manage multimodal inputs.
Model
Params
Compute (GFLOP%/ seq)
Compute/FFF FFN (MFL.PP//se) seq)
Act, & Total
AN-HOBERT-BAL BASE
161M & 161M
12.1
472
MoHAVE- BASE
189M & 359M
14.8
921
AV-HUBERT-LARGE
477M & 477M
32.2
839
MoHAVE-LARGE
553M & 1.0B
39.3
1,630
4.2. Group-level Load Biasing Loss
To autonomously manage the expert group loads without manual (de-)activation as hard routing, we introduce bias- ing loss that directs the load towards a certain group. This load biasing loss encourages the inter-modal router to as- sign higher weights to E experts for audio sequences and to EV experts for video sequences. For audio sequenees within a sub-batch A, the frequency and average probability of selecting the i-th group is calculated as follows:
Our hierarchical model, MoHAVE, features a two-layer rout- ing mechanism: inter-modal and intra-modal routers, where the modal router learns to assign appropriate weights to each modality-specinn group. Figure 3(c) presents the overview of MoHAVE's routing strategy.
4.1. Hierarchical Gating Structure
gfa
1
Σ
1
(arg max g(x) = i),
(12)
LA aEA
The inter-modal router orchestrates the initial token distri- butions across expert groups. It generates logits through m(xx) V, x and determines the dispatch weights for group with (xx) sofimax(u(z)), This router dynami- cally selects the top-m expert groups, and within those, the intra-modal routers select the top-k experts, thus involving m x k experts in total. For practical efficiency, we set for each group and modify the intra-modal router's proba- bility distribution to Kronecker delta, Pij - arrrmmxx(p:)) The output from this layer integrates these selections:
Q = 1 TAI
Σ k(z). TEA
(13)
Similar calculations for of¹ and QY are made for video sequences X' E V. We designate the first group audio experts and the second group as video experts, then the load biasing loss is defined as:
Ls = Lg + L. = (1 - o¹ + oo Q.) (14)
y= Σ (x) Σ Des (x)
(Elupm(G)
jElpk(EE)
Note that L and σI are only computed over x E A and x E V. respectively.
Σ (x)
(x), where jrg max(p:) (10)
(G)
For sequences containing both audio and video, we exclude them from the load biasing loss calculation but incorporate them into the load balancing. Although these tokens are uni- formly dispatched on average, the inter-modal router finds the optimal strategy for each token based on its character- istics. Empirically, we find that MoHAVE learns to assign greater weight to the visual expert group for audio-visual in- puts under high auditory noise, and to the audio expert group for less noisy inputs (see $5.4 for details), demonstrating the model's 's adaptability under various noisy conditions.
where q is the normalization of 9 across top-m probabilities, G is the number of expert groups, and (x)(z) denotes the output from the j-th expert in the i-th group.
Focusing on audio-visual applications, we designate two ex- pert groups: audio and visual. Each token x, regardless of its modality, is processed by the intra-modal routing networks of both groups, i.e., (,)) hY (a)) = [WA, W x. The frequencies f(4.) and probabilities p(A'V) for selecting experts are computed in the same manner as Eq. (5) for all E B. Thus, the load balancing loss can be computed for both groups:
The overall loss function, combining the cross-entropy (CE) for token prediction, is formulated as:
J - Σ
La - |BA
EyY-Pj. PY (11)
Liot = LeF + cn + + Ls + ezbz.
(15)
Here, cp and ez are set to le-2 and 1e-3, respectively, in line with (Fedus et al., 2022; Zoph et al., 2022), and es is also set at le-2.
where f/ and fy denote the frequencies of token assign- ments to audio and visual experts, respectively.
5

---

MoHAAV:: Mixture of Hicrarchical Audio-Visual Experts for Robust Speech Recognition
5. Experiments and Results
description, refer to Appendix A.I.
5.1. Implementation Details
5.2. Computation Cost
Datasets. For the robust AVSR benchmark, we utilize the LRS3 dataset (Afouras et al., 20). which consists of 433 hours of audio-visual speech from 5,000+ speakers. Follow- ing the experimental setup of Shi et al. (2022b), we extract audio noise samples from the MUSAN (Snyder et al., 2015) dataset, targeting different noise types such as babble, mu- sic, and natural noises, along with speech noise from LRS3.
Table summarizes the parameter sizes and computational costs of MoHAVE. To assess actual computation costs when processing inputs, we measure floating point operations per second (FLOPs) using an audio-visual sequence of 500 frames with 50 text tokens. The entire compute cost for AV-HuBERT-BASE and MoHAVE-BASE are 12.1 GFLOPs and 14.8 GFLOPs, respectively, while for LARGE, the com- putes are 32.2 GFLOPs and 39.3 GFLOPs. This indicates slight increase in FLOPs for MoHAVE, primarily due to the MoF layers replacing FFNs in the decoder. Although the MoE layers require roughly twice the computation cost of standard FFNs (refer to Compute/ / FFN), the encoder and attention layers in the decoder remain unchanged. Conse- quently, the overall computational cost remains comparable to AV-HuBERT counterparts, ensuring scalability without significant computation overhead.
These noises are randomly augmented into the audio data, corrupting 25% of the training set with a signal-to-note ra- tio (SNR) sampled from (O, 5). We measure performance using the word error rate (WER), primarily under noisy con- ditions with SNRs of 10, -5, O, 5, 10) dB, specifically N-WER (Kim et al., 2024) which highlights the significance of visual cues in noise-corrupted environments.
For multilingual evaluations, the MuAViC dataset (An- war et al., 2023) used, featuring 1,200 hours of audio- visual content from 8,000+ speakers across languages, sourced from LRS3-TED (Afouras et al., 2018b) and m'T- EDx (Elizabeth et al., 2021). We use 8 languages (excluding English) for multilingual AVSR and 6 languages for X-to- English audio-visual speech-to-text translation (AVS2TT) tasks. We assess the models using WER for transcription and the BLEU score (Papineni al., 2002) for translation.
5.3. Robust AVSR Benchmark Results
Table presents MoHAVE's robust performance on the AVSR benchmark under diverse noisy conditions, demon- strating exceptional robustness across different noise types and SNR levels: N-WER of 5.8% for BASE and 4.5% for LARGE. This substantiates the model's potential for effectively scaling AVSR systems without incurring signifi- cant computational costs. The results also reveal that simple MoE implementations (AV-MoE in Table 2), despite their larger capacity, fail to achieve remarkable gains. Instead, the key improvement stems from leveraging expert group specialization, as evidenced by the effectiveness of hard routing. By splitting experts into audio and visual groups, MoE is enabled with more targeted and effective processing of multimodal inputs, leading to substantial performance enhancements. Without our load biasing loss, MoHAVE loses its group specialization capability, comparable to the performance of simple AV-MoEs.
MoHAVE model description. Our MoHAVE framework is developed in two configurations: BASE and LARGE. The BASE model consists of 12 Transformer (Vaswani et al., 2017) encoder layers and 6 decoder layers, while the LARGE model incorporates 24 encoder layers and 9 decoder lay- ers. Both models' audio-visual encoders are derived from the AV-HUBERT-BASS/LLLAE noise-augmented corpus of LRS3 (Afouras et al., 2018b) + VoxCeleb2 (Chung et al., 2018). Our MoE implementation activates top-2 out of 8 experts in every FFN layer within the decoder (Jiang et al., 2024), while the hierarchical ar- chitecture engages the top-1 expert from each audio and visual group. To facilitate the expert group specialization, load biasing is used with audio video randomly dropped in 25% probability.
models, pretrained on
Building upon this expert group strategy, MoHAVE en- hances its adaptability through dynamically determining the usage of each group. This adaptive routing approach allows the model to flexibly adjust to varying audio-visual scenarios, contributing to consistent gains in robustness across the benchmark, as detailed in Table 2. An in-depth analysis of this hierarchical gating approach and its impact token dispatching is discussed in $5.4, underscoring its critical role in ad advancing MoHAVE's capabilities in various AVSR environments.
As summarized in Table 1, the BASE model of MoHAVE holds 359M parameters, and the LARGE configuration con- tains IB. Specifically, for MoHAVE-BASE the encoder accounts for 103M parameters, and the decoder 256M, whereas in LARGE, the encoder holds 325M, and the de- coder 681M Despite its larger model capacity, due to the sparse activation of these parameters, only about half are active during token processing, amounting to 189M for BASE and 553M for LARGE model. This setup ensures com- putational efficiency which is comparable to the smaller AV-HuBERT counterparts. For more details on the model
Comparison with state-of-the-art AVSR methods. Ta- ble 3 shows how our MoHAVE decoder, when integrated with a range of audio-visual encoders, consistently improves
6

---

MoHAAE:: Mixture of Hicrarchical Audio-Visual Experts for Robust Speech Recognition
Table Audio-visual speech recognition performance (WER %) on the IRR3 dataset (Afouras al., 201 The numher of parameters for each model includes both encoder and decoder. For evaluation, augmented noise is sampled from the MUSAN dataset (Snyder al.. 2015), while speech noise is sampled from the held-out of LRS3. N-WER (Kim et al., 2024) averages results across four noise types and five signal-to-noise ratios, and C-WER indicates the result with clean audio signal.
Specialized Groups
Activated Params
Total Params
SNR = {-10, -5, 0, 5, 10)
Model
Experts
N-WER
C-WER
babble
speech
music
natural
AV-HUBERT-BASE
161M
161M
10.8
4.9
5.6
5.1
6.6
2.1
AV-MoF-BASE.
189M
246M
10.5
4.5
5.3
5.0
6.3
2,0
AV-MoB-BASE
189M
359M
10.5
4.5
5.3
4,9
6.3
2.1
(+) Hard Routing
189M
359M
9.9
4.4
4.6
5.9
MoHAVE- BASE (ours)
(x)(HH)
189M
359M
9.6
4.2
4.5
5.8
(-) Load Biasing
2x4 (H)
189M
359M
10.3
4.4
4.9
6.2
AV-HuBEKT-LAARE
477M
477M
9.7
4.6
4.4
4.1
5.7
1.4
AV-MoE-LARGE
X
553M
704M
10.1
3.8
4.6
4.3
5.7
1.8
AV-MoE-LARGE
X
553M
1.0B
10.1
3.8
4.6
4.2
5.7
1.8
(+) Hard Routing
V
553M
1.0B
8.3
3.3
4.0
3.7
4.8
1.5
MoHAVE-LARGE (ours)
(H)
553M
1.0B
7.7
3.0
3.7
3.4
4.5
1.5
(-) Load Biasing
(H)
X
553M
1.0B
9.9
3.6
4.6
4.3
5.6
1.7
performance compared to existing state-of-the-arr methods. While BRAVEn (Haliassos et al., 2024) typically struggles in noisy multimodal scenarios due to its original design focused on handling unimodal tasks-MoHAl boosts its accuracy.
Table 3. Performance comparison on the noisy LRS3 benchmark with prior works. We present average N N-WER, with babble (B), speech (S), music (M), and natural (N) noise types. BRAVEn implemented with separate ASR and VSR encoders, combined and jointly trained with decoder.
Other recent approaches have advanced by utilizing the noise-augmented AVSR encoder (Shi et al., 2022a), such as additionally learning temporal dynamics with cross-moda. attention modules (CMA) (Kim et al., 2024). MIR-GAN (Hu et al., 2023b), Uni (Hu et al., 2023c), and CMA are all built upon AV-HuBERT-LAAGEE matching the architecture and activated parameter count with the dense (non-MoE) baseline in Table When paired with an AV-HuBERT en- coder and trained through the CMA's self-supervised learn- ing, MoHAVE achieves remarkable performance: N-WER of 4.2%.
Method
M
Avg
Joint ASR and VSR encoders
BRAVEn (Haliassos et al., 2024)
13.5
15.7
7.4
11.0
+ MoHAVE (ours)
12.3
13.4
6.8
9.8
Noise-augmented AVSR encoder
AV-HuBERT (Shi et al., 2022h)
9.7
4.6
4.2
5.7
+ MIR-GAN (Hu et al., 2023b)
5.6
+ UniVPM (Hu et al., 2023c)
9.3
4.1
3.6
5.2
+ CMA (Kim et al., 2024)
8.1
2.9
3.7
4.6
+ MooIAVE (ours)
7.7
3.0
3.6
4.5
+ CMA MoHAVE (ours)
7.3
2.8
3.3
tions. Since hard routing relies on manually (de-)activating the audio and visual groups, for audio-visual inputs, we assign fixed equal weight (pA,p) = 0.5) to both groups. However, this equal weighting may not always be optimal in varying environments, such noise type intensity.
5.4. Expert and Group Load Analysis
MoHAVE's expert load distribution. Figure 4(a) illus- trates the expert load distribution of MoHAVE according to imput types: visial, audio-only. and video-only se- quences. For audio-visual inputs, all experts from both the audio and visual groups are selected at similar frequencies, with some layer-dependent variations. In contrast, when processing audio-only sequences, the model predominantly activates the audio expert group, while for video-only se- quences, the visual expert group is mainly utilized. This distribution validates the effectiveness of our load biasing loss in guiding the inter-modal router to assign appropriate weights based on input modality.
As shown in Figure 4(b), increasing reliance on the audio group under babble noise degrades performance, with an optimal weight for the audio group heing 0.3. Unlike babble noise, which confuses the model with multiple overlapping speech signals, natural noise is more distinet from speech, leading higher reliance on the audio group (p > 0.5) preferable. These results indicate that an ideal routing strat- egy for audio-visual data should be dynamically adjusted.
Figure 5 further illustrates MoHAVE's group load distri- bution across different noise levels. The model adaptively adjusts its reliance between the audio and visual expert groups-unde under high noise conditions (low SNRs), it shifts
Expert group utilization in noisy AVSR.
To analyze the effectiveness of hierarchical gating in AVSR, we first exam- ine the limitations of hard routing ($3.2) under noisy condi-

---

MoHAAV:: Mixture of Hicrarchical Audio-Visual Experts for Robust Speech Recognition
Expert Expert
Expert 2 Expert
Expert 3 Expert
Expert Expert
Audio Expert Group
babble noise
Visual Expert Group
15.0
Audio-Visual Sequence
Audio-only Sequence
Video-only Sequenie
VIES SWR
WER SM0
100%
1979
1091
SNR 9-0
Percembage
Percenage
Percentag
natural noise
50%
309
50%
6.3
6.2
WER 500
6.1
6.0
5.9
0.1
0.3
0.5
0.7
0.9
Layer
Layat
Luget
(sudio group)
Figure 4. (a) Expert load distribution in MoHAVE according to input modalities, with expert selection frequencies weighted by the inter-modal router's output probability. (b) Performance of the hard routing strategy under different weight assignments audio expert group. The visual expert group is weighted by p p²
Layer
Layer
Layer
Layer
Layer 2
Layer
Layer
Layer
1.0%
100/9
100%
100%
1095-
1635
79%
75%
75%
37%
99%
25%
20%
25%
255
25%
30
0%
0%
30
800
75
95
SNR, babble
SNR, babble
SNR, babo
SNR, babble
SNR, natural
SNR. natura
SNR, natural
SNR, natura
Figure 5. Expert load distribution in MoHLAVE for the audio group (solid bars) and visual group (dashed bars) noisy audio-visual sequences under babble (left) and natural (right) noise. Full layer-wise results are provided in Appendix C.1.
more tokens to the visual group, while in cleaner condi- tions (high SNRs), the audio group is more actively utilized. This behavior also adjusts to noise types, as observed with babble and natural noise, demonstrating the MoHAVE's adaptability and robustness.
achieves superior performance in both AVSR and AVS2TT. Whisper (Radford et al., 2023), a leading multilingual ASR model, is known to perform poorly in noisy setup due to its lack visual understanding for robustness. While multilin- gual AV-HuBERT (Anwar et al., 2023) underperforms the state-of-the-ar models like XLAVS-R (Han al., 2024), have re- re-implemented it using the pretrained model from signif- Choi et al. (2024), which has been pretrained icantly larger dataset including 7,000 hours of speech in 100+ languages. This model outperforms (38.9% average WER) or remains competitive (18.5% average BLEU) with much larger XLAVS-R 2B. When integrated with this ver- sion, MoHAVE further improves performance, achieving 37.4% % average WER and 19.5% average BLEU, setting new benchmarks in almost every language being evaluated.
5.5. Multilingual Audio-Visual Speech Tasks
MoEs have demonstrated effectiveness in multilingual speech tasks (Hu al., 2023a; Wang et al., 2023), as MoE. capable of enabling more diverse routing paths for dif- ferent language tokens. To evaluate MoHAVE's multilin- gual capabilities, we train multilingual model and assess its performance on the MuAViC benchmark (Anwar et al., 2023), evaluating separately for each language. Following Han et al. (2024), we introduce multilingual babble noise SNR dB to 50% of the input samples during training, where the noise clips are sampled from the MuAViC train apply beam search with beam size of 5 and normalize text by punctuation removal and lower- casing before calculating WER. For AVS2TT evaluation, we use SacreBLEU (Post, 2018) with its built-in 13a tokenizer. To simulate noisy test conditions, we inject babble noise sampled from the MuAViC test set.
6. Conclusion
set. For inference,
In this study, we propose MoHAVE, hierarchical MoE framework for AVSR, designed to enhance scalability and robustness. By training an inter-modal router that dynami- cally assigns weights audio and visual expert groups, Mo- HAVE enables adaptive group selection based on input context. Evaluations on robust AVSR benchmarks demon- strate its state-of -fte----tthhttn performance, with superior noise
Table 4 summarizes the results, where MoHAVE-LARGE
8

---

MoHAAV:: Mixture of Hicrarchical Audio-Visual Experts for Robust Speech Recognition
Table Multilingual audio-visual speech task performance, with non-English speech recognition (WER) and X-Fn speech-to-text translation (BLEU score). in noisy environment with multilingual babble noise (SNR = 0). 'Results obtained from Han et al. (2024). tte-implemented using pretrained model from Choi (2024).
Source
Model
Pretrain data
Ar
De
Es
Avg
Speech Recognition, Test WER
Whisper large-v2 (Radford ct al., 2023)
680k hrs, 100+ langs
197.9
244.4
113.3
116.3
172.3
172.4
223.6
126.2
170.8
u- U-HuBERT' (Hsu & Shi, 2022)
1.7k hrs, English
102.3
73.2
69.7
43.7
43.2
48.5
47.6
67.0
61.9
mAV-HuBERT'S (Anwar et al., 2023)
1.7k hrs. English
82.2
66.9
62.2
40.7
39.0
44.3
43.1
43,1
52.7
XT.S- 300M (Babu et al., 2022)
1.2k hrs, 9 langs
97.3
69.8
74.8
47.6
37.1
47.9
54.4
59.8
61.1
XLAVS-R SSSRSRR. 300M (Han et al., 2024)
8.3k hrs, 100+ langs
91.9
53.5
49.6
28.8
29.3
32.2
32.5
46.1
45.5
XLAVS-R 2B (Han et al., 2024)
1.2k hrs, 9 langs
93.5
58.5
38.6
23,9
23.5
24,6
26.1
41.0
41.2
mAV-HuBERT
7.0k hrs. 100+ langs
88.7
51.3
37.2
20.7
22.6
24.2
23.8
42.4
38.9
mAV-HuBERT's + Hard Routing
7.0k hrs, 100+ langs
93.4
49.3
35.7
20.3
23.6
23.4
24.1
44.7
39.3
MoHAVE-LARGG (ours)
7.0 hrs, 100+ langs
92.9
47.3
35.3
18.7
21.6
21,9
40.6
37.4
X-En Speech-to-Test Translation, Test BLEU 1
Whisper large-12 (Radford et al., 2023)
680K hrs, 100+ langs
0.1
0.4
0.7
0.1
0.1
0.2
0.3
HAV-HUBRRT (Anwar et al., 2023)
1.7k hrg, English
4.2
12.8
15.0
12.5
14.8
4.6
10.7
XLAVS-R 300M (Han et al., 2024)
8.3k hrs. 100+ langs
13.2
17.4
23.8
18.7
21.8
9.4
17.4
XLAVS-R 2B (Han et al., 2024)
1.2k hrs, langs
15.7
19.2
20.1
22.3
10.4
18.7
mAV-HuBERT*
7.0k hrs, 100+ langs
8.9
21.5
26.5
21.2
24.2
8.8
18.5
mAV-HuBERT + Hard Routing
7.0k hrs, 100+ langs
6.7
19.9
24.7
19.6
23.0
7.2
16.8
MoHAVE-LARGE (ours)
7.0k hrs, 100+ langs
11.4
22.3
27.1
22.1
25.1
9.2
19.5
resilience, further supported by flexible expert load distri- butions across diverse noisy conditions. This work estab- lishes an adaptive modality-aware MoE paradigm, advane- ing larger-scale multimodal speech recognition systems.
use and fairness in model deployment remain critical in AVSR systems for ensuring equitable applications.
References
Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint 0rX7722303.88774, 2023.
Acknowledgements
This work was supported by Institute of Information & communications Technology Planning & Evaluation (UTPP) grant funded by the Korea government (MSIT) [No.2022-0- 00641, XVoice Multi-Modal Voice Meta Learning, 90%] and (No.2019-0-00075 Artificial Intelligence Graduate School Program (KAIST). 10%].
Afouras, T., Chung, S., Senior, A., Vinyals, O., and Zis- serman, Deep audio visual speech recognition. IEFE transactions on pattern analysis and machine intelligence, 44(12):8717-872 2018a.
Impact Statement
Afouras. T., Chung, J. S., and Zisserman, Lrs3-ted: large-scale dataset for visual speech recognition. arXiv preprint arXiv: (eXXv::809.0049990 00496, 2018b.
MoHAVE addresses the scalability and robustness chal- lenges multimodal speech processing, enabling more efficient and adaptive expert selection. This advancement highlights the potential for further scaling AVSR models in terms of capacity, training, and datasets while extending to more diverse real-world applications without incurring excessive computational costs.
Ainslie, J., Lee-Thorp. J., de Jong, M., Zemlyanskiy, Y.. Lebron, F., and Sanghai, S. Gqa: Training generalized multi-query transformer models from multi-head check- points. In Proceedings of the 2023 Conference Empir- ical Methods in Natural Language Processing, pp. 4895- 4901, 2023.
While our research contributes to the broader field of mul- timodal learning and robust speech recognition, we not foresee significant ethical concerns beyond those inherent to AVSR systems, such as potential linguistic biases in speech models trained on imbalanced datasets or privacy issues when handling human speech video data. Responsible data
Anwar, M., Shi, B., Goswami, V., Hsu, W.-N., Pino, J., and Wang, C. Muavic: A multilingual audio-visual corpus for robust speech recognition and robust speech-to-tens translation. In Interspeech. International Speech Commu- nication Association, 2023.

---

MoHAVE: Mixture of Hierarchical Audio-Visual Experts for Robust Speech Recognition
Babu, A., Wang. C., Tjandra, A., Lakhotia, K., Xu, Q.. Goyal, N., Singh, K., von Platen, P., Saraf, Y., Pino, J., et al. XIs-r: Self-supervised cross-lingual speech repre- sentation learning at scale. In Interspeech. International Speech Communication Association, 2022.
Clark, A., de Las Casas, D., Guy, A., Mensch, A., Paganini, M., Hoffmann, J., Damoc, B., Hechtman, B., Cai, T, Borgeaud, S., et al. Unified scaling laws for routed lan- guage models. In International conference on machine learning, pp. 4057-4086. PMLR, 2022.
Baevski, A., Zhou, Y., Mohamed, A., and Auli, M. wav2vec 2.0: A framework for coff-uueee learning of speech representations. Advances in neural information process- ing systems, 33: 13:1124402 2449-12460, 2020.
Dai, D., Dong. L.., Ma, S., Zheng, B., Sui, 7.., Chang, B., and Wei, F. Stablemoe: Stable routing strategy for mixture of experts. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 7085-7095, 2022.
Barrault, I.., Chung, Y.-A., Meglioli, M. C., Dale, D., Dong. N., Duquenne, P.-A., Elsahar, H., Gong, H., Heffernan, K., Hoffman, J., et al. cotmmnessmm--mmsssie multilin- gual & multimodal machine translation. arXiv preprint arXiv:2308. 1596, 2023.
Dai, Y., Chen, H., Du J., Wang, R., Chen, S., Wang, H., and Lee, C.-H. A study of dropout-induced modality bias on robustness to missing video frames for audio-visual speech recognition. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 27445-27455, 2024.
Burchi, M. and Timofte, R. Audio-visual efficient con- former for robust speech recognition. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp.
Dupont, S. and Luettin, J. Audio-visual speech modeling for continuous speech recognition. IEEE transactions on multimedia, 2(3):141-155 1151,,,,1111111 2000.
2023.
Cappellazzo, U., Kim, M., Chen, H., Ma, P., Petridis, S., Falavigna, D., Brutti, A., and Pantic, M. Large language models are strong audio-visual speech recogmition learn- ers. arXiv preprint arXiv:2409. 12319, 2024.
Elizabeth, S., Matthew, W., Jacob, B., Cattoni, R., Negri, M., Turchi, M., Oard, D. W., and Matt, P. The multilingual tedx corpus for speech recognition and translation. In Proceedings of Interspeech 2021, pp. 3655-3659, 2021.
Chen, C., Hu, Y., Zhang, Q.. Zou, H., Zhu, B., and Chng,, E. S. Leveraging modality-specifft representations for audio-visue speech recognition via reinforcement learn- ing. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 12607-12615, 2023.
Fedus, W., Zoph. B., and Shazeer, N. Switch transformers: Scaling to trillion parameter models with simple and ef- ficient sparsity. Journal of Machine Learning Research, 23(120): 2022.
Chen, S., Wang, C., Chen, Z., Wu, Y., Liu, S., Chen, Z., Li, Kanda, N., Yoshioka, T., Xiao, X., et al. Wavlm: Large- scale self-supervised pre-training for full stack speech processing. IEEE Journal of Selected Topics in Signal 2022.
Fu, D., Cheng, X., Yang. X., Wang, H., Zhao, Z., and Jin, T. Boosting speech recognition robustness to modality- distortion with contrast- contrast-augmented prompts. In ACM Multimedia, 2024.
Processing, 16(6):1505-1
Cheng, Y., Li, Y., He, J., and Feng, R. Mixtures of experts for audio-visual learning. Advances in Neural Information Processing Systems, 2024.
Gulati, A., Qin, J., Chiu, C.-C., Parmar, N., Zhang, Y., Yu, J., Han, W., Wang, S., Zhang, Z., Wu, Y., et al. Con- former: Convolution-aagmentee transformer for speech recognition. In Interspeech. International Speech Com- munication Association, 2020.
Chiu, C.-C., Qin, J., Zhang, Y., Yu, J., and Wu, Y. Self- supervised learning with random-projectin quantizer for speech recognition. In International Conference on PMLR, 2022.
Guo, D., Yang, D.,, Zhang, H.,, Song, J., Zhang, R., Xu, R., Zhu, Q., Ma, S., Wang, P., Bi, X., et al. Deepseek-rl: In- centivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint ar arXiv:2501.1294/ 2025.
Machine Learning, pp.
Choi, J., Park, S. J., Kim, M., and Ro, Y. M. Av2av: Di- rect audio-viees visual speech to audio-visual speech transla- tion with unified audio-visual speech representation. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pp. 27325-27332 2024.
Haliassos, A., Ma, P., Mira, R., Petridis, S., and Pantic, M. Jointly learning visual and auditory speech repre- sentations from raw data. The Eleventh International Conference on Learning Representations, 2023.
Chung. J. S., Nagrani, A., and Zisserman, A. Voxceleb2: Deep speaker recognition. In Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH, volume 2018, pp. 1086- 1090, 2018.
Haliassos, A., Zinonos, A., Mira, R., Petridis, S., and Pan- tic, M. Braven: Improving self-supervised pre-training for visual and auditory speech recognition. In ICASSP
10

---

MoHAVE: Mixture of Hicrarchical Audio-Visual Experts for Robust Speech Recognition
2024-2024 IEEE International Conference on Acous- fics, Speech and Signal Processing (ICASSP): pp. 11431- 11435. IEEE, 2024.
audio-visual speech recognition. In 32nd International Joint Conference on Artificial Intelligence, IJCAI 2023, pp. 5076-5084, International Joint Conferences on Artifi- cial Intelligence Organization, 2023d.
Han, H., Anwar, M., Pino, J., Hsu, W.-N., Carpuat, M., Shi, B., and Wang, C. XLAVS-R: Cross-lingual audio- visual speech representation learning for noise-robust speech perception. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 12896-12911. Association for Computational Linguistics, August 2024.
Jacobs, R. A., Jordan, M. I., Nowlan, S. J., and Hinton, G. E. Adaptive mixtures of local experts. Neural computation, 3(1):79-88) 87, 1991.
Jiang. Q. Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, S., Casas, D. d. I., Hanna, E. B., Bressand, F,, et al. Mixtral of experts. arXiv preprint arXiv:2401.04088 2024.
Hong, J., Kim, M., Yoo, D., and Ro, Y. M. Visual context- driven audio feature enhancement for robust end-to-end visual speech recognition. In Proceedings of the Annual Conference of the International Speech Commu- nication Association, INTERSPEECH, volume 2022, pp. 2842, 2022.
Jordan, M. I. and Jacobs, R. A. Hierarchical mixtures of experts and the em algorithm. Neural computation, 6(2): 181-214, 1994.
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language models. arXiv preprint arXiv:2001. 08361, 2020.
Hong, J., Kim, M., Choi, J., and Ro, Y. M. Watch listen: Robust audio-visual speech recognition with visual cor- ruption modeling and reliability scoring. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pultern Recognition, pp. 18783-18794, 2023.
Kim, S., Jang, K., Bae, S., Kim, H., and Yun, S.-Y. Learning video temporal dynamics with cross-modat attention for robust audio-visual speech recognition. IEEE Spoken Language Technology Workshop (SLT), 2024.
Hsu, W.-N. and Shi, B. u-hubert: Unified mixed-modal speech pretraining and zero-shot transfer to unlabeled modality. Advances in Neural Information Processing Systems, 35:21157-21 17000 2022.
Kim, S., Cho. S., Bac, S., Jang. K., and Yun, S.-Y. Multi- task corrupted prediction for learning robust audio- visual speech representation. In The Thirteenth In- ternational Conference on Learning Representations, 2025. URL htt.hhtttt///ppttttt http:///ppereeeeeeeeeeeee net/ forum? i id=wEgL5keDnB.
Hsu, W.-N., Bolte, B., Tsai, Y.-H. H., Lakhotia, K., Salakhut- dinov, R., and Mohamed, A. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM Transactions Audio, Speech, and Language Processing, 29:3451-3460 2021.
Kingma, D. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.665 2014.
Hu, K., Li, B., Sainath, T. N., Zhang, Y., and Beaufays, F. Mixture-of-ee -x-eee conformer for streaming multilingual asr. In Interspeech. International Speech Communication Association, 2023a.
Lee, B.-K., Park, B., Won Kim, C., and Man Ro, Y. Moai: Mixture of all intelligence for large language and vision models. In European Conference on Computer Vision, pp. 273-302. Springer, 2025.
Hu, Y., Chen, C., Li, R., Zou, H., and Chng, E. S. Mir-gan: Refining frame-level modality-invariant invariant representations with adversarial network for audio-visus speech recog- nition. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 11610-11625, 2023b.
Lepikhin, D., Lee, H., Xu, Y., Chen, D., Firat, O., Huang, Y., Krikun, M., Shazeer, N., and Chen, Z. (GS) /hard Scal- ing giant models with conditional computation and auto- matic sharding. In International Conference on Learning Representations, 2021. URL https: /pwp//vvie..eeeeeeeeeee net / forum? d
Hu, Y., Li, R., Chen, C., Qin, C., Zhu, Q.-.,, and Chng, E. S. Hearing lips in noise: Universal riseme-phonent map- ping and transfer for robust audio-visual speech recog- nition. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1:
Li, J., Li, C., Wu, Y., and Qian, Y. Unified cross-modal attention: Robust audio-visual speech recognition and beyond. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 32: 12:1941- 2024a.
Long Papers), pp. 15213-15232, 2023c.
Li, Y., Hui, B., Yin, Z., Yang. M., Huang, F., and Li, Y. Pace: Unified modal dialogue pre- pre-training with progres- sive and compositional experts. In Proceedings of the 61st
Hu, Y., Li, R., Chen, C., Zou, H., Zhu, Q., and Chng. E. S. Cross-modal global interaction and local alignment for

---

MoHAVE: Mixture of Hicrarchical Audio-Visual Experts for Robust Speech Recognition
Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp- 13402-13416, 2023.
Pan, X., Chen, P., Gong, Y., Zhou, H., Wang, X., and Lin, Z. Leveraging unimodal self-supervised learning for mul- timodal audio-visual speech recognition. In Proceed- ings of the 60th Anmual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 4491-4503, 2022.
Li, Y., Jiang, S., Hu, B., Wang, I.,, Zhong, W., Luo, W., Ma. L., and Zhang, M. Uni-moe: Scaling unified mul- timodal Ilms with mixture of experts. arXiv preprint WrXiv:240S.11211 2024b.
Papineni, K., Roukos, S., Ward, T., and Zhu, W.-J. Bleu: a method for automatic evaluation of machine transla- tion. In Proceedings of the 40th annual meeting of the Association for Computational Linguisties, pp. 311-318, 2002.
Lian, J., Baevski, A., Hsu, W.-N., and Auli, M. Av-data2vec: self-supervised learning of audio-visure speech represen- tations with contextualized target representations. In 2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pp. 1 1-8. IEEE, 2023.
Post, M. A call for clarity in reporting bleu scores, In Pro- ceedings of the Third Conference on Machine Translation: Research Papers, pp. 186-191, 2018.
Lin, B., Tang, Z., Ye, Y., Cui, J., Zhu, B., Jin, P., Zhang, J., Ning, M., and Yuan, L. Moe-llava: Mixture of ex- perts for large vision- ssiiunlanguage models. arXiv preprint arXiv:2401. 15947, 2024.
Qu, L., Weber, C., and Wermter, S. Lipsound2: Self- supervised pre-training for lip-to-speech reconstruction and lip reading. IEEE transactions on neural networks and learning systems, 35(2):2772-27822 2022.
Ma, P., Mira, R., Petridis, S., Schuller, B. W., and Pantic, M. Lira: Learning visual speech representations from audio through supervuupriis.nn In Interspeech. International Speech Communication Association, 2021a.
Radford, A., Kim, J. W., Xu, T., Brockman, G., McLeavey, C., and Sutskever, I. Robust speech recognition via large- scale weak supervision. In International conference on machine learning, pp. 28492-28518. PMLR, 2023.
Ma, P., Petridis, S., and Pantic, M. End-to-end audio- visual speech recognition with conformers. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 7613-7617. IEEE, 2021b.
Ren, S., Du, Y., Lv, Han, G., and He, S. Learning from the master: Distilling cross-mode advanced knowledge for lip reading. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pullern Recognition, pp. 13325- 13333, 2021.
Ma. P., Haliassos, A.. Feraanezz-oopez A., Chen, H., Petridis, S., and Pantic, M. Auto-avsr: Audio-visual speech recognition with automatic labels. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2023.
Schneider, S., Baevski, A., Collobert, R., and Auli, M. wav2vee: Unsupervised pre-training for speech recog- nition. In Interspeech. International Speech Communica- tion Association, 2019.
Makino, T., Liao, H., Assacl, Y., Shillingford, B., Garcia, B., Braga, O., and Siohan, O. Recurrent neural network transducer for audio-visual speech recognition. In 2019 IEEE automatic speech recognition and understanding workshop (ASRU), pp. 905-912. IEEE, 2019.
Sco, P. H.,, Nagrani, A., and Schmid, C. Avformer: Injecting vision into frozen speech models for zero-shot av-asr. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2023, Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q. Hinton, G., and Dean, J. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In International Conference on Learning Representations, 2017. URL https:ttp/// http:///pppeeeeeeeeee/eeeeeeeee. forum?? id-BlckMDqIg.
McKinzie, B., Gan, Z-, Fauconnier, J.-P., Dodge, S., Zhang, B., Dufter, P., Shah, D., Du, X., Peng, F., Belyi, A., et al. Mml: methods, analysis and insights from multimodal Ilm pre-training. European Conference on Computer Vision, pp. 304-323. Springer, 2025.
Mustafa, B., Riquelme, C., Puigcerver, J., Jenatton, R., and Houlsby, N. Multimodal contrastive learning with limoe: the language-image mixture of experts. Advances in Neural Information Processing Systems, 0195564-9576 2022.
Shen, S., Yao, Z., Li, C., Darrell, T., Keutzer, K., and He, Y. Scaling vision-language models with sparse mixture of experts. In Findings of the Association for Computational Linguisties: EMNLP 2023, pp. 11329-11344 2023.
Shi, B., Hsu, W.NN., Lakhotia, K., and Mohamed, A. Learn ing audio-visual speech representation by masked mul- timodal cluster prediction. International Conference on Learning Representations, 2022a.
Noda, K., Yamaguchi, Y., Nakadai, K., Okuno, H. G., and Ogata, Audio-visual speech recognition using deep learning. Applied intelligence, 42:722-737, 2015.
12

---

MoHAVE: Mixture of Hicrarchical Audio-Visual Experts for Robust Speech Recognition
Shi, B., Hsu, W.-N., and Mohamed, A. Robust self- supervised audio-visual speech recognition. In Inter- speech. International Speech Communication Associa- tion, 2022b.
Zhang, J.-X., Wan, G., Ling, Z.-H., Pan, J., Gao, J., and Liu, C. Self-supervised audio-visual speech representa- tions learning by multimodal self-distillation. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2023.
Snyder, D., Chen, G., and Povey, D. Musan: A music, speech, and noise corpus. arXiv preprint arXiv:1510.08484, 2015.
Zhu, J., Zhu, X.,, Wang, W., Wang, X., Li, H., Wang, X., and Dai, J. Uni-perceiie nni-perceiver-men moe: Learning sparse general- ist models with conditional moes. Advances in Neural Information Processing Systems, 35:2664- 2678, 2022.
Thiemann, J., Ito, N., and Vincent, E. The diverse environ- ments multi-channel acoustic noise database (demand): A database of multichannel environmental noise recordings. In Proceedings of Meetings on Acoustics, volume 19. AlP Publishing, 2013.
Zhu, Q. Zhou, L., Zhang, Z., Liu, S., Jiao, B., Zhang, J., Dai, L.,, Jiang, D., Li, J., and Wei, F. Vatlm: Visual- audio-text pre-training with unified masked prediction for speech representation learning. IEEE Transactions on Multimedia, 2023.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, t.., and Polosukhin, I. Atten- tion is all you need. Advances in neural information processing systems, 30, 2017.
Zoph, B., Bello, I., Kumar, S., Du, N., Huang, Y., Dean, J., Shazeer, N., and Fedus, W. St-moe: Designing stable and transferable sparse expert models. arXiv preprint WrXiv:2202.08966, 2022.
Wang, J., Pan, Z., Zhang, M., Tan, R. T., and Li, H. Restor- ing speaking lips from occlusion for audio-visual speech recognition. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 19144-19152, 2024.
Wang, W., Ma, G., Li, Y., and Du, B. Language-routing mixture of experts for multilingual and code-switching speech recognition. In Interspeech. International Speech Communication Association, 2023.
Watanabe, S., Hori, T., Kim, S., Hershey, J. R., and Hayashi, T. Hybrid ctc/attention architecture for end-to-end speech recognition. IEEE Journal of Selected Topics in Signal Processing, 11(8):2240-2253, 2017.
Wu, Y., Peng. Y., Lu, Y., Chang, X., Song, R., and Watanabe, S. Robust audiovisual speech recognition models with mixture-of-experte IEEE Spoken Language Technology Workshop (SLT), 2024.
You, Z., Feng, S., Su, D., and Yu, D. Speechmoe: Scaling to large acoustic models with dynamic routing mixture of experts. In Interspeech. International Speech Communi- cation Association, 2021.
You, Z., Feng, S., Su, D., and Yu, D. Speechmoe2: Mixture- of-experts model with improved routing. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 7217-7221. IEFF, 2022.
Zhang, F., Zhu, Y., Wang, X., Chen, H., Sun, X., and Xu, L. Visual hallucination elevates speech recognition. In Proceedings of the AAAI Conference on Artificial Intelli- gence, volume 38, pp. 19542-19550) 2024.
13

---

MoHAAVE: Mixture of Hicrarchical Audio-Visual Experts for Robust Speech Recognition
MoHAVE: Mixture of Hierarchical Audio- Visual Experts for Robust Speech Recognition
Supplementary Material
A. Experimental Setup
A.1. Model Description
As described $5.1, MoHAVE implemented in two configurations: BASE and LARGE, following the architecture of AV-HuBERT-BASE and AV-HuBERT-LARGE respectively. The encoder maintains the same structure AV-HuBERT, while the decoder incorporates MoE layers by replacing every feed-forward network (FFN) layer with expert modules. Each expert in the MoE layers follows the same bottleneck structure with FFN, consisting of two fully-connected layers with an activation function.
To encourage expert group specialization, we apply load biasing, where either audio or video is randomly dropped with a probability 25%. This allows the model learn modality-aware expert utilization. For expert selection, a router network assigns tokens subset of experts, ensuring efficient computation. The router probabilities are always normalized as sum to when computing the output y- The routing networks V, and W, are parameterized matrices, with dimensions matching the hidden dimension size by the number of experts.
For comparison, we evaluate multiple MoE-based AVSR models: AV-MoE: A simple MoE implementation the same implementation of sparse MoE as (Dai et al., 2022; Jiang et al., 2024).
AV-HuBERT, activating top-2 out of 4 or 8 experts per token. We follow
AV-MoE with Hard Routing: Uses top-2 out of 4 experts for unimodal inputs (audio-only or video-only). For multimodal (audio-visual) inputs, it activates top-1 from each expert group and averages their outputs. This model does not have an explicit router for groups, but within each group, there an mndra- router, i.e., W.A W.V.
Employs top-1 expert per group, with an inter-modal router dynamically adjusting group weight assignments and an intra-modal router uniformly dispatching the tokens to modality-specific experts.
MoHAVE:
A.2. LRS3 Benchmark Experiments
We initialize our model using the pretrained checkpoint from (Shi et al., 2022a) and fine-tune on the LRS3 train set for 120K steps. The encoder remains frozen for the first 90K steps, allowing only the AVSR decoder be trained, after which the entire model is fine-tuned for the remaining 30K steps. Our fine-tuning setup follows the configurations from (Shi et al., 2022b). We employ connectionist temporal classification (CTC) decoding (Watanabe et al., 2017). The Adam optimizer (Kingma, 2014) is used with a learning rate of 5e-4 and a polynomial decay schedule with an initial warmup. Each training step processes 8,000 audio-visual frames, equivalent to 320 seconds of speech data.
negative log-likelihood loss for predicting the next text token, without using
For inference, we beam search with beam size of 50. The AVSR performance is evaluated using word error rate (WER) across five signal-to-noiss ratio (SNR) levels: {-10, -5, O, 5, 10} (lower value means higher noise level). We use audio noise sampled from MUSAN (babble, music, natural) and I.RS3 speech noise, ensuring speaker overlap between training and test sets. Since Table presents SNR-averaged results for each noise type, we provide the full results across all SNR levels in Table 5. MoHAVE--ARGE achieves 5.0% WER on LRS3 with speech noise at SNR=- 10, yielding a 56.1 % relative WER improvement over AV-HUBERT-LARGE 36.7% over AV-MoE-LARGE, and 25.4% over the hard-routing variant. This indicates that MoHAVE correctly predicts over half of the words that AV-HuBERT misses.
A.3. MuAViC Benchmark Experiments
We evaluate MoHAVE on the MuAViC benchmark (Anwar et al., 2023) for multilingual AVSR and to-Englishh AVS2TT tasks, For multilingual AVSR, the dataset includes non-English languages: Arabic (Ar), German (De), Greek (EI), Spanish

---

MoHAAE:: Mixture of Hierarchical Audio-Visual Experts for Robust Speech Recognition
Table Audio-visual speech recognition performance (WER %) on the I.R33 dataset (Afouras al., 201 The numher of parameters for each model includes both encoder and decoder. For evaluation, augmented noise is sampled from the MUSAN dataset (Snyder et al., 2015), while speech noise is sampled from the held-out set LRS3, AV-MoE and MoHAVE use experts.
Babble, SNR (dB)
Speech,
SNR (dB)
Music, SNR (dB)
Natural, SNR (dB)
Model
-10
AVG
-10
-5
0
5
10
AVG
-10
AVG
-10
-5
0
5
10
AVG
AV-IOBBRTT-BASS
27.6
14.1
6.1
38
2.7
10.8
8.6
588
3
3.3
2.8
4.9
12.2
6,4
3.8
2.8
2.5
5.6
10.9
5.4
4.0
288
2.3
5.1
AV-MoE-BASE
26.3
13.7
6.2
3.5
2.7
10.5
8.4
5.3
3.4
2.8
2.3
4.5
11.5
5.3
6.0
3.4
2.8
2.5
4.9
(+) Hard Routing
25.2
13.2
5.6
3.2
2.4
9.9
8.2
5.2
3.4
2.7
2.3
11.0
5.0
5.3
3.2
2.5
2.3
4.6
MoHAVE-BASE
25.3
12.2
5.3
2.9
2.3
9.6
7.9
5.1
3.3
2
2.3
4.2
10.3
5.6
3.3
2.3
2.0
4.7
9.7
5.1
3.2
2.4
2.2
4.5
Lo)d Biasing
26.5
13.6
5.6
3.2
2.4
10.3
8.2
5.2
3.5
2.6
2.4
4.4
11.1
6.4
3.4
2.7
2.6
5.2
10.3
5.6
3.4
2.7
2.4
4.9
AV-HUBERT-LARGE
27.0
12.4
4.7
2.4
1.8
9.7
114
4.6
2.9
2.2
1.8
4.6
10.5
4.9
2.0
1.6
4.4
9.6
4.7
2.5
2.0
1.8
4.1
AV-MoE-LARGH
28.1
12.5
5.0
2.7
2.1
10.1
7.9
4.0
2.9
2.4
2.0
3.8
10.4
5.4
2.9
2.3
1.9
4.6
8.9
4.8
3.1
2.0
2.0
4.2
(+) Hard Routing
22.9
10.8
3.8
2.4
1.
8.3
6.7
3.9
2.4
1.8
1.8
3.3
9.9
4.3
2.3
1.8
1.9
4.0
8.3
4.1
2.4
1.9
1.7
3.7
MoHAVE-LARGE
21.0
9.8
4.1
2.2
1.6
7.7
5.0
3.6
2.3
2.0
1.9
3.0
8.2
4.0
2.6
1.8
1.8
3.7
7.3
3.7
2.6
1.9
1.6
3.4
(-) Load Biasing
27.8
12.4
4.5
2.6
2.0
9.9
6.7
4.0
3.1
2.1
1.9
3.6
10.6
5.3
3.0
2.1
1.8
4.6
9.9
4.9
2.8
2.1
2.0
4.3
(Es), French (Fr), Italian (It), Portuguese (Pt), and Russian (Ru), encompassing approximately 700 hours of training data from 3,700 speakers. For X-En AVS2TT, the dataset covers 6 languages: Greek, Spanish, French, Italian, Portuguese, and Russian, where each sample includes audio-visual speech with corresponding English transcriptions.
A single multilingual model is trained for each task, capable of detecting the source language and generating target transcriptions accordingly. The evaluation is conducted on each language separately, as seen in Table 4. Using the pretrained multilingual AV-HuBERT from (Choi et al., 2024), we fine-tune the model for 120K steps, unfreezing the encoder after 10K steps. Inference is performed with beam size of 5, and the samples with empty ground-truth transcriptions are removed from the evaluation set.
B. Comparison with Previous Works
Empirical comparisons of MoHAVE with AVMoE (Cheng et al., 2024) and EVA (Wu et al., 2024) are unfortunately infeasible due to fundamental differences in target tasks and methods. Both AVMoE and EVA primarily address audio captioning for visual contexts (e.g., narrating sports game scenes), while our work specifically targets typical AVSR tasks, where both audio and visual inputs directly involve human speech.
Moreover, AVMoE (Cheng al., 2024) employs a dense MoE; unlike sparse expert structures commonly used in modern LLMs Transformers, AVMoE's s MoE is actually implemented weighting between unimodal and cross-modal adapters, rather than selecting sparse FFN experts. Specifically, AVMoE uses two entirely separate MoEs for audio encoder and visual encoder, infeasible for processing multimodal tokens. Our approach fundamentally differs by employing sparse multimodal MoE, dynamically routing tokens based on audio-visual inputs.
Closer to our work is EVA (Wu et al., 2024), which simply applics a sparse MoE structure into an audio-visual encoder. Although exact implementation details are unavailable (code and checkpoints unreleased), EVA's structure aligns closely with our basic MoE implementation which we evaluated as AV-MoE in Table 2, except ours is in the decoder. As demonstrated in our study (Table 10), applying MoE at the encoder-level like EVA falls behind multimodal decoder approach. Thus, EVA likely cannot achieve comparable robustness or efficiency.
C. Expert Group Utilization
C.I. Group Load Analysis for Entire Layers
In the main paper (Figure 5), we have presented expert load distribution for selected layers. Figure 6 provides the distribution across all MoE layers, illustrating how MoHAVE dynamically adjusts expert groups based on noise conditions.
C.2. Language-wise Analysis on Multilingual Tasks
We additionally provide language- wise analysis on multilingual AVSR in Figure 7. Our analysis indicates eeguuage-deendeenn differences the expert allocation within MoHAVE, For example, Arabic tokens tend to be routed more frequently toward
15

---

MoHAVE: Mixture of Hicrarchical Audio-Visual Experts for Robust Speech Recognition
visual experts, whereas Spanish French tokens rely more heavily on audio experts. However, also that these trends vary by layer. Also, within each expert group, the intra-modal router's load-balancing ensures uniform expert utilization across data samples. Thus, there is explicit language-specifie expert selection within groups, consistent with observations found in Zoph et al. (2022). We suppose that more detailed investigation into expert load distribution across languages and its relation to linguistieppraaliggisti characteristics would serve as valuable future work.
babble,
babble,
babble, -NR=0
babble, SNR=5
babble, SNR-10
75%-
75%
75%
75%
75%
Percentage
Percentage
PercenLage
50%
50%
50%
50%
50%
757
2556
MoE Layer
MoE Layer
MoE Layer
MoE Layer
MoE Layer
SNR=-10
natural, SNR=0
natural,
SNR=10
Percenage
Percentage
Percentage
Percentago
50%
5056
30%
50%
5078
2556
2554
255
25%
250
09
054
0%
Layer
MoE Layer
Layer
MoE Layer
Layer
Figure Expert load distribution in MoHAVE for the audio group (solid bars) and visual group (dashed bars) across noisy audio-visual sequences under babble (first row) and natural (second row) noise. The frequency of cach expert has been weighted by the inter-modal router's output probability.
Layer
Layer
Layer 3
Layer 4
300%
75%
75%
50%
50%
2554
25%
De 21 Es
P: Ru
Ar
D6
E5 FT
PC RU
Ar De
E3 Fr
PC
A7 De E
RV SNR-0 O, m-babble
E5
Fr 12
SNR-0 m-babble
SNR-0, m-babble
SNR 0, m-babble
Layer
Layer
Layer 7
Layer 8
100%
300%
300%
700%
72%
75%
75%
75%
50%
32%
52%
50%
2556
25%
75%
25%
07
0:
51
0%
8%
m-babble
5NR m-babble
SNR-0
m-babble
m-babble
Figure Expert load distribution multilingual MoHAVE the audio group (solid bars) and visual group (dashed bars).
16

---

MoHIAVE: Mixture of Hicrarchical Audio-Visual Experts for Robust Speech Recognition
D. Additional Results
D.I. Evaluation Under Real-world Noise Conditions
Following the standard practice in robust AVSR works (Hong et al., 2023; Kim et al., 2024), we have introduced various noise conditions (babble, speech, music, and natural) to evaluate our MoHAVE's robustness and adaptability, To better reflect real-world noise conditions, we conduct further evaluations by augmenting LRS3 with realistic background audio from the DEMAND dataset (Thiemann et al., 2013), which contains recordings from diverse indoor and outdoor environments, e.g. cafeteria meeting room. As summarized in Table 6, on this enhanced benchmark, MoHAVE consistently outperforms AV-HuBERT across various real-world settings, achieving an average WER of 3.6% for 18 environments. These results further confirm MoHAVE's performance under realistic audio-visual conditions.
Table Performance comparison on LRS3 (Afouras et al., 2018b) with audio noise sampled from the DEMAND dataset (Thiemann et al., 2013). For each noisy environment, WER (%) measured by randomly sampling the SNR value from the range [- 10, OB.
TARK
RIVER
CAFE
RESTO
CAFETER
METRO
STATION
MEETING
KITCHEN
LIVING
WASH
FIBLD
HALL
POTTCE
OFTIC
SQUARE
TRAFFIC
BUS
CAR
AVC.
Model
AV-HUBERT-LARGE MoHAVE-LARGE
6.1
6.4
13.1
4.7
5.7
1.6
1.7
2.1
2.7
3.0
1.9
4.4
5.4
11.9
6.4
4.1
4.5
2.2
1.9
1,9
2.3
2.5
1.8
D.2. Multilingual Tasks Clean Environments
Table 7 outlines the MuAViC benchmark results in a clean environment, without auditory noise added. The experimental setup remains consistent with Table utilizing the same models. Unlike the noisy setting, we observe that MoHAVE does not yield significant performance improvements clean speech tasks. This is primarily because MoHAVE enhances AVSR under noisy conditions by dynamically adjusting the utilization of audio and visual expert groups.
Indeed, in clean speech recognition and translation tasks, encoder capacity-particularyy when pretrained on large-scale audio data- -ail-pllsss more crucial role than decoder-speciff training methods. In addition, visual information is less essential in noise-free environments, as demonstrated by the strong ASR performance of the Whisper-large-v2 model (Radford et al., 2023). Even the smaller ASR model, XLS-R 300M (Babu et al., 2022), surpasses AVSR models such as mAV-HuBERT (Anwar et al., 2023) or u-HuBERT (Hsu & Shi, 2022) this setting, underscoring that the advantage of using AVSR models emerges most clearly in robust speech recognition.
D.3. Number Activated Experts
By default, MoHAVE selects one expert from each group- audio and visual activating total of two experts per token. This design is match the com- pute of standard MoE implementations, which uti- lizes top-2 out of 8 experts. A natural question arises: does activating more experts improve per- formance, or does it simply increase computational costs without substantial gains?
Table Impact the number activated experts on AVSR performance.
(k44 k
babble
speech
music
natural
N N-WER
C-WER
(1,1)
9.6
4.2
4.7
4.5
5.8
1.8
(1,2)
9.3
4.1
4.8
4.4
5.7
1.9
(2,
10.1
4.5
2.4
(2,2)
11.0
5.2
5.8
5.5
6.9
3.0
Table 8 presents the results when activating more experts of MoHAVE-BAN where top-k- experts from the audio group and kV experts from the visual group are selected. Interestingly, increasing the number of audio experts significantly degrades performance, implying that the model might be confused by employing another sub-optimal expert.
In contrast, activating two visual experts while keeping one audio expert improves performance (N-WER of 5.7%) compared to the default setting of single visual expert. Particularly under the babble noise, WER has decreased from 9.6% to 9.3%. This suggests that adding an additional visual expert can be beneficial in noisy environments, likely due to the increased robustness from visual information challenging audio conditions.
17

---

MoHAAV:: Mixture of Hicrarchical Audio-Visual Experts for Robust Speech Recognition
Table Multilingual audio-visual speech task performance, with non-English speech recognition (WER) and X-Fn speech-to-text translation (BLEU score), in clean environment without auditory noise. 'Results obtained from Han et al. (2024). 'Re-implemented using the pretrained model from Choi et al. (2024).
Source
Model
Pretrain data
Ar
De
Es
Fr
It
Pt
Ru
Avg
Clean Speech Recognition,
Test WER
Whisper large-v2 (Radford et al., 2023)
680k hrs, 100+ langs
91.5
24,8
25.4
12.0
12.7
13.0
15,5
31.1
28,2
u- HuBERTI (Hsu & Shi, 2022)
1.7k hrs, English
89.3
52.1
46.4
17.3
20.5
21.2
21.9
39.1
mAV-HuBERT's (Anwar et al., 2023)
1.7k hrs, English
69.3
47.2
41.2
16.2
19.0
19.8
19.9
38.0
33.8
XLS-R 300M3 (Babu et al., 2022)
1.2k hrs, 9 langs
85.6
44.0
34.4
13.2
15.1
14.3
16.2
34.4
32.2
XLAVS-R 300M (Han al., 2024)
8.3k hrs, 100+ langs
80.0
38.0
28.1
11.7
15.3
13.8
14.4
31.2
29.1
XI.AVS-R 2B (Han et al., 2024)
1.2k hrs, 9 langs
79.3
44.4
19.0
9.1
12.3
10.6
11.2
25.0
26.4
MAV-HuBERT?
7.0k krs, 100+ langs
78.3
41.4
25.5
11.9
16.2
14.8
14.3
31.6
29.3
MoHHAVE-LARGE (ours)
7.0k hrs, 100+ langs
85.1
38.9
25.9
11.2
14.6
14.0
13.8
30.0
29.2
Clean X-En Speech-to-Tex Translation, Test BLEU 1
Whisper large-v2 (Radford al., 2023)
680k hrs, 100+ langs
24.2
28.9
34.5
29.2
32.6
16.1
29.9
mAV-HuBERT (Anwar et al., 2023)
1.7k hrs, English
7.6
25.2
20.0
24.0
XLAVS-R 300M (Han et al., 2024)
8.3k hrs, 100+ langs
18.3
23.9
29.8
25.1
28.9
12.1
23.0
XLAVS-R 2B (Han et al., 2024)
1.2k hrs, 9 langs
21.6
25.1
30.6
26.6
29.9
13.9
24.6
mAV-HuBERT
7.0k hrs, 100+ langs
11.5
24.2
29.2
23.9
28.1
10.4
21.2
MoHAVE-LARGE (ours)
7.0k hrs, 100+ langs
13.8
24.9
30.8
25.0
28.7
10.9
22.4
D.4. Unimodal Task Results
Table presents unimodal task results, evaluating model performance on video-only (VSR) sequences and audio-only (ASR) sequences. BRAVEn (Haliassos et al., 2024) and VSR performance, as these models are specifically pretrained for the VSR task. While using an L.LM decoder is highly effective in VSR, since LLMs are able to refine and correct recognition errors, ASR performance largely determined by the encoder's pretraining strategy as BRAVEn and Whisper encoders. As an adaptive audio-visual model, MoHAVE does not specialize in unimodal tasks but instead performs robustly in multimodal AVSR. It only exhibits slight improvement in VSR over AV-HuBERT. These results indicate that unimodal performance is primarily influenced by the effectiveness of the encoder pretraining strategy rather than the MoE-based multimodal approach.
(Cappellazzo et al., 2024) models achieve the best
Table Comparison on the unimodal ASR VSR task performance.
Method
Encoder Decoder
BRAVEn (Haliassos et al., 2024)
BRAVEn Transformer
26.6
1.2
Llama3.1 Bamma3.--BB-AA -B-AVSR (Cappellazzo et al., 2024)
AV-HuBERT + LLM
25.3
Llama3.1-8B. 1-B--1----B-S (Cappellazzo et al., 2024)
Whisper LLM
AV-HUBERT-LARGE (Shi al., 2022a)
AV-HuBERT + Transformer
28.6
1.4
MoHAVE-LARGE (ours)
AV-HuBERT + MoE
28.2
1.4
D.5. Variations of MoHAVE Implementations
MoHAVE in the encoder. We have implemented MoHAVE by integrating MoE into the decoder to facilitate text token processing while enhancing multimodal fusion. Since the AVSR decoder incorporates information from both audio and visual modalities along with text tokens, the decoder-based MoHAVE is expected to be the most effective strategy. An alternative approach is to apply MoHAVE within the encoder, by pretraining the encoder using the AV-HuBERT masked prediction strategy (Shi et al., 2022a). For this, we initialize the pretrained encoder (with standard Transformers) and convert the FFN layers into MoF layers by copying the FFN parameters into all the expert modules. Since the BASE model consists of 12 encoder layers, we convert of them alternatively to match the number of MoE layers in the decoder MoHAVE. During ane-tuning, all MoE layers in the encoder are also trained following the same procedure.
There are two options pretraining strategies: (1) pretraining only the MoE layers initialized from the FFN parameters,
18

---

MoHAVE: Mixture of Hicrarchical Audio-Visual Experts for Robust Speech Recognition
and (2) pretraining the entire encoder with MoE layers. As shown in Table 10, the latter approach significantly outperforms the former, suggesting that encoder MoHAVE requires full pretraining for effective learning. However, even with full pretraining, encoder MoHAVE performs inferior to decoder MoHAVE, This is because the encoder only processes audio and visual tokens, whereas the decoder directly integrates audio-visual embeddings with text, finding optimal strategies for text token dispatching that best improves speech recognition. addition, applying MoHAVE to both the encoder and decoder leads to degraded performance despite the increased computational cost.
Decoder uptraining. We also explore a successive training strategy for the decoder, referred to as uptraining (Ainslic al., 2023), where decoder MoHAVE undergoes additional training after the fine-tuning phase of standard Transformers, Table 10, uptraining does not yield further improvements compared training from scratch, even after additional 120K training steps. In fact, observed the shorter uptraining steps leading degraded performance. This suggests that training the decoder MoHAVE requires comprchensive learning phase rather than incremental fine-tuning, as MoE may fundamentally alter the processing pathways of tokens.
However,
Table 10. Performance comparison of MoHAVE applied to the encoder, decoder, and both.
Method
babble
speech
music
natural
N N-WER
C-WER
Encoder MoHAVE (pretrain only MoF)
5.0
5.0
6.5
2.4
Encoder MoHAVE
10.0
4.4
5.1
4.5
6.0
1.9
Encoder Decoder MoHAVE
10.1
4.7
5.3
4.7
6.2
2.0
Decoder MoHAVE
9.6
4.2
4.7
4.5
5.8
1.8
Decoder MoHAVE (uptrain)
9.7
4.2
4.8
4.5
5.8
1.8
19


Reviews:
[{"review_id": "lLYEbZ2FTn", "reviewer": "Reviewer_ikv4", "summary": "This paper enhances audio-visual speech recognition based on MoE with audio/visual hierarchical modeling. This paper attaches the audio-visual MoE fusion parts to the decoder part and controls the audio and video expert groups, respectively, based on the group-level load biasing loss so that each modality contributes to audio-visual speech recognition in a balanced manner. The experimental results show that this balanced (hierarchical) manner performs reasonably from the hard routing-based approaches. The method also achieved state-of-the-art performance in the noisy LRS3 benchmark and multilingual audio-visual speech recognition/translation.\n\n## update after rebuttal\nI checked the rebuttal, but I still could not fully convince myself that this paper has significant novelty from AV-MoE (Cheng et al.) and EVA (Wu et al.) in terms of the methodology, as the major concept is similar. The actual experimental comparisons will make their claims stronger. Thus, I want to keep my score as it is.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Weakness 1: The proposed method and its improvement are incremental\n\nA1: Thank you for your valuable comments. We would like to clarify that MoHAVE introduces several key innovations in both scalability and robustness for AVSR systems, which go beyond existing works:\n\nMoHAVE is the **first AVSR framework that scales up to ~1B params**, through using a sparse MoE architecture to enable efficient scaling with low computational overhead. To mitigate the model’s inherent bias toward audio, we also introduced **expert group specialization**, followed by a **novel hierarchical gating mechanism** that dynamically routes tokens based on modality reliability and input characteristics. Unlike previous multimodal MoEs relying on modality fusion or assigning fixed roles to each expert, MoHAVE explicitly adjusts expert group usage dynamically—enhancing both adaptability and robustness.\n\nMoHAVE achieves **state-of-the-art results across robust AVSR benchmarks** (Tables 2 and 3). While the improvements in Table 1 may seem modest on average, the gains under severe noise conditions are substantial. As shown in Appx. A.3 (Table 5), MoHAVE-Large achieves 5.0% WER on LRS3 with speech noise at SNR=-10—yielding a **56.1% relative WER improvement over AV-HuBERT-Large, 36.7% over AV-MoE-Large, and 25.4% over the hard-routing variant.** This indicates that MoHAVE correctly predicts over half of the words AV-HuBERT misses.\n\nWe believe these contributions pose a significant breakthrough in AVSR scalability and adaptive learning, and our hierarchical routing design offers broad potential for other multimodal MoE applications as well.\n\n> Weakness 2: The paper should have more comparisons with hard routing\n\nA2: We initially introduced hard routing into AVSR to utilize visual experts for noise robustness. However, it lacks adaptability since the allocation of expert groups must be manually fixed (50% audio / 50% visual in our implementation). This approach is sub-optimal depending on audio-visual input quality (as discussed in Fig. 4(b)), and the limitation becomes clearer under challenging environments. Under severe conditions (e.g., Table 5, SNR=-10, -5) hard routing cannot dynamically adjust expert usage, performing much worse than MoHAVE. In response to the reviewer’s suggestion, we additionally evaluated the hard routing model on MuAViC, as **provided below**. Here, hard routing substantially underperforms MoHAVE and even mAV-HuBERT. \n\nModel (Task)|Ar|De|El|Es|Fr|It|Pt|Ru|Avg\n-|-|-|-|-|-|-|-|-|-\nHard Routing (AVSR)|93.4|49.3|35.7|20.3|23.6|23.4|24.1|44.7|39.3\n**MoHAVE** (AVSR)|92.9|47.3|35.3|18.7|21.2|21.6|21.9|40.6|**37.4**\nHard Routing (AVS2TT)|-|-|6.7|19.9|24.7|19.6|23.0|7.2|16.8\n**MoHAVE** (AVS2TT)|-|-|11.4|22.3|27.1|22.1|25.1|9.2|**19.5**\n\n**Regarding Fig. 4 and 5:** computing expert load distribution for hard routing would be trivial. By design, expert usage is manually set depending on the input: 100% audio experts for audio-only, 100% visual for video-only, and 50/50 for audio-visual (finding optimal split is heuristic). Unlike MoHAVE, there is no data-driven or noise-aware expert selection. Thus, hard routing would trivially display static distributions without dynamic behavior.\n\n> Weakness 3: Experimental comparisons with (Cheng et al., 2024; Wu et al., 2024)\n\nA3: Direct comparisons with AV-MoE (Cheng et al.) and EVA (Wu et al.) are unfortunately infeasible due to fundamental differences in target tasks and methods. Both AV-MoE and EVA primarily address audio captioning for visual contexts (e.g., narrating sports game scenes), while our work specifically targets typical AVSR tasks, where both audio and visual inputs directly involve human speech.\n\nMoreover, AV-MoE employs a dense MoE; unlike sparse expert structures commonly used in modern LLMs or Transformers, AV-MoE’s \"MoE\" is actually implemented as weighting between unimodal and cross-modal adapters, rather than selecting sparse FFN experts. Specifically, AV-MoE uses two entirely separate MoEs for audio encoder and visual encoder, infeasible for processing multimodal tokens. Our approach, MoHAVE, fundamentally differs by employing a sparse multimodal MoE, dynamically routing tokens based on audio-visual inputs.\n\nCloser to our work is EVA, which simply applies a sparse MoE structure into an audio-visual encoder. Although exact implementation details are unavailable (code/checkpoints unreleased), EVA’s structure aligns closely with our basic MoE implementation which we evaluated as *AV-MoE* in Table 1 (*AV-MoE-Base* and *AV-MoE-Large*), except ours is in the decoder. As demonstrated in our study (Table 9 in Appx. B.5), applying MoE at the encoder-level—like EVA—falls behind our multimodal decoder approach. Thus, EVA likely cannot achieve comparable robustness or efficiency.\n\n> Suggestion 1: Rewrite the equations\n\nA4: Thank you. We will revise the equations to clearly distinguish the expert set from its cardinality, defining $E_i$ as the output of the $i$-th expert.", "claims_and_evidence": "- The proposed MoHAVE (Mixture of Hierarchical Audio-Visual Experts) shows performance improvements from hard routing-based approaches. However, its margin is small.\n- The method shows the state-of-the-art performance on two public benchmarks (audio-visual noisy speech recognition based on LRS3 and multilingual audio-visual speech recognition/translation based on the MuAViC benchmark)\n- The paper also shows how much each modality contributes by checking the MoE posterior values, but this does not have comparisons, and it is difficult to validate better interpretability.", "methods_and_evaluation_criteria": "- The paper uses two public benchmarks (audio-visual noisy speech recognition based on LRS3 and multilingual audio-visual speech recognition/translation based on the MuAViC benchmark). It is based on the established token (word/character) error rate and BLEU score, and its evaluation criteria are valid.", "theoretical_claims": "- This paper does not have a theoretical component.", "experimental_designs_or_analyses": "- I found that the straightforward approaches based on the hard-routing method are very competitive. The paper should provide the hard-routing method in the following experiments (e.g., Tables 2 and 3).\n- The expert load distribution results (Figures 4 and 5) are intuitive and interpretable. However, there are no comparisons, and it is difficult to discuss whether this behavior is better or not. Is it possible to compute that of the hard-routing method? Then, we can discuss how the proposed MoHAVE is more reasonable.", "relation_to_broader_scientific_literature": "- multi-modal processing (vision, text, and speech) has become very important in recent AI technologies. Also, MoE has become a very active research topic in ML now. So, this research that combines them would have a good broader scientific impact.", "essential_references_not_discussed": "The paper references prior AV speech recognition methods based on MoE (Cheng et al., 2024; Wu et al., 2024) and distinguishes its focus on general and speech video scenarios. However, this distinction is insufficient, as all approaches (MoHAVE, Cheng et al., 2024; Wu et al., 2024) are applicable to both scenarios. Given the methodological similarities, the paper should include experimental comparisons with these prior works to strengthen its claims.", "other_strengths_and_weaknesses": "Strengths\n- showing the state-of-the-art performance in two public audio-visual speech recognition/translation benchmarks, including the multilingual setup\n- the modality contribution is intuitive\n\nWeaknesses\n- the proposed method and its improvement are incremental. \n- hard-routing methods seem to be strong, and the paper should have more comparisons with them\n- AV ASR has already been studied in various literature (e.g., (Cheng et al., 2024; Wu et al., 2024)) and its contribution is marginal. The experimental comparisons with (Cheng et al., 2024; Wu et al., 2024) may somewhat mitigate this weakness.", "other_comments_or_suggestions": "- Equation (2): $E_i(x)$ suddenly appears without any explanation. Also, it is confusing because $E$ is used as the number of experts in equation (1). I recommend the authors rewrite these equations.\n- This is just a note. The TED organization changed its policy last year and did not want researchers to use the TED data for AI development. The MuAViC dataset is based on the TED data, and it would be difficult to publish the results based on this data in the future.", "questions_for_authors": "Figures 4 and 5: Can you compute and compare the expert load distribution in the hard-routing methods versus the proposed methods? If the proposed methods demonstrate greater interpretability, this would further substantiate the method’s interpretability claims.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4BScearUrP", "reviewer": "Reviewer_QYg3", "summary": "The paper introduces MoHAVE, a novel Audio-Visual Speech Recognition (AVSR) framework leveraging a Mixture of Experts (MoE) architecture. By dynamically selecting modality-specific experts through a hierarchical gating mechanism. Experimental results on benchmark datasets demonstrate its effectiveness, outperforming existing models in challenging noisy environments.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Weakness 1: The model is evaluated on synthetic data and not on real-world conditions\n\nA1: We acknowledge your concern regarding evaluation with synthetic noise data. While standard AVSR benchmarks such as LRS3 and MuAViC typically offer curated datasets with high-quality audio and clear visual information, **these benchmarks alone cannot fully represent real-world noisy conditions.** Therefore, following standard practice in robust AVSR research [1,2], we have introduced various noise conditions to evaluate our MoHAVE's robustness and adaptability.\n\nAdditionally, to better reflect real-world noise conditions, we conducted further evaluations by augmenting LRS3 with realistic background audio from the DEMAND dataset [3], which contains recordings from diverse indoor and outdoor environments, e.g., cafeteria. On this enhanced benchmark (at SNR=-10~0), MoHAVE consistently outperformed AV-HuBERT across various real-world settings including **cafeteria (WER: 6.4 vs. 8.6), restaurant (11.9 vs. 13.1), meeting room (4.5 vs. 5.7), and river (4.4 vs. 6.1)**, achieving an **average WER of 3.6% vs. 4.1%** across all 18 environments. These results further confirm MoHAVE’s performance under realistic audio-visual conditions.\n\n\n> Weakness 2: CMA and UniVPM are missing from Table 1\n\nA2: The CMA and UniVPM models in Table 2 are all built upon AV-HuBERT-Large, **matching the architecture and activated parameter count with the dense (non-MoE) baseline in Table 1.** Furthermore, both Table 1 and Table 2 evaluate under identical experimental setups and noise configurations (i.e., babble, speech, music, and natural noises), while Table 2 reports average results for music and natural. For detail, please refer to the table below.\n\nMethod|# Experts|Groups|Activated Params|Total Params|babble|speech|music|natural|N-WER|C-WER\n-|-|-|-|-|-|-|-|-|-|-\nUniVPM|-|-|478M|478M|9.3|4.1|3.6|3.6|5.2|1.2\nCMA|-|-|500M|500M|8.1|2.9|3.8|3.6|4.6|1.5\n\nWe note that Table 1 primarily demonstrates the effectiveness and efficiency of MoHAVE, **by comparing different MoE variants**, including standard MoE, hard routing, and hierarchical MoE. In contrast, CMA and UniVPM in Table 2 utilize special modules for cross-modality, which are orthogonal to the MoE framework, focusing instead on audio-visual fusion or feature enhancement strategies independent of expert routing mechanisms. Yet, recognizing the importance of comprehensive comparisons, we have included the result of incorporated CMA + MoHAVE in Table 2.  To improve clarity, we will revise by merging Table 1 and Table 2.\n\n> Question 1: Can MoHAVE handle asynchronous speech and lip movements?\n\nA3: We have not yet evaluated MoHAVE under audio-visual asynchronous conditions. Our current framework is optimized for scenarios where audio and video are temporally aligned, as is standard in most AVSR works. Prior works that address the audio-visual asynchrony have proposed solutions like external synchronization module [4], which explicitly model temporal offsets between audio and visual streams. While MoHAVE does not currently model asynchrony, we believe combining MoHAVE with methods that explicitly handle asynchronous inputs could be a valuable extension for future research.\n\n\n> Question 2: How do different languages affect expert selection and hierarchical MoE routing?\n\nA4: Thank you for this insightful question. Our analysis indicates language-dependent differences in expert allocation within MoHAVE. For example, Arabic tokens tend to be routed more frequently toward visual experts, whereas French or Spanish tokens rely more heavily on audio experts (please see this [**anonymized link**](https://figshare.com/s/a45cbf35654ae37b2ce9)). However, we also note that these trends vary by layer. Also, within each expert group, the intra-modal router’s load-balancing ensures a uniform expert utilization across data samples. Thus, there is no explicit language-specific expert selection within groups, consistent with observations found in [5]. We suppose that more detailed investigation into expert load distribution across languages and its relation to linguistic/paralinguistic characteristics would be valuable future work.\n\n---\nReferences:\n\n\n[1] Hong et al. \"Watch or listen: Robust audio-visual speech recognition with visual corruption modeling and reliability scoring.\" CVPR, 2023.\n\n\n[2] Kim et al. \"Learning video temporal dynamics with cross-modal attention for robust audio-visual speech recognition.\" SLT, 2024.\n\n\n[3] Thiemann et al. \"The diverse environments multi-channel acoustic noise database\" Proceedings of Meetings on Acoustics, 2013.\n\n\n[4] Li et al. \"Unified cross-modal attention: robust audio-visual speech recognition and beyond.\" TASLP, 2024.\n\n\n[5] Zoph et al. \"St-moe: Designing stable and transferable sparse expert models.\" arXiv, 2022.", "claims_and_evidence": "The main claims of the paper are: \n\n1. MoE architecture for scaling AVSR systems that effectively scales AVSR models while maintaining computational efficiency \n2. Hierarchical gating for adaptive expert utilization\n3. Robust AVSR performance \n\nThese claims are generally well-supported by experimental results. However, a comparison with all relevant baseline models (e.g. CMA, UniVPM) is important to support claim 3.", "methods_and_evaluation_criteria": "The proposed method and evaluation criteria are well-defined. The hierarchical Mixture-of-Experts (MoE) approach is a reasonable architectural choice for improving model scalability and the evaluation benchmarks (LRS3 and MuAViC) are widely used in the AVSR research community. However, the evaluation mostly relies on synthetic noise additions rather than real-world conditions. It would be interesting to show how the model performs on real-world data.", "theoretical_claims": "The theoretical claims are well-supported particularly in the discussion of MoE routing mechanisms and hierarchical gating. The paper clearly defines the load balancing loss and load biasing loss to improve expert selection. The hierarchical gating strategy for inter-modal and intra-modal routing is mathematically justified, and the empirical results show effectiveness the approach.", "experimental_designs_or_analyses": "The experimental design and analysis are well-structured with the use of two AVSR benchmarks: LRS3 and MuAViC. The authors compare their approach with multiple baseline models (e.g. AV-HuBERT, AV-MoE),  perform ablation analysis (load biasing loss, hard routing, and number of activated experts), along with a computational cost analysis that show the efficiency of the proposed model. However, some baseline models (e.g. CMA, UniVPM) are not included in Table 1 for comparison, despite being mentioned in the related work and included in Table 2.", "relation_to_broader_scientific_literature": "The paper’s key contributions build upon prior work in Audio-Visual Speech Recognition (AVSR) using Mixture-of-Experts (MoE), including models like AV-HuBERT, AV-data2vec, and Auto-AVSR, which utilize self-supervised learning for audio-visual speech processing. However, MoHAVE advances this approach by incorporating a hierarchical MoE framework, enhancing scalability and robustness while maintaining computational efficiency.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "Strengths\n\n1. Scalability without excessive computational cost.\n2. Adaptive expert selection improves generalization across different noise conditions.\n3. Comprehensive benchmarking across various AVSR datasets and multilingual settings.\n\nWeaknesses\n\n1. The model has been evaluated on synthetic data and not on real-world conditions\n2. Some relevant baseline models (e.g. CMA, UniVPM) are missing from the comparison.", "other_comments_or_suggestions": "NA", "questions_for_authors": "1. Why don’t you compare MoHAVE with the CMA (Kim et al., 2024) and UniVPM (Hu et al., 2023c)  in Table 1?\n2. Can MoHAVE handle asynchronous speech and lip movements (e.g. delays in video frames relative to audio)?\n3. How do different languages affect expert selection and the hierarchical MoE routing?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GiYJvaAW0H", "reviewer": "Reviewer_Fwbw", "summary": "This paper proposes an adaptive hierarchical routing mechanism in mixture of experts model for audio-visual (AV) speech recognition and AV language X speech to English text translation. As compared to a hard routing of modalities into modality specific expert groups, this paper uses a combination of inter-modal and intra-modal routers while still keeping audio and video experts separate. This way the model can utilize A-only, V-only and AV data in one model. The paper also implements some load balancing techniques to balance the mixture weights. \n\nFor AVSR, LRS3 dataset is used and the model performance is reported under various noise conditions. For cross-lingual speech task, MuAViC benchmark is used. The model is compared agains Base AV MoE method and AV-HuBERT method. In the AVSR experiments, it is shown that the WER on noisy datasets improve over AV-HuBERT and that the load biasing is a crucial implementation detail that affect the final WER. In the X-En Speech-to-Text Translation, the large MoHAVE model outperforms XLAVS-R models for most of the test languages in terms of the BLEU score.\n\n## update after rebuttal\nI would like to keep my score after rebuttal.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Weakness 1: The notation in the equations could be improved\n\nA1: Thank you for the detailed review. We will revise the notations, especially in Eq. (11), to clearly distinguish the set of experts from its cardinality.\n\n\n> Suggestion 1: The computational cost analysis is only discussed in Appendix\n\nA2: We agree with your suggestion that computational efficiency is indeed one of the main contributions of MoHAVE, especially given its sparse MoE architecture that activates only a fraction of parameters. To better highlight this contribution, we will move the computational cost comparison as well as its discussion currently presented in Appx. A.2 into the main body.\n\n> Question 1: What was the reason for the following choice? *For sequences containing both audio and video, we exclude them from the load biasing loss calculation but incorporate them into the load balancing.*\n\nA3: The load biasing loss is designed specifically to encourage modality specialization for a subset of experts. Audio-visual multimodal tokens do not inherently favor one modality-specific expert group over another. Therefore, these tokens are included in the load balancing, ensuring uniform expert group loads across every multimodal token.\n\n> Question 2: Do the batches contain a mix of audio-only, video-only, and AV data points in a single batch?\n\nA4: Yes, each batch is constructed by sampling a total of 320 seconds of data, structured as follows: 25% of the data points in the batch are audio-only (video dropped), another 25% are video-only (audio dropped), and the remaining 50% are multimodal, containing both audio and video.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "Equations in the main text mostly seem to be correct. In Eq. (11), $E^{A}$ is used both as the set of experts and the number of experts in summation, hence the notation is a little confusing.", "experimental_designs_or_analyses": "Went through the tables but have not checked the details of Figs. 4 and 5 (which have an expected pattern of relying less on audio and more on video in low SNR conditions). \n\nOne of the main contributions is presented as \"robust AVSR benchmarks including multilingual tasks, delivering high accuracy while maintaining computational overhead. \" However, the computational efficiency is only discussed in the appendix. It could have been useful to include that analysis in the main text and Fig. 4 or 5. could have been moved to the Appendix.", "relation_to_broader_scientific_literature": "In line with previous studies, load balancing is crucial for the success of the MoE implementation. The analysis of the mixture weights per modality was also supporting earlier observations around relying more on the visual component when the acoustic noise is heavy. \n\nSince multimodal systems are becoming more popular, the proposed hierarchical multimodal MoE might be helpful for other studies in other multimodal applications.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "+ Strengths: A variant of the MoE routing is proposed which might be useful for other multimodal applications. Results show some improvement over the AV-HuBERT baseline for AVSR and MoHAVE also can handle some multilingual tasks.\n\n- Weaknesses: The notation in the equations could be impoved, and the paper might benefit from including the computational cost analysis in the main text.", "other_comments_or_suggestions": "1) Eq. 11, please check the notation as described above\n\n2) One of the main contributions is presented as \"robust AVSR benchmarks including multilingual tasks, delivering high accuracy while maintaining computational overhead. \" However, the computational efficiency is only discussed in the appendix. It could have been useful to include that analysis in the main text and to save space Fig. 4 or 5. could have been moved to the Appendix. This is a minor suggestion.", "questions_for_authors": "- What was the reason for the following choice? \"For sequences containing both audio and video, we exclude them from the load biasing loss calculation but incorporate them into the load balancing.\"\n\n- How are the batches constructed during training? Do the batches contain a mix of audio-only datapoints, video-only datapoints and AV data points in a single batch?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "NA"}}, {"review_id": "QdRXWsAZjF", "reviewer": "Reviewer_VMgS", "summary": "MoHAVE is a novel audio-visual speech recognition system that addresses the scalability challenges in traditional AVSR models. The paper introduces a sparse Mixture-of-Experts (MoE) framework combined with a hierarchical gating mechanism that dynamically routes audio-visual inputs to modality-specific expert groups. Here’s a brief summary:\n\nMoHAVE leverages a sparse MoE architecture to scale model capacity while keeping computational costs low. Its hierarchical gating system consists of an inter-modal router that assigns weights to audio and visual expert groups based on input characteristics, and intra-modal routers that further select the top experts within each group. This design enables the model to adapt to varying noise conditions by shifting reliance between modalities—using more visual cues in high auditory noise and vice versa. The paper demonstrates that MoHAVE achieves state-of-the-art performance on robust AVSR benchmarks like LRS3 and in multilingual speech recognition and translation tasks, all while activating only a fraction of its total parameters during inference.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for reviewing our paper thoroughly and for recognizing the key strengths, including the **suitable hierarchical MoE architecture for AVSR, a novel gating mechanism, and strong experimental results demonstrating robust performances**.\n\nWe greatly appreciate your thorough evaluation from the perspective of AVSR, even though MoE lies outside your primary research field. As such, we would like to briefly highlight our contributions of MoHAVE to the MoE framework:\n\n- **Novel hierarchical gating mechanism**: An inter-modal router dynamically allocates tokens to audio or visual expert groups based on input characteristics, and an intra-modal router further dispatches tokens to appropriate experts within these groups.\n- **Adaptive and robust MoE**: Previous works for multimodal MoEs have relied on modality fusion strategies or assigning fixed roles to each expert. While they lack adaptability and robustness to dynamically changing noisy environments, our work addresses this limitation by dynamically adjusting the usage of expert groups.\n\nWe also kindly encourage you to go through other reviewers' comments as well as our responses, and if any additional concerns arise, please let us know through the discussion phase. We would be glad to provide any further clarification.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "Yes", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "A scalable sparse MoE framework tailored for audio-visual speech recognition.\nA novel hierarchical gating mechanism that dynamically adjusts expert group utilization based on input context.\nSuperior robustness and accuracy under noisy conditions, setting new performance benchmarks on standard AVSR tasks.", "essential_references_not_discussed": "The key contribution is a MOE-based AVSR model, and cite mainstream prior works.", "other_strengths_and_weaknesses": "Overall, I think this is a good piece of work; the MoE technique is well-suited for AVSR, and previous studies have shown that simply fusing the two modalities can lead to models that overly depend on audio. From an AVSR perspective, the experimental results achieved in this paper are excellent, representing a robust algorithm. However, since my own research field does not focus on MoE, I cannot provide a particularly meaningful evaluation of the innovations in the MoE component.", "other_comments_or_suggestions": "Na", "questions_for_authors": "Na", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "Na"}}]

Author rebuttal:

Meta-review: The authors propose a MoHAVE (Mixture of Hierarchical Audio-Visual Experts) framework in this paper for audio-visual speech recognition and translation. A sparse MoE architecture is introduced to efficiently scales up the AVSR model capacity. The hierarchical gating design which includes inter-model and intra-model routing can dynamically assign the expert groups for different modalities based on the input context. The authors report the SOTA performance of the proposed MoHAVE on AVSR benchmark LRS3 and multilingual AV speech recognition and translation benchmark MuAViC. Overall, all reviewers consider the work interesting and performance of MoHAVE is good although the technical novelty of using MoE for AVSR is not overwhelmingly significant. The authors' rebuttal has cleared up most of the concerns raised by the reviewers.
Decision: Accept (poster)