Efficient Audiovisual Speech Processing via MUTUD: Multimodal Training and Unimodal Deployment
Authors: Joanna Hong
Sanjeel Parekh
Honglie Chen
Jacob Donley
Ke Tan
Buye Xu
Anurag Kumar
Venue: Accepted by TMLR ()

Abstract: Building reliable speech systems often requires combining multiple modalities, like audio and visual cues. While such multimodal solutions frequently lead to improvements in performance and may even be critical in certain cases, they come with several constraints such as increased sensory requirements, computational cost, and modality synchronization, to mention a few. These challenges constrain the direct uses of these multimodal solutions in real-world applications. In this work, we develop approaches where the learning happens with all available modalities but the deployment or inference is done with just one or reduced modalities. To do so, we propose a Multimodal Training and Unimodal Deployment (MUTUD) framework which includes a Temporally Aligned Modality feature Estimation (TAME) module that can estimate information from missing modality using modalities present during inference. This innovative approach facilitates the integration of information across different modalities, enhancing the overall inference process by leveraging the strengths of each modality to compensate for the absence of certain modalities during inference. We apply MUTUD to various audiovisual speech tasks and show that it can reduce the performance gap between the multimodal and corresponding unimodal models to a considerable extent. MUTUD can achieve this while reducing the model size and compute compared to multimodal models, in some cases by almost 80%.

Published in Transactions on Machine Learning Research (12/2025)
Efficient Audiovisual Speech Processing via MUTUD: Multi- modal Training and Unimodal Deployment
Joanna Hong Meta
joarnohong@heee.or
Sanjeel Parekh
sanjeel@@eeta.
Meta
Honglie Chen
honglicchen@metta.cmm
Meta
Jacob Donley
jdonley@metta.com
Meta
Ke Tan
tanke 1 1t111116mmma...ommmm com
Meta
Buye Xu
sub@meta.com com
Meta
Anurag Kumar Meta
anurogkr@ieee.org
Reviewed on OpenReview:
https: http:://openneeveau net/ forum? id= 5bshBYBRD
Abstract
Building reliable speech systems often requires combining multiple modalities, like audio and visual cues. While such multimodal solutions frequently lead improvements in performance and may even be critical certain cases, they come with several constraints such as increased sensory requirements, computational cost, and modality synchronization, to mention few. These challenges constrain the direct uses of these multimodal solutions in real-world applications. In this work, we develop approaches where the learning happens with all available modalities but the deployment or inference is done with just one or reduced modalities. To do s0, we propose Multimodal Training and Unimodal Deployment (MUTUD) framework which includes a Temporally Aligned Modality feature Estimation (TAME) module that can estimate information from missing modality using modalities present during inference. This innovative approach facilitates the integration of information across different modalities, enhancing the overall inference process by leveraging the strengths of each modality to compensate for the absence of certain modalities during inference. We apply MUTUD to various audiovisual speech tasks and show that it can reduce the performance gap between the multimodal and corresponding unimodal models to a considerable extent. MUTUD can achieve this while reducing the model size and compute compared to multimodal models, in some cases by almost 80%.
Work done while Meta, Google DeepMind

---

Published in Transactions on Machine Learning Research (12/2025)
1 Introduction
Unimodal (audio-only) approaches well-known speech problems such as speech enhancement, speaker separation, and automatic speech recoguition, have made rapid progress using deep learning. At the same time, multimodal approaches to these tasks are also increasingly gaining significance (Mira et al. 2023 Xu
et al. 2020 Ma et al. 2021b Hong et al. 2022 2023). While the additional modality may come in different. forms such as text, contact microphones, IMUs, etc., visual modality is the most widely used in these speech tasks. This bears similarity to humans as we also innately rely on visuals to perceive sounds and speech (Schwartz et a (2044). In fact, people with hearing impairments have also been shown to rely on visuals for better perception of speoch (Burnham et al. 2013). Given the significance of multimodal perception of speech by humans, it is natural that multimodal learning has shown impressive gains over unimodal systems for various speech tasks. The role of visuals in speech understanding becomes much more critical in acoustically difficult scenarios such as noisy environments or situations where the speech signals on their own are not reliable for the task at hand (Weninger et al. 2015 Tan & Wang 2019
Wang et al., 2020 Braun et al. 2021
While multimodal systems can extract supplementary and complementary information from different modalities [Baltrušaitis et al.) 2018 Lu 2023), leading to performance improvements, certain challenges with multimodal models can restrain their uses in real-world systems. These include but are not limited to (1) Multimodal models are often computationally much more expensive compared to their unimodal counterparts and the performance gain might not justify the substantial increase in computational cost. This is especially relevant for real-time and on-device applications (e.g., speech enhancement). In fact, in several cases, this can prohibit the deployment of multimodal systems. (2) Multimodal data comes at a significantly higher cost. Acquisition of multimodal data requires complex sensory devices working together scamlessly. Alignment, synchronization, and annotation efforts in multimodal data are far more challenging than audio-only data. More importantly, such aligned and synchronized multimodal data is required even during inference, necessitating the availability of all sensory devices and the processing power to align and synchronize the captured signals. This can make multimodal systems impractical in several real-world applications. (3) Lastly, it might not be feasible to use multiple modalities for a speech task due to practical constraints such as privacy or difficulties in getting signals for all modalitics. For example, while multimodal ASR could improve audio-only ASR in noisy conditions, getting the visual signals during real-world uses might not be possible.
The above discussion highlights benefits of multimodal learning over unimodal learning, yet there are certain constraints which can make unimodal models preferable over multimodal despite lower performance. Motivated by this, the primary question we ask is how do we learn from multimodal data while enabling unimodal uses of the model? In this framework, we still want to learn from the rich information available in multimodal data but unimodal inference removes the constraints around uses of the multimodal system. Note that, unlike works on robustness to missing modality we develop a fundamental approach for MUltimodal Training and Unimodal Deployment (MUTUD, pronounced "mauled"). In modality robustness, the model behavior remains the same during training and inference, and hence the challenges of multimodal systems outlined before are not rectified. MUTUD is driven by architectural and training novelties, which addresses those challenges. MUTUD framework is built using a novel Temporally Aligned Modality feature Estimation (TAME) module. The TAME module is designed to estimate deep representations of modalities which are absent during inference using the representations of modalities present during inference. TAME achieves this by having codebooks for each modality and linking pross-modal pairs of codebooks in a way that enables modality feature recall using the codebooks and the features of available modalities. Keeping in mind the importance of temporal information, TAME is designed to temporally align modalities sampled at different. time resolutions.
We apply our framework for 3 well-known tasks in the speech processing domain and do multimodal (audiovisual (AV)) training and unimodal inference; speech enhancement, speech recognition, and active speaker detection. Speech enhancement in particular may have light real-time and low-compute requirements for several applications. In all the tasks, we show that MUTUD achieves unimodal inference with a significantly better performance compared to the counterpart models trained on unimodal data. Morcover, compared to the full multimodal systems, our model has significantly lesser parameters and compute and yet gives competitive performance.
2

---

Published in Transactions on Machine Learning Research (12/2025)
2 Related Works
Audiovisual speech processing. Analogous to humans, AV learning for speech-related tasks naturally results in methods that are more robust to noisy scenarios such as acoustic SNR degradation, poor lighting conditions, motion blur, etc. In this paper we focus on three AV speech problems namely, speech enhancement
Gabbay et al.
2017
Afouras et al.;
2018a Gao & Grauman 2021 Mira et al., 2023 Yang et al.
2022
Owens & Efros
2018
Hou et al.
2018
speech recognition (Huang & Kingsbury 2013] Mrouch et al.
2015
Noda et al. 2015
Stewart et al.
2013;
Ma et al. 2021b; and speaker detection Gargg et al. 2000; Cutler &
Davis 2000
Chakravarty et al.
2016 Roth et al.) 2020 The reader is referred to excellent survey papers
for a detailed overview of different methodologies (Michelsanti et al.) 2021 Potamianos et al.) 2017 As
alrcady highlightod, traditional AV approaches suffer from several constraints such as sensor requirements, computational cost, and modality synchronization which limit their applicability in real-world applications.
Resource-conssraanee learning. Considerable progress has been made in resource-constranee audio-only speech procossing (Kim et al., 2020: Lee ct al., 2021 Maayah et al.) 2023). even though such multimodal
methods are relatively smaller. Typical stralegies include lightweight nelwork design ( (Maayah el al. 2023).
quantization and pruning (Tan et al. 2021) and knowledge distillation (Thakker et al.) 2022]. Gogate
et al. (2020) build a robust language-dependent audiovisual model called CochleaNet for real-time speech enhancement through audiovisual mask estimation. LAVSE (Chuang et al.) (020)) proposed a visual data compression technique for speech enhancement. Our focus in this work is very different. We intend to develop efficiency in multimodal learning by allowing resource-heavy modalities to be absent during prediction when deployed.
Learning with missing modality. Multimodal learning for robustness to missing modality is a practical problem that has been explored in some works before. Each work differs in the modality considered to be missing, the phase (Training or testing) which this information is absent, and whether the loss of information is partial or complete (Hegde et al., 2021 Ma et al. 2021a] Chang et al., 2022; Woo al.) 2023 Ma et al.] 2022: Lee et al., 2023). The methods often tailor-made for the scenarios in consideration. For brevity, here we limit our discussion to AV speech-related tasks. Some studies rely on memory architecture to retrieve missing modality via associated bridging mechanism (Kim et al.] 2021b Hong et al. 2021 Kim et al. 2021a). These related works serve as inspiration for MUTUD. Further, AV-HUBERT (Shi et al. 2022 and u- H-RTTERT (Hsu & Shi. 2022) presented a self-supervised pre-training framework that can leverage both multimodal and unimodal speech with a unified masked cluster prediction objective, achieving zero-shot modality generalization for multiple speech processing tasks. While these works have made significant progress on various speech processing problems, they are very different from ours their focus is on self-supervised training of large models with massive amounts of unlabeled data. The learned models are then fine-tuned for tasks like ASR. These models are not designed for unimodal deployment with compute/ compute/memory memory efficiency in mind. Furthermore, it is difficult to adapt AV-HuBERT/ HUBBBBBB for tasks like speech enhancement, especially in cansal settings.
Unlike these works, we are driven by the challenges of multimodal learning outlined before. We focus explicitly on multmmodal learning for unimodal prediction and real-world deployment, which addresses those challenges. Our approach is fairly generic and can be applied to many common multinodal learning methods and tasks.
3 MUTUD: Multimodal Training and Unimodal Deployment
We describe proposed method, which call Multimodal Training and Unimodal Deployment (MUTUD). Our goal is to design network that leverages multinodal sensory inputs during training, but only takes in a subset of them during inference. In section 3.1 we first describe MUTUD in its general setting, where an arbitrary number of modalities are considered, followed by a discussion targeted to the audiovisual speech domain. In section 3.2 we introduce our proposed TAME Modulc, which the key component to enable unimodal predictions. Finally in section 3.3. we describe the training objectives. The left panel in Figure] shows the difference between MUTUD and conventional multinodal learning.
3

---

Published in Transactions on Machine Learning Research (12/2025)
Video Encoder
Traiding Only
Video Encoder
Fv
Module
video
Input
Fasioon
Audio-basid estimated
Compatd
Concot
LSTM
Inpur audio
Input speech
Enhanced speech
Audio Encoder
Skip connacttors
Multimodal Learning: Training and Prodiction Process
(b) Audiovisual Speech Enhancement through
Training Only
F
loss
TAME Module
f2
Imput
Linear
Downstroom Task
F
audio
MUTUD: Training and Prediction Process Different
(c) Temporally Aligned Modality feature Estimation (TAME) Module
dulluig poining
Figure (a) The left panel shows a comparison between conventional audiovisual speech processing and MUTUD. TAME module enables audiovisual learning without doing video processing during prediction. (b) The upper half in the right panel illustrates MUTUD for AVSE model. After training the video encoder is
discarded. (c) The bottom half in the right panel shows the estimation of video representations using TAME.
The illustration is for 0 in Eq 5
3.1 MUTUD Overview
Let. D be dalaset where each sample X DD is characterized by M different modalities X = (Xm,). i = 1: M. M = {m1, m., m is the set of modalities. Conventionally, multimodal learning operates with the assumption that the model always imputs all M modalitios, during training as well as for predictions. Let MX = (Xm,, m. E d) a deep neural network (DNN) based multimodal system (parameterized by Φ). In MUTUD, M modalities of X are available during training, but only a subset, M, C M, are available during real-world deployment or for inference.
To this end, design MUTUD with two crucial characteristics in mind. Let h(:0) be the MUTUD system. (1) Since M(:0) processes only [M,l modalities for prediction, we expect it to have fewer parameters and be computationally more efficient in real-world deployment. Ideally, we would like h(:0) () to have inference size and compute similar to M.M.(X = {Xm,, m, E M.) v), model counterpart of h (X; with M M,, (2) On the performance end, MM (: ()) should have superior performance compared to I: due utilization of more modalitics the learning process. We expect h(; 0) have superior performance compared
M. (X; and closer to that of h (X;
In typical multimodal model, all Xm, are encoded by network, these representations through various mechanisms (concatenation, attention, etc. Kalkhorani et al. (2023): : Wei et al. (2020): Ma et al. (2021b); Lee et. al. (2020); Praveen el al. (2023)). The fused representations are further processed by more neural layers to solve the task at hand. We operate in a similar setting. To achieve our goal, we develop an efficient and effective mochanism associate and relate missing modalitics, (M - -Ma), to those in Ma; such that the representations of Xm, E M-MM, can be recalled using those of Xm, E Mg. We propose a Temporally Aligned Modality feature Estimation (TAME) module. TAME learns pair of codebooks (C"..
then fused
4

---

MUTUD for Audio Visual Speech Processing. We focus on audiovisual speech tasks where MUTUD is designed to use only one of them during deployment. For a succinct and clear description of MUTUD and TAME, we explain through the task of Audiovisual Speech Enhancement (AVSE) but the method similarly adapts other tasks. The right panel in Figure outlines the base AVSE model (MM()))) The speech and video encoders produce F, RT, and F2 IR' T representations, respectively. Ta and T, represent time dimensions and depend on the frame rates of speech and audio. The frame rate of speech is K times of video (T, T. * K) and hence F. is upsampled by a factor of K to match the size along temporal direction before the concatenation step. The concatenated representations are then decoded by the decoder to produce the enhanced speech. The (; model the Audio-only model, where everything is the same except that there are no visual inputs, and the decoder decodes the encoded audio representations to output enhancod Published in Transactions on Machine Learning Research (12/2025)
C for each pair of modality in {(m, my), Vm, -M Vmy M,3. The training objectives link these codebooks in way that enables estimation of representations for Xm, EM M, during inference.
speech. Under MUTUD, our goal is train with both visual and audio inputs but deploy an Audio-only model. Hence, we design and train TAME module to estimate video representations during prediction.
3.2 TAME Module
The core of the TAME consists of sppelity- codebooks (MSCs) for audio and video. These used to associate and relate modalities through their respective representations during training. During inference, the audio representations are used to retrieve the video representations through these MSCs. The MSCs are designed to capture temporal alignment and synchronized relations between the audio and the video. Since the audio representations frame rate (in Fa) is higher by a factor of K, we design TAME keeping this temporal relation in consideration. That is the 4th video frame feature in Fu, fú: is associated with audio features (JE-+t /K-+41, ..... j...,,jKtt4K- (11 in Fa- Besides keeping the temporal alignment between audio and video representations intact, this temporal coupling between the audio and video is also necessary for learning to estimate video features using audio.
TAME formulates this through blocks of codebooks in each MSC, represented as C E RRXNxD and C" E R for the audio and vidco respectively, (sec Figure 1 N is the number of codes in each set of codebooks in Ca and C
All features in consideration (ft for video and f1 = (fkk+, fkk fk+++1 ..... fo fk-t+k ..... f ff.., for audio) are first embedded through their respective MSC. This relationship between ft and 1 codebook in C is established through the vectors kvt. For improved readability, represent the left superscripts (k) kyd
vectors as of
( (cg ft
(1)
| la ||5 2
where 2 = C (k, n, :], n {0, 10..,..NNN NJ.
of is computed for all K codebooks (k {0, K - 13) using Eq Similarly, the audio features are related to its codebooks C as,
kk1++ ( (2g fok fkit+k
(2)
|| l2 p + 1/2
where 2 = C" [k; :]. n = 1..1....,NN N}
The temporal steps t are [0.1...., - 1}. Note that, for audio the t codebook of C linked with Ath audio feature in fi. Eq [ and 2 embed the audio and vidco information into their respective MSCs. A softmax across the number of codes gives the probability distribution of the relationship between the codebooks and the corresponding modality representations,
5

---

Published in Transactions on Machine Learning Research (12/2025)
p' exp (T inn)
(3)
プド= exp (T (9.)
exp (T 0n +++)
gkath where {0, :,. N}
(4)
exp (T E
j t
T the temperature for the softmax function. These distributions are computed for each E {O.l...., "o.L,...,,I11 K-1} The modality-spociftt information captured by ø' and d' are used to relate and associate the two modalitics well relrieve the video representations using the audio representations.
Audio-to- Video Representations. The bottom half in the right panel of Figure I shows the schematics for obtaining vidco representations using audio. The hth feature in fa directly estimates "interleaved" represenlations for video using the hth codebook in C
f fk+++h = linear
Σ on
K-t+k c 0:
(5)
where Or are the parameters of the linear layer, in practice, this linear layer includes batch-normalization [Ioffe & Szegedy 2015). The fk.tt= (instead) are concatenated with fa fK-t+ and then decoded by the decoder to produce enhanced speech. Note that, in the base AVSE model Ta video features are simply repeated to upsample by a factor of K and then concatenated to audio features. TAME helps estimate video information at lower temporal resolution, which can be crucial for precise replacement of video representations.
Clearly, the video encoder is discarded during inference and as long the size and compute of the TAME mod- ule is significantly smaller than the video encoder, the whole model is much more efficient compared to the full audiovisual model.
3.3 Training Objectives
TAME Specific Losses. To train the proposed TAME module, we propose three different training objectives. First, we need to ensure that the relationship between the video features and video codebook C" well- structured that C" gets embedded with video information. This achieved through self-modality recall of ft for each codebook in C kft ( Ph 2h 01). A reconstruction loss then guides the learning
T--K-1
Σ
Σ
k
(6)
Next, a reconstruction loss between the estimated video representations fKit+ and ft enforces retrieval of video information through audio representations.
1-K Σ 1= K=
La-s0 =
Σ
f
pkt+
- -fell2
(7)
Lastly, we establish cross-modal association by linking the two MSCs through the distribution captured by I' and gKtt+k Let pe (captured by f²) and Q² (captured by Ketth) be the distributions over the codes for Eth codebook in C and C respectively.
K
Lc,-cc
D (P*11t)))))
(8)
k=1

---

Published in Transactions on Machine Learning Research (12/2025)
The loss function in Eq the distribution of codes in each codebook of C matches the corresponding ones in C This is necessary the codebooks in C are probed using audio representations embedded in q to obtain video representations.
Task-Specific Loss Functions. The overall training MUTUD inchudes task-specific loss functions which in this case are speech enhancement losses. In this work, the outputs of the enhancement models are complex spectrograms (E) of the enhanced speech. The time-domain waveform (e) from E is obtained using the Inverse-Short Time Fourier Transform. The speech enhancement loss functions we use
taskk
| - Cl1 - SI-SDD((,,
(9)
where C' is the complex STFT of target clean speech and is the time-domain target clean speech. The <
S1-SDR loss is defined as SI-SDR(e, c) = 10 logı0 [|oc-e]3 where a = T
e The enhancement losses in Eq 9
are computed using both f and ft as inputs to the decoder and the overall Lusk is the sum these losses. This necessary to warrant that the video encoder learns meaningful representations in the end-to-end training.
The total loss function is
MMTTUD = Lu-+o + La-se + Lca-Co + ALtask
(10)
where À is the weight given to the task loss.
A few points are worth noting here. The TAME which is enabling MUTUD seamlessly fits into the base AVSE framework and can be easily adopted for many common multimodal methods and tasks. In our experiments, we evaluate MUTUD for 3 multimodal tasks; AVSE, audiovisual speech recognition (AVSR), and andiovisual active speaker delection (AV-ASD).
4 Experimental Setup
We evaluate MUTUD under 3 multimodal tasks; AVSE, audiovisual speech recognition (AVSR), and ego- centric audiovisual active speaker detection (AV-ASD). AVSE is of key focus as this task is often desired to be deployed in real-time communication and on-device, which exacerbates the multimodal challenges outlined earlier in the paper.
4.1 Datasets
For AVSE and AVSR lasks, we utilize the LRS3-TED corpus (Afouras et al. 2018b;, large-scale audiovisual dataset for speech tasks. For the AV-ASD task, we use EasyCom, a challenging real-world egocentric dataset. (Donley et al.) (021). Overall, this allows for comprehensive evaluation of MUTUD under wide variety of acoustic and visual noise conditions.
LRS3-TED. LRS3-TED corpus Afourrs et al. 2018b is a large-scale dataset of TED and TEDx videos. LRS3-TED consists of audio-visual pairs and correspouding text transcriptions for 151,819 utterances, totaling 439 hours. Following the original splits, we use ~131,000 utterances for training and ~1,300 utterances for testing. For AVSE, the clean speech samples are taken from LRS3 and the noise samples are taken from Reddy al. (2021) noise set. The videos are 25 fps with 224 224 resolution. During pro-processing, we center-crop at the mouth with size of 88 88.
EasyCom. We employ EasyCom (Dnnley et al. 2021 for the AV-ASD task. This dataset contains ~ 5 hours of natural conversations recorded in a noisy restaurant-like environment. The ego-centric nature of the data makes it extremely challenging as the sensory devices (camera and microphone on wearable glasses) are always moving. The ego-motions make it difficult to learn from the video and the audio is corrupted by noiso, making audiovisual active speaker detection (AV-ASD), challenging on this dataset. The datasct. inchudes annotated voice activity, speech transcriptions, head bounding boxes, target speech, and source identification labels. We train-test splits from Hsu et al. 2022

---

Published in Transactions on Machine Learning Research (12/2025)
4.2 Implementation Details for AVSE
Data processing. For LRS3, we crop the lip regions, resize the cropped frames into 88x88, and transform them to grayscale following Kim et al. (2021c). The audio, sampled at 16kHz, is converted into a spectrogram using a window size of 20 ms and a hop length of 10 ms. We augment the video data by applying random spatial erasure and time masking for effective modeling of the visual context (Mira et al., 2022
All models are trained using noisy-clean speech pairs where, speech samples from LRS3 are mixed with noise samples from the DNS Challenge (Reddy et al., 2021) noise set. The noisy mixture is obtained by randomly mixing up to 5 different noise samples. The SNR. range for mixing is -15 dB to 10 dB. We report results under 2 test conditions, (a) 3 background noises (3-BN) are present in the noisy mixture, and (b) 5 background noises (5-BN) are present. Evaluations are done at five different SNRs (in dB): 5, 0, -10, and -15.
Architectural and Training details. We use different backbones of audio-only/A sullyaudiovisual models for comprehensively evaluating our proposed MUTUD on the speech enhancement task. of the backbone models are inspired from the U-Net architecture design of gated convolutional recurrent network (GCRN) (Tan & Wang 2019) and the corresponding audiovisual model (Mira et al., 2023). The Audio-only enhancement model here is U-Net style encoder-decoder architecture. The input to the audio model is complex spectrogram of the audio. The audio encoder is composed of stacking of 4 gated convolutional blocks; which consists of two 2D convolutional layers, where the outputs of each convolutional layer, one followed by Sigmoid activation, are multiplied. The decoder includes an LSTM layer. The Audiovisual model (Mira et al. 2023) is built on top of this Audio-only model by employing a 3D convolutional layer followed by a ResNet (He et al.) 2016) as the video encoder. The video and audio encoder outputs are concatenated and forwarded through the decoder to produce complex spectrograms of the enhanced audio. The concatenated audio features and video features are taken into a 2- 2-layer Grouped LSTM. The decoder consists of 5 deconvolutional layers with a skip connection like U-net architecture. The encoder-decoder structure is designed in symmetric way, where the number of kernels progressively increases in the encoder and decreases in the decoder. To aggregate the context along the frequency direction, a stride of 2 is adopted along the frequency, dimension in all convolutional and up-convolutional layers. For MUTUD, K = 4 and we set the number of codes, in the MSCs to 32 after conducting an ablation study for different N (Sec. 5.5). We train using AdamW optimizer (Kingma& & Ba, 2014) with a learning rate of 10-4, We adopt a cosine scheduler (Loshchilov & Hutter 2016), adding warmup for 20 epochs. Loss function hyperparameter À is set to 0.01.
We also use VisualVoice (Gao & Grauman 2021 another type of backbone for audiovisual speech enhancement and follow the original architecture details and implementations.
Evaluation metrics. We utilize three standard speech quality and intelligibility metrics for AVSE: Short Time Objective Intelligibility (STOT) (Taal et al. (010)), Scale-Invariant Signal-to-DDisoortio Ratio (STSDR) (Le Roux et al., 2019), Perceptual Evaluation of Speech Quality (PESQ) (Rix et al. (000)..
5 Results and Discussions
5.1 Effectiveness of MUTUD
Table I presents quantitative results for the AVSE task under 3-background noise test conditions. A few important details about the reported methods are in order. For a fair comparison, in addition to Audio-only, we also report. the performance of Audio-only (w. matched params), that is, a model with the number of parameters matched with MUTUD. This is important to establish that the proposed TAME module is in fact providing crucial information not present in the audio modality and cannot be compensated for by simply adding more parameters to the Audio-only model. We show results for two versions of MUTUD representing two different training mechanisms: One where we train the entire model from scratch, denoted by w.o. pretrained TAME. Another is where we first pre-train the TAME module solely with clean audio and video frames and then fine-tune the entire model for the enhancement task. This is done to better guide the TAME module to store modality-speciffe information in the MSCs.
It clear from Table D that our proposed framework MUTUD outperforms both the Audio-only and the A Audio-only with matched parameters over all metrics and SNRs. This shows that the model has learned
8

---

Published in Transactions on Machine Learning Research (12/2025)
Table Speech Enhancement performance comparison of different models for 3-BN test condition.
STOI (%)
SISDR (dB)
PESQ
Method
-10
-15
-5
-10
-5
-15
Noisy Audio
82.6
72.4
60.5
48.8
38.9
5.00
0.01
-5.02
-10.03
-15.07
1.24
1.12
1.07
1.07
1.07
Audio-only
92.7
88.1
80.1
67.5
51.5
13.64
10.55
7.08
2.88
-2.82
2.18
1.80
1.48
1.27
1.14
Audio-only
93.0
88.3
80.4
68.0
52.6
13.75
10.58
7.05
2.86
-2.62
2.30
1.87
1.53
1.30
1.16
w. similor # poroms
MUTUD
93.4
89.1
81.6
69.5
53.6
14.07
10.99
7.54
3.32
-2.24
2.37
1.93
1.58
1.33
1.17
n.o0 pretoiined TAME
MUTUD
93.5
89.2
81.8
69.8
54.0
14.11
11.02
7.56
3.38
-2.19
2.36
1.92
1.57
1.32
1.17
Audiovisual
93.5
83.3
74.0
62.7
13.92
10.90
7.61
3.81
2.35
1.38
MUTUD Audio-3 63M Audio
92.7
91.3
86.0
74.2
52.9
95.4
92.4
87.3
76.2
57.6
Method
100.0
97.7
93.4
83.3
63.4
Audiovizual
100.0
100.0
100.0
100.0
100.0
SNR Level (dB)
Figure 2: MUTUD bridges the gap between Audiovisual and Audio-only models. Performance (in %) of gain in STOI through different methods relative to the gain Audiovisual brings in average intelligibility (STOT) of noisy speech samples. MUTUD is able to get most of performance gains of the Audiovisual model across different SNRs. For example, at -5dB SNR Audio-only gives gives 86.0% of the gain of the Andioivisual model whereas MUTUD at 93.4% of Audiovisual model.
with visual information available during training and MUTUD is able to estimate video encodings and use them for better enhancement. It is worth mentioning that for extremely low SNRs of -10dB and -15dB, where multimodal models heavily rely on visual information for speech enhancement, MUTUD continues consistently perform better than the Audio-only model. Figure shows each model's performance relative the Audiovisual model, the gain Audiovisual model brings in speech intelligibility (STOI) at different SNRs. As one can see at each SNR, the MUTUD model is able to recover most of the Audiovisual performance, without directly using visual signal during inference. This further highlights the TAME module's 's ability to estimate relevant visual information at prediction time. While we do not expect the MUTUD model to outperform or fully match the performance of the Audiovisual model, it does an excellent job reducing the gap between the unimodal and multimodal models. Except for extremely low SNR (-15dB), MUTUD fairly competitive with the Audiovisual model on all 3 metrics. This further argues for our multinodal training and unimodal deployment strategy. We also observe that the pre-trained TAME module is slightly superior to the one simply trained from scratch.
We conduct additional experiments to further verify the effectiveness of MUTUD, We tackle a more challenging condition with a 5-background noise test. Shown in Table 2 we observe similar trends for the 5-background noise test conditions showing that MUTUD can be successfully employed such extreme noise conditions.
Spectrogram Visualization: We visualize the spectrogram of each model illustrate the improvements over the baseline in Figure The advantage of MUTUD over Audio-only is most visible at SNR -10/- - 15 dB:

---

Published in Transactions on Machine Learning Research (12/2025)
Table Speech Enhancement performance comparison of different models for 5-BN test condition.
STOI (%)
SISDR (dB)
PESQ
Method
-5
-10
-15
0
-10
-15
-5
-10
-15
Noisy Audio
81.7
70.8
58.2
46.0
36.3
5.00
0.00
-5.00
-00.00
-15.02
1.21
1.10
1.06
1.06
1.08
Audio-only
92.2
87.0
78.3
64.4
47.0
13.28
10.08
6.47
1.99
-4.17
2.16
1.74
1.43
1.23
1.11
MUTUD
92.7
87.7
79.3
65.5
48.1
13.45
10.32
6.72
2.27
-3.88
2.24
1.8
1.47
1.25
1.12
pretrained TAME
MUTUD
92.8
88.0
79.6
65.6
48.0
13.60
10.43
6.85
2.33
-3.86
2.23
1.80
1.47
1.25
1.12
Audiovisual
92.9
88.6
81.6
71.3
58.9
13.43
10.37
6.95
2.90
-2.23
2.25
1.85
1.53
1.30
1.16
SWR 5 dB
SNR dB
SNR 15 dB
SNR -10
SNR 15 dB
9002
2302
0900
cnly
0901
0003
Audiorbuun
1000
1003
055
Figure 3: Spectrograms for noisy audio, Audio-only, MUTUD, Audiovisual, and clean audio (rows) at SNR
5, 0, 5, 100 155 dB (columns).
MUTUD preserves thinner, more continuous harmonics and darker inter-harmonic valleys (lower noise floor), and it shows a cleaner FO trajectory with reduced low-frequency clutter. At -5 dB, MUTUD further reduces high-frequency speckle and keeps consonant transitions (short vertical stripes) sharper than Audio-only. Across all SNRs, MUTUD remains consistently closer to Audiovisual than Audio-only.
Different Backbone Models: We analyze the robustness the TAME module with different architectural backbones. The first one smaller Audio-only architectural backbone inspired again from the GCRN (Tan & Wang 2019) Audio-only model. To do so, we reduce the output dimension of the last two layers of the the audio encoder from 128 to 64. The visual part of the corresponding Audiovisual model remains same before. This results in Audio-only and Audiovisual models with 0.815M and 12.195M parameters, respectively. Rosults in Table B verify the robustness of TAME module which showcases quantitative trends similar to those observed before, MUTUD is able to bridge the gap between the Audio-only model and the Audiovisual model. Finally, we adopt a different baseline, VisualVoice (Gao & Grauman 2021), in order demonstrate our method's flexibility with the underlying network architectures for the AVSE task. We successfully verify that, shown in Table g our proposed model can be adapted a different bascline model achieving superior performance to the Audio-only model.
10

---

Published in Transactions on Machine Learning Research (12/2025)
Table 3: Speech Enhancement performance comparison for Backbone 2. A smaller backbone, that is, the Audio-only and the corresponding audiovisual models used here are smaller and weaker.
STOI (%)
SISDR (dB)
PESQ
Method
-5
-10
-15
-5
-10
-15
-5
-10
-15
Noisy Audio
82.6
72.4
60.5
48.8
38.9
5.00
0.00
-5.0
-10.08
-15.06
1.24
1.12
1.07
1.07
1.07
Audio-only
91.9
86.6
77.7
64.5
49.3
12.98
9.68
5.99
1.58
-4.14
2.12
1.73
1.44
1.24
1.13
MUTUD
92.3
87.3
78.9
66.0
50.3
13:31
10.07
6.45
2.08
-3.65
2.20
1.79
1.49
1.27
1.14
W.o. pretrained TAME
MUTUD
92.3
87.3
78.9
66.0
50.5
13.29
10.04
6.43
2.06
-3.66
2.22
1.81
1.49
1.27
1.14
Audiovisual
92.6
88.2
81.1
71.1
59.9
13.23
10.10
6.65
2.72
-2.01
2.15
1.80
1.51
1.31
1.18
Table Speech Enhancement performance comparison for Backbone VisualVoice as backbone for Audio-only, audiovisual, and MUTUD.
STOI (%)
SISDR (dB)
PESQ
Method
-5
-10
-15
5
0
-5
-10
-15
5
0
-5
-10
-15
Noisy Audio 82.6
72.4
60.5
48.8
38.9
5.00
0.00
-5.00
-10.03
-5.066
1.24
1.12
1.07
1.07
1.07
Audio-only
91.8
85.9
76.5
64.0
48.8
11.67
8.62
5.17
1.08
-.90
2.05
1.60
1.32
1.17
1.09
MUTUD
93.5
89.0
82.0
71.3
56.5
12.72
9.68
6.53
2.98
-2.04
2.40
1.94
1.58
1.33
1.18
Audiovisual
94.0
90.1
84.1
75.4
64.3
12.92
9.94
6.90
3.54
-0.86
2.51
2.03
1.65
1.39
1.21
5.2 Generalization to Audio-only Datasets
Our MUTUD models as well as the Audio-only and Audiovisual models are trained and evaluated on the LRS3 audiovisual dataset. To understand generalization capabilities of these models we evaluate them on an out-of-domain Audio-only dataset and also compare them with other state-Che-arl methods this dataset. We use the well-established DNS Challenge eval set (Reddy et al. 2020) for this evaluation. There are few points worth noting from Table 5 Unlike other methods in the table, our Audio-only is not trained on DNS Challenge set, yet the performance is competitive with other state-of-the-arl methods. Moreover, our model is causal and relatively small, unlike some of the other models in the table. Overall, it shows that our Audio-only speech enhancement experimental backbone is a strong and competitive backbone model. The MUTUD model improves over the Audio-only model on this evaluation set well. Furthermore, this independent, evaluation on Audio-only dataset shows that MUTUD generalizes well and not limited to audiovisual data seen during training. It also evidences practicality and usefulness of MUTUD in real-word, as this test setting is how the model will be used in real-world.
5.3 Efficiency Analysis
Table 6 shows parameter and Multiply Accumulate Operations (MAC) counts for all models. The MUTUD model is comparable in size and compute to both Audio-only models. In fact, the MAC for MUTUD is around 13% lower compared to even the Audio-only model with a matching parameter count. However, we saw Table [ that MUTUD much more superior compared to these models. With respect to the Audiovisual model, MUTUD is smaller almost by a factor of and has smaller size and MAC by around 83% and 77% respectively. This shows the massive gain in efficiency one can achieve through our MUTUD learning framework.
11

---

Published in Transactions on Machine Learning Research (12/2025)
Table 5: Performance comparison on DNS Challenge evaluation (no-reverb) set.
Method
PESQ SI-SDR STOI # Params
Noisy
1.58
9.07
0.92
FullSubnet Hao et al.
2021
2.77
17.29
0.96
5.6M
CleanUNet Kong et al.
2022
3.15
0.96
46.07M
Demucs
Defossez al.
2020
2.66
0.97
33.53M
NSNet (iia et al.
2020
2.15
15.61
0.94
5.1M
Audio-only (Ours)
2.32
16.21
0.94
2.98M
MUTUD (Ours)
2.56
17.50
0.96
3.63M
Table 6: Number of Parameters and Multiply Accumulate Operations (MACs) for all models.
Audio-only matched params
Audio-only
Audiovisual MUTUD
# of Param.
2.978M
3.627M
15.736M
3.635M
MACs
1.381G
1.821G
9.324G
1.593G
Inference Time (ms) 98.1 + 2.87
100.2 + 2.50
206.0 + 8.1
108.0 2.1
5.4 TAME Module Analysis
To analyze the estimated video features from the TAME module, we measure how similar they are to the original video and audio features. We compute the average cosine similarity and & distance between video features and estimated video features (F, vs. Fe), video features and audio features (F, vs. F.), and estimated video features and audio features (F. vs. Fa) for SNRs ranging from 5dB to -1dBB. Figure 4 clearly indicates that the cosine similarity between the estimated video features and the original vidco fcatures is high, around 0.94, while the similarity between audio and original (estimated) video features is low, s 0.40 (== 0.42). The l distances show a similar trend where the audio and video features are further apart, and the estimated vidco features and the original ones are much closer. The high similarity between the estimated and original video features, while having low similarity between the estimated video and audio features evidence that TAME is not just regurgitating audio features but is actually functioning as designed (use audio information to get video information).
In addition, in Figure 5 we show the t-SNE visualization of the estimated video features, the original video features, and the audio features for all SNRs. Analyzing the clusters, we can clearly observe that the audio features F. form a distinct group, separate from the estimated video features F, and the actual video feature F., demonstrating that the TAME can differentiate between modality-specific characteristics. More importantly, the estimated video features F. and the actual video features Fg are clustered closely together the feature space, implying that the TAME module can accuratcly retricve vidco fcatures from the memory block, closely mirroring the actual video features even the SNR levels decrease.
We further analyze the distribution across the audio and video codebooks for all K. Figure 6 shows a visualization of all the learned codebooks C and C for audio and video. We visualize the mcan of all 32 codebooks for each k. One key inference from this visualization is that all codes are well-represented and the training formulation does not lead to mode collapse. This is further evidenced through the visualization of the probability distribution q for a sample noisy audio frame Figure 7 Figure 7 shows the variation the usage pattern of different codebooks by just single frame of audio and shows that the codes are not collapsing and are learning to capture the expected information.
12

---

Published in Transactions on Machine Learning Research (12/2025)
Cosine Similarity
L2 Distance
1.1
7.0
0.9
0.5
vs.
4.0
0.3
w..F
3.0
0.1
2.0
0.
1.0
0
-10
-15
-10
-15
Figure 4: Cosine similarity (red) and la distance (blue) between video features and estimated video features, vidco and audio features, and estimated vidco and audio features for different SNRs.
snr-10
Figure 5: TSNE visualization of the estimated video features Fer the actual video features Fu. and the audio features Fa for SNRs ranging from 5dB to -15BB.
5.5 Ablation for Codebook Size
We perform ablation for the size of the codebooks in MSCs. We experiment with 4 different codebook sizes, N (8, 16, 32, and 64) in each MSC of the TAME module. Table 7 indicates that as the size increases, more gain in speech enhancement performance is achieved, meaning that larger number of codes in the MSCs can contain more meaningful features. We see that the = 64 does not get much performance gain over = 32. N = 32 sufficient for embedding the audio and video information into the codebooks and then relating them enable estimation of vidco represcntations using audio.
Table Ablation on different numbers of codes, N, in each MSC of TAME.
STOI (%)
SISDR (dB)
PESQ
# of codes
0
-5
-10
-15
5
0
-5
-10
-15
5
-5
-10
-15
93.3
88.7
80.9
68.6
53.0
13.91
10.71
7.20
3.02
-2.39
2.36
1.91
1.56
1.32
1.17
16
93.4
88,8
81.1
68.9
53.3
13.98
10.81
7.30
3.10
2241
2.35
1.92
1.57
1.32
1.17
89.2
81.8
54.0
14.11
11.02
7.56
3.38
-2.19
2.36
1.92
1.37
1.32
1.17
64
93.6
89.1
81.6
69.6
54.0
14.11
11.00
7.51
3.31
-2211
2.38
1.95
1.59
1.34
1.18
Table 8: MUTUD effectiveness in Audiovisual Speech Recognition (AVSR) task.
WER (%)
Method
5
-5
-10
-15
Audio-only
12.24
17.836
31.37
60.64
93.32
Audiovisual
5.26
7.088
11.01
21.12
36.56
MUTUD
11.71
16.299
24.99
44.07
73.56
13

---

Published in Transactions on Machine Learning Research (12/2025)
kel
kr2
krd
0.004
0.60
1344
01003
0.01
0.303
0.02
0.000
0.000
-0.001
0000
Audio codebook
kd7
2.0
20
2.0
1.5
1.5
1.5
1/0
1.0
0.5
0.0
180
0.
111
00
10 25
25
-05
10
15
20
25
10
16
20
25
10
15
20
25
Video codebook
Figure 6: Visualization of the learned audio and video codebooks (Co and C*)) The plots show the mean of each code in all K(= 4) codebooks.
012
0.20
0.20
025
0.15
6.15
6.15
0.10
010
606
010
8m
Probabillyy distribution the relationship between audio codebook and the corresponding audio modality representation
Figure 7: Visualization of the probability distribution (for each k) for a sample noisy audio frame.
5.6 Audiovisual Speech Recognition (AVSR)
To showcase our method's versatility, we additionally show results on different downstream tasks: AVSR and AV-ASD to verify the effectiveness of the proposed TAME module. In this subsection, we will show the experimental details for the AVSR task and the performance analysis.
5.6.1 Implementation Details
bascline architecture. The vidco encoder in the V-CAFE architecture consists of a 3D convolution layer with Batch Normalization and Max-pool followed by ResNet-18 He et al.) 2016), and the audio encoder contains two 2D convolution layers followed by one ResBlock for the audio front-end. The input shape of the model is the same as the Audiovisual Speech Enhancement model. VCAFE consists of a cross-modd modal attention followed by a noise reduction mask. The noise reduction mask consists of two convolution layers with ReLU and Sigmoid activation respectively. The mask is multiplicated to the audio fcatures fa, and the masked audio features are summed with the original features to obtain the enhanced audio features. Finally, with the enhanced audio features and the visual features are concatenated with the linear layer and taken into Conformer (Gulati et al. 2020 for the encoder and Transformer (Serdyuk et al. 2022) for the decoder for predicting the speech.
For AVSR we adopt V-CAFE (Hong al. 2022)
The Conformer (Gulati et al. 2020 sequence encoder is composed of hidden dimensions of 512, leed-forward
dimensions of 2048, 12 layers, attention heads, and a convolution kernel size of 31. The Transformer [Serdyuk et al. 2022) sequence decoder contains hidden dimensions of 512, feod-forward dimensions of 2048, layers, and attention heads are employed. Note that the video features are upsampled with the nearest neighbor interpolation to match the size of the audio features when taken into the proposed TAME Module.
14

---

Published in Transactions on Machine Learning Research (12/2025)
Table 9: MUTUD effectiveness in Audiovisual Active Speaker Detection (AV-ASD) task.
Method
mAP (%)
SyncNet CChung & Zissorman,
2017
82.1
TalkNet (Too et al. 2021
79.9
SPELL+ (Min et al. 2022
85.9
Audiovisual EgoASD (uuh et al.
2025
87.6
Video-only EgoASD
82.3
MUTUD EgoASD
86.5
V-CAFE achieves a Word Error Rates (WER) 2.9% on LRS3 tests when trained on only LRS3 dataset, which is competitive with other works such as (Ma al. 2021b). Other works such as (Shi et al.) 2022] Serdyuk et al. 2022) much larger additional training data for slightly better WER. This shows that V-CAFE is simple and reliable backbone for uses in experiments.
To make results more insightful we show results under noisy conditions. We utilize background noises in diverse environments of DEMAND (Thiemann et al. 2013) dataset with SNR range randomly chosen from -15dB to 15dB for training. For testing, we report the testing performance at five different SNRs (in dB): 5, 0, -10, and -15, measuring speech recognition quality through (WER).
5.6.2
Performance Analysis
As shown in Table 8 MUTUD, while not outperforming the AV approach, shows a substantial reduction in WER compared to the Audio-only method, highlighting TAME module's contribution learning to leverage visuals even if it is available only during training. This is especially true for low-SNRs where visuals play more important roles and MUTUD can help reduce WER by a considerable margin (6% for -5dB and 16% for -1dBB in absolute terms). The reduced WER across various levels of background noise indicates that the TAME module effectively utilizes visual information to complement audio input, thus enhancing overall speech recognition accuracy.
5.7 Audiovisual Active Speaker Detection (AV-ASD)
We additionally conduct the experiment the AV-ASD task. In the AV-ASD task, we assume the absence of audio modality at inference time, instead of the visual modality as done in previous experiments. We show results on the EasyCom dataset, a considerably more challenging real-world noisy dataset than the LRS3-TED dataset.
5.7.1 Implementation Details
The AV-ASD model follows Huh et al. 2025 It consists of mouth keypoint detector to crop the lip region, CNN-based video, and audio encoders, and a fusion layer followed by causal temporal layer to incorporate a longer temporal past context. For the mouth keypoint detector, we adopt the ground truth facial per speaker manually checked by annotators. From the keypoints, we generate a new face crop by cropping the region by half of the width of the face region horizontally and also crop quarter of the height downwards and threo-quarters of the height upwards from the center of the mouth. We also generate a lip region, cropping the same way horizontally but cropping quarter of height up and down from the mouth center.
The audio encoder is adopted from VGG-M (Chatfield et al. 2014) operating 13-dim Mel-Frequent Cepstral Coefficient (MFCC). For the video encoder, we use a spatio-temporal VGG-M (Chatfield et al. 2014 composed of a 3D convolutional layer followed by a stack of 2D convolutions. We also adopt a Self-Attentive Pooling (SAP) layer (Bhattacharya et al. 2017) for fusing the output audio and video features. Lastly, we set unidirectional LSTM layer for temporal modeling to scquentially process consocutive embeddings from the fusion layers predict speech activity corresponding to the latest frame followed by projection layer and a sigmoid activation to derive activity predictions for each target speaker. For TAME module integration, like
15

---

Published in Transactions on Machine Learning Research (12/2025)
the AVSR model, the video features are upsampled with the nearest-neighbor interpolation to match the number of the audio features.
For training, we apply horizontal flipping, random rotation within -1 ~ +15", and motion blur augmentation with kernels randomly from 10, 25, 50, and 100. Due to the limited size of Easycom, we firstly pretrain the model with a larger dataset, VoxCeleb2 (Chung et al.] 2018), to produce a better performance and generalization. We train using SGD optimizer (Robbins & Monro 1951] with learning rate of 5 with a weight decay of 5- We evaluate the performance using the mean Average Precision (mAP).
5.7.2 Performance Analysis
As indicated in Table g. the model showcases a mean Average Precision (mAP) of 86.50, which sits comfortably between the Vidco-only method at 82.25% and the AudioVisual approach 87.60%. Notably, this outcome demonstrates the TAME's capability properly retrieve audio features, complementing the previously illustrated proficiency in video feature retrieval. Therefore, the comparative performance indicates that the proposed TAME is not only effective in leveraging visual information but also exhibits reciprocal competence in audio feature retrieval, thereby reinforcing its applicability in multimodal scenarios. Thus, the experimental results underscore the versatility of the proposed TAME module.
6 Conclusion
This work motivated address practical challenges in using multimodal solutions in real-world applications. We build and train the models keeping in mind that inference will be unimodal multimodal training but unimodal deployment strategy, and propose MUTUD. In MUTUD, the model learns associate and relate different modalities through modality-specifie codebooks. Once this is achieved during training, the representations of modality absent during inference are obtained using the one present during inference. We show substantial gains over corresponding unimodal models and efficiency gains over full multimodal counterparts while retaining performance to a considerable exlent. Moreover, our framework and TAME are fairly generic and can be easily adapted for other common multimodal learning tasks and models. We can also extend MUTUD to more than two modalities through pairwise MSCs.
References
Triantafyllos Afouras, Joon Son Chung, and Andrew Zisserman. The conversation: Deep audio-visual speech enhancement. arXiv preprint 1xXx:1804.0442// 2018a.
Triantafyllos Afouras, Joon Son Chung, and Andrew Zisserman. Lrs3-ted: a large-scale dataset for visual speech recognition. arXiv preprint orXiv: 00496, 2018b.
Tadas Baltrušaitis, Chaitanya Ahuja, and Louis-Philipee Morency. Multimodal machine learning: A survey and Laxonomy. IEEE transactions on paltern analysis and machine intelligence, 41 41(2):423 443, 2018.
Gautam Bhattacharya, Md Jahangir Alam, and Patrick Kenny. Deep speaker embeddings for short-duration speaker verification. In Interspeech, pp. 1517-1521, 2017.
Sebastian Braun, Hannes Gamper, Chandan KA Reddy, and Ivan Tashev. Towards efficient models for real-time deep noise suppression. In ICASSP 2021-2021 IEFE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 656 660. IEEE, 2021.
Douglas Burnham, Ruth Campbell, G Away, and BJ Dodd. Hearing eye I1: the psychology of speechreading and auditory-visual speech. Psychology Press, 2013.
Punarjay Chakravarty, Jeroen Zegers, Tinne Tuytelaars, and Hugo Van hamme. Active speaker detection with audio-visual co-training. In Proceedings of the 18th ACM International Conference on Multimodal Interaction, pp. 312-316, 316, 2016.
Oscar Chang, Otavio Braga, Hank Liao, Dmitriy Serdyuk, and Olivier Siohan. On robustness to missing video for audiovisual speech recognition. Transactions on Machine Learning Research, 2022. TSSN 2835-8856.
16

---

Published in Transactions on Machine Learning Research (12/2025)
Ken Chatfield, Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Return of the devil in the details: Delving deep into convolutional nets. arXiv preprint or 4rXi::1405.333,, 2014.
Shang-Y Yi Chuang, Yu Tsao, Chen-Chou Lo, and Hsin-Min Wang. Lite audio-visual speech enhancement. arXiv preprint vrXiv:2005.1166. 2020.
Joon Son Chung and Andrew Zisserman. Out of time: automated lip syne in the wild. In ACCV 2016 Workshops, pp. 251-263. Springer, 2017.
Joon Son Chung, Arsha Nagrani, and Andrew Zisserman. Voxceleb2: preprint ar Xiv: 05622, 2018.
Deep speaker recognition. arxiv
Ross Cutler and Larry Davis. Look who's talking: Speaker detection using video and audio correlation. In 2000 IEEE International Conference on Multimedia and Expo. ICME2000. Proceedings. Latest Advances in the Fast Changing World of Multimedia (Cat. No. 00TH8532), volume 3, pp. 1589 1592. IEEE, 2000.
A Alexandre Defossez, Gabriel Synnaeve, and Yossi Adi. Real time speech enhancement in the waveform domain. arXiv preprint arXiv:2006.128 2020.
Jacob Donley, Vladimir Tourbabin, Jung-Suk Lee, Mark Broyles, Hao Jiang, Jie Shen, Maja Pantic, Vamsi Kr- ishna Ithapu, and Ravish Mehra. Easycom: An augmented reality dataset to support algorithms for easy communication in noisy environments. arXiv preprint arXim:2107.04177, 2021.
Aviv Gabbay, Asaph Shamir, and Shmuel Peleg. Visual speech enhancement. arXiv preprint Xiv: mr.Xi::711.08755 2017.
Ruohan Gao and Kristen Grauman. Visualvoice: Audio-visual speech separation with cross-modal consistency. In 2021 IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 15500. TEEE, 2021.
Ashutosh Garg, Vladimir Pavlovic, and James M Rehg. Audio-visual speaker detection using dynamic bayesian networks. In Proceedings Fourth IEEE International Conference on Automatic Face and Gesture Recognition (Cal. No. PR00580), pp. 384 390..90.10 IEEE, 2000.
Mandar Gogate, Kia Dashtipour, Ahsan Adeel, and Amir Hussain. Cochleanet: A robust language-indepeedent audio-visual model for real-time speech enhancement. Information Fusion, 63:273 285, 2020.
Anmol Gulati, James Qin, Sluug-Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, et al. Conformer: Convolution-augnenntel transformer for speech recognition. arXiv preprint Xrxvv:2005. 08100, 2020.
Xiang Hao, Xiangdong Su, Radu Horand, and Xiaofei Li. Fullsubnet: A full-band and sub-band fusion model for real-time single-channel speech enhancement. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6633-6637. IEEE, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770 (007778, 2016.
Sindhu B Hegde, KR Prajwal, Rudrabha Mukhopadhyay, Vinay P Namboodiri, and CV Jawahar. Visual speech enhancement without real visual stream. In Proceedings of the IEEE/CVF Winter Conference Applications of Compuler Vision, pp. 1926-19669 19066195.1111 2021.
Joanna Hong, Minsu Kim, Se Jin Park, and Yong Man Ro. Speech reconstruction with reminiscent sound via visual voice memory. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:3654-3668 2021.
Joanna Hong, Minsu Kim, Dachun Yoo, and Yong Man Ro. Visual context-driven audio feature enhancement for robust end-to-end audio-visual speech recognition. aix preprint arXiv:2207.000 06020, 2022.
17

---

Published in Transactions on Machine Learning Research (12/2025)
Joanna Hong, Minsu Kim, Jeongsoo Choi, and Yong Man Ro. Watch or listen: Robust audio-visur speech recognition with visual corruption modeling and reliability scoring. In Proceedings of the TEEE/CR /VF/ Conference on Computer Vision and Pattern Recognition, pp. 18783 18794, 2023.
Jen-Cheng Hou, Syu-Siang Wang, Ying-Hui Lai, Yu Tsao, Hsiu- Chang, and Hsin-Min Wang. Audio-visual speech enhancement using multimodal deep convolutional neural networks. IEEE Transuctions on Emerging Topics in Computational Intelligence, 2(2):117-122) 2018.
Wei-Ning Hsu and Bowen Shi. u-hubert: Unified mixed-modal speech pretraining and zero-shot transfer to unlabeled modality. Advances in Newral Information Processing Systems, 35:21157 21170, 2022.
Wei-Ning Hsu, Tal Remez, Bowen Shi, Jacob Donley, and Yossi Adi. Revise: Self-supervised speech resynthesis with visual input for universal and generalized speech enhancement. arXiv preprinl arXiv:2212.1133 2022.
Jing Huang and Brian Kingsbury, Aisial deep learning for noise robust speech recognition. In 2013 IEEE international conference on acoustics, speech and signal processing, pp. 7596-7599. IEFE, 2013.
Jaesung Huh, Juan Azcarreta Ortiz, Anurag Kumar, Ashutosh Pandey, Ali Aroudi, Daniel D.E. Wong, Francesco Nesta, Buye Xu, and Jacob Donley. Advancing active speaker detection for egocentric videos. IEEE, 2025.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pp. 448-456. pmlr, 2015.
V Ahmadi Kalkhorani, Anurag Kumar, Ke Tan, Buye Xu, and D Wang. Time-domain transformer-based audiovisual speaker separation. In Proc. INTERSPEECH, pp. 3472 3476, 2023.
Chanwoo Kim, Dhananjaya Gowda, Dongsoo Lee, Jiyeon Kim, Ankur Kumar, Sungsoo Kim, Abhinav Garg, and Changwoo Han. A review of on-device fully neural end-to-end automatic speech recognition algorithms. In 2020 54th Asilomar Conference on Signals, Syslems, and Compulers, pp. 277-283 IEEE, 2020.
Minsu Kim, Joanna Hong, Se Jin Park, and Yong Man Ro. Cromm-vsr: Cross-modal memory augmented visual speech recognition. IEEE Transactions on Multimedia, 24:4342-4355, 2021a.
Minsu Kim, Joanna Hong, Se Jin Park, and Yong Man Ro. Multi-modality associative bridging through memory: Speech sound recollectod from face video. In Proceedings of the IEEE/CVF International Conference on Compuler Vision, pp. 296-3006 306, 2021b.
Minsu Kim, Joanna Hong, and Yong Man Ro. Lip to speech synthesis with visual context attentional gan. Advances in Newral Information Processing Systems, 34, 2021c
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. ar preprint orXiv:1412.688 6980, 2014.
Zhifeng Kong, Wei Ping, Ambrish Dantrey, and Bryan Catanzaro. Speech denoising in the waveform domain with self-attention. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 7867-7871. IEEE, 2022.
Jonathan Le Roux, Scott Wisdom, Hakan Erdogan, and John R Hershey. Sdr half-baked or well done? In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 626-606666666 630. IEEE, 2019.
Jun-Tac Lec, Mihir Jain, Hyoungwoo Park, and Sungrack Yun. Cross-attentional audio-visual fusion for weakly-supervised action localization. In International conference on learning representations, 2020.
Lukas Lee, Youna Ji, Minjac Lee, Min-Seok Choi, and Naver Coporation. Demucs-mobile On-device lightweight speech enhancement. In Interspeech, pp. 2711 2711-2715, 2715, 2021.
18

---

Published in Transactions on Machine Learning Research (12/2025)
Yi-Lum Lee, Yi-Hsuan Tsai, Wei-Ches Chiu, and Chen- Yu Lee. Multimodal prompting with missing modalities for visual recognition. In Proceedings of the IEFE/CVF Conference on Compuler Vision and Paltern Recognition, pp. 14943 14952, 2023.
lyya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint urXiv:1608.0388 2016.
Zhou Lu. A theory of multimodal learning. Advances in Neurol Informalion Processing Systems, 2023.
Mengmeng Ma, Jian Ren, Long Zhao, Sergey Tulyakov, Cathy Wu, and Xi Peng. Smil: Multimodal learning with severely missing modality. In Proceedings of the AAAT Conference on Artificial Intelligence, volume 35, pp. 2302 2310, 2021a.
Mengmeng Ma, Jian Ren, Long Zhao, Davide Testuggine, and Xi Peng. Are multimodal transformers robust missing modality? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18177-1888 18186, 2022.
Pingchuan Ma, Stavros Petridis, and Maja Pantic. End-to-end audio-visual speech recognition with conformers. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 7613 7617. IEEE, 2021b.
Marina Maayah, Ahlam Abunada, Khawla Al-Janahi, Muhammad Ejaz Ahmed, and Junaid Qadir. Limitaccess: on-device tinyml based robust speech recognition and age classification. Discover Artificial Intelligence, 3 (1):8, 2023.
Daniel Michelsanti, theng-HHa Tan, Shi-Xiong Zhang, Yong Xu, Meng Yu, Dong Yu, and Jesper Jensen. An overview of deep-learning- audio-visual speech enhancement and separation. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:1368 19:1368-1396, 2021.
Kyle Min, Sourya Roy, Subarna Tripathi, Tanaya Guha, and Somdeb Majumdar. Learning long-tern spatial-temporal graphs for active speaker detection. In European Conference on Computer Vision, pp. 371-387. Springer, 2022.
Rodrigo Mira, Alexandros Haliassos, Stavros Petridis, Björn W Schuller, and Maja Pantic. Svts: Scalable video-to-speech synthesis. orXiv preprint ar arXiv:2205.0000 2022.
Rodrigo Mira, Buye Xu, Jacob Donley, Anurag Kumar, Stavros Petridis, Vamsi Krishna Ithapu, and Maja Pantic. La-voce: Low-snr audio-visual speech enhancement using neural vocoders. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023.
Youssef Mroueh, Etienne Marcheret, and Vaibhava Goel. Deep multimodal learning for audio-visual speech recognition. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 2130 2134. IEEE, 2015.
Kuniaki Noda, Yuki Yamaguchi, Kazuhiro Nakadai, Hiroshi G Okuno, and Tetsuya Ogata. Audio-visual speech recognition using deep learning. Applied Intelligence, 2015.
Andrew Owens and Alexei A Efros. Audio-visual scene analysis with self-supervised multisensory features. Tn Proceedings of the European conference on computer vision (ECCV), pp. 631 648, 2018
Gerasimos Potamianos, Etienne Marcheret, Youssef Mroueh, Vaibhava Goel, Alexandros Koumbaroulis, Argyrios Vartholomaios, and Spyridon Thermos. Audio and visual modality combination in speech processing applications. In The Handbook of Multimodal-Multiseenno Interfaces: Foundations, User Modeling, and Common Modality Combinations- Volume 1, pp. 0499-54..2 2017.
R Gnana Praveen, Patrick Cardinal, and Eric Granger. Audio-visual fusion for emotion recognition in the valence-arousal space using joint cross-attention. IEEE Transactions on Biometrics, Behavior, and Identity Science, 2023.
19

---

Published in Transactions on Machine Learning Research (12/2025)
Chandan KA Reddy, Vishak Gopal, Ross Cutler, Ebrahim Beyrami, Roger Cheng, Harishchandra Dubey, Sergiy Matusevych, Robert Aichner, Ashkan Aazami, Sebastian Braun, et al. The interspeech 2020 deep noise suppression challenge: Datasets, subjective testing framework, and challenge results. In INTERSPEECH, 2020.
Chandan KA Reddy, Harishchandra Dubey, Kazuhito Koishida, Arun Nair, Vishak Gopal, Ross Cutler, Sebastian Braun, Hannes Gamper, Robert Aichner, and Sriram Srinivasan. Interspeech 2021 deep noise suppression challenge. Xiv preprint or 2ir::111101..0 100.0000222 01902, 2021.
A.W. Rix, J.G. Beerends, M.P. Hollier, and A.P. Hekstra. Perceptual evaluation of speech quality (pesq)-a new method for speech quality assessment of telephone networks and codecs. In 2001 IEEE International Conference on Acoustics, Speech, and Signal Processing. Proceedings (Cat. No.01 volume 2, pp. 749 752 vol.2, 2001. doi: N0.1109/ICCASSP200 2001.9402..
Herbert Robbins and Sutton Monro. A stochastic approximation method. The annals of mathematical statistics, pp. 400-407, 1951.
Joseph Roth, Sourish Chaudhuri, Ondrej Klejeh, Radhika Marvin, Andrew Gallagher, Liat Kaver, Sharadh Ramaswany, Arkadiusz Stopezynski, Cordelia Schmid, Zhonghua Xi, et al. Ava active speaker: An audio-visual dataset for active speaker detection. In ICASSP 2020- 2020 IEFE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 4492 4496. IEEE, 2020.
Jean-Luc Schwartz, Frédéric Berthommier, and Christophe Savariaux. Seeing to hear better: evidence for carly audio-visual interactions in specch identification. Cogmition, 2004.
Dmitriy Serdyuk, Otavio Braga, and Olivier Siohan. Transformer-based video front-ends for audio-visual speech recognition. arXiv preprint orXiv:2201.104 10439, 2022.
Bowen Shi, Wei-Ning Hsu, Kushal Lakhotia, and Abdelrahman Mohamed. Learning audio-visual speech representation by masked multimodal cluster prediction. arXiv preprint 3xXxx:2201.02180 2022.
Darryl Stewart, Rowan Seymour, Adrian Pass, and Ji Ming. Robust audio-visual speech recognition under noisy audio-video conditions. JEEE transactions on cybernetics, 44(2):175-184 2013.
Cees H Taal, Richard C Hendriks, Richard Heusdens, and Jesper Jensen. A short-time objective intelligibility measure for time-frequency weighted noisy speech. In 2010 IEEE internalional conference on acoustics, speech and signal processing, pp.
IEEE, 2010.
Ke Tan and DeLiang Wang. Learning complex spectral mapping with gated convolutional recurrent networks for monaural speech enhancement. IRFE/ACM Transactions on Audio, Speech, and Language Processing, 2019.
Ke Tan, Xueliang Zhang, and DeLiang Wang. Deep learning based real-time speech enhancement for dnal-microphone mobile phones. IEEE/ACM transactions audio, speech, and language processing, 29: 1853-1863 2021.
Ruijie Tao, Zexu Pan, Rohan Kumar Das, Xinyuan Qian, Mike Zheng Shou, and Haizhou Li. Is somcone speaking? exploring long-term temporal features for audio-visual active speaker detection. In Proceedings of the 29th ACM international conference multimedio, pp. 3927-3327 0935-,95,-- 2021.
Manthan Thakker, Sefik Emre Eskimez, Takuya Yoshioka, and Huaming Wang. Fast real-time personalized speech enhancement: End-to-end enhancement network (c3net) and knowledge distillation. arXiv preprint arXiv:2204.00771, 2022.
Joachim Thiemann, Nobutaka Ito, and Emmanuel Vincent. Demand: collection of multi-channel recordings of acoustic noise in diverse environments. In Proc. Meetings Acoust, pp. 1 6, 2013.
Zhong-Qiu Wang, Peidong Wang, and DeLiang Wang. Complex spectral mapping for single-and multi-channel speech enhancement and robust asr, IEEE/ACM transactions on audio, speech, and language processing, 08:1778-1781 2020.
20

---

Published in Transactions on Machine Learning Research (12/2025)
Liangfa Wei, Jie Zhang, Junfeng Hou, and Lirong Dai. Attentive fusion enhanced audio-visual encoding for transformer based robust speech recognition. In 2020 Asia- Paia-Pacific: Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC), pp. 638 643. IEEE, 2020.
Felix Weninger, Hakan Erdogan, Shinji Watanabe, Emmanuel Vincent, Jonathan Le Roux, John R Hershey, and Björn Schuller. Speech cnhancement with lstm recurrent neural networks and its application to noisc- robust asr. In Lalent Vuriable Analysis and Signal Separalion: 12th International Conference, LVA/ICA 2015, Liberec, Czech Republic, August 25-28, 2015, Proceedings 12, pp. 91-99. Springer, 2015.
Sangmin Woo, Sumin Lee, Yeonju Park, Muhammad Adi Nugroho, and Changick Kim. Towards good practices for missing modality robust action recognition. In Proccedings of the AAAT Conference Artificial Intelligence, volume 37, pp. 2776 2784, 2023.
Yangyang Xia, Scbastian Braun, Chandan KA Reddy, Harishchandra Dubey, Ross Cutler, and Ivan Tashev. Weighted speech distortion losses for neuraa-network-basse real-time speech enhancement. In ICASSP 2020-2020 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 871 875. IEEE, 2020.
Bo Xu, Cheng Lu, Yandong Guo, and Jacob Wang. Discriminative multi-modality speech recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pp. 14433 14442, 2020.
Karren Yang, Dejan Marković, Steven Krenn, Vasu Agrawal, and Alexander Richard. Audio-visual speech codecs: Rethinking audio-visual speech enhancement by re-synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pp. 8227-8237, 2022.
21


Reviews:
[{"review_id": "e8jSRmPDgo", "reviewer": "Reviewer_2T6j", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces a new framework called Multimodal Training and Unimodal Deployment (MUTUD), which allows models to leverage information from multiple modalities, such as audio and visual data, during training, but requires only a single modality during inference. Central to this approach is the Temporally Aligned Modality feature Estimation (TAME) module, which can estimate the features of missing modalities using information from the available ones at inference time. By using cross-modal codebooks, the TAME module enables the model to recall and reconstruct representations of absent inputs. This framework addresses key challenges in deploying multimodal systems, such as increased computational demands, complex data synchronization, and the need for multiple sensors during inference, by allowing unimodal deployment without significant loss in performance. The paper demonstrates that MUTUD can substantially narrow the performance gap between unimodal and multimodal models, while also achieving considerable reductions in model size and computational cost—sometimes by as much as 80% compared to full multimodal models. The approach is validated across several speech processing tasks, including speech enhancement, speech recognition, and active speaker detection, and consistently shows improved unimodal inference compared to models trained only on single modalities.", "claims_and_evidence_yn": "Yes", "claims_explanation": "1. The paper introduces codebooks that effectively link features between visual and audio modalities, enabling the estimation of missing features during inference to compensate for the absence of one modality.\n\n2. Experimental results demonstrate that the proposed model can reduce computational costs by up to 80% in certain cases.\n\n3. The experiments, as presented in Tables 1, 2, and 3, show that the method significantly improves performance even when only a single modality is available during inference.", "audience_yn": "Yes", "audience_explanation": "This method can also be extended to a wider range of audio-visual applications, such as video understanding, where one modality may be unavailable during real-world deployment.", "requested_changes": "The paper lacks relevant visualizations to illustrate the improvements over the baseline. The authors could enhance their analysis by visualizing the spectrograms of both the predictions and the ground-truth, providing a clearer comparison of model performance.", "broader_impact_concerns": "N/A", "additional_comments": "N/A", "extra_scores": {}, "extra_text": {}}, {"review_id": "0QBFqQyfbV", "reviewer": "Reviewer_fbgS", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduce a multi-modal alignment between the video and audio so that the model can leverage one or a few modalities during the inference. Extensive experiments are conducted to validate the performance of the algorithm \n\nStrength:\n- The idea of leveraging the video to assist the audio speech processing is interesting and the authors demonstrated the performance improvement \n- The organization and formulation of the problem setup is clear.\n\nWeakness:\n- Multimodal alignment with a CLIP-like loss is not new and it's not suprising to see that the modalities after the alignments can be used without the help of the other modalities. \n- The elaboration on the temporal information is not clear and needs further clarification.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The claim is accurate with clear formulation.\n\nThe only concern is that the math formulations are kind of messy to read, for example, the subscript on the left side always confusing my reading.", "audience_yn": "Yes", "audience_explanation": "This is aligned with the TMLR audience's interest.", "requested_changes": "- math formulations are kind of messy to read, for example, the subscript on the left side always confusing my reading, also it would be helpful changing the inner product from $<,>$ to $\\langle, \\rangle$\n- The temporal information should be highlighted besides the regular modality alignment \n- Experiments should be highlighted with the best response in boldface to help understanding the performance difference. \n- Needs to highlight the difference between the modality alignment and the representation learning based on these related works.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "VjqYzc7EHM", "reviewer": "Reviewer_Hswm", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper tackles the significant discrepancy between the multimodal training and unimodal deployment of speech processing systems. The authors propose MUTUD, a framework designed to retain the benefits of multimodal learning in a practical, unimodal inference setting. The core of this framework is the Temporally Aligned Modality feature Estimation (TAME) module, which employs modality-specific codebooks to learn cross-modal associations during training. At deployment, TAME can then estimate the features of a missing modality (e.g., video) using only the available modality (e.g., audio), thus bridging the performance gap while maintaining unimodal efficiency.\n1) Strengths\n- The paper studies a critical and practical problem: the discrepancy between training and deployment in audiovisual speech processing.\n- The effectiveness of the proposed MUTUD is well-supported by extensive experiments and in-depth analyses across multiple audiovisual tasks and datasets.\n\n2) Weaknesses\n- The core idea, using a learnable module to impute missing modalities, is not new. For example, DCP[1] directly adopts a neural network to predict missing modalities via contrastive learning.\n- The proposed framework introduces too many hyperparameters (loss weights $\\alpha, \\beta, \\gamma, \\lambda$, codebook size $K$, and temperature $\\tau$). This complexity, coupled with the lack of tuning guidance, may pose a challenge for real-world applications.\n\n[1] Lin, Yijie, et al. \"Dual contrastive prediction for incomplete multi-view representation learning.\" IEEE Transactions on Pattern Analysis and Machine Intelligence 45.4 (2022): 4447-4461.", "claims_and_evidence_yn": "Yes", "claims_explanation": "Performance Improvement: Tables 1, 2, 3, 4 (Speech Enhancement), Table 8 (Speech Recognition), and Table 9 (Active Speaker Detection) consistently show that MUTUD outperforms the audio-only baselines by a significant margin across all evaluation metrics.\n\n Inference Efficiency: The claim of high efficiency at inference is clearly demonstrated in Table 6. MUTUD has a parameter count and computational cost that are comparable to the unimodal model, while being drastically more efficient than the full audiovisual model.", "audience_yn": "Yes", "audience_explanation": "The paper addresses the \"gap between training and deployment\", a fundamental and practical problem in machine learning where models that are powerful during training are infeasible for deployment. The proposed MUTUD framework is presented as a generic approach that could be adapted to other multimodal domains beyond speech processing (e.g., video-text, audio-text), making the idea highly valuable to the wider community.", "requested_changes": "The paper would be strengthened by an ablation study on the components of the TAME loss function. The contribution of each term, particularly the cross-modal association loss $\\mathcal{L}_{C_a\\rightarrow C_v}$, is not experimentally justified. Such an analysis would provide better insight into the design of the training objective.\n\nKey hyperparameters, such as the loss weights ($\\alpha, \\beta, \\gamma, \\lambda$) and the softmax temperature ($\\tau$), are simply set to fixed values without justification. The absence of a sensitivity analysis makes it unclear how robust the model is to these choices and offers limited guidance for researchers aiming to reproduce or adapt the method for other applications.", "broader_impact_concerns": "N/A", "additional_comments": "N/A", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)