Unified Speech Recognition: A Single Model for Auditory, Visual, and Audiovisual Inputs
Authors: Alexandros Haliassos
Rodrigo Mira
Honglie Chen
Zoe Landgraf
Stavros Petridis
Maja Pantic
Venue: NeurIPS 2024 poster (2024)
Keywords: Speech recognition
lipreading
self-supervised learning
semi-supervised learning
Primary area: speech_and_audio
TL;DR: We propose a unified speech recognition method for ASR, VSR, and AVSR.

Abstract: Research in auditory, visual, and audiovisual speech recognition (ASR, VSR, and AVSR, respectively) has traditionally been conducted independently. Even recent self-supervised studies addressing two or all three tasks simultaneously tend to yield separate models, leading to disjoint inference pipelines with increased memory requirements and redundancies. This paper proposes unified training strategies for these systems. We demonstrate that training a single model for all three tasks enhances VSR and AVSR performance, overcoming typical optimisation challenges when training from scratch. Moreover, we introduce a greedy pseudo-labelling approach to more effectively leverage unlabelled samples, addressing shortcomings in related self-supervised methods. Finally, we develop a self-supervised pre-training method within our framework, proving its effectiveness alongside our semi-supervised approach. Despite using a single model for all tasks, our unified approach achieves state-of-the-art performance on LRS3 for ASR, VSR, and AVSR compared to recent methods. Code will be made publicly available.

Unified Speech Recognition: A Single Model for Auditory, Visual, and Audiovisual Inputs
Alexandros Haliassos Imperial College
Rodrigo Mira Imperial College rs25170ic. uk
Honglie Chen
Meta AI
ah22148ic.ace ac.uk
hongliechen@meta.. com
Zoe Landgraf
Stavros Petridis Meta AI / Imperial College stavrospêmeta.com com
Maja Pantic
Meta Al zoelandgraf @meta..emmmmmmmmm com
Meta AI Imperial College maj aeaaticcnemeeeee com
Abstract
Research in auditory, visual, and audiovisual speech recognition (ASR, VSR, and AVSR, respectively) has traditionally been conducted independently. Even recent self-supervised studies addressing two or all three tasks simultaneously tend to yield separate models, leading to disjoint inference pipelines with increased memory requirements and redundancies. This paper proposes unified training strategies for these systems. We demonstrate that training single model for all three tasks enhances VSR and AVSR performance, overcoming typical optimisation challenges when training from seratch. Moreover, we introduce greedy pseudo- lebelllabel approach to more effectively leverage unlabelled samples, addressing shortcomings related self supervieed methods. Finally, we develop a self-supervised pre- training method within our framework, proving its effectiveness alongside our semi-supervised approach. Despite using a single model for all tasks, our unified approach achieves state-of-the-ar performance compared to recent methods on LRS3 and LRS2 for ASR, VSR, and AVSR, as well as on the newly released WildVSR dataset. Code and models are available at https: https://githa tighub....... com/ ahaliassos/us:
Introduction
Speech recognition can be achieved using auditory signals (known auditory/automatic speech recognition; ASR) [1, 2], visual cues from lip movements (visual speech recognition; VSR) [3, 4], or both (audiovisual speech recognition; AVSR) [5, 6]. Audio typically offers the most relevant information in videos of talking faces, but lipreading can greatly enhance recognition, especially when the audio is noisy or wholly unavailable [6]. Despite the similarities between ASR, VSR, and AVSR, research in these fields has largely developed independently [7, 8, 3, 3.
The Transformer architecture's 's versatility [10, 11, 12] has spurred efforts to unify speech recognition by pre-training a single model on various unlabelled inputs (visual, auditory, and audiovisual) through self-supervision [13, 14, 15]. However, these methods often require separate fine-tuning stages for ASR, VSR, and AVSR, leading to separate models for each task, which increases computational load and complexity. u-HuBERT [16] shows that single pre-trained model can be fine-tuned for all three tasks, yet does not reach the performance of separately fine-tuned models [17, 18].
In this paper, delve deeper into strategies for unified speech recognition (USR) by training single model to perform ASR, VSR, and AVSR. We find that training such a model from scratch on the LRS3 dataset [19] achieves competitive performance on all tasks. This is notable given the known
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

optimisation difficulties in VSR training, which previously required self- self-supervised pre-training [13], supervised feature extractor pre-training [6], or curriculum learning strategies [9]. Our findings suggest that including audio improves the optimisation landscape for VSR and AVSR supervised training, as observed in a different context by [20].
Furthermore, we propose semi-supervised pscudo-labelling approach to leverage unlabelled audiovi- sual data, addressing shortcomings of standard fine-tuning self-supervised methods [13, 14, 17, 18]. Fine-tuning often leads to overfitting due to using fewer samples than pre-training, requiring various "tricks" reach optimal performance [13, 17]. This issue is particularly pronounced in encoder- decoder architectures where usually only the encoder is pre-trained, and attempts to pre-trau tre-train the decoder have yielded inconsistent results [21, 22]. Our -upervviee approach generates pseudo- labels via an encoder-decode momentum-bon -based teacher [23] to leverage unlabelled samples through- out training, effectively mitigating overfitting. Training on all three modalities simultaneously helps alleviate the computational cost of pseudo-labelling as the cost is amortised across the inputs.
Lastly, inspired by recent self-supervised works, we design a pre-training method within our unitied framework. We combine pre-training with pseudo-labelling and show that our semi-supervised approach complementary to self-supervision. Our final unified models achieve state-of-the-an results across multiple settings, surpassing existing methods that use separate models for each task.
2 Related Work
Audiovisual self-supervised speech representation learning. Recent interest in audiovisual self- supervised learning for speech recognition has focused on leveraging the correspondence between audio waveforms and silent lip movements to capture shared semantic content across the modali- ties [13, 17. 14, 15, 18]. These methods employ cross-modal learning and masked prediction [24] to develop contextualised representations from large unlabelled datasets, which are more readily available than transcribed datasets. After trainirinnng, a randomly initialised decoder is appended to the encoder, often with optional CTC layer [25]. The system is then fine-tuned on smaller set of labelled samples for tasks such as ASR, VSR, and AVSR, usually resulting in different mod- els for each task [13, 15]. However, these methods may fail to leverage unlabelled samples fully since the pretext tasks are not directly aligned with speech recognition. Furthermore, the decoder, trained on limited data during fine-tuning, is highly susceptible overfitting. necessitating strategies such as freezing encoder layers [13] or employing variable learning rates across layers to optimise performance [17, 26].
Pseudo-labelling for speech recognition. Pseudo-labelling has been explored in audiovisual speech recognition literature, with methods such as offline pseudo-labelling [9, 27] and frame-wise distillation using frozen teacher models [28]. While these approaches rely on frozen external ASR models trained on large-scale datasets [7, 291, our USR method eliminates this dependency using a randomly initialised teacher model that improves throughout training.
Iterative pseudo- weeudo-abblllng has shown promise for ASR. Some employ multiple rounds of pseudo- labelling using costly beam search and filtering strategies [30, 31, 32, 331, while others continuously and efficiently update pseudo-labels using a CTC-only loss [34, 35J. However, eliminating filtering and attention losses can impact training due to low-quality pseudo-labels, as observed in recent method [36] that aims to apply these approaches for ASR, VSR, and AVSR but lags behind the state-of-the-arr (see Appendix J). In contrast, USR uses an encoder-decoder architecture to generate CTC and attention pseudo-labels at each iteration through a greedy approach, while pseudo-label quality is maintained via a coken-wise filtering mechanism inspired by the supervipdd FixMatch technique [37] in image recognition. We note that sharing the same pseudo-labe leeuls across auditory, visual, and audiovisual inputs amortises generation costs, leading to efficient CTC-attention training.
Single model for multiple modalities. An earlier study [38] trained single recurrent neural network [39] for ASR, VSR, and AVSR, but noted significant performance differences compared to modality-speciffe models. Recent works have shown that the Transformer architecture [10] can handle multiple modalities using the same weights, with minimal performance degradation [11, 12]. In speech recognition, some [13, 14, 15] use the same Transformer encoder for auditory, visual, and audiovisual inputs during pre-training, but then separately fine-tune the parameters for ASR, VSR, and AVSR, resulting in separate models during deployment. u-HuBERT [16] uses the same weights for all

---

Semi-supervised Training
Pseudo-label Filtering
Legend
CTC Preudo-labels
Parudo labels
Attention
Pretdlo-abels
Audihory
Pogado labul Fitering
Panudtadatbel
Visual
Decoder
Teacheer
CTC
Decoder
Student
Attention Pooudo labels
Stop gradient
Encoder
Preude-labels
or
Encoder
or
along batch
Concatenate
Concatonate
Unmasked Features
Labels
Masked Features
at
abblls
Output Frobabilities
Self- superviedd Pre-training
Multi-modal Feature Extraction
Self-supervieed Losses
Targets
Predictor
Linear
Linear s
nicct
Encodler
Encoder
Audio Feature
Video Feature
Audio Feature
Video Feature Extractor
Unmasleed features
AM)
+0
Figure 1: Unified Speech Recognition. Our USR method combines self-supervised pre-training with semi-supervised fine-tuning. For semi-supervised training. pseudo-labels are generated from unmasked audiovisual features using an EMA (exponential moving average)-based teacher. The student, intaking masked inputs, predicts pseudo-labels for unlabelled data and ground-truth labels for labelled data. To obtain the pseudo-labels, an argmax operation is applied to the CTC and attention teacher output probabilities; the tokens with predicted probability below a fixed threshold discarded. For self-supervised pre-training, student encoder processes masked visual, auditory, and audiovisual samples and predicts targets, generated by an EMA-based teacher intaking unmasked audiovisual samples, via shallow predictor. The targets the average outputs of the teacher blocks. The resulting student weights are used to initialise the student and teacher in semi-supervised fine- tuning. Feature extraction achieved through modality-specific feature extractors, whose features are concatenated along the channel dimension to produce the audiovisual inputs. The auditory, visual, and audiovisual student inputs are batched together for training efficiency.
modalities when fine-tuning pre-trained AV-HuBERT backbone [13], demonstrating the viability of a unified model. However, it encounters limitations common to other self-supervised approaches, such as proneness overfitting during supervised fine-tuning. Our proposed semi-supervised approach leverages unlabelled samples during the fine-tuning stage, significantly alleviating these concerns.
3 Unified Speech Recognition
Our unified method trains a Pre-LN [40] Transformer [10] encoder-decoder model for ASR, VSR, and AVSR. Section 3.1 describes the task of unified speech recognition using supervised training, where we have ground-truth annotation for each audio-visua pair. Sections 3.2 and 3.3 then introduce our proposed idea, which employs semi-supervisee training and self- pre-training to effectively utilise unlabelled samples. An overview of USR's 's components is depicted in Figure I.
3.1 Unified Supervised Training
Inputs. Let { [(v.. ag. yb) b E [1, B}} be batch of B labelled samples, where Va denotes a Tę- frame video of lip movements, a, denotes the corresponding (raw) audio waveform of T, 640T, frames!, and ys denotes the label sequence of length T,. Following [9, 17], and are zero-masked with a maximum duration of 0.4 and 0.6 seconds for each second of video and audio, respectively.
We assume the video is sampled at 25 frames per second and the audio 16.000KHZ.
3

---

Multi-modal feature extraction. The raw video and audio are fed into ResNet-11 [41] architectures: 2D version with a 3D stem [42] for video and ID version for audio, sub-sampling the audio to match the video's s sampling rate [17]. Linear layers follow the feature extractors to produce the visual and auditory features. The audiovisual features are formed by concatenating the feature extractor outputs along the channel dimension and applying a linear transformation. Finally, the features from the three modalities are concatenated along the batch dimension for efficient processing. We provide the model with all three input types, enabling it to perform well on ASR, VSR, and AVSR.
Losses The encoder outputs pass through a linear + softmax layer to yield output probabilities Cb,m. for each modality m {v,a, av). The CTC loss for each modality is given by
1
C.
Σ b=1
Leac CC
B
where lee is the standard CTC loss [25]. Further, let av.m denote the attention probabilities from the outputs of the decoder in teacher forcing mode [43]. The batch attention loss can be expressed as
1 Σ
Am
Lco(ab.m
Mob. Vt).
(2)
where lee is the summed cross-entropy loss for each token. The CTC and attention losses combined to obtain
L. Amnem + >ac)AAm:
where Aae the relative weight placed on the CTC loss versus the attention loss. We set dete to 0.1, following [27, 17, 18]. The overall labelled loss is given by
C X,C, (1 Xx) + Lav),
where X, controls the weight of the video loss relative to the audio/audiovisual losses. We do not separate weights for the audio/audiovisual losses due to similar training dynamics observed in preliminary experiments.
3.2 Unified Semi-supervised Training
We introduce a student-tent teacher labell-ab framework to utilise unlabelled samples alongside labelled examples. The student, equipped with labelled losses, mirrors the model in Section 3.1.
Inputs. In addition to the labelled batch from Section 3.1, we now also have 3 unlabelled video and audio samples {(V), a b E [1, B.). The student inputs are masked as before.
Pseudo-labels. The teacher, sharing the same architecture as the student, generates pseudo-labels for unlabelled samples. The student optimised as usual, no gradients are passed to the teacher. Instead, the teacher's weights 0g are updated at each iteration via an exponential moving average (EMA) the student's weights 02 [44]: 0g + pd, + (1 - 4)0,, where μ increases throughout training from 0.999 to using cosine scheduler.
For an unmasked audiovisual sample, let ē and â, denote the CTC probabilities from the teacher encoder and the attention probabilities from the teacher decoder, respectively. The CTC and attention pseudo-labels are given by arg max(@;) and arg max(āx), respectively, where arg max is applied token-wist wise. Hence, the pseudo-labels correspond to units with the maximum probability across the vocabulary for each input/output time-step. The attention targets are generated auto-regressively by selecting, at each time-step, the most likely unit the input for the next time-step, without using costly beam search strategy. Our greedy approach allows for efficient label generation.
Filtering. The teacher may not consistently generate high-quality predictions, especially early in training. We propose a straightforward token-wise filtering mechanism, creating masks (1,) > T) and 1l (max(ā,) 2 T), where the operations are applied token-wise. We thus discard a pseudo- label for given time if its confidence falls below certain threshold r. This mechanism draws inspiration from image recognition literature [37] and is adapted to sequences.

---

Unlabelled losses. The unlabelled losses are computed via the cross- entropy between the student
predictions and the teacher pseudo-labels. That the per-modality CTC losses are given by
192
Σ B
(max((6) 7) Ise (C) arg max(e:)),
where @ denotes the Hadamard product and Jm the student outputs. The attention losses Am are computed similarly. The unlabelled losses Lm are obtained as in Eq. 3:
C AmnC + (1 Aec Aam)...
(6)
Final loss. The total semissupervised loss f combines the per-modality labelled (see Eq. 3) and unlabelled losses (see Eq. 6):
_ -odbe + 7a(1 Xo) + + (1-))))) -0-
Cgv),
(7)
where % and % weigh the contribution of the labelled loss versus the unlabelled loss for au- dio/audiovisual and visual inputs, respectively. In Section 4.2, we show the benefits of using separate weights for each modality rather than single weight for both.
3.3 Unified Self-supervised Pre-training
Transformers typically benefit from self-supervised pre-training [45, 13, 17, 15], even with the same data used during fine-tuning [46, 45]. Inspired by recent work [17, 18, 15], we propose self-supervised method within our framework that can precede semi-supervised fine-tuning.
Inputs. For pre-training, we use only the unlabelled 8" samples from Section 3.2. Following [17], we mask the student inputs by selecting each video frame index as the start of three-frame mask with 0.4 probability, applying corresponding enlarged mask to the audio in temporal alignment. The elements of the mask h2 are set and for unmasked and masked tokens, respectively.
The targets are generated by an EMA-based teacher encoder model from unmasked audiovisual inputs, similarly to Section 3.2. Following [15, 18]. the targets e,, are generated by averaging the outputs from all encoder blocks and applying instance normalisation [47]. Using only audio targets, as [15], can make the student's final layers more relevant to speech, which has proven beneficial for fine-tuning with few samples, where there is high chance of overfitting [17]. Our fine-ture tueitunngg process instead uses abundant unlabelled data with pseudo-labes labelss which help reduce overfitting and allow the network to learn from rich audiovisual targets.
Targets.
Predictor. Following [17], we employ a 122dimensional two-block Transformer predictor that processes student encoder outputs and mask tokens to produce predictions Ps.m Unlike the separate predictors for video and audio used in [17], we use single predictor for all inputs.
Loss. The loss for modality m can be expressed as
B²
Cself
Σ hs C cos(pt.m:e)). 6=1
(8)
Bu
where cos denotes cosine similarity, applied token-wise. Thus, the student aims to predict the teacher targets corresponding to the masked inputs. The supervupee loss = is then
s
C + Ax + Lielt).
4 Main Properties
In this section, we investigate the behaviour of our unified model. For experiments, we use 12-block Base model with hidden size of 512 (see Appendix C.4 for model details). We report test set word error rates (WER) for direct comparison with the main results. Note that we used the validation set from [13] in the exploration stage to avoid overfitting the test set.
5

---

Table Supervised ablations on the full LRS3 dataset using our Base model. Default settings are in gray in all tables of the paper.
Modality sampling. Ran- dom sampling trained more epochs as it sees one-third
(c) Relative weight for video
(a) Sharing model parameters vs.
loss.
using mosality-speeifie models.
of the data at each iteration.
WER (%)
WER (%)
A
Params
WER (%)
V
A
AV
A
AV
Mod
V
A
AV
0.1
42.9
2.2
1.9
Shared
36.4
2.3
2.1
Unshared
85.5
2.1
63.4
Rand
36.2
2.3
2.2
0.3
36.4
2.3
2.1
0.5
35.2
2.4
2.2
All
36.4
2.3
2.1
4.1 Unified Supervised Training
In Table 1, we investigate properties of training our unified model from scratch on the full LRS3 dataset [19] (see Section 3.1). Training details are provided in Section C.5.
Sharing weights. Table la studies the impact of weight sharing versus separate models per task (ASR, VSR, AVSR). While using only auditory inputs yields strong performance, training VSR and AVSR models from scratch encounters optimisation challenges, in line with prior research [13, 17]. Interestingly, these hurdles are overcome with weight sharing, resulting in robust VSR and AVSR performance without self--uupr pre-training [17] or training techniques like curriculum learning [9]. This is likely due audio containing denser verbal information than video, enhancing the optimisation landscape for visual modalities [20].
Modality sampling. We employ weighted average to combine the per-modality losses (see Eq. 4). In contrast, other methods [13, 15] randomly sample, at each iteration, imput types with different probabilities, which may vary during training. Table 1b shows that our approach performs similarly with random sampling when training the latter for 3 more epochs. Our approach offers benefits such as sharing computational costs among feature extractor forward passes and amortising the cost of pseudo-label generation across input types (see Section 3.2), as all modalities use the same targets.
Input type weight. Table Ic studies the effect of using different weights for the visual modality. We observe that using higher Av for the VSR loss improves VSR but worsens ASR/AVSR. We choose Ay 0.3 as our default setting, striking balance in performance among the different tasks.
4.2 Unified Semi-supervised Training
In Table 2, we ablate various components to better understand our unified semi-supervised framework (see Section 3.3). We adopt the common low-resource setting [13]: the 30-hour "trainval" partition of LRS3 serves as our labelled dataset, while the remaining portion of LRS3 (without labels) provides our unlabelled samples. See Appendix C.5 for training details.
Filtering predicted tokens. Figure 2 investigates the impact of the threshold parameter T E (0, 0.8, 1}. We plot (from left right) (1) the proportion of tokens exceeding T, (2) the validation attention accuracy of the decoder using teacher forcing, and (3) the CTC loss, as function of the epoch number. We also show the final WER. We observe that
1, where only labelled samples contribute to training. results in poor attention accuracy, high CTC loss, and high WER across input types. Conversely, т implying no filtering (i.e., all tokens are considered regardless of confidence level), yields competitive performance, suggesting some robustness to low-quality Finally, for t = 0.8, the proportion of tokens with confidence over begins at a low level and steadily increases throughout training as the teacher network improves. This yields improved performance in terms of attention accuracy, CTC loss, and final WER, demonstrating the efficacy of filtering via simple confidence threshold. A more fine-grained analysis of values given in Scction D.I.
secudolaabel..

---

Proportion Over Throshold
1.0
0.36
r-0.5
0.8
1.56-
T-13
WER (%)
CTC
V
AV
Altention
40.7
4.9
4.7
0.00
0.8
37.8
4.0
3.9
00
0.86
61.8
8.9
40
20
40 Epoch
Epoch
Epoch
Figure 2: Pseudo-label filtering threshold. Left: Validation plots for different values of threshold
Right: Final WER for different values of r.
Table 2: Semi-supervised ablations under the LRS3 low-resource setting using our Base model.
(a) Relative labelled weight for (b) Teacher's EMA momentum
(c) CTC vs. CTC-attention losses.
audio and video.
parameter.
WER (%)
Loss type
WER (%)
WER (%)
7
V
A
AV
V
AV
V
AV
CTC
45.6
5.2
5.0
0.5
0.5
42.3
4.1
4.0
0
38.9
4.1
4.0
CTC-att
37.8
4.0
3,9
0.2
0.2
38.0
4.2
4.1
0.999
37.8
4.0
3.9
0.5
0.2
37.8
4.0
3.9
Quantity/quality trade-off. Pseudo-labels tend to be abundant but noisy, while ground-truth transcriptions scarce yet high-quality. The balance between quantity and quality is adjustable via the hyperparameters and in Eq. 7. Table 2a explores different values for 7 and %, revealing better performance when % %s Noisy pseudo-labels generated from audiovisual samples may suffice for VSR, which often performs worse than ASR/AVSR and benefits from data abundance. Conversely, ASR/AVSR is less prone to overfitting and may suffer with excessive reliance on low- quality pseudo-labels, requiring higher relative weight on labelled losses.
Momentum. Table 2b shows the effect of updating the teacher's s weight via EMA (μ 0.999) compared to simply copying the student's weights at every iteration (μI = 0). Using EMA results in better performance, yet good results achieved even without it.
Loss types. CTC and attention- encoder-decodee frameworks are dominant approaches in speech recognition. While attention typically outperforms CTC, it may struggle with proper alignment prediction, requiring tuming of various decoding hyperparameters [48]. To address these challenges, we adopt CTC-attention hybrid framework [48], as in [17, 9, 27]. The costly auto-regressive attention pseudo-label generation made computationally feasible via our greedy strategy and multi- modal feature extraction (which amortises pseudo-labe generation costs). Table 2c demonstrates significant improvement in results by using both CTC and attention compared CTC alone.
4.3 Unified Self-supervised Pre-training
Table 3 examines the main properties of our self-supervised method (see Section 3.3). We fine-tune pre-trained models with different hyperparameters using our semi-supervised approach (Section 3.2). We use the LRS3 low-resource setting, as in Section 4.2. See Appendix C.6 for training details.
Target modality. In Table 3a, we evaluate our method with targets derived from the different input modalities. Across all cases, pre- pre-training outperforms training from scratch, highlighting the complementarity of semi- and self-supervised training. Visual targets enhance VSR but diminish ASR/AVSR performance compared to auditory targets; overall, audiovisual targets consistently perform best. These results suggest that cross-modal-onf, pre-training may lose crucial modality- specific information, reducing generalisation when fine-tuning on all data (ineluding unlabelled samples), i.e., via pseudo-labelling. Our observations in contrast to previous findings with
7

---

Table 3: Self-supervised ablations under the LRS3 low-resource setting using our Base model.
(a) Target type. "Seratch" refers semi-supervised training only.
(b) Averaging blocks vs. using only last encoder block.
(c) Predictor depth.
Target
WER (%)
Depth
WER (%)
V
A
AV
Target
WER (%)
V
A
AV
Scratch
37.8
4.0
3.9
A
AV
37.0
3.2
3.0
36.2
3.7
3.4
Last block
37.2
3,4
3.1
36.0
3.2
3.0
A
37.3
3.2
3.1
Avg blocks
36.0
3.2
3.0
36.9
3.1
2.9
AV
36.0
3.2
3.0
supervised fine-tuning, where visual audiovisual pre-training targets tend to underperform [13, 17, 15]. See Appendix F for an in-depth analysis comparing supervised and semi-supervised fine-tuning.
Averaging targets. [15, 18] demonstrate that using the average of encoder blocks as targets outperforms using the last block alone. Table 3b confirms this finding in our setting.
Predictor depth. In Table 3c, we study the influence of predictor depth. A deeper predictor yields more abstract encoder representations, while shallower one retains more task-specific features [17]. We observe strong performance our default depth of 2. Notably, our semi-supervised fine-tuning approach is less sensitive to predictor depth than standard methods [17, 18].
5
Comparisons with Previous Results
5.1 Comparisons with Self-supervised Methods
Table 4 compares our approach on LRS3 [19] with self-supervised methods under similar model sizes and data settings. We combine pre-training (Section 3.3) with standard fine-tuning (Section 3.1) when using identical pre-training and fine-tuning data, and with semi-supervised fine-tuning (Section 3.2) when using extra unlabelled data. In addition to the low-resource labelled data setting outlined in Section 4.2, we test in a high-resource setting using the full 433-hour LRS3 dataset for fine-tuning. Our pre-training employs either LRS3 alone or combined with 1,326-hour English-only version of VoxCeleb2 [49, 13J. We experiment with Base, Base+, and Large Transformers (see Appendix C.4).
Low-resource. Using the Base model and LRS3 for pre-training, our approach significantly exceeds the previous tate-of-theeaar across VSR, ASR, and AVSR, when fine-tuning on 30 hours. Increasing the pre-training data and model size enhances performance, demonstrating our method's scalability. With the Large model and LRS3+ Vox2 as pre-training data, we achieve 26.9% WER for VSR and 2.4% WER for both ASR and AVSR, matching BRAVEn on ASR and surpassing it on VSR. Unlike other methods, which use separate models for each task, USR employs single model for all tasks.
High-resource. In the high-resource setting, our results are comparable to modality-specific models for ASR/AVSR and superior for VSR across all settings. Our top model obtains 22.3% WER for VSR, 1.2% WER for ASR, and 1.1% WER for VSR, significantly outperforming u-HuBERT, which also uses single model for all modalities. Furthermore, USR's low-resource VSR performance is superior to u-HuBERT's high-resource VSR result.
5.2 Comparisons with the State-of-the-Art
LRS3. In Table 5, we compare our best model against the state-of-the-an on LRS3. We present our USR results with a language model incorporated via shallow fusion [17, 27], improving VSR performance from 22.3% to 21.5%. Despite using a shared model for tasks, our performance exceeds multiple supervised methods and approaches top results [27, 52, 53], which use significantly more labelled data. USR surpasses Auto-AVSR on VSR (21.5% vs. 23.5% despite the latter using more total data and external ASR models for transcription. Finally, outperform self-supervised

---

Table 4: Comparisons with self-supervised methods. LRS3 results for the low-resource (LR) and high-resource (HR) labelled data settings, with 30 and 433 hours of labelled data, respectively. Best results in bold, second-bes underlined.
Pre-train data
Shared params
WER (%) LR
WER (%)
HR
Method
AV
V
A
AV
Base(+) models
AV-HuBERT [13]
LRS3
51.8
4.9
4.7
44.0
3.0
2.8
VATLM [14]
LRS3
X
48.0
3.6
RAVEn [17]
LRS3
47.0
4.7
39.1
2.2
AV-data2vec [15]
LRS3
X
45.2
4.4
4.2
39.0
2.0
1.8
Lip2Vec [20]
LRS3
49.5
42.0
BRAVEn [18]
LRS3
43.4
4.0
4.0
36.0
1.9
USR
LRS3
36.0
3.2
3.0
34.3
1.9
1.6
Base(+) models
AV-HuBERT [13]
LRS3+Vox2
46.1
4.6
4.0
34.8
2.0
1.8
VATLM [14]
LRS3+VoX2
42.6
3.4
34.2
1.7
RAVEn [17]
LRS3+Vox2
40.2
3.8
33.1
1.9
AV-data2vec [15]
LRS3+
37.8
3.7
3.3
32.9
1.7
1.4
Vec [20]
LRS3+ Vox2
40.6
34.1
BRAVEn [18]
LRS3+ Vox2
35.1
3.0
28.8
1.4
USR
LRS3+VoX2
28.4
2.6
2.5
26.5
1.6
1.3
Large models
AV-HuBERT [13]
LRS3+VoX2
X
32.5
2.9
3.3
28.6
1.3
1.4
VATLM [14]
LRS3+ Vox2
31.6
2.7
28.4
L.2
RAVEn [17]
LRS3+VoX
32.5
2.7
28.2
1.4
AV-data2vec [15]
LRS3+ Vox2
30.8
2.7
2.7
28.5
1.3
1.3
Lep2 [20]
LRS3+VoX2
31.2
26.0
BRAVEn [18]
LRS3+VoX2
30.8
2.3
26.6
1.2
u-HuBERT [16]
LRS3+VoxI
29.1
1.5
1.3
USR
LRS3+ Vox2
26.9
2.4
2.4
22.3
1.2
1.1
Table 5: Comparisons with the state-of-the-art on LRS3. "Labels include automatic transcriptions
from ASR models trained on large-scale, often non non-publle public datasets. "ST" desnote offline self- self-training.
Labelled
Unlabelled
Language
Shared params
WER (%)
Method
hours
hours
model
AV
Supervised
V2P [50]
3,886
X
55.1
RNN-T [38]
31,000
/
33.6
4.8
4.5
VTP [51]
2,676
X 30.7
Auto-AVSR [27]
1,902
X 23.5
1.0
1.0
Auto-AVSR [27]
3,448
X 19.1
1.0
0.9
ViT3D-CM [52]
90,000
X 17.0
1.6
Synth VSR [53]
6,720
X
16.9
LP Conf [54]
100,000
X
X
12.8
0.9
Self/semi-superviee.
AV-HuBERT w/ ST [13]
433
1,326
X
28.6
RAVEn w/ ST [17]
433
1,326
V
X
23.1
1.4
USR
433
1,326
V
21.5
1.2
1.1

---

Table 6: Comparisons with the state-of-the-art on LRS2. "Includes methods that use automatic
transcriptions from ASR models trained large-scale datasets. "ST" stands for self self-training.
Labelled hours
Unlabelled hours
Language
Shared params
WER (%)
Method
model
A
AV
Supervised"
CM-seq2seq [6]
380
37.9
3.9
3.7
CM-aux [9]
1,459
x
25.5
VTP [51]
698
x
28,9
VTP [51]
2,676
X
22.6
Auto-AVSR [27]
818
27.9
2.6
Auto-AVSR [27]
3,448
14.6
1.5
1.5
Self/semi-supervised
Uni-AVSR [55]
223
60,000
44
43.2
2.7
2.6
LiRA [56]
223
433
38.8
X
RAVEn [17]
223
1,759
23.2
2.5
RAVEn w/ ST [17]
223
1,759
17.9
2.3
USR
223
1,759
16.0
2.0
1.9
USR
223
1,759
15.4
1.9
1.9
methods [13, 17] using self-training that require a costly beam search strategy combining CTC, attention, and language model scores. Our simpler, greedy approach is effective, and we aim to explore additional offline pseudo-labelling for USR in future work.
LRS2. We also compare with the state-of-the-an art on the LRS2 dataset [57] (see Table 6). We train our model using the same hyperparameters as for the high-resource LRS3 setting. Consistent with our LRS3 results from Table 5, USR surpasses all other self-supervised methods across ASR, VSR, and AVSR, and outperforms strong supervised methods [27] trained with >> 4x more labelled data (433 vs. 1,759 hours). Results on the WildVSR dataset are in Appendix E.
6 Conclusion
Despite their similarities, research in VSR, ASR, and AVSR has typically focused on developing sep- arate models for each task. In this paper, we propose unified training strategies that use a single model to address all three tasks simultaneously. Our USR approach combines self-supervised learning with greedy
-upervipe technique achieve state-of-the- results, surpassing related methods that use separate models for each task. Future work could explore alternative encoder architectures, strategies to improve pseudo-label quality, and methods incorporate extra audio-only data. We hope to inspire further efforts towards consolidating ASR, VSR, and AVSR systems.
Acknowledgements
Only Imperial College co-authors downloaded, accessed, and used the datasets. Imperial College authors conducted all of the dataset pre- pre-processing at Imperial College.
References
O. Abdel-Hamid, A.-r. Mohamed. H. Jiang, L. Deng, G. Penn, and D. Yu, "Convolutional neural networks for speech recognition," TEEE/ACM Transactions on audio, speech, and language processing, vol. 22, no. 10. pp. 1533-15 1545, 2014.
[2] C.-C. Chiu, T. Sainath, Wu, R. Prabhavalkar, Nguyen, Chen, Kannan, Weiss, K. Rao, E. Gonina et al. "ftate-of-the- art speech recognition with ceruuuence-t-sequeeee models," in 2018 IEEE international conference on acoustics, speech and signal processing (ICASSP). JEEE, 2018, pp. 4774- 4778.
10

---

[3] Y. M. Assael, B. Shillingford. S. Whiteson, and N. De Freitas, "Lipnet: End-to-end sentence-level lipreading," " arXiv preprint arXiv:161..01550 2016.
[4] B. Martinez, P. Ma, S. Petridis, and M. Pantic, "Lipreading using temporal convolutional networks," in ICASSP 2020-2020 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2020, pp. 6319-6323.
[5] S. Petridis, T. Stafylakis, P. Ma, F. Cai, G. Tzimiropoulos, and M. Pantic, "End-to-end audiovisual speech recognition." in 2018 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE, 2018, pp.
[6] P. Ma, S. Petridis, and M. Pantic, "End-to-end audio-visual speech recognition with conformers," in ICASSP 202/-202 JEEE International Conference on Acousties, Speech and Signal Processing (ICASSP). IEEE, 2021, pp. 7613-7617.
[7] A. Baevski, Y. Zhou, A. Mohamed, and M. Auli, "wav2vec 2.0: A framework for self-supervised learning of speech representations"" Advances in neural information processing systems, vol. 33, pp. 12 449 12460, 2020.
[8] W.-N. Hsu, B. Bolte, Y.HH. H. Tsai, K. Lakhotia, R. Salakhutdinov, and Mohamed, "Huhert: Self- supervised speech representation learning by masked prediction of hidden units," IEFF/ACM Transactions on Audio, Speech, and Language Processing, vol. 29, pp. 3451-3460, 2021.
[9] P. Ma. S. Petridis. and M. Pantic, "Visual speech recognition for multiple languages the wild," Nature Machine Intelligence, vol. 4, no. 11, pp. 930-939--000 90000999922000 2022.
[10] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, "Attention is all you need," " Advances in neural information processing systems, vol. 30, 2017.
[11] H. Akbari, L. Yuan, R. Qian, W.-H. Chuang. S.-F. Chang. Y. Cui, and B. Gong, "Vatt: Transformers for multimodal superviid learning from raw video, audio and text," Advances in Neural Information Processing Systems, vol. 34, pp. 24 2422122 2021.
[12] R. Girdhar, M. Singh, N. Ravi, L. van der Maaten, Joulin, and I. Misra, "Omnivore: A single model for many visual modalitics," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022. pp. 16 102-16 112.
[13] B. Shi, W.-N. Hsu, K. Lakhotia, and A. Mohamed, "Learning audio-visual speech representation by masked multimodal cluster prediction." arXiv preprint arXiv:2201. 021144 2022.
[14] Q. Zhu, I.. Zhou, 7. Zhang, S. Liu, B. Jiao, J. Zhang, L. Dai, D. Jiang, J. Li, and F. Wei, "Vatlm: Visual-audio-text pre-training with unified masked prediction for speech representation learning," IEFE Transactions on Multimedia, 2023.
[15] Lian, A. Baevski, W.-N. Hsu, and M. Auli, "Av-data2vec: Self-supervised learning of audio-visual speech representations with contextualized target representations." arXiv preprint arXiv:2302.066 2023.
[16] W.-N. Hsu and Shi, "u-hubert: Unified mixed modal speech pretraining and zero-shot transfer to unlabeled modality." Advances Neural Information Processing Systems, vol. 35, pp. 21 157-21 170, 2022.
[17] A. Haliassos, P. Ma. R. Mira. Petridis, and M. Pantic, "Jointly learning visual and auditory speech representations from raw data." arXiv preprint arXiv:2212.06246 2022.
[18] A. Haliassos, A. Zinonos, Mira, S. Petridis, and M. Pantic, "Braven: Improving self-supervised pre- training for visual and auditory speech recognition," in ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEBE, 2024, pp. 11 111-1111 435.
[19] T. Afouras, S. Chung, and A. Zisserman, "Las3-ted: large-scale dataset for visual speech recognition." " arXiv preprint arXiv:1809.000
2018
[20] Y. D. Djilali, S. Narayan, Boussaid, E. Almazrouei, and Debbah, "Lip2vee: Efficient and robust visual speech recognition via latent-to-latent visual audio representation mapping." in Proceedings of the IEEF/CVF International Conference on Computer Vision, 2023, pp. 13 790-13 801.
[21] Ao, Zhang, L. Zhou, S. Liu, H. Li, T. Ko, L. Dai, J. Li, Y. Qian, and F. Wei, "Pre-training transformer decoder for end-to-end ast model with unpaired speech data," arXiv preprint arXiv:2203.17113 2022.
[22] A. Elkahky, W.-N. Hsu, P. Tomasello, T.-A. Nguyen, R. Algayres, Y. Adi, Copet, Dupoux, and A. Mohamed, "Do coarser units benefit cluster prediction based speech pre-training"" in ICASSP 2023- 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEBE, 2023, pp. 1-5.
[23] A. Tarvainen and H. Valpola, "Mean teachers are better role models: Weight--veraged consistency targets improve semi-supervised deep learning results," Advances in neural information processing systems. vol. 30, 2017.
11

---

[24] J. Devlin, M.-W. Chang, K. Lee. and K. Toutanova, "Bert: Pre-training of deep bidirectional transformers for language understanding." " arXiv preprint
2018.
[25] A. Graves, S. Fernández, F. Gomez, and J. Schmidhuber, "Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks," in Proceedings of the 23rd international conference on Machine learning, 2006, pp. 369-376.
[26] K. Clark, M.-T. Luong, Q. Le, and C. D. Manning, "Electra: Pre-training text encoders as discriminators rather than generators," arXiv preprint Xiv::200 10555, 2020.
[27] P. Ma, A. Haliassos, A. Fernandez-Lopez. H. Chen, S. Petridis, and M. Pantie, "Auto-avsr: Audio-visual speech recognition with automatic lahels," " in ICASSP 2023-2023 IEEE International Conference on Acousticx, Speech and Signal Processigg (ICASSP). TEEE, 2023, pp. 1-5.
[28] T. Afouras, J. S. Chung, and A. Zisserman, "Asr is all you need: Cross-modal distillation for lip reading," in ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2020. pp. 2143-2147.
[29] V. Panayotov, G. Chen, D. Povey, and S. Khudanpur: "Librispeech: an asr corpus based on public domain audio books." in 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP).
IEEE, 2015, pp. 5206-520222 5210.
[30] D. S. Park, Y. Zhang, Y. Jia, W. Han, C.-C. Chiu, B. Li, Y. Wu, and Q. Le, "Improved noisy student training for automatic speech recognition." " arXiv preprini arXiv:2005.00:: 09629, 2020.
[31] Q Xu, T. Likhomanenko, J. Kahn, A. Hannun, G. Synnaeve, and R. Collobert, "Iterative -abulligabeling for speech recognition." " arXiv preprint arXiv:2005, 000667 2020.
[32] Y. Zhang, J. Qin, D. S. Park, W. Han, C.-C. Chiu, R. Pang, Q. V. Le, and Y. Wu, "Pushing the limits of semi-supervised learning for automatic speech recognition," " arXiv preprini arXiv:2010. 10504, 2020.
[33] Kahn, Lee, and Hannun, "Self-training for end-to-end speech recognition," in ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2020, pp- 7084-7088.
[34] T. Likhomanenko, Q. Xu, J. Kahn, G. Synnaeve, and R. Collobert, "slimipl: Sanguage-model-fee iterative
pseudo-labeling," arXiv preprint arXiv:2010.1 0t..20000111224 11524, 2020.
[35] Y. Higuchi, N. Moritz, J. L. Roux, and T. Hori, "Momentum pscudo-labeling for semi-supervised speech recognition." arXiv preprint arXiv:2 00..000..0002 006.0822,, 2021.
[36] A. Rouditchenko. R. Collobert, and T. Likhomanenko, "Av-cpl: Continuous pseudo-labeling for audio- visual speech recognition." arXiv preprint arXiv:2309.13995 2023.
[37] K. Sohn, D. Berthelot, N. Carlini, Z. Zhang, H. Zhang. C. A. Raffel, E. D. Cubuk, A. Kurakin, and C.-L. Li, "Fixmatch: Simplifying semi-supervised learning with consistency and confidence," Advances in neural information processing systems, vol. 33, pp. 596-608, 2020.
[38] T. Makino, H. Liao. Y. Assael, B. Shillingford, B. Garcia, O. Braga, and O. Siohan, "Recurrent neural network transducer for audio- visual speech recognition." in 2019 IEEE automatic speech recognition and understanding workshop (ASRU). IAEE, 2019, pp. 905-912.
[39] A. Sherstinsky, "Fundamentals of recurrent neural network (mnn) and long short-term memory (Istm) network." Physica D: Nonlinear Phenomena, vol. 404, p. 132306, 2020.
[40] R. Xiong, Y. Yang, D. He, K. Zheng, S. Zheng, C. Xing, H. Zhang, Y. Lan, L. Wang, and T. Liu, "On layer normalization the transformer architecture," in International Conference on Machine Learning. PMLR, 2020, pp. 10 10.52410055 533.
[41] K. Hc, X. Zhang, S. Ren, and J. Sun, "Deep residual learning for image recognition," in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016. pp. 770-778.
[42] T. Stafylakis and G. Tzimiropoulos, "Combining residual networks with Istms for lipreading." arXiv preprint arxiv: 3tt:::/770.000 703,04105. 2017.
[43] R. J. Williams and Zipser, "A learning algorithm for continually running fully recurrent neural networks." Neural computation, vol. 1, no. 2, pp. 270-280, 1989.
[44] J.BB Grill, F. Strub, F. Altché, C. Tallec, P. Richemond, F. Buchatskaya, C. Doersch, B. Avila Pires, Z. Guo, M. Gheshlaghi Azar et al., "Bootstrap your own latent-a new approach self-supervised learning." " Advances in neural information processing systems, vol. 33, pp. 21 271--1 284, 2020.
[45] K. Hc, X. Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick, "Masked autocncoders are scalable vision Iearners," in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022,
pp. 16 000-160009
[46] M. Caron, H. Touvron, L. Misra, H. Jégou, J. Mairal, P. Bojanowski. and A. Joulin, "Emerging properties in self supervised vision transformers." in Proceedings of the IEEE/CVF international conference on computer vision, 2021. pp. 9650-
12

---

[47] D. Ulyanov, A. Vedaldi, and V. Lempitsky, "Instance normalization: The missing ingredient for fast stylization." arXiv preprint arXiv:1607.08022 2016.
[48] S. Kim, T. Hori, and S. Watanabe, "Joint cic-attention based end-to-end speech recognition using multi-task learning." in 2017 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE, 2017, pp. 4835 -4839.
[49] J. S. Chung. A. Nagrani, and A. Zisserman, "Voxceleb2: Deep speaker recognition." arXiv preprint arXiv: 180.00020,,,,,, 2018.
[50] B. Shillingford, Y. Assacl, M. W. Hoffman, T. Paine, Hughes, U. Prabhu, H. Liao, H. Sak, K. Rao,
L. Bennett et al., "Large-scale visual speech recognition," " arXiv preprint arXiv: 05162, 2018.
[51] K. Prajwal, I. Afouras, and A. Zisserman, "Sub-word level lip reading with visual attention." in Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition, 2022, pp. 5162-5172.
[52] D. Serdyuk, O. Braga, and O. Siohan, "Trranformer-basee vidco front-ends for audio-visual speech recognition for single and multi-person video." arXiv preprint arXiv:2201. 10439, 2022.
[53] X. Liu, E. Lakomkin, K. Vougioukas, P. Ma, H. Chen, R. Xie, M. Doulaty, N. Moritz, J. Kolar, S. Petridis et al., "Synthysr: Scaling up visual speech recogmition with synthetic supervision." in Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, 2023, pp. 18 806-18815.
[54] O. Chang, H. Liao, Serdyuk, Shaly,, and O. Siohan, "Conformer all you need for visual speech recognition," in ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal P Processing (ICASSP). IEEE, 2024, pp. 10 136-1010 140.
[55] X. Pan, P. Chen, Y. Gong, H. Zhou, X. Wang, and Z. Lin, "Leveraging unimodal sule-super learning for multimodal audio-visie visual speech recognition," arXiv preprint arXiv:2203.0799 2022.
[56] P. Ma. R. Mira, S. Petridis, B. W. Schuller, and M. Pantic, "Lira: Learning visual speech representations from audio through self supervision," " arXiv preprint arXiv:2106.09177 2021.
[57] J. Son Chung, A. Senior, O. Vinyals, and A. Zisserman, "Lip reading sentences in the wild," in Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 6447-6456.
[58] A. Haliassos, K. Vougioukas, S. Petridis, and M. Pantic, "Lips don't lic: A generalisable and robust approach to face forgery detection," in Proceedings of the IEEE/CV conference on computer vision and pattern recognition, 2021, pp. 5039-5049.
[59] Y. A. D. Djilali, S. Narayan, F. LeBihan, H. Boussaid, E. Almazrouei, and M. Debbah, "Do vsr models generalize beyond Ins3?" in Proceedings of the IEEF/CVF Winter Conference on Applications of Computer Vision, 2024, pp. 6635-6644.
[60] T. Kudo and J. Richardson, "Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing," arXiv preprint arXiv: 1808. 1808606226, 2018.
[61] G. Huang. Y. Sun, Z. Liu. D. Sedra. and K. Q. Weinberger. "Deep networks with stochastic depth." in Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11 11-14, 2016, Proceedings, Part IV 14. Springer. 2016, pp. 646-661.
[62] I. Loshchilov and F. Hutter, "Decoupled weight decay regularization." arXiv preprint 171:0:171..05101 2017.
[63] P. Goyal, P. Dollár, R. Girshick, P. Noordhuis, L. Wesolowski, A. Kyrola, A. Tulloch, Y. Jia, and K. He. "Accurate, large minibatch sgd: Training imagenet in hour," arXiv preprint (7XX::1706.02677 2017.
[64] I. Loshchilov and Hutter, "Sgdr: Stochastic gradient descent with warm restarts," arXiv preprint arXiv: 011116688039888 2016.
[65] S. Watanabe, T. Hori, S. Karita, Hayashi, Nishitoba, Y. Unno, N. Enrique Yalta Soplin, J. Heymann, M. Wiesner, N. Chen, A. Renduchintala, and T. Ochiai, "ESPnet: End-to-end speech processing toolkit," in Proceedings of the 19th Annual Conference of International Speech Communication Association (INTERSPEECH), 2018, pp. 2207-2211.
[66] S. Watanabe, T. Hori, S. Kim, J. R. Hershey, and T. Hayashi, "Hybrid etc/attention architecture for end- to-end speech recognition." IEEE Journal of Selected Topics in Signal Processing, vol. 11, no. 8, pp. 1240-1253, 2017.
[67] A. Varga and H. Steeneken, "Assessment for automatic speech recognition: li. noisex-92: A database and an experiment to study the effect of additive noise on speech recognition systems," Speech communication, vol. 12, no. 3, pp. 247-251, 1993.
13

---

A Limitations
USR uses unlabelled samples during fine-tuning via pseudo-labelling, which is more computationally intensive than standard supervised fine-tuning due to (1) the increased data volume and (2) the high cost of pseudo-labelling outperforms state-of-the-art self-supervised methods (37.8% vs. 43.4% WER [18] in the LRS3 low-resource setting). Additionally, our approach efficiently generates pseudo-labels using a simple thresholding mechanism. Despite this, higher-quality labels are known improve speech recognition, often enhanced by techniques like beam search, language modelling, and combining CTC and attention scores. We do explore alternative filtering mechanisms, which we defer to future work.
However, our semi-supervised approach without pre-training still
B Societal Impact
Speech recognition technology can greatly benefit people with disabilities who may struggle to interact with devices using traditional input methods like keyboards. Visual speech recognition can assist individuals with aphonia, who cannot produce voiced speech. It has also been shown that models trained for visual speech recognition also aid in detecting fake videos by understanding natural mouth movements [58].
However, speech recognition technology also poses societal risks. It can be exploited for surveillance through, e.g., CCTV, necessitating appropriate government regulations. As in other machine learning applications, there may be biases in the datasets used to train the models. Biases related to gender, age, or ethnic background can lead to reduced performance for underrepresented groups. Addressing this requires training models balanced data employing bias-reduction techniques.
C Experiment Details
C.1 Dataset Details
LRS3. The LRS3 dataset [19] is the largest publicly accessible audio-visual dataset for continuous speech recognition with transcriptions. It includes approximately 430 hours of spoken sentences from TED Talks and features vocabulary of over 50,000 words spoken by thousands of different speakers. The dataset is collected by automatically tracking faces, synchronising the video/audio streams, and splitting the videos into individual sentences. The test set comprises roughly 1 hour of utterances from speakers not included in the training set.
LRS2. The LRS2 dataset [57], totalling 223 hours of footage from BBC programs, is the second- largest transcribed audio-visual dataset available for continuous speech recognition. The test is around 0.5 hours long. Like LRS3, LRS2 features an unrestricted vocabulary and includes thousands of diverse speakers. However, LRS3 tends contain videos of more variable quality, making it more challenging dataset for VSR.
WildVSR. WildVSR [59] is a recent VSR dataset, created by closely following the LRS3 dataset curation processes. The VSR dataset contains more challenging samples compared with LRS3, leading to significant drops in the VSR performance of models evaluated on WildVSR. The test set contains around hours footage.
VoxCeleb2. VoxCeleb2 [49] large- scale audio-visual dataset containing talking faces of celebri- ties, with about 6,000 speakers and over 2,400 hours of footage. The dataset includes elements like laughter. cross-talk, music, and other interference, with an unconstrained vocabulary. Since VoxCeleb2 is multilingual, we use an English-only version curated by [13], which consists of 1,323 hours of footage.
C.2 Data Licenses
LRS3 [19], VoxCeleh2 [49], and WildVSR [59] are licensed under CC BY-NC-ND 4.0. LRS2 [57] allows for academic, non-commercial research.
14

---

Table Configuration of our models. Unlike in [13, 17, 18]. the number of parameters includes the whole model, including the decoder and feature extractors.
Base
Base+
Large
Parameters (M)
86
171
503
Encoder blocks
12
12
24
Decoder blocks
6
6
Attention dimension
512
768
1024
Attention heads
8
12
16
MLP size
2048
3072
4096
Table 8: Supervvisa//emiisspperviee training settings.
Hyperparameter
Value
Training epochs
75
Warmup epochs
20
Optimiser
AdamW
Learning rate
3e-3 (LRS3), 2c-3 (LRS3+
Optimiser (B1, .22)
(0.9, 0.98)
Weight decay
0.04 Cosine decay
Learning rate schedule
Drop rate [61]
0.1 (Base), 0.2 (Large)
Gradient clipping threshold
3.0
Video augmentations
RandomCrop HorizontalFliip 155 (low-resource), 700 (high-resource) 2,400 (LRS3), 1,400 (LRS3+Vox2)
Frames per GPU (labelled)
Frames per GPU (unlabelled)
C.3 Pre-processing
We follow the video pre-processing protocol from related works [9, 13, 17, 18]. We remove motion jitter from the videos, crop a 96 x 96 region centred around the mouth for each frame, and apply a grayscale transformation. We note that audio used without pre-processing. As in [13, 17, 18), we tokenise the targets using SentencePiece [60] subword units with a vocabulary size of 1,000.
C.4 Model Configurations
Following [18], we use three model sizes: Base, Base+, and Large. While the Transformer encoders and decoders vary in size, the feature extractors remain unchanged, consistent with [17, 18] (which use the same feature extractors). The configuration of the models is summarised in Table 7. Base+ corresponds the Base models used in similar works [13, 14, 15]. We train our Base, Base+, and Large models on 32, 64, and 128 A A100 40GB GPUs, respectively.
C.5 Superrised/Semissuprrisee Training Settings
We use consistent settings across supervised training (Section 3.1) and semi-supervised training (Section 3.2). We train our models using AdamW [62] for 75 epochs with 20-epoch linear warmup [63] and a cosine learning rate decay [64]. We use gradient clipping and drop path [61] for regularisation. In addition the masking discussed in the main text, we also perform random spatial cropping (size 88 88) and horizontal flipping (probability 0.5) on the videos in a temporally consistent manner, as in [17, 18). The hyperparameter details are presented Table 8. We fix the seed to 42. It takes approximately 12 hours to train the Base model on the labelled data (32 GPUs). It takes around one, four, and six days train the Base (32 GPUs), Base+ (64 GPUs), and I Large (128 GPUs) models, respectively. Note that Base is trained LRS3, and Basc+ and Large LRS3+Vox: Vox2.
15

---

Table 9: Settings for pre-training.
Hyperparameter
Value
Training epochs
150 (LRS3), 75 (LRS3+
Warmup epochs
40 (LRS3), 20 (LRS3+VoxCeleb2)
Optimiser
AdamW
Learning rate
5e-3 (LRS3), 2e-3 (LRS3+ (ox33ee)x))l)b2)
Optimiser (31 (22)
(0.9, 0.98)
Weight decay
0.04 Cosine decay 0.1 (Base), 0.2 (Large)
Learning rate schedule
Drop rate [61]
Gradient clipping threshold
3.0
Video augmentations
RandomCrop HorizontalFlip
Frames per GPU
2,400 (Base), 1,800 (Base+), 900 (Large)
Table 10: More semi-supervised ablations under the LRS3 low-resource setting using our Base model (includes self-supervised pre-training).
(a) Filtering thresholds Tas Tan for CTC and at-
tention, respectively.
WER (%)
Tete
Talt
(b) Hard versus soft sampling.
A
AV
WER (%)
0.60
0.60
37.2
3.3
3.1
Sampling
0.80
0.80
36.0
3.2
3.0
AV
0.95
0.95
36.6
3.3
3.1
Hard
36.0
3.2
3.0
0.60
0.80
36.7
3.1
2.9
Soft
37.5
3.4
3.4
0.95
0.80
36.2
3.3
3.2
0.80
0.60
37.7
3.2
3.0
0.80
0.95
36.5
3.3
3.1
C.6 Pre-training Settings
The pre-training settings are similar. We use a longer schedule (in terms of number epochs) for LRS3 with 150 total training epochs and 40 warmup epochs. We also use higher learning rate of 5 10-3, The full settings are given in Table It takes approximately two days pre-train all models.
C.7 Decoding
We the ESPNet framework [65] for decoding, as in [17, 18), employing beam search with beam size of 40. The final beam search score
S = aSe + a) 3Sm.
(10)
where Sat and San are scores from the CTC and attention branches, respectively, and Si is the optional score from pre-trained language model, which is incorporated through shallow fusion [66]. Following [27, 17], we set a for all experiments. When using a language model, we select
from (0.1, 0.2, 0.3, 0.4] based on the validation set.
D More Ablations
D.1 Semi-supervised ablations.
Confidence threshold. Our default setting uses pseudo-labelling confidence threshold 7 of 0.8 both for the CTC and attention losses, for simplicity. In Table 10a, we investigate different threshold values, including the use of separate thresholds for the two losses. We observe that USR's 's performance
16

---

Table 11: More self-supervised ablations under the LRS3 low-resource setting using our Base
model.
(a) Mask probability.
(b) Pre-training target types.
WER (%)
Mask probability
WER (%)
V
A
AV
Target
A
AV
0.2
37.1
3.4
3.2
0.4
36.0
3.2
3.0
AV
36.0
3.2
3.0
0.6
36.7
3.1
2.9
A+V+AV
36.2
3.2
3.0
0.8
38.0
3.3
3.1
remains consistent across a range of different thresholds, with no clear improvement when using separate thresholds.
Hard versus soft sampling. Our greedy attention pseudo-labelling strategy involves choosing at each generation step the most likely pseudo-label according to the probability distribution given by consider alternative "soft sampling" approach well. We use
the decoder. For comparison, weighted sampling at each generation step, drawing label based on the entire distribution given by the decoder. Each label has chance of being selected proportional its estimated probability. This approach increases the variety of pseudo-labels but may reduce their quality since low-probability pseudo-labels more frequently used.
In Table 10b we compare the two approaches. We observe that hard sampling outperforms soft sampling for all three modalities. Future work can explore alternative methods to effectively increase pseudo-label variety.
D.2 Self-supervised ablations.
Mask probability. Table 1la, we compare different mask probabilities for pre-training, low mask probability can result in a trivial learning task, whereas high probability can make the task overly challenging. We find that probability between 0.4 and 0.6 achieves a good balance.
Combining targets. During pre-training, targets generated from audio-visual input and pre- dicted by students using masked auditory, visual, and audio-visual inputs. We explore predicting the combined targets from all input types by summing the corresponding outputs from the teacher, but as shown in Table 11b, this does not yield improvements over simply predicting the audio-visual targets.
E Comparisons with the State-of-the-Art WildVSR
WildVSR [59] is recent test set featuring more challenging "in- "in-the-wild" samples than LRS3. In Table 12, we evaluate our Large model WildVSR, trained using the high-resource setting (see Table 5). Our unified approach achieves similar VSR results to the modality-specift RAVEn when the latter uses an additional self-training stage.
Supervised vs. Semi-supervised Fine-tuning
In Table 13, we closely evaluate the differences between supervised and semi-supervised fine-tuning.
Supervised fine-tuning with few labelled samples is prone to overfitting, necessitating various training "tricks" to improve performance. For example, [17, 18] use a smaller decoder for the low low-resource setting, different learning rates for the encoder and decoder, and layer-wise learning rate decay [26]. use our Base model and low-resource setting to evaluate supervised and semi-supervised (our default) fine-tuning, with and without these strategies. As shown in Table 13a, while these "tricks" significantly benefit supervised training (consistent with [17]), they actually hurt semi-supervised fine-tuning. This suggests that semi-supervised training is less prone to overfitting, making these
17

---

Table 12: WildVSR results. We test our model from Table for VSR.
Labelled hours
Unlabelled
Shared params
WER (%)
Method
hours
Supervised
CM-seq2seq [6]
1,459
X
58.4
VTP [51]
698
75.6
VTP [51]
2,676
X
68.7
Auto-AVSR [27]
661
62.3
Auto-AVSR [27]
1,759
49.3
Auto-AVSR [27]
3,448
X
38.6
Seff/semi-supervised
AV-HuBERT [13]
433
1,326
51.7
AV-HuBERT w/ self-training [13]
433
1,326
X
48.7
RAVEn [17]
433
1,326
52.2
RAVEn w/ self-training (17)
433
1,326
X
46.7
USR
433
1,326
46.4
Table 13: Comparisons between supervised and our semi-supervised fine-tuning. We the LRS3 low-resource setting and our Base model.
(a) Fine-tuning "tricks" for supervised and semi- supervised fine-tuning.
(b) Pre-training target types for supervised fine- tuning.
WER (%)
Fine-tuning
WER (%)
V
A
AV
Target
V
A
AV
Sup
52.5
5.8
5.4
Sup w/ tricks
45.6
5.2
5.0
V
63.2
9.0
8.9
43.9
4.8
4.6
Semi
36.0
3.2
3.0
AV
45.6
5.2
5.0
Semi w/ tricks
39.3
3.2
3.0
regularisation methods unnecessary. In general, we noticed that using semi-supervised fine-tuning results in less sensitivity to pre-training hyperparameters (e.g., compare Tables 13b and 3a).
In Table 3a, observed that semi-supervised fine-tuning benefits most from audiovisual targets. Here, we fine-tune the same pre-trained model using only labeled data to assess the influence of target type supervised fine-tuning. Table 13b shows that audio-only targets perform best for supervised fine-tuning, consistent with findings from other works [15, 17]. As discussed in the main text, semi-supervised fine-tuning allows the model to leverage the rich and diverse information in audiovisual targets, which supervised fine-tuning struggles to achieve.
Table 14: Experiments with auditory noise. We compare USR with the
BRAVEn method on LRS3 with different nognal-to-noise-ratt (SNR) levels. We use Base models trained under the low-resource setting.
SNR (dB)
Clean
5
-5
BRAVEn (A) BRAVEn (AV)
4.0
15.6
24.6
99.0
4.0
12.4 J3.2
15.0 19.6
48.5 _30.5
USR (A)
3.2
14.3
26.9
100.4
USR (AV)
3.0 jo.2
6.1
10.1 (18.8
35.7 .64.7
18

---

Table 15: Error bars. We report the mean and standard deviation over five runs with random seeds. We use our Base model with LRS3 as the pre- pre-training dataset.
WER (%)
Setting
AV
Low-resource
36.2 + 0.40
3.25 + 0.10
3.02 + 0.04
High- High-resource
34.2 + 0.56
1.77 0.18
1.68 + 0.11
G Experiments with Auditory Noise
We have demonstrated that AVSR slightly outperforms ASR on the clean LRS3 test set. However, it in the presence of auditory noise that AVSR truly excels, as visual cues help clarify ambiguous utterances. Table 14 presents ASR and AVSR results under varying levels of audio babble noise from the NOISEX dataset [67]. We employ our Base model under the low-resource setting with LRS3 as the pre-training dataset. Notably, the noise is added to the LRS3 test set, and the model is not trained on noisy data. We observe that as noise levels increase (and the signal-to-noiss ratio decreases), the performance gap between AVSR and ASR widens. Interestingly, this gap is more pronounced for USR compared to the modality-specifft BRAVEn.
H
Error Bars
Due to high computational demands and in line with previous studies [13, 17, 18, 15, 16], we do include error bars for our main results. To assess the variability of our method across multiple training runs, Table 15 presents the mean and standard deviation over five runs with different random seeds for our low- and high-resource settings, using our Base model with LRS3 as the pre-training dataset. We observe that the results are consistently stable around the mean.
I Qualitative Differences between Self-supervised Pretext Tasks
Our pre-training method shares similarities with recent audio-visual self-supervised tasks, RAVEn [17], BRAVEn [18], and AV-data2vec [15]. These methods employ an EMA-based teacher to generate targets from unmasked data, which the student predicts using masked inputs. Here, we compare and contrast our USR pretext task with these methods.
I.1 Comparisons with RAVEn/BRAVEn
RAVEn and BRAVEn pre-train separate Transformer encoders for visual and auditory inputs, which are then fine-tuned for ASR and VSR. AVSR can be performed through shallow fusion of visual and auditory features. In contrast, USR pre-trains single student Transformer encoder for auditory, visual, and audiovisual inputs, significantly reducing training and inference costs.
We adopt the approach of using shallow Transformer encoder predictor, which has been shown to improve representation learning [17]. However, while RAVEn and BRAVEn use separate predictors for visual and auditory features (with BRAVEn also using differently-sizer predictors), use single predictor for modalities, simplifying the architectural design.
I.2 Comparisons with AV-data2vec
AV-data2vec also unifies pre-training by using a single Transformer encoder for modalities. However, while AV-data2vec employs random modality sampling, we compute all med-modalyty losses each iteration, amortising the cost of target generation (see Section 4. AV-dat22vec's use of a scheduler for modality probabilities increases the complexity of the pre-training process. Furthermore, AV-data2vec uses audio-only targets, whereas we use audiovisual targets, which are shown to perform best for our semi-supervised fine-tuning (see Section 3.2).
19

---

Table 16: Comparison with AV-CPL. LRS3 results for the low-resource (LR) and high-resource
(HR) labelled data settings. We show results for the Large model using LRS3+Vox2 as the pre-training dataset.
WER (%)
LR
WER (%)
HR
Method
A
AV
V
A
AV
AV-CPL [36]
56.7
10.0
10.4
47.4
2.3
2.2
USR
26.9
2.4
2.4
22.3
1.2
1.1
Table 17: Summary the impact of semi- and self-supervised training under the LRS3 low- resource setting using our Base model. We compare four approaches: supervised training on 30 hours of labelled data, seff-supervised pre-training with supervised fine-tuning, semi-supervised training, and self-supervised pre-training with remi-supervised fine-tuning.
Self-supervised pre-training
WER (%)
Setting
Fine-tuning
V
A
AV
Only labelled data Self-supervised
X
Supervised
61.8
8.9
8.4
Supervised
43.9
4.8
4.6
Semi-supervised
X
Semi-supervised
37.8
4.0
3.9
Self- + semi-supervised
suni-supervised
36.0
3.2
3.0
J Comparison with AV-CPL
As mentioned in Section 2, the recent AV-CPL method [36] uses pseudo-labelling to train a single model for ASR, VSR, and AVSR, similar to our semi-supervised approach described in Section 3.2. Table 16 compares USR with AV-CPL on the low- and high-res high-resource labelled data settings using the Large model and LRS3+Vox2 between the two methods, which we attribute to USR' 's use of CTC-attention training, self-supervised pre-training, and pseudo-label filtering, among other design choices studied in Section 4.
the pre- pre-training dataset. We observe dramatic WER differences
K Summary of the Impact of Semi- and Self-supervised Training
Sections 4.2, 4.3, and Appendix F demonstrate the impact of self- and semi remi-supervised learning on speech recogmition performance. Table 17 summarizes the contributions of each component. Self-supervised pre-training on the full LRS3 dataset, followed by supervised fine-tuning on 30 hours of LRS3 (see Appendix F), outperforms supervised training on the same 30 hours alone, as expected. Additionally, semi-supervised training (without pre-training) significantly surpasses the self-supervised baseline. Combining self-supervised pre-training with semi-super fine-tuning yields the best results.
L Failure Cases
Table 18 presents some failure cases from the LRS3 test set. We evaluated our Large model trained in high-resource setting with LRS3 and VoxCeleb2. While VSR tends to produce more errors than ASR and AVSR, these errors are often related to phonetically similar sounds, such as "this" vs. "these" "disguised" "denies." Additionally, using both auditory and visual modalities (AVSR) can improve the model's ability to distinguish challenging samples, such as "Mali Wear" vs. "malware."
20

---

Table 18: Failure cases on the LRS3 test set. We use the Large model trained in the high-resource setting with LRS3+ VoxCeleb2.
Source
Transcription
Groundtruth
And all of this matters greatly because public safety to me is the most important function
VSR
And all of these matters are crazy because public safety to me is the most important function
ASR
And all of this matters greatly because public safety me is the most important function
AVSR
And all of this matters greatly because public safety me is the most important function
Groundtruth
I'm here to tell you the story of crazy love, a psychological trap disguised as love I'm here to tell you the story of crazy love, psychological trap denics the love
VSR
ASR
I'm here to tell you the story of crazy love, a psychological trap disguised as love I'm here tell you the story of crazy love, psychological trap disguised as love
AVSR
Groundtruth
It took six days to deploy a global malware campaign
VSR
It took six days to deploy our global market campaign
ASR
It took six days to deploy our global Mali Wear campaign
AVSR
It took six days to deploy global malware campaign
Groundtruth
It worked for the Oakland A's and it worked in the state of New Jersey It worked for the Oaklands and it worked in the state of New Jersey It worked for the Oakland Asia and it worked in the state of New Jersey
VSR
ASR
AVSR
It worked for the Oakland A's and it worked in the state of New Jersey
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: See Sections 1, 3, and 5.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
See Appendix A.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
21

---

The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't It acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: See Section 4 and Appendix C.I.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
22

---

While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c)
If the contribution is new model (e.g., , a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification:
We provide code as part of the supplementary material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cc)
heaeiic/ggie////eeee /tt.dd//oooeeeuuiiiseeeesesseees oeiionicyyc for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: / //nips. wwtpcc/pppppc.ccc.cc.c.ccc..tttt htliiicgguiee///ee Coode/ubieeiieeece ssionPolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: See Section 4 and Appendix C.I.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
23

---

The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: See Appendix H.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula,
call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: See Appendices C.4, C.5, and C.6.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't It make it into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: //tearr////////////eee hwtpppplwww..hccccceieeeeeeeeeee
Answer: [Yes]
Justification:
The research does not violate the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

---

If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: See Appendix B.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deploymentt of technologics that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path any negative applications, the authors should point it out. For example, it is legitimate point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
25

---

Answer: [Yes]
Justification: See Section C.2.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors encouraged to reach out the asset's ' creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
26

---

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
27


Reviews:
[{"review_id": "UEyo2TCWtG", "reviewer": "Reviewer_d2RY", "summary": "This paper unifies the ASR, VSR, and AVSR tasks in a single model and shows the performance benefits of a single model in LRS3 data. There are several attempts at unifying these three models, but I think this is the first successful trial of realizing it. The paper proposes an effective training strategy to avoid losing performance on each task. Together with their self-supervised training, the model archives SOTA performance in a similar range of the training data.", "questions": "- Page 4, line 110: I'm a bit confused about \"We set $\\lambda_{\\text{ctc}}$ to 1.\" Do you mean that you always set $\\lambda_{\\text{ctc}}$ to 1? No attention weights? Is it related to Table 2-d? Please clarify it.\n- Equation (4): Why didn't you prepare a different weight for a and av?\n- Section 3.2, Filtering: Did you use the same threshold for CTC and ATT? The dynamic range of c and a could be different, and I'm not sure that using the same threshold is optimal.\n- Section 4: Did you only use a Transformer architecture? How about using a Conformer architecture?\n- It is not a question but a suggestion. I recommend you emphasize the results of the multiple databases in the abstract to claim the generalization of this work across the database.", "limitations": "The paper has independent sections about limitations and Societal Impact, which describe the current issue due to the computational cost, the importance of the VSR, and the risk of general speech recognition technology.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- the first successful method of realizing the ASR, VSR, and AVSR tasks in a single model while maintaining/improving the performance for each task\n- Good reproducibility based on the code release, use of the public data, and detailed experimental configurations/analyses.\n- Easy to read. Although the technique is a little bit complicated with a lot of terms depending on the architecture (CTC, attention, modality, training modes (self-supervised/supervised), the paper always provides some rationales (e.g., from the reference or experiments) to justify their methods\n  - detailed ablation experiments support their design choices and strategies. \n- The paper also shows the effectiveness with multiple databases (LRS3, LRS2, and WildVSR)", "weaknesses": "- the technical novelty is not very strong. Most techniques are well-known or straightforward (e.g., the use of CTC, pseudo-label filtering, etc.).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review and time. Below we address the key concerns raised. \n\n> the technical novelty is not very strong. Most techniques are well-known or straightforward (e.g., the use of CTC, pseudo-label filtering, etc.).\n\nWhile individual components of our work have been previously used in other studies (as discussed in the Related Work), we believe that the USR framework as a whole represents significant novelty. As noted by the reviewer, earlier efforts to unify ASR, VSR, and AVSR have often lagged behind modality-specific approaches. Our research demonstrates, for the first time in the literature, that a combination of self-supervised and semi-supervised learning can produce a unified model that achieves state-of-the-art performance across all tasks. This success is attributed to key technical design choices, including (but not limited to) the use of a greedy, computationally efficient attention-based pseudo-labelling approach; a multi-modal feature extraction step that enables amortisation of the pseudo-label generation costs across the three modalities; and multi-modal (audio-visual) target prediction in self-supervised pre-training, which previously proved unsuccessful with supervised fine-tuning. Furthermore, we believe that the straightforward and intuitive nature of USR enhances its utility and potential impact in the community.\n\n> Page 4, line 110: I'm a bit confused about \"We set to 1.\" Do you mean that you always set to 1? No attention weights? Is it related to Table 2-d? Please clarify it.\n\nWe apologise for the confusion; this is a typo. The intended value for the CTC loss weight was 0.1, not 1. We always use CTC-attention training, except for the ablation in Table 2d where we study the performance of a CTC-only loss. We have fixed this typo in the paper.\n\n> Equation (4): Why didn't you prepare a different weight for a and av?\n\nIn preliminary experiments, we observed that the training dynamics for ASR and AVSR were very similar, and hence decided to use a combined weight for the two modalities in Eq. 4 and Eq. 7 in order to reduce the number of hyperparameters. We have now added this point to the paper (Section 3.1).\n\n> Section 3.2, Filtering: Did you use the same threshold for CTC and ATT? The dynamic range of c and a could be different, and I'm not sure that using the same threshold is optimal.\n\nSimilarly, we use the same threshold for CTC and attention for simplicity. However, we agree that the dynamic ranges between the two could be different and have run experiments with separate thresholds to investigate this point. The results are in Table 1a in the attached rebuttal PDF. We observe that USR's performance remains consistent across a range of different thresholds, with no clear improvement when using separate thresholds. We added this experiment in the Appendix.\n\n> Section 4: Did you only use a Transformer architecture? How about using a Conformer architecture?\n\nWe used Transformer-based architectures with convolutional frontends to align with closely related works [13-18, 20] for fair comparisons (e.g., see Table 4). However, given that USR is agnostic to the choice of encoder architecture, we agree it would be interesting to explore other architectural variants, including Conformers, which could improve results even further. Still, we believe this direction lies outside the scope of our present work and therefore defer it to future research. This idea for future work has now been included in the Conclusion.\n\n> It is not a question but a suggestion. I recommend you emphasize the results of the multiple databases in the abstract to claim the generalization of this work across the database.\n\nThank you for this suggestion. We have updated the abstract accordingly.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QHUA4lUi4x", "reviewer": "Reviewer_d9WG", "summary": "This paper proposes USR, a unified speech recognition model that leverages pseudo labels during fine-tuning. It introduces a single model capable of handling three tasks—ASR, VSR, and AVSR—simultaneously, delivering state-of-the-art performance.", "questions": "1. During pretraining, have you explored using audio-only targets? If so, what was the performance like compared to AV targets? How does it compare to AV-HuBERT?\n\n2. Why do you incorporate all three features (audio, video, audio-visual) during fine-tuning? Is there a rationale or experimental evidence supporting this approach?\n\n3. There's no need to adhere strictly to the architectures like AV-HuBERT or AV-data2vec. Consider experimenting with more advanced video encoders since visual features are often not well-extracted in previous studies.\n\n4. For pseudo label sampling, why opt for a greedy search? Have you considered trying soft sampling instead?", "limitations": "The limitations have been discussed in the paper.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-organized. Although the USR system is relatively complex, the paper presents each module with detailed descriptions and clear illustrations, making it easy for readers to follow.\n\n2. The experiments, including ablations, are extensive. All experimental details are included, making it easy to reproduce the results.\n\n3. The USR system leverages pseudo labels during the fine-tuning stage. While pseudo labeling is not a novel technique in ASR or AVSR, USR enhances the performance of ASR, VSR, and AVSR through carefully designed training procedures. The illustration of the pseudo labeling process is also clear.\n\n4. The system achieves nearly state-of-the-art performance across all tasks.\n\n5. The literature review is thorough.", "weaknesses": "1. While not a unique weakness to this paper, the complexity of training current SSL-based VSR or AVSR systems remains a challenge. Introducing additional modalities significantly increases complexity compared to speech-only SSL systems. Notably, the reduction in GPU hours is minimal compared to previous works, and the convergence speed is exceedingly slow. Future work should address these issues.\n\n2. Performance is highly sensitive to certain configurations, such as the ratios of pseudo labels and the use of EMA. However, the paper lacks an analysis of why this sensitivity occurs or suggestions on how to mitigate it. These are common weaknesses in related work.\n\n3. The results do not consistently achieve state-of-the-art performance. The authors should experiment with other hyperparameters, such as learning rates, during fine-tuning to improve outcomes.\n\n4. Failure cases were not discussed too much.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review and time. Below we address the key concerns raised. \n\n> The complexity of training current SSL-based VSR or AVSR systems remains a challenge. \n\nWe recognise that VSR and AVSR systems present unique challenges compared to audio-only systems, and one of our future goals is to improve the computational efficiency of multi-modal speech recognition. Despite these challenges, we believe that audio-visual speech representation learning is an exciting and promising area of research. Leveraging visual modalities like lip movements offers significant advantages in disambiguating difficult utterances, particularly in noisy environments or when audio is unavailable (see Table 12, Appendix). VSR and AVSR can also improve communication for individuals who have difficulty producing voiced speech. Additionally, evidence suggests that incorporating multiple modalities during pre-training can enhance the performance of audio-only systems (e.g., see [13, 17]).\n\n> Performance is highly sensitive to certain configurations, such as the ratios of pseudo labels and the use of EMA.\n\nWhile self-supervised learning systems can indeed be sensitive to hyperparameters, we believe that USR significantly reduces this sensitivity. Our semi-supervised pseudo-labelling framework is sensitive to extreme threshold values (see Table 2a) and somewhat sensitive in terms of VSR to the weighting of labelled and unlabelled losses (see Table 2b). In Section 4.2, we hypothesise that this sensitivity arises from the inherent trade-off between label quantity and quality, which must be balanced. However, as shown in Table 3, USR’s performance remains stable across a range of pre-training hyperparameter configurations. We attribute this stability to our semi-supervised method, which leverages abundant unlabelled samples during fine-tuning. Appendix E demonstrates that our method is more robust to pre-training target types than supervised fine-tuning (compare Table 11b and Table 3a) and does not require additional training tricks for strong performance (see Table 11a), unlike other works (e.g., [13, 17, 18]). We have updated the paper to better highlight USR's reduced sensitivity to pre-training hyperparameters.\n\n> The results do not consistently achieve state-of-the-art performance.\n\nWe have indeed carefully tuned hyperparameters, such as the learning rate, to obtain our final results. In the well-established and highly competitive LRS3 high- and low-resource benchmarks (see Table 4), our model surpasses (sometimes by a large margin) or matches the previous state-of-the-art modality-specific models in 16 out of 18 cases. In the remaining two cases, we are marginally behind (1.6% vs. 1.4% and 2.4% vs. 2.3% WER). *Crucially, we achieve these results using a single model for ASR, VSR, and AVSR, while other methods require separate models for each modality, resulting in ~3x the number of weights during inference.* We also achieve SotA results on WildVSR (Table 9) and LRS2 (Table 10).\n\n> Failure cases were not discussed too much.\n\nThank you for the suggestion. We added a discussion on failure cases in the Appendix. See Table 3 in the rebuttal PDF. We observe that, while VSR tends to produce more errors than ASR and AVSR, these errors are often related to phonetically similar sounds, such as \"this\" vs. \"these\" or \"disguised\"\" vs. \"denies.\" Additionally, using both visual and auditory modalities (AVSR) can improve the model's ability to distinguish challenging samples, such as \"Mali Wear\" vs. \"malware\". \n\n> During pretraining, have you explored using audio-only targets? If so, what was the performance like compared to AV targets? How does it compare to AV-HuBERT?\n\nWe have explored using audio-only targets for pre-training with our semi-supervised fine-tuning in Table 3a, where we observe that AV targets work best. Still, with audio-only targets, we achieve WERs of 37.3% for VSR, 3.2% for ASR, and 3.1% for AVSR, significantly outperforming AV-HuBERT's 51.8%, 4.9%, and 4.7%, respectively. Additionally, we experimented with supervised fine-tuning (see Appendix E), observing WERs of 43.9%, 4.8%, and 4.6% for the three tasks (see Table 11b). Notably, AV-HuBERT uses a separate model for each task, whereas we use a single unified model.\n\n> Why do you incorporate all three features (audio, video, audio-visual) during fine-tuning?\n\nOur goal was to develop a single, unified model capable of performing well on audio-only (ASR), video-only (VSR), and audio-visual (AVSR) data during inference, thereby reducing the computational and memory redundancies associated with separate models per task. To achieve this, we fine-tune the model using all three types of data, enabling it to effectively perform each task. We have now emphasised this point in Section 3.1.\n\n> Consider experimenting with more advanced video encoders.\n\nWe used Transformer-based architectures with convolutional frontends to align with closely related works [13-18, 20] for fair comparisons (e.g., see Table 4). However, given that USR is agnostic to the choice of encoder architecture, we agree it would be interesting to explore other architectural variants, which could improve results even further. Still, we believe this direction lies outside the scope of our present work and therefore defer it to future research. This idea for future work has now been included in the Conclusion.\n\n> For pseudo label sampling, why opt for a greedy search? Have you considered trying soft sampling instead?\n\nWe initially chose a greedy search for its efficiency and effectiveness. Based on your suggestion, we now also experimented with a soft sampling approach, where we used weighted sampling at each generation step. The results are in Table 1b of the rebuttal PDF. We see that hard sampling outperforms this variant of soft sampling but believe that exploring more sophisticated methods to effectively increase pseudo-label variety is a promising direction for future research.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8CyQfJzuuT", "reviewer": "Reviewer_Fi7g", "summary": "This paper proposes a training methodology for a *single* model which can use *either* audio, visual, or audiovisual features as input for automatic speech recognition. This is done by enforcing a training batch always includes (feature,label) pairs of all three modalities, using a 1D/2D ResNet-18 feature extractor for audio and video, respectively. These features are processed by a Transformer encoder-decoder model to obtain an ASR prediction. Furthermore, the authors explore a semi-supervised fine-tuning approach and a self-supervised initialization stage, both using a student-teacher approach, and within the same unified methodology. This allows the authors to produce a model which is competitive with state-of-the-art models while using a significantly less data.", "questions": "In which settings/experiments is the transformer decoder used?\n\nIn table 3 (A), is there a reason for not trying targets A + V + AV, as during fine-tuning?\n\nYou state in line 103 that features from the 3 modalities are concatenated along the batch dimension for efficient processing. However, Table 1 (B) shows that random sampling of modalities performs much worse, requiring 3x more epochs for similar performance. So it seems to me it's not only done for efficient processing, but also for effective optimization? \n\nAlso, do [13, 15] in line 179 share parameters for each task or not? According to Table 4 they do not, but if you use random sampling of modalities, how does this explain their relevance to Table 1 (B)?\n\nWhat is the CTC attention in Table 2 (C)? Is this simply equation 3 with $\\lambda_{ctc} < 1$? I might have missed it, but it seems to me the method section does not explain these 2 different loss types?", "limitations": "The methods requires all data to be audio-video paired. An interesting future direction could be the inclusion of audio-only data in the framework.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "I think the proposed method is interesting for researchers in the audio-visual ASR domain and will spur future work. The paper is well-written with clear English, barring some questions I have stated below. The authors do a good job presenting their results, referring to details in the appendix where required. The ablation experiments clearly show readers how their proposed methodology behaves and why certain design decisions were made. The authors also shared their code and model checkpoints, which significantly increases the reproducibility and impact of this paper.", "weaknesses": "The model architecture seems a bit unclear to me. Specifically, line 88 states the use of a transformer encoder-decoder model. However, line 104 states a single FC layer on top of the encoder for vocabulary predictions, while line 107 states to use the decoder output sequence, which is subsequently not used as $1 - \\lambda_{ctc}=0$. So the decoder is not actually used during fine-tuning? How is inference actually done?\n\nI see no mention of a fixed random seed for running experiments, are all model initialized equal? This seems important as the paper does not have error bars/does not run experiments multiple times\n\nMinor editing comments:\n* Table titles must appear above the table as per the formatting instructions. \n* The table/figure combinations on Page 6 are confusing. Could you separate the figures as not part of a (sub)table?\n* A small description of LRS3 would be desirable for those not familiar with the dataset (e.g., how many hours does the unlabeled portion have (line 190), what is the data source, how was it collected, how large is the test set?)\n* line 97: 0.4 and 0.6 seconds for each second of ...", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review and time. Below we address the key concerns raised. \n\n> Line 104 states a single FC layer on top of the encoder for vocabulary predictions, while line 107 states to use the decoder output sequence, which is subsequently not used as 1−λctc=0. So the decoder is not actually used during fine-tuning? How is inference actually done?\n\nWe apologise for the confusion; this is a typo. The intended value for the CTC loss weight was 0.1, not 1, and the decoder is indeed used during fine-tuning. Inference is then performed with both CTC and attention scores (see Appendix C.6). We have now fixed this typo in the paper.\n\n> I see no mention of a fixed random seed for running experiments, are all model initialized equal?\n\nThank you for raising this point. We indeed use a fixed random seed (42) for our experiments, and have now clarified this in the paper. Due to high computational demands and in line with previous studies [13-18, 20], we do not include error bars for our main results. However, we do show error bars for a subset of our main experiments in Table 13, where we observe that the results are consistently stable around the mean.\n\n> Minor editing comments\n\nThank you for the editing suggestions, which have now been incorporated into the paper. Specifically, we made the following changes: we placed the captions above all tables; we separated Table 2a and the rest of Table 2 into distinct figures and tables; we added a description of the datasets in the Appendix; and we fixed the typo you pointed.\n\n> In which settings/experiments is the transformer decoder used? What is the CTC attention in Table 2 (C)?\n\nWe apologise again for the confusion caused by the typo regarding the CTC loss weight, which should be 0.1. In Table 2d, CTC-attention (i.e., including the Transformer decoder) is our default loss. The CTC-only configuration, which corresponds to a CTC loss weight of 1, is used solely for the purposes of this ablation study.\n\n> In table 3 (A), is there a reason for not trying targets A + V + AV, as during fine-tuning?\n\nPlease note that during fine-tuning, we use only the AV targets (see Figure 1, top-left) because they provide the richest information and allow us to amortise the pseudo-labelling cost across the three modalities. Incorporating all three targets during fine-tuning is challenging, as it is unclear how to efficiently combine / predict the per-modality pseudo-label sequences, which are generated auto-regressively by the decoder. However, during pre-training, which involves only the encoder, we can more easily combine the targets from the three modalities. In Table 2b of the rebuttal PDF, we compare audio-visual targets (our default) with the sum of the per-modality targets. We observe that combining the per-modality targets results does not outperform predicting only the AV targets. Additionally, predicting all targets is more computationally expensive because the teacher encoder must process auditory, visual, and audio-visual features rather than just the audio-visual ones. We added this experiment in the Appendix.\n\n> You state in line 103 that features from the 3 modalities are concatenated along the batch dimension for efficient processing. However, Table 1 (B) shows that random sampling of modalities performs much worse, requiring 3x more epochs for similar performance. So it seems to me it's not only done for efficient processing, but also for effective optimization?\n\nRandomly sampling the modalities means that, on average, the model is exposed to only one-third of the modalities at each iteration compared to concatenating all three modalities along the batch dimension. As a result, it would require approximately three times more epochs to achieve similar performance. However, an advantage of our approach is that it allows us to use the same pseudo-labels for all three modalities at each iteration, thus amortising the pseudo-label generation cost across modalities, which would not be possible with random modality sampling. We have updated the caption of Table 1b to explain why random sampling is trained for longer.\n\n> Also, do [13, 15] in line 179 share parameters for each task or not? According to Table 4 they do not, but if you use random sampling of modalities, how does this explain their relevance to Table 1 (B)?\n\n[13, 15] do share parameters during pre-training, hence their relevance to Table 1b, but they separately fine-tune the resulting model on ASR, VSR, and AVSR, resulting in a separate model for each task during inference. In contrast, USR yields a single unified model that is capable of performing all three tasks during inference (section 2, “Single model for multiple modalities” includes a relevant discussion).\n\n> The methods requires all data to be audio-video paired. An interesting future direction could be the inclusion of audio-only data in the framework.\n\nIndeed, thank you for this interesting suggestion, which we have now added to our Conclusion.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "C9JloFz0hw", "reviewer": "Reviewer_WdRe", "summary": "This paper proposes a unified architecture and training method for auditory/visual speech recognition. Building upon this model, the authors introduce a semi-supervised pseudo-labeling method to leverage unlabeled audio-visual data, as well as self-supervised pre-training to enhance model performance. Experiments indicate that the model achieves state-of-the-art performance on A/V/AVSR.", "questions": "1. How is the weight of the teacher model in self-supervised pretraining initialized? Is it initialized randomly or with pretrained weight on another task?\n\n2. Did the author make a comparison between the teacher-student self-supervised pretraining in the paper with masked-autoencoding training of audio and/or visual features? Is the proposed pretraining method superior?\n\n3. Did the author investigate the effect of different masking ratios?", "limitations": "The authors discuss the limitation of their work in appendix A.", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. This work for the first time proposes an effective model and training procedure for unifying auditory and visual speech content recognition, which is of high novelty and practical significance. \n\n2. The author conducted comprehensive and extensive ablation studies, verifying the characteristics of the model and the effectiveness of each step in the training paradigm. The experimental results are robust and credible, offering significant guidance for related research.", "weaknesses": "The article has no obvious flaws, but there are some questions that I hope the authors can clarify (see questions).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review and time. Below we address the key concerns raised. \n\n> How is the weight of the teacher model in self-supervised pretraining initialized? Is it initialized randomly or with pretrained weight on another task?\n\nThe teacher model is randomly initialised in pre-training and improves throughout training via an exponential moving average operation through bootstrapping [44], as in related works [15, 17, 18]. We will emphasise this more in the paper.\n\n> Did the author make a comparison between the teacher-student self-supervised pretraining in the paper with masked-autoencoding training of audio and/or visual features? Is the proposed pretraining method superior?\n\nWe are not aware of works in audio-visual self-supervised learning for speech recognition that employ masked autoencoding (MAE) for pre-training, i.e., predicting raw audio and/or visual features as in [45]. Although the investigation of MAE in the context of audio-visual speech recognition is outside the scope of our work, we agree that it is an interesting direction for future research. Our pre-training task, which combines masked prediction (similarly to MAE) with teacher-student training, aligns more closely with recent successful models in this area, such as RAVEn [17], AV-data2vec [15], and BRAVEn [18].\n\n> Did the author investigate the effect of different masking ratios?\n\nInitially, we conducted a set of preliminary experiments and chose to use the masking ratio from [18]. However, we agree that exploring different masking ratios is a valuable ablation study, and thank you for prompting us to investigate this further. We have now conducted experiments with different masking ratios for pre-training (see Table 2a in the rebuttal PDF). We find that the model performs well with a masking probability of 0.4-0.6 and is, interestingly, quite robust to the exact choice of masking ratio. We added this ablation in the Appendix.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely thank the reviewers for their thoughtful comments, which have greatly contributed to improving our paper. We are pleased that the reviewers recognise the effectiveness of our method (Reviewers d2RY, d9WG, WdRe), the quality of our experiments (Reviewers d2RY, d9WG, Fi7g, WdRe), and the reproducibility of our results (Reviewers d2RY, d9WG, Fi7g). We also appreciate their acknowledgment of the potential impact and practical significance of our work (Reviewers Fi7g, WdRe), as well as the quality of our writing and presentation (Reviewers d2RY, d9WG, Fi7g, WdRe).

We have addressed the reviewers' concerns with individual responses to each review. Please see the attached rebuttal PDF, which includes new experimental results. Key changes to the paper are summarised as follows:

* We included in the Appendix more ablations for self-supervised pre-training (different mask probabilities and target types) as well as for semi-supervised fine-tuning (additional filtering thresholds and comparisons between hard and soft sampling).  
* We provided in the Appendix detailed descriptions of the datasets used in the paper.
* We added failure cases and a corresponding discussion in the Appendix.
* We moved captions above the tables and separated Table 2a and the rest of Table 2 into distinct figures / tables.
* We highlighted our state-of-the-art results on LRS2 and WildVSR in the Abstract.
* We improved the clarity of the text, for example, by emphasising our method’s reduced sensitivity to pre-training hyperparameters and clarifying that the teacher is randomly initialised in the pre-training stage.
* We added further ideas for future work in the Conclusion, including exploring alternative encoder architectures and the use of extra audio-only data.
* We fixed typos identified by the reviewers, including an error in the CTC loss weight, which was mistakenly listed as 1 instead of 0.1 in the original paper.

Meta-review: all 4 reviewers vote to accept this paper.
The paper is well written, easy to read and is of great interest to the community. 
The main weakness is that the novelty of the paper can be improved.
Decision: Accept (poster)