Pay Attention to CTC: Fast and Robust Pseudo-Labelling for Unified Speech Recognition
Authors: Alexandros Haliassos
Rodrigo Mira
Stavros Petridis
Venue: ICLR 2026 Poster (2026)
Keywords: Speech Recognition
Audiovisual Learning
Lipreading
Semi-Supervised Learning
Pseudo-Labeling
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Unified Speech Recognition (USR) has emerged as a semi-supervised framework for training a single model for audio, visual, and audiovisual speech recognition, achieving state-of-the-art results on in-distribution benchmarks. However, its reliance on autoregressive pseudo-labelling makes training expensive, while its decoupled supervision of CTC and attention branches increases susceptibility to self-reinforcing errors, particularly under distribution shifts involving longer sequences, noise, or unseen domains. We propose CTC-driven teacher forcing, where greedily decoded CTC pseudo-labels are fed into the decoder to generate attention targets in a single forward pass. Although these can be globally incoherent, in the pseudo-labelling setting they enable efficient and effective knowledge transfer. Because CTC and CTC-driven attention pseudo-labels have the same length, the decoder can predict both simultaneously, benefiting from the robustness of CTC and the expressiveness of attention without costly beam search. We further propose mixed sampling to mitigate the exposure bias of the decoder relying solely on CTC inputs. The resulting method, USR 2.0, halves training time, improves robustness to out-of-distribution inputs, and achieves state-of-the-art results on LRS3, LRS2, and WildVSR, surpassing USR and modality-specific self-supervised baselines.

Published as a conference paper at ICLR 2026
PAY ATTENTION TO CTC: FAST AND ROBUST PSEUDO- LABELLING FOR UNIFIED SPEECH RECOGNITION
Alexandros Haliassos' .2 Rodrigo Mira², Stavros Petridis 1.2 NatWest AI Research Imperial College London
al lexandres.. wwwahallsssoo....m. wtw.wsss.sstttttt......s........ con.
@natwest........................
ABSTRACT
Unified Speech Recognition (USR) has emerged as a semi-supervised framework for training single model for audio, visual, and audiovisual speech recognition, achieving statc-of-the- results on in-distribution benchmarks. However, its reliance on autoregressive pseudo-labelling makes training expensive, while its decoupled supervision of CTC and attention branches increases susceptibility to self-reinforcing errors, particularly under distribution shifts involving longer sequences, noise, or unseen domains. We propose CTC-driven teacher forcing, where greedily decoded CTC pseudo-labels are fed into the decoder generate attention targets in single forward pass. Although these can be globally incoherent, in the pseudo-labe labellll setting they enable efficient and effective knowledge transfer. Because CTC and CTCC-drven attention pseudo-labels have the same length, the decoder can predict both simultaneously. benefiting from the robustness of CTC and the expressiveness of attention without costly beam search. We further propose mixed sampling to mitigate the exposure bias of the decoder relying solely on CTC inputs. The resulting method, USR 2.0, halves training time, improves robustness to out-of of-iisiributtionn inputs, and achieves state-of of-the-art results on LRS3, LRS2, and WildVSR, surpassing USR and codality-specifff self-supervise baselines.
INTRODUCTION
Speech recognition includes multiple closely related tasks: automatic speech recognition (ASR)
from audio (Abdel-Hamid et al. 2014 Chiu et al. 2018), visual speech recognition (VSR) from lip
movements (Assael et al.] [2066] Martinez et al. 2020), and audiovisual speech recognition (AVSR),
which combines both modalities (Petridis et al. 2018 Ma et al. 2021). Despite their complementary
nature, ASR, VSR, and AVSR have traditionally been studied isolation (Baevski et al., 2020 Hsu
eet al. 2021; Assael et al. 2016), leading to separate models that are not only more cumbersome to deploy, but also forgo the advantages of sharing representations across modalities (Akbari et al.
2021 Girdhar et al. 2022). Recent work in audiovisual self-supervised learning (Shi et al. 2022a Zhu al. [2023] Lian al. 2023) has explored unified pre-training across modalities but still often relies on separate fine-tuning stages for ASR, VSR, and AVSR, resulting in distinct models per task.
Unified Speech Recognition (USR) Haliassos al. 2024a) offers a promising step toward unification by incorporating unlabelled data during fine-tuning, thus reducing modality-specific sensitivity and enabling single model to achieve state-of-the-art in-distribution performance across all tasks. Its decoupled two-branch architecture combines Connectionist Temporal Classification (CTC) (Graves) (Chorowski et al.) 2015) supervision, where each student branch
al. 2006) and attention- independently guided by Seuudollabels (PLs) from the corresponding branch of momentum teacher (Tarvainen & Valpola, 2017). While effective in distribution, this design leaves USR's 's atten- tion decoder with two key limitations: (1) PL generation is bottleneck, its autoregressive (AR) decoding is slow and must run at every training step, and (2) decoupled supervision increases vulner- ability to the seiforeinn training loop (Sohn et al. 2020): the student can be harmed by noisy AR PLs from the teacher, which in turn degrade the teacher itself as it is moving average of the student.
(OOD) errors. That is, mistakes from AR decoding are compounded by
To address these limitations, we propose USR 2.0, which combines the efficiency and robustness of CTC with the expressiveness of attention-based decoding. As shown in Figure [] CTC decoding is fast

---

Published as a conference paper at ICLR 2026
60
(LAS3)
0.5
0.471
(Autorggeessive)
00D (Vox2)
Mean Decoding Time (s)
Att (Teacher Forcing)
50
0.4
CTC
WER (%6
40
0.3
30
0.2
20
10
0.1
4.6
4,6
3.5
0.050
0
3.2
3,8
2/6
0.013
50
100
150
200
250
300
350
400
Decoding Strategy
Input Length (Frames)
Figure 1: CTC vs. attention-based decoding (greedy). Left: AVSR word error rate (WER) of USR on in- (LRS3) and outtof-distribution (VoxCeleb2, automatically transcribed) samples. CTC decoding notably more robust to domain shift and long sequences, while autoregressive attention based decoding performs best in-distribution. Right: Decoding speed on H200 GPU. CTC is 440 faster than autoregressive decoding. While teacher forcing could significantly speed up attention- based decoding, it typically relies ground-truti tokens unavailable during aseudo-labellinn
and robust to domain shift due to monotonic alignment and conditional independence assumptions, while attention- baaiod decoding has higher in-distribution quality but depends on slow AR generation. USR 2.0's core idea is CTC-driven teacher forcing: instead of decoding the teacher autoregressively, we feed its greedily decoded CTC outputs into the decoder to generate attention based PLs in single forward pass, removing the AR bottleneck. While this choice may appear counterintuitive since the outputs can lack global coherence- a key insight is that in a pseudo-labelling setting coherence is unnecessary: teacher and student operate under the same forced inputs, making knowledge transfer effective. Further, because these attention-bat aligned), the student can predict both in a single decoder forward pass, coupling the branches and improving robustness to OOD samples. Finally, as CTC- teacher forcing introduces train-test mismatch (CTC inputs during training vs. AR tokens at inference). we mitigate with simple mixed sampling strategy that intermittently reintroduces AR decoding.
PLs are derived from the CTC PLs (and are hence
USR 2.0 improves robustness under domain shift, with large gains on long utterances, noisy audio, and cross-dataset evaluations across LibriSpeech (Panayotov et al. (015), WildVSR (Djilali et al. 2024). and AVSpeech (Ephrat et al. 2018). is also markedly less sensitive to beam size than USR, maintaining strong performance even under greedy decoding. These robustness improvements further translate to better -i-iistribution performance, as unlabelled data often comes from OOD sources (Chung et al. 22188] Shi et al.) 2022a).. As result, USR 2.0 achieves ttate-of-the-att WER in various semi-supervised settings across ASR, VSR, and AVSR, with nearly 2x faster training than USR. Enabled by findings,
scale USR 2.0 to Huge model trained ~550 hours of unlabelled data, yielding WERs of 17.6% (VSR), 0.9% (ASR), and 0.8% (AVSR) on LRS3, along with state-of-the thet-arr results on LRS2 and WildVSR (Appendix C.I). all using single unified model.
RELATED WORK
CTC, attention, and joint training. Speech recognition typically relies CTC (Graves al. Baevski et al. 2020; Hsu et al. (202T) or attention-based encoder-decoder models (Chorowski Chan et al. 2016 Radford et al. 2023). CTC enables efficient training and inference via monotonic alignment and conditional independence, improving robustness in noisy or out-of- distribution settings (Kim et al., 2017), whereas attention-basst models capture richer dependencies through autoregressive decoding but may suffer from alignment errors and degrade under distribution shift (Watanabe et al. 2017). Joint CTC-attention training (Kim et al. 2017) supervises a shared encoder via separate branches, with inference performed by joint beam search and CTC rescoring (see Appendix B) While this strategy has been adopted in VSR and AVSR (Ma et al. 2022 2023
2006
al. Chan
Haliassos et al. 2022a 2024a), its reliance costly beam search limits its use in iterative pseudo- labelling. aon-auuoorgeessive transformers (Ghazvininejad et al.) 2019 Tian et al. 2020 Song et al.,
202T) avoid AR decoding via parallel token generation but typically reduce accuracy, whereas
2

---

Published as a conference paper at ICLR 2026
method applies CTC guidance to the attention-basse decoder only during eseudo-labelliigg where output sequence coherence is unnecessary -while retaining accurate AR beam search at inference.
Self-supervised audiovisual speech models.
Audiovisual self-supervised learning has shown promise for speech recognition by learning contextualised representations from large-scale unlabelled audiovisual data (Shi et al. 2022a Zhu et al. 2023 Lian et al. (023). Pre-training typically relies on masked prediction (Baevski et al. 2022; He al. 2022) or audiovisual correspondence (Chung & 2021). followed by fine-tuning with a CTC layer and/or attention-
Zisserman Morgado et al.
based decoder (Cappellazzo et al. 2024, Rouditchenko et al. (024). Although recent methods unify
modalities in single encoder during pre-training (Shi et al. 2022a Zhhu et al. (023), the sensitivity
of supervised line-tuning to aask-specific hyperparameters often complicates joint optimisation across modalities (Hsu & Shi 2022, (Shi et al. (022a). As result, fine-tuning is typically performed separately for ASR, VSR, and AVSR, increasing deployment costs (Haliassos et al.) 2022a 2024b)
and forgoing the potential benefits of shared representations across modalities (Akbari et al. 2021
Pseudo-labelling for speech recognition. Pseudo-labelling is effective strategy for semi- supervised speech recognition, where model-generated transcriptions for unlabelled data act as additional supervision (Sohn et al.] (020). Offline pseudo-labelling with frozen model is effi- cient (Afouras et al.) 2020 Ma et al.) 2023), but requires externl models and does not support iterative refinement. In contrast, self-training methods update pseudo-labels during training using the model itself, often with beam search (Park et al., 2020; Xu et al., 2020) Kahn et al.) (020). However, beam search is computationally expensive, especially at scale, prompting some works to adopt CTC-only supervision (Likhomanenko et al.) 2020; Higuchi et al. (02TT) at the cost of reduced sequence modelling capacity (Rouditchenko et al. 2023). USR (Haliassos et al. 2024a) shows that fine-tuning with unlabelled data using greedy CTC and attention-based pseudo-labelling mitigates the task-specific sensitivity observed in self-supervised approaches, enabling a single model for ASR, VSR, and AVSR. Building on this, revisit USR's decoupled pseudo-labellinn scheme to improve training efficiency, robustness to distribution shifts, and in-distribution performance.
BACKGROUND: USR
USR (Haliassos et 2024a) is semi-supervised mudent-teacher framework for training a single model to perform ASR, VSR, and AVSR. It uses shared encoder with two output heads-a a CTC layer and an attention- based ancoder-and leverages unlabelled data via iterative pseudo-labelling (see Figure 2 left). The student receives masked audio, video, and audiovisual inputs, and is trained to match the teacher's outputs on unmasked audiovisual inputs, while also learning from ground-truth labels where available. Key components of USR are outlined below (more details in Appendix B.2).
Teacher-student setup. Let Og and Or be the parameters of the student and teacher models. The student is optimised via gradient descent, and the teacher is updated as an exponential moving average of the student via Of t- 70, + (1 r)0s, where follows cosine schedule from 0.998 to 1.
Pseudo-labelling. Given an unlabelled audiovisual speech input represented feature sequence of length I., the teacher generates two types of pseudo-labels (PI.s) via greedy decoding. The CTC head produces frame-level PLs ot 5CCC for t E [1, L). and the decoder (attention branch) autoregressively produces token-level PLS /A for u E [1, UAR], where UAR is the decoder output length:
CTC Ut
arg max Perc a. 0r), Vr
g
arg max PAu PAu (ya Vo
1 zx; fr).
(1)
Loss unlabelled data.
The student is trained match the teacher's 's PLs for all masked modalities.
Let gre. denote the student's 's output from head w E {CTC, Att) for modality m E {A, V, AVJ. USR uses the cross-entropy loss (frame-wise or token-wise depending on the head w):
_ m CE
where we omit time and token indices and u for ease of notation. The total unlabelled loss is a weighted combination across heads and modalities, and is jointly optimised with the supervised loss.
3

---

Published as a conference paper at ICLR 2026
Pseudo-Labo Generation (Teacher)
USR CTC-Driven Mode
USR
Pseudo- Prediction (Student)
USR Drivan
USR 2.0. Mode
cTC
Oitione
Legend
Teachar
Stadert
Oupers
Dutpu
Discc Prodicion
Coats-
Predicior
Teacher Fording
Figure 2: attention-based pseudo-labels (PLs) from unmasked audiovisual inputs: CTC PLs are generated in parallel, while attention- based PLs require autoregressive decoding. The student predicts each target type independently, leading to decoupled supervision. USR 2.0 introduces two modes for tighter integration. In CTC-driven mode (centre), attention based PLs are generated by feeding collapsed CTC PLs into the decoder via teacher forcing, avoiding autoregression. The student decoder predicts both types of PLs. In AR mode (right), the teacher operates autoregressively as in USR, and the student's ' CTC branch predicts both CTC and attention based PLs. USR 2.0 alternates between modes at each iteration: CTC-driven mode improves efficiency and robustness, while AR mode mitigates exposure bias. Note: B refers to batch size, and typically, Ucrc, UAR < T. (not to scale in the figure).
in USR and USR 2.0 In USR (left), the teacher generates CTC and
Limitations. We argue that a key limitation of USR is its decoupled supervision: the student's CTC and decoder branches are trained independently using their respective PL.s. This design makes the attention decoder brittle OOD inputs (e.g., noisy audio or long utterances), because (1) greedy autoregressive (AR) decoding can fail and produce cascading errors (Kim et al. 2017: Fan et al.)
2018), and (2) errors are reinforced in the training loop, as noisy AR PLs supervise the student, which can then degrade the teacher via EMA updates. We validate this effect empirically in Section [
Equally problematic is the computational cost of generating attention- based PLs: because they are produced autoregressively, one forward pass is required per output token. Despite key-value caching, this makes pseudo-labelling slow, especially since decoding is performed every training step. This inefficiency becomes a major bottleneck for scaling to larger models and longer sequences, and it limits the practical utility of semi-supervised training on large unlabelled corpora.
4 PROPOSED METHOD
To overcome these limitations, we propose USR 2.0 for faster and more robust pseudo-labelling. The key idea is CTC-driven teacher forcing, where greedily decoded CTC PLS replace slow AR decoding to generate attention targets in single forward pass. This design both removes the AR bottleneck and transfers the robustness of CTC to the decoder via joint CTC-attention PL prediction. To address the resulting train-tes, discrepancy, we propose simple mixed sampling strategy, intermittently reintroducing AR decoding. Together, these components preserve the modelling flcxibility of attention while improving efficiency and robustness under distribution shift. An overview is in Figure
4

---

Published as a conference paper at ICLR 2026
4.1 CTC-DRIVEN TEACHER FORCING
We propose to leverage the teacher's CTC outputs to guide the generation of attention-based PLs. Given an unlabelled audiovisual input x,, first greedily decode the teacher's CTC head, then apply a merge-and-collapse operation by removing blank tokens and merging repeated symbols:
jt = arg max Pete | x;; 0r),
y g = collapsc(U1.1)
where collapse(-) denotes the standard CTC post-processing operation (Graves et al. 2006). This produces a sequence of of length Uere & L,, which we then feed as input to the teacher decoder:
Vu
arg max PAu C x;; 0r), u = u=,,...,Ucr
Vu.
replacing the autoregressive conditioning on past attention outputs with the fixed CTC token prefix Jeu. This removes the sequential dependency, allowing the entire sequence of attention- PLs to be generated in parallel with a single forward pass. As a result, the modelling strength of attention- based pseudo-labelling is preserved while avoiding the high cost and fragility of autoregression.
Aligned targets. Since the attention-based PLs are generated by conditioning the decoder directly on the CTC pseudo-labels, they inherit the same sequence length Ucrc- This shared alignment ensures that both pseudo-labed types correspond position-wist -wise, allowing the student decoder to predict them simultancously single forward pass under teacher forcing on the CTC PLs. This design lets the decoder inherit robustness from predicting CTC pseudo-labels while retaining flexibility from predicting attention pseudo-labelo
Global coherence. In CTC-driven teacher forcing, each decoder output S/ is conditioned on the fixed prefix of CTC tokens 8/2u rather than past decoder outputs / A natural concern is that the resulting sequence T may lack coherence (see Appendix C.4 for discussion). While this prevents its use for inference-time decoding, the student decoder learns effectively during self-training with token-wise cross-entropy because of matched conditioning: teacher and student are conditioned on the same CTC-derived sequence, which is itself coherent. The student is trained to predict the teacher decoder's 's most likely next token under this shared input. The student decoder therefore learns a stable mapping from coherent CTC prefix to the teacher's conditionally valid next-token prediction. Because inference is autoregressive and relies on this learned prefix-to-next-toke relation, global incoherence of the teacher-generated sequence does not hinder learning.
4.2 MIXED SAMPLING
While CTC-driven teacher forcing allows attention-based PLs to be generated efficiently, it introduces a mismatch: the decoder trained using inputs derived from the teacher's CTC predictions, whereas at inference time it autoregressively conditions on its past outputs. This discrepancy can be seen as a form of "exposure bias," which can hurt performance (Ranzato et al. 20155). addition to this input-side mismatch, CTC's conditionnllinddeenneenn assumptions may also occasionally yield imperfect CTC predictions on challenging, in-distribution segments, and these errors can propagate through the teacher to produce weaker attent pseudo-labed labels. To address these issues, introduce a simple yet effective mixed sampling strategy, similar to scheduled sampling (Bengio et al.) 2013) but for a different type of exposure bias. At each training step, we randomly sample mode: with probability 0.5 we use the CTC-driven mode, otherwise we adopt the standard AR mode""
In CTC-driven mode, the student CTC branch outputs jCTC.. for modality m, while the decoder outputs ÿ conditioned on the same CTC PLs jCTC used to condition the teacher. The losses are:
cCTCm = CTC(gCIC.. "egeerC,...r.)).). ,gTT), CAu.m = 0.5 CE(g AA + 0.5 CE(gAl,, "aaaaa....CCC grrc)..
(5)
The decoder is supervised with both the teacher's attention-base pseudo-labels ğAtt and the CTC targets GCTC to leverage the strengths of both PL types. The student's CTC branch is supervised only with J since شAAl may lack coherence (as explained) and could degrade CTC training.
'We also experimented with an adaptive sampling schedule (Appendix C.2 where the probability of AR pseudo-labelling increases over training. This strategy performed similarly the fixed 0.5 probability used in USR 2.0, so we adopt the simpler approach in the main method.
5

---

Published as a conference paper at ICLR 2026
120
AUINDERRT
120
50-4
Relative Decading Timee
100
100
USR
(8)
Scan Lengtte
WER (96)
Scan Lengito
40
2.0
60
60
30
40
40
20
100
200
300
400
500
600
100
200
300
400
500
600
10-
10-
Input Length (Frames)
Input Length (Frames)
Beam Size (Log Scale)
(a) Greedy decoding
(b) Beam size 30
(c) WER vs. Beam Size
Figure 3: Robustness long utterances. (a) Greedy decoding: USR 2.0 maintains robustness to longer imput lengths, significantly outperforming other models. (b) Beam search (beam size 30, joint CTC-attention decoding) improves USR robustness but still lags behind USR 2.0. (c) Increasing beam size reduces the WER gap between USR and USR 2.0, but at significant computational and memory cost. The size of the markers corresponds to the relative memory cost with batched beam search.
In AR mode, the attention PLs j are generated via standard AR decoding to alleviate train-test mismatch. The student decoder trained to predict yA while the CTC branch is jointly supervised with both je and iÂ both now represent coherent sequences. The corresponding losses are:
CCTC,m 0.5
,TTC + 0.5 CTC( """"" L CE ,y""), (6)
Note that only the CTC branch receives both types of pseudo-labels supervision: zeTe and yAu may differ in length, making it infeasible to supervise the decoder on both in single forward pass.
4.3 IMPLEMENTATION DETAILS
Our implementation follows USR, with the main difference being our proposed pseudo-labelling strategy. We summarise the setup below; full details, including model sizes and hyperparameters, are given Appendix
Architecture. We use a transformer encoder-decoder with modality-specific ResNet-18 back- bones (He al. 2016 Stafylakis & Tzimiropoulos (017)) for audio and video. The audio encoder subsamples to match the visual frame rate (25 Ips). Audio and visual features are projected via linear layers into shared transformer dimension, while audiovisual features are formed by concate- nation and applying separate projection (Haliassos et al.) (024a).. The resulting representations are processed by a transformer encoder with relative positional embeddings (Dai et al. 2019). We experiment with four model sizes (Base, Base+, Large, Huge); details are in Table
Loss weights. We follow joint attenitno training (imm al., 2017). combining the CTC loss (weight 0.1) with attention-based cross-entropy loss using label smoothing 0.1 (Szegedy et al. 2016)). Following USR, we use modelity-specific weights 0.3 (visual) and 0.7 (audio and audiovisual), and unlabelled-fo-labbelees loss ratios of 0.97 (visual) and 0.75 (audio and audiovisual).
Thresholding. We adopt confidence-based filtering as in USR, with a threshold of 0.8. For collapsed CTC PLs, sequence-level confidence is computed as the average log- probabibbiiy over token predictions.
Inference. Unless otherwise stated, inference uses joint CTC-attention decoding via ESPnet (Watan abe et al. 20188). with beam size 40 and CTC weight 0.1. We use a 1,000-token SentencePiece vocabulary (Kudo & Richardson 2018) trained the labelled data.
5 OUT-OF-DISTRRBUTION RESULTS
nut-of-distribution robustness is critical not only for deployment but also for semi-supervised learning, since unlabelled data often comes from domains outside the labelled distribution. Better

---

Published as a conference paper at ICLR 2026
Table 1: Comparison of WER ( (%) across different SNR levels (dB) for ASR and AVSR on the
LRS3 test set (beam size 30). None of the methods saw noise-augmented samples during training.
All Samples
Samples with
100 Frames
Method
10 dB
5 dB
0 dB
-5 dB
Avg
10 dB
dB
d
-5 dB
Avg
ASR
AV-HuBERT
8.4
18.9
54.2
94.6
44.0
6.5
16.2
49.3
97.2
42.3
BRAVEn
7.2
16.8
50.1
99.2
43.3
6,4
49.2
108.4
44.5
USR
14.3
104.4
43.3
4,9
12.6
48.1
111.7
44.3
USR 2.0
5.2
13.4
44.0
94.4
39.3
3.8
10.6
42.8
98.3
38.9
AVSR
AV-HuBERT
5.8
10.5
24.3
56.9
24.4
4.4
8.0
18.0
48.7
19.8
BRAVEn
7.2
14.0
32.2
54.6
27.0
6.5
12.7
30.9
55.5
26.4
USR
4.0
6.4
14.8
36.5
15.4
2.7
4.6
11.6
29.1
12.0
USR 2.0
3.7
5.6
14.0
33.1
14.1
2.6
4.3
10.4
26.0
10.8
robustness yields higher-quality pseudo-labels and thus more reliable training. To assess USR 2.0's robustness, we evaluate the Base model under long utterances, noisy audio, and unseen datasets, comparing against USR and strong self-superrvsed baselines AV-HuBERT (Shi al.) 2022a) and BRAVEn (Haliassos et al. 2024b). All experiments are conducted in the low-resource setting (Shi al. 2022a). where the labelled is the 30-hour "trainval" partition of LRS3 (Afouras al. 2018 and the remaining LRS3 samples are treated as unlabelled data.
5.1 ROBUSTNESS TO LONG UTTERANCES
We first evaluate robustness to long sequences using ~~2000 automatically transcribed samples from VoxCeleh2 (Chung al. 2018), treating Whisper (Radford et al. 2023). trained on far more data, as an oracle for WER. The test set is bucketed into ~~000-sample groups by input length (50-600 frames, in 50-frame increments). The longest labelled training utterance is 155 frames, while unlabelled samples reach 600, inputs beyond 155 represent distributional shift relative to the labelled data.
Greedy decoding. Figure 3al shows WER versus input length under greedy (attention-based) decoding, which is important for pseudo-labelling All methods perform similarly for inputs under 150 frames, but WER rises sharply beyond that for BRAVEn, AV-HuBERT, and especially USR, likely due to compounded autoregressive errors. While longer sequences should theoretically help by providing more context, distributional mismatch dominates. In contrast, USR 2.0 remains stable due to its integrated CTC and attention supervision, which mitigates autoregressive drift.
Beam search. We next evaluate joint CTC- decoding with beam size 30. For AV-HuBERT, which lacks CTC, we use attention-only decoding. As shown in Figure 3b) USR benefits strongly from beam search, as CTC scoring helps correct decoder errors. However, performance still drops beyond 400 frames, suggesting that inference-tng time correction does not fully compensate for poorly trained decoder. In contrast, USR 2.0 remains strong, indicating greater underlying robustness.
Beam size. Finally, we vary the beam size and plot WER for USR and USR 2.0 across samples (Figure 3ch. At small beams, USR 2.0 clearly outperforms USR, highlighting its robustness under fast decoding. As beam size increases, the gap narrows, with very large beams compensating for USR's weaker decoder at the cost of significantly higher latency and memory. Overall, USR 2.0 performs far better with small beams, making it well suited for pseudo-labelling and low-latency applications.
5.2 ROBUSTNESS TO NOISE
In Table we evaluate generalisation to noise on LRS3 with additive babble noise from NOI- SEX VVarga & Steeneken 1993) SNRs from 10 dB to 5 dB. As most LRS3 test samples are short, we also report results on those longer than 100 frames to better reflect werl-world inputs. Importantly, none of the models were trained with noise augmentation, making this zero-shot evaluation.
7

---

Published as a conference paper at ICLR 2026
Table In-distribution comparisons with self-supervised methods. LRS3 WER (%) results for low- (left) and high-resource (right) labelled data settings. Best results in bold. We specify model size and pre-training dataset for each setting bold. "Shared params" denotes whether unified model is used. 'Trained on LRS2+LRS3 (labelled) and English-only VoxCeleb2+AVSpeed (unlabelled).
Shared params
Shared params
Method
V
A
AV
Method
A
AV
Base, LRS3
Base+, LRS3+Vox. Vox2
AV-HuBERT
51.8
4.9
4.7
AV-HuBERT
X
34.8
2.0
1.8
RAVEn
47.0
VATLM
34.2
1.7
AV-data2vec
45.2
RAVEn
33.1
1.9
BRAVEn
43.4
4.0
4.0
AV-data2vec
1.7
1,4
36.0
3.2
3.0
Lip2Vec
USR 2.0
36.2
3.0
2.9
BRAVEn
X
28.8
1.4
USR
26.5
1.6
1.3
Base+, LRS3+Vox2
USR 2.0
24.8
1.4
1.2
AV-HuBERT
46.1
4.6
4.0
RAVEn
40.2
3.8
Large, LRS3+Vox2
AV-data2vec
37.8
AV-HuBERT
28.6
1.3
1.4
BRAVEn
35.1
VATLM
28.4
1.2
USR
28.4
2.6
2.5
RAVEn
USR 2.0
26.4
2.5
2.4
AV-data2vec
X
28.5
1.3
1.3
Lep2Vec
X
26.0
Large, LRS3+Vox2
BRAVEn
X
26.6
1.2
AV-HuBERT
32.5
3.3
u-HuBERT
29.1
1,5
1.3
RAVEn
32.5
USR
22.3
1.2
1.1
AV-data2vec
30.8
USR 2.0
21.5
1.3
1.0
BRAVEn
30.8
USR
26.9
2.4
2.4
Huge"
USR 2.0
V
23.7
2.3
2.2
USR 2.0
17.6
0.9
0.8
For ASR, USR 2.0 clearly outperforms all baselines moderate noise levels (10 dB), showing strong robustness. At -5 dB SNR, where the audio distribution shifts substantially. its performance is comparable AV-HuBERT. On longer samples, the relative gap between USR 2.0 and USR slightly widens, suggesting that USR 2.0's decoder may be more resilient to accumulated errors over time.
Consistent with prior works (Ma al. 202T: Haliassos al. 2024b a). AVSR outperforms ASR methods (USR
under noise by leveraging visual information. Across all SNRs, the and USR 2.0) significantly outperform the self-supervised baselines (BRAVEn and AV-HoBERT). In- terestingly, the improvement of USR 2.0 over USR remains stable across sequence lengths, suggesting that AVSR gains are less dependent on input duration than ASR.
5.3 ROBUSTNESS TO OOD DATASETS
Table 3: Performance on OOD
We evaluate robustness to OOD datasets with substan- tial gaps between training and test distributions, includ- ing shifts in vocabulary, speaker accents, background noise, and recording conditions. We use LibriSpeech test- clean (Panayotov et al. 2015) for ASR, WildVSR (Dji- lali et al. (024) for VSR, and 1,000 manually filtered AVSpeech (Ephrat et al., 2018) samples for AVSR, tran- scribed using Whis Whisper (Radford al. 2023). As shown in Table USR 2.0 outperforms all baselines by a wide mar-
datasets: LibriSpeech (LibriS), Wild- VSR, and AVSpeech (AVS). We report WER (%) under greedy decoding.
Method
LibriS
WildVSR
AV-HuBERT
29.1
82.4
26.0
BRAVEn
38.4
81.2
44.6
80.0
34.7
USR
15.4
73.7
25.0
gin under greedy decoding across all tasks, highlighting the robustness of its improved decoder to diverse, real-world shifts.
IN-DISTRIBUTION RESULTS
We now evaluate performance the LRS3 benchmark, which serves as in-distribution setting where the labelled and test data come from the same domain. However, because the unlabelled data often comes from other sources (e.g., VoxCeleh2), pseudo-labelling robustness remains critical.
8

---

Published as a conference paper at ICLR 2026
Table 4: AVSR WER (%) for different pseudo- label types for each branch in CTC-driven and AR modes. Default settings are in gray.
1.6
Rel. Time per training Step
0OD (AVSpoech)
1.5
26.2
1.4
WER (%)
24.2
24.3
CTC Head Preds
Decoder Preds
1.3
ID
0OD
CTC PL.
Att PL.
CTC PL
Att PL.
1.2
CTC-Driven Mode
X
3.2
24.2
1.1
x
X
3.3
35.1
12
2,9
2,8
28
2,9
1.0
/
3.2
24.3
0.0
0.2
0.4
0.6
0.8
1.0
x
3.6
25.5
AR Mode Sampling Probability
AR Mode
X
2.9
40.1
Figure 4: AVSR WER (%) AR mode
X
X
3.0
45.1
pling probability on in-distribution (LRS3) and out-of-distribution (AVSpeech) samples.
/
X
2.8
52.3
Low- and high-resource settings. Table 2 presents WERs for low- (30h) and high-resource (433h) settings (Shi et al.) 2022a), comparing against self-supervised methods, including RAVEn (Haliasoss et al. 2022a). AV-data2vec (Lian et al. 2023), VATLM (Zhu et al. 2023). Lip2 Djiilaii et al. 2023), and u- H-HUBERT (Hsu Shi] 2022). :. USR 2.0 matches or outperforms the state of the art, even methods that train separate models per task. Gains over USR more pronounced with VoxCeleb2 pre-training, particularly for VSR, which relies more heavily on pseudo-labels. These results support hypothesis that improving pseudo-labelling benefits even in in-distribution performance.
Training efficiency. Figure plots WER for VSR against wall-clock training time for USR and USR 2.0 across different model scales and pre-training datasets. observe approximately 2x faster training for each set- ting, driven by two key factors: (i) faster training steps due to CTC-driven teacher forcing, and (ii) faster convergence, requiring fewer epochs (50 75; see Appendix C.5 These gains make USR 2.0 substantially more efficient for semi-supervised training at scale.
38
36
Dase (LAS3)
WER (96)
32
30
28
Swote RS1=WWe7)
26
arge (URS3-WW2
24
Large (URS3
Scaling. Building on these results, we scale USR 2.0 to a Huge model trained on the combined LRS2 (Son Chung 2017) and LRS3 labelled sets, along with large unlabelled corpus from English-only VoxCeleb2 and AVSpeech. The model achieves WERs of 17.6%, 0.9%. and 0.8% on VSR, ASR, and AVSR, respectively, demon- strating USR 2.0's scalability. Comparisons with additional methods are in Appendix C.I
20
40
80
100
120
140
Training Time (hours)
Figure 5: VSR WER training time for the low-resource setting.
ABLATIONS
We ablate the choice of pseudo-label (PL. targets for each branch under the two modes: CTC-driven and AR. To isolate their effects, disable AR mode when evaluating CTC-driven mode, and vice versa. All experiments use the low-resource setup with the Base model (see Section 5p. We report AVSR WER on both in-distribution (ID; LRS3 test) and outto--distribution (OOD; AVSpeech) data. More ablations are given in Appendix C.2
CTC-driven mode (Table 4). By default, the CTC head predicts CTC PLs, while the decoder predicts both CTC and attention-baan based PLs. Removing CTC supervision from the decoder harms OOD performance, confirming its value for robustness. Dropping iatention-basse targets reduces ID performance, underscoring their importance for sequence modelling. Thus, both are erucial. Interest- ingly, supervising the CTC head with attention- based PLs does not noticeably degrade performance, suggesting that AR inconsistencies may be offset by higher-quality conditional predictions.

---

Published as a conference paper at ICLR 2026
Finally, conditioning the decoder on CTC pseudo-labels in this mode avoids conditioning solely on AR ARggenerated pseudo-labels, which can be error-prone under domain shift due to cascaded autoregressive mistakes. This effect is reflected in the large OOD gap between rows 2 of "CTC- Driven Mode" (3).1%) and "AR Mode" (45.1%)
AR mode 4). Here, the default assigns only attention-based PLs to the decoder (due the target length mismatch) and both PL types to the CTC head. Removing inrention-basee targets from the CTC head reduces performance. Using only attention- based PLs in both branches degrades OOD performance, reaffirming the role of CTC PI.s for robustness. Overall, ID and OOD trends are largely uncorrelated, and AR performs significantly worse than CTC-driven mode on OOD data.
Effect of mixed sampling probability (Figure 4. We evaluate the effect of varying the probability of sampling the AR mode during training. As this probability increases, ID performance improves, likely due to reduced exposure bias. OOD performance remains relatively stable across most of the range but degrades sharply when approaching AR-only training. Training time per iteration also increases steadily with higher AR probability due to the cost of AR decoding. The sampling probability therefore controls the balance between ID accuracy, OOD robustness, and training efficiency, and may be tuned match application needs, with our default of 0.5 offering strong overall balance.
8 CONCLUSION
We presented USR 2.0, which addresses two key limitations of USR that prevent scaling to larger models and diverse unlabelled data: the computational cost of autoregressive pseulo-labelling and the decoupled design that reduces robustness to out-of of-distribution inputs. By introducing CTC-driven teacher forcing and mixed sampling, USR 2.0 combines the efficiency and robustness of CTC with the modelling flexibility of attention-based decoding. This design improves generalisation to long utterances, noisy conditions, and unseen datasets, while reducing training time by nearly half. These robustness gains also translate to improved in-distribution performance, since unlabelled data often comes from -ittt--distriuttion sources. Scaling both model and corpus, USR 2.0 achieves state-of- the-art performance across ASR, VSR, and AVSR with a single model. Beyond USR, our insights can be applied to other speech recognition tasks, including audio-only, streaming, and multilingual ASR, which active research areas where large unlabelled datasets common. More broadly, the CTC-driven teacher-forcing paradigm could be applied to other the input and output follow the same temporal order but lack explicit frame-level alignment, such as handwriting recognition, music transcription, or DNA/protein sequencing. We hope our insights will prove useful for future work on scalable requence-to-sequenee self-training beyond speech.
settings where
ACKNOWLEDGEMENTS
Data collection, processing, and experiments were conducted at Imperial College London.
REFERENCES
Ossama Abdel-Hamid, Abdel-rahman Mohamed, Hui Jiang, Li Deng, Gerald Penn, and Dong Yu. Convolutional neural networks for speech recognition. IEEE/ACM Transactions on audio, speech, and language processing, 22(10):1533 1545, 2014.
Triantafyllos Afouras, Joon Son Chung, and Andrew Zisserman. Lis3-ted: large-scale dataset for visual speech recognition. arXiv preprint arXiv: 180,00800000490 2018.
Triantafyllos Afouras, Joon Son Chung, and Andrew Zisserman. Asr is all you need: Cross-modal distillation for lip reading. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 2143-2147. IEEE, 2020.
Hassan Akbari, Liangzhe Yuan, Rui Qian, Wei-Hong Chuang. Shih-Fu Chang. Yin Cui, and Boqing Gong. Vatt: Transformers for multimodal self-supervised learning from raw video, audio and text. Advances in Neural Information Processing Systems, 14:2206-222211 2021.
10

---

Published as a conference paper at ICLR 2026
Yannis M Assael, Brendan Shillingford, Shimon Whiteson, and Nando De Freitas. Lipnet: End-to-end sentence-leve lipreading. arXiv preprint arXiv:1611.01999 2016.
Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in neural information processing systems, 33: 11:11249911222 124600, 2020.
Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, and Michael Auli. Data2vec: general framework for supe--upe learning in speech, vision and language. In International Conference on Machine Learning, pp. 1298-1312. PMLR, 2022.
Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. Scheduled sampling for sequence prediction with recurrent neural networks. Advances in neural information processing systems, 28, 2015.
Umberto Cappellazzo, Minsu Kim, Honglic Chen, Pingchuan Ma. Stavros Petridis, Daniele Falavigna, Alessio Brutti, and Maja Pantic. Large language models are strong audio-visual speech recognition learners. arXiv preprint arXiv:2409. 12319, 2024.
William Chan, Navdeep Jaitly, Quoc Le, and Oriol Vinyals. Listen, attend and spell: A neural network for large vocabulary conversational speech recognition. In 2016 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp. 4960 4964. IEEE, 2016.
Oscar Chang. Hank Liao, Dmitriy Serdyuk, Ankit Shahy, and Olivier Siohan. Conformer all you need for visual speech recognition. In ICASSP 2024-2024 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 10136-10140, IEEE, 2024.
Chung-Cheng Chiu, Tara N Sainath, Yonghui Wu, Rohit Prabhavalkar, Patrick Nguyen, Zhifeng Chen, Anjuli Kannan, Ron Weiss, Kanishka Rao, Ekaterina Gonina, et al. State-of-the speech recognition with acoustics, speech and signal processing (ICASSP), pp. 4774 4778. IEEE, 2018.
models. In 2018 IEEE international conference
Jan K Chorowski, Dzmitry Bahdanau, Dmitriy Serdyuk, Kyunghyun Cho, and Yoshua Bengio. Attention-baser models for speech recognition. Advances in neural information processing systems, 28, 2015.
Joon Son Chung and Andrew Zisserman. Out time: automated lip sync in the wild. In Computer Vision-ACCV 2016 Workshops: ACCV 2016 International Workshops, Taipei, Taiwan, November 20-24, 2016, Revised Selected Papers, Part II 13, pp. 251-263. Springer, 2017.
Joon Son Chung, Arsha Nagrani, and Andrew Zisserman. Voxceleb2: Deep speaker recognition. arXiv preprint 1806.05622. 2018
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V Le, and Ruslan Salakhutdi- nov. Transformerxll: Attentive language models beyond a fixed-length context. arXiv preprint arXiv:1901.02866.. 2019.
Yasser Abdelaziz Dahou Djilali, Sanath Narayan, Haithem Boussaid, Ebtessam Almazrouei, and Merouane Debbah. Lip2vec: Efficient and robust visual speech recognition via latent-to seent-to-latenn visual to audio representation mapping. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 77790-13801, 2023.
Yasser Abdelaziz Dahou Djilali, Sanath Narayan, Eustache LeBihan, Haithem Boussaid, Ebtesam Almazrouei, and Merouane Debbah. Do models generalize beyond Irs3? In Proceedings of the JEFF/CVF Winter Conference on Applications of Computer Vision, pp. 6635-6644, 2024.
Ariel Ephrat, Inbar Mosseri, Oran Lang. Tali Dekel, Kevin Wilson, Avinatan Hassidim, William T Freeman, and Michael Rubinstein. Looking to listen at the cocktail party: A speaker-indepennent audio-visual model for speech separation. arXiv preprint arXiv: I 1804 03619, 2018.
Angela Fan, Mike Lewis, and Yann Dauphin. Hierarchical neural story generation. arXiv preprint arXiv: 1805. 04833, 2018.
11

---

Published as a conference paper at ICLR 2026
Marjan Ghazvininejad, Omer Levy, Yinhan Liu, and Luke Zettlemoyer. Mask-predict: Parallel decoding of conditional masked language models. arXiv preprint arXiv:1904.09322, 2019.
Rohit Girdhar, Mannat Singh, Nikhila Ravi, Laurens van der Maaten, Armand Joulin, and Ishan Misra. Omnivore: A single model for many visual modalities, In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition. pp. 16102-16112, 2022.
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in hour. arXiv preprint arXiv: (alp::176.0677.. 2017.
Alex Graves, Santiago Fernandez. Faustino Gomez, and Jürgen Schmidhuber. Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In Proceedings of the 23rd international conference Machine learning, pp. 369-376, 2006.
Alexandros Haliassos, Konstantinos Vougioukas, Stavros Petridis, and Maja Pantic. Lips don't 't lie: A generalisable and robust approach to face forgery detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 5039-5049, 2021.
Alexandros Haliassos, Pingchuan Ma, Rodrigo Mira, Stavros Petridis, and Maja Pantie. Jointly learn- ing visual and auditory speech representations from raw data, arXiv preprint arXiv:2212.06246, 2022a.
Alexandros Haliassos, Rodrigo Mira, Stavros Petridis, and Maja Pantic. Leveraging real talking faces via sel--supeo for robust forgery detection. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 14950-14962, 2022b.
Alexandros Haliassos, Rodrigo Mira, Honglic Chen, Zoe Landgraf, Stavros Petridis, and Maja Pantic. Unified speech recognition: single model for auditory, visual, and audiovisual inputs. arXiv preprint arXiv:2411.0223 2024a.
Alexandros Haliassos, Andreas Zinonos, Rodrigo Mira, Stavros Petridis, and Maja Pantic. Braven: Improving self-supervised pre-training for visual and auditory speech recognition. In ICASSP 2024-2024 JEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 11431 1-31-11435. IEEE, 2024b.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 770-778, 2016.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pallern recognition, pp. 16000-16009, 2022.
Yosuke Higuchi, Niko Moritz, Jonathan Le Roux, and Takaaki Hori. Momentum pseudo-labeling for semi-supervised speech recognition. arXiv preprint XXiv:21006089222 2021.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text 2019.
degeneration. arXiv preprint
Wei-Ning Hsu and Bowen Shi. u-hubert: Unified mixed-modal speech pretraining and zero-shot transfer to unlabeled modality. Advances in Neural Information Processing Systems, 35:21157- 21170, 2022.
Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM Transactions Audio, Speech, and Language Processing, 29:3451-3460, 2021.
Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger. Deep networks with stochastic depth. In Computer Netherlands, October 11-4, 2016, Proceedings, Part IV 14, pp. 646-661. Springer, 2016.
2016: 14th European Conference, Amsterdam, The
12

---

Published as a conference paper at ICLR 2026
Jacob Kahn, Ann Lee, and Awni Hannun. Self-training for end-to-end speech recognition. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 7084-70888 IEEE, 2020.
Suyoun Kim, Takaaki Hori, and Shinji Watanabe. Joint ctc-attention based end-to-end speech recognition using multi-task learning. In 2017 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp. 4835-4839, IEEE, 2017.
Taku Kudo and John Richardson. Sentencepicce: A simple and language independent subword tokenizer and detokenizer for neural text processing. arXiv preprint arXiv:1808.06226, 2018. 2018.
Jiachen Lian, Alexei Baevski, Wei-Ning Hsu, and Michael Auli. Av-data2vec: Self-supervised
learning audio-visual speech representations with contextualized target representations. arXiv preprint arXiv:2302.064/4 2023.
Tatiana Likhomanenko, Qiantong Xu, Jacob Kahn, Gabriel Synnaeve, and Ronan Collobert. slimipl: segguage-mddel-free iterative pseudo-labeling. arXiv preprint arXiv:2010. 11544, 2020.
Xubo Liu, Egor Lakomkin, Konstantinos Vougioukas, Pingchuan Ma, Honglie Chen, Ruiming Xie, Morrie Doulaty, Niko Moritz, Jachym Kolar, Stavros Petridis, et al. Synthvsr: Scaling up visual speech recognition with synthetic supervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18806-18815, 2023.
Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv: 160.:366,..,383, 2016.
Hya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv: 1,0100100, 2017.
Pingchuan Ma. Stavros Petridis, and Maja Pantic. End-to-end audio-visual speech recognition with conformers. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 7613-7617. IEEE, 2021.
Pingchuan Ma, Stavros Petridis, and Maja Pantic. Visual speech recognition for multiple languages in the wild. Nature Machine Intelligence, 4(1 ((11):930-93)) 2022.
Pingchuan Ma, Alexandros Haliassos, Adriana Fernandez-Loper Honglic Chen, Stavros Petridis, and Maja Pantic. Auto-avsr: Audio-visual speech recognition with automatic labels. In ICASSP 2023-202 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2023.
Takaki Makino, Hank Liao, Yannis Assael, Brendan Shillingford, Basilio Garcia, Otavio Braga, and Olivier Siohan. Recurrent neural network transducer for audio-visual speech recognition. In 2019 IEEE automatic speech recognition and understanding workshop (ASRU), pp. 905-912. IEFE, 2019.
Brais Martinez, Pingchuan Ma, Stavros Petridis, and Maja Pantic. Lipreading using temporal convolutional networks. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6319-6322 63233... IEEE, 2020.
Pedro Morgado, Nuno Vasconcelos, and Ishan Misra. Audio-visual instance discrimination with cross-modal agreement. In Proceedings of the IEEE/CVF conference computer vision and paltern recognition, pp. 13475-12486, 2021.
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: an asr corpus based on public domain audio books. In 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp. 5206-5210. IEEE, 2015.
Daniel S Park, Yu Zhang, Ye Jia, Wei Han, Chung Cheng Chiu, Bo Li. Yonghui Wu, and Quoc V Le. Improved noisy student training for automatic speech recognition. arXiv preprint ar arXiv:2005.09623 096999 2020.
13

---

Published as a conference paper at ICLR 2026
Stavros Petridis, Themos Stafylakis, Pingehuan Ma, Feipeng Cai, Georgios Tzimiropoulos, and Maja Pantic. End-to-end audiovisual speech recognition. In 2018 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp. 6548 6552. IEEE, 2018.
KR Prajwal, Triantafyllos Afouras, and Andrew Zisserman. Sub-wort level lip reading with visual attention. Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition. pp. 5162-5172, 2022.
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. In International Conference Machine Learning, pp. 18492-285188 PMLR, 2023.
Marc' Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. Sequence level training with recurrent neural networks. arXiv preprint arXiv:1511.0673 2015.
Andrew Rouditchenko, Ronan Collobert, and Tatiana Likhomanenko. Av-cpl: Continuous pseudo- labeling for audio-visual speech recognition. arXiv preprint arXiv:2309. 17395, 2023.
Andrew Rouditchenko, Yuan Gong, Samuel Thomas, Leonid Karlinsky, Hilde Kuehne, Rogerio Feris, and James Glass. Whisper-flamingo: Integrating visual features into whisper for audio-visual speech recognition and translation. arXiv preprint arXiv:2406. 100202 2024.
Dmitriy Serdyuk, Otavio Braga, and Olivier Siohan. Transformer-based video front-ends for audio- visual speech recognition for single and multi-person video. arXiv preprint arXiv:2201.1044 10439, 2022.
Bowen Shi, Wei-Ning Hsu, Kushal Lakhotia, and Abdelrahman Mohamed. Learning audio-visual speech representation by masked multimodal cluster prediction. arXiv preprint arXiv:2201 022201.02/88 184, 2022a.
Bowen Shi, Wei-Ning Hsu, and Abdelrahman Mohamed. Robust self-supervised audio-visual speech recognition. arXiv preprint arXiv:2201.0/76. 2022b.
Brendan Shillingford, Yannis Assael, Matthew W Hoffman, Thomas Paine, Cían Hughes, Utsav Prabhu, Hank Liao, Hasim Sak, Kanishka Rao, Lorrayne Bennett, et al. Large-scale visual speech recognition. arXiv preprint arXiv:1807.05162. 2018.
Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Do- gus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying semi- supervise learning with consistency and confidence. Advances in neural information processing systems, 33:596-608, 2020.
Joon Son Chung, Andrew Senior, Oriol Vinyals, and Andrew Zisserman. Lip reading sentences in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6447-6456, 2017.
Xingchen Song, Zhiyong Wu, Yiheng Huang, Chao Weng, Dan Su, and Helen Meng. Non- autoregressive transformer asr with ete-enhanced decoder input. In ICASSP 2021 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1894-5898. IEEE, 2021.
Themos Stafylakis and Georgios Tzimiropoulos. Combining residual networks with Istms for lipreading, arXiv preprint / 2017.
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2818-2826, 2016.
Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averagel consistency targets improve semi-supervised deep learning results. Advances in neural information processing systems, 30, 2017.
14

---

Published as a conference paper at ICLR 2026
Zhengkun Tian, Jiangyan Yi, Jianhua Tao, Ye Bai, Shuai Zhang, and Zhengqi Wen. Spike- triggered non- con-aetoreer transformer for end-to-end speech recognition. arXiv preprint arXiv:2005.07903, 2020.
Andrew Varga and Herman JM Steeneken. Assessment for automatic speech recognition: li noisex- 92: A database and an experiment to study the effect of additive noise on speech recognition systems. Speech communication, 12(3):247-251, 1993.
Shinji Watanabe, Takaaki Hori, Suyoun Kim, John Hershey, and Tomoki Hayashi. Hybrid ctc/attention architecture for end-to-end speech recognition. IEEE Journal of Selected Topics in Signal Processing, 1109::240-1255, 2017.
Shinji Watanabe, Takaaki Hori, Shigeki Karita, Tomoki Hayashi, Jiro Nishitoba, Yuya Unno, Nelson Enrique Yalta Soplin, Jahn Heymann, Matthew Wiesner, Nanxin Chen, Adithya Renduchintala, and Tsubasa Ochiai. ESPnet: End-to-end speech processing toolkit. In Proceedings of the 19th Annual Conference of International Speech Communication Association (INTERSPEECH), pp. 2207-2211, 2018.
Ronald J Williams and David Zipser. A learning algorithm for continually running fully recurrent neural networks. Neural computation, 1(2):270-280, 1989.
Qiantong Xu, Tatiana Likhomanenko, Jacob Kahn, Awni Hannun, Gabriel Synnaeve, and Ronan Collobert. Iterative pseudo-labeling for speech recognition. arXiv preprint arXiv:2005.0922. 2020.
Qiushi Zhu, Long Zhou, Ziqiang Zhang, Shujie Liu, Binxing Jiao, Jie Zhang, Lirong Dai, Daxin Jiang, Jinyu Li, and Furu Wei. Vatlm: Visual-audio-text pre-training with unified masked prediction for speech representation learning. IEEE Transactions on Multimedia, 2023.
15

---

Published as a conference paper at ICLR 2026
A EXPERIMENTAL SETUP
A.1 DATASETS
LRS3. The LRS3 dataset (Afouras et al., 2018) consists of approximately 430 hours of transcribed TED Talk videos, featuring thousands of speakers and an open vocabulary. Videos are automatically segmented into sentence-level utterances, with audio and visual streams synchronised via facial tracking. The dataset exhibits rich variability in lighting, speaking styles, and head poses. The test set, used for evaluation only, contains around hour of speech by speakers held out from training. LRS3 is released under the Creative Commons BY-NC-ND 4.0 license.
LRS2. Sourced from British television broadcasts, LRS2 (Son Chung et al.) 2017) provides 223 hours of audiovisual content with sentence-level transcriptions. While similar in structure to LRS3, the video content LRS2 more controlled, with generally better lighting and resolution. The test set contains roughly 30 minutes of clips spoken by unseen speakers and is frequently used for benchmarking visual and audiovisual speech models. is available for academic, non-commercial research use.
WildVSR. WildVSR (Djilali et al. 2024) is challenging VSR benchmark collected from open- domain video content. Designed to test generalisation, it includes more extreme conditions than LRS3 or LRS2, such as diverse ethnicities, head motions, and video resolutions. It comprises around 5 hours of test material without overlap with LRS3 and is used exclusively for evaluation. WildVSR licensed under CC BY-NC-ND 4.0.
AVSpeech. AVSpeech (Ephrat et al. 2018) contains over 4,700 hours of face-aligned speech
segments harvested from over 150,000 YouTTue videos. Its scale and speaker diversity make
valuable for self- or semi-supervised learning. We follow prior work (Ma et al. 2023) in filtering dataset to retain 1,323 hours only English-speaking videos. AVSpeech is licensed under
Creative Commons Attribution 4.0 International License (CC BY 4.0).
VoxCeleb2. VoxCeleb2 (Chung et al., 2018) comprises over 2,400 hours of unconstrained, in-the- wild recordings from 6,000+ speakers. The data spans multiple languages and noisy environments such as interviews and live performances. We use an English-only subset curated in prior work (Shi et al. 2022a), which includes 1,326 hours of utterances. VoxCeleb2 is distributed under the CC BY-NC-ND 4.0 license.
We include the test-clean set of LibriSpeech (Panayotov et al. 2015) as additional audio-only evaluation benchmark. It consists of 5.4 hours of read speech from audiobook recordings, featuring high-quality audio with minimal background noise. LibriSpeech is released under the CC BY 4.0 license.
LibriSpeech.
The lists of sampled videos used for out-of-distribution evaluation AVSpeech and VoxCeleh2 included the README, md of the source code provided in the supplementary material.
A.2 EXTERNAL CODEBASES
For the robustness experiments Section 5
we directly ran official codebases of prior methods to ensure fair and consistent comparisons. Specifically, we used:
AV-HuBERT (Shi et al.
2022a): https://ggttbbocc///ceeeeresrrrr
hubert
BRRAVEn HHlliasoss et al.
2024b): httpps//ggthubb.cmm/ataalaaaas/p
USR (Haliassos al. 2024a):
ht htt.p://ggthbb..ccom/aaaaaaassss
Where available, we used the officially released pre-trained models. In cases where no suitable model was provided (i.e., audiovisual BRAVEn, audio-only AV-HuBERT, and audiovisual AV-HuBERT without noise augmentations), we trained the models ourselves using the official hyperparameters and training protocols.
16

---

Published as a conference paper at ICLR 2026
Table 5: Model hyperparameters. Parameter count includes encoder, decoder, and feature extractors.
Base
Base+
Large
Huge
Total parameters (M)
86
171
503
953
Transformer blocks (enc dec)
12 / 6
12
24/9
36 06/9
Attention dim MIP size
512
768
1024
1280
2048
3072
4096
5120
Heads
8
16
Table 6: Training hyperparameters for different pre-training datasets and model sizes. "All" refers to the combination LRS2, LRS3, VoxCeleb2, and AVSpeech. Additional hyperparameters are provided in Appendix A.5
Hyperparameter
LRS3 (Base)
LRS3+VoxCelle
All (Huge)
Learning rate
3e-3
2e-3
7e-4
Weight decay
0,04
0.04
0.04
(31,82)
(0.9, 0.98)
(0.9, 0.98)
(0.9, 0.98)
Drop path rate Frames per GPU (lab./unlab.)
0.1
0.1 (Base), 0.2 (Large)
0.3
600 4400
300 / 3200
Frames per GPU (high-resource)
1200 / 3200
700 / 2400
For all other baselines and state-of-the-art comee-of-tee-tttcommaariooos we report numbers as provided in the original publications.
A.3 PRE-PROCESSING
Video frames are first stabilised to reduce jitter, then cropped to a 96 x 96 region centred on the speaker's s mouth, and finally converted to grayscale. This pipeline follows prior work (Shi al.
2022a by Haliassos et al. 2022a 2024b
No pre-processing is applied to the raw audio.
A.4 MODEL VARIANTS
We employ four model sizes: Base, Base+, Large, and Huge. Each variant shares the same feature extractors, with the encoder and decoder size being the primary differentiating factors. Configurations are summarised Table Note that Base+ corresponds the Base model used related works (Shi
et al. 2022a
Lian et al. 2023
Zhu et al. 2023).
A.5 TRAINING HYPERPARAMETERS
Optimiser and scheduling. Models are trained 50 epochs using AdamW (Loshchilov & Hutter 2017), with 15-epoch linear warmup (Goyal et al. 2017) followed by cosine decay (Loshchilov & Hutter 2016). Drop path regularisation (Huang et al. 2016 and gradient clipping (threshold 3.0) are applied for stability.
Augmentations. Video augmentations include random cropping (88 88) and horizontal flipping with probability 0.5, applied consistently across frames, Both modalities also use temporal zero- masking for the student model enforce context understanding: video inputs are masked with max duration 0.4s, audio with 0.6s, following Ma et al. (2022): Haliassos et al. 2022a) Teacher inputs are unmasked to preserve PL quality.
Training resources. We fix seed to 42. Training takes ~ day for Base (8 H200 GPUs), ~22 days for Base+ (32 GPUs), and ~3 days for Large (32 GPUs). Base LRS3. Base+ and Large LRS3+ Cexeeeeb, and Huge uses LRS2+LRS3+ VoxCeleb2+AVSpecch over <4 days on 64 GPUs. Frame counts per GPU uned for utilisation.
17

---

Published as a conference paper at ICLR 2026
Initialisation. All models initialised from the same self-supervised checkpoints USR. For Huge, we pre-train using the Large USR hyperparameters from Haliassos et al. (2024a) before semi-supervised fine-tuning.
Summary. Table key hyperparameters. Training configs, dataset prep, and evaluation code are included in the supplementary material.
B SUPPLEMENTARY METHODOL.OGICA BACKGROUND
B.I CTC AND ATTENTION-BASES MODELS
Connectionist Temporal Classification (CTC). CTC (Graves et al. 2006) models the probability
of a target sequence (yi...., Wu) given input frames æ by marginalising over all monotonic alignment paths n
Percly 1x)
I[ x),
REB 1=1
where n. is the label (or blank) at timestep and collapses repeats and blanks recover y. This formulation (i) enforces a left-to-right monotonic alignment, since paths are built only by inserting blanks or repeats (never by reordering labels), and (ii) assumes each Ne is conditionally independent of Tht given thereby allowing the path probability to factorise as II. P( z). This explicit alignment mechanism tends be robust under distribution shifts, but the conditional independence assumption limits modelling capabilities (Watanabe et al. 2017).
Attention-based encoderrdecoder models. define the conditional probability:
contrast, attention- based encoder-decoder models
Pauly x)) = II P(ya Veuuy
(8)
Unlike CTC, attention- based decoders do not assume conditional independence across timesteps, allowing them to model long-range dependencies and complex token interactions. However, they do enforce monotonic alignment and must instead learn alignment patterns purely from data. As a result, they are highly expressive but often brittle under distribution shifts, such as longer input sequences or noisy data, where learned alignments may fail to generalise.
During training, these models are typically trained using teacher forcing (Williams & Zipser 1989) where the ground-truth tokens yсu are fed inputs to the decoder instead of its own predictions. This stabilises learning and significantly accelerates computation by enabling parallel sequence processing in transformer-bast based architectures. However, at inference time, the model must operate autoregressively, meaning each predicted token must be fed back into the decoder for the next step.
Joint CTC-attention training and decoding. To leverage the strengths of both CTC and attention- based models, hybrid architectures incorporate joint CTC-attention framework (Watanabe et al. 2017.. This approach optimises a multi-task objective:
(9) where Lete is the CTC loss (typically applied to projection of the transformer encoder outputs), LAu is the attention- cross-entropy loss (applied to the decoder outputs), and À is a tunable hyperparameter balancing the two objectives.
L = XLere + (1 (1- AAn
At inference time, joint decoding is often performed using beam search, combining CTC and attention scores:
S(y') = a log Pyrc(y' xx) + (1 a) log Pan(y' xx)
(10)
where y' is candidate hypothesis and controls the relative contribution of the CTC branch. The CTC probabilities help constrain the beam, reducing alignment errors attention- decoding and improving robustness to distribution shifts.
While effective, beam search is computationally expensive and thus impractical in iterative self- training setups, where decoding must be performed repeatedly over large unlabelled datasets.
18

---

Published as a conference paper at ICLR 2026
B.2 ADDITIONAL DETAILS ON USR
While the main paper focuses on our new pseudo-labelling generation and prediction strategy, this section supplements the brief recap Section with components of USR that are not the focus in USR 2.0, specifically the confidence filtering strategy (excluded from the main text for simplicity) and more detailed formulation of the cuni-supervised loss. For clarity, we describe the setting using one labelled and one unlabelled sample per iteration (with audio, visual, and audiovisual views); in practice, batches of by labelled and b, unlabelled samples are used, typically with by > bn
Shared parameters. USR is single model trained perform ASR, VSR, and AVSR jointly. Aside from modality-speciffe feature extractors (audio and visual frontends), the remaining components (encoder and decoder heads) are shared across modalities. These shared components operate on token sequences, allowing audio, visual, and audiovisual inputs to be processed in unified transformer architecture after appropriate modality-speecinn projections. A key benefit of this design in the self-training setup is that pseudo-labels only need to be generated once per unlabelled audiovisual sample. These shared pseudo-labels can then supervise three modality-specific inputs (audio-only, video-only, and audiovisual), amortising the cost of pseudo-label generation.
Confidence filtering, To reduce the impact of erroneous pseudo-labels during self-training, USR applies confidence-batt based filtering, discarding low-confidence predictions from the teacher model. Let cC [0, 1]4x[v| and A [0, 1]4xxx]P] denote the softmax outputs of the teacher's CTC and attention-basee heads, respectively, where V is the subword vocabulary. Frame- and token-wise masks are obtained as follows:
CTC .TCC I(max: Pi,k rt
2 T),
A
IImax Aul T)
(11)
with confidence threshold 0.8.
In USR 2.0, the CTC PLs are token-level (rather than frame-level) due to the collapse operation, so it is difficult to use the frame-wis confidence scores cCTC max; Pi.k function. Instead, compute single sequence-level confidence:
CTC directly in the CTC loss
conf(i cxp
Σ
log ci
CTC)
(12)
L
The CTC sequence-level mask is then given by
CT = (conf CTC
27)
(13)
This reflects the fact that in USR 2.0, CTC PI. filtering must occur at the sequence level due to the way the CTC loss is computed.
Unlabelled losses. Let goom denote the student predictions from head w E [CTC,At] for modality m (A, V, AV}. In USR. the cross-entropy losses on unlabelled data (with confidence masking) are
m
CTC CE((
CTC, c ,DETC)
(14)
T.
t=1
UAr
At
An CEE -TA").
(15)
UAR
In USR 2.0, the unlabelled losses from equation All The CTC and attention unlabelled losses are then combined per modality using:
equation 6 are similarly masked using CTC and
ÀCTe L cCC... + (1 - >crc) A ACTC E [0, 1].
(16)
Labelled losses. For labelled samples, USR employs the standard joint CTC-attention objective
described in Section B The resulting per-modality labelled loss is denoted by LT
19

---

Published as a conference paper at ICLR 2026
Table 7: Comparisons with the state-of-the-art on LRS3. "Labels include automatic transcriptions from ASR models trained on large-scale, often non-public datasets. "Shared params" denotes whether a unified model is used at inference. "ST" denotes offline self-trainie training.. Uses Whisper as its encoder, trained on ~680k hours of transcribed speech data (Radford et al. 2023
Labelled hours
Unlabelled hours
Lang. model
Shared params
WER (%)
Method
V
A
AV
Supervised
V2P
3,886
X
X
55.1
RNN-T
31,000
/
33.6
4.8
4.5
VTP
2,676
V
X
30.7
Llama-AVSR
680,000
V
X
24.0
0.8
0.8
Auto-AVSR
1,902
X
23.5
1.0
1.0
Auto-AVSR
3,448
V
X
19.1
1.0
0.9
ANKIA
ViT3D-CM
90,000
17.0
1.6
Synth
6,720
16.9
LP Conf
100,000
12.8
0.9
Whisper-Fllmmingo'
680,000
X
0.7
0.8
Seff/semi-supervssed
AV-HuBERT w/ ST
433
1,326
X
28.6
RAVEn w/ ST
433
1,326
X
23.1
1.4
BRAVEn w/ ST
433
2,649
X
20.1
1.1
USR
433
1,326
V
21.5
1.2
1.1
USR 2.0
656
2,649
V
17.6
0.9
0.8
Final semi-supervised objective. Let wm [0, denote the per-modality weight and 7m E [0, control the balance between unlabelled and labelled losses. The full loss is:
Leemi
Σ
Wm 7im CT + 7im LT").
(17)
(AV.VV
As described in Section we set:
WAV 0.7, = 0.3,
7/V 0.75,
0.97.
(18)
C ADDITIONAL RESULTS
C.I FULL SHATE-OF-THE-AL ART COMPARISONS
Tables and 9 present comparisons with prior work on LRS3, LRS2, and WildVSR. We report results from strong supervised systems trained with vastly more labelled or automatically transcribed data, including V2P (Shillinffrr et al. 2018), RNN-T (Makino et al. 2019), VTP (Prajwal et al.
2022). Llama-AVSR (Cappellazzo al. 2024). Auto-AV SR (Ma a (203)) ViT3D-CM (Serdyuk et al. 2022), Synth VSR (Liu et al. 2023) LP Conf (Chang et al. (024). Whit Whisper-Flamingo ditchenko al. 2024). CM-seq2seq (M et al., 2021), and CM -uxx (Ma et al. 2022). All results
Flamingo (Rou-
for USR 2.0 refer to our Huge model, trained on LRS2 and LRS3 labelled data and unlabelled English-only VoxCeleb2 and AVSpeech, without the use of an external language model.
USR 2.0 achieves strong performance across all modalities. On LRS2 and LRS3, it outperforms Auto-AVSR, which leverages automatic transcriptions from external ASR models, and is on par with Whisper-Flamingo and Llama-AVSR, which initialise from Whisper models trained 680,000 hours transcribed speech data. On WildVSR (Table 9), which contains video clips with high visual variability, USR 2.0 obtains state-of-the- performance, surpassing prior heavily supervised systems. This highlights its robustness to mn-tee-wil conditions such as variable head pose, lighting, and background.
20

---

Published as a conference paper at ICLR 2026
Table Comparisons with the state-of-the-art on LRS22. "Labels include automatic transcriptions from ASR models trained on large-scale, often non-public datasets. "Shared params" denotes whether a unified model is used at inference. "ST" denotes offline tell-nraining. UUses Whisper as its encoder,
trained 6600 hours of transcribed speech data (Radford et al. 2023
Labelled hours
Unlabelled hours
Lang. model
Shared params
WER (%)
Method
A
AV
Supervised
CM-seq2seq
380
X
37.9
3.9
3.7
CM-aux
1,459
X
25.5
VTP
698
X
28.9
VTP
2,676
X
22.6
Auto-AVSR
818
/
X
27.9
2.6
Auto-AVSR
3,448
X
14.6
1.5
1.5
680,000
X
1.3
1.4
Self/semi-superriieed
Uni-AVSR
223
60,000
X
X
43.2
2.7
2.6
A
LiRA
223
433
38.8
RAVEn
223
1,759
X
23.2
2.5
RAVEn w/ ST
223
1,759
17.9
2.3
USR
223
1,759
15.4
1.9
1.9
USR 2.0
656
2,649
12.6
1.3
1.3
Table 9: Comparisons with the state-of-the-art on WildVSR. 'Labels include automatic transcrip- tions from ASR models trained on large-scale, often non-public datasets. "Shared params" denotes whether unified model is used at inference. "ST" denotes offline self-training.
Labelled hours
Unlabelled hours
Lang. model
Shared params
Method
WER (%)
Supervised
CM-seq2seq
1,459
58.4
VTP
698
X
75.6
X
VTP
2.676
68.7
Auto-AVSR
661
/
62.3
XXX
Auto-AVSR
1,759
/
49.3
Auto-AVSR
3,448
/
X
38.6
Selffsemi-superriieed
AV-HuBERT
433
1,326
X
ANNIK
51.7
AV-HuBERT w/ ST ST
433
1,326
48.7
XX
RAVEn
433
1,326
52.2
RAVEn w/ ST ST
433
1,326
46.7
USR
433
1,326
46.4
USR 2.0
656
2,649
X
38.5
21

---

Published as a conference paper at ICLR 2026
Table 10: Ablation results. We report WER (%) for VSR (V), ASR (A), and AVSR (AV) on in-distribution test data (LRS3) and AVSR on out-of-distribution data (sampled, automatically transcribed AVSpeech). Default settings for USR 2.0 are highlighted in gray
(a) Weight on attention vs. CTC pscudo-labels for decoder loss in CTC-driven mode.
(b) Weight on attention vs. CTC pseudo-labels for CTC loss in AR mode.
ID
OOD
ID
OOD
Weight
Weight
V
A
AV
AV
V
AV
AV
36.7
3.7
3.6
25.5
36.3
3.2
45.1
0.25
37.1
3.5
3.4
24.2
0.25
36.7
3.2
2.9
41.9
0.5
36.6
3.3
3.2
24.2
0.5
36.5
3.1
2.9
40.1
0.75
36.4
3.2
3.2
28.1
0.75
36.2
3.0
2.9
44.4
36.5
3.3
3.3
35.1
36.1
3.0
2.8
52.3
(c) Effect of CTC merge & collapse on pscudo-labels in USR.
(d) AR mode sampling schedule.
ID
OOD
Schedule
ID
OOD
Method
V
A
AV
AV
AV
AV
Constant
36.2
3.0
2.9
25.0
Without
36.0
3,2
3.0
34.7
Linear
36.5
2.9
2.9
24.8
With
36.2
3.2
2.9
36.9
Cosine
36.6
3.0
2.9
25.4
These results demonstrate the benefits of scaling both model capacity and unlabelled data within unified semi-supervised learning framework. While USR 2.0 already outperforms many task- specific baselines, we anticipate further improvements through continued scaling of compute and data. Importantly, all results obtained using single shared model for ASR, VSR, and AVSR, without relying on task-specific tuning or external language models.
C.2 ADDITIONAL ABLATIONS
We conduct additional ablations on the Base model in the low-resource setting, using LRS3 as the unlabelled dataset. We evaluate in-distribution performance on the L.S33 test set for VSR, ASR, and AVSR, and out-of-distribution performance the sampled AVSpeech test set (AVSR with greedy decoding). Results are shown Table 10
Loss weights for auxiliary lseudo-labels. Recall that in CTC-driven mode, the decoder is trained match both CTC and attention-based pseudo-labels. Similarly, in AR mode, the CTC branch is trained to predict both pseudo-labe types. By default, the coefficients balancing the contribution of these losses are set to 0.5 in both cases (see [quation[S-ccuattinn different values of these coefficients from {0, 0.25, 0.5, 0.75, }} for each prediction head.
in the main text). We ablate
As shown in Table 10 a), assigning greater weight CTC pseudo-labels in the decoder loss generally robustness but can degrade in-distribution accuracy, especially for ASR and AVSR, which are more sensitive to pseudo-label noise. This trade-off highlights the complementary inductive biases of CTC and attention: CTC promotes monotonic alignment and stability, while attention models long-range dependencies that are more beneficial clean, in- distribution settings.
improves out-of-distribun
Interestingly. Table b0]b) shows that combining CTC and attention aatention-based pseudo-labels in the CTC loss of the AR mode improves OOD performance over using either type alone, suggesting that the two supervision signals reinforce each other when used the CTC branch.
CTC merge & collapse. USR uses a cross-entropy loss directly on frame-level pseudo-labels and predictions, without collapsing repeated tokens removing blanks. In contrast, USR 2.0 applies the standard CTC posc-procs step (merging repeats and deleting blanks) to convert frame-level outputs into token-level pseudo-labels. To isolate the effect of this change, compare training USR
22

---

Published as a conference paper at ICLR 2026
BRAVEn
USR
USR 2.0
Altentinn
Adaettionn
Attention
100
CTC
CTC
CTC
Seen Lengths
Seen Lengtht
Seen (anglhs
(%)
80
60
20
100
200
300
100
500
600
100
200
300
400
500
600
100
200
300
400
500
600
Input Length (Frames)
Input Length (Frames)
Input Length (Frames)
Figure 6: WER vs. input length on VoxCeleb2 (automatically transcribed using Whisper). Models are trained in the low-resource LRS3 setup, where labelled utterances are no longer than around 150 video frames. We evaluate CTC-only and attention-only greedy decoding for BRAVEn, USR, and USR 2.0.
with and without merge & collapse. As shown in Table [0]c), this choice has negligible impact performance in USR. The benefit of collapsing in USR 2.0 stems not from improved label quality, but from enabling compatibility with teacher forcing in the decoder: it significantly reduces sequence length and maps the targets into the subword token space required by the decoder input.
Sampling schedule. USR 2.0 alternates between CTC-driven and AR modes when training on unlabelled data. By default, training. A natural hypothesis is to begin with more frequent use of the more robust CTC-driven mode, then gradually increase the use of AR mode improve expressiveness. To test this, evaluate both linear and cosine schedules for the AR sampling probability, increasing from 0 to over the course of training. As shown in Table [O)d), neither schedule outperforms the simpler constant strategy, suggesting that fixed mixing is sufficient in this setting.
constant sampling probability of 0.5 for AR mode throughout
C.3 OUT-OF-DISTRIBUTTIN ROBUSTNESS
Length generalisation of decoding branches. Figure 6 evaluates the robustness of CTC-only and attention-only decoding across input lengths for BRAVEn, USR, and USR 2.0. We use the Base models trained in the low-resource setting (on LRS3), where the longest labelled utterances span roughly 150 frames. Evaluation is performed on out-of-distribution samples from VoxCeleb2, grouped into length buckets (e.g.. 150-200, 200-250, etc), as in Figure from the main text.
All models perform stably under CTC-only decoding, likely due to the robustness of monotonic alignment. However, attention-onle decoding reveals notable differences. Both BRAVEn and USR exhibit sharp increases WER as input length grows, reflecting the decoder's 's fragility on long, out- of-distribution utterances. Notably, USR deteriorates more severely than BRAVEn, likely due its self-training loop, where autoregressive pseudo-labels are generated by a teacher that is itself derived from the student, compounding early decoding errors. In contrast, USR 2.0 maintains significantly better performance, suggesting that its training strategy (i.e., the transfer of alignment information from the CTC branch to the decoder) successfully stabilises attention based predictions.
Qualitative examples. To illustrate this effect, Table presents qualitative decoding examples on long, out-of- distiittibtttionn utterances. Transformer decoders are known to struggle with such inputs under greedy decoding, often producing truncated or repetitive outputs (Fan al., 2018 Ho foltzman et al. 2019). These issues are amplified in pseudo-labelling setups, where standard remedies like beam search with length penalties are too costly to apply. USR, we observe such failures on utterances longer than those seen during supervised training, typically manifesting omissions or degeneracies.
USR 2.0 mitigates these issues by jointly supervising the decoder with both CTC and attention-based pseudo-labels. This promotes more stable and complete sequence generation, even under greedy decoding. As shown in Table II USR 2.0 produces fluent hypotheses, while USR suffers from repetition and content loss. These findings echo our quantitative results in Figures and 3 of the main text, where USR 2.0 maintains robustness long utterances.
23

---

Published as a conference paper at ICLR 2026
Table 11: Qualitative analysis of USR vs. USR 2.0 on long, out-of-distribution utterances. We show examples where USR either omits substantial segments (top) or degenerates into repetitions (bottom), both of which USR 2.0 resolves. Segments omitted by USR but correctly transcribed by USR 2.0 are highlighted in green; repetition errors by USR are highlighted in red.
(a) Omission Failures
GT
because with them we can break this cycle of suffering without them the cycle will continue I'll lcave you with the word that we use at the end of every yoga class which also illustrates this fundamental truth I really believe in that we are all one because with them we can break this cycle of every yoga class which also illustrates this fundamental truth I really helieve in that we are all one because with them leave you with the word that use at the end of every yoga class which also illustrates this fundamental truth really believe is that we are all one
USR
USR 2.0
break this cycle of suffering without them the cycle will continue I'll
GT'
so the evidence is there and it's going to be designed to interpret it correctly will understand for example that books are very biased in the evidence they contain they only talk about kings and princes and elite white male people doing stuff so it's a complicated problem hut as it learns more about our objectives it will become more and more so it's a complicated problem but as it learns more about our objectives it will hecome more and more
USR
USR 2.0
the evidence there it's going be designed to interpret it correctly it will understand for example that books are very biased in the evidence they contain they only talk about kings and princes and elite white male people doing stuff so it's a complicated problem but as learns more about our objectives it will become more and more
GT
you may find these stories inspirational have to be perfectly honest I find them daunting a little bit unsettling because I look at my own life and I ask is this going to happen to me and if it does will I recognize it and if I recognize it will I have the guts to make the leap myself so I took look some of these leaps into the void because if super successful peoples' success in fact does not hinge upon living on the edge
USR
so I took look at some of these leaps into the void because if super successful people success in fact does not hinge upon living on the edge
USR 2.0
you may find these stories inspirational have to be perfectly honest find them daunting a little bit unsettling because I look at my own life and I ask is this going to happen to me and if it does will I recognize it and if I recognize it will I have the guts to make the leap myself so I took look at some of these leaps into the void because if super successful people's 's success in fact does not hinge upon living on the edge
(b) Repetition Failures
GT
here's striking thing though here's what Shery says Shery says yeah it's an incredibly painful experience but she wouldn't 't have it any other way hecause what it's taught her is here's the striking thing though here's what Sharry says Sharry says Sharry says Sharry says Sharry says Sharry says Sharry says Sharry says...
USR
USR 2.0
here's 's the striking thing though here's 's what Shary says Shary says yes incredihly painful experience but she wouldn't 't have any other way because it's what it's laught her is
GT
at the pebhle the bottom of the stream the stream is continuously moving and turbulent and that makes il very difficult to see the pebble the botlom of the stream very much in the same way it's very difficult see astronomical sources
USR
at the pubble on the bottom of stream the stream the stream the stream the stream the stream the stream the stream the stream the stream...
USR 2.0
at the puble on the bottom of stream the stream is continuously moving and turbulent and that makes it very difficult to see the puble on the bottom of the stream very much in the same way it's very difficult to see astronomical sources
GT
your grandmother was on TEDx Youth and that was like so cool back then I want go like I went this and I did that and saw this want tell them stories
USR
your grandmother was on TEDx Youth and that was like so cool back then see I want to go like I
went to this and I did that and I saw this and saw this and I saw this and I saw this...
USR 2.0
your grandmother was on TEDx Youth and that was like so cool back then see I want to go like I
went to this and I did that and I saw this and I wanted to tell them story
24

---

Published as a conference paper at ICLR 2026
Example 1
Ground-truth sentence
WE EXCISE FROM OUR BODIES
CTC pseudo-labels
<508>
CTC-driven attention pseudo-labels
WE
.EX
CI
SE
JT
FROM
.UUR
B0
Di
ES
<EOS>
Example 2
Ground-truth sentence:
EVERYONE
CTC pseudo-labels
-808-
EVERYONE
S
DOKAY
CTC-driven attention
pseudo-labels
EVERYONE
_OKAAY
-EOS-
Example 3
Ground-truth sentence
DO WANT TALK ABOUT IT
CTC psaudo-labels
<808
DOO
WANT
TO
.TLK
ABBUUT
.T
CTC-driven attention pseudo-labels
ABOUT
Figure 7: Examples of AR inconsistencies in decoder outputs under CTC-driven teacher forcing. In each case, the decoder conditioned on CTC-derived prefixes, sometimes leading to repeated or malformed outputs the sequence level, even though each token prediction is plausible given its input. These artefacts do not affect training, as the student learns to match the teacher's outputs under the same input.
C.4 CTC- CTC-CONDITIONND SEQUENCE- INCONSISTENCIES
When using CTC-driven teacher forcing to generate decoder targets for self-training, observe that the resulting attention based outputs may exhibit sequence-level inconsistencies, even though each individual token prediction is conditionally valid. Specifically, the decoder is conditioned the sequence of subwords predicted by the CTC branch, and the decoder produces the most plausible next token given each CTC-derived prefix. If the decoder and CTC branches diverge on certain tokens, the resulting attention-bacn based output may contain repeated or malformed phrases when interpreted as complete sequence.
Figure 7 iillustrates examples of this phenomenon. In the first case, the CTC output incorrectly includes prefix ending in "EXCISE" the
an extra subword ("ES" after "EXCISE"). When conditioned decoder correctly predicts "IT". However, for the subsequent prediction, the decoder sees the CTC- based prefix ending "EXCISEES" and again predicts "IT". which reasonable output under the given context. This results in duplicated "IT" in the final sequence. In another case, the CTC branch outputs "EVERYONE'S" leading the decoder to generate "IS" when conditioned on the prefix "EVERYONE", followed by "S" under the prefix "EVERYONE" resulting in the malformed word "ISS". Finally, we show an example where the CTC output incorrectly predicts "DON'T" instcad of "DO", prompting the decoder generate "DO'T" due to the prefix "DON".
We emphasise again that these The student decoder is trained to predict the teacher's 's outputs under the same CTC-driven inputs, making the decoder's predictions consistent given its inputs, even they appear inconsistent at the sequence level.
artefacts do not undermine the training process.
25

---

Published as a conference paper at ICLR 2026
Table 12: WER % for USR 2.0 under different decoding strategies inference. We compare CTC-only decoding, attention-only greedy decoding, joint CTC + attention decoding with beam search (beam size 30), and CTC-driven teacher forcing
Decoding Method
A
AV
CTC-only (greedy)
45.6
4.1
4.0
Attention-only (greedy)
41.8
3.8
3.9
CTC + attention (beam size 30)
36.2
3.0
2.9
CTC-driven teacher forcing
47.1
4.2
4.1
VSR
ASR
AVSR
0.98
0.98
Decoder Accuracy
0.96
0.96
0.7
0.94
0.94
0.6
0.92
0.92
0.90
0.90
4.0
1.0
1.0
0.8
Loos
3.0
0.6
0.6
0.4
0.4
2.0
10
20
30
40
50
0
10
20
30
40
50
0
10
20
30
40
50
Epach
Epach
Epech
Figure Validation curves for USR and USR 2.0. Top row shows decoder accuracy curves; bottom row shows CTC validation loss. Each curve shows mean + std over three seeds.
C.5 IN-DISTRIBUTION BEHAVIOUR
Impact of decoding strategy at inference (in-distribution). Table 12 presents WER results for USR 2.0 under different decoding strategies the in-distribution LRS3 low-resource benchmark. We observe that attention-only greedy decoding outperforms CTC-only decoding across ASR, VSR, and AVSR, reflecting the decoder's 's strong sequence modelling capabilities. As expected, the best performance is achieved using joint CTC + attention decoding with beam search (albeit at much higher computational and memory cost). On the other hand, as we have seen in Section C attention decoding tends to degrade on out-of fitt-f--disrrbuttinn inputs such as longer utterances, while CTC-only decoding remains stable. This contrast underscores the complementary strengths of the two branches.
We also evaluated whether CTC--riven teacher forcing can be used an inference-ting decoding strategy. As shown in Table T2, this approach yields higher WER than standard greedy or beam-search decoding. This is expected: the benefit of CTC-driven teacher forcing arises during self self-training from matched teacher-student conditioning. At inference, where the model must generate coherent outputs autoregressively, CTC-driven decoding does not perform better than conventional CTC-based or AR methods.
Validation curves and convergence. Figure g compares the validation dynamics of USR and USR 2.0 across VSR, ASR, and AVSR, using Base models trained in the low-resource setting over 50 epochs with LRS3 as the unlabelled data source (in-distribution setting). We report decoder accuracy (top row) and CTC validation loss (bottom row), averaged over three random seeds. Across all tasks, USR 2.0 consistently achieves higher decoder accuracy and lower CTC loss throughout training, indicating more effective learning within fewer total epochs.
26

---

Published as a conference paper at ICLR 2026
Table 13: Training Duration Comparison. We report LRS3 low-resource WER (%) results for 50- and 75-epoch training. Mean and standard deviation over three random seeds are reported for the 50-epoch runs.
Method
Epochs
AV
USR
50
37.6
0.7
3.4 0.1
3.2 0.1
USR 2.0
50
36.2 + 0.5
3.0 + 0.1
2.9 + 0.1
USR
75
36.0
3.2
3.0
USR 2.0
75
36.1
3.0
2.9
As shown in Table extending training for USR 75 epochs allows it to close much of the gap in this in-distribution setting (where even the unlabelled data is in distribution), suggesting that USR 2.0 reaches comparable or better performance with fewer epochs. This contributes to its overall training efficiency, which stems from both faster training steps and reduced number of total steps (see Figure in the main text).
C.6 FAILURE CASES
shows representative failure cases for the Huge USR 2.0 model on the LRS3 test set. Although the model performs strongly across modalities, some error patterns remain. First, VSR produce visually plausible but semantically incorrect substitutions (e.g., "break" "prod", "conscious" vs. "cautious"), which ASR and AVSR typically correct. Second, there are cases where "They want treatment"), but the
Table 14
ASR can drift semantically (e.g.. "It wasn't treatment" audiovisual model resolves the ambiguity, illustrating how visual cues can compensate for uncertain acoustics. Finally, small number errors persist across all modalities. These tend be phonetically understandable (e.g.. "Signfeld" vs. "Seinfeld", or "had" vs. "ought"",
D DISCUSSION LIMITATIONS
Despite its advantages, USR 2.0 also presents limitations. First, while our proposed modifications significantly reduce training time compared to USR (nearly halving it), they do not match the efficiency of self-supervised approaches that rely fully supervised fine-tuning stage (Shi et al. 2022a Haliassos al. 2022a 2024b). In low-resource settings, where the labelled dataset is small, such methods benefit from training the model on limited labelled data, making them substantially faster in wall-clock time. In contrast, USR and USR 2.0 iteratively optimise on both labelled and unlabelled data through self-training, leading to longer overall training durations. However, this trade-off enables unified learning across ASR, VSR, and AVSR tasks. By leveraging across modalities, USR 2.0 reduces hyperparameter sensitivity and achieves strong performance on all tasks using a single model, without sacrificing inference speed and with significantly lower deployment cost compared to training and maintaining separate models.
Second, we observe that the benefit of additional unlabelled data is more pronounced for VSR than for ASR AVSR. This is likely due the higher WERs VSR, which allow it benefit from increased data volume even if pseudo-labels are noisy. In contrast, ASR and AVSR, which already perform well in low-resource settings, may be bottlenecked by label quality rather than quantity. As result, greedy decoding (used here to keep may limit further improvements in ASR and AVSR, where higher-quality supervision could be more beneficial. We believe that developing strategies for improving pseudo-label quality without significantly compromising efficiency is challenging but important area for future work.
computationally feasible)
Third, CTC-driven teacher forcing is specifically designed for iterative self-training, where the teacher evolves over time and pseudo-labels must be generated efficiently at every iteration. In this regime, matching the teacher and student conditioning is more important than global sequence coherence, and the method works well in practice. However, in non-self self-training scenarios, such as offline ascudo-labelling with frozen pre-trained model or inference-time decoding, coherence becomes desirable. In these settings, standard autoregressive or beam-search decoding is more appropriate, as they (though slower) are run only once and do not dominate training cost.
27

---

Published as a conference paper at ICLR 2026
Table 14: Representative failure cases for USR 2.0 on the LRS3 test set. We show outputs from the Huge model. Correct fragments are shown in green; errors are in red.
GT
We push and pull and poke and prod things We push and pull and poke and break things
V
A
We push and pull and poke and prod things We push and pull and poke and prod things
AV
GT
You're more cautious You're more conscious
V
A
You're more cautious You're more cautious
AV
GT
And one activist actually found a contract from a western company for the sale of surveillance
V
And one activist asked if could contract from a western company for sale surveillance And one activist actually found a contract from a western company for the sale of surveillance
A
AV
And activist actually found a contract from western company for the sale of surveillance
GT
They want treatment They want that treatment It wasn't 't treatment They want treatment
V
A
AV
GT
I said we work together think work the said worked together I said work together
V
A
AV
GT
What difference does it make if they talk like Jerry Seinfeld What difference does it make if they talk like church they fail What difference does it make if they talk like Jerry Signfeld What difference does it make if they talk like Jerry Signfeld
A
AV
GT
That's something we ought to celebrate
V
That's something we can't celebrate That's something we had to celebrate
A
AV
That's something we had to celebrate
E BROADER IMPACT
Unified speech recognition systems like USR 2.0 have the potential to expand accessibility across wide range of real-world settings. ASR can facilitate communication for individuals with motor impairments reading difficulties, while VSR AVSR offer an alternative for users with aphonia or environments with poor audio quality. Moreover, the techniques developed here may contribute to broader advances in multimodal machine learning, including audiovisual forensics. For instance, it has been shown that the ability to accurately interpret audiovisual speech can support the detection of manipulated media, such deepfakes (Haliassos et al. 2021 2022b).
However, this technology not without risks. Audiovisual speech recognition may be misused for mass surveillance covert monitoring through video feeds, particularly in contexts where individuals have not consented to such analysis. This raises important questions around privacy, informed consent, and the need for regulatory oversight. Additionally, as with most large-scale machine learning systems, performance disparities may arise if models are trained on imbalanced datasets that underrepresent certain demographic groups. This could lead to inequitable outcomes, with
28

---

Published as a conference paper at ICLR 2026
worse recognition rates for users based gender, age, accent, ethnicity, which warrants careful consideration.
LLM USAGE DECLARATION
In accordance with the ICLR 2026 policy on large language models (LLMs), we disclose that LLMs were used solely for polishing the language of this paper. Specifically, we employed LLM to rephrase certain sentences for clarity and conciseness. No part of the paper's technical content, experimental design, analysis, conclusions was generated by an LLM.
29


Reviews:
[{"review_id": "2d4pvImyha", "reviewer": "Reviewer_3D7N", "summary": "The paper presents USR 2.0, an improved version of the Unified Speech Recognition (USR) framework that integrates ASR, VSR, and AVSR into a single model. The original USR is limited by the high cost and error accumulation of autoregressive decoding for attention-based pseudo-labels. USR 2.0 addresses this with CTC-driven teacher forcing, which uses CTC outputs to generate attention pseudo-labels in a single forward pass. A mixed sampling strategy balances the expressiveness of autoregressive training with the robustness of CTC. As a result, USR 2.0 achieves faster training, improved robustness to distribution shifts, and state-of-the-art performance on LRS2, LRS3, and WildVSR using a unified model.", "questions": "* Mixed sampling uses AR decoding in only half of the training steps, yet inference is fully AR. Have the authors compared this to a model trained entirely with AR decoding? It would clarify how much performance is being traded for efficiency.\n\n* Is there any potential to use CTC-driven attention decoding at inference time? While coherence may be limited, it might still be useful in latency-sensitive scenarios.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "* The authors' overall intuitions are reasonable and well-supported by design choices. USR 2.0 introduces a CTC-driven pseudo-labelling approach that effectively removes the autoregressive bottleneck in attention-based decoding, resulting in significantly faster training and improved inference efficiency.\n\n* The model unifies ASR, VSR, and AVSR within a single architecture, and demonstrates strong robustness to long inputs, noise, and domain shifts, while maintaining competitive performance on ID benchmarks.\n\n* The paper is further strengthened by a clear training strategy, a simple yet effective mixed sampling method, and a comprehensive set of experiments including ablations, OOD tests, and qualitative analysis.", "weaknesses": "* Since the attention decoder must operate autoregressively during inference, the benefit of test-time parallelism is limited, and the speedup primarily applies to the training phase.\n\n* Although attention pseudo-labels generated from CTC-driven decoding may lack global coherence, the authors convincingly argue that this does not hinder learning during self-training, as both teacher and student are conditioned on the same token sequence. However, this could limit reuse of such pseudo-labels in non-self-training settings, where inference coherence is required.\n\n* Minor weakness :\n    * In Figure 2, the line representing teacher forcing looks quite similar to the other lines. Please change it to a different color for better visibility.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cn1wxW31IZ", "reviewer": "Reviewer_t5vy", "summary": "The paper proposes a framework, for semi-supervised universal speech recognition, USR 2.0 building upon USR. Hybrid CTC–attention training in USR suffers from inefficiency and instability: autoregressive (AR) decoding is slow and error-prone when generating pseudo-labels for unlabeled data, and accumulated prediction errors can degrade model quality. The authors identify that the CTC branch—though less expressive—is inherently more stable and monotonic, making it a good candidate for guiding the attention branch.\n\nTo exploit this, the paper introduces CTC-driven teacher forcing, where the teacher model’s CTC predictions are first greedily decoded and merge-collapsed and then used as fixed token prefixes to condition the attention decoder. This approach removes the need for sequential AR decoding, allowing attention-based pseudo-labels to be generated in parallel in one forward pass. Although such CTC-conditioned sequences may not be globally coherent, they still provide consistent and aligned supervision for training, since both teacher and student operate under identical conditioning.\n\nHowever, this introduces a mismatch between training and inference: during training, the decoder conditions on CTC tokens, while at inference it uses its own AR outputs. To reduce this exposure bias, the paper adds a mixed-sampling strategy, randomly alternating between CTC-driven mode and standard AR mode. The CTC-driven mode emphasizes efficiency and robust alignment, while the AR mode preserves linguistic coherence and matches inference conditions. Together, these yield faster training, improved long-form stability, and better robustness to out-of-distribution data, while retaining the modeling flexibility of attention decoding.", "questions": "See Weaknesses above.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. CTC-driven teacher forcing is an elegant idea that leverages the stability and monotonic alignment properties of CTC to guide the attention decoder. It enables parallel pseudo-label generation without slow autoregressive decoding, thereby reducing computational cost and eliminating cascading AR errors.\n\n2. Improved training efficiency: The paper shows a 2× reduction in training time, which is significant for multimodal setups (audio, visual, audiovisual). This is achieved without compromising accuracy, highlighting the practicality of the approach.\n\n3. Strong robustness across conditions: The mixed-sampling strategy strikes a balance between efficiency and exposure-bias mitigation. The model demonstrates improved out-of-distribution (OOD) robustness, particularly for long-form or noisy inputs, compared to both the baseline USR and modality-specific self-supervised systems.\n\n4. Thorough experimentation: The experiments are well-structured, spanning ASR, VSR, and AVSR tasks and several benchmarks. The ablations clearly quantify the contribution of each component.\n\n5. Conceptual clarity and reproducibility: The paper gives a clear conceptual link between the CTC and attention paradigms, and explains the motivation for bridging them. The code and implementation details are provided, making the approach accessible to replicate.", "weaknesses": "The proposed approach relies heavily on the quality of CTC-generated pseudo-labels. While the paper acknowledges that the student decoder is trained to predict the teacher’s outputs under the same CTC-driven inputs, the method still assumes that these pseudo-labels are sufficiently coherent to guide decoder learning. For challenging or noisy segments, however, CTC errors can propagate through teacher forcing since the decoder conditions directly on these imperfect sequences. Consequently, the learning signal may not always be optimal, potentially limiting the broader applicability of the approach beyond the specific experimental settings demonstrated in the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "c3MkjyB30g", "reviewer": "Reviewer_Qtvc", "summary": "This paper introduces USR 2.0, an enhanced version of the Unified Speech Recognition (USR) framework. It effectively addresses its predecessor's limitations (high training cost, poor robustness, etc.) to deliver a more efficient and reliable semi-supervised model for ASR, VSR, and AVSR. The core innovation is CTC-driven teacher forcing, which speeds up pseudo-label generation approximately 40x and couples the two supervision signals, enabling the model to achieve new state-of-the-art results while halving the total training time.", "questions": "1.\tAre there any failure cases for USR 2.0?\n2.\tWhy weren’t other methods, especially USR, trained with the highest-resource setting (Huge)?\n3.\tBeyond speech recognition, could this CTC-driven teacher forcing paradigm be applied to other sequence-to-sequence tasks?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1.\tThe paper itself is clearly written and easy to follow.\n2.\tThe paper directly addresses the problems and limitations of USR.\n3.\tThe results are robust compared to baselines.", "weaknesses": "There are no obvious flaws in this paper. Only a few points require clarification (see Questions).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lTgXIKTtdZ", "reviewer": "Reviewer_Q3PZ", "summary": "This paper introduces USR 2.0, a new semi-supervised framework for unified speech recognition (ASR, VSR, AVSR). It directly addresses two issues in the original USR: the high training cost of autoregressive (AR) pseudo-labeling and the model's brittleness on out-of-distribution (OOD) data. The core contribution is a mechanism called CTC-driven teacher forcing, where fast and robust pseudo-labels from the teacher's CTC branch are used as input prefixes to the teacher's decoder to generate attention-based targets in a single, non-autoregressive pass. This is combined with a \"mixed sampling\" strategy to mitigate train-test mismatch. The authors claim this approach halves training time, dramatically improves OOD robustness, and achieves state-of-the-art results with a single model.", "questions": "- Could you elaborate on the mechanism that allows the decoder to learn from \"incoherent\" attention targets (as shown in Fig 7 ) and still produce coherent AR outputs at inference? Is it fair to characterize this as the decoder learning a mapping from a coherent CTC prefix to the teacher's (conditionally valid) next-token prediction, and that this mapping task itself is what transfers the CTC's robustness, regardless of the global coherence of the resulting target sequence?\n\n- Figure 5 shows ID and OOD performance are optimized at opposite ends of the AR sampling probability spectrum. This suggests a simple 50/50 mix is a compromise. Did you consider a staged training approach? For example, (1) training only in the fast and robust CTC-driven mode (p=0.0) to get a robust model, and then (2) fine-tuning only in AR mode (p=1.0) to recover the last bit of in-distribution performance? This might be more effective or faster overall than mixing at every step.\n\n- You state the 2x training speedup comes from (i) faster training steps (due to non-AR pseudo-labeling) and (ii) faster convergence (requiring 50 vs. 75 epochs). Could you provide a rough breakdown of this? How much of the 2x gain is from (i) vs. (ii)? For instance, how does the wall-clock time of a 75-epoch USR 2.0 run compare to the 75-epoch USR run?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The paper targets a clear and significant problem: the slowness and unreliability of AR pseudo-labeling in a strong existing framework.\n- The \"CTC-driven teacher forcing\" method is an elegant solution. It cleverly uses the CTC branch's speed and robustness to guide the attention decoder, effectively solving the speed and stability issues simultaneously.\n- The claims are not just supported by standard in-distribution (ID) benchmarks but by a comprehensive suite of OOD tests, including:\n   - Demonstration of clear superiority over USR on long sequences, where USR's performance collapses.\n   - Shows strong performance on data with additive noise.\n- The 2x speedup claim is validated with wall-clock time comparisons, showing the method is both faster per step and converges in fewer epochs .", "weaknesses": "- The ablations show that the 50% mixed sampling is a compromise . The pure CTC-driven mode is best for OOD robustness, while the pure AR mode is best for ID accuracy. The final method balances these, but it's a trade-off, not a solution that maximizes both simultaneously.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)