Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
Authors: Bowen Shi
Wei-Ning Hsu
Kushal Lakhotia
Abdelrahman Mohamed
Venue: ICLR 2022 Poster (2022)
Keywords: audio-visual speech recognition
lip reading
speech recognition
self-supervised learning

Abstract: Video recordings of speech contain correlated audio and visual information, providing a strong signal for speech representation learning from the speaker’s lip movements and the produced sound. We introduce Audio-Visual Hidden Unit BERT (AV-HuBERT), a self-supervised representation learning framework for audio-visual speech, which masks multi-stream video input and predicts automatically discovered and iteratively refined multimodal hidden units. AV-HuBERT learns powerful audio-visual speech representation benefiting both lip-reading and automatic speech recognition. On the largest public lip-reading benchmark LRS3 (433 hours), AV-HuBERT achieves 32.5% WER with only 30 hours of labeled data, outperforming the former state-of-the-art approach (33.6%) trained with a thousand times more transcribed video data (31K hours) (Makino et al., 2019). The lip-reading WER is further reduced to 26.9% when using all 433 hours of labeled data from LRS3 and combined with self-training. Using our audio-visual representation on the same benchmark for audio-only speech recognition leads to a 40% relative WER reduction over the state-of-the-art performance (1.3% vs 2.3%). Our code and models are available at https://github.com/facebookresearch/av_hubert.

Published as a conference paper at ICLR 2022
LEARNING AUDIO-VISUUS SPEECH REPRESENTATION BY MASKED MULTIMODAL CLUSTER PREDICTION
Bowen Shi
Wei-Ning Hsu
Kushal Lakhotia²
Abdelrahman Mohamed
Toyota Technological Institute at Chicago
Meta AI
bshi@ttic.en
{wnhsu, kushall, @eddooff. com
ABSTRACT
Video recordings of speech contain correlated audio and visual information, pro- viding strong signal for speech representation learning from the speaker's lip movements and the produced sound. We introduce Audio- Visual Hidden Unit BERT (AV-HuBERT), self-supervised representation learning framework for audio-visual speech, which masks multi-stream video input and predicts automat- ically discovered and iteratively refined multimodal hidden units. AV-HuBERT learns powerful audio-visual speech representation benefiting both lip-reading and automatic speech recognition. On the largest public lip-reading benchmark LRS3 (433 hours), AV-HuBERT achieves 32.5% WER with only 30 hours of labeled data, outperforming the former sfate-of-the-ar approach (33.6%) trained with a thousand times more transcribed video data (31K hours) (Makino et al. 2019). The lip-reading WER is further reduced to 26.9% when using all 433 hours labeled data from LRS3 and combined with self-training. Using audio-visual representation on the same benchmark for audio-only speech recognition leads 40% relative WER reduction vs 2.3%) Our code and models are available at https://gitte com/ Facebookresearct//ar hubert
the state-of-the-art performance (1.3%
INTRODUCTION
Human perception of speech intrinsically multimodal, involving audition and vision. The speech production is accompanied by the movement of lips and teeth, which can be visually interpreted to understand speech. Visual cues of speech not only play an essential role in language learning for pre-lingual children (Meltzoff & Moore 1977, Davies al. 2008), but also improve speech understanding in noisy environment (Sumby & Pollack 1954) and provide patients of speech im- pairment with means of communication. Furthermore, perceptual studies (McGurk & MacDonald 1976) have shown that such visual cues can alter the perceived sound.
For machine learning models, the tight coupling between audio and visual lip movement informa- tion emerges as a natural source for supervision to learn speech representations, which has not been extensively utilized yet in the self-supervised speech representation learning literature. Recent
cessful representation learning frameworks for speech (c.g., APC (Chung et al. 2019). CPC (Oord al. 2018: Kharitonov al.) 2021). wav2vec 2.0 (Baevski et al. 20200] Hsu et al. 2021b), De- CoAR2.0 (Ling & Liu (200)) HuBERT (Hsu et al. 2021c a)) are mostly built entirely on audio. The fundamental research question addressed in this paper is whether a supervupe audio-visual speech representation learned from the lip movement information, alongside the audio signal in video recordings, captures cross-modal correlations and improves downstream performance for sual speech recognition (i.e., lip reading) and automatic speech recognition (ASR) tasks. Existing ML models for lip-reading rely heavily on text transcriptions to achieve an acceptable level of accu- racy. The state-of-the-au lip-reading model (Makino et al. 2019) requires 3IK hours of transcribed video data for training. Such large amounts of labeled data are expensive and hard to obtain for most of the world's 's 7,000 languages. The benefit from robust visual speech representation leamn ing framework goes beyond lip-reading. Additionally, it can benefit a vast range of applications,
Work done Meta AT

---

Published as a conference paper at ICLR 2022
including but not limited to keyword spotting in sign language (Albanie et al. 2020). speech en- hancement (Xu et al. 2020) and talking face generation (Chen et 2018).
In this paper, we present Audio-Visual Hidden Unit BERT (AV-HuBERT), multimodal self- supervised speech representation learning framework. It encodes masked audio and image se- quences into audio-visual features via hybrid ResNet-transformer architecture predict the pre- determined sequence of discrete cluster assignments. The target cluster assignments are initially generated from signal processing- ing the features learned by the audio-visual encoder via k-means clustering. AV-HuBERT simul- taneously captures linguistic and phonetic information for unmasked regions from both the lip- movement and audio streams into its latent representations, then encodes their long-range temporal relationships to solve the masked-pre preedcttreeiition task.
acoustic features (e.g.. MFCC) and iteratively refined us-
The contextualized representations learned by AV-HuBERT show excellent transferability to the lip- reading task, where only the visual modality is available. Pre-training on audio and visual input streams led to substantially better results than only visual input. In the low-resource setup using only 30 hours of labeled data from LRS3 (Afouras et al. 2018b), our model achieves a lip-reading WER of 32.5%, outperforming the previous state-of-the-art model (33.6%) trained on 3 31,000 hours of transcribed videos (Makino et al.) 2019). Using the complete 433 hours from LRS3 further reduces WER to 28.6%. We further show AV-HuBERT and self- self-training are complementary each other: combining both sets a new lip-reading WER record of 26.9%. In addition, we show that the multimodal clusters derived from AV-HuBERT can be used to pre-train HuBERT model for audio-based speech recognition, outperforming the previous state-of-the-art model (2.3%) and the unimodal HuBERT pre-trained on audio clusters (1.5%) by a large margin (1.3%).
2
RELATED WORK
The strong correlation between video modalities provides an effective means for self-supervised representation learning on videos, which has been explored in many prior works and is still active research area. This work draws inspiration from two lines of previous research:
Multimodal general video representation focuses on learning self-supervised audio-visual repre- sentations of general videos to solve high-level semantic tasks, c.g., action recognition and audio event detection (Arandjelović & Zisserman 2017 Bruno et al., 2018 Morgado et al.) 2021 Chen
et al. 2020 Lee et al. (021). Owens& & Efros (2018) learns a multimodal network to predict whether the audio and visual streams of a video are temporally synchronized while |Pam al. (2019) ap- plies cyclic translation between different modalities. PPiergiovanni et al. 2020) learns the visual representation through a multi-tasking framework that incorporates a series pretext tasks such reconstruction and temporal ordering prediction. Sharing our work's 's inspiration of DeepClus- tering (Caron et al. 2018). XDC (Alwassel et al.] (2020) and AV-BERT (Chan et al. 2021)) learn cross-modal representations through predicting cross-modal or cluster assignments. In contrast to XDC, AV-HuBERT is trained with a BERT--ike masked prediction loss, which forces the model to learn the structure within the multimodal input and was shown in Hsu et al. (2021c) to be more resilient to bad cluster assignments compared to unmasked cluster prediction. On the other hand, AV-BERT focuses on learning utterance-level multimodal environment embeddings that serves as the global context for ASR, while our objective is to learn frame-level audio-visun speech represen- tations and pre-train a model that can be fine-tuned for downstream tasks with either modality.
Semi- and self-supervised audio-visual speech representation learning focuses on improving lip-reading with untranscribed audio-visual speech data. To solve isolated visual word recognition, Chung et al. (0020) learns visual embeddings using a contrastive loss based on audio-visune synchro- nization. Ma et al. (2021a) learns visual speech representations by minimizing the distance between its latent features and off-the-shel audio embeddings. Using an external supervised ASR to tran- scribe unlabeled audio, Afouras et al. (2020) trains their lip-reading model on the augmented labeled and pseudo-labela data. Unlike Ma et al. (02 and Afouras al. (2020), model is trained from scratch and encouraged to learn contextualized representations with a masked prediction task. Moreover, method does not rely on any external pre-trained models.
2

---

Published as a conference paper at ICLR 2022
3 METHOD
3.1 PRELIMINARY: AUDIO HUBERT
Our research builds on Audio HuBERT (Hsu et al. (021a) which is a self-supervised learning frame- work for speech and audio. It alternates between steps: feature clustering and masked predic- tion. In the first step, a discrete latent variable model (e.g., k-means) is applied to a sequence of acoustic frames AI.T producing sequence of frame-level assignments zi.r. Clusters of signal- processing-basee acoustic features, e.g., Mel-frequency cepstral coefficients (MFCC), exhibit non- trivial correlations with the inherent acoustic units of speech inputs. Using (A1:T zi:r) pairs, the second step learns new feature representations by minimizing masked prediction loss, similar to masked language modeling in BERT (Devlin al. 2019). The pressure predict cluster assign- ments of masked audio regions forces the model to learn good local acoustic representations for unmasked regions and long-range temporal dependencies between latent features. Repeating these two steps improves the cluster quality and consequently the quality of the learned representations.
3.2 SINGLE-MODAL & CROSS-MODAL VISUAL HUBERT
The most naïve way to extend HuBERT the visual domain is by generating targets using visual features. Formally, given an image sequence [1T we first cluster the image features into a sequence of discrete units zi.r via k-means: 2 = k-means(G(I.)) E {1,2.... .... V}. where G' is visual feature extractor and V is the codebook size. The cluster assign- ments žir serve as the prediction targets of the model. Initially, G can be an engineered image feature extractor such Histogram of Oriented Gradients (HoG), analogous to MFCC in audio HuBERT. The intermediate layers of the HuBERT model are used as G in later iterations.
Single-modal Visual HuBERT:
To perform the masked prediction task, the model first encodes I using ResNet into an in- termediate visual feature sequence fi.7. which is then corrupted into firr via a binary mask M. Specifically, A7 E M, f/' is replaced with learned masked embedding. We adopt the same strategy in HuBERT to generate span masks. The masked visual features fü.r are encoded into a sequence of contextualized features C1.7' via transformer encoder followed by a linear projection layer. The loss is computed over the masked regions and optionally over unmasked ones (when a 2 0):
P: Softmax We: + bb,15ts7
L =
Σ log))(2)) - a Σ Coggr(z))
(1)
EEM
K&M
Where (W& E RdxV b E V are parameters of the projection layer which maps features into logits predicting the cluster assignments.
The single-modal visual HuBERT aims to learn visual speech rep- resentation through gradually refined image features. However, does not employ the audio stream of the video. Presumably, audio features, e.g., MFCC or a pre-trained audio HuBERT model, cor- relate with phones better than vanilla image features (e.g., HoG) do. To this end, we train an audio encoder based on the aligned audio frame sequence A in parallel to the visual encoder. The iter- ative training alternates between the two encoders. In each iteration, an audio encoder E" is utilized to generate target cluster assignments z The visual encoder E is trained subsequently with (II:, (..r). The 2 also used train the next iteration of the audio encoder F. for refinement.
Cross-modal Visual HuBERT:
The eross-modality visual HuBERT can be seen as modeling visual inputs by distilling knowledge from the audio stream, where ž represents the audio-side knowledge. We hypothesize that the audio feature is more favorable speech representation learning than the visual feature, which is validated the Section E.. Critical for the lip-reading downstream task, the masked prediction objective used by HuBERT forces the model to capture temporal relationships, which facilitates prediction of homophemes, which are groups of sounds with identical visual shapes (c.g., 'p'-'b', "f"""", 'sh'-" "sh'""ch") that impossible distinguish using a single image frame.
3

---

Published as a conference paper at ICLR 2022
AULTIMODAL CLUSTER 155
C11
C15
C80
C80
C27
masked entropy)
CONTEXTUALIZE AURIO-VISUAL
FRAME
FRAME
Illustration of AV-HuBERT. Masked prediction losses are only computed for the three middle frames, because at least one modality is masked for those frames. See section A for its comparison between single-modal and cross-modal visual HuBERT.
Figure 1:
3.3 AUDIO-VISUAL HUBERT
Our primary model in this work is Audio- Visual HuBERT (AV-HuBERT), shown in figure) [ which is trained iteratively by alternating between feature clustering and masked prediction in similar way the Visual HuBERT but with four main improvements:
Audio-visual input: The AV-HuBERT model consumes both acoustic and image frames for the masked prediction training. which enables better modeling and distillation of the correlations be- tween the two modalities. Specifically, image sequences and acoustic features pass through their light-weight modality nodality-specific encoders to produce intermediate features, which are then fused and fed into a shared backbone transformer encoder to predict masked cluster assignments. The targets are generated from clustering audio features features extracted from the previous iteration of the AV-HuBERT model. When fine-tuned for lip-reading, we drop the audio input to work solely with the visual input. The input discrepancy addressed by modality dropout described next.
Modality dropout: andio-visual speech recognition models can relate audio input to lexical output more effortlessly than the visual input stream, as observed in the literature (Afouras al.) 2018a] Ma et al.] 2021b). This causes the audio modality to dominate model decisions. The problem is aggravated in our setting because the target cluster assignments are initially generated from acoustic the audio stream in joint model, only use linear layer to encode acoustic input to force the audio encoder to learn simple features.
features. To prevent the model's over-reliance
Additionally, before fusing audio and visual inputs into the backbone transformer encoder, dropout applied to mask the full features one modality: we refer to it as modality dropout. With a probability Pm: both modalities is selected with a probability of pa. Formally, given the encoded audio and visual feature sequence fürr and fi equation 2 shows feature fusion equipped with modality dropout:
used as input. When only modality is used, the audio stream
concat (f) f()
with with (1 Pm Pmm)P
concat (ff 0)
concat(O,, (0,
with (1 11-Pan (1-P))(1 Pa)
where concat denotes channel-wise concatenation. Note that modality drop out is applied at the sequence level instead of at the frame-level, which effectively tasks AV-HuBERT to perform masked prediction with visual-only, audio-only, or audio-visual input. Modality dropout prevents the model from ignoring video input and encourages the model to produce the prediction regardless of what modalities are used as input. Furthermore, since the fine-tuning and inference phases use the visual stream alone (no audio input), this modality dropout mechanism bridges the gap between pre- training (multimodal) and fine- time--uningiiffrenne
A similar dropout mechanism is used in prior work (Zhang al. 2019a; Makino et al., 2019 Neverova et al., 2014: Abdelaziz jet al., (220)) to increase the robustness in multi-modal settings. We verify the modality dropout effectiveness in Section
4

---

Published as a conference paper at ICLR 2022
Audio-visual clustering: One benefit of pre-training both modalities is the ability to generate multimodal cluster assignments that serve as target labels for the masked prediction task of the next iteration. In contrast to the Cross- modall Visual HuBERT where targets generated from audio-based features or a prior Audio HuBERT model, the targets for AV-HuBERT are naturally multimodal after the first iteration. Lip movement sequences provide complementary information to the audio stream. Combining both modalities produces cluster assignments of higher quality for AV-HuBERT, as shown Section E.1
Masking by substitution: We propose a novel masking strategy for AV-HuBERT that masks seg- ments in the visual stream by substituting them with random segments from the same video. More formally, given an input video Firr an imposter video I and a mask consisting of intervals M (ss fi)ssssn we corrupted Fer into Iir by setting:
(3) where is a sampled integer offset from the interval [0,T, 44 + s:]. Now, to solve the task, the model needs to first identify the fake frames and then infer the labels belonging to the original frames. Since the "filled-in" segments are from real video segments and temporally smooth, the fake segment detection sub-task becomes less trivial compared when using vanilla masking or substitution with non-consecutive frames. We show an ablation confirming the advantage of the proposed masking strategy in Section [
I = n. Pcpp V1 sisn
The audio and visual segments are masked independently using two different masking probabilities m, and my. We hypothesize that the difficulty of the masked prediction task differs for each modal- ity: inferring the masked targets given the audio stream is more straightforward than using the lip movement stream. Setting a high masking probability for acoustic frames is essential to help the whole model capture the language characteristics. On the contrary, setting high masking proba- bility for the visual input provide the model with more imposter segments than the original ones, hurting its ability learn meaningful features (studied in Section [ of the appendix). Given the output probability pırr and target cluster assignments Z1:7, the AV-HuBERT pre-training loss is:
= Σ
-
Σ REM"UM"
log
BEM"UM"
where M and M denotes the frames that masked for the audio and the visual stream. is hyperparameter weighting the contribution of the unmasked regions in the overall objective.
3.4 FINE-TUNING
The proposed pre-training approaches can be fine-tuned for visual speech recognition using any sequence classification loss. In this paper, we focus on fine-tuning with the connectionist temporal classification (CTC) (Graves et al., 2006) and attention-based sequence-to-sequeece cross-entropy loss (Bahdanau et al. 2016) (S2S for brevity), which are the most popular choices. Assume that the feature sequence output of our pre-trained model is e1.7 and the ground-truth transcription is W1, W2. .... Wg. For CTC, a projection layer is used to map the visual feature sequence into the output probabilities: p: = Softmax /W/e, + b(t,, where W E Rdx(U++1), b E RU+1 and U the output vocabulary size (+1: plus blank symbol). The model is trained with CTC loss: Lote = - log > (w) p(T(e1:7), where B maps an alignment sequence π to w. For S2S, tranformer decoder is appended to the pre-trained encoder to autoregressively decode the feature sequence e1:7 into target probabilities p(w, p(w/(W1a-1, e.:T). The whole model is trained with cross entropy loss: T,22- Et = log p(w: (W//W/-111 e1.T).
4 EXPERIMENT
4.1
SETUP
We conduct experiments two datasets: LRS3 (Afouras al.) 2018b) with 433 hours of tran- scribed English videos and VoxCeleb2 (Chung et al.) 2018) with 2442 hours of unlabeled mul- tilingual videos. We only use the English portion of VoxCeleb2, which amounts 1,326 hours
5

---

Published as a conference paper at ICLR 2022
of content. The inputs to our backbone model are lip Regions-Of-Interes (ROIs) for the visual stream and log filterbank energy feature for the audio stream. The image encoder is a modified ResNet- 18, which has been used in prior work (Ma et al.] 2021b Martinez et al. 2020 Stafylakis & TZimiropoulos/ 2017). The audio encoder is simply a linear projection layer. We consider two model configurations: BASE with 12 transformer blocks and LARGE with 24 transformer blocks. For BASE and LARGE, the embedding dimension/feeddforeee transformer block are 768/3072/12 and 1024/4096/16 respectively. The number of parameters in BASE and LARGE are 103M and 325M respectively. in equation
dimension/attention heads in each
is set to O.
The model uses five iterations of feature clustering and masked prediction during pre-training See Section B.4 E for details on clustering. For fine-tuning, we use phone targets for the CTC loss and unigram-bases subword units (Kudo) 2018 for the S2S loss. For decoding the CTC-trained model, we use a 4-gram language model trained on LRS3 training text. For the S2S fine-tuned model, we rely only on its own decoder module to incorporate language information, with no external language model employed during inference. To further show the complementary relationship between AV- HuBERT and existing approaches of using unlabeled data, we also experiment on combining AV- HuBERT with self- self-training. Specifically, we generate pseudo labels for unlabeled data using a fine- tuned HuBERT, and fine-tune the pre- trained AV-HuBERT model with the combination of pseudo- labeled videos and original labeled videos. Note that no additional data is used when combined with self-training. More details about the used datasets, data pre-processing, and model training are in Section B
4.2
MAIN RESULT
Table T compares the performance of our AV-HuBERT pre-training approach to previously pub- lished supervised, semi-supervised, and supe-super lip-reading systems using different amounts of labeled and unlabeled data. Since the CTC and S2S fine-tuning approaches have similar trends, only S2S results are shown in table Complete results of CTC fine-tuning are in Table C.4
Using 1,759 hours unlabeled data for pre-training and only 30 hours of labeled data for fine-tuning, HU- outperforms all the prior lip-reading models, including the model in (Makino ct al. 2019) which is trained with 1000 times more labeled data. Fine-tuning with the whole training set of LRS3 further reduces WER. Combining our method and self-training achieves new SOTA result with only 7% of the data used for training the model in Makino et al. (2019). Furthermore, it shows that HuBTTB and self-training are complementary to each other. Note the overall gain is attributed mainly to AV-HuBERT as self-training alone leads to much worse performance (> 50% WER). More details can be found section C.3 Many prior models pre-train their visual front-end, e.g., ResNet- 18. using word-level annotated lip-reading videos, which is costly to collect since they require word boundary information. In contrast to these models, our models are fully pre-trained from seratch using the proposed approach.
Compared to the sumi-super approach JD (Afouras et al. 2020). which transcribed 334 hours of the English data in VoxCeleb2 using pre-trained ASR system our best model achieves 29% lower absolute WER benefiting from VoxCeleb2 for pre-training. Even when limiting our model the LRS3 data for pre-training and fine-tuning, our model surpasses their semi-supervised system by 18% Compared to LiRA et 2021a), a recently proposed self-supervised model for lip-reading, AV-HuBERT-BASE provides 17.5% lower absolute WER on average for low-resource and high-resource settings. The implementation details of LiRA are provided in Section B.5
With the same network architecture, our pre-training approach significantly reduces WER com- pared to training from scratch, in both low-resource (92.3% - 32.5%. LARGE) and high-resource (62.3% + 28.6%, LARGE) settings. qualitative view of the improvement can be found in Sec- tion F Additionally, we notice that our AV-HuBERT pre-training helps under a fully supervised set- ting. Using the LRS3 data only (433 hours), pre-training followed by fine-tuning (41.6%, LARGE) outperforms training a lip-reading model from scratch (62.3%, LARGE) to predict the output text.
'The prior work in (Ma al. 2021b) uses outdated version of LRS3 (before 2018) with speaker overlap
in training and test data, which is no longer publicly available. Its hest results the current version are included & C.4. For comparison, we simulate the old closed-speaker setting in Section C
table
2The gap data amount (334hr vs 1,759hr) is due to the different parameters and ASR model used for
filtering -nnglihlish data in VoxCeleb2.

---

Published as a conference paper at ICLR 2022
Table I: WER (%) of our models and prior work on the LRS3 dataset. tWe re-implemented Ma
et al. 2021a with the same architecture since the author source code was not provided.
Labeled iso (hrs)
Labeled utt (hrs)
Unlabeled data (hrs)
Method
Backbone
Criterion
WER (%)
Supervised
Afouras 2020
CNN
CTC
433
68.8
Zhang 20T9b
CNN
S2S
157
698
60.1
Alouras
2018a
Transformer
S2S
157
1,362
58.9
Xii 20201
RNN
S2S
157
433
57.8
Shilligffodd al.
2019
RNN
CTC
3,886
55.1
Ma et al
202
lb
Conformer
CTC+S2S
433
46.9
Ma et al.
20
Conformer
CTC+S2S
157
433
43.3
Makino ct dl.
2019
RNN
Transducer
31,000
33.6
Semii-Sppervised & Self-Superviie
Afouras et al.
2020
CNN
CTC
157
433
334
59.8
30
433
71.9
Ma et al.
2021a
Transformer BASE
S2S
433
1,759
49.6
Proposed SSple-Supervised Self- Self-Supervised Semi-Supervised]
30
94.3
30
433
$1.8
30
1,759
46.1
Transformer BASS
S2S
433
60.3
433
433
44.0
AV-HuBERT
433
1,759
34.8
30
92.3
30
433
44.8
30
1,759
32.5
Transformer-L.Ae
S2S
433
62.3
433
433
41.6
433
1,759
28.6
30
1,759
28.6
AV-HuBERT + Self-Trrining
Transformer-LRRGe
$28
433
1,759
26.9
AV-HuBERT, pre-trained video-audio pairs, learns better fine-grained visual representation than the scratch model trained on video-text pairs. The benefits of AV-HuBERT pre-training in various labeled data setups can be found in Section C.I
4.3 AV-HUBERT VS. VISUAL HUBERT
We compare AV-HuBERT against suite of alternatives, including the Single-modal and Cross- modal Visual HuBERT Table All the models are BASE pretrained on 433 hours of unlabeled data and fine-tuned on 30 hours of labeled data. For this comparison, to its computational efficiency and given their similarity results trends to S2S.
CTC fine-tuning due
Table 2: Fine-tuning performance (in WER, %) AV-HuBERT and visual HuBERT on different target labels. Init: feature in the initial iteration, sub: feature in subsequent iterations. AV: AV- HuBERT, V: Visual-HuBERT, A: Audio-HuBERT.
Bar-
Fan2
Iteration
AMFCC
Model/init-ssub
2
WMFCC A
WUFCC A
AV/MFCC-AV
71.5
63.6
60.9
58.8
58.2
AV/MFCC-A
71.5
64.3
63.5
V/MFCC
75.4
69.4
69.1
RUMFCC
AWVFCCC
V/MFCC-V
75.4
72.6
72.3
V/HoG-VV
80.3
80.1
WMRFD k
WUROD A
As shown in table 2 target cluster assignments driven from a single modality, either audio or visual, do not provide much WER reduction beyond the second iteration. Training the AV-HuBERT model

---

Published as a conference paper at ICLR 2022
using targets driven from audio-visual features keeps improving for more iterations and achieves better final performance. As mentioned in Section 3 visual information is complementary to audio input; hence, using both produces higher quality target clusters. Mcasuring the quality of different target labels using cluster quality metrics such as purity and NMI shows the same trend observed from lip-reading WER (See appendix [.
Fixing target labels used for the masked- preskcddreeeittion pre-training, AV-HuBERT (AV/MFCC->A) out- performs the Cross-modal Visual HuBERT (V/MFCC+A) by a large margin. AV-HuBERT effec- tively transfers knowledge from audio into the visual encoder and the backbone transformer model benefit visual-only fine-tuning and inference. In contrast to AV-HuBERT, iterative pre-training brings much smaller gains to single-modalin visual HuBERT ("V/MFCC+VV, "V/HoG-V"))
Starting with audio features is critical for learning effective target labels for the masked-predictiee task. Phonetic information, which is crucial for lip-reading, is primarily present in the audio stream. All the models considered so far are based on audio feature MFCC clustering in their initial iteration. As is shown in the "V/HoG-VV" row, using hand-engineered visual features provides a lousy starting point for iterative learning. Visual features such as HoG mainly incorporate low-level visual cues such edges and luminance, which is irrelevant to the downstream recognition task. Using features of a higher correlation with phonetic units are more likely to benefit the final lip-reading model. Indeed, clustering MFCC features show much higher purity (30.3%) than HoG clusters (16.4%) if one considers phone labels as the target units, as shown in Table E.T
4.4 MULTILINGUAL VS. MONOLINGUAL
Since the correlation between lip movements and the produced sound is governed by the vocal apparatus that is language-agnostic the proposed AV-HuBERT can utilize multi-lingual data for such learning. This would be particularly beneficial for low-resource languages. Nevertheless, the masked language modeling aspect in AV-HuBERT is still language-dependent, implying that mixing other languages would increase the language domain discrepancy. To study how these two factors affect AV-HuBERT, we compare using monolingual English data only versus multilingual videos in the pre-training phase. In particular, we vary the amount of English data we use pre-training while the number of non-English utterances is fixed to 1. 16 hours in all the settings. For simplicity, we train AV-HuBERT (BASE) for one iteration with MFCC clusters and fine-tune with CTC using 30 hours of labeled data.
Table 3: WER (%) with different amounts of unlabeled English utterances in pre-training, Non-En data: 1,116 hours. Labeled data for fine-tuning: 30 hours.
Hours of En data for Pre- Training
0
100
400
800
1759
Pre-train on En only, WER (%)
84.1 I
77.8
68.9
67.9
59.9
Pre-train on En 1,116 hr of non-En, WER (%)
70.6
68.4
67.4
66.6
64.3
As is shown in table 3 using non-English data in pre-training significantly reduces WER when there are no very little English data in pre-training (< 100 hours). As we increase the amount of English data, the gain diminishes because the out-of-domain effect brought by non- non-English data outweighs the benefit of the overall increase in pre-training data. Using the whole English data only for pre-training is better than combining it with other languages (59.9% vs. 64.3%). Training with iterations leads to similar results (47.3% vs. 48.9%). This experiment highlights the importance of the domain match between pre-training and fine-tuning data. For zero/low-resoure settings, merging data from other languages in pre-training benefits the downstream task. When the unlabeled data from the target language is abundant, limiting the pre-training data to one language is beneficial.
4.5 ASR PERFORMANCE
The multimodal clusters produced by AV-HuBERT, which have higher quality than targets, can also benefit speech pre-training. To test hypothesis, we trained an audio- HuBERT, with only audio input during the masked beekeed-reeicttoon pre-training, for one iteration with cluster as- from scratch
signments driven from AV-HuBERT features. We also pre-trained
8

---

Published as a conference paper at ICLR 2022
using clusters driven from the MFCC features for three iterations. The two pre-trained models evaluated on a downstream ASR task.
Table shhows the performance of different models fine-tuned the ASR task. We only include the performance of S2S fine-tuning for our models as it consistently outperforms the CTC fine-tuning. An audio-HuBERT pre-trained using targets generated by AV-HuBERT features outperforms the vanilla audio-HuBERT in low-resource (30h) and high-resource settings (433h) fine-tuning settings different model architectures. With the same amount of labeled data, our best model (1.4%) outperforms the prior SOTA (2.3%) even without an external language model during inference.
Given that the AV-HuBERT model utilizes both modalities at its input, it can be fine-tuned, in prin- ciple, for the ASR downstream task. In practice, we notice pre-training an Mudio-HuBERT with audio-visual cluster leads to better ASR performance (3.8%) than pre-trained AV-HuBERT (4.6%), potentially due to its hyperparameters being selected based on lip reading rather than ASR. In fact, audio- HubiRT can be treated special case of AV-HuBERT with Pm 0, Pa
Table 4: ASR WER (%) of audio-HuBERT pre-trained with audio-onlyaaudoovvisall clusters and their comparison to prior work on the LRS3 dataset.
Labeled data (hrs)
Unlabeled data (hrs)
Method
Backbone
Criterion
LM
WER (%)
Supervised
Afouras et al.
201 8a
Transformer
S2S
1,362
8.3
Alouras al.
22188a 8018a
Transformer
CTC
1.362
8.9
Xu
2020
RNN
S2S
433
7.2
Ma ct al.
20216
Conformer
CTC+S2S
433
2.3
Self-Superiied
433
5.4
Transformer-Bas Base
$2S
30
1,759
5.0
Hsu al. (202
433
1,759
2.4
(A/MFC CC-AA
30
433
4.5
Transformer-Large
$2S
30
1,759
3.2
433
1,759
1.5
Proposed (SpeeSSupe))ssed)
30
433
4.9
Transformer- -Base
$2S
1,759
3.8
433
1,759
2.0
A/
433
4.2
Transformer-Large
S2S
30
1,759
2.9
433
1,759
1.3
5
CONCLUSION
We presented multiple pre-training models for visual speech recognition. Our AV-HuBERT model leverages the strong correlation between the audio and lip movement streams for sulf-supervised audio-visual speech representation learning. Our pre-training approaches iteratively alternate be- tween feature clustering and learning new features through HuBERT model consumes masked image and audio frames to predict target cluster assignments. The targets are initially generated from MFCC features and gradually refined through iterative train- ing. Experiments on visual speech recognition show that AV-HuBERT achieves SOTA using 433 hours of text transcriptions, two orders of magnitude less than the 31. 1,000 hours of labeled data used in the prior best approach. When using only one-thousandth of labeled data, the lip-reading perfor- mance outperforms the prior SOTA by more than 10% (relative). AV-HuBERT also improves the representation for the ASR downstream task. An audio-HuBERT model trained with targets gener- ated by an AV-HuBERT model shows superior performance, achieving the SOTA in the audio-bann based speech recognition in the LRS3 dataset. As future work, AV-HuBERT can be applied for multilin- gual lia-reading in low-resource languages. Additionally, our approach can be extended to other applications of visual speech representation, such speech enhancement and generation.
masked--rredcction loss. The AV-

---

Published as a conference paper at ICLR 2022
ETHICAL STATEMENT
All the data used in this paper are publicly available and are used under the following three licenses: the TED terms of use, the Creative Commons BY-NC-ND 4.0 license and Creative Commons Attri- bution 4.0 International License. Through spot-checking, we find the datasets are gender balanced and cover wide range races and ages. However, the distribution of speakers in the data may not be representative of the global human population. Please be cautious of unintended societal, gender, racial and other biases caused by the fact. To maintain anonymity, only the mouth area of a speaker is visualized wherever used in the paper. The proposed method can be applied in several areas in- cluding security and crime investigations. However can also be used for malicious purposes such surveillance and wiretapping. We are committed to distributing our code and model carefully, with special attention to any potential security and privacy concerns.
REPRODUCIBILITY STATEMENT
Our code and models are publicly available. In the meantime, we include as many implementation details as we can in the paper.
REFERENCES
The CMU pronouncing dictionary.
http://ww..sest cs. edu. cgi-bin./ bbir/
cmudict
Ahmed Hussen Abdelaziz, Barry-John Theobald, Paul Dixon, Reinhard Knothe, Nicholas Apos- toloff, and Sachin Kajareker. Modality dropout for improved performance-driven talking faces. In ICMI, 2020.
Triantafyllos Afouras, Joon Son Chung, A. Senior, Oriol Vinyals, and Andrew Zisserman. Deep audio-visual speech recognition. IEEE transactions on pattern analysis and machine intelligence, 2018a.
Triantafyllos Afouras, Joon Son Chung, and Andrew Zisserman. LRS3-TED: a large-scale dataset for visual speech recognition, 2018b. arXiv: arXiv:1889.00444 1809.00496..
Triantafyllos Afouras, Joon Son Chung, and Andrew Zisserman. ASR all you need: Cross-modal distillation for lip reading. In ICASSP, 2020.
Samuel Albanie, Gül Varol, Liliane Momeni, Triantafyllos Afouras, Joon Son Chung, Neil Fox, and Andrew Zisserman. BSL-IK: Scaling up co-articulated sign language recognition using mouthing cues. In ECCV, 2020.
Humam Alwassel, Dhruv Mahajan, Bruno Korbar, Lorenzo Torresani, Bernard Ghanem, and Du Tran. Self-supervised learning by cross-modal audio-video clustering. In NeurIPS, 2020.
Relja Arandjelović and Andrew Zisserman. Look, listen and learn. In ICCV, 2017.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arX1v:1607.06450, 2016.
Alexei Baevski, Henry Zhou, Abdel rahman Mohamed, and Michael Auli. wav2vec 2.0: A frame- work for self-supervised learning of speech representations. In NewrIPS, 2020.
Dzmitry Bahdanau, Jan Chorowski, Dmitriy Serdyuk, Philemon Brakel, and Yoshua Bengio. End- to-end attention-based large vocabulary speech recognition. In ICASSP. 2016.
Korbar Bruno, Tran Du, and Torresani Lorenzo. Cooperative learning of audio and video models from self-supervised synchronization. In NeurlPS, 2018.
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsu- pervised learning of visual features. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 132-149, 2018
10

---

Published as a conference paper at ICLR 2022
David M. Chan, Shalini Ghosh, Debmalya Chakrabarty, and Bjöm Hoffmeister. Multi-modal pre- training for automated speech recognition. arXiv preprint arX:0:2110.09890, 2021.
Lele Chen, Zhiheng Li, Ross K. Maddox, Zhiyao Duan, and Chenliang Xu. Lip movements gener- ation at glance. In ECCV, 2018.
Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation learning. In ECCV, 2020.
Joon Son Chung, Arsha Nagrani, and Andrew Zisserman. Voxceleb2: INTERSPEECH, 2018.
Deep speaker recognition. In
Soo-Whan Chung, Hong Kang, and Joon Son Chung. Seeing voices and hearing voices: learning discriminative embeddings using cross-modal self-supervision. In Interspeech, 2020.
Yu-An Chung, Wei-Ning Hsu, Hao Tang, and James Glass. An unsupervised autoregressive model for speech representation learning. arXiv preprint arXiv:190..03244 2019.
Rebecca Davies, Evan Kidd, and Karen Lander. Investigating the psycholinguistic correlates of speechreading in preschool age children. International journal of language and communication disorders, 44:164 64-74, 06 2008.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In NAACL, 2019.
Angela Fan, Edouard Grave, and Armand Joulin. Reducing transformer depth demand with structured dropout. ICLR, 2020.
Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. Connectionist tempo- ral classification: labelling unsegmented sequence data with recurrent neural networks. In ICML, 2006.
Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. Hubert: Sele-supervised speech representation learning by masked prediction of hidden units. arXiv preprint wrx.iv:2106.7440 2021a.
Wei-Ning Hsu, Anuroop Sriram, Alexei Baevski, Tatiana Likhomanenko, Qiantong Xu, Vineel Pratap, Jacob Kahn, Ann Lee, Ronan Collobert, Gabriel Synnaeve, et al. Robust wav2vec 2.0: Analyzing domain shift in self-supervised pre-training. arXiv preprint arXiv:2 2021b.
Wei-Ning Hsu, Yao-Hung Hubert Tsai, Benjamin Bolte, Ruslan Salakhutdinov, and Abdelrahman Mohamed. Hubert: How much can bad teacher benefit ast pre-training? In ICASSP. 2021c
Eugene Kharitonov, Morgane Rivière, Gabriel Synnaeve, Lior Wolf, Fimaan-E Mazaré, Matthijs Douze, and Emmanuel Dupoux. Data augmenting contrastive learning of speech rep- resentations in the time domain. In 2021 IEEE Spoken Language Technology Workshop (SLT), pp. 215-222. IEEE, 2021.
Davis E. King. Dlib-ml: A machine learning toolkit. Journal of Machine Learning Research, 10: 1755-1758, 2009.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. CoRR, abs/1412.6980, 2015.
Taku Kudo. Subword regularization: Improving neural network translation models with multiple subword candidates. In ACL, 2018.
Sangho Lee, Youngjae Yu, Gunhee Kim, Thomas Breuel, Jan Kautz, and Yale Song. Parameter Efficient Multimodal Transformers for Video Representation Learning. In ICLR, 2021.
Shaoshi Ling and Yuzong Liu. Decoar 2.0: Deep contextualized acoustic representations with vector quantization. arXiv preprint arXiv:20/2.06659, 2020.
Pingchuan Ma, Rodrigo Mira. Stavros Petridis, Bjöm Schuller, and Maja Pantic. LiRA: Learning visual speech representations from audio through self-supervision. In Interspeech, 2021a.

---

Published as a conference paper at ICLR 2022
Pingchuan Ma, Stavros Petridis, and Maja Pantic. End-to-end audio-visual speech recognition with conformers. In ICASSP, 2021b.
Takaki Makino, Hank Liao, Yannis Assael, Brendan Shillingford, Basilio Garcia, Otavio Braga, and Olivier Siohan. Recurrent neural network transducer for audio-visual speech recognition. In Interspeech, 2019.
Brais Martinez, Pingchuan Ma, Stavros Petridis, and Maja Pantic. Lipreading using temporal con- volutional networks. In ICASSP, 2020.
Harry McGurk and John MacDonald. Hearing lips and seeing voices. Nature, 264:746-748, 1976.
Andrew N. Meltzoff and M. Keith Moore. Imitation of facial and manual gestures by human neonates. Science, 198:75-78, 1977.
Pedro Morgado, Nuno Vasconcelos, and Ishan Misra. Audio-visual instance discrimination with cross-modal agreement. In CVPR, 2021.
Natalia Neverova, Christian Wolf, Graham Taylor, and Florian Nebout. ModDrop: adaptive multi- modal gesture recognition. IEEE Trunsuctions on Puttern Analysis and Machine Intelligence, 2014.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding, arXiv preprint arXiv:1807.03745. 2018.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng. David Grangier, and Michael Auli. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of NAACL-HLT 2019: Demonstrations, 2019.
Andrew Owens and Alexei A Efros. Audio-visual scene analysis with self-supervised multisensory features. In ECCV, 2018.
Hai Pham, Paul Liang. Thomas Manzini, Louis-Philippe Morency, and Barnabas Poczos. Found in translation: Learning robust joint representations by cyclic translations between modalities. In AAAI, 2019.
AJ Piergiovanni, Anelia Angelova, and Michael Ryoo. Evolving losses for unlabeled video repre- sentation learning. In CVPR, 2020.
Mireo Ravanelli, Jianyuan Zhong, Santiago Paseual, Pawel Swietojanski, João L. Monteiro, Jan Trmal, and Yoshua Bengio. Multi-task self-supervised learning for robust speech recognition. In ICASSP, 2020.
Brendan Shillingford, Yannis Assael, Matthew Hoffman, Thomas Paine, Cían Hughes, Utsav Prabhu, Hank Liao, Hasim Sak, Kanishka Rao, Lorrayne Bennett, Marie Mulville, Ben Cop- pin, Ben Laurie, Andrew Senior, and Nando Freitas. Large-scale visual speech recognition. In Interspeech, 2019.
Themos Stafylakis and Georgios Tzimiropoulos. Combining residual networks with Istms for lipreading. In Interspeech, 2017.
W. H. Sumby and Irwin Pollack. Visual contribution to speech intelligibility in noise. Journal of the Acoustical Society of America, 26:212-215, 1954.
Bo Xu, Cheng Lu, Yandong Guo, and Jacob Wang. Discriminative multi-modality speech recogni- tion. In CVPR, 2020.
Shiliang Zhang, Ming Lei, Bin Ma, and Lei Xie. Robust audio-visual speech recognition using bimodal dfsmm with multi-condition training and dropout regularization. ICASSP, 2019a.
Xingxuan Zhang, Feng Cheng, and Shilin Wang. Spatio-temporal fusion based convolutional se- quence learning for lip reading. ICCV, 2019h.
12

---

Published as a conference paper at ICLR 2022
A MODEL ILLUSTRATION
Figure A.1: Comparison between the proposed AV-HuBERT with single-modal and cross-modal
visual HuBERT
Viezal Cluster 10s
C11
C15
C80
C80
C27
loss entropy)
transformer
FRAME
FRAME
1
(a) Single-modal visual HuBERT
Audio Cluster 10s
C15
C80
C11
C15
C80
C80
C27
masked prediction tion loss (cross entropy)
masked prediction loss icross entropy)
transformer
transformer
FRAME
ResNet
3
(b) Cross-modal visual HuBERT
MULTIMODAL CLUSTER 105
C11
C80
C80
C27
masked predict ton loss (cross entropy)
CONTEXTUAL REPRESENTATIONS
audio-visual fusion
MASKED
ResNet
FFN
FRAME
2
(c) Audio-visual HuBERT (proposed)
13

---

Published as a conference paper at ICLR 2022
B DETAILED EXPERIMENTAL SETUP
B.I DATASETS
LRS3 (Afouras et al. 2018b) is the largest publicly available sentence-level lip reading dataset to date. It consists of over 400 hours of video, extracted from TED & TEDx talks in English from YouTube. In the original dataset, the training data is split into two partitions: pretrain (403 hours) and trainval (30 hours) Both parts transcribed at the sentence level and come from the same source as the test set. The pretrain set differs from trainval in that the duration of its video clips of much wider range and can be shorter or longer than a full sentence. In the low-resource setup, we only use trainval as the labeled data. As no official development set is provided, we randomly select 1,200 sequences from trainval as the validation set (about hour) for early stopping and hyperparameter tuning.
VoxCeleb2 (Chung al. 2018) is originally created for multilingual audio-visual speaker recogni- tion and it contains over 2,442 hours of utterances of over 6,000 speakers extracted from YouTube videos. The dataset naturally serves our purpose it does contain ground-trute transcriptions. The VoxCeleb2 has substantial domain discrepency to the LRS3 data as its utterances are from multi- ple languages and includes videos larger variety of domains including interviews, talks, excerpts under indoor and outdoor environments. In VoxCeleb2, by default we only use the English portion for pre-training. As no ground-trutt language label is given in the VoxCeleb2, we use a simple off-the-shelf character-basee
heuristic to choose the English samples. Specifically, model (Hsu et al.) aa) trained on Librispeech which achieves 1.9%/3.5% WER on clean/other test set. We run greedy decoding on the VoxCeleb2 and the proportion of valid English words to determine a target utterance is English or not. An utterance is only selected if the proportion of valid English words is higher than 60%. The total amount of unlabeled data after filtering is 1 1,326 hours.
ASR
B.2 DATA PREPROCESSING
For each video clip, we detect the 68 facial keypoints using dlib (King 2009) and align each frame to reference face frame via affine transformation. We crop 96 x 96 region-offinterest (ROI) centered on the mouth. Each image frame converted to grayscale. We randomly cropped 88 x 88 from the whole ROI and randomly flipped it horizontally with probablity of 0.5 during training. At test time, 88 x 88 ROI is center cropped and does not go through horizontal flipping. The preprocessing steps remain same as prior works in lip reading (Martinez al., 2020; Ma al. 2021b). For the associated audio, we extract the 26- 26-dimensional log filterbank energy feature at a stride of 10 ms from the raw waveform and use it as input to the model. As the image frames are sampled 25Hz, we stack the 4 neighboring acoustic frames to synchronize the two modalities.
B.3 AV-HUBERT MODEL ARCHITECTURE
In the modified ResNet- 18 (Ma et al.) 2021b Martinez et al. 2020; Stafylakis & Tzimiropoulos,
2017), the first convolutional layer substituted by 3D convolutional layer with kernel size x 7 7. The visual feature tensor is flattened into single-dimensional vector through a 2D average pooling layer in the end. We use one linear projection layer as the audio encoding module. The acoustic features are normalized by per-frame statistics before feeding into the network (Ba et al.) 2016). We use dropout of = 0.1 after the self-attention block within each transformer layer, and each transformer layer is dropped (Fan et al. 2020 rate of 0.1.
B.4 TRAINING AND INFERENCE
Pretraining Our models are implemented with fairseq (Ot et al. 2019). The whole network is randomly initialized before pre-training. In pre-training, the model is trained for five iterations in total. For the initial iteration, we generate the targets by running k-means clustering algorithm on 39-dimensional MFCC feature (13 coefficients with its first- and second-order derivatives) extracted from raw audio waveform. For the subsequent iterations, the feature from an intermediate layer of the model in the previous iteration is used for clustering. The layer index (one-based) used for clustering in iteration 1-4 are {9, 12, 12, 12). The number of features clustered to (100, 100,
14

---

Published as a conference paper at ICLR 2022
500, 1000, 2000} respectively for the 5 iterations. See section E.3 for analysis. To save training time, we always use the BASE model to generate clusters and LARGE model is only trained in the 5th iteration.
We set both pm and Pa to 0.5 for modality dropout at training time. To extract features for clustering, both modalities are used. We adopt the strategy used in wav2vec 2.0 (Baevski al., 2020) to generate masks, where p% of all frames are randomly selected as start and subsequent frames are masked. In iteration 1-4, we mask the fused features and set pll be 8/10 respectively as we observe such practice generates higher quality cluster assignments (see section E.2 In the last iteration, we set pll be 6/5 for video and 8/10 for audio (see section D)
We train the model with Adam (Kingma & Ba 2015). warming up the learning rate for the first 8% of updates to a peak of 0.002 and then linearly decay it. Videos are batched together to not exceed 1, 1,000 image frames (40 seconds) per GPU. Both BASE and LARGE models are updated for 400K and 600K steps at each iteration, respectively in 433h/1759h unlabeled settings. We train on 32 and 64 VI00-GPUs for BASE and LARGE. On average, each iteration takes ~ 2.0/3.0 days for BASE and ~ 2.4/3.6 days for LARGE in using 433h/1759h unlabeled data for pre-training.
Fine-tuning After pre-training, we fine-tune the AV-HuBERT model on labeled (video, text) pairs. The audio encoder is removed and its output is replaced by a zero-vector. In CTC fine-tuning, a ran- domly initialized projection layer is added on top of the transformer map features into phonemes. The lexicon is constructed with CMUDict (cmu). In fine-tuning with S2S, we use a 6-layer/9-layer transformer decoder on BASE and LARGE model to decode features into unigram-basse based subword units (Kudo 2018). The vocabulary size in CTC and S2S are 46 and 1000 respectively.
In CTC, the pre-trained model is updated from the initial iteration without any freezing. The model is fine-tuned for 30K/100K steps respectively in 30h/433h setting. In S2S, the pre-trained model (i.e., encoder) is frozen for the first N% updates N is 100 and 50 for 30h and 433h labeled setting respectively. The entire model is trained for 18K/45K steps in the 30h/433h setting. Both models are trained with Adam, with the learning rate being warmed up for the first P% of updates to a peak of 0.001 and linearly decayed. P is tuned among { 10, 30, 50}. All hyperparamters are tuned on the validation set.
Decoding For CTC, we use a grammm language model trained on text data in the LRS3 training set. The perplexity of the 4-gram LM on test set 110.5. No language model is used for S2S decoding. For CTC, we tune the beam width among (5, 10, 20, 50, 100, 155). the language model weight among (0, 1, 2, 4, 8) and word insertion penalty among (+-4, +22, +1,0). 03. For S2S, The beam width and length penalty are tuned among {5. 10, 20, 50} and {0, #13. The tuning is done on the validation set.
Self-Training We apply self-training LARGE AV-HTBERT. Specifically, the fine-tuned LARGE HuBERT model(A/MFCC visual data. In 30h/433h setting, the amount of data for fine-tuning A/MFCC+AV are 30h and 433h respectively. The pre-trained AV-HuBERT LARGE is fine-tuned with the pseudo-labeled videos and videos with ground-truth text labels (30h/433h). Note the data used here is exactly same with the case of using AV-HuBERT only.
Table 4 is used to assign pseudo- labels to the unlabeled audio-
B.5 LIRA IMPLEMENTATION
We re-implemented the LiRA (a et al. 2021a training objective in our framework, as there does not exist publicly available implementations and we aim to focus the comparison on the pre-training objective rather than the architectural difference. We use the same backbone architecture as the BASE AV-Hubert except the output layer being a linear project layer with an output dimension of 256. The 256-dimensional frame PASE+ feature is extracted from the audio with its official implementation in (Ravanelli et al. (020)). The original PASE+ feature is downsampled to 25Hz for synchronization with the visual stream. The pre-trained model is fine-tuned in both CTC and S2S. The optimizer and learning rate schedule in pre-training, hyperparameter search in fine-tuning and decoding remain the same as AV-HUBERT. Note with our implementation, the WER is reduced by 23% (94.3% + 71.9%) using LiRA when the percentage of labeled data is 6.9% (30h labeled, 433h in total), while Ma et al. (202 la) achieves ~ 10% improvement in a similar setting.
15

---

Published as a conference paper at ICLR 2022
C ADDITIONAL LIP-READING RESULTS
C.1 AMOUNT OF LABELED DATA
Table C.I shows the effect pre-training on different amount of labeled data for fine-tuning. We use 433 hours of unlabeled data (LRS3 only) and randomly selected 1, 10 and 100 hours of labeled data for fine-tuning. Overall pre-training brings large and consistent gains across different amount of labeled data. Specifically CTC-based fine-tuning outperforms S2S-based fine-tuning in low- resource settings (I-hour and 10-hour). The larger number of parameters as well as the lack language model for decoding makes S2S model more likely to overfit especially when the amount of fine-tuning data is small.
Table C.I: WER (%) in using different amount of labeled data for fine-tuning (BASE, 433 hours unlabeled)
WER (%)
Labeled (hrs)
Unlabeled (hrs)
Criterion
LM
w/o pretrain
w/ pretrain
433
CTC
4-gram
98.6
68.8
S2S
98.9
92.0
433
CTC
4-gram
90.8
57.6
S2S
97.6
63.1
100
433
CTC
4-grmm
77.8
54.2
S2S
84.3
48.1
C.2 PERFORMANCE ON SEEN SPEAKERS
The current LRS3 benchmark is under the open-speaker setting, where the speaker identities in training and test set do not overlap. To test the lip reading performance for a fixed set of speakers which the case for an early versions of LRS3 used before October 2018, we randomly choose 5 groups of utterances from the trainval partition of LRS3 test set and repeat experiments for each group independently. Each group contains 1322 utterance, which is of the same amount as the original test set. The model compare the AV-HUBERT' LARGE pre-trained with 1,759 unlabeled data. As is shown in table C.2, the average WER achieved by our model for seen speakers 18.0 + 0.5%, which is significantly lower than the WER for unseen speakers (30.5%) under the open-speaker setting.
Table C.2: WER (%) under closed-speaker setting for 5 randomly sampled test sets and their average
Test set (seen speakers)
AVG
5
WER (%)
17.4
18.6
18.5
17.5
18.3
18.0 0.5
C.3 PERFORMANCE OF SELF-TRAINING ONLY
Table C.3 shows the performance only applying self-training. The WER of self-training only model significantly higher than AV-HuBERT and self-trained AV-HuBERT, which suggests that the gain of the combined approach is primarily from AV-HuBERT.
C.4 FULL RESULTS WITH CTC FINE-TUNING
Table C.4 shows the full results on LRS3, which includes the CTC fine-tuning performance for all the models we implemented. In general, the conclusions we draw from S2S (e.g.. the benefits of pre-training approach in different settings, the improvement over LiRA) in section 4.2 holds for
CTC as well.
16

---

Published as a conference paper at ICLR 2022
Table C.3:
Comparison of WER (%) among model trained from scratch, self-training only, AV-
HuBERT only and self-trained HUBETT All models are Transformer-ARGGE.
Labeled (hrs)
Unlabeled (hrs)
Method
WER (%)
30
1,759
w/o pre-training Self-training
92.3
53.0
AV-HuBERT
32.5
AV-HuBERT + Self- Self-training
28.6
433
1,759
w/o pre-training
62.3
Self-training
51.7
AV-HuBERT
28.6
AV-HuBERT + Self Self-training
26.9
17

---

Published as a conference paper at ICLR 2022
Table C.4: WER (%) of our models and the comparison with prior works on LRS3-TED dataset. We re-implemented Ma al. (2021a) using the same model architecture approach to have more fair comparison.
Labeled iso (hrs)
Labeled utt (hrs)
Unlabeled data (hrs)
Method
Backbone
Criterion
WER (%)
Supervised
Afouras 2020
CNN
CTC
157
433
68.8
Zhang 20196
CNN
S2S
157
698
60.1
ATouras
2018a
Transformer
S2S
157
1,362
58.9
Xi
2020
RNN
$2S
157
433
57.8
Shillngford al.
2019
RNN
CTC
3,886
55.1
MB et al.
202 15
Conformer
CTC+S2S
4,3
46.9
Ma at al
20
Conformer
CTC+S2S
157
433
43.3
Makıno ct ti.
2019
RNN
Transducer
31,000
33.6
Semi-Superrised & Sele-Super
Afouras et al.
2020
CNN
CTC
157
433
334
59.8
30
433
72.8
CTC
433
1,759
58.4
Ma 2021a
Transformer BASE
30
433
71.9
S2S
433
1,759
49.6
Proposed Self-Superriised Self-Supervised Semi Semi- Superviipr)))e))
30
83.7
30
433
55.3
30
1,759
47.3
CTC
433
62.5
433
433
49.3
Transformer -BASE
433
1,759
43,0
30
94.3
30
433
51.8
30
1,759
46.1
82S
433
60.3
AV-HuBERT
433
433
44.0
433
1,759
34.8
30
92.2
30
433
48.4
30
1,759
40.7
CTC
433
61.9
433
433
44.3
Transforee-LARGE
433
1,759
38.6
30
92.3
30
433
44.8
30
1,759
32.5
$28
433
433
433
41.6
433
1,759
28.6
HV-HBEETT + Self-Training Transforme-LARRE
S2S
30
1,759
28.6
433
1,759
26.9
18

---

Published as a conference paper at ICLR 2022
D ABLATION STUDIES
The ablation studies in this section are done in the last iteration of the AV-HuBERT, pre-trained with 433 hours unlabeled data. The model is fine-tume tine-tuned with 30 hours labeled data using CTC.
Table D.1: Ablation study for hyper-parameteen The ablations are done in the last iteration of AV-
HUBERT. m the probability of acoustic/image frame being masked.
Masking
Modality Dropout
Loss
WER
Where
How
ma
17k
Pm
Pa
dev
test
Input
Sub (same, seg)
0.8
0.3
0.5
0.5
0.0
46.8
55.3
Sub (same, frm) Sub (diff, seg)
47.2
55.8
47.6
56.1
Learned Embedding
52.6
57.8
Gauss, Noise
52.4
57.9
Feature
Learned Embedding
55.2
58.2
Input
Sub (same, seg)
0.8
0.3
0.5
0.5
0.0
46.8
55.3
0.8
0.8
59.3
61.6
0.3
0.3
54.9
58.2
Input
Sub (same, seg)
0.8
0.3
0.5
0.5
0.0
46.8
55.3
1.0
n/a
55.2
57.0
Input
Sub (same, seg)
0.8
0.3
0.5
0.5
0.0
46.8
55.3
1.0
46.7
55.7
compare the proposed masking strategy against several alternatives. Feature masking applies span mask at feature level and leads to worst per- formance, which is due to the leakage of information to ResNet. Directly masking image sequence with random Gaussian noise or a learned embedding slightly improves the performance by pre- venting the prior issue. However, those artificial frames also corrupts the raw image sequence and enlarges the domain gap videos between pre-training and fine-tuning. On the other hand, our pro- posed method achieves better performance. Specifically, using segments from the ("Sub, (same, seg)") as the imposter leads to the best result compared to sampling from a different utterance ("Sub, (diff, seg)") sampling non-consecutive frames from the same utterance ("Sub, (same, frm)") The filled-in fake images are visually similar to the raw images and substitution with segment keeps the temporal smoothness making the replaced segment more realistic, which enforces the ResNet to encode more fine-grained visual details.
Masking Strategy In the first part of table D.I
utterance
Masking probability We set two different masking probabilities for audio and visual stream in the last iteration. The probabilities of an acoustic frame and image frame being masked are 0.8 and 0.3 respectively. As is shown in the second part of table [D.]] setting masks for audio and visual stream independently is essential because the optimal masking probability for audio and visual stream different. Audio encoder tends to deteriorate into a simple acoustic feature extractor when mask length is small. On the other hand, long visual mask will lead to the model lacking context to distinguish between fake and real images.
Modality dropout The third part of table D.I compares the model performance with and without modality dropout. Randomly dropping audio sequence prevents the model from over-relying on audio for masked prediction and helps the visual representation learning.
Where to compute prediction loss In the last part of table D.I we compare the choice masked prediction vs. prediction. The loss weight on unmasked region does have large impact on the fine-tuning performance. This is different from the findings in Audio HuBERT Hsu et al. (202 Ia), where masked prediction leads much better performance. Given image frames as input, the pre- diction of cluster assignments. which are mostly determined by the accompanied audio stream, helps encode phonetic information into the visual representation. The task much less trivial than
TTo avoid using original frames for substitution, in equation is selected from |0. 2a; tg] U [4 T s.] if the imposter is from the same sequence.
19

---

Published as a conference paper at ICLR 2022
single-modal model (audio-HuBERT), where setting non-zero weight on unmasked prediction can easily make the model deteriorate into an acoustic feature extractor. In addition, the high quality of targets in the last iteration also makes such prediction more helpful.
20

---

Published as a conference paper at ICLR 2022
E ANALYSIS ON CLUSTERING
E. MEASURING CLUSTERING QUALITY
For analysis, use frame-level phonetic labels as the ground-truth and match its correlation be- tween cluster assignments. The phonetic labels are obtained via forced alignement from a mono- phone based HMM-GMM ASR model trained LRS3. In particular, and Normalize Mutual Information (NMI) as the evaluation metrics. Table 1. shows that (1) The quality of cluster assignments consistent with fine-tuning performance across different models (2) Hand-engineered audio feature (MFCC, NMI: 21.5%) has much stronger correlation with phonetic labels than the visual feature (HoG, NMI: 1.6%) (3) Audio-visual clusters (NMI: 44.2%) are of better quality than pure audio-based clusters (NMI: 39.7%). (4) In single-modality visual Hubert (V/HoG-V V), feature quality is improved negligibly through iteration training.
clustering purity
Table E.1: Quality of different cluster assignments. Each number is in the format of Purity (NMI).
The metrics of cluster assignments and WER last iteration of each model are boldface.
Target K
Model
Iter
WER (%)
Feature
Purity (%),1
NMI (%),1
1
MFCC
100
30.3
21.5
71.5
2
AV/MFCC-AV (it1, 1.9)
100
47.3
37.7
63.6
AV/MFCC-XA
3
AV/MFCC-AV (it2. L12)
500
61.5
42.6
60.9
(Proposed)
4
AV/MFCC-AV (it3, L12)
1000
65.6
43.7
58.8
5
AV/MFCC- AV (114. L12)
2000
68.8
58.2
1
MFCC
100
30.3
21.5
71.5
AV/MFCC-A
2
A/MEC- (itl, L9)
100
47.0
36.7
64.3
3
A/MFC- (it2, L12)
500
56.5
39.7
63.5
1
MFCC
100
30.3
21.5
75.4
V/MECC-A
2
A/MFCC-A (itl.
100
47.0
36.7
69.4
3
A/MFCC-A (i12, L12)
500
56.5
39.7
69.1
1
MFCC
100
30.3
21.5
75.4
V/MFCC-V
2
V/MFCC VV (itl, L9)
100
32.8
22.9
72.6
3
V/MFCC-V (it2, L12)
500
33.0
22.8
72.3
1
HoG
100
16.4
1.6
80.3
v/HoG+VV
2
V/HoG~V (itl, L9)
100
16.4
1.8
80.1
E.2 FEATURE MASKING PRODUCES BETTER FEATURES FOR CLUSTERING
In iteration 1 1-4, we apply the mask in the fused feature. We observe such practice generates targets of higher quality, thus helping future iterations more. Table E.2 shows comparison between such two different masking strategies. Input-level masking enhances the learning of visual representation (see table D.I) while produces worse audio-visur feature (NMI: 27.2% In contrast, the two streams of original input are better aligned in feature-level masking which is consistent with the cluster generation process, thus leading to better audio-visual clusters (NMI: 37.7%).
Table E.2:
Impact of masking strategy quality of cluster assignments (purity/NMI: quality of cluster assignments used to train the model)
Feature
Purity (%)
NMI (%)
MFCC
100
30.3
21.5
AV/MFCC-AV (itl, L9) w/ Feature Masking
100
47.3
37.7
AV/MFCC-+VV (itl, L9) w/ Input Masking
100
34.5
27.2
E.3 CLUSTERING QUALITY ACROSS LAYERS
Figure E.I shows the clustering quality of features of different layers in different iterations. The cluster assignment quality generally improves with more iterations. In the first iteration, features in
21

---

Published as a conference paper at ICLR 2022
the middle layers show higher quality than the other layers. The target for the first iteration (MFCC clusters) is of worse quality, thus later layers that are more correlated with targets do not yield the best cluster. Target quality improves with more training iterations. thus the best feature layer shifts towards the end. Setting a larger number of clusters increases the clustering quality as can be seen from the comparison between "varied clusters" and "2K clusters". In terms of the 12th layer which we choose, the highest NMI (44.2%) is achieved in the last iteration. In addition, more iterations of training improves the overall quality of clusters produced by a model though the highest NMI/purity among all layers does not necessarily increase in later iterations. Therefore, setting larger number of iterations brings stable gains which are more robust to the index of layer chosen for clustering. It is important as the purity/NMI, whose measurement rely on supervised model, are not used for hyperparameter search practice.
Figure F. Quality of feature clusters from different layers across different iterations (BASE, 433 hours unlabeled data). (Iter i, Layer j): cluster quality of layer-j feature of iter-i model. Upper row: 100, 500, IK, 2K clusters for iterations. Bottom row: 2K clusters for all iterations. Purity/NMI of the initial MFCC clusters: 30.3%
NMI vs. Layer, Varied clusters
Purity Layer, Varied clusters
45.0
Ite--
Iee-2
65
Iter-2
40.0
Iter-3
60
Iter-4
58
37.5
50
32.5
40-
35
27.5
10
12
Layer
Layer
NMI Layer, 2K clusters
Layer, duustess
45.0
Ier--
Itar
65
Itar2
40.0
Iter
60
Iter
55
37.5
50
22.5
40
27.5
22

---

Published as a conference paper at ICLR 2022
F QUALITATIVE EXAMPLES
Figure F.2 shows the example outputs from different models. Our self-supervised model is the LARGE AV-HUBERT pre-trained with 1,759 hours unlabeled data and fine-tuned with 433 hours labeled data. The baseline model is the supervised baseline trained with 433 hours labeled data. Both models use the S2S criterion for supervised training and have the same number of parameters. Qualitatively, our approach provides transcriptions with much higher quality. The baseline approach confuses among words of similar sound while our model output more semantically sound sentences. Figure F.2 also shows typical erross made by our model. We noticed many errors are on short sentences. This is mainly because lip reading relies heavily on the context for recognition due to the existence of homophomes. Thus the error rates in lip reading utterances, which differs from ASR, as can be seen from figure F.T Substitution among words with homophemes ('fiction' 'vision' in a.4, 'part' 'bunk' in b.5) is another source of error made by the model.
notably higher in short
Figure F.l: WER vs. sentence length for lip reading (left) and ASR (right)
3.0
2,0
WER (18)
1.0
0.0
50
15. 10)
[10. 15)
(15.20)
[20.25]
25
10
[5,10)
(10.15)
[15. (15
120, 25)
words per sentence
words per sentence
23

---

Published as a conference paper at ICLR 2022
Figure F.2:
Transcriptions from different lip-reading models. GT: ground-truth Proposed: self-
supervised model, Supervised: supervised model. Red: wrong words in the output
(a) Self-supervised Supervised
GT:
why ask all of the states to do that instead why not ask all of these things to do that instead
Proposed:
Supervised:
why can't actually all of these things do things and
GT:
indeed we run the risk of making things worse indeed want the risk of making things worse
Proposed:
Supervised:
in india we roughly receive money the health world
GT:
my desire to dísappear was still very powerful my desire to disappear was still very powerful
Proposed:
Supervised:
my son is speaking with children about food
GT:
the silent majority does not need to be silent the same majority does not need to be silent this time the total disaster needs be designed
(4)
Proposed:
Supervised:
GT:
mortality is not going down it's going up mortality is not going down it's going up
Proposed:
Supervised:
we're we're seeing this not only carrying slowly how
(b) Failure cases
GT:
it's win all around he's win on the ground
Proposed:
GT:
sort of leadership by humiliation
Proposed:
the leadership by communication
GT:
is it about equality ask about quality
Proposed:
GT:
science fiction one of the greatest and most effective forms of political writing stage vision is one of the greatest and most effective forms of political writing
Proposed:
GT:
can't identify with that part we can't 't identify with that bunk
(5)
Proposed:
24


Reviews:
[{"review_id": "kKu-JLu8koB", "reviewer": "Reviewer_BjAV", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strength:\n\n++ This paper is basically well-written with detailed supplementary.\n\n++ The idea of leveraging Bert for self-supervised training is different from the traditional audio-visual alignment pertaining.\n\n++ Detailed improvements such as the audio-visual stream formulation and Masking by substitution have been proposed.\n\n++ The results on LRSv0.4 outperform the previous SOTA.\n\n## Weakness:\n\n-- This kind of formulation, though novel on the topic of audio-visual speech recognition, has been used on other topics. The whole model is built upon HuBert, which limits the contribution.\n\n-- Why use only the LRS3 dataset for evaluation? The LRS2 dataset should be available. More dataset evaluations could be more comprehensive than one.\n\n-- About the **Masking by substitution**: Why would \"the fake segment detection sub-task improves the learned features of the ResNet encoder\" when the “filled-in frames are from real videos\"? More intuition should be given. It is said that your motivation is to keep \"masked region smooth temporally\". I understand that it would be smoother than random noise masking, however, segments from the same video cannot guarantee temporal smoothing.\n\n-- I am curious about the \"MULTILINGUAL VS. MONOLINGUAL\" experiments in Sec. 4.4. The model is kept in a setting with one 1 iteration training and 30h of labeled data. While I understand the limitation in resources, the authors are suggested to show the multilingual results under the full setting. With extra data involved, it might take a longer training time. After all the method is \"agnostic to the spoken language\".\n", "summary_of_the_paper": "This paper proposes audio-visual HuBert based on the original HuBert model. The general idea is similar to HuBert that predicts clusters of extracted features. The authors further propose the modality dropout and masking by substitution module to further fine-tune the design. Extensive experiments and ablation studies are carried out on the LRS3 dataset, which validate the effectiveness of the proposed method.", "summary_of_the_review": "Overall this paper introduces techniques that the field of audio-visual speech recognition has rarely leveraged into this field, and makes task-specific modifications. The idea is not thoroughly new but the application is good. As I am no expert in this field, it is possible that I have missed something.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "gAca2R-IFSV", "reviewer": "Reviewer_9sxE", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n1.\tThe authors propose a powerful method to learn speech representations from multi-modal data which can be utilized for both ASR and VSR, in a self-supervised manner.\n2.\tAV-HuBERT achieves state-of-the-art performance in terms of WER in a sentence-level audio-visual dataset, LRS3. Moreover, even in a low-resource setting (training with 30 hours labeled data + 1,759 hours unlabeled data), their model shows a comparable VSR result with a previous state-of-the-art method that trained with 31,000 hours labeled data.\n3.\tTheir proposed method is well verified in various views with extensive experiments through both manuscript and supplementary. (Performances 1) with different model sizes, 2) with different loss functions (CTC and S2S), 3) with different cluster targets, 4) using multi-lingual data, 5) of both ASR and VSR, and 6) of ablation studies.)\n\nWeakness:\n1.\tEven the proposed AV-HuBERT shows impressive performances, but it is an expansion of HuBERT to work in multi-modal data.\n2.\tThere are many typos and errors in the manuscript. Please refer to below detailed comments. Moreover, Figure 1 should be improved, the arrows are arranged confusingly.\n\nQuestions:\n1. The authors use concatenation as the default fusion operator. Which dimension is utilized for concatenation, temporal or channel? If the temporal dimension is used for multi-modal fusion, better to refer to the two papers [R1, R2].\n2. For the loss function in equation (4), their final performance seems obtained by setting the alpha as 0. However, it is hard to find because it is placed in the results of ablation studies in the appendix. Since setting the alpha as zero (the same as omitting the second loss term) is better performed, it would be better to describe the effect of alpha in AV-HuBERT after equation (4) or at the experimental setup, briefly.\n3. The ASR performance of AV-HuBERT is not presented. It is just described as the AV-HuBERT performs worse than HuBERT in text. Moreover, the last sentence in Sec 4.5 seems not enough to explain why AV-HuBERT is not better than HuBERT on ASR, even if multi-modal data is utilized during training. Have the authors re-examined hyper-parameters (m_a, m_v, p_m, p_a, alpha) for ASR task?\n4. 2nd paragraph in Sec A.4. What does the sentence “Both modalities are used at test time for feature extraction” mean? There is no downstream task utilizes both modalities.\n\nErrors:\n1. Sec 3.2 line #4. In “The cluster assignment z_(1:T)^a~”, Does z_(1:T)^a mean z_(1:T)^i? \n2. 3rd paragraph on page 4. “When only modality is used”  -> \"only one modality\".\n3. Last paragraph on page 5. “where B consists of all possible...” -> “where B^-1 maps all possible...”. B seems a mapping function thus the word “consist” seems not appropriate.\n4. 2nd paragraph on page 6. “we rely on the joint decoder module”. What does the joint decoder mean? Do the authors use a joint CTC/S2S decoder?\n5. 2nd paragraph on page 8. ‘iterative pe-training’ -> ‘iterative pre-training’\n6. 1st paragraph on page 14. CMUDict (cmu) -> maybe missing reference.\n\n[R1] Chen, Yen-Chun, et al. \"Uniter: Universal image-text representation learning.\" European conference on computer vision. Springer, Cham, 2020.\n[R2] Lee, Sangho, et al. \"Parameter Efficient Multimodal Transformers for Video Representation Learning.\" International Conference on Learning Representations. 2020.", "summary_of_the_paper": "This paper presents a new self-supervised learning framework for audio-visual speech. Their framework is trained with BERT-like training that predicts representations in the masked regions using audio and visual inputs. Extensive experiments are performed using two popular audio-visual datasets. Especially, with a small amount of labeled training data (30 hours), their method achieved comparable results with a previous state-of-the-art method which trained with a huge amount of labeled data (31,000 hours). Moreover, when trained with more labeled data (433 hours), it surpasses the previous state-of-the-art method (trained with 31,000 hours) by 3.1% WER.", "summary_of_the_review": "Overall, authors approach is promising for reducing the need for a large labeled visual dataset for training VSR models, and the result shown in the paper is significant.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "BXMW44UlMwm", "reviewer": "Reviewer_vjDT", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The ideas are intuitive and very suitable for the problem. The authors take good advantage of a recent advancement of pre-training in ASR (HuBERT), and applies it to lip reading, aka visual speech recognition. The visual-only HuBERT is a simple adaptation, but the cross-modal iterative refinement is a useful addition. \n\nThe performance is strong on both low and full resource settings, and the results clearly demonstrate the advantages of self-supervised pre-training.\n\nIt would be good to also show the effectiveness of the embeddings for other downstream tasks, such as audio-visual speech recognition and speech separation. \n\nI think Table C1 in the appendix is quite useful for justifying design choices and at least some of it would be beneficial to show in the main paper. It would also be good to show lip reading results after visual-only single-modality pre-training (like audio HuBERT), without cross-modal training.\n\nAs an application paper, the paper might be better suited for CVPR or Interspeech, but I still vote for acceptance since the contributions are significant and the results are state-of-the-art.\n", "summary_of_the_paper": "The paper proposes a strategy for self-supervised pre-training for lip reading. To this end, they propose Av-HuBERT which learns embeddings by masking video input and predicting iteratively refined hidden units. The authors propose two sub-strategies for this: visual-only and cross-modal. The pre-trained embeddings are fine-tuned on the lip reading tasks, in the style of wav2vec 2.0 and HuBERT for automatic speech recognition. The authors perform experiments on both limited and full resource settings, on both of which they demonstrate strong performance. ", "summary_of_the_review": "The authors propose an effective adaptation of HuBERT for lip reading, by adding cross-modal iterative training. The contributions are logical and the results are strong.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "8cNVmnMv3fE", "reviewer": "Reviewer_3PrM", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n\na) Simple but very effective extension of masked self-supervised training to multi-modal tasks\n\nb) State of the art lip-reading and ASR results on the LRS3 benchmark\n\nc) Demonstrates value of providing multi-modal input to the model even when only a single modality is present at test time.  Alternatives that consider only that modality as input (image of lips in this case) are significantly worse.\n\nCons:\n\nNot a negative, but it will be interesting to see further experimentation with layer-wise supervision approaches that inform visual representation learning via audio or AV networks.  This will have the advantage that the model will only see the visual modality at training time, yet will have the advantage of supervision from audio or AV model at all layers, not just at the output.\n\n", "summary_of_the_paper": "This paper presents a multi-modal (audio-visual) pre-training approach that results in models that can be fine tuned to achieve state of the art performance on both lip-reading (visual only) as well as speech recognition (audio only) on the LRS3 dataset which is the largest public lip-reading benchmark. It extends the idea of masked self-supervised learning to multi-modal scenarios where units that are masked and predicted are automatically discovered via clustering and refined as training progresses.  While the proposed approach is a relatively straight forward extension of previously published HuBERT model it is demonstrated to be very effective.  Paper also presents a number of alternative approaches for leaning embeddings of the visual modality and very interestingly demonstrates the value of multi-modal input to the model even when a single modality is present at test time.\n", "summary_of_the_review": "Very well written paper, presenting an effective approach for learning embeddings for multi-modal data.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: PAPER: This paper introduces an extension of the HuBERT audio-only model for the audio-visual setting, allowing for self-supervised pre-training of multimodal model which also performs well on the unimodal tasks (lip-reading and ASR). The paper applies the idea of modality dropout to their multimodal pre-training setup and introduce the idea of masking with substitution as a way to improve visual representation learning. A strong aspect of the paper is its experimental section, showing strong improvement for lip-reading tasks, bringing a new state-of-the-art performance. The experiments also show improvement for ASR task. 
DISCUSSION: All reviewers seemed to appreciate the experimental results, with new state-of-the-art performance on both unimodal task, when performing multimodal pre-training. The paper does bring some technical novelty, but primarily because of its application to the audio-visual domain. The modality dropout idea was already explored for other audio-visual tasks such as speech-driven face animation (Abdelaziz et al., ICMI 2020), but the idea of “masking by substitution” seems novel and helps learning better visual representations. The authors were able to address many questions and concerns expressed by reviewers. All reviewers took the time to read these responses and acknowledge them.
SUMMARY: This paper brings an interesting extension of the audio-only HuBERT model for the audio-visual setting. The strength of the paper is in its evaluation, with strong performances, establishing many new state-of-the-art results.  All reviewers supported the acceptance of this paper.
Decision: Accept (Poster)