Jointly Learning Visual and Auditory Speech Representations from Raw Data
Authors: Alexandros Haliassos
Pingchuan Ma
Rodrigo Mira
Stavros Petridis
Maja Pantic
Venue: ICLR 2023 poster (2023)
Keywords: self-supervised learning
lipreading
speech recognition

Abstract: We present RAVEn, a self-supervised multi-modal approach to jointly learn visual and auditory speech representations. Our pre-training objective involves encoding masked inputs, and then predicting contextualised targets generated by slowly-evolving momentum encoders. Driven by the inherent differences between video and audio, our design is asymmetric w.r.t. the two modalities' pretext tasks: Whereas the auditory stream predicts both the visual and auditory targets, the visual one predicts only the auditory targets. We observe strong results in low- and high-resource labelled data settings when fine-tuning the visual and auditory encoders resulting from a single pre-training stage, in which the encoders are jointly trained. Notably, RAVEn surpasses all self-supervised methods on visual speech recognition (VSR) on LRS3, and combining RAVEn with self-training using only 30 hours of labelled data even outperforms a recent semi-supervised method trained on 90,000 hours of non-public data. At the same time, we achieve state-of-the-art results in the LRS3 low-resource setting for auditory speech recognition (as well as for VSR). Our findings point to the viability of learning powerful speech representations entirely from raw video and audio, i.e., without relying on handcrafted features. Code and models are available at https://github.com/ahaliassos/raven.

Published as a conference paper at ICLR 2023
JOINTLY LEARNING VISUAL AND AUDITORY SPEECH REPRESENTATIONS FROM RAW DATA
Alexandros Haliassos' Pingchuan Ma' Rodrigo Mira' Stavros Petridis Maja Pantic
Immeeral College London
Meta Al
a exandros ha assossl (eipssoss@@mmperillll. ac.
ABSTRACT
We present RAVEn, self-supervised multi-moda approach to jointly learn vi- sual and auditory speech representations, Our pre-training objective involves en- coding masked inputs, and then predicting contextualised targets generated by slowly-evolving momentum encoders. Driven by the inherent differences be- tween video and audio, our design is asymmetric w.r.t the two modalities' pre- text tasks: Whereas the auditory stream predicts both the visual and auditory targets, the visual sults in low- and high-resource labelled data settings when fine-tuning the vi- sual and auditory encoders resulting from single pre-training stage, which the encoders are jointly trained. Notably, RAVEn surpasses all self-supervised methods on visual speech recognition (VSR) on LRS3, and combining RAVEn with self-training using only 30 hours of labelled data even outperforms recent semi-supervised method trained on 90,000 hours of non-public data. At the same time, we achieve state-of-the
predicts only the auditory targets. We observe strong re-
results in the LRS3 low-resource setting for au- ditory speech recognition (as well as for VSR). Our findings point to the viability of learning powerful speech representations entirely from raw video and audio, ie.. without relying on handcrafted features. Code and models are available at https://qithubs com http:/aaaaaasssssaaaaaaa
INTRODUCTION
The sound of someone articulating words coincides with the sight of movements in and around their mouth. Both a recording of a speech waveform and a corresponding silent video of mouth motion provide rich- but not identical information on which words were uttered. Despite the difficulty of interpreting lip movements compared with an audio waveform, the task of visual speech recognition (VSR; also known lipreading) has important applications, ranging from recognising utterances in a noisy environment (Ma et al. 2021b Afouras et al. 2018a, Martinez et al.) 2020); Makino et al., 2019) and aiding people suffering from aphonia (an inability speak), transcribing archival silent films and detecting DeepFad Fake videos (Haliassos et al.) 2021).
Auditory (also known automatic) speech recognition (ASR) and VSR benefit greatly from the combination of high-capacity neural networks and large datasets. Rapid advances of modern hard- ware are enabling the use of ever-growing, data-hungry networks, but the effort required for tran- scription hinders the scaling of labelled data along with the models. One way to leverage unlabelled videos for VSR is to use an external ASR model for pseudo-labelling (Afouras et al. 2020 Ma et al. 2022). However, this requires large amount of labelled data to train strong ASR model in the first place, and supervised VSR training with long sequences often poses optimisation problems, requiring costly curriculum learning strategies (Chung et al. 2017 Ma al. 2022) or pre-training the feature extractor with isolated words (Afouras et al. 201 Sa Ma et al. 2021b).
A solution is first learn, in self-supervised way, general representations from large corpora of unlabelled data, and then fine-tune them on smaller labelled datasets (Mohamed et al. (022). The fine-grained correspondence between the (synchronised) visual and auditory modalities provides
Work done Meta AI.

---

Published as a conference paper at ICLR 2023
natural source of self superviun,,,,,on, and can produce highly semantic representations invariant to noise not shared between the modalities. However, approaches leveraging this correspondence either (1) only work for word-level samples rather than continuous speech (Chung & Zisserman 2016 Chung
le al. 2019 2020); (2) use handcrafted features (e.g., spectrograms or MFCCs) as their inputs or
targets Ma et al. 2021a Shi et al. 2022). which contain inductive biases that may influence the
learned representations; (3) use multi-stage pre-training procedures (Ma et al.) [02 Shi et al.
2022
Pan al., 2022); y and/or (4) use separate pre-training strategies for VSR and ASR (Shi al.
2022
complicating the process of obtaining representations suitable for both tasks.
In this paper, we present single-stage self-supervised approach that jointly learns visual and au- ditory speech representations from raw video and audio only. We dub our approach RAVEn (Raw Audio-Visual Speech Encoders). It involves pair of student-teacher networks for each modal- ity, whereby the students encode temporally-masked inputs, and, through the use of lightweight Transformer-base predictors, regress outputs of nomentum-based teachers (Grill et al., 2020] Caron et al., 202I) that are presented with unmasked inputs. Further, given that audio contains more information relevant to speech than video, we propose a learning strategy that accounts for the expected difference in the quality of targets between the modalities. Namely, while the audio student predicts outputs from both video and audio teachers (cross- and within- modal learning), the video student predicts only auditory targets (cross-modal learning). As show, this setup leads to better downstream performance for both VSR and ASR as opposed to other strategies.
We conduct experiments with models and datasets of different sizes. We find that, when fine-tuning our pre-trained models for VSR and ASR with only 30 hours of labelled data from LRS3 (Afouras) al. 2018b), RAVEn surpasses recent self-supervised methods by large margin in most settings. Coupling pre-training with self-training reaches 23,8 23.8% WER for VSR on LRS3, even outperforming a method trained on 3000x more transcribed hours (Serdyuk et al.) 2021). At the same time, we are better than or on par with the recent AV-HuBERT method (Shi et al. 2022) on ASR, without using a dask-dependent pre-training strategy nor handcrafted features. Using the full 433-hour L.S3 dataset for fine-tuning pushes the results even further, achieving 23.1% 1% / 1.4% WER for VSR / ASR, respectively. Similarly strong performance is observed on the LRS2 dataset (Appendix BB.
2 RELATED WORK
Masked prediction.
The pre-training task of predicting missing content given masked inputs has
proven successful in various domains, such as natural language processing (Devlin al.
2019
Radford et al. 2018; 2019 Brown et al. 2020), image recognition (He et al. 2021 Xie et al.
2022
Bao al. [022]] and speech recognition (Baevski et al. 2020 Hsu et al. 2022 || Shi et al. 2022). An
important aspect of masked prediction is the nature of the targets. Some works (ee et al. 2021 1 Xie
ett al. 2022) use pixels targets; others use pre-trained tokenisers (Bao al. 2021
modality-
specific handcrafted features (Wei et al.) [022]] Hsu et al. [2021] Shi et al.) 2022). Our method, in
contrast, generates targets from raw video and audio using momentum encoders.
Self-distillation for unsupervised representation learning. RAVEn is partly inspired by the suc- cess of self-distillation in self-supervised learning with visual data (Grill et al., 2020) Caron et al. 202II: such works target invariance w.r.t. image-specific augmentations. In contrast, RAVEn does not rely on domain-specific augmentations but rather on a combination of cross- moosl learning and masked prediction to drive representation learning for visual and auditory speech signals. data2vec (Baevski et al. 2022) combines masked prediction with a momentum encoder, but, aside from being uni-modal, it is different methodologically in multiple ways. For example, it applies ad-hoc normal- isation and averaging techniques to the targets to prevent representation collapse, while our targets are simply the outputs of the encoders, which are regressed via Transformer- heads.
Self-supervised audiovisual learning. Audiovisual correspondence has been used to learn global representations for action recognition through the use of clustering (Alwassel et al.] 2020 Asano et al. 2020). contrastive learning (Arandjelovic & Zisserman, 2017: 2018 Korbar et al. 2018
Patrick et al. 2020 Morgado et al. 2021 Ma et al., al.] 2021c)), or representation matching (Recasens)
et al. 2021). Cross-modal learning has also found uses in biometric matching (Nagrani et al.)
201 Sa b), emotion recognition (Shukla et al. 202I), and DeepFake detection (Haliassos et 20.2)
We employ cross- and within-modal losses to learn camporally-varying speech representations.
2

---

Published as a conference paper at ICLR 2023
Used
Fealure
Festure Estrator
Tranffomer Enooder
CMC
Tranffrmer Enooder
mask
tokan
Video
audo teacher
Shallow Predecers in
Sh Preciator
teacher
exponential svenage
DMA
inference
Transfomer Encoder
CHA
Transfomeet Encoder
cross modal prediction
within-moda prediction
Feature Extrstor
video
video teacher
Figure I: RAVEn overview. Given masked video and audio, students predict outputs of unmasked momentum teachers, via shallow Transformer predictors that intake mask tokens. The audio student predicts outputs from both audio and video teachers; the video student predicts only audio targets. losses applied all features; the within-modal loss is computed only on masked features. Only the student encoders are fine-tuned for VSR/ASR. Frames blurred for anonymity.
Self-supervised audiovisual learning for speech recognition. Earlier audiovisual self- supervised works (Chung & Zisserman 2016 Chung et al.] 2019 2020) tended to focus on word-
level lipreading. Recently, some attention has been paid to the more realistic task of continuous speech recognition (Ma et al. 2021a Shi et al. 2022 Sheng et al. 2021 Pan al. 2022 Ma al.,
2021c). Sheng et al. (2021): Ma et al. (2021c) use contrastive learning and apply their method to VSR. Ma et al. (2021a) predict speech features using an external PASE+ encoder (Ravanelli al., 2020 Pan et al. (2022) transfer pre-trained visual and auditory encoders, which were separately trained via contrastive losses. AV-HuBERT (Shi et 2022) predicts iteratively
cluster as- signments from masked audiovisual inputs and achieves impressive VSR and ASR performance. It employs multiple stages of alternating between offline clustering and cluster assignment prediction, and relies on hand-crafted audio features (MFCCs) for cluster initialisation, which is shown to be crucial the performance (Shi et al., (022). We demonstrate that is possible to jointly learn effective representations for VSR and ASR in single stage simply from raw video and audio.
3 METHOD
3.1 PRE-TRAINING
Our architecture consists of a pair of student-teacher networks per modality (see Figure D. The students intake masked inputs and predict targets formed by teachers receiving unmasked inputs.
Masking. We employ masking encourage the students to take context into account when solving the task. Given a grayscale video x" E R xxx with resolution (H, W) and T frames, and an audio sample xº E R of length N. randomly sample with probability 0.2 each video frame be the starting mask index, and if selected, then the consecutive three frames are zeroed out (ablation in Section 4.4. A similar mask applied to the auditory input, except that it is enlarged by factor of 16 000 25- 640 (since the audio is sampled at 16,000 Hz and the video at 25 fps).
Encoders. The masked video and audio, 8" and Ha respectively, are fed to their corresponding student encoders f and fe. yielding features S (τ²) R and fo (3 E RTxD, where D is
3

---

Published as a conference paper at ICLR 2023
the dimensionality of each feature. Both f" and fe consist of mrodality-specinne convolutional feature extractor followed by a temporal encoder, as in related VSR/ASR works (Ma et al. 2022 Baevski et al.] 2020). The video feature extractor is 2D ResNet18 (He et al.) 2016) with 3D convolutional stem (Petridis et al.] 2018), outputring an embedding per frame. On the audio side, ID ResNet18 which produces features at 25 fps, to match the video sampling rate (see Appendix A.5)) The temporal encoder for each modality is a Transformer (Vaswani et al., 2017) (without a classification head) with hidden size D. We use relative positional embeddings (Dai al. 2019). and following Chen et al. (202T), we replace layernorm (Ba et al., (016)) with batchnorm (loffe & Szegedy 2015) before each multilayer pereeptron (MLP) block (see Appendix C.1 for comparison).
Predictors. The students contain Transformer predictors, which regress targets given 1) the en- coder outputs corresponding to the unmasked portions of the inputs and 2) mask tokens associated with the masked portions. Note that mask tokens are applied to the predictors rather than the en- coders (He et al. (021). This reduces the discrepancy between pre-training and fine-tuning: In both cases, the encoders do not see mask tokens. A predictor that takes representations corresponding to modality mi E {vr a} and predicts targets associated with modality m2 E {u, a} is denoted as fpm fm--- For ease, the application of mask tokens to the encoder outputs is absorbed in the notation.
Unlike other works which output global representations (one embedding per sample) and thus use MLPs as predictors (Grill et al., 2020 Chen et al. (02I), we use Transformers to allow modelling temporal dynamics, which we found greatly improves results. The predictors be lightweight: Indeed, two-block Transformers with hidden size 512 work optimally in our experiments.
The targets are simply the outputs of momentum- -based teachers g" and g" (Grill et al. 2020 Caron et al., 202I), which are given as input the unmasked video or audio, in order to force the students to predict the missing information. Each teacher is architecturally identical to its student encoder counterpart. Denoting the parameters of the student encoders and teachers as s and tm respectively, each iteration the following update is performed:
Targets.
(1) where E {v, a} specifies the modality and J momentum parameter following cosine sched- ule from 0.999 to 1. A high value of leads to slowly-varying teachers and stable targets. The use of momentum-basee teachers obviates the need for handcrafted targets or multi-stage training.
1T <- put" + (1 -p)=";
Prediction tasks. The auditory modality contains more information relevant to speech than the visual one: Mouth motion is inherently more ambiguous than a speech waveform due the presence of homophemes (Chung et al., 2017). We propose a loss structure which reflects this asymmetry between the modalities. The audio student predicts the targets from both the video and audio teacher, thus benefiting from the ability of cross moosl learning to induce semantic representations, while the same time being encouraged to retain information from the auditory input that is absent from the visual one. As result, two predictors are associated with the audio student, one for each target type. On the other hand, the video student only predicts the auditory targets, which are inevitably of higher quality.
Losses. The loss function the negative cosine similarity (Grill et al. 2020), denoted as sim. Due to the temporal alignment of the inputs, the cosine similarity is applied between pairs of correspond- ing features and then summed across the time dimension. For the within-mode task (audio-to-audio prediction), the loss is applied only on targets corresponding to masked portions of the input (De- vlin et al. 2019). For the cross-modal tasks, the loss is applied on all targets, which we found to work better. Note that predicting the unmasked portions in cross-modal learning is non-trivial task (unlike in within modal learning) and can bolster representation learning.
Denoting the set of mask token indices for audio Ma. the andio-to-audio prediction loss and cross-modal losses can be respectively expressed as
ca-aa
Σ sim (57=*a (fe (I")),, sg (g" (I") nEM.
(2)
L
Σ sim (fmm -22 pm (z )), sg (g m2
(3)
4

---

Published as a conference paper at ICLR 2023
where mı, my (v.a), my # ma, and sg denotes the that no gradient passed back to the teacher networks..
operation, which indicates
Objectives. At each iteration, the objectives for the video and audio students are, respectively,
C. = L La -ave + Lama.
(4)
The teachers are updated via Equation
3.2 FINE-TUNING
For fine-tuning, keep the pre-trained student encoders and discard the rest. We append linear layer and a Transformer decoder for joint CTC / attention decoding (Watanabe et al. 2017). as in Ma al. (2021b). Following Ma al. (2021b) we set the CTC weight to 0.1. We use SentencePiece (uudo & Richardson 2018 subword units with vocabulary size of 1,000 as our targets.
Self-training. Combining pre- pre-training with self-training tends improve results over using either strategy in isolation (Xu et al., 2021; (Shi et al. (022). To that end, we first fine-tune our pre-trained audio encoder on the labelled data, and then use the model for pseudo-labelling the unlabelled data. The pre-trained video and audio models are then fine-tuned using both the labels and pseudo-labels.
4 EXPERIMENTS
4.1 SETUP
Datasets. For pre-training, we conduct experiments with LRS3 (Afouras et al. 2018b) (without the labels) well as combination of LRS3 and English-only version of VoxCeleb2 (Chung et al. 2018) curated pShi et al. (2022). which we refer to as Vox2ee. The former features 433 hours of footage and the latter 1,759 hours. For fine-tuning, we use the full LRS3 with the labels as our high-resource labelled data setting, as well as a 30-hour subset (the "trainval" partition) as our
low-resource setting. We present results for the LRS2 dataset (Chung al. 2017) in Appendix B
Transformer encoder. We show results for two configurations of the Transformer encoders, Base and Large, with 12 and 24 blocks respectively. The hidden size/MLP size/attention heads are 512/2048/8 for Base and 1024/4096/16 for Large, amounting to 41 million (M) and 328 M pa- rameters, respectively. We note that our Base model around half the size of the one used by Shi et al. (022)) and the Large models are similar in size. Training details are provided in Appendix
4.2 LOW-RESOURCE LABELLED DATA SETTING
We pre-train our models on LRS3 and/or LRS3+ Vox2-en and then fine-tune them on the 30-hour LRS3 subset to evaluate performance when labels scarce. We reports results in Table
Compared with training from scratch, RAVEn pre-training leads to dramatic performance improve- ments in all configurations. Notably, increasing the model size hurts VSR performance when train- ing from scratch, but improves it when using pre-training.
Our Base variant outperforms all related methods VSR. It surpasses the Base AV-HuBERT model by 4.8% and 5.9% WER when using LRS3 and LRS3+ RRS3+Vox2-ee Vox2-en, respectively, despite having roughly half the number of parameters. The Large model provides significant boosts over the Base model (32.5% vs 40.2% WER) when using Vox2-en Vox2-en for pre-training, keeping the number of labelled data points fixed. Self-training further improves WER by 7.7%, indicating its complementarity with RAVEn pre-training. Finally, using language model (see Appendix A.4 details) leads to WER of 23.8%, better than a method (Serdyuk et al., 2021) trained on 000 hours non-public data.
On ASR, RAVEn significantly outperforms the audio-only Hubert (Hsu et al. 2021) model, and in all cases is better than or on par with AV-HUBERT Our best ASR model without self-trainne traininng achieves 2.7% WER AV-HuBERT's 2.9% WER, despite AV-HuBERT using different pre-trale rre-training strat- egy for VSR than ASR. For example, using the same pre-training hyperparameters increases AV. HuBERT's WER for ASR from 3.8% to 4.6% with the Base model (Shi et al. 2022). In contrast,
5

---

Published as a conference paper at ICLR 2023
WER (%)
Method
Encoder
LM
Unlab hrs
Lab hrs
VSR
ASR
supervised
Afouras ot al.
20
Transformer
/
1,519
58.9
8.3
Xu al. 2020
RNN
X
590
57.8
7.2
Shillingford
2019
RNN
V
3,886
55.1
Ma 2022
Conformer
V
813
34.7
Makino el al.
2019
RNN
X
31,000"
33.6
4.8
Prajwal
2022
Transformer
V
2,676'
30.7
Serdyuk
202
Transformer
90,000"
25.9
2.3
Serdyuk et al.
2022
Conformer
X
90,000
19.3
1.6
from scratch
Base model
Transformer
30
93.4
18.5
Large model
Transformer
x
30
95.5
9.9
self-supervised
Base models, less pre-training data
Ma
2021a
Transformer
X
433
30
71.91
Zhang ct al. 2022
Transformer
X
433
30
67.8
10.9
1lsu et al.
Transformer
X
433
30
5,4
Shi et
2022
Transformer
X
433
30
51.8
4.9
RAVEn
Transformer
X
433
30
47.0
4.7
Base models, more pre-training data
Hsu al. 2021
Transformer
x 1,759
30
5.0
Shi et al. 2022
Transformer
x 1,759
30
46.1
4.6 3.8
RAVEn
Transformer
x
1,759
30
40.2
3.8
Large models, more pre-training data
Hsu et al. 2021
Transformer
X 1,759
30
3.2
Shi ct al.
2022
Transformer
X
1,759
30
32.5
2.9
Shu et al.
2022 w/ self-training
Transforner
X
1,759
30
28.6
RAVEi
Transformer
X
1,759
30
32.5
2.7
RAVEn w/ self-training
Transformer
X
1.759
30
24.8
2.3
RAVEn w/ self- self-training
Transformer
1,759
30
23.8
1.9
Table 1: LRS3 low-resource setting. We report results on the test set when fine-tuning on 30 hours of LRS3 with different model sizes number of unlabelled data hours (Unlab hrs). LM denotes whether or not language model was used during decoding. We also provide baselines for training our models from scratch (without our pre- gre-training) and results from suply-sue methods trained more labelled data hours (Lab hrs) for reference. "Includes non can--uulllly-vailllll taken from Shi al. (2022) Resullt with the same pre-training strategy for VSR and ASR.
data. 'Result
the video and audio encoders we use for fine-tuning are the result of single pre-training phase, where they were jointly learned.
All in all, our results suggest that transcribed data, which are costly to obtain, can be largely substi- tuted with unlabelled audiovisual data.
4.3 HIGHH-RESORRG LABELLED DATA SETTING
Table reports results when fine-tuning on the full 433 hours of LRS3. Despite increasing the la- belled data, training the model from scratch still leads to poor performance, as the low-resource setting. This is likely related to the long utterances in the LRS3 dataset, which may cause optimisa- tion difficulties (Ma et al. 2022). A potential remedy to employ curriculum learning, as proposed by Ma et al. (022)). by training the network in multiple stages with increasingly longer sequences.
We observe that this strategy indeed reduces WER from 87.3% to 39.8% for the Base model. Even so, pre-training with the same data used for fine-tuning leads better WER of 39.1%, without requiring a curriculum learning procedure. This suggests that self-supervised pre- training can aid optimisability. The impact of pre-training is more pronounced with the Large model.

---

Published as a conference paper at ICLR 2023
WER (%)
Method
Encoder
LM
Unlab hrs
Lab hrs
VSR
ASR
xemi-supervised (using external models for pseudo-labelling)
Afouras al. (0000)
CNN
344
433
59.8
Ma et al. 2022
Conformer
641
818
31.5
from scratch
Base model
Transformer
433
87.3
2.5
Base model w/ curriculum
Transformer
X
433
39.8
Large model
Transformer
X
433
85.8
2,2
Large model w/ curriculum
Transformer
433
38.8
self superviedd
Base models, less pre-training data Shi ct al. (2022)
Transformer Transformer
433
433
44.0
RAVEn
X
433
433
39.1
2.2
Base models, more pre-training data
Ma ct al.
2021a
Transformer
1,759
433
49.61
Hsu et al.
2021
Transformer
X 1.759
433
2.4
Shi et al.
2022
Transformer
X
1,759
433
34.8
2.0
RAVEn
Transformer
1,759
433
33.1
1.9
Large models,
pre-training data
Hsu al.
2021
Transformer
1,759
433
1.5
Sh ci al.
2022
Transformer
X
1,759
433
28.6
1.3
Shi et al.
2022
w/ self-training
Transformer
X
1,759
433
26.9
RAVEn
Transformer
1,759
433
27.8
1.4
RAVEn w/ self-training
Transformer X
1,759
433
24.4
1.4
RAVEn w/ self-training
Transformer
1,759
433
23.1
1.4
Table 2: LRS3 high-resource setting. We report results on the test set with different model sizes and number of unlabelled data hours (Unlab hrs). Lab hrs denotes the number of labelled hours, and LM denotes whether or not language model was used during decoding. We provide baselines for training models from scratch (without pre-training). Ressutt taken from Shi et al. (2022)
RAVEn outperforms AV-HuBERT under all configurations VSR. Our best result is 23.1 %, achieved using self-training and language model. We note that unlike methods that use exter- nal ASR models (Afouras al.) 2020; Ma al. 2022) for pseudo-labelling, we do not require extra data for the self-training phase.
We are on par with the tfete-rr for ASR in the high-resource setting, achieving WER of 1.4% with the Large model. This is despite using raw audio as input (rather than spectrograms which Shi al. (2022) use). We notice that additionally including self-training and a language model does not reduce the WER, suggesting that the ASR performance may have saturated in our environment.
4.4 PRE-TRAINING ABLATIONS
Ablations are performed with our Base model in the low-resource setting with LRS3 pre-training using the validation set from Shi et al. (2022) (as there is no official development set), For more ablations, see Appendix C
Prediction tasks. We study in Table the impact of cross- and within-modal prediction on the leamed representations. Figure 2aaree2illinn FigureZjillusrates illustrates the various prediction tasks we consider. We observe following. The within-modal variant performs decently for ASR but poorly for VSR, which can be explained by the assumption that audio contains more speech information than video. The cross- modal variant performs better than within-modal for both modalities. Since lip movements and the corresponding waveform are correlated in terms of lexical content, the representations resulting from cross cross-modal modal prediction are expected to capture rich semantic information. At the same time, they are likely to be, large extent, invariant factors unshared between the modalities,

---

Published as a conference paper at ICLR 2023
Wihin-moool
Cross-modal
within-mode'
within-modal
video
student
video
A
A
target
audic
Prediction tasks. We consider different choices for our prediction tasks, combining cross- and within-modi losses. We find that applying both cross- and within-modd losses for the audio student, and only a cross-modal loss for the video student works best.
Figure 2:
Prediction tasks
WER (%)
Setting
V-V
A
V
A
A A
VSR
ASR
Within-
/
X
92.7
15.5
Cross-modal
X
X
/
V
40.8
14.0
Cross- + within-modal
/
/
49.0
14.0
Cross- video widhin-modal
X
/
55.3
16.4
Cross- + audio wodhin-modal
X
V
V
/
32.9
12.2
Table 3: Prediction task ablations under the LRS3 low-resource setting using Base model. V + A means that the video student predicts the audio teacher representations, Each prediction loss is associated with a separate predictor.
such as visual or auditory noise, benefiting generalisation. Combining cross- with within-modal learning hurts VSR relative to only cross-modal prediction. However, cross-modal with within- modal learning only for audio achieves the best results. We hypothesise that predicting the auditory targets forces the auditory stream keep relevant information absent from the visual stream. We note that removing the video-to-vidde prediction (from row 3 to row 5 in Table 3) also improves the audio student, as the quality of the visual targets improves. Finally, cross- modal with within-modal learning only for video does not perform well, validating that the asymmetry only works one way.
Masking sampling. Table 4a studies the effect of different masking strengths by varying the mask length and probability that given index is chosen as the mask start. We observe that probability of 0.2 and a length of three video frames (corresponding to 1,920 audio samples) works well. Less masking allows the student focus less context and degrades both VSR and ASR. Interestingly, although more masking hurts VSR, it helps ASR, suggesting that an asymmetric masking strategy w.r.t. the two modalities may further improve results. We leave this exploration future work.
Our masking is less aggressive than what was found to be optimal in related self-supervised image and action recognition literature (where 75% or even 90% of the input masked) (He et al. [021]] Tong et al.) (2022). We hypothesise that mouth movements are farying and do not contain much temporal redundancy, and thus such strong masking makes our pretext task overly difficult.
WER (%)
WER (%)
WER (%)
Position
Loss
Prob
Length
VSR
ASR
VSR
ASR
VSR
ASR
Encoder
43.1
14,0
Masked
44.3
12.6
0.1
41.1
15.6
Predictor
32.9
12.2
All
32.9
12.2
0.2
32.9
12.2
0.4
35.7
11.7
(b) Mask token position. Ap- (c) Cross-modal loss. Applying plying the mask tokens in the pre- the cross cross-modal loss for all inputs dictor works better than the en- is superior to applying only for coder.
(a) Masking strength in terms of starting index probability and mask length.
masked inputs.
Table 4: Masking ablations under the LRS3 low-resource setting using Base model.
8

---

Published as a conference paper at ICLR 2023
WER (%)
Predictor type
Params
VSR
ASR
WER (%)
None
coll.
coll.
Att dim
MLP dim
Linear
0.3M
64.4
17.7
VSR
ASR
MLP
4.2M
62.6
19.0
256
1024
44.0
13.3
Transformer 1 block
3.9M
46.7
14.1
512
2048
32.9
12.2
Transformer 2 blocks
7.4M
32,9
12.2
1024
4096
39.1
12.3
Transformer 4 blocks
14.2M
46.2
13.4
(b) Predictor width. A moderate predictor width works optimally.
(a) Predictor type. A lightweight Transformer pre- dictor works better than a linear layer or 2-layer MLP design. *Denotes representation collapse.
Table 5: Predictor capacity ablations under the LRS3 low-resource setting using our Base model.
Mask token position. We investigate in Table 4b the optimal position of the mask tokens. clear that better performance is achieved when applying them in the predictors rather than the encoders. Forgoing the use of mask tokens in the encoders leads to no input discrepancy between pre-training and fine-tuning, since mask tokens used during fine-tuning.
Mask loss. It is common to apply the loss only masked inputs for within- losses (Hsu et al., 2021] He et al. (021), since predicting targets corresponding to unmasked inputs may be trivial. However, this not the case for cross-modal prediction, where the targets are not related to the inputs in an obvious way. Indeed, Table 44c shows that applying the loss for both masked and unmasked inputs outperforms applying only for masked inputs.
Predictor design. Table 5a shows the effect of different predictor types. We note that using no predictors at all leads to representation collapse, where all outputs are constant (Grill et al.) [2020). We compare linear layers and 2-layer MLPs (applied independently to each encoder output) with Transformers of varying capacity. Following Grill et al. (2020), the MLPs have hidden dimension of 4096 with batchnorm followed by ReLU (Nair & Hinton] 2010) after the hidden layer.
We find that a Transformer predictor works significantly better, even when the number of parame- ters in the (1-block) Transformer and the 2-layer MLP is similar. Interestingly, the ability for the predictors to model temporal dependencies seems to be crucial to our representation learning phase.
two-block Transformer works optimally (Table 5a). A shallower Transformer likely results in representations too specialised to the pretext task. A deeper might place lot of the burden the predictors, leaving the encoder representations too general. Similar conclusions can be drawn regarding the Transformer width (see Table 5b). Overall, the lightweight predictor design means that using three predictors (one on the video student side and two on the audio) has relatively little effect on the total computational cost.
5 CONCLUSION
We presented RAVEn, single-stage method that jointly learns visual and auditory speech repre- sentations entirely from raw data. employs momentum encoders to generate targets, which, given masked inputs, are predicted by Transformer encoders and predictors. Especially salient to the quality of the representations are appropriate masking, lightweight Transformer predictors, and an asymmetric loss structure w.r.t. the visual and auditory modalities. RAVEn achieves strong perfor- mance under many settings without requiring multiple pre-training stages, handcrafted audio targets, separate pre-training strategies for VSR and ASR.
As future work, would be interesting to examine the effect of sharing weights between the visual way of reducing the memory requirements for pre-training. We would
and auditory encoders also like to apply RAVEn to other tasks related to speech. We hope our study inspires future research extending beyond speech recognition.

---

Published as a conference paper at ICLR 2023
ETHICS STATEMENT
Despite numerous positive applications, our method can also be misused. For example, lipreading technologies be employed for surveillance, compromising the public's privacy and trust. This problem is likely to be exacerbated as the quality of CCTV cameras improves over time. Appropriate government regulations will need be put in place limit such concerns. Another issue relates to the potential biases embedded in the datasets used. Such biases may have to do with gender, age, or ethnic backgrounds. specific group being under-represented in the data would likely result in reduced model performance samples belonging to said group. Making sure that the models are trained on balanced data, or using other bias-reduction techniques, can address such issues.
Our work used datasets that were made public for research purposes, i.e., LRS2, LRS3, and Vox- Celeb2. In particular, we used the cropped face .mp4 videos that were available on the official dataset webpages and which complied with the following licenses: Creative Commons BY-NC-ND 4.0 license and Creative Commons Attribution 4.0 International License, TED terms of use, and BBC's terms of use.
REPRODUCIBILITY
To ensure reproducibilityy we provide as many implementation details possible in the main paper as well tables showing the hyperparameter values the appendix. Moreover, we plan on making the code and pre-trained models publicly available.
ACKNOWLEDGEMENTS
Only non-Meta co-authors downloaded, accessed, and used the LRS2 dataset. Only non-Meta au- thors conducted any of the dataset pre-processing (no dataset pre-processing took place on Meta's servers or facilities).
REFERENCES
Triantafyllos Afouras, Joon Son Chung, Andrew Senior, Oriol Vinyals, and Andrew Zisserman. Deep audio-visual speech recognition. IEEE Transactions on Pattern Analysis and Machine In- telligence, 2018a.
Triantafyllos Afouras, Joon Son Chung, and Andrew Zisserman. LRS3-TED: a large-scale dataset for visual speech recognition. arXiv preprint 1809 00496, 2018b.
Triantafyllos Afouras, Joon Son Chung, and Andrew Zisserman. Asr is all you need: Cross-modal distillation for lip reading. In Proceedings of the 45th IEEE International Conference on Acous- fies, Speech and Signal Processing (ICASSP), pp- 2143-2147, 2020.
Humam Alwassel, Dhruv Mahajan, Bruno Korbar, Lorenzo Torresani, Bernard Ghanem, and
Du Tran. Self-supervised learning by cross-modal audio-video clustering. Proceedings of the 33rd Advances in Neural Information Processing System (NIPS), 33:9758-977 2020.
Relja Arandjelovic and Andrew Zisserman. Look, listen and learn. In Proceedings of the 16th IEEE/CVF International Conference Computer Vision (ICCV), pp. 609-617, 2017.
Relja Arandjelovic and Andrew Zisserman. Objects that sound. In Proceedings of the 15th European Conference on Computer Vision (ECCV), pp. 435-451, 2018.
Rosana Ardila, Megan Branson, Kelly Davis, Michael Kohler, Josh Meyer, Michael Henretty, Reuben Morais, Lindsay Saunders, Francis M. Tyers, and Gregor Weber. Common voice: A maasively-mullilingg speech corpus. In Proceedings of the 12th Language Resources and Eval- uation Conference, pp. 4218 4222, 2020.
Yuki Asano, Mandela Patrick, Christian Rupprecht, and Andrea Vedaldi. Labelling unlabelled videos from scratch with multi-modal self-supervision. Proceedings of the 33rd Advances in Neural Information Processing System (NIPS), 1844660-4677 2020.
10

---

Published as a conference paper at ICLR 2023
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450, 2016.
Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A frame- work for self-supervised learning of speech representations. In Proceedings of the 33rd Advances in Neural Information Processing System (NIPS), volume 33, pp. 2249-114400, 2020.
Alexei Bacvski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, and Michael Auli. Data2vec: A general framework for self-supervised learning in speech, vision and language. arXiv preprint arXiv:2202.03555, 2022.
Hangbo Bao, Li Dong, and Furu Wei. Beit: Bert pre-training of image transformers. arXiv preprint arXiv:2/06.08254 2021.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Proceedings of the 33rd Advances in Neural Information Processing System (NIPS), 33: 13:1:87744 1877-1901, 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the 18th IEEE/CVF International Conference on Computer Vision (ICCV), pp. 9650-9660, 2021.
Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. In Proceedings of the 18th IEEE/CVF International Conference on Computer Vision (ICCV), pp. 9640-9649, 2021.
Joon Son Chung and Andrew Zisserman. Out of time: automated lip sync in the wild. In Proceedings of the 13th Asian Conference on Computer Vision (ACCV), pp. 251-263, 2016
Joon Son Chung, Andrew Senior, Oriol Vinyals, and Andrew Zisserman. Lip reading sentences in the wild. In Proceedings of the 30th IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3444-3453, 2017.
Joon Son Chung, Arsha Nagrani, and Andrew Zisserman. Voxceleb2: Deep speaker recognition. In Proceedings of the 19th Annual Conference of International Speech Communication Association (INTERSPEECH), pp. 1086-1090, 2018.
Chung, J. Son Chung, and H. Kang. Perfect match: Improved cross-modal embeddings for audio- visual synchronisation. In Proceedings of the 44th IEEE International Conference on Acousties, Speech and Signal Processing (ICASSP). pp. 3965-3969, 2019.
Soo-Whan Chung, Hong-Goo Kang, and Joon Son Chung. Sceing voices and hearing voices: learn- ing discriminative embeddings using cross- moosl self-supervision. In Proceedings of the 1 Annual Conference of International Speech Communication Association (INTERSPEECH), pp- 3486-3490, 2020.
Kevin Clark, Minh- Minh-Thang Thang Luong, Quoe V Le, and Christopher D Manning. ELECTRA: pre-training text encoders as discriminators rather than generators. In Proceedings of the 8th International Conference on Learning Representations (ICLR), 2020.
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime G. Carbonell, Quoe Viet Le, and Ruslan Salakhutdi- nov. Transformer-xl: Attentive language models beyond a fixed-length context. In Proceedings of the 57th Conference of the Association for Computational Linguistics (ACL), pp. 2978-2988, 2019.
Jacob Devlin, Ming-Wos Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language
Technologies, NAACL-HLI, pp. 4171-4186, 2019.
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, An- drew Tulloch, Yangging Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in 1 hour. arXiv preprint arXiv: 2017.

---

Published as a conference paper at ICLR 2023
Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernaro Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self- learning. In Proceedings of the 33rd Advances in Neural Information Processing System (NIPS), volume 33, pp. 21271-212888 2020.
Alexandros Haliassos, Konstantinos Vougioukas, Stavros Petridis, and Maja Pantic. Lips don't lie: A generalisable and robust approach to face forgery detection. In Proceedings of the 34th IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5039-5049, 2021.
Alexandros Haliassos, Rodrigo Mira, Stavros Petridis, and Maja Pantic. Leveraging real talking faces via self-supervision for robust forgery detection. In Proceedings of the 35th IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 14950-14962, 2022.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the 29th IEEE/CVF Conference on Computer Vision and Pattern Recog- nition (CVPR), pp. 770-778, 2016
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. arXiv preprint arXiv:2111.06377, 2021.
François Hernandez, Vincent Nguyen, Sahar Ghannay, Natalia A. Tomashenko, and Yannick Estève. TED-LIUM 3: Twice as much data and corpus repartition for experiments on speaker adaptation. In International Conference on Speech and Computer, volume 11096, pp. 198-208, 2018.
Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:3451-3460, 2021.
Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger. Deep networks with stochastic depth. In Proceedings of the 14th European Conference on Computer Vision (ECCV), pp. 646-661, 2016.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Proceedings of the 32nd International Conference on Machine Learning (ICML), volume 37 of JMLR Workshop and Conference Proceedings, pp. 448-456, 2015.
Kazuki Irie, Albert Zeyer, Ralf Schlüter, and Hermann Ney. Language modeling with deep trans- formers. arXiv preprint orXiv:1905, 042266, 2019.
Bruno Korbar, Du Tran, and Lorenzo Torresani. Cooperative learning of audio and video models from self-supervised synchronization. Proceedings of the 31st Advances in Neural Information Processing System (NIPS), 31, 2018.
Taku Kudo and John Richardson. Sentencepiece: simple and language independent subword tokenizer and detokenizer for neural text processing. In Proceedings of the 56th Conference of the Association for Computational Linguistics (ACL). pp. 66-71, 2018
Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. In Pro-
ceedings of the 5th International Conference Learning Representations (ICLR), 2017.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In Proceedings of the 7th International Conference on Learning Representations (ICLR), 2019.
Pingchuan Ma, Rodrigo Mira, Stavros Petridis, Bjöm W. Schuller, and Maja Pantic. LiRA: Learning visual speech representations from audio through self-supervision. In Proceedings of the 22nd Anmual Conference of International Speech Communication Association (INTERSPEECH), pp- 3011-3015, 2021a.
Pingchuan Ma. Stavros Petridis, and Maja Pantic. End-to-end audio-visual speech recognition with conformers. In Proceedings of the 46th IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 7613-7617, 2021b.
12

---

Published as a conference paper at ICLR 2023
Pingchuan Ma, Stavros Petridis, and Maja Pantic. Visual speech recognition for multiple languages in the wild. arXiv preprint arXiv:2202.13068 2022.
Shuang Ma, Zhaoyang Zeng, Daniel McDuff, and Yale Song. Contrastive learning of global and 2021c.
local audio visual representations. arXiv preprint
Takaki Makino, Hank Liao, Yannis Assael, Brendan Shillingford, Basilio Garcia, Otavio Braga, and Olivier Siohan. Recurrent neural network transducer for audio-visual speech recognition. In Proceedings of the IEEE Automatic Speech Recognition and Understanding (ASRU) Workshop, pp. 905-912, 2019.
Brais Martinez, Pingchuan Ma. Stavros Petridis, and Maja Pantic. Lipreading using temporal con- volutional networks. In Proceedings of the 45th IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6319-6323, 2020.
Rodrigo Mira, Alexandros Haliassos, Stavros Petridis, Bjöm W Schaller, and Maja Pantic. Svts: Scalable video-to-speech synthesis. arXiv preprint arXiv:2205.020 02055 2022.
AbdeIrahman Mohamed. Hung-yi Lee, Lasse Borgholt, Jakob D Havtorn, Joakim Edin, Christian Igel, Katrin Kirchhoff, Shang-Wen Li, Karen Livescu, Lars Maalee, et al. Self-supervised speech representation learning: A review. arXiv preprint arXiv:2205.10643 2022.
Pedro Morgado, Nuno Vasconcelos, and Ishan Misra. Audio-visual instance discrimination with cross-modal agreement. In Proceedings of the 34th IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pp. 12475-12486, 2021.
Arsha Nagrani, Samuel Albanie, and Andrew Zisserman. Learnable pins: Cross-modal embeddings for person identity. In Proceedings the 15th European Conference on Computer Vision (ECCV), pp. 71-88, 2018a.
Arsha Nagrani, Samuel Albanie, and Andrew Zisserman. Seeing voices and hearing faces: Cross- modal biometric matching. Proceedings of the 31 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8427-8436. 2018b.
Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 17th International Conference on Machine Learning (ICMI., 2010.
Xichen Pan, Peiyu Chen, Yichen Gong, Helong Zhou, Xinbing Wang, and Zhouhan Lin. Lever- aging uni-modal self-supervised learning for multimodal audio-visual speech recognition. In Proceedings of the 60th Conference of the Association for Computational Linguistics (ACL), pp- 4491-4503, 2022.
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: An ASR corpus based on public domain audio books. In Proceedings of the 40th IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5206-5210, 2015.
Mandela Patrick, Yuki M Asano, Polina Kuznetsova, Ruth Fong, Joao F Henriques, Geoffrey Zweig, and Andrea Vedaldi. Multi-modal self-supervision from generalized data transformations. arXiv preprint arXiv:2003. 20t.:20000000498., 2020.
Stavros Petridis, Themos Stafylakis, Pingchuan Ma, Georgios Tzimiropoulos, and Maja Pantic. Audio-visual speech recognition with hybrid ctc/attention architecture. In 2018 IEEE Spoken Language Technology Workshop (SLT), pp. 513-522 2018.
K. R. Prajwal, Triantafyllos Afouras, and Andrew Zisserman. Sub-word level lip reading with visual attention. In Proceedings of the 35th IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
Alec Radford, Karthik Narasimhan, Tim Salimans. and Ilya Sutskever. Improving language understanding with unsupervised learning. 2018. [Online; accessed 20-September-2022).
Larguagg-uusupervik
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9 2019.
13

---

Published as a conference paper at ICLR 2023
Mirco Ravanelli, Jianyuan Zhong, Santiago Pascual, Pawel Swietojanski, Joao Monteiro, Jan Trmal, and Yoshua Bengio. Multi-task self-supervised learning for robust speech recognition. In Pro- ceedings of the 45th IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6989-6993, 2020..
Adria Recasens, Pauline Luc, Jean- Caan-Baptiste Alayrac. Luyu Wang, Florian Strub, Corentin Tallec, Mateusz Malinowski, Viorica Pätrăucean, Florent Altché, Michal Valko, et al. Broaden your views for sef-suueee video learning- In Proceedings of the 18th IEEE/CVF International Conference on Computer Vision (ICCV), pp. 1255 1265-1265, 2021.
Sucheng Ren, Yong Du, Jianming Lv, Guoqiang Han, and Shengfeng He. Learning from the master: Distilling cross-modal advanced knowledge for lip reading. In Proceedings of the 34th IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 13325-13333 2021.
Pierre Richemond, Jean-Bastien Grill, Florent Altché, Corentin Tallec, Florian Strub, Andrew Brock, Samuel Smith, Soham De. Razvan Pascanu, Bilal Piot, et al. Byol works even without batch statistics. arXiv preprint arXiv:2010. 2020.
Antony W Rix, John G Beerends, Michael P Hollier, and Andries P Hekstra. Perceptual evaluation of speech quality (pesq)-a new method for speech quality assessment of telephone networks and codecs. In 2001 IEEE international conference acoustics, speech, and signal processing. Proceedings (Cat. No. 01CH37221), volume 2. pp. 749-752. IEEE, 2001.
Dmitriy Serdyuk, Otavio Braga, and Olivier Siohan. Audio-visual speech recognition is worth 32x32x8 voxels. In Proceedings of the IEEE Automatic Speech Recognition and Understanding (ASRU) Workshop, 2021.
Dmitriy Serdyuk, Otavio Braga, and Olivier Siohan. Transformer-based video front-ends for audio- visual speech recognition. CoRR, abs/2201.10039) 2022.
Changchong Sheng. Matti Pietikäinen, Qi Tian, and Li Liu. Cross-modal self-supervised learning for lip reading: When contrastive learning meets adversarial training. In Proceedings of the 29th ACM International Conference on Multimedia, pp. 2456-2464, 2021.
Bowen Shi, Wei-Ning Hsu, Kushal Lakhotia, and Abdelrahman Mohamed. Learning audio- speech representation by masked multimodal cluster prediction. In Proceedings of the 10th Inter- national Conference on Learning Representations (ICLR), 2022.
B. Shillingford, Y. Assacl, M. W Hoffman, T. Paine, C. Hughes, U. Prabhu, H. Liao, H. Sak, K. Rao, L. Bennett, et al. Large-scale scale visual speech recognition. In Proceedings of the 20th Annual Conference of International Speech Communication Association (INTERSPEECH), pp. 4135- 2019.
Abhinav Shukla, Stavros Petridis, and Maja Pantic. Does visual self-supervision improve learning of speech representations for emotion recognition. IEEE Transactions on Affective Computing, pp. 1-1, 2021.
Cees H Taal, Richard C Hendriks, Richard Heusdens, and Jesper Jensen. An algorithm for intelligi- bility prediction of time-frequency weighted noisy speech. IEEE Transactions on Audio, Speech, and Language Processing, 19(7):2125-2) 2236, 2011.
Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. Videomae: Masked autoencoders are data- efficient learners for self-supervised video pre-training. arXiv preprint arXiv:2203. /2602, 2022.
Hugo Touvron, Matthieu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Hervé Jégou. Going deeper with image transformers. In Proceedings of the 18th IEEE/CVF International Conference on Computer Vision (ICCV), pp. 32-42, 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proceedings of the 31st Annual Conference on Newral Information Processing Systems (NIPS), pp. 5998 6008, 2017.
14

---

Published as a conference paper at ICLR 2023
Shinji Watanabe, Takaaki Hori, Suyoun Kim, John R Hershey, and Tomoki Hayashi. Hybrid ctc/attention architecture for end-to-end speech recognition. IEEE Journal of Selected Topics in Signal Processing, 11(8):1240-1253, 2017.
Shinji Watanabe, Takaaki Hori, Shigeki Karita, Tomoki Hayashi, Jiro Nishitoba, Yuya Unno, Nelson Enrique Yalta Soplin, Jahn Heymann, Matthew Wiesner, Nanxin Chen, Adithya Renduchintala, and Tsubasa Ochiai. ESPnet: End-to-end speech processing toolkit. In Proceedings of the 19th Annual Conference of International Speech Communication Association (INTERSPEECH), pp- 2207-2211, 2018.
Chen Wei, Haoqi Fan, Saining Xie, Chao- Wu, Alan Yuille, and Christoph Feichten- hofer. Masked feature prediction for supervid visual pre-training. arXiv preprint 1rXiv:2112.09133, 2021.
Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: A simple framework for masked image modeling. In Proceedings of the 35th IEEE/CVF 2022.
Conference on Computer Vision and Pattern Recognition (CVPR), pp.
Bo Xu, Cheng Lu, Yandong Guo, and Jacob Wang. Discriminative multi-modalit speech recogni- tion. In Proceedings of the 33rd IEEE/CVE Conference on Computer Vision and Pattern Recog- nition (CVPR), pp. 14433-14445 2020.
Qiantong Xu, Alexei Baevski, Tatiana Likhomanenko, Paden Tomasello, Alexis Conneau, Ronan Collobert, Gabriel Synnaeve, and Michael Auli. Self-training and pre-training are complementary for speech recognition. In Proceedings of the 46th IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 3030-3034, 2021.
Jianwei Yu, Shi-X Shi-Xiong Zhang, Jian Wu, Shahram Ghorbani, Bo Wu, Shiyin Kang, Shansong Liu, Xunying Liu, Helen Meng, and Dong Yu. Audio-visual recognition of overlapped speech for the Irs2 dataset. In Proceedings of the 45th IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 5984-6988, 2020.
Zi-Qiang Zhang, Jie Zhang, Jian-Shu Zhang, Ming-Hul Hui Wu, Xin Fang, and Li-Rong Dai. Learn- ing contextually fused audio-visus, representations for preprint XXXx:22202.07422 2022.
speech recognition. arXiv
15

---

Published as a conference paper at ICLR 2023
Hyperparameter
Value
Training epochs Warmup epochs Optimiser
150
40 (LRS3), 30 (LRS3+ AdamW
Learning rate Optimiser (91, 52)
3e-3 (Base), 2e-3 (Large) (0.9, 0.999) 0.04 Cosine decay
Weight decay
learning rate schedule
Drop path
0.05 (Base), 0.1 (Large) 5.0 RandomCrop + HorizontalFlip
Gradient clip
Video augmentations
Table 6: Pre-training settings.
A DATASET IMPLEMENTATION DETAILS
A.1 DATASETS
LRS3. LRS3 (Afouras et al. 2018b) is the largest publicly available transcribed audio-visual dataset for continuous speech recognition. consists of around 430 hours of spoken sentences from TED talks, with vocabulary of more than 50,000 words uttered by thousands of speakers. The test set contains around hour of utterances with speakers separate from those in the training set.
LRS2, The 223-hour LRS2 dataset (Chung et al. 2017), collected from BBC programmes, the second-largest publicly available transcribed audio-visual dataset for continuous speech recognition. As LRS3, it contains an unconstrained vocabulary and thousands of diverse speakers.
VoxCeleb2. VoxCeleb2 (Chung al., 2018) is non-transcribed dataset containing YouTube- downloaded videos. It consists of around 2,500 hours of utterances with over 6,000 speakers. Since VoxCeleb2 multi-lingual, as mentioned the main text, we use an English-only version curated by Shi et al. (2022). amounting to 1,759 hours.
A.2 DATASET PRE-PROCESSING
We follow common practices in the literature for dataset pre-processing (Ma et al., 2022 Shi al., 2022 Martinez et al. 2020). We crop a 96 x 96 region centred around the mouth and transform to grayscale. Raw audio is used without any pre-processing normalisation. Utterances longer than 24 seconds are split into smaller constituents.
A.3 PRE-TRAINING
Table provides the default setting for pre-training. We use the AdamW (Loshchilov & Hutter 2019 optimiser with linear learning rate warmup (Goyal al. 2017) and a cosine decay schedule
Loshchilov & Hutter 2017). During training, we apply random spatial cropping of size (88 x 88) followed by horizontal flipping with probability 0.5. These augmentations are applied in time-consistent manner across the video clips to maintain temporal coherence. We do not use any augmentations for the raw audio. We also use stochastic depth (Huang et al. (016) for regularisation, well as LayerScale (Touvron al.) 2021 with coefficient 0.I. We train the Base model with 32 A100 GPUs, and the Large model with 128. takes around 15 minutes hour per epoch to pre- train Base models on LRS3 LRS3+Vox2-oe and 2 hours per epoch for Large model on LRS3+Vox22
Our batching process is as follows: The samples are sorted based their length to minimise zero- padding, and then samples with the same length are randomly shuffled at the beginning of each training epoch. Each batch contains maximum of 96 36 seconds of footage for Base Large.
16

---

Published as a conference paper at ICLR 2023
Hyperparameter
Value
Training epochs
(HR), 50 (LR)
Warmup epochs Optimiser
20
AdamW
Learning rate encoder Learning rate decoder Optimiser (81,33)
2e-3
6e-3
(0.9, 0.98)
Weight decay
0.04 (HR), 0.1 (LR)
Learning rate schedule Layer-wise learning rate decay
Cosine decay 0.75 (HR), 0.5 (LR) le-5
Minimum learning rate
Drop path
0.2
Gradient clip
5.0
Video augmentations
RandomCrop + HorizontalFlip + TimeMask TimeMask
Audio augmentations
Decoder blocks
6 (Basc), 9 (Large)
Decoder hidden size
256 (LR), 512 (HR, Base), 1024 (HR, Large)
Decoder MLP size
1024 (LR), 2048 (HR, Base), 4096 (HR, Large) 4 (LR), 8 (HR, Base), 16 (HR, Large)
Decoder attention heads
Table Fine-tuning settings. HR denotes high-resource labelled data setting and LR low-resource.
A.4 FINE-TUNING / DECODING
The protocol for fine-tuning is similar that for pre-training with some exceptions (see Table 7 We use higher learning rate for the decoder than for the pre-trained encoder, since the decoder is randonly-inittalisee. We also employ layer-wise learning rate decay (Clark et al. 2020), which we apply
found to reduce overfitting. In addition to the augmentations from the pre-training stage, time masking (Ma et al., 2022) for both video and audio clips. Specifically, for each second of the sample, we use cero-masking with duration that is uniformly sampled between 0 and 0.4 seconds. Fine-tuning Base (for both VSR and ASR) takes around 30 seconds per epoch the low-resource setting and 7 minutes per epoch in the high-resource setting. Fine-tuning Large takes approximately minute and 20 minutes per epoch in the low- and high- setting, respectively.
CTCaattention decoding. We use joint CTC/attention decoding map the input sequence æ of length I" to target sequence = (y1,...., VL) of size L. The CTC loss during fine-tuning is given by
T [
L.x = log Σ
po (aa))))
AE:...y
where Ang is the set of valid alignments, A is such alignment, and a, the token time-step
The attention loss, computed using the Transformer decoder outputs, can be expressed
Luu
Σ 1=1
log (ulvi..... ,1,
(6)
The final loss during fine-tuning is given by L aLe + (1 Cats where a = 0.1.
We use the ESPnet framework (Watanabe et al. 2018) for decoding. We set the beam size to 40. The final score used to choose the most likely sequence is given by = XSex (1=)) Sae and Smn denote the scores from the CTC and attention branches, respectively, and X = 0.1. SLM an optional score from the language model, incorporated through shallow fusion (Watanabe et al.) 2017). When using language model, 39 is chosen from {0.1, 0.2, 0.3, 0.4) using the validation set.
where
Inference on one A100 GPU (without batching) takes around 3 seconds to decode 10 seconds of footage for Base and 5 seconds for Large.
17

---

Published as a conference paper at ICLR 2023
stage
filters
output size
stage
Bitters
output size
conv1
80, 64, stride
160T
conv1
x 7
stride 2x
X
pool:
x x stride
x 2
x 22 22
TCS1
3, 64
3, 64
x
1607
3 64
TC51
3 x 34
x
T 22 x
3. 128
105g
x 128
3, 128
x
1x11x11
re82
3, 128
807
3, 256
IC5g
3, 256
x
Tx6x6
ICS3
3, 256
2
40T
3, 512
resg
512
x
Tx8x3
TCS4
3, 512
x 2
20T
pool,
globul spatial average pool
Tx1x1
poolz
average pool, stride
(a) Visual feature extractor.
(b) Auditory feature extractor.
Table 8: Feature extractors, We provide details on the convolutional architectures used as the feature extractors for the visual and auditory modalities. Note that the output sizes for both networks match.
Language model. We 16-block Transformer-based language model, as proposed by Irie et al. (2019). The hidden size/MLP size/attention heads are 512/2048/8. The language model is trained on the combination of the following datasets: Librispeech (Panayotov et al., 2015). LRS2/3, TED-LIUM 3 (Hernandez et al. 2018). VoxForge and Common Voice (Ardila et al. 2020). The total number of characters is 166 million.
A.5 FEATURE EXTRACTORS
The details of the visual and auditory convolutional feature extractors provided in Tables 8a and respectively.
8b
B LRS2 EXPERIMENTS
We report results on the test set of the LRS2 dataset (Chung et al. 2017) in Table After pre-training on the LRS3 or the Vox2-en datasets, we fine-tune on the "pre-training" and "training" sets of LRS2. We observe similar trends as in the LRS3 experiments (Table namely that performance benefited from large models, large unlabelled datasets, and self-traming. We significantly out- perform all other methods, including one (Pan et al.) 2022) pre-trained on 60,000 hours of audio data.
C MORE ABLATIONS
C.I MORE PRE-TIAINING ABLATIONS
Momentum parameter. Table L0a shows the effect of varying the momentum parameter for up- dating the teacher networks. We show results at three coarse levels: 0 (teacher is copy of the student at each iteration), 0.999 (with a cosine schedule to 1), and (teacher does not get updated during training). We that using momentum value of ( leads representation collapse. At the other extreme, value of does not allow the teacher targets to improve during training, leading to poor representations. slowly-evolving momentum encoder most effective.
Pre-MLP normalisation. As mentioned in the main text, we use batchnorm before each MLP module rather than layernorm, choice inspired by Chen et al. (202I). Table 10b shows that our method still works with layernorm, but lags behind batchnorm. This is an interesting observa- tion worthy of future exploration. A preliminary hypothesis is that batchnorm may improve the
18

---

Published as a conference paper at ICLR 2023
WER (%)
Method
Encoder
LM
Unlab hours
Lab hours
VSR
ASR
supervised
Chung et al.
(2017)
LSTM
223
70.4
Petridis et
(2018
LSTM
380
63.5
8.3
Ren et al.
202T
Transformer
X
818
49.2
Yu al.
2020
CNN
223
48.9
6.7
Ma al.
202Ib
Conformer
380
37.9
3.9
Prajwal ct al. (2022)
Transformer
698
28.9
Ma ct al. (022)
Conformer
818
27.3
Prajwal et al. (2022)
Transformer
2,676
22.6
semi-supervised (using external models for pseudo-labelling
Afouras al. 2020)
CNN
777
223
51.3
Ma et al. 2022)
Conformer
641
818
25.5
superviid
Pan et
2022
Transformer
60,000
223
43.2
2.7
Ma et al.
2021a
Conformer
/
433
223
38.8
RAVEn
[Bas))
Transformer
433
223
32.1
3.9
RAVEn (Large)
Transformer
X
1,759
223
23.2
2.5
RAVEn (Large) w/ self-training
Transformer
1,759
223
19.3
2.3
RAVEn (Large) w/ self-training
Transformer
1,759
223
17.9
2.3
Table 9: LRS2 results. We report results on the test set with different model sizes and number of unlabelled data hours (Unlah hours). Lab hours denotes the number of labelled hours, and LM de- notes whether or not language model was used during decoding. "Includes non-publicly available data.
WER (%)
Momentum
WER (%)
VSR
ASR
Norm
VSR
ASR
coll.
coll.
0.999*
32.9
12.2
BN
32.9
12.2
75.0
74.0
LN
39.0
12.7
(a) Momentum parameter. It is important for the momentum encoders slowly evolve during train- ing. FFllows a cosine schedule with 0.999 as the starting value. Denotes representation collapse.
(b) Pre-MLP normalisation. Using batchnorm (BN) before the MLPs in the encoder works better than using layernorm (LN).
Table 10: More pre-training ablations under the LRS3 low-resource setting using our Base model.
19

---

and Published as a conference paper at ICLR 2023
WER (%)
Encoder LR
Decoder LR
VSR
ASR
WER (%)
Learning rate decay
same I.R
VSR
ASR
10-3
1 10
36.1
13.1
1.0
42.4
12.5
3 x 10-3
3 10
34.0
12.2
0.75
33.1
11.7
5 x 10-3
5 x 10-3
34.8
12.2
0.50
32.9
12.2
different LR
0.25
33.0
12.5
1 x 10
5 10
32.9
12.2
(b) Learning rate decay. Reducing the learning
as the encoder depth decrcases improves per- formance.
(a) Learning rates (LR) for encoder and decoder. Using larger learning rate for the decoder than the encoder is beneficial for VSR.
WER (%)
Tokens
LM
VSR
ASR
Character
35.8
13.6
Character
/
32.7
11.1
Subword
32.9
12.2
Subword
30.4
10.5
(c) Tokenisation. Using subword (SentencePiccee) units targets is better than using characters. A language model (LM) also helps.
Table 11: Fine-tuning ablations under the LRS3 low-resource setting using our Base model.
conditioning of the networks at initialisation, leading to better targets at the beginning of training RRihhemond et al. 2020
C.2 FINE-TUNING ABLATIONS
Learning rates. Table laa studies the effect of using different learning rates for the encoder and decoder. We find that it is beneficial for VSR use higher learning rate for the decoder than the encoder, likely because the encoder is pre-trained, whereas the decoder is thus requires larger learning rate to search for a more effective local optimum.
Learning rate decay. Table Tlb shows the influence of decaying the encoder's learning rate the depth of the model decreases. The learning rate at block b, rb, is given by TgdB-b, where B the index of the last block and is the learning rate decay (Clark al. 2022)). decay less than I works well for both VSR and ASR, suggesting that during fine-tuning it is useful to employ larger
learning rates for the deeper layers, which are more task-specific.
Tokenisation. In Table 1lc. compare the use of characters with SentencePiece (Kudo & Richardson 2018) subword units (vocabulary size of 1 1,000) as our target tokens. We find that using subword units leads superior results than character units. This may be explained by the language priors embedded in subword units, which facilitate speech recognition.
C.3 AUDIO-VISUAL FINE-TUNING
possible to use the learned auditory and visual representations for audio-visual speech recog- nition. To that end, following Ma et al. (2021b), concatenate the outputs of the two encoders, feed the resulting embeddings to a 2-layer MLP module with hidden size 4096 and batchnorm, and then fine-tune for speech recognition, as described in Section 3.2 We initialise the video and audio encoders with weights obtained from uni-modal fine-tuning and randomly initialise the rest. We fine-tune for 30 epochs with the hyperparameters used for audio-only fine-tuna fine-tuning, shown in Table
Table results in the low- and high-resource settings for the Large model with LRS3+ Vox2- en pre-training. Audio-visual training gives marginal improvements (if any) compared audio-only
20

---

Published as a conference paper at ICLR 2023
Fine-tune setting
Resource
WER (%)
Audio-only
Low
2.6
Audio-visual
Low
2.5
Audio-only
High
1.4
Audio-visual
High
1.4
Table 12: Audio vs audio-visual learning on LRS3 test set.. Including the visual modality for fine-tuning has very little influence clean conditions.
Model
Transcription
Groundtruth
So what do you think happens to these gals So what did the thing I would see this talk So what do you think happens this is an accident what you think happens these So what do you think happens these gals
VSR, low-resource, base, LRS3
VSR, high-resource, large, LRS3+ Vox2-en
ASR, low-resource, base, LRS3
outs
ASR, high-resource, large, Vox2-nn
Groundtruth
project really to find photographs that were taken before something photographs on which can before something process really find photographs that taken before something project really find photographs that taking before something project really to find photographs that taken before something
VSR, low-resource, base, LRS3
projects really has
VSR, high-resource, large, LRS3+Vox2-20
ASR, low-resource, base, LRS3
ASR, high-resource, large, LRS3+ Vox2-en
Groundtruth
Why
embedded social networks Why we admitted social networks
VSR. low-resuure, base, LRS3
VSR, high-resource, large, LRS3+ Vox2-en
Why are we embedded in social networks
ASR, low-resource, base, LRS3
Why are we in better social networks Why are we emhedded in social networks
ASR, high-resource, large, -RS3++oxx2een
Groundtruth
They won the game Because I wasn' happy
VSR, low-resource, base, LRS3
VSR, high-resource, large, JRS3+Vox2220
They want to come
ASR, low-resource, buse, LRS3
They want the game
ASR, high-resouree, large, L.R53+VoX2-0
They won the game
Groundtruth
Has it gotten better
VSR, low-resource, base, LRS3
It's not better It's gotten better
VSR, high-resource, large, LRS3++oo2-200
ASR, low-resource, base, LRS3
it gotten better
ASR, high-resource, large, Vox2-en
Has it gotten hetter
Table 13: Transcription errors.
when the audio is clean (i.e., not noisy), consistent with findings Ma et al. (2021b). Investigating the impact of audio-visuu training and testing on audio corrupted with various noise types, is outside the scope of this work, but interesting to take up future work.
D ANALYSIS OF TRANSCRIPTION ERRORS
We provide examples of transcription errors in Table 13 We consider the Base VSR and ASR models fine-tuned in the low-resource setting (47.0% and 4.7% WER, respectively) and the Large VSR and ASR models fine-tuned in the high-resource setting with self-training and language model (23.13 % and 1.4% WER, respectively). Although the worst model sometimes makes surprising errors (e.g., "They won the game" "Because I wasn't 't happy"), most often the emrors related to words that are phonetically similar (e.g., "were taken" - "we're taking", "embedded" - "in better", "wonn" "want"). As expected, the quality of the transeriptions is higher for ASR than VSR, and it improves as we increase the model size and number of unlabelled data points.
E RAVEN FOR VIDEO-TO-SPEEC SYNTHESIS
We evaluate here whether RAVEn pre-training can benefit tasks other than speech recognition by considering video-to-speech synthesis, which aims to output the speech waveform given the corre- sponding silent video of lip movements. We follow the protocol of SVTS (Mira et al. 2022 the
21

---

Published as a conference paper at ICLR 2023
Method
Encoder
Training dataset
PESQ
STOI
ESTOI
Mira et 2022)
Conformer
LRS3
1.25
0.51
0.27
Mira et al.
(2022)
Conformer LRS3+Vox2242
1.26
0.53
0.31
SVTS from scratch
Transformer
L.RS3+Vo22-B8
1.26
0.53
0.30
SVTS w/ RAVEn
Transformer
VRS3+vx
1.30
0.56
0.36
Table 14:
Video-to-speech synthesis on LRS3 test set. Using RAVEn pre-training for the video
encoder provides significant boosts in performance.
current state-of-the-art method, to perform video-to-speech the LRS3 test set after training LRS3+Vox2-en. We study the effect of initialising the video encoder with the weights from our Large pre-trained model. We the same hyperparameters as Mira et al. (2022), except that
a learning rate 2e-4 and train only for 30 epochs (rather than 150). We train the "from scratch"
baseline with learning rate of 7e-4 for 50 epochs, as training longer resulted in overfitting.
Table 14 reports performance based on PESQ Riix et al. 2001), which measures the perceptual quality the generated samples, as well as STOI and ESTOI (Taal et al. 2011), which measure their intelliggblitty The results show that using RAVEn pre-training results in significant improvements performance, compared with training from scratch, across all metries. In fact, the performance boosts due to RAVEn pre- pre-traine training (0.04 / 0.03 / 0.06 for PESQ STOI ESTOI) are larger than those observed by Mira al. (2022) (0.01, 0.02, 0.04) when increasing the training set size by around factor of 4 (from LRS3 to LRS3+ V-x2-en).
22


Reviews:
[{"review_id": "zDjRVdGgep", "reviewer": "Reviewer_ykA5", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper explores a method for pretraining an Audio-Visual speech recognition model directly from raw video with both audio and visual signals. It does so by asymmetrically applying two student-teacher networks 1. the audio student learns to predict both audio and visual targets generated by respective teachers, 2. the video student only learns to predict the audio features. The rationale for this is that the audio provides a much stronger signal than the video. The results are competitive with state-of-the-art for self-supervised learning with small amount of fine-tuning data, e.g. for VASR on TED-LRS3 24.4% better vs 26.9% (Shi ICLR 2022), and for AVASR on TED-LRS3 1.9% is worse than the 1.3% in Shi ICLR 2022.", "summary_of_the_review": "Overall the paper makes a nice contribution to the growing area of unsupervised training on Audio-Visual data. They provide an incremental way of training an AV model in a simpler model than in the past and achieve results comparable to the state-of-the-art so I support accepting this paper for publication at ICLR.", "strength_and_weaknesses": "Strengths:\n- the paper presents a different approach to previous work on learning audio-visual representations in an unsupervised manner compared to the work in Shi ICLR 2022. In Shi's AV-HUBERT, they required a complicated multistage process of first using MFCC features to cluster, before learning an AV representation from raw data. In this work, a one-stage training regime is achieved. \n- the provides extensive analyses on the various aspects of their approach. of note, the ablations in Table 3 nicely motivate the use of asymmetric student-teacher training for best results.\n- the paper achieves state-of-the-art results for unsupervised training for visual-speech-recognition on TED-LRS3\n\nWeaknesses\n- from a certain point of view, the paper is incremental over Shi's ICLR 2022 work. it achieves similar results, however in a simpler manner. \n- the difference of 1.9% in this paper for TED-LRS is a significantly off from the 1.3% found in Shi 2022.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "YQ8MghV7HLi", "reviewer": "Reviewer_XgXU", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a merthod called RAVEn, which is a self-supervised multi-modal approach to jointly learn visual and auditory speech representations. The paper uses masked inputs and momentum encoders. The proposed approach is asymmetric in terms of the two modalities and their pretext tasks. Strong results are obtained. ", "summary_of_the_review": "Please see my reviews above", "strength_and_weaknesses": "Strengths:\n\n1- The paper tackles an important and timely problem, and the solution is relatively novel.\n\n2- The paper is well-written and structured.\n\n3- It achieves strong results.\n\n4- Implementation details and hyperparameters are well-presented, and the authors aim to make the code public.\n\nWeaknesses:\n\n1- Some editing issues are still present in the paper. For example, the heading \"Introduction\" is missing after the abstract.\n\n2- The datasets that have been used could have been expanded \n\n3- As the paper mentions, the setup is asymetric in terms of the two modalities, i.e., the auditory stream predicts both the visual and auditory targets, but the visual one predicts only the auditory targets. It is perfectly fine to have a final solution thagt is asymetric. But it would have been nice to include experiments, e.g. cross-modal + video within modal is missing.\n\n4- I understand the novelty, but it would have been nice to include a bit more intuition regarding the results and experiments to show why the performance is obtained. E.g., how does the cross-modal loss result in better representations?\n\n5- From what I understand, the downstream is done on multimodal. How is the performance on unimodal? I.e., does the teacher-student approach result in better unimodal learning too?\n\n* Update: after the rebuttal, I have now increases my score.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "EF2WmaKKGg", "reviewer": "Reviewer_CL4k", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The focus of this work is learning representations of auditory and visual speech from the underlying raw signals, as opposed to learning a representation from hand-crafted features as is typically done.  The experiments include an ablation over design choices.  The combination of within-modality modeling (for audio) and cross-modality modeling (for audio and vision) provide a powerful representation, as demonstrated by applying to automatic speech recognition and visual speech recognition.  The results are impressive and set a new state of the art.", "summary_of_the_review": "This paper proposes new representations of auditory and visual speech that lead to very low WERs in each domain.  The results are impressive and the experiments clearly test the contribution of the different components of the system.", "strength_and_weaknesses": "+The paper is well written.\n\n+Thorough evaluation of the approach.\n\n+Impressive results beating existing state of the art.\n\n-The main text refers the reader to Appendix A.3 for details of the language model, but there is no discussion of what the language model actually is.  There is a term in the overall loss,  but it is not clear how L_lm is calculated.\n\n-Are numbers available about run-time complexity?\n\n-The learned representations are applied only to A/V speech recognition -- it would be useful to show their utility in a wider range of AV tasks (although this is listed as future work),\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "CTeHxpqK54", "reviewer": "Reviewer_Hokd", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors propose a self-supervised multi-modal approach: RAVEn to jointly learn visual and speech representations. The key components in RAVEn take good practices (e.g., masked prediction, Transformer, and Knowledge Distillation) in existing self-supervised learning approaches. But differently, it leverages audio-visual cross-modal correspondence to train the teacher-student learning model. Extensive experiments show that RAVEn can help to learn powerful visual and auditory speech representations for visual and auditory speech recognition. ", "summary_of_the_review": "Most of the key components in the proposed method, such as SSL with Knowledge Distillation, masked prediction, momentum-based teacher, and audio-visual SSL, can be found in previous approaches. Due to the limited technical novelty, my current rating for this paper is reject. \n\n***Post-Rebuttal***\n\nThank the authors for responding to my questions! Most of my concerns have been clarified by the rebuttal. Although the technical contributions are incremental, the proposed RAVEn can be a strong baseline for joint visual-speech representation learning. \n\nMy rating is upgraded to 6 from 3.", "strength_and_weaknesses": "Pros:\n\n\\+  The proposed cross-modal learning mechanism is well-motivated. Considering that audio contains more information relevant to speech than visual, the authors propose a learning strategy that accounts for the modality difference. Its audio student predicts outputs from both audio and video teacher, whereas the video student predicts only audio teacher outputs. \n\n\\+ Extensive experiments and ablation studies in terms of ASR and VSR tasks can validate the effectiveness of the proposed multi-modal self-supervised approach.\n\n\nCons:\n\n\\- The technical novelty is limited. I did not find any novel and significant technical components in the proposed approach. The key designs in the proposed approach, such as SSL with Knowledge Distillation, masked prediction, and momentum-based teacher, can be found in previous works. To me, the proposed RAVEn is like an audio-visual version of the Self-Supervised Vision Transformer: DINO [1]. \n\n[1] Mathilde Caron, Hugo Touvron, Ishan Misra, Herv´e J´egou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the 18th IEEE/CVF International Conference on Computer Vision (ICCV), pp. 9650–9660, 2021.\n\n\\-  In real-world videos, audio and visual content are not always matched. For example, a speaking person can be out of the screen. How can the proposed method defend against the issue?\n\n\\- Since the proposed method can jointly learn audio and visual representations, it is straightforward to perform audio-visual speech recognition using the learned representations. But, in the paper, only ASR and VSR are explored. In addition, we only see very small improvements in the ASR task (LRS3 high-resource setting).  ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster