u-HuBERT: Unified Mixed-Modal Speech Pretraining And Zero-Shot Transfer to Unlabeled Modality
Authors: Wei-Ning Hsu
Bowen Shi
Venue: NeurIPS 2022 Accept (2022)
Keywords: multimodal speech
audio-visual speech
multimodal self-supervised learning
zero-shot
speech recognition
speech translation

Abstract: While audio-visual speech models can yield superior performance and robustness compared to audio-only models, their development and adoption are hindered by the lack of labeled and unlabeled audio-visual data and the cost to deploy one model per modality. In this paper, we present u-HuBERT, a self-supervised pre-training framework that can leverage both multimodal and unimodal speech with a unified masked cluster prediction objective. By utilizing modality dropout during pre-training, we demonstrate that a single fine-tuned model can achieve performance on par or better than the state-of-the-art modality-specific models. Moreover, our model fine-tuned only on audio can perform well with audio-visual and visual speech input, achieving zero-shot modality generalization for multiple speech processing tasks. In particular, our single model yields 1.2%/1.4%/27.2% speech recognition word error rate on LRS3 with audio-visual/audio/visual input.

u-HuBERT: Unified Mixed-Modal Speech Pretraining And Zero-Shot Transfer to Unlabeled Modality
Wei-Ning Hsu Meta AI wnhsuêmeta. com
Bowen Shi
Meta AI bshi@meta..m com
Abstract
While audio-visual speech models can yield superior performance and robustness compared to audio-only models, their development and adoption are hindered by the lack of labeled and unlabeled audio-visual data and the cost to deploy model per modality. In this paper, present HuBERT, self- supervised pre-training framework that can leverage both multimodal and unimodal speech with unified masked cluster prediction objective. By utilizing modality dropout during pre-training, we demonstrate that a single fine-tuned model can achieve models. performance par better than the state-of-the-ar modality-specie Moreover, our model fine-tuned only on audio can perform well with audio-visuse and visual speech input, achieving gero-shot modality generalization for multiple speech processing tasks. In particular, our single model yields 1.2% 99%/2.2%% speech recognition word error rate LRS3 with audio-visullaadiivvssuu input.'
Introduction
Speech processing has been moving from developing domain domain-specinf specific models toward building general model for all domains. For example, early studies presented models dedicated for robust speech recognition [Kingsbury al., 1998] and distant speech recognition [Feng et al., 2014. Kim et al., 2017] by taking into account acoustic properties for each domain. In contrast, recent work has built general domain models by pooling data from different domains and even different languages, including labeled ones for supervised learning [Chan et al., 2021, Likhomanenko et al., 2020, Pratap al., 2020] and unlabeled ones for self-supervised learning [Kawakami et al., 2020, Hsu al., 2021b, Conneau et al., 2020]. These studies show that in addition to removing the need of deploying and maintaining multiple models, a single general domain model can in fact achieve performance on par or superior to domain-specift models, especially for domains with less data, and improve generalization performance to unseen domains.
In contrast, there are few attempts and even fewer successful ones on developing versatile model that can process many speech modalities [Makino et al., 2019, Sari et al., 2021]. general, researchers develop models with different input speech modalities, such as visual speech [Chung and Zisserman, 2016], audio-visual speech [Xu et al., 2020], multi-channe speech [Haeb- Umbach al., 2021], 3D talking faces [Richard et al., 2021], Electromyographic (EMG) muscle signals [Diener and Schultz, 2018), ultrasonic oral cavity scan [Hueber al., 2010], or ultrasonic microphone [Livescu al., 2009] in isolation, despite the fact that many of them are used to solve the same task. Moreover, while additional modalities, such as visual speech, can often boost the performance and robustness on top of audio speech, models taking such input are less studied and deployed in real world application. This is mainly due to the lack of both labeled and unlabeled data for these alternative modalities, hindering the development of high-perrformin self celf-supervieed or supervised modal-specific models that are
Codes and models are available at htttps///gituuu..cm/aa/acoooooee
hubert.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Pseudo labeling
Multimodal/ / unimmdall input
Modality agnoosicc features
Pseudo labels
With NMW
U-BUBERT
Joint clustering
[13,13,5,5,3] yo
AND/OR
U-HUBERT
5,,35, Y
X
U-HUBERT
14,4,4,9,5,51
Y
Figure 1: u-HuBERT learns modality-agnon features that are clustered to produce pseudo labels with shared codebook for different speech modalities. See Section 3.2 for more information.
trained exclusively on one type of input. We argue that to tackle the data scarcity issue, essential to utilize not only the speech from the target modality, but also speech in other modalities to build general speech model for many modalities. By doing so, more data can used to learn components shared across different modalities, which is expected to bring bigger gains to low-resource modalities.
Audio-visual HuBERT (AV-HuBERT) [Shi al., 2022a] recent work progressing toward creating a versatile speech model: by pre-training on audio vvdial speech, it demonstrates strong performance for not only audio-visual downstream tasks, but also audio-only and video-only tasks. However, it only explores pre-training exclusively on audio-visual speech, and fine-tuning and testing on matching modalities. In this paper, we generalize AV-HuBERT and present unified hidden unit BERT (u-HuBERT), which leverages unlabeled speech data of many different modalities for pre-training, including both unimodal and multimodal speech. The model trained with a unified masked cluster prediction objective for both unimodal and multimodal speech data. The pre-trained u- u-HuBERT can be fine-tuned with either multimodal or unimodal data, and we demonstrate that in both cases the resulting model can be used for many speech modalities. Following AV-HuBERT, modality dropout [Neverova et al., 2014] to multimodal data to simulate different combinations of input modalities. We present extensive analysis in this paper show that modality dropout is crucial for learning modality-agnostie frame-level representation, which the key to successful mixed modality pre-training and achieving good performance when fine-tuning and testing modalities mismatch. We verify the effectiveness on two popular speech processing tasks: speech recognition and speech translation. To the best of our knowledge, this is the first single model for many speech modalities that yields performance on par with the state-of-the-art modality-specific models and is capable of zero-shot modality generalization.
apply
2 Related Work
Comparison to [Shi et al., 2022a] u-HuBERT generalizes AV-HuBERT to utilize both multimodal and unimodal speech for pre- pre-training by mapping various input to modality-agnosti feature space and creating a shared target space for masked prediction. In contrast, because AV-HuBERT is audio-visual speech, it always generates targets using features of the same
pre-trained exclusively modality (audio in the first iteration and audio-visual for the subsequent iterations). Therefore, learning modality-agnostii features to produce shared target space is not essential for AV-HuBERT.
Moreover, this paper presents novel evaluation scenario: zero-shot transfer to unlabeled modalities, which is of great practical importance because while labeled data may not always be available in the target modality, it will most likely be available some other modalities. We demonstrate this paper that modality dropout is key to achieving good zero-shot transfer performance.
Joint unimodal and multimodal pre-training Recent cross-modal contrastive approaches such as CLIP [Radford al., 2021] and ALIGN [Jia al., 2021] have shown impressive results yet they can only be trained with multimodal data and require a large-scale dataset, which can be hard to obtain for many speech modalities (e.g., paired EMG-audio speech). FLAAVA [Singh et al., 2021] and SLAM [Bapna et al., 2021] are two frameworks that utilize both unimodal and multimodal data
2 AV- HuBERT predicts the same audio-visual cluster targets C regardless of whether modality dropout is activated. See the top half of Figure 2.

---

for image-text and speech- text pre-training, respectively. There are three key differences between u-HuBERT and these two models. First, for multimodal speech, different modalities such as audio and visual speech are temporally aligned. Instead of concatenating the feature sequences from unimodal encoders, we fuse the features frame-by-frame while all three models utilize masked token prediction objective, u- u-HuBERR considers shared target token space for all unimodal and multimodal data, obtained by clustering modality-agnostie features extracted from the u-HuBERT model itself. In contrast, SLAM uses w2v-BERT [Chung et al., 2021] tokens for speech and SentencePiece [Schuster and Nakajima, 2012] for text. FLAVA on other hand uses separately trained dVAE tokenizer for image and the same as SLAM for text, and it predicts tokens for unimodal and multimodal data at the output of different modules, differing from u- HuBERT. In addition, both FLAVA and SLAM pre-train with number of auxiliary objectives, while u-HuBERT only uses one, which simplifies hyperparameter selection. Third, different modalities of speech are often used to solve the same task. Our evaluation focuses building a single model that can solve the same task with different modalities despite not having labeled data for each modality, which is not considered by SLAM or FLAVA.
before passing it to the shared encoder. Second,
Single model for many input modalities Makino et al. [2019] explored building a single model for supervised sitiiio-iisullluuiovviiii comparing unified models with modality-specific ones. Furthermore, these models are single-task supervised and hence not capable of zero-shot modality transfer when trained on unimodal data. Omnivore [Girdhar et al., 2022] is a single supervised model for many vision modalities, including image (RGB). 3D image (RGBD), and video (RGB sequence), which also considers a model archi- tecture with modality-speciin backend [Liu et al., 2021]. Both Omnivore and u-HuBERT demonstrates cross-modal generalization. but Omnivore achieves it through supervised learning with different tasks, and u- u-HuBERT achieves with unified self- superviued learning. Finally, if we considered different languages as different modalities, our work is also related to mBERT [Devlin et al., 2019], mBART [Liu et al., 2020], TLM [Yao et al., 2021], and XLM [Lample and Conneau, 2019], fine-tuning which enables generaliz- ing natural language understanding [Pires et al., 2019] and machine translation [Johnson et al., 2017] to languages without labeled data. Pre-training on multimodal speech analogous to pre-training on bitext, while the former requires less manual effort to obtain by collecting more measurements when speech is produced. In addition, fume-synchronicity between modalities provides additional self-supervision, enabling u-HuBERT to more casily learn frame-level modality- agnostic features.
speech recognition. The studies show significant degradation
encoders (RGB and depth) followed by shared Swin Transformer
3 Method
3.1 Background: audio-visual HuBERT
We build our framework upon AV-HuBERT [Shi et al., 2022a], a self-supervised representation learning model designed to pre-train a single type of multimodal speech data; audio-visual speech. AV-HuBERT first encodes temporally aligned audio speech X" RTxXC. and visual speech X" E x into feature sequences ha E R xD' and h² E Tx of the same lengths, which are then fused frame-by-frame FUSE(h" h²) E R and processed by Transformer backend [Vaswani et al., 2017] to produce contextualized audio-visual representations eav E R xD:
This model is trained by iterating two steps. First, K-means clustering model is trained on some audio-visual features to produce frame-level targets yau E [1, K]T, where K is the number of clusters. Second, the AV-HuBERT model is trained with a masked cluster prediction task, where spans from the audio-visual input are randomly masked and the model learns to predict the clusters of the masked frames given the context. The model is foreed to learn the underlying linguistic structure of audio-visual speech to infer the content in the unseen regions based on the context. The intermediate feature of a trained model, which is better than previous iteration, is used to produce targets for the subsequent iteration.
3.2 Self-supervised pre-training with many modalities
We extend the framework to utilize not only multimodal data, but also unimodal data that can be more casily obtained in large quantities. The main idea is to apply modality dropout to multimodal
3

---

Pre-training (predict pseudo labels of highlighted frames)
With muttimodal data
Shared
-
Linear [13.33,5.5.5]
KiN Huih iddu HHl)
Audio encoder
оаиио N
FOSE A
Yo
Drapout flsw
OR
Shared Transformarr
||||-
Linear Softmax
(13.33.5.5.51
Video encoder
DOMN h²
OR
FUS 0
CH
y
Shared Transformer
||||-
Linsar Softmax
5,
FUSE
Y
With unimodal data (e.g. audio)
Heve
Audio encoder
Dummy
Sune
Shared Transfoormer
Linear+ Softmax
132.5.5.33
A
PU 0)
Y
Figure u- u-HuBERT pre-trains on multimodal and unimodal speech with unified objective: predicting cluster assignments of the frames that are masked at the input (highlighted by yellow shades). For multimodal data, subset of modalities are randomly selected to predict the same targets.
speech along with shared Transformer (Vaswani al., 2017] to learn modality-agnostr features that can be quantized by a shared codebook to produce pseudo labels for masked prediction.
Given unimodal and multimodal speech data from many modalities, assume that there is anchor modality that is paired with all other modalities, For simplicity, assume the anchor modality is audio, and we have unlabeled audio, visual, and audio-visual speech. In the first iteration, because modality- agnostic feature extractor is not available yet. we cluster the anchor modality features to produce targets for unimodal and multimodal speech that contain the anchor modality. Concretely speaking, this produces pseudo paired data for audio speech X and audio-visue speech (X", X") with audio- based cluster assignment Y as the target. Like AV-HuBERT, each modality has its feature extractor: l for audio and l for video, and feature sequences are fused -r-mamby before being passed onto the Transformer g to produce contextualized features e g (FUSE(({(X*) J" (x"X"))() When a modality is absent, its feature sequence is simply replaced by zero vectors, for example, c² = g (FUSE( (X')), 0)). Following HuBERT [Hsu et al., 2021a], this model is trained with a masked prediction objective.
To encourage the model to learn podallty-aggostio features, modality dropout is applied to multimodal data to randomly drop a subset of the modalities, effectively creating multiple copies of the data with the same target but different input modalities, depicted in the top half of Figure 2. While modality dropout was also used in AV-HuBERT, it was implemented for a different reason to reduce the discrepancy between pre-training and fine-tuning for unimodal downstream tasks, which resulted in only minor improvement (reducing the WER from 57.0% to 55.3% as shown in Table D. in Shi al. [2022a]). In contrast, we will demonstrate that modality dropout is essential for mixed modality pre-training and zero-shot generalization.
After the first iteration, the model can be used to extract modality- agnostic features for all unimodal and multimodal data, which are clustered to produce the pseudo label for next iteration of pre-training as illustrated in Figure 1. For multimodal data like audio visual speech, one produce pseudo labels using features extracted from the complete multimodal input (e.g.. you from audio- audio-visual input), those extracted from input with subset of modalities (e.g.. Y" using only the video stream). We opt for the former and study the impact of these choices in Secton 4.2.
3.3 Multimodal and unimodal fine-tuning with partial modalities
Once the model is pre-trained, we remove the cluster prediction head (the linear layer on top of transformer used to predict cluster assignment distribution) and add a downstream task-specific prediction head. The pre-trained model can be fine-tuned on labeled multimodal speech, unimodal speech, or speech with mixed modalities. More importantly, the modalities included in the fine- tuning data do not necessarily cover the modalities seen during pre-training, yet the fine-tuned model can still handle the downstream task on all pre-trained modalities, which we refer to as zero-shot modality generalization. This because the pre-trained model has successfully learned modality- agnostic representations, where same speech manifested in different modalities are mapped to similar

---

representations. Thus mapping from the representation of one modality (e.g,, audio) to a target output (e.g., text) can be directly applied to other pre-trained modalities (e.g.. video).
Similar to mBART [Liu et al., 2020] for zero-shot machine translation, catastrophic forgetring (Lee et al., 2019, Üstün et al., 2021] may occur during fine-tuning and hence hinders its generalization across modalities. Particularly, this is because the distributions of Transformer input and output at earlier layers are less modaaity To alleviate the issue, we apply modality dropout when multimodal data is used, and explore a few techniques to prevent the representations from drifting too much, including freezing the entire pre- pre-trained model or freezing a number of initial layers.
Experiments
4.1 Pre-training setup
We evaluate u- u-HuBERT on audio, visual, and audio- visual speech, where larger quantities of unlabeled and labeled data are available. Two audio-visual and one audio-only datasets are used for pre-training: (1) LRS3 [Afouras et al., 2018] 8] with 433 hours of English audio-visual speech, (2) VoxCeleb2-En (VC2-En) with 1,326 hours of English YouTube audio-visual speech filtered from VoxCeleb2 [Chung et al., 2018] by Shi et al. [2022a]. and (3) TED-LIUM release 3 (TD) [Hernandez et al., 2018] with 452 hours of English audio collected from the same domain as LRS3, To improve noise robustness. we apply online noise augmentation where each utterance is corrupted with additive noise at OdB sampled from MUSAN [Snyder et al., 2015] with a probability of 0.25. More details can be found in Appendix A. These datasets are solely used to perform experiments for the purpose of comparing with previous work in a fair setup.
We adopt the AV-HUBERT-LARGE model Shi et al. [2022a]²: the video encoder is a modified Re model [He al., 2016) that takes image frames of 88x88 pixels sampled at 25Hz at input and produce a sequence of 512 dimensional embeddings at the same frame rate; audio is represented as filter bank features sampled at 100 Hz, stacked every four frames to match the video frame rate, and projected with a linear layer to 1024-D features. FUSE concatenates features crame-by-frang and linearly projects them to the embedding dimension of the shared Transformer. The shared Transformer has 24 layers, with 16 heads, embedding dimension 1024, and FFN dimension 4096, using the pre-norm residual connection setup [Nguyen and Salazar, 2019].
To directly compare with the publicly available fifth iteration noise- auise-augmented AV-HUBERT-LARGE model trained on LRS3 and VC2-En, the same feature extractor (fourth iteration BASE model) and the same codebook (learned from multimodal features (aar) are used to generate pseudo labels for multimodal LRS33 and VC2-Fn and unimodal TD. We then pre-train u-HuBERT on the combined data for 1M updates on 64 32GB V100 GPUs using the Adam optimizer [Kingma and Ba, 2015] and a learning rate of 0.002. Gradient norm is clipped at 1.0. to stablize training. A batch size of maximal 40 seconds per GPU is used. When preitrained on multimodal data, audio and video are dropped with probability of 0.25 and 0.25, respectively.
4.2 Learning modality-agnosti features and codebooks with modality dropout
We argue that modality dropout leads to learning modality-agnostic representations through forcing the model to predict the same target with different subsets of input modalities given multimodal input, which in turn enables construction of a shared codebook for all modalities. To verify this, we compare two models that are identical except for whether modality dropout is activated: the public noise-augmented AV-HuBERT LARGE model trained on LRS3 and VC2-En with modality dropout and its counterpart. For each model, we infer audio, visual, and audio-visual features for each LRS3 validation utterances, randomly sample 500 frames, pool the features, and run t-SNE [Van der Maaten and Hinton, 2008] dimension reduction for visualization (Figure 3). It is visually evident that without modality dropout, features extracted from different modalities have different distributions (bottom row), contrary to those from the model with modality dropout (top row).
We next adopt the phone normalized mutual information (PNMI) metric presented in Hsu et al. [2021a] to quantify cluster quality. For each model, we create four codebooks by clustering audio-
$ httpps///giihuu..oo//toeboboo noise noise_large_ve large_vorg www_oiterr....eeeeeeeeeeeeeeeeee yaml
5

---

visual features Ca", audio features C", visual features C", or all three combined Um C"T, quantize each type of features with each codebook, and measure the normalized mutual information E [0. 1] between frame-level phone labels and cluster assignment (Table 1). Because features inferred from different modalities exhíbit similar distributions with modality dropout, all four codebooks lead to similar PNMI for given feature. In contrast, without modality dropout, cross- tross-quantization (e.g., Yau with C codebook) often leads to significantly worse PNMI, and PNMI worse for all (codebook, feature) combinations compared with that from the modality dropout model. Note for given codebook, the PNMI still differs among modalities which due to the discrepancy of information carried per modality.
Input
Input
Input
wio PT mad drop wy PT mod drop
K-means training
PNMI (1)
PT mod-drop
w/o mod-drop
data
yev
Ya
Yv
yav
Y
Y
Um Cm
43.60
43.72
38.01
41.89
36.59
21.78
Cue
42.89
42.96
36.89
42.83
38.46
26.17
C
43.18
43.28
37.11
42.87
37.28
20.99
C
42.23
42.57
37.68
6.85
13.68
23.54
Table 1: Cluster quality (PNMI) of each modality yau ///////YY yY/Ye using codebooks learned from features of different modalities, obtained from models pre-trained with and without modality dropout.
Figure 3: t-SNE projection of AV/A/V speech representations learned by models differing in modality dropout.
4.3 Speech recognition
Fine-tuning setup LRS3 trainval and pretrain, combined for 433 hours, are used for training, with the same 1,200 utterances as Shi et al. [2022a] split for validation. Results on the test split are reported. Noise augmentation with MUSAN used pre-training also adopted during fine-tuning. 9-layer randomly initialized Transformer decoder with 8 attention heads and 1024/4096-D embedding/FFN appended to the pre-trained HuBBER and the entire model fine-tuned with a cross-entropy loss predicting the next text token given the input and the previous text tokens. Text is represented as unigram-based subword units [Kudo, 2018] with a vocabulary size of 1,000. Modality dropout multimodal input. Additionally,
is applied when fine-tuning of updates, steps to freeze, and layers to freeze throughout fine-tuning. We evaluate each model, regardless of the modality used for fine munitun all kinds of input modalities and report the word error rates (WERs) without fusing external language models. To gauge the robustness to noise, we follow the same protocol in Shi et al. [2022b] to noisy audio and audio-visual test sets by adding babble noise at decibel (dB) to the audio stream for evaluation. Detailed hyperpamater setups can be found in Appendix A, and more ablation studies can be found in Appendix B.I.
sweep learning rate, number
Modality dropout is essential for building a single model Table compares fine-tuning from scratch, as well as from pre-trained models with and without modality dropout, using labeled audio, visual, or audio-visuu visual speech, and highlights the results of zero-shot modality transfer where labeled data in test modality are not used. First, models without pre-training are significantly worse. Second, when fine-tuning significantly on unimodal data if pre-training modality dropout not applied (e.g., 1.7% to 21 21.4% on clean audio). This can be alleviated with fine-tuning modality dropout, improving the average WER from 24.5% to 12.3%, but is still behind applying it in both pre-training and fine-tuning (11.4% average). We also note that fine-tuning dropout leads to consistent improvement on noisy AV test set, likely because occasionally dropping the audio stream encourages the model to rely more on the visual input which improves noise robustness.
audio-visual data without pre-training modality dropout, performance drops
The benefit of applying pre- pre-training modality dropout is magnified in unimodal fine-tuning tue-tuning: the model fine-tuned only on labeled audio data yields the same performance on clean and noisy audio-visum input (1.3% and 4.6%) as the model fine-tuned with labeled audio-visual data, while being slight behind on video input compared to the model fine-tuned with labeled video data (31.6% vs. 28.7 78.7%). When it is compared to the model pre-trained without modality dropout and fine-tuned also with labeled audio, huge gains observed zero-shot scenarios, for example 96.8% versus 31.6%

---

Table Speech recognition results on LRS3 PT indicates model pre-traines (on LRS3 and VC2-En). PT mod-drop-p and FT-drop-p denote if modality dropout is applied in pre-training and fine-tuning, respectively. FT mod denotes the labeled data used for fine-tuning: audio-visual (AV), audio (A), or visual (V). Zero-shot scenarios are highlighted with blue shades.
FT
AV-WER
A-WER
PT
FT mod
V-WER
Avg-WER
mod-drop
mod-drop
Clean
Noisy
Clean
Noisy
fine-tuned on 433h
X
n/a
AV
3.8
17.2
28.2
87.6
83.4
44.0
AV
1.3
4.8
21.4
52.6
42.3
24.5
/
AV
X
1.2
5.2
1.7
25.5
32.4
13.2
X
n/a
AV
/
3.6
15.9
4.6
44.8
63.7
26.5
AV
1.3
4.1
1.8
23.1
31.0
12.3
AV
1.3
4.6
1.5
20.5
29.1
11.4
X
a/a
A
n/a
X
X
4.0
37.3
X
X
X
A
n/a
1.5
18.0
1.6
20.9
96.8
27.8
/
A
n/a
1.3
4.6
1.4
19.3
31.6
11.6
X
n/a
V
n/a
X
X
X
X
60.3
V
n/a
11.3
21.8
80.3
97.7
28.0
47.8
/
V
n/a
2.1
5.1
2.3
20.9
28.7
11.8
V-WER and 18.0% versus 4.6% noisy AV-WER. Similar trend can be observed when fine-tuning with labeled visual speech.
Zero-shot modality transfer not achieved through memorization Because the dataset used for fine-tuning is subset of the pre-training data, there could be suspicions that the model memorizes the audio-visual pairs from pre-training and associates them with the labeled unimodal data to enable zero-shot transfer. To avoid cheating, we fine-tune model LibriSpeech [Panayotov et al., 2015], an audio-only read audiobook dataset that is out-of-domain relative to the pre-training data composed of oratory talks (Table 3). Despite the degradation compared to fine-tuning on in-domain LRS3 data, the model still performs decently on zero-shot transfer scenarios compared to models pre-trained without modality dropout in Table 2 and 3.
Table 3: Fine-tuning on out-of-domain labeled audio speech that does not overlap with the pre-training data, and evaluating on LRS3 test. Zero-shot scenarios are highlighted with blue shades.
PT data
PT mod-drop
FT data
FT mod
AV (Clean)
AV (Noisy)
A (Clean)
A (Noisy)
V
VC2-4n LRS3+VC2-En
LibriSpeech
A
6.2
11.0
20.9
65,9
64.7
LibriSpeech
A
4.5
8,9
4.8
27.6
39.4
Comparison with state-of-the-art models Table compares u-HuBERT with models from the literature, which are crodality-speciffe In contrast, u-HuBERT the only single model that be fine-tuned any modality and test all modalities while achieving performance par better than the best modality-specific models using similar amounts of labeled data. Furthermore, compared AV-HuBERT, u-HuBERT can be pre-trained additional unimodal data to yield even better performance.
4.4 Speech translation
We repeat the experiment speech translation study if u-HuBERT enables construction of single model and achieves zero-shot generalization on a different task.
Fine-tuning setup To simulate speech translation setting, we translate the English transcriptions into Spanish create paired (English audio, English video. Spanish text) data. The 30 hour LRS3 trainval split is used for fine-tuning. We use the same decoder for speech translation as for speech recognition, where 9-layer randomly initialized Transformer is appended to the pretrained u- u-HuBERT encoder to decode English speech into Spanish text. Unigram-based sentencepiece units with vocabulary

---

Table Comparison with state-of- of-tth--rrt art audio, visual, and audio- audio-visual speech recognition results reported on LRS3. Zero-shot scenarios are highlighted with blue shades. u- u-HuBERT pre-trained without unlabeled unimodal data effectively AV-HuBERT but fine-tuned with hyperparameters optimized for performance on all three modalities. refers filtered public datasets where the list of used utterances are not public. 4 refers to non-public datasets.
Method
Unlab
Lab
(%)
AV (hr)
A (hr)
Mod
Dataset (hr)
AV
A
V
supervised learning
Ma al. [2021]
AV A
LRW, LRS3 (590)
Ma ct. [2021]
LRW, LRS3 (590)
X
2.3
X
[2021]
LRW, LRS3 (590)
43.3
Xu et al. [2020]
AV
LRW, LRS3 (590)
6.8
X
X
Ma et al. [2022]
LRW. LRS2, LRS3, AVSpeech (1,459)
X
X
31.5
Afouras al. [2021]
LRS2, LRS3, MV-LRS* TEDx (2,676)
X
X
30.7
Makino [2019]
YT3I (31,000)
Makino al. [2019]
YT3IK² (31,000)
4,8
Makino et al. [2019]
YT31K² (31,000)
X
X
33.6
Sendyuk al. [2021]
YT90k (90,000)
X
X
25.9
Serdyuk al. [2021]
LRS3. YT90K (90,433)
semi-sapersised and self-supervised learning
[Shi et al., 20226]
LR$3, VC2-Ei (1,759)
AV
LRS3 (433)
1.4
X
[Shi al., 2022h]
IRR53, VC2-FFn (1,759)
LRS3 (433)
1.6
[Shi et al., 2022a]
LRS3, VC2-En (1,759)
LRS3 (433)
X
X
28.6
Afouras al. [2020]
VC2-clean| (334)
LRW. LRS3 (590)
u-HuBERT
LRS3, VC2-En (1,759)
AV
LRS3 (433)
1.5
29.1
u-HuBEKT
LRS3, VC2-En (1,759)
A
LRS3 (433)
1.3
1.4
31.6
u-HuBERT
LRS3, VC2-EE (1,759)
LRS3 (433)
5.1
u-HuBERT
1R83, VC2.FFn (1,759)
TD (452)
AV
LRS3 (433)
1.2
1.4
27.2
size of 1,000 used decoding output units. The overall fine-tuning and decoding paradigms remain same as speech recognition. Detailed hyperparameters can be found in Appendix A. BLEU score Papineni et al. [2002] used evaluate the translation model.
Translation performance The performance of u-HuBERT on speech translation is shown in Table 5 As is similar the observation in speech recognition, the model hardly produces meaningful results its encoder is randomly initialized (Avg-BLEU: 3.3). u-HuBERT pre-training not only greatly improves translation quality by roughly 10 fold (Avg-BLEU 3.3- but also shows strong performance in zero-shot modality transfer. For instance, the model ine-tuned with audio-only speech achieves 34.3 BLEU score when it is tested using audio-visual speech under noisy environment, outperforming its audio-only counterpart (26.5). tasks such as speech translation, where labeled multimodal data be difficult to harvest, our pre-training approach shows potential in enhancing the noise robustness supplied by the extra visual modality at test time even without exposure to any labeled visual data during training. Within u-HuBERT, modality dropout an essential component for acquiring such capacity, as is suggested by improvement in average BLEU score it brings (A: +4.7, V: 17.0-+3.77), In cases with available labeled multimodal data, a single audio-visual model is able to achieve performance superior or on par to a modality-specific model (A: 42.2 vs. 42.9, V: 27.6 vs. 26.4).
4.5 Ablation studies
Pre-training modality dropout parameters To better understand the impact of pre-training modality dropout configurations zero-shot modality transfer, we conducted pre-training ablation studies with a reduced setup: for each configuration, a BASE model pre-trained on 433 hours of LRS3 data using 32 GPUs for 200K updates and then fine-tuned on 30 hours of LRS3 labeled data (the "trainval" split). Four modality dropout configurations are considered, represented by the probability of using both streams (pav). audio stream (pa) and video stream (p,) in Table 6. The first configuration (pae. Pa. p.) (1.00, 0.00, 0.00) is equivalent to not using modality dropout.
When fine-tuning audio-visual or audio-only data, among the models treitrained with modality dropout, setting the pe higher leads to slightly worse performance on vidio-visual and audio-only test sets. In contrast, when fine-tuning visual-only data, setting the probability of V-only higher leads
'The results of models finetuned with 433 hours of labeled speech can be found in Table B9 the Appendix.

---

Table 5: Speech translation results on LRS3 Headers are the same as Table 2.
PT
FT mod-drop
AV-BLEU Clean
A-BLEU
PT
mod
V-BLEU
Avg-BLEU
mod-drop
Noisy
Clean
Noisy
fine-tuned on 30h
n/a
AV
3.6
5.6
3.5
2.4
1.3
3.3
x
AV
39.5
34.8
39.2
24.5
25.1
32,6
/
AV
42.4
36.5
42.2
26.0
27.6
35.0
n/a
A
n/a
X
X
7.2
4.0
X
X 27.3
X
n/a
38.3
26.8
38.0
22.9
10.8
n/a
43.4
34.3
42.9
26.5
26.3
34.7
X
n/a
n/a
X
0.5
X
X
n/a
24.9
20.8
16.6
7.2
15.2
17.0
V
V
V
n/a
38.1
33.2
37.1
23.1
26.4
31.7
slightly better average performance. Nevertheless, all three configurations achieve significantly better performance compared to the model pre-trained without modality dropout, confirming that activating modality dropout is the key to successful zero-shot modality transfer.
Table 6: Ablation study for pre- are-training modality dropout probabilities with a reduced training setup.
PT
FT
AV-WER Clean
A- A-WER
V-WER
Avg-W WEg
Poe
Po
Pa
mod
dodpdrop
Noisy
Clean
Noisy
1.00
0.(0)
0.00
AV
5.54
14.57
17.83
67.66
59.72
33.06
0.70
0.15
0.15
AV
5.24
15.11
6.21
47.42
50.65
24.93
0.50
0.25
0.25
AV
X
5.17
15.36
6.45
48.59
51.42
25.40
0.50
0.15
0.35
AV
6.04
17.59
7.46
51.78
51.06
26.79
1.00
0.00
0.00
A
n/a
6.26
15.47
10.67
60.81
66.89
32.02
0.70
0.15
0.15
A
n/a
5.83
14.23
6.10
41.29
52.13
23.92
0.50
0.25
0.25
A
n/a
5,50
14.84
5.96
41.72
48.72
23.35
0.50
0.15
0.35
A
n/a
6.22
16.00
6.54
43.71
48.71
24.24
1.00
0.00
0.00
V
n/a
17.45
26.68
27.21
79.01
66.65
43.40
0.70
0.15
0.15
V
n/a
13.28
20.23
13.59
48.30
52,40
29.56
0.50
0.25
0.25
V
n/a
11.01
18.18
11.54
47.96
49.19
27.58
0.50
0.15
0.35
V
n/a
10.69
18.74
11.69
50.10
49.11
28.07
Pre-training dynamics with additional unimodal data While mixed modality pre-training allows the model to be trained on larger quantities of data, it also dilutes the in in-domain data with respect audio-visual or other unimodal tasks, which may lead to worse performance on those tasks. We pre-train audio-visual data and mixed-modal data (audio-visual and audio-only) respectively for different numbers updates and study if degradation occurs to audio-visual and visual speech recognition in the mixed-modal case.
The baseline (PT on AV) is pre-trained on audio-visual LRS3, and the proposed mixed-modal pre- training (PT on AV+A) uses the same audio-visual data as the baseline plus the VC2-En audio data. For each setup we pre-train the model for 1100, 200, 400, 600} updates. Results Figure 4 showed that adding unlabeled audio data consistently improves the audio speech recognition performance all number of updates. In contrast, the performance of "PT on AV+A" lags behind "PT on A" on audio-visual and visual speech recognition when model is trained for fewer than 400K updates. Nevertheless, when a model trained for longer (600K updates), mixed-modal pre-training outperforms the baseline all three tasks. The observation suggests that u- u-HuBERT can benefit from training on more data and continue improving with more updates, while the performance the baseline saturates earlier due to the lack of data.

---

FT and test on AV
FT and test on A
FT and test on V
52
7.5
+
Pi AV
WER (%)
WER (%)
WER (%)
50
AV
48
6.5
46
44
100200
400
600
100200
400
600
100200
400
600
PT total updates (K)
PT total updates (K)
PT total updates (K)
Figure 4: Ablation study comparing pre-training on audio-visual data (LRS3) versus on both audio-
visual data (LRS3) and audio-only data (audio from VC2-En) for different numbers of steps.
5
Discussion
We assume that there is anchor modality where all other modalities appear in some multimodal data paired with it. This enables the model to predict shared targets and learn modality-agnel representations for all modalities, such that non-anchor unimodal data can be used in the subsequent iterations. While this may seem limiting, most multimodal speech are paired with audio [Richard al., 2021, Hueber et al., 2010, Richmond et al., 201 I, Livescu et al., 2009] since audio is the primary measurement for speech and other modalities are often supplementary. This makes the assumption of having audio the anchor modality practical. Furthermore, the idea be easily extended to settings where any modality can be connected to the anchor via some intermediate modality. In short, modality that is N-hop away from the primary anchor can be used in the (N + 1)-th iteration.
The only case not supported u-HuBERT is pre-training on modalities that cannot be connected through any multimodal data, for example, speech. A shared codebook cannot be derived for EMG and the rest the modalities, and is unclear if the model still benefit from joint training. The setup has also been found much more challenging on other pairs of modalities: for example, SLAM Bapna et al. [2021] shows that without paired text and speech data, joint pre-trum pre-training compared to a model pre-trained only on speech. We leave such exploration to future work. In addition, also include ethical discussion in Appendix C.
collection of audio, visual, audio-visual and EMG
unlabeled speech and text degrades the performance
Conclusion
Recent studies in applied machine learning trend toward building things that are general, for example, general perception module [Jaegle et al., 2021], general self-supervised objective [Baevski et al., 2022], single model for many tasks [Brown et al., 2020, Kaplan et al., 2020, Aghajanyan al., 2022] and/or many input [Girdhar et al., 2022, Hu and Singh, 2021] instead of optimizing for performance very specific setups. Our work contributes this line of research by presenting unified self-supervised objective to pre-train a model on speech of many modalities. Specifically, can be fine-tuned for many different speech tasks on labeled data in any combination of pre-trained modalities, resulting in single model that can process all combinations of pre-trained modalities to perform the fine-tuned task.
We envision our work can bring substantial benefit to multimodal speech processing. Given there are significantly more unlabeled audio compared to unlabeled multimodal speech, being able to utilize unlabeled audio can greatly improve the quality of multimodal speech representations. Furthermore, there few even no labeled multimodal speech for many tasks, such speech translation. Hence, it is essential to build a model capable of zero-shot input modality transfer to perform these tasks with modalities without labeled data.
References
T. Afouras, S. Chung, and Zisserman. LRS3-TED: large-scale dataset for visual speech recognition, 2018. arXiv:1809 1.0.09969.
10

---

T. Afouras, J. S. Chung, and A. Zisserman. ASR is all you need: Cross-modal distillation for lip reading. ICASSP, 2020.
T. Afouras, A. Zisserman, et al. Sub-word level lip reading with visual attention. arXiv preprint arXiv:21 10Xiv:210007603, 2021.
A. Aghajanyan, B. Huang, C. Ross, V. Karpukhin, H. Xu, N. Goyal, D. Okhonko, M. Joshi, G. Ghosh, M. Lewis, et al. Cm3: A causal masked multimodal model of the internet. arXiv preprint
arXiv:2201.07520, 2022.
A. Baevski, W.-N. Hsu, Q. Xu, A. Babu, J. Gu, and M. Auli. Data2vec: A general framework for self-supervised learning in speech, vision and language. arXiv preprint arXiv:220..03555, 2022.
A. Bapna, Y.aa Chung, N. Wu, A. Gulati, Y. Jia, J. H. Clark, M. Johnson, J. Riesa, A. Conneau, and Y. Zhang. Slam: A unified encoder for speech and language modeling via speech-text joint pre-training, arXiv preprint arXiv:2110.10329 2021.
T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners, Advances in neural information processing systems, 33:1877-19011 2020.
W. Chan, D. Park, C. Lee, Y. Zhang, Q. Le, and M. Norouzi. Speechstew: Simply mix all available speech recognition data to train one large neural network. arXiv preprint 1xXxv:2004.02133, 2021.
J. S. Chung and A. Zisserman. Lip reading in the wild. In ACCV, 2016.
J. S. Chung, A. Nagrani, and A. Zisserman. Voxceleb2: Deep speaker recognition. In INTERSPEECH, 2018.
Y.-. Chung, Y. Zhang, W. Han, C.-C. Chiu, J. Qin, R. Pang, and Y. Wu. w2v-bert: Combining contrastive learning and masked language modeling for self-supervised speech pre-training. 202 IEFE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 2021.
A. Conneau, A. Baevski, Collobert, A. Mohamed, and M. Auli. Unsupervised cross-lingual representation learning for speech recognition. arXiv preprint urXiv:2006.1397) 2020.
Devlin, M.-W. Chang, Lee, and K. Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In NAACL, 2019.
L. Diener and T. Schultz. Investigating objective intelligibility in real-time emg-to- eng-to-speech conversion.
In INTERSPEECH, 2018
X. Feng. Y. Zhang, and Glass. Speech feature denoising and dereverberation via deep autoencoders for noisy reverberant speech recognition. In 2014 IEEE international conference acoustics, speech and signal processing (ICASSP), pages 1759-1763. IEEE, 2014.
R. Girdhar, M. Singh, N. Ravi, L. van der Maaten, A. Joulin, and I. Misra. Omnivore: A Single Model for Many Visual Modalities. In CVPR, 2022.
R. Haeb- Umbach, J. Heymann, L. Drude, S. Watanabe, M. Delcroix, T. N. P. University, H. Germany, A. J. Aachen, J. H. University, Baltimore., Usa, N. C. S. Laboratories, Kyoto, and Japan. Far-field automatic speech recognition. Proceedings of the IEEE, 109:124-148, 2021.
K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778, 2016.
F. Hernandez, V. Nguyen, S. Ghannay, N. A. Tomashenko, and Y. Estève. Ted-lium 3: twice as much data and corpus repartition for experiments on speaker adaptation. ArXiv, abs/1800 805.04699, 2018.
W.-N. Hsu, B. Bolte, Y.H. H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed. Hubert: Self- supervised speech representation learning by masked prediction of hidden units. arXiv preprint arXiv:2106.07447, la.

---

W.-N. Hsu, A. Sriram, A. Baevski, T. Likhomanenko, Q. Xu, V. Pratap. J. Kahn, A. Lee, R. Collobert, G. Synnaeve, et al. Robust wav2vec 2.0: Analyzing domain shift in self-supervised pre-training. arXiv preprint 1eXiv:2104.01000 2021
R. Hu and A. Singh. Unit: Multimodal multitask learning with unified transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1439-1449 2021.
T. Hueber, E.-L. Benaroya, G. Chollet, B. Denby, G. Dreyfus, and M. Stone. Development of silent speech interface driven by ultrasound and optical images of the tongue and lips. Speech Communication, 52(4):288-300 2010.
A. Jaegle, F. Gimeno, A. Brock, O. Vinyals, A. Zisserman, and Carreira. Perceiver: General perception with iterative attention. In International Conference Machine Learning, pages 4651-4664. PMLR, 2021.
C. Jia, Y. Yang, Y. Xia, Y.T. Chen, Z. Parekh, H. Pham, Q. V. Le, Y.H. Sung, Z. Li, and T. Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In ICML, 2021.
M. Johnson, M. Schuster, Q. V. Le. M. Krikun, Y. Wu, Chen, Thorat, F. Viégas, M. Wattenberg. G. Corrado, et al. Google's multilingual neural machine translation system: Enabling zero-shot translation. Transactions of the Association for Computational Linguisties, 5:339-351, 2017.
J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001 000061.,.66, 2020.
K. Kawakami, L. Wang, C. Dyer, P. Blunsom, and A. v. d. Oord. Learning robust and multilingual speech representations. arXiv preprint 1xXiv:2001.11128 2020.
S. Kim, I. R. Lane, S. Kim, and I. Lane. End-to-end speech recognition with auditory attention for multi-microphone distance speech recognition. In Interspeech, pages 3867-3871, 1,, 2017.
D. Kingma and J. Ba. Adam: method for stochastic optimization. CoRR, abs/ 14 112.69900, 2015.
B. E. Kingsbury, N. Morgan, and S. Greenberg. Robust speech recognition using the modulation spectrogram. Speech communication, 25(1-3):117-132 1998.
T. Kudo. Subword regularization: Improving neural network translation models with multiple subword candidates. In ACL, 2018
G. Lample and A. Conneau. Cross-lingual language model pretraining. In NeurIPS, 2019.
C. Lee, K. Cho, and W. Kang. Mixout: Effective regularization to finetune large-scale pretrained language models. arXiv preprint arXiv: 1909. / 1299, 2019.
T. Likhomanenko, Q. Xu, V. Pratap, P. Tomasello, J. Kahn, G. Avidov, R. Collobert, and G. Synnaeve. Rethinking evaluation asr: Are our models robust enough? arXiv preprint 11X:v:2010.11745. 2020.
Y. Liu, J. Gu, N. Goyal, X. Li, S. Edunov, M. Ghazvininejad, M. Lewis, and L. Zettlemoyer. Multilingual denoising pre-training for neural machine translation. Transactions of the Association for Computational Linguistics, 8:726-742, 2020.
Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo. Swin transformer: Hierar- chical vision transformer using shifted windows. In Proceedings of the IEFF/CVF International Conference on Computer Vision (ICCV), 2021.
K. Livescu, B. Zhu, and J. R. Glass. On the phonetic information in ultrasonic microphone signals. 2009 IEEE International Conference on Acoustics, Speech and Signal Processing, pages 4621- 4624, 2009.
Ma, S. Petridis, and M. Pantic, End-to-end audio-visual speech recognition with conformers. In ICASSP. 2021.
12

---

P. Ma, S. Petridis, and M. Pantic. Visual speech recognition for multiple languages in the wild. arXiv preprint 2tti::2202.138844 2022.
Makino, Liao, Y. Assael, B. Shillingford, B. Garcia, O. Braga, and O. Siohan. Recurrent neural network transducer for audio-visual speech recognition. In Interspeech, 2019.
A. Nagrani, J. S. Chung, and A. Zisserman. Voxceleb: large-scale speaker identification dataset. arXiv preprint arXiv: 08612, 2017.
N. Neverova, C. Wolf, G. Taylor, and F. Nebout. Moddrop: Adaptive multi-modal gesture recogmition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 38, 12 2014. doi: 10.1 109/ TPAML.2015.266154
T. Q. Nguyen and J. Salazar. Transformers without tears: Improving the normalization of self- attention. arXiv preprint arXiv: 2019.
V. Panayotov, G. Chen, D. Povey, and S. Khudanpur. Librispeech: corpus based public domain audio books. In 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 5206-5210. IEEE, 2015.
K. Papineni, S. Roukos, T. Ward, and W.-. Zhu. Bleu: method for automatic evaluation of machine translation. In ACL, 2002.
T. Pires, E. Schlinger, and D. Garrette. How multilingual is multilingual bert? arXiv preprint 100, 2019.
V. Pratap, A. Sriram, P. Tomasello, Hannun, V. Liptchinsky, G. Synnaeve, and R. Collobert. Massively multilingual asr. 50 languages, model, 1 billion parameters. arXiv preprint arXiv:2007.03001, 2020.
A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, Goh, S. Agarwal, G. Sastry, A. Askell, Mishkin, J. Clark, G. Krueger, and I. Sutskever. Learning transferable visual models from natural language supervision. In ICML, 2021.
A. Richard, M. Zollhöfer, Y. Wen, de la Torre, and Y. Sheikh, Meshtalk: 3d face animation from speech using cross-modality disentanglement, In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 113-1 182, October 2021.
K. Richmond, P. Hoole, and S. King. Announcing the electromagnetic articulography (day 1) subset of the mnguO articulatory corpus. In Twelfth Annual Conference of the International Speech Communication Association, 2011.
L. Sari, K. Singh. J. Zhou, L. Torresani, N. Singhal, and Y. Saraf. A multi-view approach to audio-visual speaker verification. 2021.
M. Schuster and K. Nakajima. Japanese and korean voice search. 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5149-5152, 2012.
D. Serdyuk, O. Braga, and O. Siohan. Audio-visual speech recognition is worth 32 x 32 x 8 voxels. arXiv preprini arXiv:2109.095: 2021.
B. Shi, W.N. Hsu, K. Lakhotia, and A. Mohamed. Learning audio-visual speech representation by masked multimodal cluster prediction. arXiv preprint arXiv:2201.02784, 2022a.
B. Shi, W.-N. Hsu, and A. Mohamed. Robust self self-supervised audio-visual speech recognition. arXiv preprint arXiv:2201.01766 2022b.
A. Singh, R. Hu, V. Goswami, G. Couairon, W. Galuba, M. Rohrbach, and D. Kiela. Flava: A foundational language and vision alignment model. ArXiv, abs/2112.044482, 2021.
D. Snyder, G. Chen, and D. Povey. Musan: A music, speech, and noise corpus. ArXiv, abs/15100088844 2015.
A. Üstün, A. Bérard, Besacier, and M. Gallé. Multilingual unsupervised neural machine translation with denoising adapters. arXiv preprint arwiv:2110.104111111 2021.
13

---

L. Van der Maaten and G. Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.
A. Vaswani, N. M. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and
I. Polosukhin. Attention is all you need. ArXiv, abs/1706.03762. 2017.
B. Xu, C. Lu, Y. Guo, and J. Wang. Discriminative multi-modality speech recognition. In CVPR, 2020.
X. Yao, Y. Zheng, X. Yang, and Z. Yang. NIp from scratch without large-scale pretraining: A simple and efficient framework. ArXiv, abs/2111.041300 2021.
Zhang, J. Xue, C. Lan, W. Zeng, Z. Gao, and Zheng. Eleatt-mn: Adding attentiveness neurons in recurrent neural networks. IEEE Transactions on Image Processing, 29:1061-1077 2019.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes] See Section 4
(b) Did you describe the limitations of your work? [Yes] See Section 5
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See
Appendix
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section 4 and Appendix A
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes] See Appendix B
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Section 4 and Appendix A
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] See Section
(b) Did you mention the license of the assets? [Yes] See Appendix
(c) Did you include any new assets either in the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [Yes] See Appendix A
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [Yes] See Appendix A
5. If you used crowdsoureing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "typvR9lAnBb", "reviewer": "Reviewer_yjaa", "summary": "This paper presents a self-supervised learning framework, termed u-HuBERT, that can utilize both multimodal and unimodal unsupervised speech data including audio-visual, audio and visual to pre-train a single model by utilizing modality dropout. If this pre-trained model is fine-tuned on audio-visual data with modality dropout, it can achieve comparable or better results than the state-of-the-art modality-specific models. In addition, the authors show that the modality dropout used in pre-taining is important for learning modality agnostic features, which makes the model only fine-tuned on audio perform well on audio-visual or visual speech data, achieving zero-shot modality generation for speech recognition and speaker verification.", "questions": "Questions/suggestions:\n- The proposed framework assumes the multiple modalities features align in a frame-by-frame fashion. How can it be applied to the case where the assumption does not hold, e.g. between the audio and text modalities?\n- Line 194: Is the pre-training modality dropout probability 0.25/0.25 tuned? How sensitive is its value to the results?\n- In Table 2, modality dropout on fine-tuning also improves AV-WER for the models fine-tuned on AV data. It would be better to explain why that is the case.\n- Line 259: For the claim of u-HuBERT \"achieving on par or better than the best modality-specific models\", it would be better to mention the caveat as well: it's slightly worse than the best modality-specific visual speech WER (27.2 vs. 25.9) although using much less labeled visual speech training data.\n\nTypos:\n- Line 37: it essential -> it is essential\n- Line 327: supported u-HuBERT -> supported by u-HuBERT", "limitations": "Yes, the authors have addressed the limitations in Section 5 and the potential negative societal impact of their work in Appendix C.\n\nFor completeness, the authors should also mention another limitation of the system that assumes all the modalities align in a frame-by-frame fashion (as mentioned in my question above), or discuss possible extensions to address this limitation for cases where this assumption does not hold, e.g. between audio and text modalities.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Originality:\n\n- Previous work on audio-visual ASR are mainly focusing on modality-specific training / fine-tuning. This work, using a shared target token space for all unimodal and multimodal data in a single model, and find-tuning it with labeled data only on a subset of the modalities seen during pre-training but perform well on other modalities in test to achieve zero-shot modality generalization, is a novel idea.\n\n- Even though modality dropout has been used in the literature, the analysis to show that it is important for learning modality agnostic features and achieving huge gains in zero-shot modality generalization scenarios is new.\n\nQuality:\n\n- The authors show that the proposed single model achieves on par or better results than state-of-the-art modality-specific models for speech recognition. It is very promising that the model fine-tuned only on audio data performs well on audio-visual and visual speech data with comparable results to the models fine-tuned on the modality-specific data.\n\n- The experiment using out-of-domain labeled audio data from LibriSpeech for fine-tuning shows convincing results to support the claim that the model's zero-shot modality transfer is not achieved through memorization.\n\n- The proposed model also shows the zero-shot modality generalization ability but only to some extent. The authors suggest this is due to the nature of the speaker verification task, where each modality can only infer a subset of the speaker features. This demonstrates a limitation to the zero-shot transfer.\n\nClarity:\n\n- The paper is written clearly. The presentation is very well organized.\n\nSignificance:\n\n- This work contributes to the community significantly, by showing an approach to pre-train on both multimodal and unimodal speech data to not only improve the state-of-the-art results in audio-visual ASR but also achieve high-quality zero-shot modality generalization. In particular, it brings significant value to the tasks with modalities without much labeled data.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "izRY6jqm4g", "reviewer": "Reviewer_aZc1", "summary": "This paper presented unified hidden unit BERT (u-HuBERT), which leverages unlabeled speech data of many different modalities for pre-training, including both unimodal and multimodal speech. The u-HuBERT model is composed of a set of modality-specific feature extractors and a shared transformer backend. With this architecture design, u-HuBERT can learn modality-agnostic frame-level representations, and can be fine-tuned with either multimodal or unimodal data. Experiment results show the effectiveness of u-HuBERT on two speech processing tasks: speech recognition and speaker verification.\n\n-> After reviewing the author rebuttal, I changed my ratings to 5.", "questions": "In section 4.3, when showing the results of fine-tuning on OOD labeled LibriSpeech data in Table 3, what's the performance of baseline methods? The author claimed the zero-shot modality transfer performance is decent on LibriSpeech, however, without comparing with baselines, it's hard to convince the readers.\n\nMore descriptions need to be added when comparing with SOTA models in section 4.3 (Table 4). Several questions are:\n* What are the baseline model settings?\n* What are the pre-training/training/fine-tuning approaches used by baseline models?\n* What are the datasets?", "limitations": "The author had limitation discussion in section 5.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Built upon AV-HuBERT, u-HuBERT can generalize to utilize both multimodal and unimodal data by mapping different inputs into a modality-agnostic feature space using the proposed modality dropout method. Modality dropout also allows zero-shot modality generalization, which means the fine-tuned model can handle modalities in the downstream task that do not cover the ones seen during pre-training. Even though the author showed efficiency of the proposed method in speech recognition and speaker verification tasks, there are still some studies and discussions missing in the paper. For example, the modality dropout rates during pre-training and fine-tuning and what's the effect of different dropout rates in each stage?\n\nTypo\n* Table 2 caption \"FT-drop-p\" should be \"FT mod-drop-p\".", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "_4OK7Xc2Zyt", "reviewer": "Reviewer_dtiD", "summary": "The paper introduces a self-supevised method in the HuBERT family designed for audio-visual speech, which can be trained on different modalities for pre-training, including single modality audio or visual, and multi-modalities (audio-visual).\nu-HuBERT is closely related to AV-HuBERT, but incorporates masked cluster prediction for both unimodal and multimodal data, via masking on modalities, (\"modality dropout\") which requires the model to predict the same targets using different sets of input modalities.", "questions": "I am under the impression that  w2v-BERT produces better representations than HuBERT.  Can you comment on your selection of HuBERT versus w2v-BERT, and whether your proposed additions to AV-HuBERT are applicable to the case of w2v-BERT?  ", "limitations": "Limitations are discussed in section 5", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\n\nStrengths: \n\nThe proposed model provides the ability to fine tune on any modality and work well on all modalities, with comparable performance to modality-specific models.\n\nEvaluation is thorough.  \nImprovements are small in absolute terms on LRS3, but baseline error rates are very small (e.g., 1.4% WER for AV ASR), so a reduction to 1.2% which is a large relative WER reduction of 14%.  Improvements like this are typically very difficult to achieve.  \nThese are useful results.  \n\nThe paper is well written and quite clear. \n\n\nWeaknesses:\n\nThe paper is not self-contained, and most of the architectural details are contained in other references. \n\nNovelty relative to AV-HuBERT is not large, but is important. \n\nThe proposed methods are practical, but not based on any deep theory developments.  ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "9qM64z-0neP", "reviewer": "Reviewer_BiUF", "summary": "This paper presents an approach to handle speech signal in multiple modalities within a single unified model that can be pre-trained and fine-tuned with datasets of mixed speech modalities.  The model includes modality specific feature extraction followed by a shared transformer module that learns modality agnostic representations.  Audio and visual modalities are considered and results are presented on speech recognition as well as speaker verification tasks.  It is also shown that the proposed model exhibits strong performance in zero-shot generalization to modalities for which it was not fine-tuned.\n\n\n", "questions": "Please see comments above.", "limitations": "n/a", "rating": 3, "confidence": 4, "soundness": 4, "presentation": 2, "contribution": 1, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This paper is very close to the recently published AV-HuBERT paper [1], so much so that it is unclear to me if the additional contributions are significant enough for publication.  For instance,\n\n* A key novelty of the paper, as mentioned in lines 54-55 and lines 103-104, is that AV-HuBERT pre-trains with audio-visual multimodal data only whereas the proposed approach pre-trains with multimodal as well as unimodal data.  However, as discussed by She et al. [1], a modality dropout is applied during training to make the model learn how to deal with unimodal inputs, thus effectively pre-training on unimodal data as well.\n\n* Lines 121-123 state that “The main idea of this paper is to apply modality dropout to multimodal speech…”.  However modality dropout was applied in [1] as well, and it is unclear how use of modality dropout in current model differs from that in [1].\n\n* Empirical results are also quite close to those reported by Shi et al. [1].  The best ASR results reported in the current paper are AV/V WER of 1.2/27.2% (these include the TD audio-only corpus during pre-training in addition to 1759 hours of audio-visual data) whereas Shi et al. report AV/V WER of 1.3/26.9% where pre-training is done on 1759 hours of audio-visual data.\n\n[1] B. Shi, W. Hsu, K. Lakhotia, A. Mohamed, “Learning audio-visual speech representation by masked multimodal cluster prediction,” ICLR 2022.\n\nThe zero-shot generalization results as well as application to speaker-verification task are novel; we don’t have corresponding results from the AV-HuBERT model.\n\nMinor typos:\nLine 37: it essential -> it is essential\nLines 54-62: provide reference to AV-HuBERT", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept