XLAVS-R: Cross-Lingual Audio-Visual Speech Representation Learning for Noise-Robust Speech Perception (2024)

Abstract: Speech recognition and translation systems perform poorly on noisy inputs, which are frequent in realistic environments. Augmenting these systems with visual signals has the potential to improve robustness to noise. However, audio-visual (AV) data is only available in limited amounts and for fewer languages than audio-only resources. To address this gap, we present XLAVS-R, a cross-lingual audio-visual speech representation model for noise-robust speech recognition and translation in over 100 languages. It is designed to maximize the benefits of limited multilingual AV pre-training data, by building on top of audio-only multilingual pre-training and simplifying existing pre-training schemes. Extensive evaluation on the MuAViC benchmark shows the strength of XLAVS-R on downstream audio-visual speech recognition and translation tasks, where it outperforms the previous state of the art by up to 18.5% WER and 4.7 BLEU given noisy AV inputs, and enables strong zero-shot audio-visual ability with audio-only fine-tuning.

heading: 1 Introduction
text: Speech recognition and speech-to-text translation, two core speech perception tasks, have witnessed rapid developments in the past two years. Still, the performance of state-of-the-art (SOTA) models such as Whisper (Radford et al., 2023) and SeamlessM4T (Communication, 2023b) degrades sharply in noisy environments (Anwar et al., 2023; Communication, 2023a). Audio-only approaches to make such systems more robust (Ng et al., 2023; Zhu et al., 2023) remain challenged by frequent noise types such as intense babble noise and overlapped speech (Shi et al., 2022b).
Complementing audio inputs with visual signals such as video frames of speaker’s lips offers an alternative approach (Sumby and Pollack, 1954; Potamianos et al., 2004; Mroueh et al.,
∗ Work done at Meta AI
2015; Chung et al., 2017), with promising results in audio-visual speech recognition (AVSR) and audio-visual speech-to-text translation (AVS2TT) tasks (Afouras et al., 2022; Ma et al., 2021; Shi et al., 2022b; Anwar et al., 2023). Yet, as summarized in Table 1, prior works are either restricted to English (Hsu and Shi, 2022; Zhu et al., 2024; Seo et al., 2023) or to a limited amount of audiovisual data (Shi et al., 2022a; Choi et al., 2023; Lian et al., 2024), limiting language coverage and performance on downstream tasks.
To bridge this gap, we present XLAVS-R, a cross-lingual audio-visual speech representation learning model for noise-robust speech perception across over 100 languages. Its training strategy is designed to maximize the efficiency of multilingual AV pre-training by injecting audio-visual (AV) signals (§3.2) after pre-training on audioonly (A-only) data that is easier to scale in size and language coverage (§3.1). We improve AVHuBERT (Shi et al., 2022a) with simplified training
12896
protocol and updated model architecture, and scale model size up to 2B parameters for performance in the multilingual setting.
We conduct extensive evaluations on the MuAViC benchmark (Anwar et al., 2023). XLAVSR yields SOTA performance on speech recognition in 9 languages and speech-to-text translation in 6 X-to-English pairs (§5.1, §5.2), notably improving their robustness to noisy inputs. Furthermore, XLAVS-R enables zero-shot audio-visual ability in downstream tasks with audio-only fine-tuning (FT): fine-tuning our XLAVS-R 2B model without supervised AV downstream data achieves the best audio-visual performances in noisy settings even compared to the corresponding audio-visual supervised fine-tuned model (§5.4). This confirms the benefits of our pre-training strategy and relaxes the data requirements for downstream tasks, making it possible to tackle AVSR and AVSTT tasks without labeled—transcribed or translated—AV data.

heading: 2 Related Work
text: Self-supervised audio-only speech representation. Self-supervised learning (SSL) for speech aims to establish a general speech representation with unlabeled speech data applicable to various downstream applications, including speech recognition and spoken language understanding tasks (wen Yang et al., 2021). Many of today’s successful SSL approaches (Baevski et al., 2020; Hsu et al., 2021; Baevski et al., 2022; Chiu et al., 2022) are based on masked span prediction. Specifically, wav2vec 2.0 (Baevski et al., 2020) applies a contrastive predictive coding-like (van den Oord et al., 2018) loss on masked speech utterances. HuBERT (Hsu et al., 2021) utilizes masked speech features to predict hidden units derived from layerwise features. Instead of using discrete units, Data2vec (Baevski et al., 2022) directly regresses dense features from an exponential moving average (EMA) teacher.
SSL approaches notably reduce the need for labeled speech data. In ASR benchmarks (Panayotov et al., 2015), they yield performance on par with or better than fully supervised models with far fewer transcriptions. This has motivated sustained efforts toward multilingual SSL models such as XLSR53 (Conneau et al., 2021), XLS-R (Babu et al., 2022), and MMS (Pratap et al., 2023), which particularly excel in low-resource language scenarios.
Self-supervised audio-visual speech representation. Audio-visual SSL approaches draw heavy
inspiration from their audio-only counterparts. AVHuBERT (Shi et al., 2022a) extends HuBERT (Hsu et al., 2021) to the audio-visual setting by taking the masked audio-visual stream as input and predicting the hidden units initialized with MFCC clusters, iteratively refining them with layerwise features. This framework has proven effective for multiple downstream tasks, including lip reading (Shi et al., 2022a), audio-visual speech recognition and translation (Shi et al., 2022b; Anwar et al., 2023). In AV2AV (Choi et al., 2023) and our work of XLAVS-R, it has been extended to a multilingual setting. RAVEn (Haliassos et al., 2023) leverages modality-specific EMA teachers to generate targets for masked prediction, thus avoiding an iterative refinement process. Similarly, AV-data2vec (Lian et al., 2024) is based on data2vec (Baevski et al., 2022) and regresses multimodal features with an audio-visual EMA teacher. AV2vec (Zhang et al., 2023) further combines AV-data2vec with the masked prediction objective in AV-HuBERT. Some prior works also investigated the use of unpaired unimodal data, notably audio, to improve multimodal representation. u-HuBERT (Hsu and Shi, 2022) augments AV-HuBERT with audio-only speech during pretraining, resulting in improved audio-visual speech recognition performance. VATLM (Zhu et al., 2024) further added text-only data and trained the model on an arbitrary modality stream. Our work builds on the insights of AV/u-HuBERT with adaptations for continual SSL training with visual modality injection to A-only model and for the multilingual setting.
Audio-visual adaptation of audio-only speech models. Recent work has begun to explore the adaptation of audio-only speech models into audiovisual models. MixSpeech (Cheng et al., 2023) develops a visual speech translation model based on a pre-trained audio-only speech translation model by minimizing the discrepancy between the probabilities from audio-only and multi-modal streams. AVFormer (Seo et al., 2023) incorporates lightweight modules into an audio-only speech recognizer to adapt it into visually grounded speech recognition through two-stage fine-tuning. FAVA (May et al., 2023) directly fine-tunes a pre-trained BESTRQ (Chiu et al., 2022) encoder for audio-visual speech recognition with a randomly initialized visual encoder. FAVA connects to our work by applying this technique to a multilingual audio-only
model. However, its evaluation focuses on English with labeled AV speech data for supervised finetuning, while our approach focuses on AV SSL to achieve competitive zero-shot results without AV labeled data as well as state-of-the-art supervised multilingual performance.

heading: 3 XLAVS-R
text: Our approach (Figure 1) requires first training an audio-only SSL model, which is used to generate unit targets for audio-visual SSL pre-training. We inject visual modality into the resulting model to continue training with audio-visual SSL pretraining. Through this process, XLAVS-R introduces three key improvements over baseline approaches. First, we leverage the abundant availability of audio-only speech data to enhance overall performances and achieve greater robustness on domain shift (§3.1), and inject visual modality to build an audio-visual pre-training model on top of audio-only pre-training model (§3.2). Second, we use a learnable audio feature extractor to better capture the various phonetic information of multilingual audios (§3.3). Third, we improve the training efficiency of the audio-visual pre-training stage by single-round training with unit targets from audioonly contextualized representation (§3.4).

heading: 3.1 Audio-Only Pre-Training
text: Instead of training an audio-visual model in one step with the mix of audio-only data and audiovisual data, we first train an audio-only speech model and adapt it to an audio-visual one via selfsupervised continual pre-training with audio-visual data. This is motivated by the facts that audio-only data is much more plentiful than audio-visual data, particularly for low-resource languages, and that audio-only models are much more computationally efficient than audio-visual models with similar architecture and size because of the cost of visual feature extraction and bi-modal feature fusion. We concentrate our computational resources on the first stage of A-only SSL since audio-only data has greater volume and the audio modality naturally contains richer semantic information than the visual modality. We refer to the outcome model of this audio-only self-supervised learning as XLS-R, as we follow the XLS-R training settings (Babu et al., 2022) and adopt its wav2vec 2.0 architecture (Baevski et al., 2020).

heading: 3.2 Continued Audio-Visual Pre-Training with Visual Modality Injection
text: After obtaining XLS-R through A-only pretraining, we add a ResNet-based visual feature extractor (VFE) and a linear projection-based feature fusion module to build up an audio-visual model and continue AV pre-training.
For audio-visual self-supervised learning, we follow Shi et al. (2022a,b) using AV-HuBERT pre-training loss. Given temporally aligned audio input Ia ∈ RTa×Ca and video speech input Iv ∈ RTv×Cv×W×H , each modality feature extractor encode inputs into same length feature sequences fa, fv ∈ RT×D. Here, audio and visual frames are masked independently by separate masking probabilities. Frame-wisely fused audio-visual features Fusion(fa, fv) ∈ RT×D′ are processed by the Transformer (Vaswani et al., 2017) to generate contextualized audio-visual representation. Target units z ∈ {1, . . . ,K}T for each frame are assigned by an unsupervised clustering method. Given the output probability p from the fused representation and the target units z, the model is trained with the following loss:
L = − ∑
t∈M log pt(zt)− α
∑ t/∈M log pt(zt), (1)
where M is a union set of masked frames for the
audio and visuals stream and α is weighing factor for the unmasked regions.
As in AV/u-HuBERT, we apply modality dropout to encourage the fusion of audio and visual representation space. In the modality fusion, either sequence of fa or fv can be dropped out by the modality dropout probability pm and filled with zero. In the case of dropping out one of the modalities, fa is selected with an audio dropout probability pa to have Fusion(0, fv).
With the masked prediction objective on audioonly targets, the continued training of audio-visual SSL aligns audio-visual and visual-only representations to corresponding audio-only representation, which is established in the previous A-only SSL.
Besides the audio-to-visual representation alignment, clean-to-noisy audio representation is also aligned during the continued AV SSL training phase by using potentially noised audio as audio inputs. This is implemented by randomly adding noises to clean speech audio during the training.
After self-supervised learning, we fine-tune with sequence-to-sequence cross-entropy loss for downstream tasks.

heading: 3.3 Learnable Audio Feature Extractor
text: XLAVS-R has the same model architecture as AV/u-HuBERT except for the nature of the audio features provided as inputs to the Transformer encoder. We jointly train a convolutional audio feature extractor (AFE) as that in wav2vec 2.0 (Baevski et al., 2020) for high-dimensional audio inputs. This differs from AV/u-HuBERT which uses lower-dimensional filterbank features. Our approach provides more capacity for multilingual models and their cross-lingual inferences.

heading: 3.4 Single-Round AV Training with Self-Supervised Audio-Only Targets
text: AV-HuBERT (Shi et al., 2022a,b) and u-HuBERT (Hsu and Shi, 2022) require multiple pre-training rounds, with training targets switching from quantized audio-only local features to quantized audiovisual contextualized representation. In each round, the AV model is re-trained from scratch. Hence, the model quality gains across training rounds come from the updated training targets. While these iterative rounds help the model encode more advanced contextual and bi-modal information in the later stage, the resulting training process is complex and computationally expensive.
To improve training efficiency, we propose to create first-round training targets from a contextualized audio-only representation instead of local features that have noisier, lower-level information. This accelerates the masked prediction learning of high-level semantic information in the first round and reduces the necessity for additional training rounds. We obtain audio-only contextualized representation from a self-supervised multilingual speech model described in Section 3.1.

heading: 4 Experiments
text: Data. For AV pre-training, we leverage a total 1.2K hours of data in 9 languages: English (En), Arabic (Ar), German (De), Greek (El), Spanish (Es), French (Fr), Italian (It), Portuguese (Pt) and Russian (Ru). We also experiment with additional data for domain and language coverage. On top of the above 1.2K hour data, we add 7.1K hours of inhouse AV data in 100+ languages and train XLAVSR with a total of 8.3K hours in 100+ languages, as summarized in the last row of Table 1. The number of hours we use in the training of each experiment setup for 9 language we focus on are shown in Table 6.
For A-only pre-training, we follow Babu et al. (2022) for training on 436K-hour data in 128 languages.
For all the pre-trained models, we perform multilingual fine-tuning on MuAViC labeled data for audio-visual speech recognition (AVSR) in all 9 languages and audio-visual speech-to-text translation (AVS2TT) in 6 language pairs into English— El, Es, Fr, It, Pt, Ru.
We use MuAViC for in-domain evaluation on both audio-only and audio-visual test modes. We use FLEURS (Conneau et al., 2023) for audio-only out-of-domain (OOD) evaluation. We report an average of all 9 languages for AVSR and 6 Xto-English language pairs for AVS2TT in both indomain and out-of-domain evaluation.
We remove extremely short utterances (less than 0.2 seconds) and long utterances (more than 20 seconds) for better training stability.
Modeling. We build XLS-R (§3.1) and XLAVSR (§3) models in two model sizes: 300M and 2B. The number of encoder layers/embedding dimension/feed forward dimension are 24/1024/4096 and 48/1920/7680 respectively. For audio-visual training targets of XLAVS-R, we extract audio-only speech representation from the 36th layer of XLS-
R 2B and quantize it with 2000 k-means clusters. For fine-tuning AVSR and AVS2TT models, we follow Anwar et al. (2023) to add a Transformer decoder that has 6 layers, an embedding dimension of 768, and feed forward network dimension of 3072. The visual feature extractor of our model is modified ResNet-18 as in AV-HuBERT and prior lipreading works (Stafylakis and Tzimiropoulos, 2017; Martinez et al., 2020; Ma et al., 2021). For XLAVS-R, we set both pm and pa to 0.5 for pretraining which is the same with AV/u-HuBERT. For fine-tuning all pre-trained models, we set pm and pa to 0.5 and 0, where the models use 50% AV modality and 50% A-only modality. We set α as 1.0 following the AV-HuBERT models1.
Noise Augmentation. Following Shi et al. (2022b) and Anwar et al. (2023), we randomly augment the input samples with additive noises at 0dB of signal-to-noise ratio. Noise audio clips are sampled from the MUSAN dataset (Snyder et al., 2015) as well as drawn from MuAViC train set for babble type. We augment 25% of the input in
1https://facebookresearch.github.io/av_hubert
pre-training and 50% in fine-tuning with noises. Evaluation. We evaluate AVSR and AVS2TT in audio-only (“A”) and audio-visual (“AV”) test modes, where the former leverages only audio modality in inference while the latter leverages both audio and visual modalities. We select the best checkpoint by validation accuracy for inference. We use beam search with default hyper-parameters and beam size of 5. For AVSR, we apply Whisper text normalizer (Radford et al., 2023) before calculating WER (word error rate). For AVS2TT, we compute BLEU (Papineni et al., 2002) using SacreBLEU (Post, 2018) with default options and the built-in 13a tokenizer. For the simulation of noisy test environment, we add babble type of noises drawn from MuAViC test set.

heading: 5 Results

heading: 5.1 Multilingual Speech Recognition
text: We compare our XLAVS-R representations against the AV-HuBERT and u-HUBERT English-only representations when fine-tuned for multilingual speech recognition in Table 2, reporting WER on
both clean and noisy inputs in the upper and lower half of the Table respectively.
In the clean setup, the XLAVS-R models perform on par or better than the baselines in both audio-only and audio-visual test modes. The XLAVS-R 300M model outperforms the AVHuBERT and u-HuBERT, English-only AV pretrained models of similar size, by up to 9% WER and 8.7% WER respectively for average audio-only and audio-visual test modes. Our larger XLAVS-R 2B outperforms the English-only baseline by an average 11.5% WER and the average of 11.4% WER respectively for the two modes. XLAVS-R 300M and 2B both outperform the baselines by large margins in every non-English language. English WER slightly increased in XLAVS-R 300M model compared to the English-only pre-trained baseline by 0.5% WER. We attribute this effect to inter-language competition (Conneau et al., 2020; Chang et al., 2023), since XLAVS-R 300M covers 128 languages with the same capacity as the English-only baselines (Table 1). Increasing the XLAVS-R model capacity to 2B achieves the lowest WER of all models in both A and AV modes. The OOD results on FLEURS confirm that our
models also maintain reasonable Audio-only performance on other existing benchmarks.
In the noisy setup, where we add multilingual babble noises to clean speech inputs, the XLAVS-R models improve performance across the board, in all languages and in both modes. In the AV mode, the average WER for XLAVS-R 2B drops significantly to 37.3%, which remarkably approaches that of the AV/u-HuBERT baseline in clean settings.
Overall, XLAVS-R models outperform the two baselines by a large margin on average in clean and noisy settings, while remaining competitive with English-only pre-trained models in English. Results of experiments with additional AV data for domain and language coverage during AV pretraining are available in Appendix 5.5 and Table 4. We also show that the findings are consistent with different types of noise in Appendix A, Table 7-8.

heading: 5.2 Multilingual Speech-To-Text Translation
text: Next, we evaluate models fine-tuned for the X-En AVS2TT tasks (Table 3), reporting BLEU scores in clean (top) vs. noisy settings (bottom).
In the clean setup, XLAVS-R 300M outperforms the English-only pre-trained AV/u-HuBERT by up to 2.2 and 2 BLEU average scores for audio-
only and audio-visual modes. It consistently outperforms the baseline by large margins in all directions and test modalities, suggesting that multilingual pre-training is essential for cross-lingual ability in the downstream tasks. Our best model, XLAVS-R 2B outperforms the English-only baselines by even wider margins (up to 3.9 and 3.4 BLEU average scores respectively for the two modes).
In the noisy setup, we simulate noisy environments as in AVSR tasks by adding multilingual babble noises to clean speech sources. XLAVS-R 300M outperforms the English-only baselines of similar size largely by up to 5.4/3.5 average BLEU improvement on A/AV, and 6.6/4.8 average BLEU for 2B model.
Overall, these results show that our XLAVS-R models outperform two baselines by a large margin in all translation directions for both clean and noisy settings, both A-only and AV test modes, and both in-domain and OOD. Results of experiments
with additional AV data for domain and language coverage during AV pre-training are available in Appendix 5.5 and Table 5.

heading: 5.3 Ablation Experiments of XLAVS-R
text: In Figure 22, we validate the effectiveness of the key changes from AV-HuBERT to XLAVSR, step-by-step starting from AV-HuBERT pretrained on MuAViC-En. First, simplifying the training process – single-round AV-HuBERT training with self-supervised contextualized audio-only units from XLS-R (+ Single-round w/SSL units) instead of the original 5 training rounds – actually yields moderate improvements. Second, switching to a learnable audio feature extractor (+ Learnable AFE) shows the biggest improvements in both clean and noisy settings, especially in the low-resource languages (El and
2Appendix Table 9 contains the exact numbers plotted.
Ru). Third, augmenting AV pre-training with multilingual data (+ MuAViC non-English) is the second-highest improvement indicating that multilingual audio-visual representation learning is critical in the self-supervised pre-training stage. Last, the introduction of the audioonly pre-training stage and multilingual audioonly resources (+ A-only pre-training (XLAVS-R)) further improves both A and AV performance in the in-domain evaluations. It yields even higher improvements in OOD evaluations, suggesting that audio-only pretraining may enhance robustness against domain shift.
In sum, all of the design contributions of XLAVS-R are shown to be effective.

heading: 5.4 Zero-shot Audio-Visual Inference
text: We compare the performance of A-only fine-tuning vs. AV fine-tuning: this matters when labels for a given language in downstream task are not available for AV fine-tuning, and thus we need to rely on the zero-shot audio-visual ability from the pretrained AV representation. In A-only FT, we exclude any visual modalities for all languages.
For 300M models (Figure 3a), XLAVS-R shows only a mild performance degradation in the noisy AV mode from AV FT to zero-shot A-only FT (41.2 WER to 48.2 WER, 17% increase rate) and almost none in clean. By contrast, the AV-HuBERT base-
line shows severe degradation even in the clean setting—56.1 to 79.8, 42.25% in noisy, and 27.7% in the clean setting. The XLAVS-R model thus effectively transfers knowledge from the pre-trained AV representation and leverages it for downstream tasks, even though there is no visual modality involved at all during the supervised fine-tuning.
For 2B models (Figure 3b), although AV FT has better recognition performance in a clean setting (23.6 vs 25.0 WER in AV mode), A-only FT model has even better performances over AV FT mode in noisy settings tested on AV mode (37.2 vs 37.3 WER), showing that the increased model capacity improves its zero-shot ability. The result of the scaled-up model suggests that XLAVS-R relaxes the need for labeled AV datasets of downstream tasks to have noise robustness.

heading: 5.5 Results with Additional Training Data
text: We also experiment with additional data for domain and language coverage during the audio-visual selfsupervised learning phase.
Table 4 shows AVSR performances of 9 languages on OOD and in-domain test sets. On the indomain test set, the model pre-trained on a scaledup AV training set shows improved AV average performances of 0.2% WER for both clean and noisy environments. In contrast to the slight improvement or even the degradation for noisy A-only
mode in the in-domain, the 8.3K model shows significant improvements in OOD for both clean and noisy settings by up to 6.1% WER on average.
Table 5 shows AVS2TT performances of six Xto-En pairs on OOD and in-domain test sets, where the trend of the prominent improvement in OOD is similar to that of AVSR comparison. Additional AV SSL dataset improves the average performance of in-domain noisy setting by 0.7 BLEU for A-only test mode and 0.4 BLEU for AV test mode, while slightly degrades in clean settings by 0.1 BLEU in A-only test mode and 0.2 in AV test mode on average. In out-of-domain evaluation, augmented data in AV SSL enhances the translation quality by up to 1.5 BLEU on average in noisy environments.
Overall, scaling up audio-visual pre-training data enhances robustness against domain shift while also remains effective for noisy robustness with audio-visual input in the in-domain.

heading: 6 Conclusion
text: We introduced XLAVS-R, a cross-lingual audiovisual speech representation for noise-robust speech perception in over 100 languages. Since audio-visual data is harder to come by than audioonly data, we injected the visual modality by continuing training an audio-only pre-trained model. We also design a simpler yet more effective training scheme and improved architecture compared to previous state-of-the-art models. Extensive evaluation on the MuAViC benchmark shows XLAVS-R achieves SOTA performance on both audio-visual speech recognition and translation tasks, yielding particularly large improvements in noisy settings. We also show that XLAVS-R effectively leverages unlabeled audio-only multilingual speech data and audio-visual data in self-supervised learning resulting in enhanced zero-shot audio-visual ability for downstream tasks, and extended AV pre-training data augments robustness against domain shift.
Limitations
Our findings are inherently limited by the settings of our empirical evaluation. For instance, we simulate noisy environments only with the “babble” sound in testing experimental setup (§4), and it remains to be seen how other types of noise might impact our model. In addition, while we include a diverse set of languages with varying amounts of resources in our evaluation, we did not evaluate translation out-of-English and between nonEnglish languages.

heading: A Generalization to Different Types of Noise
text: As we mentioned in the Limitation section, the evaluation phase in the main tables is conducted on a single type of synthetic noise, Babble. To confirm the general effectiveness of XLAVS-R on noise robustness, we further experiment AVSR with different types of noise, Music and Noise from MUSAN dataset following Shi et al. (2022b).
Table 7 and 8 show the results in average and language-wise WER of each models on different types of noise. Overall, the noise robustness of Audio-Visual speech processing still holds with different kinds of noises. All AV models manage to get a higher quality of speech recognition with visual information than A-only mode in all new types of noise. Babble turns out to be the toughest noisy environment as shown in the above table, while Music and Noise are even able to achieve scores that are close to the clean environment with AV mode, which is consistent with Shi et al. (2022b).

heading: B Future Works: Visual-only Experiment
text: The main focus of this research is on having the best noise-robust speech recognition and translation performance by leveraging stronger audio components and visual information, rather than visualonly input. Therefore, the presented AV models may not provide significant visual-only speech recognition capabilities, especially in non-English languages. The results of visual-only experiment in English are 52.4 WER for AV-HuBERT and 55.9 WER for u-HuBERT, and non-English results are around 90-100 WER. We plan to conduct comprehensive VSR experiment with XLAVS-R and further explore how to develop zero-shot visual speech recognition (VSR) ability while maintaining state-of-the-art audio-visual performances as a future work.

heading: Future Work
text: We plan to conduct comprehensive VSR experiment with XLAVS-R and further explore how to develop zero-shot visual speech recognition (VSR) ability while maintaining state-of-the-art audio-visual performances as a future work