Uni-Dubbing: Zero-Shot Speech Synthesis from Visual Articulation (2024)

Abstract: In the field of speech synthesis, there is a growing emphasis on employing multimodal speech to enhance robustness. A key challenge in this area is the scarcity of datasets that pair audio with corresponding video. We employ a methodology that incorporates modality alignment during the pre-training phase on multimodal datasets, uniquely facilitating Zero-Shot generalization through the process of freezing the video modality feature extraction component and the encoder module within the pretrained weights, thereby enabling effective cross-modal and cross-lingual transfer. We have named this method ‘Uni-Dubbing’. Our method finely tunes with both multimodal and single-modality audio data. In multimodal scenarios, it achieves a reduced word error rate (WER) of 31.73%, surpassing the previous best of 33.9%. It also excels in metrics like tone quality and synchronization. With single-modality audio, it achieves a WER of 36.08%, demonstrating adaptability to limited data. Its domain generalization capabilities are proven across various language tasks in video translation and audio generation. Trained on 433 hours of audio data, it surpasses techniques using 200 hours of audiovisual data. The code and demo are available at https://diracer.github.io/unidubbing.

heading: 1 Introduction
text: With the widespread use of short videos and online meetings in daily life and the workplace(Gupta et al., 2023), the barrier of cross-linguistic communication has become an urgent problem, and thus multimodal technologies have attracted much attention(Yemini et al., 2023). Recently, many researchers have conducted corresponding studies in this area, such as lip reading task(Assael et al., 2016; Jin et al., 2023; Li et al., 2023) that transfers video domain to text domain, Lip task(Prajwal
**Corresponding author
et al., 2020; Kim et al., 2021; Michelsanti et al., 2021; Mira et al., 2022b) that transfers video domain to audio domain, and lip translation(Huang et al., 2023) that converts to the target language directly based on lips. In the case of the field of visual tasks, the biggest challenge for researchers is the extreme scarcity of training data. In addition, the relationship between lips and speech is not always a simple one-to-one mapping; for example, the same word may have very different lip shapes for people with different accents(Choi et al., 2023a). Therefore, maintaining accurate intonation poses a significant challenge, and this has led to the emergence of many important research findings.
For these reasons, we adopt the strategy of using discrete units as intermediate targets, i.e., transforming audio and video data into discrete units for alignment, which can effectively circumvent the disadvantage of insufficient paired audio and video data. On top of this, we employ the RVQ(Défossez et al., 2022) module thus enabling the method to achieve better timbre preservation, i.e. high fidelity, after Full-Shot training. Furthermore, in order to cope with the lack of data for contemporary visual tasks, we also use mHubert(Polyak et al., 2021) and K-means of re-combining with discrete units, which enables our model to achieve better semantic consistency and reach Zero-Shot capability. As mentioned earlier, the barriers to cross-language communication are equally significant challenges and a lot of good work has emerged, but unfortunately none of the current methods have been able to achieve Zero-Shot cross-language video translation yet. We further explored learning cross-language and cross-modal Lip2Wav mappings from the audio domain, i.e., Zero-Shot transspeech, based on the Zero-Shot Lip2Wav model, have verified that the method is capable of crosslanguage migration.
In summary, our goals in the current crosslanguage video-to-speech translation are twofold:
10082
1) High quality and low error: the requirement to be able to recognise the gender in a video so as to generate the corresponding tones with minimal error is very challenging. 2) Zero-Shot: the ability of the reasoning process to achieve Zero-Shot is crucial for practicality when considering video translation.
Based on these two goals, in this paper, the innovation of this study lies in proposing a framework that requires only cross-linguistic audio speech training, without the need for visual speech training inputs, to achieve direct synthesis of visual speech to cross-linguistic audio speech. This framework can predict the corresponding audio speech output by analyzing an individual’s lip movements, and this prediction is not limited to the language system of the input visual speech. Our method utilizes an advanced Zero-Shot learning strategy (Cheng et al., 2023b) that aligns audio and visual phonemes with audio data alone during the training process, thus enabling the prediction of audio outputs in a target language that has not been seen before in seemingly impossible cross-modal scenarios. The main contributions of this paper are:
• Our cross-modal Zero-Shot transfer approach for the Lip2Wav task, trained exclusively with target audio, matches top Full-Shot models in WER, sound quality, and synchronization. • Our method in the Lip2Wav task on the LRS3 dataset attains state-of-the-art results in WER, ESTOI, LSE-C, and LSE-D, achieving partial timbre preservation to distinguish voice characteristics of unseen speakers. • Our cross-lingual audio generation technology creates target language audio from singlelanguage videos, eliminating the need for duallanguage video training. This streamlines training and lessens the need for extensive datasets in cross-lingual dubbing, while also reducing noise.

heading: 2 Related Work
text: In our paper, for the cross-language Lip2Wav synthesis task we mainly divide it into two steps: first implementing high-fidelity video-to-speech synthesis, followed by Zero-Shot cross-language videoto-speech translation. A great deal of excellent research work has preceded our study.

heading: 2.1 Video to Speech Synthesis
text: Video speech synthesis techniques(Cooke et al., 2006; Afouras et al., 2018a; ?; Cheng et al., 2023a)
that dub silent videos have received a great deal of attention from researchers in the recent past. Prajwal et al. (2020) presented the Lip2Wav, which utilizes a sequence-to-sequence architecture, enabling it to accurately capture contextual information and generate precise audio. Hong et al. (2021) trained a multimodal memory network, VVMemory, to store and recall audio features corresponding to visual inputs so that audio information can be accessed exclusively through visual inputs during inference. Vougioukas et al. (2019) introduced an end-to-end temporal model based on GAN, capable of generating speech that synchronizes seamlessly with silent videos, presenting a convincing and difficult-to-distinguish quality. Additionally, there have been several recent papers based on GANs(Kim et al., 2021; Hong et al., 2022; Mira et al., 2022b). Most recently, a new method based on diffusion, called DiffV2S, has been proposed by Choi et al. (2023a) who introduced a novel speaker embedding extractor guided by visual information and simultaneously developed a diffusion-based video-to-speech synthesis model. Choi et al. (2023b) built upon the Lip2Wav model by incorporating quantized supervised speech representations, namely speech units, for synthesizing intelligible speech from silent videos.
However, despite the fact that all the aforementioned related methods have their own merits, the problem of lack of training data for the visual task mentioned in the previous section remains unsolved. With this in mind, we train our model by using discrete units as intermediate comparison targets in the audio and video domains, thus no longer relying on paired audio and video data.

heading: 2.2 Cross-language Translation
text: The task of cross-language translation is also a very challenging and important endeavour that also receives a lot of attention.(Lavie et al., 1997; Wahlster, 2000; Nakamura et al., 2006; ITU, 2016). Tjandra et al. (2019) introduced a discrete representation of the source language to target speech into the cascaded S2ST system, where this discrete representation is predicted by a separately trained VQVAE and subsequently utilized by the VQVAE decoder to generate the target speech spectrogram. Zhang et al. (2021) proposed the XLVAE model to enhance the discretization and reconstruction capabilities of VQVAE through cross-linguistic speech recognition. Lee et al. (2021) utilizes a separately trained vocoder, which includes a duration predic-
tor, to directly predict waveforms from discrete representations. Jia et al. (2019) first introduced a model based on a sequence-to-sequence architecture capable of end-to-end training and inference. To improve translation quality and overgeneration, Jia et al. (2022) presented Translatotron2, which consists of a speech encoder, a language decoder, an acoustic synthesizer, and a single attention module that connects them together. There is also some work that attempts to introduce visual speech to enhance robustness in the translation process(Huang et al., 2023).
To the best of our knowledge, paired crosslingual audio-video datasets are currently very sparse. This scarcity results in only one existing model capable of achieving cross-lingual Lip2Wav translation. Instead, in direct contrast with the methods mentioned above, our innovative discrete-unit-based approach can successfully cross these dataset barriers, thus learning cross-language visual-phoneme mappings with Zero-Shot crosslanguage lip-synthesis translation capability.

heading: 3 Method

heading: 3.1 Overview
text: The overview of this paper is depicted in Figure 1. Figure 1a) describes the training process for high-fidelity speech synthesis, while Figure 1b) illustrates the training flow for two tasks: crossmodal and cross-language. The main differences between these tasks lie in the modality used during training, the method for generating discrete units, and the treatment of predicted discrete units for synthesizing speech. Additionally, for Zero-Shot training, it is necessary to freeze the encoder to retain the visual knowledge acquired during the pretraining phase.

heading: 3.2 High-Fidelity Lip2Wav
text: While the state-of-the-art ReVISE model (Hsu et al., 2023) achieves leading performance in Lip2Wav synthesis on the LRS3 dataset, it does not preserve the speaker’s timbre during speech synthesis. To address this issue, we propose a novel approach that utilizes acoustic tokens derived from
the Hifi-Codec (Yang et al., 2023). The Hifi-Codec consists of an audio encoder, a Residual Vector Quantizer (RVQ), and an audio decoder: Consider an audio signal x with a length of d and sampled at a rate of sr, resulting in a total duration of T = d/sr.
1) Initially, the audio encoder E, comprising multiple convolutional blocks, processes the input audio. This encoder extracts features and outputs a latent representation z. 2) Subsequently, the Residual Vector Quantizer Q employs vector quantization layers to convert z into a discrete representation zQ. In this process, the speech utterance x is encoded as a sequence of acoustic tokens [a1, a2, ..., aT ], where each token ai is an element of the set {0, 1, ...,K2−1}, with 1 ≤ i ≤ T . These acoustic tokens are the discrete units that we focus on in our training. 3) The audio decoder G reconstructs the signal x̂ from the highly compressed latent representation zQ. This algorithm efficiently quantizes the encoder output by iteratively refining the residual, which helps in preserving important information while reducing redundancy. Further, to address the challenges of temporal synchronicity in Lip2Wav tasks, we have innovated upon the existing AV-Hubert model. We have replaced the AV-Hubert decoder with a new structure.
Our adaptation involves a unique decoder structure, which includes three transposed convolutional layers. Each layer has a kernel size (K) of 4, a stride (S) of 2, padding (P ) of 1, and output padding (Op) of 1. This configuration is meticulously designed to more accurately align lip movements with the generated speech, thereby enhancing the synchronicity that is crucial for effective Lip2Wav synthesis. The output size (O) of each transposed convolutional layer is calculated using the formula:
O = ((I − 1)× S +K − 2× P ) +Op (1) where I denotes the input size.

heading: 3.3 Zero-Shot Lip2Wav Model Adaptation
text: To overcome the challenge of scarce paired audiovisual datasets, we loaded the pre-trained weights of AV-Hubert and focused on fine-tuning with pure audio data. To validate the effectiveness of our approach, we adopted the same Zero-Shot configuration on the LRS3 dataset as uHubert. The AV-Hubert model, pre-trained on paired audiovisual data, achieves multimodal alignment by mapping visual speech and audio speech to the same
phoneme space. During the fine-tuning phase with pure audio data, we froze the decoder and only trained the final transposed convolution layer to preserve the multimodal alignment knowledge acquired during pre-training. In the inference process, the model processes silent lip videos, predicting the corresponding speech discrete units solely based on lip movements. This Zero-Shot learning strategy enables the model to effectively synthesize speech from unseen lip movements, enhancing its robustness in diverse scenarios.
To further validate the effectiveness of our method, we fine-tuned the model using discrete units generated in other languages (e.g., Spanish, French), which were languages not encountered during pretraining. This approach not only enables the model to generate speech from lip movements but also to translate it into different languages. For example, during inference, an English spoken video could be decoded into the audio of another language, simplifying the process of speech synthesis and translation without the need for separate models for each task.
In these two tasks, our model does not contain any speaker embeddings and is unable to implicitly acquire visual feature embeddings of the speaker during the fine-tuning phase, eliminating the need to replicate the speaker’s acoustic information. Therefore, we used semantic tokens generated by the mHubert and kmeans methods as target units. Compared to acoustic information, semantic information has broader applicability, making the use of semantic tokens more conducive to generalization in cross-modal and cross-language Zero-Shot tasks.

heading: 3.4 Training Object
text: In this study, the focus is on predicting discrete units, for which the cross-entropy loss function
L = ∑
t
C∑
j=1
zjt log f j t (x̃a, xv)
is consistently employed. This formula calculates the loss by summing over all frames (t) and across the C units in the vocabulary. The term zjt denotes the one-hot encoded label of the j-th unit in the t-th frame, and f jt (x̃a, xv) represents the predicted probability distribution over the discrete units for the same frame and unit, as outputted by the enhancer.

heading: 4 Experiment

heading: 4.1 Datasets
text: LRS3 Dataset LRS3 (Afouras et al., 2018b) is an extensive and open-source benchmark collection for visual speech recognition research, commonly known as lip-reading. This dataset is the successor to the LRW (Chung and Zisserman, 2017a) and LRS2 (Afouras et al., 2018a) datasets and features a vast array of labeled video content with corresponding textual transcriptions, primarily sourced from TED Talks.
LRS3-T Dataset LRS3-T (Huang et al., 2023) is a new audio-visual translation dataset that has been generated from the LRS3 dataset through a cascading process, combining Neural Machine Translation (NMT) and Text-to-Speech (TTS) technologies. This intricate processing sequence culminated in a parallel audio-visual translation dataset comprising 200 hours, encompassing both the original source videos and the translated speech in the target language.
MUSAN Dataset MUSAN (Snyder et al., 2015) is a collection of music, speech, and noise recordings suitable for audio processing tasks such as speech activity detection and machine learning applications. It features 60 hours of speech from various sources, over 42 hours of diverse music tracks, and 6 hours of environmental and technical noises. We used it to generate various types of noise which were added to the original audio, in order to test the translation task’s resistance to noise interference.

heading: 4.2 Evaluation
text: In our study, we evaluate Lip2Wav and audio-video translation using key metrics. For semantic accuracy, we use WER , and for sound quality, we em-
ploy the Extended Short-Time Objective Intelligibility (ESTOI). Synchronization is measured using LSE-D (predicted audio-video temporal distance) and LSE-C (prediction confidence), as per SyncNet (Chung and Zisserman, 2017b). Our method approximates the speaker’s voice, thus we use the Mean Opinion Score (MOS) for evaluating timbre. To ensure consistency with other studies, we adopted a scoring system ranging from 1 to 5, with increments of 0.5 points. For each model, we randomly selected 50 samples for evaluation. We recommend listening to our website’s audio samples for a practical understanding.
For language translation, we apply the BLEU (Papineni et al., 2002) score to evaluate the accuracy and fluency of speech generation in different languages, comparing machine-generated text to reference texts.

heading: 4.3 Results

heading: 4.3.1 High-Fidelity Video-to-speech synthesis
text: Unlike other datasets that may concentrate on short phrases or isolated words, LRS3 offers longer sequences of speech, enabling more complex and contextually rich lip-reading tasks. Since most speakers only give a TED talk once, the LRS3 dataset is multi-speaker, with no overlap between the speakers in the test set and those in the training set. Consequently, most methods using fixed ID speaker embeddings are ineffective for the LRS3 dataset without altering its test set. This reflects real-world application needs more accurately, as the models we train should be effective for unseen speakers. This paper focuses on speaker generalization on the original LRS3 dataset, aiming to generate audio that is perceptually credible for speakers it has never encountered before.
As shown in Table 1, DiffV2s and ReVISE significantly outperform various previous methods,
with both achieving a WER below 40% and superior sound quality as evidenced by the ESTOI metric. Our results clearly surpass all prior work in these two measures, achieving a WER of 0.296 and an ESTOI of 31.96%. This is because acoustic units preserve finer details, making the generated audio easier for automatic speech recognition (ASR) systems to understand. In terms of synchronization, our model also achieved the highest rankings on the LSE-C and LSE-D metrics, surpassing all previous methods. This achievement is primarily attributed to our modifications to the original AV-Hubert decoder. We transformed it from a sequence-to-sequence model to one utilizing transposed convolutions. This change effectively ensures that the ratio between the input and output lengths of the model remains constant, thus maintaining a consistent proportional relationship between the generated audio length and the input video length. If the original AV-Hubert decoder were used, the LSE-C and LSE-D scores would be 4.65 and 9.21, respectively. Although our WER has only increased by 1.17% relative to the ReVISE, the additional fine-grained acoustic information plays a crucial role in improving synchronization. This allows our method to outperform ReVISE in terms of synchronization even when using the same transposed convolution decoder.
While quantitative metrics are important, they are not the key focus of our task. The primary contribution of our work lies in generating audio that retains partial speaker information without using the identity of the speaker. In contrast, ReVISE produces audio in a single female voice for all outputs, regardless of whether the video features a male speaker. Due to the absence of explicit speaker identity information, our method is unable to fully replicate the unique acoustic characteristics of individual speakers. However, due to its use of implicit visual embeddings and acoustic discrete units, the system is capable of generating distinct male or female voices, depending on whether the videos feature male or female speakers as protagonists. While the synthesized voices may not precisely match those of the original speakers, they do preserve certain overarching characteristics, such as gender distinctions and, to some extent, age differences. We believe this aspect is significant. In cases where humans have not seen the speaker, they cannot deduce the exact timbre from the video but can infer such general voice characteristics. The voices generated by our model align with human
perception, thus meeting human expectations and requirements.Benefiting from this approach, our MOS evaluation achieved an optimal score of 4.16.

heading: 4.3.2 Zero-Shot from Audio to Video
text: Table 1 reveals that our method achieves impressive results even when trained solely with audio, without using any video data. The sound quality, measured by the ESTOI, is 0.235. This performance is comparable to the previous three works, ranking just behind DiffV2S and ReVISE. Surprisingly, despite the absence of video data during training, the synchronization of our generated audio is quite good, significantly surpassing the Full-Shot VCAGAN and Multi-task methods, and comparable to other approaches. Most importantly, our method achieves a WER of 36.08%, which is only slightly inferior to ReVISE’s 33.9% and better than all previous Full-Shot methods. These results indicate that our approach effectively utilizes the knowledge embedded in the pre-trained model to achieve outstanding performance, while significantly reducing data collection costs, requiring only pure audio data without corresponding lip-synced video.
Due to the mHubert audio encoder operating at 50 frames per second and the AV-Hubert video encoder at 25 frames per second, we employed a convolutional layer to align the two. It was imperative to set the stride of this transposed convolution to 2, a fixed requirement. However, the size of the convolutional kernel significantly impacted the final results. To determine the optimal kernel size,
we conducted multiple experiments. For comparison, we also tried the alignment method used in AV-Hubert pre-training, which involves downscaling the audio labels’ discrete units to 25 frames per second by extracting them at intervals. In this scenario, we set the stride of the transposed convolution to 1 and chose a convolutional kernel of an odd size.
As shown in Figure 2, all models using oddnumbered kernel sizes performed worse in terms of WER compared to those using even-numbered kernels. Specifically, smaller even-numbered kernels, such as 2 and 4, significantly reduced accuracy. However, the performance improvement became marginal when the kernel size increased to 8 or larger. Based on this finding, we selected a kernel size of 8, balancing optimally between temporal resolution and computational efficiency, crucial for effective synchronization between audio and video modalities. Additionally, we experimented with the original fully connected (FC) layer. The results indicated that using an FC layer instead of transposed convolutions yielded the worst outcomes, highlighting the effectiveness of transposed convolutions in extracting local information for our task.
A noteworthy observation is that methods comparable to Zero-Shot in terms of ESTOI generally have a WER exceeding 60%. This implies that Zero-Shot is capable of acquiring a substantial degree of semantic knowledge from pre-training, but it slightly lags in generating audio quality, failing to reach a level commensurate with its semantic proficiency.

heading: 4.3.3 Translate from Video
text: Building on the concepts discussed earlier, collecting audio and its corresponding lip-synchronized video data presents significant challenges. These challenges further escalate when the task is extended to multiple languages. Our objective is to utilize datasets composed of video-audio pairs in a single language, combined with multilingual audio datasets, to make this approach applicable to multilingual audio generation. This strategy aims to efficiently utilize existing resources while addressing the challenges of multimodal and multilingual datasets.
In our study, we compared the performance of existing Full-Shot methods with our Zero-Shot method in English to Spanish (En-Es) and English to French (En-Fr) translation tasks, with detailed results presented in Table 2. We also tested the
robustness of our model under different modalities and specific noise conditions. Firstly, we found that under given noise conditions, the BLEU scores using both visual and audio modal inputs were consistently higher than those using only audio input. This demonstrates the auxiliary role of visual information in enhancing audio in noisy environments, highlighting the importance of visual data. Especially under babble noise conditions, with a signal-to-noise ratio (SNR) of -5, the BLEU score for pure audio input was even lower than that for pure visual input, further emphasizing the significance of lip-reading translation. We also provided experimental data under various noise types and intensities in the appendix. In pure visual translation, Full-Shot methods typically outperform Zero-Shot methods. However, the Zero-Shot method still performs commendably in terms of BLEU scores and MOS, achieving BLEU scores of 16.99 and 19.90, and MOS of 3.73 and 3.70, respectively.
We replicated Av-Transpeech and fine-tuned it using multimodal data of varying durations, with detailed results shown in Figure 3. The figure demonstrates that the BLEU score obtained by finetuning with 433 hours of pure audio data is roughly equivalent to that achieved with just 220 hours of audiovisual data. During the pre-training phase, we mapped the audiovisual data to the same phoneme space. This result indicates that the knowledge in this phoneme space is equally applicable to crosslingual audio, enabling us to align the source language video with the target language audio through
pure audio fine-tuning, resulting in the current BLEU scores. This finding not only validates the effectiveness of our method but also emphasizes the feasibility of using a large amount of pure audio data as an alternative in scenarios where it is challenging to collect extensive multimodal data.
In our study, as illustrated in Table 2, we additionally conducted an experiment to investigate the translation results obtained using our Zero-Shot method without freezing the encoder. This part of the experiment primarily aimed to assess the role of freezing the encoder in preserving pre-trained knowledge. Under this setup, we observed a significant phenomenon: the BLEU scores for model inference on pure video were zero in both En-Es and En-Fr translation tasks. This result implies that the majority of the visual knowledge acquired during the model’s pre-training phase has been substantially forgotten in subsequent processes.
Furthermore, compared to models that kept the encoder frozen during the inference phase, the models with unfrozen encoders also showed lower resistance to noise. This difference not only reveals the importance of freezing the encoder for maintaining model stability but also reflects the criticality of preserving knowledge acquired during pre-training when dealing with complex and variable visual inputs. Freezing the encoder effectively retains the visual information learned during the pre-training phase, which is crucial for enhancing the model’s accuracy and robustness in parsing and understanding visual data. Therefore, our study not only emphasizes the importance of managing the state of
the encoder in implementing Zero-Shot learning methods but also provides valuable insights for future model design in the intersection of vision and language domains.

heading: 5 Conclusion
text: This paper introduces Uni-Dubbing, an innovative approach trained on multimodal audio-video datasets, which achieved the best WER, ESTOI, and synchronization metrics on the LRS3 dataset. Additionally, by utilizing implicit visual embeddings and acoustic tokens, we successfully preserved partial speaker information on the crossspeaker LRS3 dataset. We then implemented a Zero-Shot strategy, transitioning from audio to video modalities in cross-modal Lip2Wav tasks, and cross-lingual Lip2Wav translation tasks. This method significantly reduces the dependency on multimodal datasets and demonstrates potential for application in a wider range of tasks.
To further validate the practicality of this method, our research utilized only the audio portion of existing multimodal datasets. In future work, we plan to explore the use of larger single-modality audio datasets, aiming to further expand the applicability and enhance the effectiveness of this method. Through such research, we hope to deepen our understanding and utilization of single-modality audio data in multimodal tasks, thereby paving new paths for development in this field.

heading: 6 Ethics Statement
text: In the context of our research, we acknowledge that lip-reading technology holds considerable potential in a multitude of applications, such as facilitating silent commands in noisy environments or enhancing communication for individuals with hearing impairments. The OpenSR system is designed to democratize the development of lip-reading models, particularly for domains where resources are scarce, thereby promoting equality in technology application across different fields and languages.
However, we recognize the ethical implications surrounding the use of speech recognition technology, including the potential for unintended information exposure. It is important to note that effective lip-reading by our model demands specific video criteria, such as front-facing, high-resolution imagery with sufficient frame rates to ensure clear visibility of lip movements. Typically, such conditions are met in environments with close-range cameras or during virtual meetings, not in scenarios where video footage is obtained from a distance or without clear visibility of the mouth region, like most surveillance contexts.
Therefore, while our model advances the field of speech recognition, it is engineered with inherent limitations that naturally restrict its use in situations that could compromise individual privacy. We maintain a commitment to ethical research practices, prioritizing the beneficial impacts of our work while actively mitigating potential risks of misuse that could infringe on personal privacy or be deemed invasive. Our ongoing research includes a strong focus on developing safeguards and protocols to ensure that the technology is used responsibly and ethically.

heading: 7 Limitations
text: The present study is limited to the use of just two modalities: video and audio, thus neglecting the potential benefits of incorporating further modalities. Furthermore, the approach of applying singlemodality Zero-Shot learning, although it minimizes reliance on extensive datasets, inherently results in the inadvertent omission of some portions of the previously acquired knowledge. Consequently, this methodology is not entirely effective in preserving the full spectrum of multimodal alignment knowledge that was initially obtained during the training phase.

heading: A Additional quantitative Results
text: Zero-Shot Kernel Size. The results of cross-modal Zero-Shot experiments conducted on the LRS3 dataset are closely related to the kernel size of the last layer of transposed convolution. Table 3 details these results, including ESTOI, WER, and synchronization rate metrics.
Zero-Shot Translate Data Size. For the Zero-Shot translation task, we present in Table 4 the performance of AV-Transpeech after fine-tuning with varying amounts of data. We compare the results of inference using both audiovisual data and video-only data. We found that for both AVST (Audio-Visual Synchronous Translation) and VST (Video Synchronous Translation) tasks, the effectiveness of our method is similar to that achieved by fine-tuning with a 200-hour multimodal audiovisual dataset.
Zero-Shot Translate Noise Robust. In the main text, we only present the performance of the model under partial noise conditions. Table 5 and Table 6 respectively showcase the results of the Zero-Shot model under frozen and no frozen states across various noise conditions.

heading: B Additional qualitative Results
text: LRS3 Dataset in Lip2Wav Implementation. In Figure 4, we display visualizations of four samples each from the ground truth, our Full-Shot and Zero-Shot methods, and ReVISE, to compare their respective mel-spectrogram outputs. These methods generate mel-spectrograms whose backbone structures maintain a certain degree of similarity, resulting in low WER and minimal differences in retained semantic information for the synthesized speech. However, in comparison, our Full-Shot method produces melspectrograms that more closely resemble real data (Ground Truth) in detail, displaying finer frequency variations and a more continuous temporal sequence structure. This indicates that the Full-Shot approach achieves higher accuracy in audio reconstruction, capturing more of the acoustic features of real speech signals beyond just semantic information. Additionally, our Zero-Shot method shows greater similarity to ReVISE, demonstrating that even when fine-tuned using only audio data, it can retain a considerable level of semantic information. This validates the effectiveness of our method in modal transfer.
In Table 7, we present the results of several audio samples processed through Lip2Wav and subsequently analyzed using ASR. The errors generated by these methods are largely similar, likely stemming from the inherent confusability of the Lip2Wav approach itself. This is because the majority of errors originate from phonetically similar words or phrases, which are exceedingly difficult to overcome in subsequent processing.
LRS3-T Dataset in Cross-Lingual Lip2Wav Translation. In Figure 5, we display the actual spectrograms for En-Es and En-Fr samples, along with the corresponding spectrograms generated by AvTranspeech and our Zero-Shot method. The mel-spectrograms generated by Av-Transpeech show a high degree of similarity to those produced by our method, but both exhibit certain differences from the GT. This is primarily because both methods use discretized units generated in the same way as training targets, hence the information they carry is quite similar, primarily focusing on semantic information. On the LRS3-T dataset, the similarity of the mel-spectrograms generated by these two methods further confirms the Zero-Shot capabilities of our approach.
Table 8 shows GT, Av-Transpeech, and our En-Es and En-Fr translation results. Our translations contain more erroneous words compared to Av-Transpeech, as reflected in the lower BLEU scores reported in the text. However, the locations of errors are similar for both methods, suggesting that pure audio fine-tuning might achieve semantics similar to Full-Shot for the main body of sentences, but there could be confusion in some details. Further research and exploration in this area are needed.

heading: C Zero-Shot configuration
text: On the LRS3 dataset, our applied Zero-Shot configuration is consistent with that of uHubert (Hsu and Shi, 2022). One concern arises: the model might memorize audio-visual pairs from the pre-training period and associate them with unimodal data for Zero-Shot learning, as the dataset used for fine-tuning is a subset of the pre-training data. To address this issue, uHubert conducted experiments on non-LRS3 audio datasets, demonstrating the effectiveness of this configuration. Therefore, we did not seek another out-of-domain audio dataset for experimentation in this task. We directly conducted Zero-Shot experiments on LRS3-T, whose audio data is not only excluded from the pre-training but also differs in language type. Furthermore, ablation experiments regarding whether to freeze the encoder layers also validated the Zero-Shot capability of our method.

heading: D More implementation details.
text: Experiment hyperparameters. Table 9 displays the training hyperparameter configurations for each task in our study, noting that audio masking was not employed in any of the tasks.
ASR toolkit for Evaluation. In this paper, the English ASR used is cited from (Ma et al., 2023). For Spanish and French, we utilize open-sourced ASR models within the fairseq framework (Ott et al., 2019) to transcribe the audios, which is consistent with the ASR used by Av-Transpeech.

heading: Future Work
text: In future work, we plan to explore the use of larger single-modality audio datasets, aiming to further expand the applicability and enhance the effectiveness of this method. Through such research, we hope to deepen our understanding and utilization of single-modality audio data in multimodal tasks, thereby paving new paths for development in this field