AV-TranSpeech: Audio-Visual Robust Speech-to-Speech Translation
Year: 2023

Abstract:
Direct speech-to-speech translation (S2ST) aims to convert speech from one language into another, and has demonstrated significant progress to date. Despite the recent success, current S2ST models still suffer from distinct degradation in noisy environments and fail to translate visual speech (i.e., the movement of lips and teeth). In this work, we present AVTranSpeech, the first audio-visual speech-tospeech (AV-S2ST) translation model without relying on intermediate text. AV-TranSpeech complements the audio stream with visual information to promote system robustness and opens up a host of practical applications: dictation or dubbing archival films. To mitigate the data scarcity with limited parallel AV-S2ST data, we 1) explore self-supervised pre-training with unlabeled audio-visual data to learn contextual representation, and 2) introduce crossmodal distillation with S2ST models trained on the audio-only corpus to further reduce the requirements of visual data. Experimental results on two language pairs demonstrate that AVTranSpeech outperforms audio-only models under all settings regardless of the type of noise. With low-resource audio-visual data (10h, 30h), cross-modal distillation yields an improvement of 7.6 BLEU on average compared with baselines.1

heading: 1 Introduction
text: Speech-to-speech translation (S2ST) models (Tjandra et al., 2019; Zhang et al., 2020; Jia et al., 2021) relying on speech data have achieved high performance and significantly broken down communication barriers between people not sharing a common language, which attracts broad interest in the machine learning community (Huang et al., 2022c; Huang et al.). Among them, direct systems (Lee
∗Equal contributions †Corresponding author
1Audio samples are available at https:// AV-TranSpeech.github.io/.
et al., 2021a,b; Huang et al., 2022d) leverage recent progress on self-supervised discrete units learned from unlabeled speech for building textless S2ST, further supporting translation between unwritten languages (Chen et al., 2022).
As speech production (Huang et al., 2023; Lam et al., 2021; Huang et al., 2022b) is accompanied by the movement of lips and teeth, it can be visually interpreted to understand speech. In recent years, significant research (Shi et al., 2022a; Prajwal et al., 2022) has introduced joint modeling of spoken language and vision: Shi et al. (2022b) investigate to learn lip-based audio-visual speaker embeddings, where the speaker’s mouth area is used alongside speech as inputs. Chern et al. (2022) focus on audio-visual speech enhancement and separation which better integrates visual information. Despite their success, it is unclear how lip can contribute to audio-based S2ST, and how to incorporate visual modality as auxiliary information in S2ST. A visual translator may open up a host of practical applications: improving speech translation in noisy environments, enabling dictation, or dubbing archival films.
Despite the benefits of audio-visual approaches, training direct speech translation models without relying on intermediate text typically requires a large amount of training data, while there are very few resources providing parallel audio-visual speech due to the heavy workload. To mitigate the data scarcity, researchers have leveraged multitask learning (Lee et al., 2021a), data augmentation (Popuri et al., 2022), and weekly-supervised data with synthesized speech (Jia et al., 2022a) in audio S2ST.
In this work, we propose AV-TranSpeech, introducing the first AV-S2ST system without using text. As illustrated in Figure 1, our textless AV-TranSpeech inherits speech-to-unit translation (S2UT) framework (Lee et al., 2021b; Huang et al., 2022d), which consists of an audio-visual speechto-unit translation (AV-S2UT) model followed by
8590
a unit-based vocoder that converts discrete units to speech. AV-TranSpeech complements the audio stream with the auxiliary visual information, which is invariant to speaking environments and promotes system robustness. To tackle the challenges of data shortage, we 1) build upon the recently introduced Audio-Visual HuBERT (AV-HuBERT) which learns contextual representations through self-supervised masked prediction, and show that large-scale pre-training benefits AV-S2ST training; 2) introduce cross-modal distillation and leverage S2ST models trained on the audio-only corpus, which further reduces the requirements of visual data and boosts the performance of visual systems in low-resource scenarios.
Experimental results on two language pairs demonstrate the robustness of AV-TranSpeech in noisy scenarios, outperforming audio-only S2ST under all settings regardless of the SNR and the type of noise. With low-resource audio-visual data (10h, 30h), cross-modal distillation yields an improvement of 7.6 BLEU on average compared with baselines. The main contributions of this work include:
• We propose the first textless audio-visual speechto-speech (AV-S2ST) translation model AVTranSpeech and collect a benchmark dataset LRS3-T which we plan to release.
• We leverage the recent success of audio-visual self-supervised learning for contextual representations and show that large-scale pre-training alleviates the data scarcity issue in AV-S2ST systems.
• We introduce cross-modal knowledge distillation, which further reduces the requirements of visual data and boosts AV-S2ST performances in lowresource scenarios.
• Experimental results on two language pairs demonstrate the robustness of AV-TranSpeech in the noisy environment under all settings regardless of the type of noise, and cross-modal distillation yields a significant improvement compared with baselines.

heading: 2 Related Work

heading: 2.1 Speech-to-Speech Translation
text: Direct speech-to-speech translation has made massive progress to date. Translatotron (Jia et al., 2019) is the first direct S2ST model and shows reasonable translation accuracy and speech naturalness. Translatotron 2 (Jia et al., 2021) utilizes the auxiliary target phoneme decoder to promote translation quality but still needs phoneme data during training. UWSpeech (Zhang et al., 2020) builds the VQ-VAE model and discards transcript in the target language, while paired speech and phoneme corpora of written language are required. Most recently, a textless S2ST system (Lee et al., 2021a) takes advantage of self-supervised learning (SSL) and leverages speech-to-unit translation (S2UT) model followed by a unit-based vocoder that converts discrete units to speech, demonstrating the results without using text data. Popuri et al. (2022) show that self-supervised encoder and decoder pre-training with weakly-supervised data improves model performance. Huang et al. (2022d) apply speech normalization on rhythm, pitch, and energy to create deterministic training targets.
Despite their recent success, current S2ST models still suffer from distinct degradation in noisy scenarios and fail to translate visual speech. In this work, we complement the audio stream with the visual information, opening up a host of practical applications (improving speech translation in noisy environments, enabling silent dictation, or dubbing
silent archival films), which has been relatively overlooked.

heading: 2.2 Audio-Visual Speech Self-Supervised Learning
text: It has been an increasing interest in self-supervised learning in the machine learning and speechprocessing community. Wav2vec 2.0 (Baevski et al., 2020) trains a convolutional neural network to distinguish true future samples from random distractor samples using a contrastive predictive coding (CPC) loss function. HuBERT (Hsu et al., 2021) is trained with a masked prediction with masked continuous audio signals. For audiovisual representation learning, they rely on spatiotemporal CNNs consisting of multiple 3D convolutional layers or a single 3D convolutional layer followed by 2D ones. Stafylakis and Tzimiropoulos (2017) propose a residual network with 3D convolutions to extract more powerful representations.
Very recently, Shi et al. (2022a) propose AVHuBERT with a self-supervised representation learning framework by masking multi-stream video input and predicting automatically discovered multimodal hidden units. It has been demonstrated to learn discriminative audio-visual speech representation, and thus we leverage the contextual representations to enhance the AV-S2ST performance.

heading: 2.3 Transfer Learning
text: Transferring knowledge across domains is a promising machine learning methodology for solving the data shortage problem. Zhang et al. (2021) perform transfer learning from a text-to-speech system to voice conversion with non-parallel training data. Afouras et al. (2020) apply cross-modal distillation from ASR for learning audio-visual speech recognition, where they train strong models for visual speech recognition without requiring human annotated ground-truth data. Cai et al. (2020) enhance the knowledge transfer from the speaker verification to the speech synthesis by engaging the speaker verification network. Popuri et al. (2022) perform transfer learning from a natural language expert mBART for faster coverage of training speech translation models.
Our approach leverages networks trained on one modality to transfer knowledge to another. In this way, the dependence on a large number of parallel audio-visual data can be reduced for constructing AV-S2ST systems.

heading: 3 AV-TranSpeech
text: In this section, we first overview the encoderdecoder architecture for AV-TranSpeech, following which we introduce the cross-modal distillation procedure for few-shot transfer learning with lowresource data. The overall architecture has been illustrated in Figure 1, and we put more details on the encoder and decoder block in Appendix A.

heading: 3.1 Overview
text: The overall AV-S2ST pipeline has been illustrated in Figure 1, where we 1) use the SSL HuBERT (Hsu et al., 2021) to derive discrete units of target speech; 2) build the audio-visual speechto-unit translation (AV-S2UT) and 3) apply a separately trained unit-based vocoder to convert the translated units into waveform.
For audio-visual speech-to-unit translation, we adopt the encoder-decoder sequence-to-sequence model as the backbone. The audio-visual speech samples first pass through the multi-layer audio and video feature extractors, which are then fused and fed into the backbone conformer encoder. In the following, the unit decoder autoregressively predicts unit sequences corresponding to the target speech.
Training direct textless AV-S2ST models typically requires a large amount of parallel training data (Duquenne et al., 2022; Lee et al., 2021b), while resources providing parallel multimodal data could be limited due to the heavy workload. To alleviate the issue of data scarcity, we 1) build upon the recently introduced Audio-Visual HuBERT (AV-HuBERT) which learns contextual representations through self-supervised masked prediction, and show that large-scale pre-training benefits AVS2ST training; 2) introduce the cross-modal distillation with S2ST models trained on the audio-only corpus, which further reduces the requirements of visual data and boosts the performance of visual systems in low-resource scenarios (10h, 30h).

heading: 3.2 Pre-Trained Encoder
text: Audio-Visual Hidden Unit BERT (AV-HuBERT) is a self-supervised model that learns from unlabeled audio-visual speech data. AV-HuBERT comprises four modules: a feed-forward network (FFN) audio feature extractor, a modified ResNet (Stafylakis and Tzimiropoulos, 2017; Martinez et al., 2020) video feature extractor, a fusion module, and a conformer (Gulati et al., 2020) backend.
Denote the domain of visual and audio samples by V,A ⊂ R respectively. The source language is therefore a sequence of visual V = {v1, . . . , vT } and speech A = {a1, . . . , aT } samples for T timesteps. The multi-layer audio feature extractor fa and the video feature extractor fv respectively take audio A and visual frames V as input, which are then fused (i.e., element-wise addition) and fed into the backbone conformer encoder and generates contextual representations X = {x1, . . . , xT }.

heading: 3.3 Unit Decoder
text: The autoregressive decoder is assisted with an attention module, which takes the encoder output as the source values for the attention, and predicts unit sequences corresponding to the target translated speech. We use a stack of transformer layers as the decoder, along with a multihead attention (Vaswani et al., 2017). Given the T -frame contextual representations from source speech X = {x1, . . . , xT }, autoregressive model θ factors the distribution over possible outputs Y = {y1, . . . , yN} by:
p(Y | X; θ) = N+1∏
i=1
p(yi | y0:i−1, x1:T ; θ), (1)
where the special tokens y0(⟨bos⟩) and yN+1(⟨eos⟩) are used to represent the beginning and end of all target units.

heading: 3.4 Cross-Modal Distillation
text: In this part, we investigate the transfer learning from orders of magnitude audio data to boost the performance of visual systems. Specifically, we leverage the S2ST model trained on a large-scale audio-only corpus and perform cross-modal distillation with low-resource audio-visual data.
S2ST Model. We adopt the current state-ofthe-art S2ST model (Popuri et al., 2022) with pretrained wav2vec 2.0 (Baevski et al., 2020) and mBART (Liu et al., 2020). Wav2vec 2.0 is a selfsupervised framework to learn speech representations from unlabeled audio data, which is trained via contrastive loss with masked spans on the input to the context encoder. mBART has been originally proposed for denoising autoencoder over text sequences, which predicts the original text z given its noisy version, g(z), created by random masking. As such, the powerful S2ST model provides a significant initialization for training audio-visual systems.
Modality Adaptor. The encoder in S2ST model fa encodes speech representation with A = {a1, . . . , aT ′} with a stride of about 20ms, while the video stream fv generates visual feature sequence V = {v1, . . . , vT } at a stride of 10 ms from the raw waveform. To alleviate this length mismatch between the audio and visual representations, we add a randomly initialized modality adaptor layer consisting of a single 1-D convolu-
tional layer with stride 2 between the audio and video streams.
Modality Dropout. Since S2ST models provide a strong initialization for our AV-S2ST, and thus it can relate audio input to lexical output more effortlessly than the visual input stream, leading to the domination of audio modality in model decisions. To prevent the model’s over-reliance on the audio stream in our joint model, we include a modality dropout with p = 50% probabilities to mask the full features of one modality before fusing audio and visual inputs, forcing the visual encoder to learn contextual representations. We show feature fusion in our cross-modal distillation with modality dropout:
fav =    fa + fv with p = 0.5 fa + 0 with p = 0.25 0+ fv with p = 0.25
As such, modality dropout (Chern et al., 2022; Zhang et al., 2019) prevents the model from ignoring video input and encourages the model to produce the prediction regardless of what modalities are used as input.

heading: 3.5 Model Training
text: In training AV-TranSpeech, we compute the crossentropy loss (denoted as "CE") between generated and reference units. For low-resource scenarios, we group our distillation strategies into two categories: 1) for AV-S2ST, we adapt the speech encoder and the unit decoder in S2ST to the visual system; 2) for V2ST with visual-only input, we only transfer the knowledge from unit decoder in S2ST to avoid noisy and incomplete decoding. In this way, the dependence on a large number of parallel audiovisual data can be reduced for constructing visual systems.

heading: 4 Experiments

heading: 4.1 Experimental Setup
text: Following the common practice in the direct unit S2ST pipeline, we apply the publicly-available pretrained multilingual HuBERT (mHuBERT) model and unit-based HiFi-GAN vocoder (Polyak et al., 2021; Kong et al., 2020), leaving them unchanged.

heading: 4.1.1 Dataset
text: To evaluate the performance of the proposed model, we conduct experiments on two language pairs,
including English-Spanish (En-Es), and EnglishFrench (En-Fr).
LRS3-T. We construct our translation dataset by converting the transcribed English text from LRS3 (Afouras et al., 2018b) into target language using cascaded neural machine translation (NMT) and text-to-speech (TTS) systems. We remove short clips (less than 2 seconds) and discard the non-vocal segments with voice activation detection (VAD). To this end, we collect 200-hour parallel audio-visual translation data (with source videos and target speech), namely LRS3-T which we plan to release.
CVSS-C. For training S2ST models, we use the benchmark dataset CVSS-C (Jia et al., 2022b), which is derived from the CoVoST 2 (Wang et al., 2020) speech-to-text translation corpus by synthesizing the translation text into speech using a singlespeaker TTS system.
Noise. For evaluating our AV-S2ST models under different noise categories, we prepare noise audio clips in the categories of “music” and “babble” sampled from MUSAN dataset (Snyder et al., 2015), and create “speech” noise samples following Popuri et al. (2022).
The total duration of each dataset is shown in Table 1.

heading: 4.1.2 Model Configurations and Training
text: For training S2ST models, we adopt Wav2vec 2.0 LARGE pre-trained on Libri-light dataset (Kahn et al., 2020) as audio encoder and unit mBART pretrained on VoxPopuli dataset (Wang et al., 2021) as the decoder. Following the practice in unit-based S2ST (Lee et al., 2021a), we use the k-means algorithm to cluster the representation given by the normalized mHuBERT (Huang et al., 2022d) into a vocabulary of 1000 units as training targets. The inputs to AV-TranSpeech are lip Regions-Of-Interest (ROIs) for the visual stream and 80-dimensional
mel-filterbank features at every 10-ms for the audio stream. As the image frames are sampled at 25Hz, we stack the 4 neighboring acoustic frames to synchronize the two modalities. The encoders in AV-TranSpeech follow AV-HuBERT LARGE configuration with 24 transformer blocks, each with 16 attention heads and 1024/4096 embedding/feedforward dimensions. We remove the auxiliary tasks for simplification and follow the unwritten language scenario (Lee et al., 2021b). AV-TranSpeech is trained until convergence for 20k steps using the Adam optimizer (β1 = 0.9, β2 = 0.98, ϵ = 10−8) with 6 Tesla V100 GPU. A comprehensive table of hyperparameters is available in Appendix A.

heading: 4.1.3 Evaluation
text: We compare AV-TranSpeech with other systems using the publicly-available fairseq framework (Ott et al., 2019), including, 1) NMT+TTS cascaded system to simulate the construction of LRS3-T, where we adopt MMT to convert transcribed English text to target languages (regarded as reference text) and then apply TTS model for speech generation, following we transcribe the speech and compute the BLEU; 2) Unit TTS, where we first synthesize speech samples with target units, and then transcribe the speech and compute BLEU; 3) Moreover, we compare the performance of AVTranSpeech in S2ST with baseline model (Popuri et al., 2022) (denoted as Enhanced S2ST).
For translation accuracy, we use open-sourced ASR models in fairseq framework to transcribe the audios and then calculate the BLEU score (Papineni et al., 2002) between the generated and the reference text. To evaluate the naturalness of the speech output, we conduct crowd-sourced human evaluations with MOS, rated from 1 to 5 and re-
ported with 95% confidence intervals (CI) via Amazon Mechanical Turk. More details on evaluation have been attached in Appendix B.

heading: 4.2 Translation Accuracy and Speech Naturalness
text: Table 2 summarizes the translation accuracy and speech naturalness among all systems, and we have the following observations: 1) Large-scale multimodal pre-training (1 vs. 2) improves performance by a large margin, while the naive model fails to work without the self-supervised pretraining strategy. It is mainly because LRS3-T is a challenging unconstrained dataset with a large proportion of videos collected from TED talks, showing the difficulty (Zhang et al., 2020; Jia et al., 2019) of direct speech-to-speech translation without relying on intermediate texts or auxiliary multitask training. In contrast, with a pre-trained AVHubert encoder and a randomly initialized decoder, AV-TranSpeech is efficient in learning contextual representations from audio-visual signals. 2) Visual modality (2 vs. 4) has brought a gain of 2.0 BLEU points on average. It complements the audio stream with visual information, opening up a host of practical applications: enabling silent dictation or dubbing archival silent films. 3) We further compare AV-TranSpeech in S2ST with the baseline model (4 vs. 7), showing that AV-TranSpeech with audio-only input is on-par with the current state-of-the-art speech model in terms of translation accuracy. 4) For speech quality, AV-TranSpeech produces natural speech regardless of modalities input competitive with the baseline S2ST system. Since we apply the publicly-available pre-trained unit vocoder which mainly controls the naturalness of output speech and leave it unchanged, we ex-
pect AV-TranSpeech exhibits high-quality speech generation as baselines.

heading: 4.3 Visual Modality Evaluation
text: The benefit of incorporating the visual stream is more apparent in challenging scenarios (Afouras et al., 2018a; Popuri et al., 2022), and thus we evaluate our models in the noisy setting to examine the impact of input modality (audio or audio-visual). A noise category with an audio clip has been sam-
pled each time, following which we randomly mix the sampled noise with varied probabilities at five SNR levels: {−10,−5, 0, 5, 10} dB. For easy comparison, the results are presented in Table 3 and visualized in Figure 3, and we have the following observations:
1) AV-S2ST consistently outperforms audio-only S2ST under all settings regardless of the SNR and the type of noise. AV-TranSpeech complements the audio stream with visual information, which is invariant to speaking environments and promotes robustness. 2) As the volume of the noise increase with lower SNR, both languages have presented a degradation in translation accuracy. Informally, AV-S2ST models show a relatively slower BLEU drop (42% drop in SNR-10 babble noise), while a distinct decrease could be witnessed in audioonly S2ST models (99.9% drop in SNR-10 babble noise).

heading: 4.4 Low Resource Evaluation
text: Training direct AV-S2ST models without relying on intermediate text typically requires a large amount of parallel visual speech (i.e., lip) training data, while there may be very few resources due to the heavy workload. In this section, we prepare low-
resource audio-visual data (LRS3-T 10h, 30h) and leverage large-scale audio-only data (Covost) to boost the performance of visual systems (AV-S2ST, V2ST), to investigate the effectiveness of our crossmodal distillation. The results are compiled and presented in Table 4, and we have the following observations:
1) In consistent with previous practice (Duquenne et al., 2022; Tjandra et al., 2019), training speech models are faced with the significant issue of data scarcity. As training data is reduced in the low-resource scenario, a distinct degradation in translation accuracy could be witnessed in both modalities (AV-S2ST or V2ST). 2) Leveraging orders of magnitude audio-only data with cross-modal distillation, the visual systems achieve BLEU scores of 21.5 and 22.2 respectively in En-Es and En-Fr AV-S2ST, showing a significant promotion regardless of the modalities and languages. In this way, the dependence on a large number of parallel audio-visual data can be reduced for constructing visual systems.

heading: 4.5 Case Study
text: We present several translation examples sampled from the En-Fr language pair in Table 5, and have the following findings: 1) With the complemental visual information brought in, the results produced by AV-TranSpeech are noticeably more literal. 2) Moreover in challenging noisy scenarios, S2ST models suffer severely from the issue of noisy and incomplete translation, which is largely alleviated
in AV-S2ST. AV-S2ST consistently outperforms audio-only S2ST in a noisy environment.

heading: 5 Conclusion
text: In this work, we proposed AV-TranSpeech, the first audio-visual speech-to-speech translation (AVS2ST) model without relying on intermediate text. AV-TranSpeech complemented the audio stream with the visual information to promote robustness in noisy environments, opening up a host of practical applications: silent dictation or dubbing archival films. To mitigate the data scarcity for training AV-S2ST models, we 1) built upon the AV-HuBERT with a self-supervised learning framework fir contextual representations, showing that large-scale pre-training benefited the AV-S2ST training; 2) leveraged cross-modal distillation with S2ST models trained on the audio-only corpus, which further reduced the visual data requirements and boosted performance in low-resource scenarios. Experimental results on two language pairs demonstrated that AV-TranSpeech achieved significant robustness in noisy environments, outperforming audio-only S2ST models under all settings regardless of the type of noise. With low-resource audio-visual data (10h, 30h), cross-modal distillation yielded an improvement of 7.6 BLEU on average compared with baselines. We envisage that our work will serve as a basis for future audiovisual speech-to-speech translation studies, unlocking the ability for high-quality translation given a user-defined modality input.

heading: 6 Limitation and Potential Risks
text: As mentioned in our experimental setup, we provide results of AV-S2ST in LRS3-T with synthesized target speech, similar to the pioneer literature (Jia et al., 2022b) in S2ST. One of our future directions is to develop a better benchmark dataset (e.g., mined or human-annotated data) to improve translation performance.
As mentioned in our results analysis, the BLEU scores heavily depend on the ASR quality, which may not accurately reflect the speech translation performance. Future directions could be improving ASR quality or exploring other evaluation metrics without reliance on ASR models.
AV-TranSpeech lowers the requirements for audio-visual speech-to-speech translation, which may cause unemployment for people with related occupations such as interpreter and translator. In addition, there is the potential for harm from nonconsensual voice generation or fake media. The voices of the speakers in the recordings might be overused than they expect.

heading: Acknowledgements
text: This work was supported in part by the National Key R&D Program of China under Grant No.2022ZD0162000, National Natural Science Foundation of China under Grant No.62222211, Grant No.61836002 and Grant No.62072397.

heading: A Model Architectures
text: In this section, we list the model hyper-parameters of AV-TranSpeech in Table 6.

heading: B Subjective Evaluation
text: Following (Huang et al., 2021, 2022a), all our Mean Opinion Score (MOS) tests are crowdsourced and conducted by native speakers. The scoring criteria have been included in Table 7 for completeness. The samples are presented and rated one at a time by the testers, each tester is asked to evaluate the subjective naturalness of a sentence on a 1-5 Likert scale. The screenshots of instructions for testers are shown in Figure 4. We paid $8 to participants hourly and totally spent about $500 on participant compensation.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: See Section 6.

heading: 3 A2. Did you discuss any potential risks of your work?
text: See Section 6.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: See Section 1.
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 7 Did you use or create scientific artifacts? Left blank.
B1. Did you cite the creators of artifacts you used? No response.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? No response.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? No response.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? No response.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? No response.
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. No response.
C 3 Did you run computational experiments? See Section 4.

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? See Section 4.1.2 and Appendix A
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? See Section 4.1 and Appendix A.

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? See Section 4.1.2. We report BLEU on average and MOS with 95% confidence intervals (CI).

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? See Section 4.1
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? See Section 4.1.3 and Appendix B.

heading: 3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots,
text: disclaimers of any risks to participants or annotators, etc.? See Section 4.1.3 and Appendix B.

heading: 3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students)
text: and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? See Section 4.1.3 and Appendix B.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Not applicable. Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Not applicable. Left blank.

heading: Future Work
text: One of our future directions is to develop a better benchmark dataset (e. g. , mined or human-annotated data) to improve translation performance. As mentioned in our results analysis, the BLEU scores heavily depend on the ASR quality, which may not accurately reflect the speech translation performance. Future directions could be improving ASR quality or exploring other evaluation metrics without reliance on ASR models. AV-TranSpeech lowers the requirements for audio-visual speech-to-speech translation, which may cause unemployment for people with related occupations such as interpreter and translator. In addition, there is the potential for harm from nonconsensual voice generation or fake media. The voices of the speakers in the recordings might be overused than they expect