BLASER: A Text-Free Speech-to-Speech Translation Evaluation Metric
Year: 2023

Abstract:
End-to-End speech-to-speech translation (S2ST) is generally evaluated with text-based metrics. This means that generated speech has to be automatically transcribed, making the evaluation dependent on the availability and quality of automatic speech recognition (ASR) systems. In this paper, we propose a text-free evaluation metric for end-to-end S2ST, named BLASER, to avoid the dependency on ASR systems. BLASER leverages a multilingual multimodal encoder to directly encode the speech segments for source input, translation output and reference into a shared embedding space and computes a score of the translation quality that can be used as a proxy to human evaluation. To evaluate our approach, we construct training and evaluation sets from more than 40k human annotations covering seven language directions. The best results of BLASER are achieved by training with supervision from human rating scores. We show that when evaluated at the sentence level, BLASER correlates significantly better with human judgment compared to ASRdependent metrics including ASR-SENTBLEU in all translation directions and ASR-COMET in five of them. Our analysis shows combining speech and text as inputs to BLASER does not increase the correlation with human scores, but best correlations are achieved when using speech, which motivates the goal of our research. Moreover, we show that using ASR for references is detrimental for text-based metrics. 1

heading: 1 Introduction
text: Speech-to-Speech translation seeks to translate speech segments from one language into another.
∗ Equal Research Leadership Contribution 1Code is available at https://github.com/faceb
ookresearch/stopes
Historically, it has been implemented and evaluated as a concatenation of three systems: automatic speech recognition (ASR), machine translation (MT) and text-to-speech (TTS) (Lavie et al., 1997; Lazzari, 2006). In recent years, there has been increasing interest in end-to-end approaches (Jia et al., 2019; Lee et al., 2022a). While end-toend S2ST is becoming popular, researchers still rely on text-based metrics to evaluate model performance by automatically transcribing the generated speech segments (Jia et al., 2019). These cascaded metrics rely on ASR systems, which for a given language may not have enough quality or may not even be available (Javed et al., 2022). They are also inappropriate for languages lacking standardized writing systems (Salesky et al., 2021a), like Hokkien or Algerian Arabic.
In this work, we propose the text-free metric BLASER for S2ST evaluation, sidestepping the dependency on ASR systems. In particular, we use LASER encoders that support multiple languages and modalities including text (Heffernan et al., 2022) and speech (Duquenne et al., 2021). We use the LASER encoders to directly embed speech segments into vectors and compute a score estimating the quality of generation. We then construct training and evaluation datasets from more than 40k human annotations, covering seven language directions (Spanish↔English, French↔English, Russian→English, Hokkien→English, and English→German). We evaluate BLASER on these datasets on the popular benchmark of MusT-C (Di Gangi et al., 2019). We also benchmark several strong ASR-based metrics, e.g., ASR-SENTBLEU (i.e., sentence-level ASR-BLEU (Jia et al., 2019)) and ASR-COMET (i.e., applying COMET (Rei et al., 2020) on ASR outputs). There is a recent interest of supervised evaluation metrics that are trained on human quality scores (Rei et al., 2020). However, these human quality scores are precious and somehow limited or nonexistent, specially for
9064
low-resource languages. Therefore, we propose both an unsupervised and a supervised version of BLASER. The results show that on average both unsupervised and supervised BLASER outperform their corresponding baseline metrics. In particular, BLASER outperforms ASR-COMET significantly in five language directions and obtains comparable results in two other language directions. Our analysis reveals that, while BLASER can use both text and speech, encoding speech data give the most significant benefits. In addition, we show that replacing human-written source input and human-written reference with ASR-generated ones hurts performance of text-based metrics, which motivates the use of modality-agnostic metrics as BLASER.

heading: 2 Related Work
text: S2ST Evaluation. Early approaches for automatic S2ST evaluation use metrics consisting of three modules where each module is used to evaluate individual component in the cascaded S2ST pipeline: e.g., BLEU and Translation Edit Rate (Snover et al., 2006) for NMT, Word Error Rate for ASR, and Mel-Cepstral Distortion (Kominek et al., 2008) for TTS. Recent approaches have been primarily focused on adapting text-based metrics for end-to-end S2ST (Jia et al., 2019; Lee et al., 2022a). In contrast to these works, we propose a text-free metric.
MT Metrics. There is a huge amount of literature in automatic machine translation evaluation in the area of natural language processing (Papineni et al., 2002; Denkowski and Lavie, 2014; Popović, 2015, inter alia). Recent methods have approached this goal by using human ratings for training model-based metrics, such as COMET, BERTSCORE (Zhang* et al., 2020) and BLEURT (Sellam et al., 2020). These metrics have achieved remarkable performance on text (Freitag et al., 2021; Kocmi et al., 2021).
Speech Metrics. Our work involves computing semantic similarity of speech segments to evaluate translation quality. It is thus related to referencebased automatic evaluation metrics for TTS where the metrics seek to measure the quality of generated speech segments given reference speech segments e.g., Mel-Cepstral Distortion, Gross Pitch Error (Nakatani et al., 2008) and other model-based metrics (Bińkowski et al., 2020). Unlike our work,
these metrics primarily focus on the naturalness of synthesized speech.
Contemporaneous to this work, Besacier et al. (2022) propose a text-free metric for comparing two speech segments in the same language. Their work limits to comparing English speech data and they do not cover multilingual S2ST evaluation. Their work is based on synthetic datasets where ratings are generated by automatic text-based measures as opposed to human annotators. Differently, we cover S2ST evaluation and we show how our metric correlates with human annotations and how it improves over text-based metrics.
Speech and/or Text Representations. There is a large body of research on learning multilingual text embeddings for various downstream tasks. LabSE (Feng et al., 2022), SentenceBERT (Reimers and Gurevych, 2019), mUSE (Yang et al., 2020) and LASER (Artetxe and Schwenk, 2019; Heffernan et al., 2022) are popular encoders that capture the semantic information of a sentence into fixed size vector representations. In the speech modality, approaches such as wav2vec 2.0 (Baevski et al., 2020a) or Hubert (Hsu et al., 2021) allow learning embeddings at acoustic-frame level.
There has recently been increased interest in aligned speech-text representations such as mSLAM (Bapna et al., 2022), MAESTRO (Chen et al., 2022b), SAMU-XLSR (Khurana et al., 2022), and LASER (Duquenne et al., 2022). While our approach could accommodate any speech representation architecture given the right pooling strategy, we chose LASER in this work for three reasons. (1) The encoders modules are freely-available; (2) the LASER embedding space can easily be extended to new languages at a minimal cost: contrary to most multilingual encoders, the teacher-student approach does not require the whole embedding space to be retrained after including data for the new language. This makes BLASER virtually usable for any language in the future (3) the embedding space could potentially be extended to any new modality meaningful to translation use cases.

heading: 3 Approach
text: The underlying idea of our approach is to leverage the similarity between speech segments without requiring intermediate textual representations. Compared to ASR-based metrics, the advantage of BLASER is that it is text-free. In particular, given the source input speech, the translated out-
put speech of a S2ST model, and the reference speech segment, respectively, we embed them into vectors hsrc, hmt, and href. These embeddings are combined and BLASER predicts a score for each translation output, where higher scores suggest better translation quality.2
The effectiveness of BLASER depends on the quality of vector representations encoded from speech segments: it requires rich semantic information to be encoded in the speech embeddings. In this work, we use LASER speech encoders (Duquenne et al., 2022), which we describe below. We note that our approach is generic and can be extended to other encoders.
We study BLASER under the unsupervised and the supervised settings, which allows it to exploit the information of human ratings, if available.

heading: 3.1 Background: LASER Encoders
text: The LASER encoder was initially trained in a sequence-to-sequence model (Schwenk and Douze, 2017) and supported 93 languages in its follow-up publications (Artetxe and Schwenk, 2019). In recent work, a teacher-student approach was applied to incorporate more languages (Heffernan et al., 2022) and to extend the model to the speech modality (Duquenne et al., 2021). All these encoders use the same teacher model and are mutually compatible. The embeddings are of dimension 1024. The reader is referred to these papers for a detailed description. These LASER encoders were successfully applied to automatically mine semantically similar sentences, in the text (NLLB Team et al., 2022) and speech domain (Duquenne et al., 2022).

heading: 3.2 Unsupervised BLASER
text: In the unsupervised setting, we directly compute the cosine similarities between hsrc and hmt, and href and hmt. Formally, this metric is defined as follows:
BLASERu = cos(hsrc, hmt) + cos(href, hmt)
2 (1)
where cos(·, ·) is the cosine similarity function.

heading: 3.3 Supervised BLASER
text: Previous work has shown that evaluation metrics (e.g. (Rei et al., 2021)) can take advantage of human ratings for training. We follow COMET (Rei
2A straightforward corpus-level score could be obtained via averaging over sentence-level scores, which can be used to compare different S2ST models, similar to metrics like BLEU.
et al., 2020) and RUSE (Shimanaka et al., 2018) and use the following features:
• Element-wise source product: hsrc ⊙ hmt
• Element-wise reference product: href ⊙ hmt
• Absolute element-wise source difference: |hsrc − hmt|
• Absolute element-wise reference difference: |href − hmt|
We concatenate these features with the embeddings of references href and translation outputs hmt and then use it as input for a neural regressor to predict a scalar indicating the quality of the translated speech, as shown in Figure 1. This metric corresponds to the following equation:
BLASERs = nnet([href; hmt; hsrc ⊙ hmt; |hsrc − hmt|; href ⊙ hmt; |href − hmt|])
where nnet(·) is a two-layer neural network and [·; ·] represents the concatenation of vectors. We note that the dimension of concatenated input vectors to the neural regressor is 6144. The entire model except the LASER encoders (which are kept
frozen) is trained by minimizing the Mean Squared Error between the BLASERs predicted scores and human ratings. We choose to freeze LASER encoders because (1) we do not want to break the aligned embedding space; and (2) it allows us to extend to unseen languages more easily.

heading: 4 Experimental Framework
text: To show that BLASER is useful both in its unsupervised and supervised form, we compare it to several baseline metrics. In this section, we describe the experimental framework for doing this comparison, including the evaluation data, the training and implementation of both baseline and proposed metrics and their evaluation.

heading: 4.1 Data
text: We create training and evaluation data from MusT-C (Di Gangi et al., 2019), Multilingual TEDx (Salesky et al., 2021b), and TAT corpus (Liao et al., 2020). Given a source input from these datasets, we generate translated outputs using various S2ST models. We then conduct human evaluations to collect human ratings for generated speech segments. As the datasets do not have reference speech segments but provide human-written transcripts, we use TTS to synthesize speech data from these transcripts to facilitate fair comparison between our metrics and other reference-based textual metrics. While the use of synthesized audios is disadvantageous to BLASER,3 current benchmarks still use human-written transcripts because of the current dependence on the text-based metrics. We expect that, in the future, S2ST benchmarks will rely on speech references and TTS will not be
3For example, examples 2 and 3 in table 6 do not correctly synthesize SMS or PKW.
needed. In this case, BLASER will have additional advantage over text-based metrics that will have to apply ASR to references in addition to ASR to system outputs.
Each data instance in our dataset consists of a source input, a translation output, a reference, and a human evaluation score, where the source, translation output, and reference have both speech and text. Figure 2 summarizes the data sources of these components. As follows we describe the details of each data sources.
Human Annotations. We do not use crowd workers as human annotators and instead we use a vendor-managed pool of well-trained and qualified bilingual annotators who pass a qualification test for their language skills. Human annotators are instructed to rate semantic similarities between source input and generated speech segments4 on a 5-point Likert scale, where higher values are better, following annotation guidelines similar to Licht et al. (2022). More details on human evaluations are in Appendix D. Each model generation has 1~18 human ratings, leading to 4k~20k annotations per language direction. We take medians of rating scores when there are more than one score associated with a particular model generation following NLLB Team et al. (2022) and Licht et al. (2022).
Speech To Speech Translation. We evaluate the translation outputs generated with the following S2ST architectures:
1. Cascaded two-stage models with speech-totext translation and TTS. This system includes Spanish-English, English-French and Russianto-English translation directions;
2. The model presented in Lee et al. (2022b), which represents target speech as discrete units and uses a speech-to-unit translation model to convert source speech to target units followed by a code HiFi-GAN vocoder (Park and Mulc, 2019; Polyak et al., 2021) to convert units to waveform. This system includes English-Spanish and Russian-to-English translation directions;
3. The model presented in Inaguma et al. (2022), which is similar to Lee et al. (2022b) except that it is a two-pass direct S2ST architecture
4We note that the generated speech segments could be reference speech segments coming from the TTS models or translated speech segments coming from the S2ST models.
that first generates textual representations and predicts discrete acoustic units subsequently. This system includes the Spanish-to-English translation direction;
4. The model presented in Wang et al. (2022), which employs mBART (Liu et al., 2020) for unsupervised machine translation in their unsupervised cascaded speech-to-text translation pipeline. This system includes the Spanish-toEnglish translation direction.
5. The Hokkien-to-English S2ST system is threestage cascaded: a concatenation of Hokkien to Chinese speech-to-text translation + Chinese to English machine translation + English TTS (English text-to-unit + unit vocoder from Lee et al. (2022b)).
6. The English-to-German S2ST system is the MLLP-VRAIN system (Iranzo-Sánchez et al., 2022) from IWSLT 2022 (Anastasopoulos et al., 2022), which is a cascaded system of separate ASR, MT, and TTS models.
Automatic Speech Recognition. For ASR, we use the open-sourced implementation in FAIRSEQ (Ott et al., 2019),5 that provides strong models built on top of the unsupervised pretrained wav2vec (Schneider et al., 2019) or XLSR (Conneau et al., 2020a) models. In particular, for English and Russian, we use wav2vec 2.0 large (Baevski et al., 2020b) finetuned with CTC loss (Graves et al., 2006). For Hokkien, Spanish, French, and German, we use the ASR models released in Chen et al. (2022a), Grosman (2021b), Grosman (2021a), and Grosman (2022), respectively.
5https://github.com/facebookresearch/ fairseq/blob/ust/examples/speech_to_spee ch/asr_bleu
Text to Speech. For TTS, we use the toolkit released by Wang et al. (2021a), which provides a set of recent state-of-the-art speech synthesis models.
The language directions in the final dataset are Spanish-English and French-English in both directions (i.e., en→es, es→en, en→fr, and fr→en), Russian to English (ru→en), Hokkien to English (hk→en) and English to German (en→de). We split the data into training and test sets when there is enough data available (i.e., at least one thousand data instances for a language direction). We also make sure that there is no overlapping source inputs between train and test sets. Table 1 summarizes the dataset statistics.

heading: 4.2 Baseline Metrics
text: We consider a variety of baseline metrics, including BLEU and CHRF+ (Popović, 2017), which are standard metrics to evaluate textual similarities. While BLEU is by nature corpus-level, here we use the sentence-level version due to the insufficient amount of human annotations. To differentiate these two versions, we denote the sentence-level BLEU as SENTBLEU. We also benchmark BERTSCORE (Zhang* et al., 2020) and COMET, which are popular modelbased metrics that correlate well with human judgments on textual data (Kocmi et al., 2021).6 We extend these metrics to speech data by using ASR systems to transcribe the machine-translated speech segments. We prepend “ASR-” to the beginning of the names of these metrics to indicate the use of ASR systems. Table 2 summarizes the differences among the metrics.
Specifically, we use BLEU7 and CHRF+8 as im6Multilingual BLEURT (Pu et al., 2021) reports similar performance as COMET on WMT metrics tasks and therefore we decided to only include COMET in our experiments.
7SacreBLEU signature: nrefs:1|case:mixed|eff:yes|tok:13a|smooth:exp|version:2.2.0
8SacreBLEU signature:
plemented in SacreBLEU (Post, 2018).9 We normalize the reference text before computing ASRSENTBLEU and ASR-CHRF+ to match the lowercased and punctuationless ASR output. We use the official implementations for BERTSCORE10 and COMET.11 To form competitive baselines, we also train COMET from scratch on our training data (COMETretrain) and the concatenation of our training data and the direct assessments from WMT 15-19 metrics tasks (Stanojević et al., 2015; Bojar et al., 2016, 2017; Ma et al., 2018, 2019) (COMETretrain with WMT).

heading: 4.3 Training and Evaluation
text: LASER Encoders. We use the speech LASER encoders released in Duquenne et al. (2022) except for English and Hokkien.12. For Hokkien speech LASER encoder, we followed the training procedure presented in (Chen et al., 2022a) using the same pretrained model and training data. For the English speech LASER encoder, we fine-tuned XLSR 2B (Babu et al., 2021) on several ASR datasets including CoVoST2 (Wang et al., 2021c), Common Voice (Ardila et al., 2020), EuroparlST (IranzoSánchez et al., 2020), MusT-C (Di Gangi et al., 2019), Voxpopuli (Wang et al., 2021b) and Librispeech (Panayotov et al., 2015).
Training Setup and Hyperparameters. For BLASERs, the regressor has two hidden layers of sizes 3072 and 1536, similar to COMET. We keep
nrefs:1|case:mixed|eff:yes|nc:6|nw:2|space:no|version:2.2.0 9https://github.com/mjpost/sacrebleu
10We use language-specific configurations recommended in https://github.com/Tiiiger/bert_score
11We use the “wmt20-comet-da” model from https:// github.com/Unbabel/COMET
12https://github.com/facebookresearch/ fairseq/tree/ust/examples/speech_matrix
the LASER encoders fixed during training. We use a learning rate of 5× 10−5 and employ learning rate annealing with a linear schedule. When training COMET, we follow the official implementation and fine-tune the entire model from the XLM-R-LARGE model checkpoint (Conneau et al., 2020b). For both BLASERs and COMET, we train them for 20 epochs. We standardize the human ratings in our training set by subtracting them with a mean and a variance computed based on the entire training set.
Computational Cost. We trained BLASERs using 1 Quadro GV100 and the training takes less than one hour. We used 4 Tesla V100 to train COMET and the training takes more than two days.
Evaluation. We compute Pearson’s correlation at the sentence level between the automatic and human rating scores. Given that our test sets are relatively small, we perform statistical significance test using the bootstrap method from Koehn (2004).13

heading: 5 Experimental Results and Analysis
text: In this section we report the main results of our proposed metric BLASER, on two different settings (unsupervised and supervised) and we compare it to widely used baseline text-based metrics. Additionally, we report an analysis at various levels, including the impact of evaluating using different modalities and a qualitative inspection of several examples to observe scores of various metrics for particular examples.

heading: 5.1 Main Results
text: We report unsupervised and supervised results in Table 3. We note that results that fail to pass the significance test are neither better nor worse significantly than the corresponding baseline.
Generally, model-based metrics perform significantly better than string-based ones. Among the unsupervised metrics, BLASERu performance improves significantly over ASR-SENTBLEU and ASR-CHRF+ for all language directions except for en→es, showing the capabilities of BLASER in capturing semantic information even when human annotations are absent.
Among the supervised metrics, we see that BLASERs almost always performs better than the official ASR-BERTSCORE and ASR-COMET.
13https://github.com/neubig/util-scrip ts/blob/master/paired-bootstrap.py
When compared to the stronger baseline ASRCOMETretrained with WMT, BLASERs is better than the baseline significantly in four language directions and they are comparable in the other three directions.
We also find that BLASER can generalize training signal to languages where there is no training data available. Specifically, if we compare BLASERs to BLASERu, we see that BLASERs always improves over the unsupervised version. Also, for the language directions where there is no training data (i.e., hk→en, fr→en, en→fr), BLASERs still beats BLASERu. Additionally, we observe that hk→en and ru→en are two of the language directions for which BLASERs shows significant improvements over ASR-COMET, confirming the zero-shot capabilities of our proposed methods in comparison to existing metrics.

heading: 5.2 Analysis
text: Impact of Human-Written vs ASRtranscriptions. To investigate the impact of using transcripts generated by ASR systems rather than human-written inputs and references, we replace the human-written source input and reference with the ones generated by ASR systems.
We note that in this case, all the transcripts are obtained via ASR systems, simulating an evaluation setting where only audio data is available. We show the results in Table 4 where we find that the human-written transcripts are less helpful on those to-English language directions than the from-English ones. We hypothesize that this is in part due to the quality of ASR systems as these ASR-based metrics depend more on references than source inputs and English ASR systems tend to be of better quality than the non-English ones (Khare et al., 2021).
Impact of Using Source and Reference. We investigate the impact of using source and reference speech segments when computing BLASER scores. We evaluate this impact on BLASERu by reporting the performance of individual terms in Equation 1. See the results in Table 5. In general, we find the source input generates better correlations with human ratings than reference. Combining the two leads to the best performance.
Qualitative Analysis. To get a sense of the qualitative differences between BLASER and text-based scores, and better understand what kind of nuances are captured, we manually inspect sample sen-
tences. A selection is presented in Table 6. In each of these examples, the text and generated audio perfectly match, discarding any influence potentially introduced by the ASR model. In cases where the output vocabulary does not perfectly match the reference but is still valid, BLASER seems able to capture the semantics and produce a meaningful score. In the first example, ASR-SENTBLEU is very much impacted by the vocabulary mismatch, while BLASER and ASR-COMET yield high scores, in line with human evaluation. BLASER also seem to detect clear mistranslations better than either of ASR-COMET or ASR-SENTBLEU. In the second example, the end of the output sentence makes little sense. Only BLASER accounts for this properly and produces a score aligned with human judgment. In the third example, ASR-COMET returns a high score despite the mistranslated verb which heavily changes the meaning of the sentence.

heading: 6 Conclusion and Future Work
text: We have introduced BLASER, a text-free metric to evaluate speech-to-speech translation, which avoids the dependency on ASR models required by popular text-based metrics currently used in S2ST.
We explored BLASER in both unsupervised and supervised settings. Experimental results in seven language directions show that BLASER outperforms or is comparable to strong text-based metrics in terms of correlation with human scores at the sentencelevel. Moreover, our metric is effective in zero-shot scenarios.
As for future work, we want to explore the use of speech references generated by humans and the impact of synthesized references. We also want to evaluate BLASER at the system-level with a much larger number of S2ST systems, and explore different approaches to aggregate the sentence-level scores from BLASER and we want to explore different speech and text representations as alternative to LASER.
Limitations
We are evaluating S2ST in an artificial setting given that we have to synthesize the text references. In fact, since there was no metric capable of evaluating the quality in speech, there was no motivation to build such benchmarks either (the chicken-and-egg problem). However, we expect that next benchmarks for the task will have speech references be-
cause of the rise of end-to-end S2ST systems and their quality increase. BLASER paves the way so that we can take advantage of such benchmarks when they appear.
Our metric works at the sentence-level, by embedding the entire sentence into an intermediate space. We ignore how sensitive BLASER is to the length of the sentence, which is a key aspect when we want to extend to the corpus-level metric in the future. Moreover, we are aware that sometimes sentence embedding do not discriminate different numbers or words that belong to the same word family, which may disregard impactful errors such as the change of a number in the translation output.
Ethical considerations
Translation quality scores were provided by bilingual raters as mentioned in Section 4. They were all paid a fair rate. We can not open-source the data form our experiments given that our sources are shared under no-derivative license. Small human evaluation detailed in appendix D was done by volunteers.

heading: A Cross-Modal Data Analysis
text: Considering that LASER can conveniently encode text and speech data into a shared embedding space, we conduct experiments involving both text and
speech data with the text encoders from Heffernan et al. (2022) for BLASERs. In particular, we embed the source input, translation output, and reference using either the speech or text LASER encoders. That is, a data instance formed by embeddings from speech data will result in four instances in this new setting due to different modality combinations. We then evaluate the models on the speech data in our test set. The results in Table 7 show that combining supervision from different modalities does not help improve model performance. It is likely because the embedding space is shared between text and speech and therefore adding textual embeddings do not provide extra information.

heading: B Cross-Modal Supervision Analysis
text: We also look into the benefits of leveraging speech embeddings by comparing several supervised configurations for BLASERs. We report these results in Table 8 where we experiment with different modality combinations during training and testing. The results show that the best results on average are the ones using speech modality for the source input, translation output, and reference. Interestingly, every time that we replace speech with text in the modality combinations, we see performance drops. We find that replacing reference speech segment with text leads to the slightest performance drop, which is likely due to the fact that they are synthesized and thus do not provide extra information than text. We also find that replacing speech data with text for the source input and translation output makes BLASERs similar or even worse than ASRCOMETretrained with WMT, confirming the benefits of using speech data for evaluation S2ST systems.

heading: C Cross-Modal Evaluation Analysis
text: We additionally evaluate BLASERs on different modality combinations when training on speech data only. See the results in Table 9. We find that training on speech data only still allows BLASER to obtain similar performance when replacing the reference speech segments with text.

heading: D Human Evaluation
text: We provide instructions for human evaluations in Table 10.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: section of its own name
A2. Did you discuss any potential risks of your work? Not applicable. Left blank.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: in abstract and introduction
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? section 3 and 4

heading: 3 B1. Did you cite the creators of artifacts you used?
text: section 3 and 4
7 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? we are going to share our code and license details after anonymity period

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? section 4 and 5
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? we are relying on an external dataset, we refer to the sources for those details

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? section 4 reports coverage of domains and languages
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. section 4
C 3 Did you run computational experiments? section 4

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? section 4
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? section 4

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? section 4 and 5

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? section 4
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? section 5 appendix B

heading: 3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots,
text: disclaimers of any risks to participants or annotators, etc.? appendix B

heading: 3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students)
text: and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? section 5

heading: 3 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to
text: crowdworkers explain how the data would be used? ethics section
7 D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? it is quite standard protocol in the community
7 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? for the small annotation in appendix B, we relied on volunteers that do not necessarily want to share this info

heading: Future Work
text: As for future work, we want to explore the use of speech references generated by humans and the impact of synthesized references. We also want to evaluate BLASER at the system-level with a much larger number of S2ST systems, and explore different approaches to aggregate the sentence-level scores from BLASER and we want to explore different speech and text representations as alternative to LASER. Limitations
We are evaluating S2ST in an artificial setting given that we have to synthesize the text references. In fact, since there was no metric capable of evaluating the quality in speech, there was no motivation to build such benchmarks either (the chicken-and-egg problem). However, we expect that next benchmarks for the task will have speech references be-
cause of the rise of end-to-end S2ST systems and their quality increase. BLASER paves the way so that we can take advantage of such benchmarks when they appear. Our metric works at the sentence-level, by embedding the entire sentence into an intermediate space. We ignore how sensitive BLASER is to the length of the sentence, which is a key aspect when we want to extend to the corpus-level metric in the future. Moreover, we are aware that sometimes sentence embedding do not discriminate different numbers or words that belong to the same word family, which may disregard impactful errors such as the change of a number in the translation output. Ethical considerations
Translation quality scores were provided by bilingual raters as mentioned in Section 4. They were all paid a fair rate. We can not open-source the data form our experiments given that our sources are shared under no-derivative license. Small human evaluation detailed in appendix D was done by volunteers