FINDINGS OF THE IWSLT 2024 EVALUATION CAMPAIGN (2024)

Abstract: This paper reports on the shared tasks organized by the 21st IWSLT Conference. The shared tasks address 7 scientific challenges in spoken language translation: simultaneous and offline translation, automatic subtitling and dubbing, speech-to-speech translation, dialect and low-resource speech translation, and Indic languages. The shared tasks attracted 18 teams whose submissions are documented in 26 system papers. The growing interest towards spoken language translation is also witnessed by the constantly increasing number of shared task organizers and contributors to the overview paper, almost evenly distributed across industry and academia.

heading: 1 Introduction
text: The International Conference on Spoken Language Translation (IWSLT) is the premier annual scientific conference for all aspects of spoken language translation (SLT). IWSLT is organized by the Special Interest Group on Spoken Language Translation (SIGSLT), which is supported by ACL, ISCA and ELRA.
Like in all the previous 20 editions, this year’s conference was preceded by an evaluation campaign featuring shared tasks addressing scientific challenges in SLT. This paper reports on the 2024 IWSLT Evaluation Campaign, which offered the following 7 shared tasks:
• Offline SLT, with focus on speech-to-text translation of recorded conferences and interviews from English to German, Japanese and Chinese.
• Simultaneous SLT, focusing on speech-totext translation of streamed audio of conferences and interviews from English to German, Japanese and Chinese.
• Automatic Subtitling, with focus on speechto-subtitle translation of audio-visual documents from English to German and Spanish and on compression of pregenerated German and Spanish subtitles.
• Speech-to-speech Translation, focusing on natural-speech to synthetic-speech translation of recorded utterances from English to Chinese.
• Automatic Dubbing, focusing on dubbing of production quality videos from English to Chinese.
• Low-resource SLT, focusing on the translation of recorded speech from Bhojpuri to Hindi, Irish to English, Marathi to Hindi, Maltese to English, North Levantine Arabic to English, Pashto to French, Tamasheq to French, Quechua to Spanish, and Bemba to English.
• Indic Languages Track, with focus on Speech-to-Text translation of TED talk audios from English to Indic languages including Hindi, Tamil, and Bengali.
The shared tasks attracted 18 teams (see Table 1) representing both academic and industrial organizations. The following sections report on
1
each shared task in detail. Each section includes a description of the proposed challenge, the data and evaluation metrics used for training and testing systems, the received submissions, and finally a summary of the results. Detailed results for some of the shared tasks are reported in a corresponding appendix.

heading: 2 Offline SLT
text: Recent advances in deep learning are providing the opportunity to address traditional NLP tasks in new and completely different ways. One of these tasks is spoken language translation (SLT), an overarching problem that can be cast in various manners, ranging from offline to simultaneous processing, to produce either textual or speech outputs under both unconstrained and constrained conditions. This section reports on the 2024 round of the IWSLT Offline Speech Translation Track, which consists of translating audio speech from one language into text in a different target language without any specific time or structural constraints, different from the simultaneous (see §3), subtitling (§4), speech-to-speech (§5), and dubbing (§7) tasks. Under this general problem definition, the goal of the offline SLT track—the one with the longest tradition at IWSLT—is to continuously challenge this rapidly evolving technology
by gradually introducing novel aspects that raise the difficulty bar.

heading: 2.1 Challenge
text: For years, SLT has been addressed by cascading an automatic speech recognition (ASR) system with a machine translation (MT) system. More recent trends involve using a single neural network to directly translate the input audio signal in one language into text in another language, bypassing intermediate symbolic representations such as transcriptions. In light of this evolution, the challenges addressed by the 2024 round of the offline track stem from the following considerations. (1) Although the results of the recent IWSLT campaigns have confirmed that the performance of end-to-end models is approaching that of cascade solutions, it is currently not clear which of the two technologies is more effective. Moreover, (2) all recent evaluations have been based on test sets extracted from TED talks, which represent a relatively simpler application scenario compared to the variety of potential deployments of SLT technology. In this controlled scenario, a single speaker delivers a prepared speech without background noise or interaction with other speakers. Finally, (3) last year’s edition showed that introducing complexity to the scenario (e.g., including spontaneous speech, terminology, and dialogues) resulted in a
clear performance degradation compared to using the classic TED talk test set.
Therefore, in addition to addressing the question of whether the cascade solution remains the dominant technology, this year we focused on understanding whether current state-of-the-art solutions can handle more complex scenarios (e.g., spontaneous speech, terminology, different accents, background noise, and dialogues). To shed light on these aspects, participants were challenged with data representative of different domains and conditions, namely:
• TED Talks1 – the classic IWSLT evaluation material, for which fresh test data were collected also this year;
• TV series from ITV Studios2 – data featuring multiple individuals interacting in various scenarios. The speech translation system needs to deal with overlapping speakers, different accents, and background noise;
• Physical training videos offered by Peloton3 – data featuring individuals exercising in the gym. The speech translation system needs to deal with with background noise and an informal speaking style;
• Accented English conversations – data featuring conversations, each containing two friends interacting on a daily topic, such as hobbies and vacation. The speakers were selected to cover a wide range of English speakers around the globe. In addition to the variety of accents, another major challenge is the presence of spontaneous speech.
In continuity with the last two years, three language directions were proposed. Depending on the evaluation scenario, the language conditions covered are:
• English → German: TED talks, TV series, physical training videos, and accented English conversations;
• English→ Japanese: TED talks. • English→ Chinese: TED talks.
1https://www.ted.com/ 2https://www.itvstudios.com/ 3https://www.onepeloton.com/

heading: 2.1.1 Test Suites
text: To further broaden the scope of evaluation conditions and explore specific aspects relevant to SLT, this year we provided participants with the option to submit additional test suites alongside the standard evaluation setting described above. The purpose of a test suite is to assess an SLT system on particular aspects that are generally hidden or overlooked by the classic evaluation frameworks. While the official evaluation relies solely on the designated official test sets, these supplementary test suites offer a valuable means to enhance system testing across a wider spectrum of phenomena. They also provide an opportunity to pinpoint specific and challenging issues that impact SLT performance. The particular test suite composition and its evaluation were fully delegated to the interested test suite provider.

heading: 2.2 Data and Metrics
text: Training and development data. Similar to the 2023 edition, participants were offered the possibility to submit systems built under three training data conditions:
1. Constrained: the allowed training data is limited to a medium-sized framework in order to keep the training time and resource requirements manageable. The complete list4 of allowed training resources (speech, speech-to-text-parallel, text-parallel, text-monolingual) does not include any pretrained language model.
2. Constrained with large language models (constrained+LLM ): in addition to all the constrained resources, a restricted selection4 of large language models is allowed to give participants the possibility to leverage large language models and medium-sized resources. We reproduce the list of allowed LLMs in Table 2.
3. Unconstrained: any resource, pre-trained language models included, can be used with the exception of evaluation sets. This setup is proposed to allow the participation of teams equipped with high computational power and effective in-house solutions built on additional resources.
4See the IWSLT 2024 offline track web page: https://iw slt.org/2024/offline
The development data allowed under the constrained condition consists of the dev set from IWSLT 2010, as well as the test sets used for the 2010, 2013-2015 and 2018-2020 IWSLT campaigns. Besides this TED-derived material, additional development data were released to cover the three new scenarios included in this round of evaluation.
Test data. As in previous rounds of the offline track, the collection of new test data for the TED talks scenario started by isolating a set of talks (41 in total) that are not included in the current public release of MuST-C (Cattoni et al., 2021). Starting from this material, which was used to build the initial English-German test set, the talks for which Japanese and Chinese translations are available were selected to build the EnglishJapanese and English-Chinese test sets. Since further checks revealed a partial overlap between the selected talks and the TED2020 corpus5 (Reimers and Gurevych, 2020) a final cleaning step had to be applied to remove the overlapping talks (4 for en-de, 4 for en-ja, none for en-zh). After this removal, the final test sets comprise 37 talks for English-German (corresponding to a total duration of 3h:07m:14s), 30 talks for English-Japanese (2h:14m:11s), and 30 talks for English-Chinese (3h:20m:19s).
For the TV series scenario, the 7 TV series for a total duration of 06h:01m are offered by ITV 5https://opus.nlpl.eu/TED2020/en&de/v1/ TED2020
Studios.6 Each series includes multiple speakers, background noise, and different audio conditions.
For the Physical training scenario, the 9 physical training videos for a total duration of 03h:59m are offered by Peloton.7 Each video includes a single speaker in a room practicing sports activities with, often, background music and breathy voice.
For the Accent challenge scenario, the test set has 1,448 utterances that are sampled from 76 conversations in the Edinburgh International Accents of English Corpus (EdAcc, Sanabria et al., 2023). In total, the test set contains about 3.5 hours of audio data, 34k English words, 25.2k German words and 33 accents. The German translations are created from the English transcripts by our professional translators who are paid at a rate of 0.095 GBP per word. The translators, with access to the aligned audio files, were required to translate the transcripts in a fluent and faithful manner while allowing punctuation and casing. For example, hesitation tokens like “ACH” and “HMM” in the transcripts are not included in the translation. The complete translation guidelines are attached in Appendix B.1.
Metrics. Systems were evaluated with respect to their capability to produce translations similar to the target-language references. The similarity was measured in terms of multiple automatic met-
6https://www.itvstudios.com 7https://www.onepeloton.com
rics: COMET8 (Rei et al., 2020), BLEU9 (Papineni et al., 2002a), chrF (Popović, 2015). Among them, this year COMET was chosen as the primary evaluation metric based the findings of Macháček et al. (2023) and Sperber et al. (2024), which indicate its highest correlation with human judgements. The submitted runs were therefore ranked based on the COMET calculated on the test set by using automatic resegmentation of the hypothesis based on the reference translation by mwerSegmenter,10 using a detailed script accessible to participants.11 Moreover, similar to last year’s round, a human assessment was performed on the bestperforming submission of each participant in order to enhance the soundness and completeness of the evaluation.

heading: 2.3 Submissions
text: This year, 4 teams participated in the offline task, submitting a total of 38 runs. Table 3 provides a breakdown of the participation in each subtask showing, for each training data condition, the number of participants, the number of submitted runs and, for each training data condition (constrained, constrained+LLM , unconstrained), the number of submitted runs obtained with cascade and direct systems. Notably, no direct system was submitted this year.
• CMU (Yan et al., 2024) participated with cascade en-de, en-ja, en-zh systems trained under the unconstrained condition. Their model consists of an ASR system based on Whisper and an MT system based on fine-tuned NLLB models. The ASR system is enhanced by the application of a specific fine-tuning to process unsegmented recordings without the need for a separate voice-activity detection stage. The MT systems generate a set of candidate translations via epsilon-sampling that are then pooled and the 1-best translation is selected using COMET-based Minimum Bayes-Risk decoding.
8Unbabel/wmt22-comet-da 9BLEU+case.mixed+numrefs.1+smooth.exp+tok.13a +version.1.4.14 10https://www-i6.informatik.rwth-aachen. de/web/Software/mwerSegmenter.tar.gz 11https://github.com/isl-mt/SLT.KIT/blob /master/scripts/evaluate/Eval.sh
• HW-TSC (Wu et al., 2024) participated with cascade en-de, en-ja, en-zh systems trained under the constrained, constrained with Large Language Models, and unconstrained conditions. The authors used different training strategies for each different condition. Under the constrained condition, an ASR is trained from scratch testing Conformer and U2. All audio inputs are augmented with spectral augmentation), and Connectionist Temporal Classification (CTC) is added to make the model converge better. The MT system takes advantage of the Deep Transformer-Big model structure, RDrop and data selection to identify in-domain data from a large pool of parallel data. Under the constrained + LLM condition, the ASR system is a combination of the wav2vec2 encoder and mBART50 decoder, where the self-attention of the encoder and decoder are frozen and all constrained are used for finetuning. The MT system is based on Llama27B fine-tuned with parallel data and source language consistent instructions, and applying CPO. Under the unconstrained condition, the ASR system is based Whisper fine-tuned and MuST-C, while the MT model selects the 1-best translation from a pool of candidates generated both with NMT and LLM using COMET. Audio segmentation is performed using SHAS.
• KIT (Koneru et al., 2024) participated with a cascade en-de system trained under the constrained with Large Language Models condition. This submission is based on a four-step approach. The audio is first transcribed by a fine-tuned ASR, the n-best list is then processed by an LLM to generate the best hypothesis. The final transcripts is translated to generate the text in the target language. The transcript and the translation are then paired and document- level automatic post-editing is applied to improve the coherence of the translations. The ASR is based on the combination of WavLM encoder and mBART50 decoder fine-tuned on the task data. Audio segmentation is based on SHAS, but a longformer technique is also tested to use context better. The ASR refiner and the MT posteditor are fine-tuned versions of Mistral 7B Instruction-tuned LLM using QLoRA, while
NLLB 200 3.3B is used as the MT system. The post-editing step showed to be less effective when the ASR quality is low. For this reason, LLM refinement is not used for the EPTV and ITV datasets.
• NYA (Zhang et al., 2024) participated with cascade en-de, en-ja, en-zh systems trained under the unconstrained condition. The ASR is based on Whisper-v3-large, while the MT system is a wider and deeper Transformer model. The MT model is enhanced by leveraging several techniques such as R-Drop, data augmentation with backward translations, domain adaptation via data filtering, and ASR output adaptation where the humanquality transcript in the SLT data is replaced with the automatic transcript. The final MT model is an ensemble of two/three models. The audio is segmented using SHAS.

heading: 2.4 Results
text: We will analyse the different aspects of the results by language pair.
2.4.1 English to German Correlation between BLEU, COMET and DA scores Table 25 shows the aggregated result of the participated systems on the four test sets. In terms of ranking based on the BLEU score, NYA wins 3 out of 4 test sets, except on ITV which CMU and HW-TSC(U) have a tie. However, the ranking is substantially changed when COMET is used. In this case, CMU is the winning system
in all conditions, indicating that this submission achieves the best performance. But in contrast to last year when the human evaluation validated the automatic metric rankings, the correlation between the automatic rankings and the human ranking is not as good as shown in Table 18. (More details on our human evaluation using DA are provided in Appendix A.2.1.) For the human evaluation, HW-TSC(C+) achieves the best performance overall and has the best DA ranking on 3 out of four test sets. Only on the accent test set, NYA has better scores. However, it is worth noticing that no system performs significantly better than HWTSC(C+) on any dataset.
The results show that it is essential to perform a human evaluation since no automatic metric, at the moment, can predict the performance of the individual systems well. Furthermore, additional research on performing reliable automatic metrics for speech translation would be very valuable.
It is interesting to note that all the submissions are based on the cascade architecture this year. This is an important change compared to previous editions where the end-to-end architectures competed with the cascade ones.
Context Beyond Segment Level One of the participating teams, KIT, used document-level post-editing to improve the coherence of translation. We note that while document-level consistency is a critical feature of text and speech translation, our evaluation this year does not reflect it yet. All used automatic metrics are segmentoriented. As detailed in Appendix A.2.1 also
the particular setup of DA this year did not allow the annotators to consider longer context because the segments were shuffled for DA. (Two neighbouring segments were provided but only to account for segmentation errors, not for assessment of context-level phenomena.) It is therefore conceivable that the outputs of the KIT system were somewhat penalized.
Domains Similar to last year’s edition, we evaluated each submitted system on different domains. First of all, the results show that the systems perform very differently in the different domains. When looking at the human ranking, the best quality is achieved in the TED domain. This is not surprising, since research has focused on this for many years and a significant amount of training resources exist. The performance on ITV and Peloton is lower, and the Accent data set appears to be the most challenging condition, indicating that speech translation remains an unsolved problem.
The availability of human rankings of the same systems across different domains allows us also to analyse whether automatic scores can be used to assess the quality of SLT system across domains. When ranking the difficulty level of different domains, we see that COMET ranks them similar than the human ranking except that COMET shows overall lower scores for Peloton, identifying Peloton as more challenging than Accent. In constrast, string-based metrics like BLEU are not able to do this. This also shows that additional metrics might be needed to measure the quality across domains.
Data conditions On top of the above, we can also observe the improvement in both BLEU and COMET scores caused by using an additional large language model or additional data. HWTSC submitted three primary systems for each data condition, and both the unconstrained (U) and the constrained+LLM (C+) models have a noticeable gain over the constrained model (C). The two better models perform similarly in both BLEU and COMET. Interestingly, additional training data beyond the language model data does not significantly improve. In terms of DA score, the constrained+LLM model is >0.6 points better than the other models in different data conditions.
Progress compared to last year We also performed an automatic evaluation of the system on the test sets from last year from the domains TED,
EMPAC, and ACL. The results are summarized in Table 26. Although the participants optimized for different domains, for each domain and each metric this year’s submissions achieved the best performance. When comparing the best submission from this year and last year, this year’s submission is between 4.4 and 1.5 BLEU points better and 1.1 to 2.7 COMET percent points better than the best system from last year.
Performance by accents For the accent test set, we performed an additional details analysis for the different accents.
Figure 1 shows the BLEU and COMET of each system across the 33 accents. The numbers in parenthesis are audio duration in the format of “minutes:seconds”. We use the self-reported labels from the original work as the prior choice for accent labeling. Since accents could be loosely defined (e.g., multi-class), subjective, and most speakers in the annotation are not the related experts, we thus derive the labels from other attributes, such as the first language of the speaker, if necessary and refine the labels to country-level. There is one speaker who declares his accent as “Trans-Atlantic” and speaks multiple first languages. We assign this special case as “Mixed”.
The aggregated result on Table 25 shows that CMU is the winning system on Accent when COMET is used for ranking, whereas NYA would be the winner if BLEU is used instead. Does this winning situation occurs on a wide range of accents or on a small subset? The breakdown on Figure 1 shows that CMU (the blue-diamond points) has better COMET scores, especially relative to NYA, and is within Top-2 on a wide range of accents. Similar observations are found in the better BLEU scores of NYA (the yellow-star points).
For the three primary systems submitted by HW-TSC (the red points), their performances are rather consistent across the 2 metrics and the accents. In most cases, both the constrained+LLM (the circles) and the unconstrained models (the squares) perform similarly, while the constrained model (the triangles) falls slightly behind. In the North Macedonian and the Pakistani accents, the constrained model seems to be better in both BLEU and COMET, but their data sizes are rather small, i.e. <1 minute. In the constrained LLM setting, the HW-TSC system in general performs better than the KIT system in a wide range of accents, but the KIT system has a slight edge in Indonesian,
Israeli and Japanese accents. The macro-average across accents are 18.7 BLEU and 0.679 COMET. Despite their fairly large test sizes, French, Irish, Jamaican, Kenyan and Vietnamese are below average. In Brazilian, German, Mexican and South African accents, all systems perform rather poorly, i.e., <10 BLEU. Potential causes are the train-test mismatch in accents, their small test sizes and the resegmentation error in the short utterances. Additionally, these speeches contain a mix of disfluencies and named entities, e.g., food ingredients, imposing further translation challenges.

heading: 2.4.2 English to Japanese
text: For the English to Japanese direction, we only have one test condition, the TED domain. In this case, the HW-TSC is the winner in all metrics, BLEU, COMET, and human ranking. However, the order of the submissions from HW-TSC varies across different metrics. Furthermore, the other two participants perform similarly on human ranking, but CMU is clearly better on COMET and NYA is clearly better on BLEU. This again suggests that the automatic metrics do not perform sufficiently well on speech translation tasks yet. Similar to the En-De language direction, all the submitted systems are based on the cascade architecture.
When comparing the submissions from this year and last year on the two progress test sets (TED and ACL), we again see a clear improvement compared to last year’s best systems.
For the data conditions, we see again a better performance of the unconstrained (U) and the constrained+LLM (C+) submissions from HWTSC compared to the system using only constrained data. However, this does not hold for the BLEU metric and the human evaluation. In these metrics, we see no clear benefit from using more data.

heading: 2.4.3 English to Chinese
text: For the English to Chinese direction, we also have only one test condition, the TED domain. In this case, the HW-TSC is the best system in human evaluation and COMET, while NYA performed best in BLEU. While this could indicate a good correlation between human evaluation and COMET, NYA actually serves as a counterexample: it performed worst in COMET and second best in human evaluation. This again suggests
that the automatic metrics do not work reliably on speech translation tasks yet. Similar to the other language directions, all the submitted systems are based on the cascade architecture.
When comparing the submissions from this year and last year on the two progress test sets (TED and ACL), we again see a clear improvement compared to the best systems of last year.
For the data conditions, we see again a better performance of the unconstrained (U) and the constrained+LLM (C+) submissions from HWTSC compared to the system using only constrained data, when considering the COMET metric and the human evaluation.

heading: 3 Simultaneous SLT
text: Simultaneous speech translation focuses on translating speech in real-time, in manner vaguely similar to simultaneous interpreting. The system is designed to begin translating before the speaker has finished their sentence. This technology is particularly useful in scenarios such as international conferences, personal travel, or public emergency events.
This year, the task included two tracks: speechto-text and speech-to-speech, covering four language directions: English to German, English to Chinese, English to Japanese, and Czech to English—a new language direction added this year.

heading: 3.1 Challenge
text: We have retained the settings from last year’s shared task. A single latency constraint is introduced for each of the tracks:
• An average lagging of 2 seconds for the speech-to-text track.
• A starting offset of 2.5 seconds for the speech-to-speech track.
Participants are allowed to submit no more than one system per track and language direction, provided the system’s latency remains within the specified constraints. The latency performance of the systems is evaluated using the open MuSTC tst-COMMON test set (Di Gangi et al., 2019). Submissions were accepted only in the form of Docker images, which were later executed by the organizers on the blind-test set in a controlled environment. An example implementation was
provided using the SimulEval toolkit (Ma et al., 2020).

heading: 3.2 Data
text: To simplify the setting and allow participants to focus on the new modeling aspects of simultaneous translation, we adhere to the constraints with large language models as defined for the offline SLT task, see Section 2.2 above. This is the sole data condition for the task. The test data differ across different language pairs:
English to German, Chinese, and Japanese Common TED Talks, which are the same as those used in the Offline task, as described in Section 2.2.
Czech to English The devset was created from two sources:
• A subset called “context” was taken from ParCzech 3.0 (Kopp et al., 2021), consisting of consecutive recordings of Parliament of the Czech Republic.
• An entire recording of a debate about AI from the ELITR test set (Ansari et al., 2021).12
The reference translations of the devset were done by students of translation studies from the Faculty of Arts at Charles University.
The testset was gathered from mock conferences that were part of the interpreting curriculum of the Faculty of Arts at Charles University. A speaker pretends to be a celebrity or an interesting person and delivers a made-up speech on a predetermined topic. We included 13 such speeches. The reference translations were provided by professional translators. Due to confidentiality of recordings, the testset is not released to the community. The statistics of the data are displayed in Table 4. 12https://github.com/ELITR/elitr-testset /tree/master/documents/2021-theaitre-r elated/robothon-debate

heading: 3.3 Evaluation
text: We evaluate two aspects of the model: quality and latency.
Quality We conducted both automatic and human evaluation. BLEU score (Papineni et al., 2002b) is used for automatic quality evaluation. For speech output, the BLEU score is computed on the transcripts from Whisper (Radford et al., 2023) ASR model. The ranking of the submission is based on the BLEU score on the Common blind test set. The human evaluation was conducted in English-to-German/Chinese/Japanese, as described in A.1.
Latency We only conducted automatic evaluation. We report the following metrics for each speech-to-text systems.
• Average Lagging (AL; Ma et al., 2019)
• Length Adaptive Average Lagging (LAAL; Polák et al., 2022; Papi et al., 2022a)
• Average Token Delay (ATD; Kano et al., 2023)
• Differentiable Average Lagging (DAL; Arivazhagan et al., 2019)
For speech-to-speech systems, we report startoffset, end-offset and Average Token Delay. The latency metrics will not be used for ranking.

heading: 3.4 Submissions
text: Four teams in total submitted systems this year, with all teams participating in at least one language direction in the speech-to-text track. All teams entered the English-to-German track; three teams entered the English-to-Chinese and Englishto-Japanese tracks; and two teams entered the Czech-to-English track, to which we added a Whisper-based benchmark. For the speech-tospeech track, two teams submitted systems, with one team submitting for all language directions and the other only in the English-to-Japanese direction.
CMU (Xu et al., 2024) participated in the speech-to-text track for the English-to-German direction. Their system integrates the WavLMbased speech encoder (Chen et al., 2021), a modality adapter, and the Llama2-7B-based decoder (Touvron et al., 2023). The training is conducted in two stages: modality alignment and
full fine-tuning, both performed on MuST-C v2 data (Cattoni et al., 2021). The two-stage training results in an offline speech translation model, which is then adapted to a simultaneous speech translation model with a simple fixed hold-n policy.
FBK (Papi et al., 2024) participated in all language directions of the speech-to-text track. Their system is a unified multilingual simultaneous speech translation system, combining AlignAtt (Papi et al., 2023b) and SeamlessM4Tmedium (Seamless Communication et al., 2023). The SeamlessM4T model is directly used in its streaming mode without additional retraining. The generated hypotheses are further processed through AlignAtt for policy learning. Based on diverse training sources, the model can translate into approximately 200 target languages from 143 source languages.
HW-TSC (Li et al., 2024a) participated in all language directions of both the speech-to-text and speech-to-speech tracks. Except for the Czechto-English direction, all other models utilize cascaded simultaneous speech translation approaches by combining offline speech recognition, machine translation, and text-to-speech. For the Czech-toEnglish direction, they utilize the offline SeamlessM4T (Seamless Communication et al., 2023) as the backbone for speech-to-text translation, combined with a text-to-speech system. They followed their last year’s submissions as the base setting (Guo et al., 2023). Additionally, they applied online voice-activity-detection-oriented segmentation, chunk padding in the speech recognition system to achieve smaller delays, and added an ensemble strategy for machine translation to achieve better stability. For end-to-end speech-totext translation, they fine-tuned the SeamlessM4T model using the suggested data in the simultaneous SLT shared task.
NAIST (Ko et al., 2024) participated in three language directions of the speech-to-text track. Their speech-to-text system combined HuBERT (Hsu et al., 2021) and mBART (Liu et al., 2020b) in an end-to-end fashion, with a local agreement policy (Liu et al., 2020a; Polák et al., 2022). Their speech-to-speech system further applied an incremental text-to-speech module tuned with AlignAtt policy (Papi et al., 2023b).
ORGANIZER’S BENCHMARK by Charles University was prepared for the Czech-to-English direction. The system is based on Whisper (Radford et al., 2023) version large-v2. We applied an onlinization technique (Polák et al., 2022, 2023a,b) to utilize the offline Whisper model in the simultaneous regime, and applied prompting to leverage the translation history from previous segments. Due to organizational reasons, the benchmark was run on different hardware so the comparison of computationally-aware latency with other systems is not possible.

heading: 3.5 Results
text: We rank the system performance based on BLEU scores. The detailed results can be found in the respective tables in Appendix A.2.3.
Speech-to-Text The ranking of the speech-totext track is as follow
• English to German (Table 29): HW-TSC, CMU, NAIST, FBK
• English to Chinese (Table 30): HW-TSC, NAIST, FBK
• English to Japanese (Table 31): HW-TSC, NAIST, FBK
• Czech to English (Table 32): ORGANIZER’S BENCHMARK (with context of 2 segments), FBK, HW-TSC
Speech-to-Speech As mentioned in Section 3.4, two teams submitted speech-to-speech track this year. HW-TSC submitted systems on all language directions and NAIST submitted on English to Japanese Direction. We only rank the English to Japanese Direction. The rank is: HW-TSC, NAIST. See Table 33 for more details.

heading: 3.6 Conclusions
text: Over the past four years, the IWSLT has consistently featured simultaneous translation tasks, reflecting a growing interest and impressive progress in this area. The shared task also brings the establishment of standardized evaluation protocols for simultaneous translation research. The recent integration of foundation models has further expanded the potential of this task. All teams integrated such models into their submissions using different approaches. CMU and NAIST teams combined two foundation models each specialized
in one modality (speech encoder and text decoder) together using fine-tuning, while others chose existing ST models such as SeamlessM4T or Whisper and modified them for simultaneous use. Surprisingly, even large models (e.g., the CMU’s Llama2-7B-based decoder) achieved competitive computationally-aware latencies.
The only cascaded system in the competition (HW-TSC) was consistently rated first in three language pairs. Nevertheless, according to all latency measurements, this system also exhibited the highest computationally-aware latencies.
One of the interesting points this year is the newly-added Czech-to-English translation direction where we included our Whisper-based benchmark. When operating at the segment level, this benchmark performed worse than participants’ systems, but given one or two of its previous translation outputs, it improved over them. This confirms that the role of context is very important in speech translation task and the best uses of LLMs for this task are still to be found.
Several promising directions for future improvements remain. Investigating downstream tasks such as cross-lingual dialogues could provide deeper insights into practical applications of simultaneous translation. Developing more interactive evaluation methods could enhance the understanding and effectiveness of these systems. Lastly, optimizing the evaluation procedure to expedite the process remains crucial, as the current system managed by the organizers can be timeconsuming.

heading: 4 Automatic Subtitling
text: In recent years, the task of automatically creating subtitles for audiovisual content in another language has gained a lot of attention due to the rapid increase in the global distribution and streaming of movies, series, and user-generated videos. Reflecting these trends, the automatic subtitling track was introduced for the first time in 2023 as part of the IWSLT Evaluation Campaigns. Given the growing interest in this area, the task has been continued this year with the addition of a new subtrack, subtitle compression, alongside the existing automatic subtitling sub-task from the previous edition.
In the automatic subtitling task, participants were asked to generate subtitles in German and/or Spanish from English speech in audiovisual docu-
ments. In the new subtitle compression task, participants were required to automatically rephrase subtitles that did not comply with the reading speed constraint (i.e., subtitles exceeding a certain length/time ratio given in characters per second) to ensure they met the required standards.
The decision to have works focusing on this specific aspect of subtitling is highly motivated by the existing requirements posed by subtitles providers (Papi et al., 2023a). In fact, the constraint on the reading speed is a commonly adopted standard to ensure that viewers can enjoy audiovisual content without experiencing fatigue or distraction due to excessive reading demands (Kruger, 2001). Therefore, adhering to this limit is crucial, making the development of ad-hoc methods to improve automatically generated subtitles that exceed this threshold of particular interest.

heading: 4.1 Challenge
text: Automatic Subtitling. The task of automatic subtitling is multifaceted: starting from speech, not only must the translation be generated, but it must also be segmented into subtitles that comply with constraints ensuring a high-quality user experience. These constraints include proper reading speed, synchrony with the voices, the maximum number of subtitle lines, and characters per line. Most audio-visual companies define their own subtitling guidelines, which can slightly differ from each other. In the case of IWSLT participants, we asked to generate subtitles according to specific guidelines provided by TED, including:
• The maximum subtitle reading speed is 21 characters per second;
• lines cannot exceed 42 characters, including white spaces;
• Subtitles cannot exceed 2 lines.
Participants were expected to use only the audio track from the provided videos (dev and test sets), the video track was of low quality and primarily meant to verify time synchronicity and other aspects of displaying subtitles on screen. That being said, the exploitation of the video was permitted.
The subtitling sub-track required participants to automatically subtitle audio-visual documents in German and/or Spanish, where the spoken language is always English. These documents were collected, similarly to last year, from the following sources:
• TED talks;13
• Physical training videos offered by Peloton;14 • TV series from ITV Studios.15
Subtitle Compression. The objective of the subtitle compression sub-track was to engage teams interested in the subtitling task but unable to build a complete automatic subtitling system. Participants were provided with automatic subtitles (in German and Spanish) generated by a non-participating system, namely the system presented in (Papi et al., 2023a), and asked to rephrase those that exceeded the reading speed constraint (more than 21 characters per second) to make them compliant. Time boundaries were to remain unchanged: only the text within a given time span had to be compressed when necessary. The original audiovisual documents (from the ITV test24 set of the subtitling sub-track) were also provided.
Although the subtitle compression task may appear simpler than subtitling, and it certainly is from the point of view of architectural complexity, it still presents its own difficulties. These challenges include those inherent in text summarization, such as identifying the main content of the original text, which must be preserved, and distinguishing accessory information, which can be omitted if necessary. Additionally, a peculiar challenge is that the text that needs to be reformulated is potentially error-prone and often does not consist of well-formed sentences but rather spans of text representing portions of sentences or words spanning contiguous phrases. It is expected that the most effective solutions are those capable of looking at the context, in an attempt to recover as much as possible the missing information in the text being processed.

heading: 4.2 Data and Metrics

heading: 4.2.1 Automatic subtitling
text: Data. This sub-track proposed two training data conditions:
• Constrained: the official training data condition, in which the allowed training data is limited to a medium-sized framework16 to
13https://www.ted.com/ 14https://www.onepeloton.com 15https://www.itvstudios.com 16https://iwslt.org/2024/subtitling#trai ning-data-allowed-for-constrained-con ditions
keep the training time and resource requirements manageable;
• Unconstrained: a setup without data restrictions (any resource, pre-trained language models included, can be used) to allow also the participation of teams equipped with high computational power and effective in-house solutions built on additional resources.
For each language and domain, a development set and two test sets were released, that of the 2023 evaluation (tst2023), used for measuring progress over years, and a new one (tst2024). Table 5 provides some statistics on these sets.
Metrics. The evaluation was carried out from three perspectives, subtitle quality, translation quality, and subtitle compliance, through the following automatic measures:
• Subtitle quality vs. reference subtitles: – SubER, primary metric, used also for
ranking (Wilken et al., 2022);17
• Translation quality vs. reference translations: – BLEU18 and CHRF19 via sacreBLEU; – BLUERT (Sellam et al., 2020).
Automatic subtitles are realigned to the reference subtitles using mwerSegmenter (Matusov et al., 2005)20 before running sacreBLEU and BLEURT.
17https://github.com/apptek/SubER 18sacreBLEU signature: nrefs:1|case:mixed|eff: no|tok:13a|smooth:exp|version:2.0.0 19sacreBLEU signature: nrefs:1|case:mixed|eff: yes|nc:6|nw:0|space:no|version:2.0.0 20https://www-i6.informatik.rwth-aachen. de/web/Software/mwerSegmenter.tar.gz
• Subtitle compliance:21
– rate of subtitles with more than 21 characters per second (CPS);
– rate of lines longer than 42 characters, white spaces included (CPL);
– rate of subtitles with more than 2 lines (LPB).

heading: 4.2.2 Subtitle compression
text: Data. No specific training data was released for this sub-track. Any solution was allowed, without limitations on the training data, including the use of LLM prompted for text compression (e.g. chatGPT). The original audio, though potentially helpful, could either be used or not by participants; its transcription with external tools (e.g. Whisper) was also permitted.
As a development set, a minimal example taken from the EuroParl Interviews benchmark (Papi et al., 2023a)22 was released, where the nonparticipating subtitling system introduced in (Papi et al., 2023a)23 was employed to generate automatic, sometimes non-compliant subtitles, which were associated with corresponding compliant reference subtitles.
The test set consists of German and Spanish automatic subtitles for the audiovisual documents defining the ITV test24 set of the subtitling subtrack; the same non-participating subtitling system was employed to generate the subtitles to be corrected.
Metrics. Since the text in subtitles has to be compressed to fulfill the CPS requirement, but at the same time its meaning should be preserved as best as possible, both CPS and BLEURT are considered primary metrics in the evaluation of compression quality.

heading: 4.3 Submissions

heading: 4.3.1 Automatic subtitling
text: The subtitling sub-track saw the participation of three teams: APPTEK, the MT unit of Fondazione Bruno Kessler (FBK) with two different systems, and Huawei Translation Service Center (HWTSC). The details about the participants’ systems are provided below: 21https://github.com/hlt-mt/FBK-fairseq/b lob/master/examples/speech_to_text/scr ipts/subtitle_compliance.py 22https://mt.fbk.eu/europarl-interviews/ 23https://github.com/hlt-mt/FBK-fairseq/b lob/master/fbk_works/DIRECT_SUBTITLING .md
AppTek: the cascade-based subtitling system developed by APPTEK24 leveraging their inproduction automatic captioning and translation offerings. A pipeline of in-house hybrid ASR, punctuation and inverse text normalization models is used to create English captions, which are segmented into blocks and lines via a neural segmentation model in combination with hard subtitling constraints, similar to Matusov et al. (2019). Time stamps follow from the HMM alignment of the first and last word in a block. In a second step, the generated source template is translated with customized transformer-based NMT models, for which full sentences are extracted and translations are reinserted into the template using a variant of the source-side segmentation method that enforces splitting into the existing blocks. The NMT models make use of preceding sentence context, and prefix tokens are used to provide genre and formality information (e.g. “talks” + “formal” for TED) and to control the length of the translation (Matusov et al., 2020). For the primary submission, the MT component is fine-tuned on high quality media and entertainment customer data. In addition, the following newly developed features are employed: automatic MT length token selection to condense translation only where necessary due to space constraints; extension of subtitle timings for lower reading speed; improved Spanish MT model. The contrastive submissions do not use these upcoming features. The second contrastive submission is created using APPTEK’s general domain MT models, which are trained on publicly available data.
FBK-AI4CDIR (Gaido et al., 2024a): the FBK’s direct subtitling system is based on the transcription-free novel architecture, SBAAM or Speech Block Attention Area Maximization, introduced in (Gaido et al., 2024b). SBAAM leverages cross-attention scores to retrieve the timestamp information and is the first fully direct solution capable of producing automatic subtitles by eliminating any dependence on intermediate transcripts. It is the only system trained under constrained conditions, utilizing only the limited data provided by the IWSLT 2024 organizers. This includes non-subtitle material, which was automatically segmented into subtitles using the multimodal segmenter by Papi et al. (2022b). SBAAM is also employed as a reference system in the
24https://www.apptek.com/
AI4Culture EU project25 and is available at: ht tps://github.com/hlt-mt/FBK-fairseq/blo b/master/fbk_works/SBAAM.md.
FBK-AI4CCSC (Gaido et al., 2024a): the FBK’s cascade subtitling system, developed by FBK within the AI4Culture project, exploiting pretrained language models and, therefore, participating under the unconstrained conditions. The system is a cascade solution with Whisper (Radford et al., 2023) as the ASR model, and Helsinki Opus-MT (Tiedemann and Thottingal, 2020) as the MT model, together with additional components developed in-house. The cascade solution is publicly available at: https://github.com/hlt -mt/FBK-subtitler.
HW-TSC (Xie et al., 2024): the unconstrained cascade solution developed by HW-TSC, which relies on Whisper (Radford et al., 2023) to estimate both transcripts and word-level timestamps, on Bert-restore-punctuation26 for retrieving punctuation and sentence segmentation, and on wav2vec2-large-960h-lv6027 for the CTCbased force alignment between transcripts and translations, obtained by in-house MT models. The MT models (English to German and English to Spanish) were directly employed on the sentence-level ASR transcripts while the timestamps were left unchanged between transcripts and translations. Moreover, they are the only models among all participants that were specifically adapted to the domains of the audiovisual documents through ad-hoc domain adaptation.

heading: 4.3.2 Subtitle compression
text: Three teams participated in the sub-track: the FBK MT unit, the Huawei Translation Service Center (HW-TSC), and the Research Institute for Artificial Intelligence Mihai Drǎgǎnescu, Romanian Academy (RACAI). The solutions they proposed differ from each other, although they share the use of Large Language Models as a common trait. Specifically:
FBK (Gaido et al., 2024a): the primary submission exploited GPT-4 (Achiam et al., 2023), which was prompted in zero-shot mode with an instruction asking the model to shorten the input text us25https://pro.europeana.eu/project/ai4cu lture-an-ai-platform-for-the-cultura l-heritage-data-space 26Bert-restore-punctuation1 27https://huggingface.co/felflare/bert-r estore-punctuation
ing the maximum number of characters compatible with the subtitle duration (value computed offline and passed as a parameter) while preserving the original words as much as possible. In the two contrastive runs, non-compliant subtitles were compressed by deleting function words from lists of different lengths.
HW-TSC (Xie et al., 2024): the subtitle compression method for the primary run is based on MT models, which are first employed for backtranslating the non-compliant subtitles into English, and then to re-translate English into the original language (either German or Spanish) by setting a large beam size and a high length penalty, so that short translations are generated and rewarded. The still non-compliant subtitles are rewritten using the LLM Llama2 (Touvron et al., 2023), instructed with few-shot prompts to condense the input text. The two contrastive runs are variants of the primary one: in the first, the LLM is not applied and the compression is carried out only by the translation models; in the second, the subtitles of the primary run rewritten by either the MT model or the LLM which are still non-compliant are replaced by the original text.
RACAI (Gasan and Păis, , 2024): the submission involves generating multiple alternatives for the original non-compliant subtitle and selecting the one that maximizes both reading speed compliance (measured by CPS), and content similarity with the original subtitle (measured by ROUGE (Lin, 2004)). The alternatives are generated by i) rephrasing the subtitles using LLMs, specifically T5 (Raffel et al., 2020) and BART (Lewis et al., 2020), which were fine-tuned for the text summarization task, and ii) generating new subtitles through the automatic transcription of the original English audio using Whisper, translating them with NLLB (Costa-jussà et al., 2022), and then applying the LLMs as in the first method.

heading: 4.4 Results
text: The performance of runs for the two sub-tracks is presented and discussed separately in the following two subsections.

heading: 4.4.1 Automatic subtitling
text: Scores on tst2024 of all runs calculated using automatic metrics are shown in Tables 34 and 35, while Tables 37 and 38 refer to tst2023, where cumulative scores of runs submitted to the 2023 edi-
tion are also reported to allow the quantification of progresses.28
This year, unlike in the last edition, only one team (FBK-AI4CDIR) participated with a system trained under constrained data conditions. Consequently, comparing its results with those of other participants is inherently unfair, and must be acknowledged if any comparisons are made. Notably, FBK-AI4CDIR is also the only direct system in the competition, highlighting that, despite advancements in direct approaches to spoken language processing, constructing cascade subtitling systems remains prevalent.
tst2024: Looking at performance in both German and Spanish, APPTEK achieved the best compromise between translation quality and subtitle compliance, as attested by the SubER values. It is interesting to note that their primary and contrastive1 systems provide better subtitle quality than contrastive2, especially on Spanish; since the first two systems featured fine-tuning on proprietary data, it can be hypothesized that such data is somehow “close” to the domains proposed in this evaluation campaign and therefore that the adaptation has rewarded these models. Overall, the new APPTEK systems (primary and contrastive1) surpass the one currently in production (contrastive2), although surprisingly the latter shows the best global SubER on German.
Focusing on the quality of the translation, in particular in terms of BLEURT, which better correlates with humans compared to BLEU and ChrF, the performance of HW-TSC’s system is superior, likely because it is the only system explicitly finetuned on in-domain data. However, this system has not been optimized in terms of compliance, resulting in the lowest CPL score and, consequently, in high SubER scores.
The FBK cascade system, based mainly on pretrained general-purpose models, shows high translation quality, especially in Spanish, and an acceptable conformity of subtitles. This proves the feasibility of building effective subtitling systems by appropriately assembling off-the-shelf models.
The FBK direct system, the only one based on a direct architecture and trained in constrained conditions, generated German subtitles with a surprisingly competitive overall SubER, despite the qual28In 2023, the evaluation was done on the three domains still
proposed here plus one additional domain, EPTV; for the sake of comparability, in the computation of the cumulative scores of the 2023 runs, EPTV has been excluded.
ity of the translation of the ITV and Peloton documents being lower compared to other systems. The good SubER probably derives from the ability of this system to satisfy subtitle compliance, which demonstrates the potential of the innovative approach it is based on. On the other hand, the gap in terms of translation quality on the two more challenging domains is in line with what already happened last year and with expectations, since unconstrained training allows building models on data more representative of real-life content.
tst2023: On German, the best systems are those by APPTEK which however did not improve the SubER score of the last year; in fact, there is an improvement in the quality of the translation which is counterbalanced by a worst CPS. Moreover, we note that the CPS of 4 out of 5 submissions from last year is better than any 2024 primary submission.
On Spanish, the improvements in the quality of the translations and of the SubER scores are generalized, while the CPS values worsen.
The progress made by the FBK team over the past year with their direct approach is notable in various aspects and for both languages, demonstrating the potential of end-to-end solutions for automatic subtitling.

heading: 4.4.2 Human evaluation
text: This year’s edition of the automatic subtitling subtrack introduces the human evaluation of the primary submissions for tst2024 en→de. Table 24 shows the direct assessment scores obtained on a sample of 1000 subtitles randomly selected from the whole test set. The ranking differs from the automatic one based on SubER, particularly for the HW-TSC system which achieves the best DA value but the worst SubER score. This can be explained by the design of the human evaluation, which was focused on assessing the translation quality while segmentation and subtitle compliance were not directly considered. In fact, the human ranking closely agrees with the pure translation quality metrics, in particular BLEURT (see Table 24 vs. column Bleurt of Table 34). While this reassures the validity of using automatic MT metrics also for the domain of subtitle translation, in future evaluations we see the need to provide the evaluators with subtitles instead of plain text sentences so that subtitle compliance, segmentation and timing errors can be accounted for.

heading: 4.4.3 Subtitle compression
text: Table 36 shows the results of the submissions to the subtitle compression sub-track in terms of BLEURT, computed against the reference subtitles and in charge of quantifying the translation quality, and CPS, as a measure of reading speed compliance. For the sake of discussion, the table also includes the results of a simple Baseline (id=[1]) and those of the provided subtitles to compress (id=[0]). In the baseline method, the original subtitles with a non-compliant reading speed were cut at the maximum number of characters compatible with the subtitle duration and without regard to maintaining the integrity of the words, which therefore may be incomplete.
The results indicate that the participants designed methods aimed to find a trade-off between translation quality and CPS compliance, standing the working point of their systems in the area between the two extremes represented by subtitles [0] and [1], which is highlighted in Figure 2.
Between [0] and [1], the subtitles generated by the contrastive FBK ([3,4]) and by the RACAI ([8]) systems are placed according to a nearly linear relationship. HW-TSC’s and, at a lesser extent, primary FBK ([2]) submissions differ markedly from this trend, thus demonstrating that it is possible to obtain a better compromise between the two contrasting features. In particular, the family of HW-TSC solutions is the most effective, approaching (in Spanish) or even overcoming (in German) the translation quality of the original subtitles, while achieving compliance for even more than 90% of the original subtitles. However, the noteworthy result of the FBK primary run shows the potential of prompting a generative LLM (GPT-4) to shorten subtitles; consider-
ing that it was done in zero-shot modality, there should be room for further improvements.

heading: 4.5 Conclusions
text: Overall, the second edition of the subtitling track continues to highlight the challenges and particularities of the automatic subtitling task. As in the previous edition, a clear gap in subtitle quality can be observed between the well-recorded, single-speaker, mostly formal style TED talk content that has traditionally been used for SLT evaluation at IWSLT, as opposed to the variety of audio conditions, dialog settings, language styles and speaking rates encountered in other types of content such as TV shows and sport videos. While no clear advancement in terms of best achieved translation quality or subtitle compliance compared to last year can be reported, remarkable improvements were achieved in the direct approach, which due to access to audio information during translation such as prosody, speaker changes and even speaker age/gender seems especially promising for subtitling of dialogs. The aspect of high speaking rates and the resulting necessity to condense subtitles down to a comfortable reading speed has been addressed and analyzed in isolation by the introduction of the subtitle compression task. Here, using LLMs for rephrasing has emerged as one of the promising approaches which was used by all participants.

heading: 5 Speech-to-Speech Translation
text: Speech-to-speech translation (S2ST) is a highly complex process involving the conversion of audio signals from one language to another. In offline translation, the system assumes that the entire audio is available before the translation process begins. This approach allows the translation system to process the audio input as a whole, enabling more effective speech recognition, semantic comprehension, and translation.
The main objective of this task is to encourage the development of automated methods for speech-to-speech translation that can perform efficiently and accurately in offline settings. Achieving this goal will not only advance the field but also contribute to improving access to information and communication across different languages and cultures.

heading: 5.1 Challenge
text: Participants built speech-to-speech translation systems from English into Chinese using any possible method, for example with a cascade system (ASR + MT + TTS or end-to-end speech-to-text translation + TTS) or an end-to-end or direct speechto-speech system. Participants can use any techniques to boost the system performance.

heading: 5.2 Data and Metrics
text: Data. This task allowed the same training data from the Offline task on English-Chinese speechto-text translation. More details are available in Sec. 2.2. In addition to the Offline task data, the following training data was allowed to help build English-Chinese speech-to-speech models and Chinese text-to-speech systems:
• GigaS2S, target synthetic speech for the Chinese target text of GigaST (Ye et al., 2023) that was generated with an in-house singlespeaker TTS system;
• aishell 3 (Shi et al., 2020), a multi-speaker Chinese TTS dataset.
Metrics. Since there was only one participant this year, we only conducted automatic evaluation in order to save resources.
Automatic metrics. To automatically evaluate translation quality, the speech output was automatically transcribed with a Chinese ASR system29 (Yao et al., 2021), and then BLEU30 (Papineni et al., 2002b), chrF31 (Popović, 2015), and COMET32 (Rei et al., 2022) were computed between the generated transcript and the humanproduced text reference. BLEU and chrF were computed using SacreBLEU (Post, 2018).

heading: 5.3 Submissions
text: We only received submissions from one participant this year.
• HW-TSC (Wu et al., 2024) submitted three cascaded systems corresponding to three scenarios: constrained, constrained with large
29https://github.com/wenet-e2e/wenet/blo b/main/docs/pretrained_models.en.md 30sacreBLEU signature: nrefs:1|case:mixed| eff:no|tok:zh|smooth:exp|version:2.3.1 31sacreBLEU signature: nrefs:1|case:mixed| eff:yes|nc:6|nw:0|space:no|version:2.3.1 32https://huggingface.co/Unbabel/wmt22-c omet-da
language models, unconstrained. All three scenarios employ a cascaded system that consists of an Automatic Speech Recognition (ASR) model, a translation model, and a Text-to-Speech (TTS) model. In the constrained scenario, the ASR model is trained on WeNet using constrained data. The translation model is a Transformer model trained using constrained data, with data enhancement, data denoising, and domain adaptation strategies applied, followed by model ensemble. The TTS model uses the VITS architecture. In the LLM constrained scenario, the ASR model is the same as in the constrained scenario. The translation model uses multiple LLMs for model ensemble, which are finetuned on llama2-13b using different strategies. The TTS model is the same as above. In the unconstrained scenario, the ASR model uses Whisper. The translation model employs multiple NMT models and LLMs for model ensemble. The TTS model remains the same as in the previous scenarios.

heading: 5.4 Results
text: Results by automatic metrics are shown in Table 39 in the Appendix.

heading: 6 Low-resource SLT
text: The 4th edition of the Low-resource Spoken Language Translation track focused on the translation of speech from a variety of data-scarce languages. The target language is typically a higher-resource one, generally of similar geographical or historical linkages. The goal of this shared task is to benchmark and promote speech translation technology for a diverse range of dialects and low-resource languages. While significant research progress has been demonstrated recently, many of the world’s languages and dialects lack the parallel data at scale needed for standard supervised learning.

heading: 6.1 Challenge
text: This year’s task significantly expanded the typological and geogrpahical diversity of the languages, language families, and scripts represented. The eight subtasks were:
• Bhojpuri→ Hindi • Marathi→ Hindi • Irish→ English
• Maltese→ English • Bemba→ English • North Levantine Arabic→ English • Tamasheq→ French • Quechua→ Spanish Teams were allowed to submit to as few as one language pair, up to all eight. Both constrained and unconstrained submissions were allowed, to be separately ranked. For the constrained scenario, teams were only allowed to submit systems using the data provided by the shared task. For the unconstrained systems, teams were allowed to use any data as well as any pre-trained models.

heading: 6.2 Data and Metrics
text: Table 6 provides a summary of the training data that were part of the shared task. We describe in more detail the data for each language pair below.
North Levantine Arabic–English (apc-eng) Levantine Arabic, a well-established unit within the Arabic dialectal continuum, can be divided into at least three regional variants (Al-Wer and de Jong, 2017). North Levantine Arabic (also known as Syrian or Shami, ISO code: apc) is based on the urban speech of mainly Beirut and Damascus and is perceived as a separate linguistic unit (Ghobain, 2017).
Participants were provided with the UFAL Parallel Corpus of North Levantine 1.0 (Sellat et al., 2023), which includes about 120k lines of multi-parallel North Levantine-Modern Standard Arabic-English textual data, that can be downloaded from the LINDAT/CLARIAH-CZ Repository.33 For additional speech data in North Levantine Arabic, participants were pointed to two LDC resources: the BBN/AUB DARPA Babylon Levantine corpus (Makhoul et al., 2005) and the Levantine Arabic QT Training Data Set 5 corpus (Maamouri et al., 2006).
Participants were also encouraged to use the Tunisian Arabic training data used in the last two years’ shared task (LDC2022E01). This three-way parallel data corresponds to 160 hours and 200k lines of aligned audio in Tunisian speech, Tunisian transcripts, and English translations. Additionally, a number of OpenSLR resources in Modern 33http://hdl.handle.net/11234/1-5033
Standard Arabic were highlighted: Tunisian Modern Standard Arabic speech and transcriptions34, the MADCAT Arabic LDC corpus (Lee et al., 2012), the Arabic portion of theMediaSpeech corpus (Kolobov et al., 2021), and the Arabic speech to text Quran data.35
Overall, the provided resources were supposed to help participants, but only the unconstrained scenario was considered within this year’s initial run of the apc-eng language pair.
The development36 and test37 data consist of recordings of native speakers of the dialect and is a mix of spontaneous monologues and dialogues on the topics of everyday life (health, education, family life, sports, culture), living abroad, and everyday life in Syria. The transcription and translation team consisted of students of Arabic at Charles University, with an additional quality check provided by the native speakers of the dialect.
Bemba–English (bem-eng) Bemba (also known as IciBemba) is a Bantu language (ISO code: bem), spoken predominantly in Zambia and other parts of Africa by over 10 million people. It is the most populous indigenous language spoken by over 30% of the population in Zambia where English is the lingua franca and official highresourced language of communication. Bemba is native to the people of Northen, Luapula and Muchinga provinces of Zambia but also spoken in other parts of the country including urban areas such as Copperbelt, Central and Lusaka provinces by over 50% of the population (ZamStats, 2012).
The provided Bemba-English corpus (Sikasote et al., 2023a) consists of over 180 hours of Bemba audio data, along with transcriptions and translations in English. The dataset is comprised of recorded multi-turn dialogues between native Bemba speakers grounded on images.
In addition, we provided transcribed (28 hours) and untranscribed (60 hours) monolingual Bemba speech from Zambezi Voice (Sikasote et al., 2023b) and BembaSpeech (Sikasote and Anastasopoulos, 2022) datasets.
Bhojpuri–Hindi (bho-hin) Bhojpuri (ISO code: bho) belongs to the Indo-Aryan language group. It is dominantly spoken in India’s western part of Bihar, the north-western part of Jharkhand, 34https://www.openslr.org/46/ 35https://www.openslr.org/132/ 36http://hdl.handle.net/11234/1-5518 37http://hdl.handle.net/11234/1-5519
and the Purvanchal region of Uttar Pradesh. As per the 2011 Census of India, it has around 50.58 million speakers (Ojha and Zeman, 2020). Bhojpuri is spoken not just in India but also in other countries such as Nepal, Trinidad, Mauritius, Guyana, Suriname, and Fiji. Since Bhojpuri was considered a dialect of Hindi for a long time, it did not attract much attention from linguists and hence remains among the many lesser-known and less-resourced languages of India.
The provided Bhojpuri–Hindi corpus consists of 22.77 hours of Bhojpuri speech data (see Table 6) from the news domain, extracted from News On Air38 and translated into Hindi texts.39 Additionally, the participants were directed that they may use monolingual Bhojpuri audio data (with transcription) from ULCA-asr-dataset-corpus40 as well as Bhojpuri Language Technological Resources (BHLTR) (Ojha et al., 2020; Ojha, 2019)41 and Bhojpuri-wav2vec2 based model.42
Irish–English (gle-eng) Irish (also known as Gaeilge; ISO code: gle) has around 170,000 L1 speakers and 1.85 million people (37% of the population) across the island (of Ireland) claim to be at least somewhat proficient with the language. In the Republic of Ireland, it is the national and first official language. It is also one of the official languages of the European Union (EU) and a recognized minority language in Northern Ireland with the ISO ga code.
The provided Irish audio data were compiled from the news domain, Common Voice (Ardila et al., 2020),43 and Living-Audio-Dataset.44 The Irish–English corpus consists of 12 hours of Irish speech data (see Table 6), translated into English texts.
Maltese–English (mlt-eng) Maltese (ISO code: mlt) is a Semitic language, with a heavy influence from Italian and English. It is spoken mostly in Malta, but also in migrant communities abroad,
38https://newsonair.gov.in 39https://github.com/panlingua/iwslt2024 _bho-hi 40https://github.com/Open-Speech-EkStep/ ULCA-asr-dataset-corpus 41https://github.com/shashwatup9k/bho-res ources 42https://www.openslr.org/64/ 43https://commonvoice.mozilla.org/en/dat asets 44https://github.com/Idlak/Living-Audio-D ataset
most notably in Australia and parts of America and Canada.
The data release for this shared task consists of over 14 hours (split into dev and train) of audio data, together with their transcription in Maltese and translation into English. Participants were also allowed to use additional Maltese data including the text corpus used to train BERTu (Micallef et al., 2022), a Maltese BERT model, the MASRI Data speech recognition data (Hernandez Mena et al., 2020), and any data available at the Maltese Language Resource Server.45
Marathi–Hindi (mar-hin) Marathi (ISO code: mar) is an Indo-Aryan language and is dominantly spoken in the state of Maharashtra in India. It is one of the 22 scheduled languages of India and the official language of Maharashtra and Goa. As per the 2011 Census of India, it has around 83 million speakers which covers 6.86% of the country’s total population.46 Marathi is the third most spoken language in India.
The provided Marathi–Hindi corpus consists of 24.58 hours of Marathi speech data (see Table 6) from the news domain, extracted from News On Air47 and translated into Hindi texts.48 The dataset was manually segmented and translated by Panlingua.49 Additionally, the participants were directed that they may use monolingual Marathi audio data (with transcription) from Common Voice (Ardila et al., 2020),50 as well as the corpus provided by He et al. (2020)51 and the Indian Language Corpora (Abraham et al., 2020).52
Quechua–Spanish (que-spa) Quechua (macrolaguage ISO code: que) is an indigenous language spoken by more than 8 million people in South America. It is mainly spoken in Peru, Ecuador, and Bolivia where the official highresource language is Spanish. It is a highly inflective language based on its suffixes which agglutinate and are found to be similar to other languages
45https://mlrs.research.um.edu.mt/ 46https://censusindia.gov.in/nada/index. php/catalog/42561 47https://newsonair.gov.in 48https://github.com/panlingua/iwslt2023 _mr-hi 49http://panlingua.co.in/ 50https://commonvoice.mozilla.org/en/dat asets 51https://www.openslr.org/64/ 52https://www.cse.iitb.ac.in/˜pjyothi/ind iccorpora/
Language Pairs Train Set Dev Set Test Set Additional Data
Bhojpuri–Hindi bho–hi 19.88 2.07 0.82 Monolingual audio with transcription (ASR) and monolingual text Irish–English ga–eng 9.46 1.03 0.69 IWSLT 2023 test set (with references ) and MT data (monolingual and parallel corpora) Marathi–Hindi mr–hi 15.88 3.66 0.61 Monolingual audio with transcriptions (ASR), IWSLT 2023 test set (with references ) and monolingual text Maltese–English mlt–eng 10 2 2 Monolingual audio with transcriptions (ASR), monolingual text North Levantine–English apc–eng - 2.5 1.85 - Tamasheq–French tmh–fra 17 - - Untranscribed audio, data in other regional languages Quechua–Spanish que–spa 1.60 1.03 1.03 48 hours of monolingual audio with
transcriptions (ASR) and MT data (not transcribed)
Bemba–English bem–eng 167.17 5.89 5.83 28.12 hours of monolingual audio with transcriptions (ASR) and 60 hours of untranscribed audio data.
Table 6: Training, development and test data details (in hours) for the language pairs of the low-resource shared task.
like Finnish. The average number of morphemes per word (synthesis) is about two times larger than in English. English typically has around 1.5 morphemes per word and Quechua has about 3 morphemes per word.
There are two main regional divisions of Quechua known as Quechua I and Quechua II. This data set consists of two main types of Quechua spoken in Ayacucho, Peru (Quechua Chanka ISO: quy) and Cusco, Peru (Quechua Collao ISO: quz) which are both part of Quechua II and, thus, considered a “southern” languages. We label the data set with que - the ISO norm for Quechua II mixtures.
The constrained setting allowed a QuechuaSpanish speech translation dataset along with the additional parallel (text-only) data for machine translation compiled from previous work (Ortega et al., 2020). The audio files for training, validation, and test purposes consisted of excerpts of the Siminchik corpus (Cardenas et al., 2018) that were translated by native Quechua speakers. For the unconstrained setting, participants were directed to another larger data set from the Siminchik corpus which consisted of 48 hours of fully transcribed Quechua audio (monolingual).
Tamasheq–French Tamasheq is a variety of Tuareg, a Berber macro-language spoken by nomadic tribes across North Africa in Algeria, Mali, Niger and Burkina Faso. It accounts for approximately 500,000 native speakers, being mostly spoken in Mali and Niger. This task is about translating spoken Tamasheq into written French. Almost 20 hours of spoken Tamasheq with French translation are freely provided by the organizers. A major challenge is that no Tamasheq transcription is provided, as Tamasheq is a traditionally oral language.
The provided corpus is a collection of radio recordings from Studio Kalangou53 translated to French. It comprises 17 hours of clean speech in Tamasheq, translated into the French language. The organizers also provided a 19-hour version of this corpus, including 2 additional hours of data that was labeled by annotators as potentially noisy. Both versions of this dataset share the same validation and test sets. Boito et al. (2022) provides a thorough description of this dataset.
In addition to the 17 hours of Tamasheq audio data aligned to French translations, and in light of recent work in self-supervised models for speech processing, we also provide participants with unlabeled raw audio data in the Tamasheq language, 53https://www.studiokalangou.org/
as well as in other 4 languages spoken from Niger: French (116 hours), Fulfulde (114 hours), Hausa (105 hours), Tamasheq (234 hours) and Zarma (100 hours). All this data comes from the radio broadcastings of Studio Kalangou and Studio Tamani.54
Note that this language pair is a continuation of last year’s shared task, using the same test set as last year.

heading: 6.2.1 Metrics
text: We use standard lowercase BLEU with no punctuation to automatically score all submissions. Additional analyses for some language pairs are provided below. Were applicable, we also report chrF++ (Popović, 2015).

heading: 6.3 Submissions
text: The Shared Task received a record 69 submissions (for speech translation) from 12 teams for all 8 language pairs. The Shared Task also received 15 submissions for the speech recognition task of transcribing the input audio. They are described in detail below.
ALADAN (Kheder et al., 2024) provided a submission for the apc-eng direction, building upon a cascade of ASR and MT systems. The authors propose a character-level and word-level normalization process to handle the orthographic inconsistency between Arabic Dialects, merging words based on a combination of weighted Levenshtein distance and similarity of embeddings, as computed with a task-specific Word2vec model. Both ASR and MT systems are trained on a combination of public (e.g., IWSLT22 data, GALE speech corpus55 for ASR, and, e.g., the UFAL parallel dataset provided by the organizers, Global Voices, LDC2012T09 for MT) and internal data (a combination of crowd-sourced and web-scrapped resources). For ASR, TDNN-F (Povey et al., 2018) and Zipformer (Yao et al., 2023) models are considered, that are firstly trained on a generic Arabic data, and then fine-tuned on a dialect-specific speech. For MT, both encoder-decoder models and instruction-following LLMs are explored. The primary solution uses both ASR systems combined with the ROVER (Fiscus, 1997) algorithm, with the MT step performed by the fine-tuned
54https://www.studiotamani.org/ 55https://arabicspeech.org/resources
Command-R56 LLM, enhanced by MBR decoding and checkpoint averaging. Contrastive submissions differ in the MT step, with the first one using the final checkpoint of the fine-tuned LLM, and the second one using a Transformer-based NLLB model.
BITSP (Anand et al., 2024) submitted systems for the Bhojpuri to Hindi and Marathi to Hindi tasks. Their approach relied on cascading transcriptions which were piped into translation systems. They used a fine-tuned Whisper model for Marathi-Hindi and an vakyansh-wav2vec model for Bhojpuri-Hindi (Chadha et al., 2022; Gupta et al., 2021). Translation was done using finetuned NLLB for both tasks (NLLB Team et al., 2022). They also looked at using sentenceembeddings generated using the MuRIL (Multilingual Representations for Indian Languages) (Khanuja et al., 2021) model for the Marathi-Hindi task.
HW-TSC (Jiawei et al., 2024) participated in the apc-eng direction with a cascade solution based on the off-the-shelf Whisper (Radford et al., 2022) model for ASR combined with a Transformer-based MT model trained from scratch for Arabic-to-English translation. The MT system (35 encoder layers, 3 decoder layers, with dhidden = 512 and dFFN = 2048) was trained on the mix of publicly available (e.g., OpenSubtities, GlobalVoices, TED) and in-house corpora, both filtered based on sentence embeddings extracted with LaBSE (Feng et al., 2022). No dialectspecific datasets were used for training directly. Instead, an in-domain model was fine-tuned on the validation set to score the training samples using domain features (Wang et al., 2020c), with the highest-scoring subset explored for the final finetuning.
JHU (Robinson et al., 2024) provided systems for all eight language pairs. The main effort of their work revolved around fine-tuning large and publicly available models in three proposed systems, one cascaded and two end-to-end. For the cascaded system, they proposed fine-tuning Whisper transcription (not translation) and then piping that output to a fine-tuned NLLB model. For the end-to-end systems, they fine-tuned for translation directly on SEAMLESS4MT v2 and Whisper translation (not transcription). In addition, they 56https://cohere.com/command
looked at a variety of different training paradigms such as intra-distillation (Xu et al., 2022), joint training, multi-task learning, curriculum learning, and pseudo-translation. The best-performing approach, similar to the broader results of this shared task differed for different language pairs. However, fine-tuned SEAMLESSM4T v2 tends to perform best for source languages on which it was pre-trained. Additionally, while multi-task training helps Whisper fine-tuning, in general cascaded systems with Whisper and NLLB tend to outperform Whisper alone. Finally, intra-distillation was shown to help NLLB fine-tuning.
KIT (Li et al., 2024c) participated in the Maltese-to-English, Bemba-to-English, North Levantine Arabic-to-English tasks in the unconstrained condition. They leveraged pretrained multilingual models by fine-tuning them for the target language pairs, looking at SeamlessM4T, NLLB (NLLB Team et al., 2022), and MMS (Pratap et al., 2024). Due to the large size of the models, they experimented with adapter fine-tuning to reduce the number of trainable parameters using LORA (Hu et al., 2021) and package PEFT (Mangrulkar et al., 2022). They were also able to show that Minimum Bayes Risk is effective in improving speech translation performance by combining systems in all of their language pairs.
SETU-DCU (Zafar et al., 2024) presented systems for two language pairs, Irish–English and
Maltese–English. Both of their submissions, despite lower performance on the Irish (GA) task, were on the unconstrained condition configuration. There were two submissions to the Maltese (MLT) task ranging from 44.7 to 52.6 BLEU and one submission to the GA task at 0.6 BLEU.
The MLT results of 52.6 BLEU were favorable due to SETU-DCU’s primary submission based on a cascaded (ASR to MT) setup of a Whisper (Radford et al., 2022) ASR system used in conjunction with an MT system based on the NLLB (NLLB Team et al., 2022) where both systems were fine-tuned on the Maltese–English data provided. Additionally, their cascaded Contrastive 1 system which used mBart-50 for decoding, scored 44.7 BLEU showing that the use of the NLLB system augmented performance by nearly 8 BLEU points. Further results can be attributed to data preparation such as removing unnecessary data chunks from the dataset, eliminating special characters, and converting the sentences to lowercase along with the following hyper-parameter configuration: batch size of 16, learning rate of 1e-5, 500 warmup steps, 30,000 max steps, per-device eval batch size of 8, generation max length of 225, and intervals of 1,000 steps for saving and evaluating, and 25 steps for logging.
SETU-DCU’s submission for the unconstrained GA task performed poorly compared to other systems submitted. It consisted of a direct speech translation system using the Whisper small model by first resampling data at 16 khz and us-
ing the following hyper-parameter configuration: batch size of 16, learning rate of 1e-5, 500 warmup steps, 1 gradient accumulation steps, generation max length of 225, and intervals of 500 steps for saving and evaluating. The model was fine-tuned over three epochs. Their only submission used Whisper for fine-tuning; however, their claim is that since the data Whisper was trained on did not contain GA at the time of fine-tuning, generation was inconsistent.
UOM-DFKI (Rishu et al., 2024) participated in the Maltese to English shared task using two popular end-to-end pretrained models, Whisper and wav2vec 2.0. They hypothesised that Maltese shares lots of vocabulary with Arabic and Italian and would therefore have good cross-lingual transfer ability due to pretraining data in those models. In addition, they investigated other popular neural models, BERT (Devlin et al., 2019) which they decided against making a formal submission, and mBART (Liu et al., 2020b) which was used as their contrastive submission. Overall, the end-toend system performed much better than the contrastive submission.
UOM (Abela et al., 2024) participated in the constrained task of the Maltese to English translation language pair. Their approach relied on a cascaded system consisting of a pipeline containing: a DeepSpeech 1 ASR system (Hannun et al., 2014), a KenLM model to optimise the transcriptions (Heafield, 2011), and finally an LSTM machine translation model. For their ASR system, they trained using the MASRI dataset and CommonVoice and used a much smaller layer size (64) than normal due to the lack of large amounts of data. These outputs were then used to decode using a 3-gram statistical language model trained on Malti v4.0. The translation system was implemented using fairseq (Ott et al., 2019) comparing both transformer and LSTM architectures, with their best performing system using LSTMs. The authors hypothesize that this was due to the very small amount of data available as a bitext.
UM (Nabhani et al., 2024) competed in the unconstrained task for Maltese-English and North Levantine Arabic-English spoken language translation using a pipeline approach. For the ASR component of their systems, they relied on finetuning XLS-R using 50 hours of Maltese speech data. To correct outputs, they relied on the sta-
tistical toolkit KenLM (Heafield, 2011). Machine translation was then done using a fine-tuned version of the 1.3B parameter NLLB model (NLLB Team et al., 2022). They experimented with a variety of data sources such as CommonVoice, MASRI, and OPUS-100.
YMOSLEM (Moslem, 2024) The Yasmin Moslem team (independent researcher) presented an end-to-end approach for speech translation from spoken Irish to written English. Their models are based on Whisper, utilizing small, medium, and large versions. The primary system employs Whisper-large, which has been fine-tuned using the official training data, supplemented with synthetic audio data and the data augmentation technique involving white noise and voice activity detection.
The synthetic audio data was generated using Azure’s text-to-speech service, applied to the Wikimedia dataset comprising 7,545 text segments. The resulting synthetic audio dataset consists of two parts: one featuring a female voice (OrlaNeural) and the other a male voice (ColmNeural). This resulted in a total of 15,090 utterances, with each text segment used to generate a synthetic speech segment for each voice. The same approach has been applied to 3,966 text segments coming from the SpokenWords dataset.
In addition to the official IWSLT-2023 training dataset and the aforementioned synthetic audio dataset, the Irish portion of the FLEURS dataset, the Bitesize dataset, and the SpokenWords dataset were utilized to fine-tune the WhisperLarge model. Note that the Irish portion of the Spoken Words dataset has been translated into English using the Google Translation API.
QUESPA (Ortega et al., 2024) submitted six total systems consisting of three constrained and three unconstrained systems. Team QUESPA were able to improve the previous year’s results despite the data remaining the same as last year’s ranging from 1.4 to 2.0 BLEU for the constrained task and 11.1 to 19.7 BLEU for the unconstrained one. This year QUESPA provided developmental results on several models that used mel-filter bank (MFB) features extracted using Fairseq (Wang et al., 2020a) were included that show the effect of the s2t transformers model type size ranging from extra-small to large.
QUESPA’s Constrained systems did not vary
much from last year’s systems as far as system architecture is concerned. However, they were able to identify a caveat in the training data set which contains audio wav files of lengths from 1 to 30 seconds while the developmental and test sets were all of 30 seconds in length. Their opinion is that the varied length warranted a severe hyper-parameter empirical search resulting in a Primary system that scored 2.0 BLEU with the following configuration of a Fairseq (Wang et al., 2020a) speech translation model based on melFilter Bank features: extra-small transformer, 6 encoder layers, 3 decoder layers, Adam optimization, 500 epochs and a learning rate of .0002 while using an average of the last 10 checkpoints which outperformed the same model with other hyperparameters from last year. Their Contrastive 1 system, similar to the primary system, introduced a new concept of data augmentation in combination with a medium transfomer (s2t transformer), 12 encoder layers, 6 decoder layers, and 8 attention heads and 200 epochs. More importantly, in Contrastive 1 they introduced audio augmentation via LibRosa57 where the translation was the same but four audio techniques were introduced: Noise (0.009 aggregation), Roll (sr/10), Time(0.4), and Pitch (-5) to create 4-fold sets of the original. Additionally, QUESPA’s Contrastive 1 system removed SpecAugment as an audio augmentation technique. Finally, the Contrastive 2 system from Team QUESPA were identical to the primary system with the change of epochs to 400 and model type to a medium-size (s2t transformer).
QUESPA’s Unconstrained systems were a novel introduction for the QUE–SPA task and outperformed last year’s best systems. Their primary system introduced the SpeechT5 (Ao et al., 2022)
57https://librosa.org/
ASR PLM which consists of 12 Transformer encoder blocks and 6 Transformer decoder blocks, with a model dimension of 768, an internal dimension (FFN) of 3,072, and 12 attention heads. It used normalized training text from the LibriSpeech language model as unlabeled data, which consisted of 400 million sentences and fine-tuned on the competition data while optimizing with Adam and a learning rate maximum of 0.0002. Fine-tuning was performed using the SpeechT5 fine-tuning recipe58 for Speech-Translation with the same hyperparameter settings. Additionally, their primary system used a data augmentation technique (noise, distortion, duplication)59 (Ma, 2019) for total of 120h: 60h original + 60h synthetic data scoring 16.0 BLEU, higher than previous year’s results. For Contrastive 1, QUESPA introduced a combination of more data by manually translating Quechua to Spanish 55 hours of the total set along with an additional 19 minutes of Guarani and 29 minutes of Bribiri from the AmericasNLP60 shared task. On top of that, they applied two data augmentation techniques: (1) nlpaug (Ma, 2019) and (2) DA-TTS (Zevallos et al., 2022), which involves generating synthetic text and audio using a de-lexicalization algorithm and a TTS system for the source language (Quechua). These two data augmentation techniques generated 62 hours and 50 hours respectively. Altogether, they used a total of 167h and 48 min: 55h (new dataset) + 48 min (ANLP dataset) + 62h nlpaug + 50h DA-TTS. The Contrastive 1 system was QUESPA’s best system scoring 19.7 BLEU. The Contrastive 2 system was also newly
58https://github.com/microsoft/SpeechT5/ tree/main/SpeechT5 59https://github.com/makcedward/nlpaug 60https://turing.iimas.unam.mx/americasn lp/2022_st.html
introduced with the use of Whisper medium-size, multi-lingual model for ASR in a cascade approach basically replacing last year’s “fleurs” ASR system. The MT system was identical to the one they used last year called FloresMT (Ortega et al., 2023). QUESPA’s Contrastive 2 system resulted in a score of 11.1 BLEU.

heading: 6.4 Results
text: Table 8 summarizes the winning submissions for each language pair. Detailed results for all teams’ systems and settings are available in Appendix B.5.
Of the 8 language pairs, 5 different teams had the top performing system on at least one language pair. This shows how competitive the shared task was, and that a multitude of approaches are helpful for low-resource speech translation. Additionally, no team was able to beat the baseline on the Tamasheq-French direction (which corresponds to last year’s best system). This suggests that there continues to be lots of room for improvement and that this remains an active area of research.
Compared to previous iterations of the shared task, many of the language pairs had marked improvements with large gains in the official automatic metrics. For example, BLEU scores for Maltese-English and Marathi-Hindi are in the 40s and 50s. Furthermore, for North Levantine Arabic-English, Bemba-English, and BhojpuriHindi are above 20 BLEU points. Even for Quechua-Spanish, the least resourced language pair, the best submission’s BLEU score is almost 20 points.
This marks stark improvements from last year’s shared task systems for some language pairs. In Marathi-Hindi, the best system in 2023 achieved a BLEU score of 39.7, with this year’s best system improving by more than 7 BLEU points. Similarly, the improvements in the quality and quantity of the Maltese data lead to a more than 50 BLEU points improvement compared to last year. For Irish and Tamasheq, the performance increases are more modest, about 1 to 2 BLEU points in each, compared to the 2023 Shared Task.
For the language pairs included for the first time in the shared task, we find that Bemba-English and Bhojpuri-Hindi end up with decent systems, a result of high-quality data availability: for instance, Bemba-English has an order of magnitude more training data –167h– than any other language pair
in our shared task); and Bhojpuri is the second most “high-resourced” language in our set, with almost 22 hours of speech translation data.
Within the systems submitted to the initial run of the North Levantine Arabic-English language pair, all of the primary submissions are based on a pipeline approach exploring ASR and MT, with a single submission combining E2E and cascaded systems. Since the popular NLLB model explored by several submissions supports an input/output combination of dialectical Arabic/English and a large-scale, parallel textual dataset of Levantine Arabic was provided, the participating teams mainly struggled with the ASR component. The winning submission by ALADAN, which outperformed a second-place team by over 8 BLEU points, uses an internal dataset of Levantine speech to boost the performance of their ASR component. While the data used for finetuning the MT system is comparable between the submissions, ALADAN explored a much larger, prompt-driven LLM compared to the 600M/1.3B NLLB variants explored by other teams.
We note that almost all submissions followed the unconstrained setting – a clear indication that pre-trained multilingual systems seem to be the best option for building ST for low-resource languages, at least under the current data, architectural, and compute constraints.

heading: 7 Automatic Dubbing

heading: 7.1 Challenge and Test Sets
text: Dubbing is a form of speech translation where the user can not only hear the translated speech, but also can often see the original speaker. This adds numerous challenges and constraints, including isochrony (does the new translation respect the timing of the original speech), phonetic synchrony or lip sync (is the new speech compatible with the mouth movements of the original speaker, if visible), kinesic synchrony (is the new speech consistent with visible body movements of the original speakers), and others (Mayoral et al., 1988; Chiaro, 2009; Chaume, 2020; Brannon et al., 2023).
For English→Chinese, we use the ITV test set from subtitle task. We manually selected 10min sections from each of clip 15, 16, 18, 19, and 21. The 10min sections were manually selected with several goals:
1. Speech is fairly clear
2. A mix of on-/off-screen dialogues
3. A diverse set of genders and accents
4. Avoid excessive profanity
5. Avoid opening/closing credits
German→English followed the same setup as the submissions from last year (Chronopoulou et al., 2023; Pal et al., 2023; Rao et al., 2023)

heading: 7.2 Submissions
text: This task received a total of four English→Chinese submissions (see Table 9): one end-to-end dubbing submission and three participants in the offline speech translation task (speech to text) scored our challenge set (set5). For the offline submissions, we utilized the provided translations to generate dubs.
We also received one submission (Li et al., 2024b) for German→English. We chose to focus on English→Chinese for evaluation due to the availability of the offline speech systems to compare against, which should represent strong speech translation models (but not dubbing specific models).
The process of generating dubs from text translations involved several steps. First, due to the absence of source language subtitles, we downloaded subtitles from an open-source website and manually time align the five clips. Each time aligned sentence was then split at commas and full stops to create manageable segments for processing, while keeping a track of original sentences and time-stamps.
Similarly, the translations from the three submissions were also split at commas and full stops. We used Vecalign (Thompson and Koehn, 2019, 2020) a tool for sequence alignment, in conjunction with LASER-2 embeddings (Heffernan et al., 2022), to align the source language with the target language. This ensured that the meaning and context of the translated text matched the original as closely as possible. Timestamps were then projected from the source to the target language, providing a temporal map for the dubbing process.
For each sentence, we employed Amazon Polly, a text-to-speech service, to generate the corresponding speech. We also used the duration of the source speech segment as a constraint to generate target speech with Polly. Polly allowed this by adding a flag with max durations, where the
generated speech cannot go beyond maximum duration. We used Zhiyu standard voice as that allowed use of this flag via SSML wrapper. Adding duration constraint essentially ensured that the target speech did not exceed the length of the source speech. Typically, the target speech was shorter than the source speech, so we filled the remaining portion with silences to maintain synchronization.
We synchronized the start time of the target speech with the source speech using the previously obtained timestamps to ensure that the dialogue matched the visual cues accurately. Finally, we concatenated the target speech segments to form the complete clip.

heading: 7.3 Metrics and Results
text: We report speech overlap (between the original audio and the dubbed audio) in Table 10. For reference, in a large corpus of professionally dubbed media, human speech overlap between original and dubbed speech is about 0.658 (mean) and 0.731 (median) (Brannon et al., 2023). The dubbing submission HWTSC-Dubbing is similar to the human statistics, while the cascaded systems generated in part by the task organizers perform substantially worse.
We report PEAVS (Perceptual Evaluation of Audio-Visual Synchrony) score (Goncalves et al., 2024), an automatic metric with a 5-point scale that evaluates the quality of audio-visual synchronization, in Table 12. PEAVS is the only AV sync evaluation metric that is grounded in human judgements as it is trained on a large Audio-Visual synchrony benchmark for “in-the-wild” videos. In our case, we use PEAVS for evaluating the quality of synchrony in the generated dubs. As expected for a system optimized with speech timing in mind, HWTSC-Dubbing performs best here.
Table 12 also reports BLASER 2.0-QE scores. BLASER 2.0-QE is a reference-free modalityagnostic automatic metric for speech translation quality (Seamless Communication et al., 2023). It only supports short-form speech, so we segment the full speech into sentences as mentioned in Section 7.2 and report average scores. Surprisingly, the dubbing submission performs the best at this metric, even though it is optimized for both translation quality and timing. It is worth noting that the segments being evaluated are quite short, often much shorter than typical sentences in written text, and lack of domain context has been shown
to be problematic in machine translation metrics even for normal length sentences (Läubli et al., 2018; Toral et al., 2018; Vernikos et al., 2022). BLASER 2.0-QE is not trained on dubbing data, so there is likely degradation due to domain mismatch (Zouhar et al., 2024).
We report two measures of lip sync, both from Prajwal et al. (2020): LSE-D (lip-sync error distance) and LSE-C (Lip Sync Error - Confidence) (see Table 13). LSE-D measures the accuracy of audio-visual synchronization by identifying the offset with the smallest distance between audio and video features. LSE-C measures the confidence in this synchronization by comparing the best match’s distance to those of adjacent offsets, with higher values indicating greater confidence. In essence, LSE-D tells us how well the audio and video are synchronized, while LSE-C tells us how sure the model is about that synchronization. HWTSC-Dubbing performs the best at LSE-D on average, although one strange result is that the metric prefers HWTSC-Dubbing to the original audio in two of the test sets, which does not make sense. Another surprise is that CMUOffline slightly outperforms HWTSC-Dubbing on the LSE-C metric.
We also conduct human judgements to evaluate translation quality and naturalness. We evaluate the first 20 sentences of each clip based on the rubric (Table 11), and report the average score for each submission in Table 12. In general, the dubbing system produces more natural speech but sometimes less accurate translation than the offline systems. The offline systems oftentimes have to speed up the speech synthesis to match the original duration of a sentence, leading to hard-torecognize speeches.

heading: 8 Indic Languages Track
text: In the realm of spoken language processing, speech-to-text translation (ST) holds a crucial role at the intersection of natural language processing. The primary aim of ST is to convert spoken language from one linguistic context into written text
in another language. This typically involves using Automatic Speech Recognition (ASR) to convert speech in the source language into text, followed by Machine Translation (MT) to translate the source language text into the target language. ST is a multimodal task that takes speech input and produces output in text format. Furthermore, it is inherently multilingual, taking speech input in one language and generating text output in another. Traditionally, human language translators proficient in both the source and target languages have handled this task. However, the scarcity of translators fluent in multiple languages has created a pressing need for a dedicated model tailored to excel in the unique realm of ST tasks across diverse languages. Recent advancements in ST have predominantly focused on high-resource languages, leaving a significant gap for low-resource languages that face a substantial catch-up journey. The attention imbalance is primarily due to the scarcity of data for low-resource languages, as most deep-learning models depend on data abundance. Acquiring such data for low-resource languages poses a formidable challenge.
While a considerable body of research is dedicated to ST across diverse language families, a noticeable gap exists in investigating this domain concerning low-resource Indian languages. Currently, there are no datasets specifically designed for the ST task in Indian languages, covering both the Indo-Aryan and the Dravidian language families. This research aims to create either an End-toEnd (E2E) or a Cascaded ST model
This Indic track aims to establish an ST translation model that spans a diverse array of dialects and low-resource languages originating from the Indo-Aryan and Dravidian language families in India. Given that a significant portion of the data is sourced from very low-resource languages, these languages remain largely unexplored in the realm of speech translation. Compounding this challenge is the fact that many of the target languages are distantly related to English. Consequently, we anticipate that relying solely on pre-trained
models may encounter numerous obstacles. The dataset provided will serve as the inaugural benchmark and gold standard dataset, encompassing all Indian languages. We aspire for participants to develop systems capable of real-world deployment in the future.

heading: 8.1 Challenge
text: The Indic shared task consists of ST for three language pairs from English (en) to Hindi (hi), Tamil (ta), and Bengali (bn). The ST data for all these three language pairs is derived from the IndicTEDST dataset (Sethiya et al., 2024). The submissions are allowed for both the constrained and the unconstrained cases. The constrained case involves only the data provided in the task. The unconstrained case can utilize either the data provided in the challenge or any external data, along with any pre-trained models. The submissions are also allowed for the cascade and end-to-end models for all the language pairs. Thus, the task accepts the following cases for all three language pairs (en-hi, en-ta, and en-bn):
• End-to-end + Constrained
• End-to-end + Unconstrained
• Cascade + Constrained
• Cascade + Unconstrained

heading: 8.2 Data and Metrics
text: The ST task data for the Indic track encompasses three Indian languages representing diverse language families. The languages included in this shared task are Hindi (hi), Bengali (bn), and Tamil (ta), originating from the Indo-Aryan and Dravidian language families. The dataset includes speech and text (transcriptions) in English (source language) and text (translations) in Hindi, Bengali, and Tamil (target languages).
The data for this Indic track comprises a ST corpus that includes 3 low-resource Indian languages. The data is curated from the TED talks with Indic translations, usually a talk spans from 3 minutes to 15 minutes. A segmentation of the audio files in the form of YAML is provided with the data. Table 15 illustrates the consistency maintained across all corpora, with an equal number of lines in their .en, .lang, and .yaml files. However, due to inherent linguistic differences, the number of tokens in the .en and .lang files varies. The count of audio files
corresponds to the number of distinct talks, each delivered by an individual speaker. Additionally, the speech hours indicate the cumulative speech duration in a given language. Each parameter is meticulously categorized into test, train, and valid subsets, establishing a comprehensive and structured dataset.
English-Hindi: Hindi is the third most spoken language in the world, with 615 million speakers. It belongs to the Indo-Aryan language family, mainly spoken in India. It is also the official language of India, written in devanagiri script. The data contains English speech, English text (transcripts), and Hindi text (translations). The speech in English language is 103.5 hours and the text in Hindi language is 37K lines.
English-Bengali: Bengali is the 7th most spoken language in the world, with 228 million speakers. It belongs to the Indo-Aryan language family, spoken in the Bengal region of South Asia. It is also the official language of Bangladesh, written in Bengali-Assamese script. The data contains English speech, English texts (transcripts), and Bengali texts (translations). The speech in English language is 13.59 hours and the text in Bengali language is 6.9K lines.
English-Tamil: Tamil is one of the classical languages of India, spoken by 90.8 million speakers. It belongs to dravidian language family, spoken by the tamil people of South Asia. It is the official language of Tamil Nadu state of India, written in Brahmi script. The data contains English speech, English texts (transcripts), and Tamil texts (translations). The speech in English language is 22.01 hours and the text in Tamil language is 8K lines.
Metrics: Case-sensitive detokenized BLEU using sacreBLEU (Post, 2018) is used to report the performance of all the submissions.

heading: 8.3 Submissions
text: There were four teams participating in this inaugural task: Research team from National Institute of Information and Communications Technology of Japan (NICT) , the Voice Intelligence Team of Samsung (SRI-B), the Huawei Translation Service Center (HW-TSC), and a team from National Institute of Technology Kurukshetra, India (NITKKR). The participants submitted their result under various constraints, including end-to-end constrained, unconstrained, cascaded end-to-end, and unconstrained approaches. Below, we provide an overview of each team’s approach and their results.
NICT: Their submission included cascaded and end-to-end approach in unconstrained setting for all the language pairs. The cascaded system involves fine-tuning the Whisper model for ASR and fine-tuning the IndicTrans2 model for MT. This dual fine-tuning aimed to address the format mismatch between spoken and written language. For the end-to-end syatem, the IndicTrans2 model is used to generate pseudo translation data, which replaced the gold transcription data for fine-tuning the Whisper model. This strategy aimed to distill knowledge from a stronger translation model and ensure consistent formatting. In stage 1, Whisper is fine-tuned using English transcription and Indic language translation. Stage 2 involves generation of pseudo translations for all English transcriptions, and fine-tuning Whisper using English audio and the pseudo translations. During inference, the fine-tuned Whisper model performed direct end-to-end speech translation.
HW-TSC: The submission included implementation of cascaded approach in the unconstrained setting. It involves Whisper-large-v3 model for Automatic Speech Recognition (ASR) and a Transformer model for Machine Translation (MT). For MT, strategies like LaBSE for parallel corpus filtering, data diversification using multiple model predictions, forward and back translation for data augmentation, domain fine-tuning with scored data selection, and regularized dropout for enhanced training efficiency are used. The base architecture is from FAIRSEQ toolkit (Wang et al., 2020b) with hyperparameters of 2048 as batch size, learning rate of 5e-4, label-smoothingcross-entropy loss with label smoothing of 0.1,
4000 warmup steps, and Adam optimizer settings (ß1 = 0.9, ß2 = 0.98). During inference, a beam size of 4 and length penalties of 1.0 is applied to optimize translation outputs.
SRI-B: The submission included end-to-end approach in both constrained and unconstrained setting. In the constrained setting the base architecture used is from FAIRSEQ toolkit (Wang et al., 2020b). Pre-processing involves the extraction of 80 channel log mel-filter bank features with a window size of 25ms and SpecAugment for data augmentation. The s2t conformer among fairseq’s built-in architectures for speech-to-text translation is used. It consist of 16 encoder layers and 6 decoder layers with label-smoothed cross-entropy loss and the Adam optimizer with a learning rate of 2e-3 to train the models. Under the unconstrained setting, the method involves using the pre-trained SeamlessM4T v2 from Meta, a multi-lingual end-to-end model designed for various languages. The pre-trained multi-lingual model is used to directly generate text in Indic languages directly from English for evaluation.
NITKKR: The submission adopts cascaded approach in unconstrained setting to solve the task. It begins with audio preprocessing and transcription, utilizing ResembleAI for noise reduction, distortion restoration, and speech bandwidth enhancement. The processed audio is then fed into OpenAI’s Whisper model for real-time ASR. Subsequently, MT models are applied: Helsinki-NLP’s OPUS-MT for translating English to Hindi, and Facebook’s Multilingual BART (MBART) for both English to Tamil and English to Bengali translations.

heading: 8.4 Results
text: Scores on the test set of all submissions are calculated using automatic metrics and the respective settings are presented in Table 16. In the following section, we discuss results from each direction of languages.

heading: 8.4.1 En-Hi
text: Unconstrained Setting: In the E2E approach, NICT achieved a BLEU score of 33.02, significantly outperforming SRI-B, which scored 21.63. This superior performance by NICT can be attributed to their robust use of pseudo translation data aimed to distill knowledge from a stronger

heading: E2E NICT 33.02SRI-B 21.63

heading: E2E NICT 10.79SRI-B 18.13

heading: E2E NICT 13.46SRI-B 11.93
text: translation model to ensure consistent formatting. In the cascaded approach, NICT again led with a remarkable 60.54 BLEU score, significantly higher than HW-TSC at 47.14 and NITKKR at 19.77. The cascaded approach by NICT utilized the strengths of pretraining the ASR and MT model to address the format mismatch problem which leads to maximizing the performance.
Constrained: In the E2E approach, there was one submission by SRI-B, which achieved a BLEU score of 29.76.

heading: 8.4.2 En-Bn
text: Unconstrained: SRI-B with a BLEU score of 18.13 beats NICT which scored 10.79 when implementing the E2E approach. In the cascaded approach, NICT scored the highest with 52.63 BLEU, compared to HW-TSC at 35.04 and NITKKR at 4.46. The same strategy from En-Hi allowed NICT to excel in this category, demonstrating the effectiveness of their cascaded approach.
Constrained: For the E2E approach, SRI-B scored a BLEU of 2 demonstrating the challenges of the constrained setting in this language pair.

heading: 8.4.3 En-Ta
text: Unconstrained: NICT led with a BLEU score of 13.46, while SRI-B scored 11.93 for the models using E2E approach. NICT’s consistent use of Whisper for ASR and their robust translation models contributed to their leading position. For teams using the cascaded approach, NICT
again achieved the highest BLEU score of 39.84, followed by HW-TSC at 30.79 and NITKKR at 11.76. The result could be explained due to the method of addressing the format mismatch problem by NICT already mentioned above.
Constrained: In this setting there is one submission using the E2E approach, by SRI-B. They achieve a score of 0.81, which shows the limitations on this setting and language pair. The low score could be explained due to limited data and the morphologically complex structure of the Tamil language.

heading: 8.5 Conclusion
text: This is the first time that a speech-to-translation task is presented for the Indic track as one of the IWSLT tasks. The results presented in the work establish an important benchmark for the end-toend as well as cascade models for both the constrained and unconstrained conditions. This work highlights a major performance gap between the end-to-end and the cascade models. Also, a noteworthy gap is seen in the performance with the unconstrained data and pretrained models are used. We plan to include more data and more Indic languages in the next edition.

heading: Acknowledgements
text: The FBK team (Roldano Cattoni, Mauro Cettolo, Matteo Negri and Sara Papi) is co-funded by the European Union under the project AI4Culture: An AI platform for the cultural heritage data space (Action number 101100683).
Atul Kr. Ojha and John P. McCrae would like to thank Science Foundation Ireland (SFI) under Grant Number SFI/12/RC/2289 P2 Insight 2 and thanks to RTÉ/TG4 for sharing the Irish speech data. We would also like to thank Panlingua Language Processing LLP for providing the MarathiHindi, and Bhojpuri-Hindi speech translation data and for their support.
The work by Tsz Kin Lam and Barry Haddow was funded by UK Research and Innovation (UKRI) under the UK government’s Horizon Europe funding guarantee (grant number 10039436: UTTER).
The work by Mateusz Krubiński, Petr Zemánek, Adam Pospı́šil, and Pavel Pecina was funded by the European Commission via its H2020 Program (contract no. 870930: WELCOME).
The work by University of Malta was supported through H2020 EU Funded LT-Bridge Project (GA 952194) and DFKI for access to the Virtual Laboratory.
Brian Thompson’s contributions to this work were conducted outside of, and are unrelated to, his employment at Amazon.
Ondřej Bojar would like to acknowledge the support from the 19-26934X (NEUREM3) grant of the Czech Science Foundation. The work of Peter Polák has been supported by the Ministry of Education, Youth and Sports of the Czech Republic, Project No. LM2023062 (LINDAT/CLARIAH-CZ). Dávid Javorský has been supported by the grant 272323 of the Grant Agency of Charles University.

heading: A Human Evaluation
text: Human evaluation included MQM for the English-to-Japanese simultaneous speech translation task (A.1), as well as direct assessment for offline, simultaneous, and subtitling tasks (A.2).
A.1 MQM-based Human Evaluation for the English-to-Japanese Simultaneous Task For the English-to-Japanese Simultaneous Translation Task, we conducted a human evaluation using a variant of Multidimensional Quality Metrics (MQM; Lommel et al., 2014). MQM has been used in recent MT evaluation studies (Freitag et al., 2021a) and WMT Metrics shared task (Freitag et al., 2021b). For the evaluation of Japanese translations, we used JTF Translation Quality Evaluation Guidelines (JTF, 2018), distributed by Japan Translation Federation (JTF). The guidelines are based on MQM but include some modifications in consideration of the property of the Japanese language.
We hired a Japanese-native professional interpreter as the evaluator. The evaluator checked translation hypotheses along with their source speech transcripts and chose the corresponding error category and severity for each translation hypothesis on a spreadsheet. Here, we asked the evaluator to focus only on Accuracy and Fluency errors, because other types of errors in Terminology, Style, and Locale convention would not be so serious in the evaluation of simultaneous translation. Finally, we calculated the cumulative error score for each system based on the error weighting presented by Freitag et al. (2021a), where Critical and Major errors have the same level of error scores. The results are shown in Table 17.
A.2 Direct Assessment For the offline translation track (Section 2), simultaneous translation track (Section 3), and subtitling track (Section 4), we conducted a human evaluation of primary submissions based on a random selection of 1000 segments from each test set. Human graders were asked for direct assessment (DA) (Graham et al., 2013; Cettolo et al., 2017; Akhbardeh et al., 2021), expressed through scores between 0 and 100.
A.2.1 Automatic Segmentation In the case of offline and subtitling tracks, we collected segment-level annotations based on the resegmentated test data (see Section 2). Because we did not want issues from the segmentation to influence scores negatively, we followed Sperber et al. (2024) and provided translators not only with the source sentence and system translation, but also with the system translation of the previous and following segments. Annotators were then instructed as follows: “Sentence boundary errors are expected and should not be factored in when judging translation quality. This is when the translation appears to be missing or adding extra words but the source was segmented at a different place. To this end, we have included the translations for the previous and next sentences also. If the source and translation are only different because of sentence boundary issues, do not let this affect your scoring judgement. Example for a clear case for a good translation suffering only from sentence boundary issues that should not result in a poor score: Source: *you’ll see that there’s actually* a sign near the road. Translation: ein Schild neben der Straße gibt.
Previous sentence: Ich bin mir sicher, dass Sie nicht wissen, dass, wenn Sie weiter weitergehen, *Sie sehen – (Gelächter) – dass es tatsächlich* Next sentence: . . . .” No video or audio context was provided. Segments were shuffled and randomly assigned to annotators to avoid bias related to the presentation order. Annotation was conducted by professional translators fluent in the source language and native in the target language.
A.2.2 Subtitling Constraints The subtitling task (Section 4) includes cases where systems compress translations in order to match subtitling constraints, e.g. filtering out non-relevant information present in the source. This is desired in subtitling and should therefore not be penalized in human evaluation. To this end, we provided annotators with the following instructions: “When judging the translations, please consider that these are subtitles which are compressed translations of the original speech, not the translations of the subtitles in the source language. Thus, there may be significant differences in how the source and the target sentences are formulated. Subtitles are created independently for each language with the goal of good readability during the short time period when they are displayed on screen. Readability in terms of number of characters per second may differ between the source (English) and target (German). Please take this into account. The translation should convey the same meaning as the source sentence but may omit information that is not very important for getting the main message of the sentence across. It is OK if the sentence is shortened this way in order to fulfil the readability constraints.”
A.2.3 Computing rankings System rankings are produced from the average DA scores computed from the average human assessment scores according to each individual annotator’s mean and standard deviation, similarly to Akhbardeh et al. (2021). Ranks are established according to Wilcoxon rank-sum statistical significance test with p < 0.05. The below tables show the DA scores and rankings. Note that the guidelines are different for offline, simultaneous, and subtitling tasks. This makes results not directly comparable across tasks, and we consequently only present within-task rankings here. Within each of the tasks (only the offline and subtitling English-to-German have more domains), all the outputs were assessed in one annotation run, distributing the scoring items randomly to annotators across domains, with all annotators most likely seeing all the domains. This allows us to treat the DA scores across domains in a given task as comparable, so we present them in the same table.
Appendix B. Automatic Evaluation Results and Details
B.1 Offline SLT
• Systems are ordered according to the COMET score (denoted by COMET, the third column).
• The “Joint” column is computed by averaging the scores of the 4 test sets, aka macro-averaging.
• The “D” column indicates the data condition in which each submitted run was trained, namely: Constrained (C), Constrained+LLM (C+), Unconstrained (U).
• All systems are based on cascade architecture.
Translation Guidelines
In this task, we aim to obtain high quality German translations of the English transcripts. The transcripts (inside the “transcripts.txt” file) contain conversations between friends talking about a daily topic, e.g. hobbies and vacation. There are 76 conversations (recordings) in total. In each conversation, there are only two speakers, but the same pair of speakers may appear in another set(s) of conversations, see the list below. The content of each recording is independent of each other, so they could be translated independently. For each source sentence (line) to be translated, we provide metadata, such as the recording id, speaker id, the audio file and the utterance number. The utterance number indicates its order in the conversation. It begins from 0 (which is not included in the transcripts required for translation) and stands for the beginning of the conversation. In general, most recordings start from an utterance number of 15.
The general translation guidelines are: ● All translations should be “from scratch”, without post-editing from Machine Translation.
We can detect post editing so will reject translations that are post-edited. ● Translators should preserve the line structure of the source file. By this we mean that they
should not add or remove line-breaks , and each line is English should correspond to a line of German. Note that each line of the source file corresponds to one audio file. ● We need the translations to be returned in the same format. If you prefer to receive the text in a different format, then please let us know as we may be able to accommodate it. ● Translators should avoid inserting parenthetical explanations into the translated text and obviously avoid losing any pieces of information from the source text. We will check a sample of the translations for quality, and we will check the entire set for evidence of post-editing.
Since it is a conversation between friends, please pay attention to the below: ● You might need to use the context before and/or after the utterance to translate. ● [Important] There are disfluencies in the transcripts, including but not limited to, hesitation,
repetitions, and correction. We expect to have fluent and faithful translations. These disfluencies in the transcripts might be helpful for your translation, but they are not required as long as the meaning is clear. Please avoid word-by-word translation of them.
a. In general, please focus on the core meaning in the translation. You might rephrase or remove the redundant parts in the transcripts if necessary, e.g., repetitions. b. For Hesitation, some examples are below, please do NOT include them in the translation. We keep them on the transcripts as it might help signal a “pause” in the utterance.
Examples of disfluencies: ● Hesitation:
a. List of possible tokens: {"ACH", "AH", "EEE", "EH", "ER", "EW", "HA", "HEE","HM", "HUH", "MM", "OOF", "UH", "UM", "HMM"} b. Example: "YEAH I KNOW UM WAIT WHAT WAS I GONNA SAY UM SO DO YOU WANNA ASK THE QUESTION NOW"?
● Repetitions:
a. “WELL ACTUALLY ARE THEY LIKE ALL THESE ALL THESE ALL THESE DUMPLINGS OF EASTERN EUROPEAN ORIGIN”
A note on the recording_id
There are 76 conversations / recordings in total, but the same pair of speakers may show up in another conversation(s) (122 speakers in total). In spite of the same pair of speakers, the contents in each of these conversations are also independent of each other. These conversations have their id extended by “_PX” where “X” is a number. Below is the list of recodings that have “_PX” in their names:
● EDACC-C23_P1, EDACC-C23_P2 ● EDACC-C32_P1, EDACC-C32_P2 ● EDACC-C33_P1 ● EDACC-C40_P1, EDACC-C40_P2, EDACC-C40_P3 ● EDACC-C43_P1 ● EDACC-C46_P1, EDACC-C46_P2 ● EDACC-C05_P0, EDACC-C05_P1 ● EDACC-C29_P1, EDACC-C29_P2 ● EDACC-C31_P1, EDACC-C31_P2 ● EDACC-C38_P1, EDACC-C38_P2 ● EDACC-C35_P1, EDACC-C35_P2, EDACC-C35_P3 ● EDACC-C36_P1, EDACC-C36_P2 ● EDACC-C37_P1, EDACC-C37_P2 ● EDACC-C47_P1, EDACC-C47_P2 ● EDACC-C57_P1, EDACC-C57_P2
B.2 Simultaneous SLT
B.3 Automatic Subtitling
B.4 Speech-to-Speech Translation

heading: Future Work
text: We aspire for participants to develop systems capable of real-world deployment in the future