Robustness of Multi-Source MT to Transcription Errors
Year: 2023

Abstract:
Automatic speech translation is sensitive to speech recognition errors, but in a multilingual scenario, the same content may be available in various languages via simultaneous interpreting, dubbing or subtitling. In this paper, we hypothesize that leveraging multiple sources will improve translation quality if the sources complement one another in terms of correct information they contain. To this end, we first show that on a 10-hour ESIC corpus, the ASR errors in the original English speech and its simultaneous interpreting into German and Czech are mutually independent. We then use two sources, English and German, in a multi-source setting for translation into Czech to establish its robustness to ASR errors. Furthermore, we observe this robustness when translating both noisy sources together in a simultaneous translation setting. Our results show that multisource neural machine translation has the potential to be useful in a real-time simultaneous translation setting, thereby motivating further investigation in this area.

heading: 1 Introduction
text: Speech translation (ST) suffers from automatic speech recognition (ASR) errors, especially in challenging conditions such as non-native language speakers, background noise, named entities and specialized vocabulary usage (Macháček et al., 2019; Gaido et al., 2021, 2022; Anastasopoulos et al., 2022). ASR errors negatively impact translation quality, via the compounding of speech recognition and translation errors (Ruiz et al., 2017; Sperber and Paulik, 2020), thereby limiting the application of automatic speech translation in realistic settings. Fortunately, there are multilingual settings where a source is simultaneously or consecutively interpreted into multiple languages. Many documents are also dubbed or subtitled in offline mode. This simultaneous interpretation takes place either via human interpreters, in the form of dubbing or
subtitling. In a situation where the same sentence is available in multiple languages, multi-source machine translation (MT) significantly improves translation quality, especially when the two sources used separately do not yield high quality translations (Dabre et al., 2018; Zoph and Knight, 2016; Nishimura et al., 2018).
Although not yet clearly verified, multi-source MT could be useful in settings where the sources complement each other. In other words, challenges in translation posed by using each source should be independent of one another. Given that ASR is noisy and that multiple sources can help overcome limitations of individual sources, this paper asks the following question: “Can multi-source MT be leveraged for speech translation in a multilingual setting where an original source transcription and its simultaneously interpreted transcription are available?” We decompose this question into three parts where we hypothesize that (a) ASR errors in the original and interpreted transcripts are independent, which makes them complementary, (b) multi-source MT is robust to transcription errors present in individual sources, and (c) the robustness of multi-source MT continues to hold in a simultaneous translation setting. We address each question in Sections 3, 4, and 5, respectively.
To prove our hypotheses, firstly, we verify on the Europarl Simultaneous Interpreting Corpus (ESIC, Macháček et al., 2021) that original speech ASR and interpreted speech ASR are indeed complementary in terms of errors. Secondly, we simulate transcription errors in a full sentence multi-source MT setting for English and German to Czech translation. We clearly show that when both sources are noisy, using them together leads to significant improvements when compared to using them individually, in contrast to drops in quality when at least one of the sources is clean. For example, on ESIC test set with 15% WER noise in English source and 10% WER noise in German source,
3707
multi-sourcing performs 0.9 BLEU score higher than English source. Finally, we use both sources in a simultaneous translation setting and show that multi-source MT continues to be robust to transcription errors.
Our findings show that multi-source MT has strong potential in a simultaneous translation setting where multiple sources are available via ASR or interpreted ASR. We note that our current analysis is limited to the case where the multiple sources are aligned and hence available at the same time. This is a setting of, e.g., dubbed and subtitled videos where we would want to consider additional target languages. In simultaneous settings where one source is available with a delay, the synchronization of the sources would be a considerable problem, which we leave for future research.

heading: 2 Related Work
text: This paper mainly focuses on ASR errors, multilingual multi-source translation and simultaneous translation.
ASR errors often propagate to MT in cascaded ST systems and in a real time translation setting where ASR systems are used. It is a major issue that affects translation quality. Martucci et al. (2021) propose a method to tune the MT on the training data with artificial noise that mimics ASR errors, via a unigram “lexical noise model” learned on automatic–gold transcript pairs. Other authors propose similar methods for training (Sperber et al., 2017; Di Gangi et al., 2019; Xue et al., 2020; Serai et al., 2022). However, in this work, rather than mimicking ASR noise during training, we complement a noisy source with another whose ASR errors are provably independent of the first. Specifically, we use the lexical noising of Martucci et al. (2021) to simulate noise in multiple source languages and show the robustness of MT models, especially in a multi-source setting.
Multilingualism (Dabre et al., 2020) has been shown to improve translation quality in a variety of situations. In particular, multi-source machine translation has high potential for improving translation quality, but has been relatively underexplored. In the context of multi-source text-to-text translation, Zoph and Knight (2016) and Dabre et al. (2018) showed that leveraging the same sentence in different languages improves translation quality as the two sources are expected to com-
plement hard to translate phenomena in the other source. Although this approach requires multiparallel sentence-aligned data, the missing sources can be obtained by MT (Nishimura et al., 2018) or via simultaneous interpretation. Rather than training a multi-source model, Firat et al. (2016) propose the “late averaging” which needs multilingual models trained on pairwise bilingual data, which we also focus on when evaluating multi-source models. Late averaging is akin to ensembling via logits averaging, but with source sentences in different languages. These works do not consider transcription errors which are ubiquitous in speech translation, an aspect this paper focuses on.
Multi-sourcing in simultaneous translation settings has not been extensively explored. Dabre et al. (2021) have explored simultaneous multipivot translation where a source is translated into a target language via multiple pivot languages, where the pivot languages are translated using multi-source translation. Unlike them, we consider only one pivot language which is interpreted from the source and then use it together with the source to show that the translation quality into the target language improves. Additionally, they do not consider the effect of transcription noise on translation, which we do. Simultaneous translation approaches such as wait-k (Ma et al., 2019) and Local Agreement (LA-n, Polák et al., 2022) are commonly used, and we use the latter for our experiment.
An alternative multi-sourcing approach is to select and use only one source. Macháček et al. (2021) provide an analysis of using either the original, or its simultaneously interpreted equivalent as a source for simultaneous ST. Interpreting is delayed, but shorter and simpler than translationese. Interpreters also segment their speech to sentences differently than the original speakers, so it is not easy to align segments. In any case, selecting sources will involve additional effort and thus we consider using multiple sources together to be a more effective approach. In this regard, multiple language sources, both as text and speech streams, could be used in ASR (Paulik et al., 2005; Soky et al., 2022) as well as in pre-neural MT and ST (Och and Ney, 2001; Paulik and Waibel, 2008; Khadivi and Ney, 2008). Miranda et al. (2013) use them for punctuation restoration. Kocmi et al. (2021) provide a broad analysis of benefits of multilingual MT.

heading: 3 Parallel Source-Interpreted ASRs are Independent
text: We assume a multi-source setting with the original speech and its simultaneously interpreted equivalent as the two sources will improve robustness to ASR errors if the errors in the two source streams complement each other. This is not obvious because, on the one hand, the ASRs work independently where they are deployed for different languages, trained on different data, and the processing is fully independent. On the other hand, the content of the speeches is identical. Interpreters’ speech pacing also depends on the original speaker, and it may influence the quality of both ASRs the same way. Therefore, in this section, we analyze the dependency of ASR errors in the source and interpreter, on 10-hour ESIC corpus (Macháček et al., 2021) to prove that the ASR errors are indeed independent.
Methodology First, we processed ASR for English original speakers and interpreters into Czech and German. For English, we used the low-latency neural ASR by Nguyen et al. (2021). For German, we used an older hybrid HMM-DNN model trained using the Janus Recognition Toolkit, which features a single-pass decoder (Cho et al., 2013). For Czech, we used Kaldi (Povey et al., 2011) HMMDNN model trained on Czech Parliament data (Kratochvíl et al., 2020). Table 1 summarizes the transcription quality on ESIC showing that the quality is low, but to the best of our knowledge it is the best one available for this domain.
We then re-used the word alignments of gold transcripts between the original and interpretation as described in Macháček et al. (2021). 38% of tokens were aligned between English and Czech interpretations, and 40% between English and German, see Table 2. It may be caused by the characteristics of the language pair (e.g. compound words in German vs multi-word expressions in English), features of interpreting (non-verbatim translation, shortening) and by errors in automatic alignment. We only analyzed the aligned tokens further. Since
there are many tokens left in two 5-hour subsets of the corpus, we consider further analysis as valid.
Finally, we aligned gold and automatic transcripts using Levenshtein edit distance.1 We classified each token in the ASR transcript as transcribed correctly or not, both for source and interpretations.
Results We made a contingency table (Table 3) and ran a χ2 test (Pearson, 1900) of statistical independence. The results show that the parallel source and interpretation ASRs make errors independently of each other with p < 0.01, for both pairs, English-Czech and English-German, for both dev and test subsets.
We manually assessed the severity of the ASR errors and realized that most errors are only in spelling and fluency, and not in adequacy. We therefore conclude that our finding of independence of parallel ASRs may be valid only for ASRs of comparable quality to ours.

heading: 4 Multi-Source Speech Translation
text: Having established that ASR errors are independent, we now analyze whether multi-source neural machine translation (NMT) is robust to noisy sources. We focus on NMT for individual sentences, with gold sentence alignment of the sources and reference. It is a less realistic use-case than translating long speech documents without any sentence segmentation and alignment of the sources, but proving the robustness of multi-sourcing in this
1https://pypi.org/project/edlib/
setting paves the way for its application in long speech document translation.
Data For training, we use data from OPUS (Tiedemann and Nygaard, 2004), aiming at a multiway model with English and German on the source side and Czech as a target. We download all the data from OPUS, remove all sentences from IWSLT, WMT, ESIC and other test sets, filter them by language identification, and then process with dual cross-entropy scoring (JunczysDowmunt, 2018) using the bilingual NMT models from Tiedemann and Thottingal (2020). We select the top 30 million sentences for each language pair as training data, to prevent overfitting for either. It is also near the threshold that Chen et al. (2021) showed as optimal.
For NMT validation and evaluation, we use the “revised transcript and translations” from ESIC (Macháček et al., 2021). These are the texts that were originally uttered in the European Parliament, transcribed, revised and normalized for reading and publication on the website, and then translated. They are analogous, but not identical, to the gold transcripts of the original and interpretations that we used in Section 3. In addition to the version published by Macháček et al. (2021), we properly align the sentences in all the three languages. Two documents were removed because they missed German translation. The corpus is of comparable size to a usual MT test set. See size statistics in Table 4.
For a contrastive evaluation, we use Newstest11 (Callison-Burch et al., 2011). It contains 3003 sentences in 5 languages: English, German, Czech, French and Spanish, the same amount in each. Newstest11 has references that were translated directly, not through an intermediate language. We also use three additional Czech references of Newstest11 that were translated from German (Bojar et al., 2012).
Multi-Sourcing We convert Marian models to PyTorch to be used with the Hugging Face Transformers (Wolf et al., 2020) library, in which we implement late and early averaging. For both single-
and multi-sourcing, we use greedy decoding because beam search support is not implemented with multi-source.
Training details We train a multi-way NMT model using Marian (Junczys-Dowmunt et al., 2018) with English and German as sources, with language identification tokens, and Czech as the target. We use two separate SentencePiece (Kudo and Richardson, 2018) vocabularies, both sizes of 16 000. The source vocabulary is joint for German and English, and the target is only for Czech. The model is a Transformer Base (6 layers, 512 embedding size, 8 self-attention heads, 2048 filter size) trained on 8 Quadro P5000 GPUs with 16 GB memory for 17 days, until convergence.
Checkpoint selection We validate all checkpoints (every 1000 training steps, 15 minutes) on two single sources (English and German) and two multi-sourcing options: early averaging, and late averaging of a single checkpoint with two sources. Furthermore, after the training has ended, we selected top 10 checkpoints that reached the highest BLEU scores for English and German singlesource on the ESIC dev set. We evaluated all pairs of the top performing checkpoints in late averaging multi-sourcing setup. The top performing model from all validation and grid search options was selected as a final model. It is late averaging with a pair of distinct checkpoints. We also use these two checkpoints for single source evaluation.
Evaluation Metrics We estimate translation quality by BLEU (Papineni et al., 2002) and chrF2 (Popović, 2016) calculated by sacreBLEU2 (Post, 2018). We also report the current state-of-the-art metric COMET3 (Rei et al., 2020) that achieves the highest correlation with direct assessment as a kind of human judgements (Mathur et al., 2020). However, COMET requires one source on the input and is not suitable for multi-source. Therefore, we report it twice (En/De COMET) with two single sources. Note that En COMET scores assume English as source and Czech as target. Since ESIC is tri-parallel, even if the translation is obtained using German or English and German multi-source, we only use the English source as the input to the COMET model. De COMET scores are computed similarly.
2Metric signatures: BLEU|nrefs:1|case:mixed|eff:no|tok:13a|smooth:exp| version:2.2.1, chrF2|nrefs:1|case:mixed|eff:yes|nc:6|nw:0|space:no|version:2.2.1
3wmt20-comet-da model
Results with clean inputs Table 5 shows the results of multi-sourcing with clean inputs, without any speech recognition noise. One would be tempted to conclude that the translation from English is of a higher quality than the translation from German (e.g. 33 vs. 26 BLEU on ESIC dev set), but such a claim is risky. The metrics measure the match of the candidate translation with the reference sentence (and, in case of COMET, also with the source), and it is conceivable that the English served as the source for the human reference translation. The Czech reference thus may very well exhibit more traits of the English source than of the German source. While the chrF2 scores agree with BLEU, COMET scores seem to indicate that multisourcing is as good as, if not better than, using a single source. Since COMET is known to correlate with human judgements better than BLEU (Mathur et al., 2020) our results show that multi-sourcing is indeed a viable solution.
To further shed light on the impact of the source used for creating references, we evaluated the models with Newstest11 and computed the scores with three additional references that were translated only from German. The German single source achieves much higher BLEU than the English source (32.23 vs 16.62 BLEU), with multi-sourcing in between (22.47 BLEU). Similar trends are observed in chrF2 and COMET scores. This is the opposite of ESIC scores, where the reference was obtained from English. It shows that the traits of the source language such as word order, structure of clauses and terms are remarkable in automatic metrics when the reference is constructed from that source, but these effects may be negligible in human evaluation. Appendix C contains more details.
Finally, we consider a “balanced” scenario where an equal number of references comes from both sources and this shows similar scores for both single sources (23.40 vs 22.85 BLEU) with multisourcing outperforming them by 0.6 and 1.1 BLEU. We therefore conclude that our multi-source model should be well-prepared for content originating in any of the source languages, but the automatic evaluation metrics may not always capture this. Moving forward, we only use BLEU for simplicity.

heading: 4.1 Modeling Transcription Noise
text: Although multi-sourcing English and German is not very beneficial when both sources are clean, we hypothesize that it could show benefits with noisy
sources. Averaging two noisy sources can lead to cancelling the noise. Since ESIC contains triparallel sentence-aligned translations as texts and not speech, and since we want to evaluate different levels of ASR noise, and we do not have many ASRs, we generate the ASR errors artificially.
Custom WER noise model We adopt the lexical noise model by Martucci et al. (2021) and modify it to create outputs with arbitrary WER. The lexical noise model modifies the source by applying insertion, deletion, substitution, or copy operations on each word with probabilities pI , pD, and pS , respectively. The probabilities are learned from the ASR and gold transcript pairs. It thus may learn to shuffle homonyms such as “eight” and “ate”.
In the original lexical noise model by Martucci et al. (2021), the target WER is bound to the performance of the given ASR system on which it is trained, and can not be changed. WER is defined as
on Newstest11 with balanced reference source language. Green-backgrounded area is where the English single-
source outperforms German single-source. Black underlined numbers indicate the area where multi-sourcing achieves higher score than both single-sourcing options. In bold is near maximum gap from single-source, more than 2.1 BLEU. Red-colored numbers are where at least one single-source scores higher.
the number of incorrect words in the ASR transcript divided by the number of correct words in the gold transcript. The errors are either insertions, deletions, or substitutions. In the lexical noise model, insertion is applied independently on the other operations. Therefore, we can decompose WER to the sum of insertion rate and the rate of deletions or substitutions.
In the lexical noise model, the insertion rate equals to the expected number of insertions for each gold word. Since the probability of not inserting is 1 − pI , the expected number of repetitions before not inserting succeeds is pI1−pI . It is also a mean of a geometric distribution with p = 1− pI .
The rate of deletions and substitutions is pD + (1− pD)pS , where pD is the number of deletions. The words that were not deleted can be substituted,
and there is (1− pD)pS of them. In summary, the original model WER is
WER = pI
1− pI + pD + (1− pD)pS , (1)
To get a custom target WER, we rescale the learned probabilities by a constant c:
WERdesired = cpI
1− cpI +cpD+(1−cpD)cpS . (2)
We simplify the equation above to
WERdesired ≈ cpI + cpD + (1− cpD)cpS . (3)
It leads to a quadratic function where c can be found easily. Since we work with probabilities, we select the smallest non-negative root as the solution. We release our implementation online.4
4https://github.com/pe-trik/ asr-errors-simulator
Training the noise model For training the noise model, we utilize VoxPopuli (Wang et al., 2021) to retrieve around 100,000 audio and gold transcript sentences in English and 60,000 in German. They are from the same domain as ESIC, both corpora are from the European Parliament. We processed the audio with NVidia NeMo CTC ASRs5 (Kuchaiev et al., 2019; Gulati et al., 2020). Then we trained the rules of the lexical noise model and applied them on source data. Since the result is deterministic on the random seed of the lexical noise model, we perform multi-sourcing using three different seeds and report average BLEU scores with standard deviation.
Results with transcription noise Table 6 summarizes the BLEU scores of two-source MT with different levels of transcription noise in each of the sources on two sets: ESIC dev with reference translated from English, and Newstest11 with balanced reference. Appendix A contains the corresponding chrF2 scores. Table 7 shows the results on the ESIC test set for the settings where multi-source models achieved the highest improvement due to noisy inputs.
In Table 6, on both sets, we observe that the less noisy single source achieves higher BLEU than the other single source. When the difference in noise levels between the sources is small (close to diagonal in the table), then multi-sourcing reaches slightly higher BLEU than single sources. In case of balanced Newstest11, this area matches the diagonal. In case of ESIC with English original source and reference translated from English, the area of multi-source outperforming single-source is shifted. This tendency is reflected in the test set results in Table 7 as well. Only when the German source is less noisy than the English one, it does improve BLEU in multi-sourcing. We explain it by the discrepancy of source languages for MT and reference that affect BLEU the same way as in offline mode in Section 4. On Newstest11, with the references translated from German, we expect the reverse.
We also observe expected behavior that the more noise, the lower BLEU in all setups. Compare e.g. 33.3 BLEU with zero noise and 12.1 with 40% WER in both sources. With very large noise, it is possible that neither option would be usable. In ESIC dev, e.g. when English WER is 20%, we observe large span, between 5 and 25% WER in
5stt_de_quartznet15x5 and stt_en_conformer_ctc_large from https://catalog.ngc.nvidia.com/models
German, where multi-sourcing outperforms single source at least by several hundreths of BLEU. This span in Newstest11 is much more narrow, only 20 to 25% WER in German. We hypothesize that it may be caused by the domain difference. The lexical noise model is trained on Europarl. In news domain, there may be fewer words for substitution, so the noise consists more of deletions and insertions, and it might be more harmful for MT in combination of two sources. However, multi-sourcing appears to be robust to ASR errors regardless of whether we have one or both sources as original.

heading: 5 Simultaneous Multi-Source
text: In the previous section, we experimented with offline translation with artificial ASR noise and showed that multi-source models are indeed robust to noise. However, one important use case of speech translation is in a real time setting where simultaneous MT is used. We therefore evaluate the robustness of multi-source models in a simultaneous setting.

heading: 5.1 Simultaneous Machine Translation
text: Simultaneous MT is a task that simulates one subtask of a technology that translates long-form monologue speech in real-time, or with the lowest possible latency. There exist two main approaches to simultaneous MT: streaming and re-translating (Niehues et al., 2018; Arivazhagan et al., 2020). Re-translating systems generate preliminary translation hypotheses that can be updated. Both approaches have complementary benefits and drawbacks. In this paper, we focus on streaming.
We assume that simultaneous MT continuously receives an input text segmented to sentences, one token at a time, as produced by the speaker and upstream tasks. After reading each input token, the system can either produce one or more target tokens, or decide to read the next input token, e.g. to have more context for translation. The goal of simultaneous MT is to translate the input with high quality and low latency. Quality is measured on full sentences as in standard text-to-text MT, e.g. by BLEU. The standard latency measure of simultaneous MT is Average Lagging (AL, Ma et al., 2019). It is an average number of tokens behind an “optimal” policy that generates the target proportionally with reading the source.
Simultaneous MT can be created from standard text-to-text NMT by applying any simultaneous de-
coding algorithm. However, it is recommendable first to adapt NMT, so it is inclined to translate consecutive sentence prefixes with the same target prefix. We use Local Agreement (LA-n) as a decoding algorithm. It achieved good performance by the best performing system (Polák et al., 2022) in the most recent IWSLT competition (Anastasopoulos et al., 2022). Local Agreement (LA-n) means that n consecutive updates must agree on a target prefix to commit and write. The last committed prefix is then forced as a prefix to decoding the next units. Agreement size n is a parameter that controls the latency.

heading: 5.2 Creating Simultaneous MT Systems
text: In Section 4, we used multi-way models trained on full sentences, but in a simultaneous setting, these models will make mistakes when translating partial sentences using the LA-n approach. Therefore, our multi-way models should first be adapted for partial sentence translation. To this end, we used the multiway English and German to Czech MT model as a base for simultaneous MT. We fine-tuned the last trained model checkpoint for stable translation on 1:1 mix of incomplete sentence prefixes and full sentences as Niehues et al. (2018). For each source-target pair of the training data, we selected 5-times 1 to 90 % of source and target characters and rounded them to full words. Then, we ran
training for 1 day on 1 GPU. We validated BLEU score on ESIC dev and Normalized Erasure (NE, Arivazhagan et al., 2020) on all prefixes of the first 65 sentences (around 1500 words) of ESIC dev set. We ran fine-tuning with multi-way data for English and German as source languages, and for bilingual English-Czech and German-Czech MT.
We stopped training after one day when there were no improvements in stability or quality. Then, we selected one checkpoint for English and one for German that reached acceptable quality and stability values. See Appendix B for details.

heading: 5.3 Multi-Sourcing in Simultaneous MT
text: We use late averaging of the two selected checkpoints for multi-sourcing in simultaneous MT. The only aspects of multi-sourcing in simultaneous mode that differ from single-source or nonsimultaneous mode are synchronization of the sources and how to count Average Lagging.
Synchronization In a realistic use-case, it is necessary to synchronize the original speech and simultaneous interpreting. However, we leave it for further work, as our goal is to inspect the limits of multi-sourcing. Therefore, we simulate a case where the sources are optimally synchronized, aligned and parallel to sentence level.
In multi-source mode, we sort all sentence pre-
fixes by proportion of the character length to the sentence length. Each “Read” operation of the multi-source system then receives two prefixes in two languages. One of them is updated by one new token. Every such update is counted to local agreement size. We note that there are other strategies, e.g. count only English source updates to LA-n, but in this paper we have other goal than searching for the best strategy.
AL in multi-source In multi-source setup, we only count Read operations of the English source to AL calculations that we report, and not of the German source because the sources are simultaneous. Counting only German tokens differs negligibly, approximately by 0.1 tokens.

heading: 5.4 Simultaneous Multi-Source with Artifical Noise
text: We want to compare multi-sourcing model to single-sourcing with artificial ASR noise model as in Section 4.1. We evaluate each system on the latency levels with local agreement sizes 2, 5, 10 and 15. Since each evaluation takes approximately 5 hours on 2000 sentences, we report only one run, and not average and deviation on multiple randomly noised inputs.
The results on ESIC dev set are in Figure 1. We can observe the same trends as in the offline case. The single source that is noised less achieves higher BLEU. Multi-sourcing outperforms both single sources when both noise levels are similar and when the English one is lower, e.g. in the case with 10% WER in German and 20% WER in English. We explain it again by the fact that the Czech reference is translated from English, and not German.
Furthermore, on both ESIC and Newstest11 (Figure 1) we observe that multi-sourcing performs worse in the low latency modes, i.e. in AL<5 that roughly corresponds to LA<5. We assume that the proportional synchronization of the two sources is often inaccurate and may confuse late averaging. In higher latency modes, the synchronization noise at the end of input may be lowered by local agreement. Having validated the multi-source NMT is robust to ASR errors in both full sentence and simultaneous settings, we have paved the way for harder settings where multilingual interpretations of the original source available with different amounts of delay can be used for translation.

heading: 6 Conclusion
text: We have investigated the robustness of multi-source NMT to transcription errors in order to motivate its use in settings where ASRs for the original speech and its simultaneously interpreted equivalent are available. To this end, we first analyzed the 10-hour ESIC corpus and documented that the ASR errors in the two sources are indeed independent, indicating their complementary nature. We then simulated transcription noise for English and German when translating into Czech in single and multi-source NMT settings and observed that using multiple noisy sources is significantly better than individual noisy sources. We then repeated experiments in a simultaneous translation setting and showed that multi-source translation continues to be robust to noise. This robustness of multi-source NMT to noise motivates future research into simultaneous multi-source speech translation, where one source is available with a delay. We will also consider training models with simulated ASR errors to further increase their robustness, especially in multisource settings.

heading: 7 Limitations
text: Although we have shown the robustness of multisource NMT to transcription errors in a fullsentence and simultaneous settings, our work has the following limitations:
• Our work does not address the case where the additional source, typically interpreted, is available after a delay. A delayed source may reduce the gains seen by multi-sourcing.
• We have only focused on the Local Agreement (LA-n) approach for simultaneous translation and exploration of other simultaneous approaches such as wait-k remains.
• Human evaluation of translations is pending.
• Evaluation on other language pairs is pending.

heading: Acknowledgements
text: The research was partially supported by the grants 19-26934X (NEUREM3) of the Czech Science Foundation, “Grant Schemes at CU” (reg. no. CZ.02.2.69/0.0/0.0/19_073/0016935), 398120 of the Grant Agency of Charles University, and SVV project number 260 698. Part of the work was done during an internship at NICT.

heading: A chrF2 Scores with Noisy Inputs
text: There is an evidence that chrF2 correlates with human judgements better than BLEU. In Table 8, we can see that for multi-sourcing with noisy inputs on ESIC dev, chrF2 are indeed higher than single-sourcing, and this correlates with the BLEU score gains in Table 6. On the other hand, for Newstest11, chrF2 scores do not indicate any improvements. While the corresponding BLEU scores in
Table 6 indicated improvements of multi-sourcing with noisy inputs, the magnitude of these gains were minor, much smaller than those observed for ESIC. This gives us sufficient reason to believe that multi-sourcing should be useful in a setting like ESIC, where the reference is created from only one source, which is more realistic than the “balanced” use-case of Newstest11, where the reference originates from 5 languages.

heading: B Checkpoint Selection for Simultaneous Multi-Source
text: The checkpoints that we selected for simultaneous multi-source decoding (recall Section 5.2) was the multi-way checkpoint for English and bilingual one for German. Table 9 summarizes the results of finetuning for stability. BLEU decreased marginally (by 0.2 on English and 0.9 on German), while normalized erasure (NE) dropped by 40% in English and 52% on German.
Based on some outputs, we explain higher NE in German-to-Czech by discrepancy in word orders. Many erasures were caused by an incorrect presumption of the final verb. Regardless, our finetuned models exhibit significantly reduced NE and can be reliably used for simultaneous translation using the LA-n approach.

heading: C Effect of Reference Source Language
text: To explain the effect of reference source language, we run a contrastive evaluation on the subset of Newstest11 that consists only from the documents that originate in English. We compare BLEU measures with a reference translated directly from Czech, and with three additional references translated only from German (Bojar et al., 2012).
The results of simultaneous mode (recall Section 5) are in Figure 2. We observe the same trends as in offline mode in Section 4. The BLEU score is higher for the single source with the language from which the reference was translated. When this source is noised substantially more than the other, multi-sourcing outperforms both by a small margin.
In case of German references, the nearest margin to single-sourcing is much smaller than with the English references. We assume it is because the structural difference of English source and GermanCzech references is larger than German to EnglishCzech reference. It is documented also by BLEU scores with zero noise (33 and 20 on references
from English vs 16 and 30 on references from German).
the English single-source outperforms German single-source. Black underlined numbers indicate the area where multi-sourcing achieves higher score than both single-sourcing options. Red-colored numbers are where at least one single-source scores higher.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
7
A2. Did you discuss any potential risks of your work? Not applicable. Left blank.
3 A3. Do the abstract and introduction summarize the paper’s main claims? 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? 2-5

heading: 3 B1. Did you cite the creators of artifacts you used?
text: 2-5
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Not applicable. Left blank.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Not applicable. Left blank.
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? we didn’t create anything from potentially harmful data. The cited authors are responsible for the data we used.

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? 2-5
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Left blank.
C 3 Did you run computational experiments? 2-5

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? 2-5
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? 2-5

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? 2-5
C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Not applicable. Left blank.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Not applicable. Left blank.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Not applicable. Left blank.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Not applicable. Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Not applicable. Left blank.

heading: Future Work
text: This robustness of multi-source NMT to noise motivates future research into simultaneous multi-source speech translation, where one source is available with a delay. We will also consider training models with simulated ASR errors to further increase their robustness, especially in multisource settings