Venue: EMNLP 2023

# CLAD-ST: Contrastive Learning with Adversarial Data

for Robust Speech Translation

Sathish Reddy Indurthi, Shamil Chollampatt, Ravi Agrawal, Marco Turchi Zoom Video Communications sathisreddy.indurthi@zoom.us, shamil.chollampatt@zoom.us, ravi.agrawal@zoom.us, marco.turchi@zoom.us

###### Abstract

The cascaded approach continues to be the most popular choice for speech translation (ST). This approach consists of an automatic speech recognition (ASR) model and a machine translation (MT) model that are used in a pipeline to translate speech in one language to text in another language. MT models are often trained on well-formed text and therefore lack robustness while translating noisy ASR outputs in the cascaded approach, degrading the overall translation quality significantly. We address this robustness problem in downstream MT models by forcing the MT encoder to bring the representations of a noisy input closer to its clean version in the semantic space. This is achieved by introducing a contrastive learning method that leverages adversarial examples in the form of ASR outputs paired with their corresponding human transcripts to optimize the network parameters. In addition, a curriculum learning strategy is then used to stabilize the training by alternating the standard MT log-likelihood loss and the contrastive losses. Our approach achieves significant gains of up to 3 BLEU scores in English-German and English-French speech translation without hurting the translation quality on clean text.

+
Footnote †: _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_, pages 9049–9056 December 6-10, 2023 ©2023 Association for Computational Linguistics

## 1 Introduction

Neural machine translation (NMT) has made significant advancements over the past several years with claims of achieving 'human parity' Hassan et al. (2018), and 'super-human performance' Barrault et al. (2019). However, despite making tremendous progress in quality and coverage Costa-jussa et al. (2022), NMT models have been identified to lack robustness in dealing with noisy inputs Belinkov and Bisk (2018).

Robustness is especially important in _cascaded_ speech translation (ST) systems, where an NMT model works on the output of the upstream automatic speech recognition (ASR) system. In this scenario, significant MT performance degradation has been measured due to _i) error propagation_ from the ASR and _ii)_ the mismatch between _training-testing_ condition as the NMT model is typically trained on well-formed text making it weak in dealing with noisy inputs. For these reasons, there has been significant effort towards building end-to-end ST models Berard et al. (2016); Weiss et al. (2017). However, due to the lack of sufficient speech translation datasets in many languages and the flexibility to independently optimize ASR and MT systems, cascaded systems continue to be a dominant approach Anastasopoulos et al. (2022).

Prior research has tried to tackle the robustness problem in NMT models independently by (1) synthetic noise injection Belinkov and Bisk (2018), including mimicking ASR errors for the ST task Sperber et al. (2017); Li et al. (2018); Martucci et al. (2021); Wang et al. (2022) or (2) by adding noise to the continuous-space representations learned by the NMT model Sato et al. (2019); Cheng et al. (2020); Wei et al. (2022). Some of the synthetic noise injection methods replace words with probable words from ASR confusion lattices Martinoci et al. (2021); Wang et al. (2022) but fail to consider the full context and to realistically replicate ASR errors, like the phrase-level errors and segmentation errors that ASR models often make (see Table 1). On the other

\begin{table}
\begin{tabular}{|l|} \hline
**Human Transcript: I’m not sure that’s wise, given the importance of the problem, but there’s now the geoengineering discussion about: _Should_ that be in the back pocket in case things happen faster, _or_ this innovation goes a lot slower than we _expect_? \\ \hline
**ASR Output: I’m not sure that’s why it’s given the importance of the problem. But now that the geoenineering discussion about _should_ that be in the back pocket in case things happen faster _or_ this innovation goes a lot slower than we _expect_. \\ \hline \end{tabular}
\end{table}
Table 1: Example human transcript and OpenAI Whisper ASR output from the MuST-C En-De devset. The underlined text indicates erroneous phrases and _italicized_ indicates punctuation errors.



hand, the latter approaches predominantly generate arbitrary or random error distributions and overlook the real-world error signals from the actual upstream ASR model.

To address the robustness problem, particularly in the context of cascaded ST, we propose to combine the best of the two approaches. This is obtained by training the NMT model with adversarial examples generated from the ASR outputs and encouraging the encoder representations of both the ASR outputs and their corresponding human transcripts to be closer to each other. This is done via _contrastive learning_Saunshi et al. (2019), in addition to the standard log-likelihood NMT training, and through _curriculum_ learning Bengio et al. (2009) by pretraining on log-likelihood loss followed by iteratively alternating between the two objectives. Our method leads to significant improvements in the ST task on two language pairs (En-De, En-Fr) without hurting the performance on MT of well-formed texts. Our approach also follows the standard cascade approach that does not rely on any end-to-end ST data (speech aligned with their transcription and translation) unlike that of Bahar et al. (2021) and Lam et al. (2021), making it easier to apply to a wider range of language pairs.

## 2 Method

Our NMT model is a Transformer model Vaswani et al. (2017) with several encoder and decoder layers, with self-attention modules within each block and a cross-attention module to link the encoder and decoder components. Training the network parameters \(\Theta\) is done by minimizing the mean negative log-likelihood over a batch of \(n\) source-target text translation pairs \((X,Y)=({\bf x}^{(1)},{\bf y}^{(1)}),\ldots,({\bf x}^{(n)},{\bf y}^{(n)})\). This objective encourages producing a target sentence \({\bf y}\) with a sequence of tokens \(y_{1},y_{2},\dots,y_{|{\bf y}|}\) given the source sentence \({\bf x}\) with sequence of tokens \(x_{1},x_{2},\dots,x_{|{\bf x}|}\).

\[\mathscr{L}_{\text{NMT}}=-\sum_{i=1}^{n}\sum_{j=1}^{|{\bf y}^{(i)}|}\log P(y_ {j}^{(i)}|{\bf x}^{(i)},{\bf y}_{<j}^{(i)},\Theta)\]

To improve the robustness of the NMT model on noisy ASR outputs for cascaded speech translation, we use a _contrastive_ learning method Wei et al. (2021) that is aimed to bring the representations of a noisy sentence generated by the upstream ASR closer to its clean version (the human transcript for the same audio) in the semantic space modeled by the NMT encoder.

To get the encoder sentence representations efficiently for contrastive learning, a [CLS] token is prepended to the input sentences similar to the BERT model Devlin et al. (2019). We found that using encoder output corresponding to the [CLS] token is slightly better than using the mean of encoder outputs as the sentence representation. \({\bf s}_{\mathbf{x}}\), which also gets added to each of the decoder outputs before softmax (see Figure 1).

Contrastive learning uses speech transcription corpora (i.e., speech paired with human transcripts). The speech input is passed through the upstream ASR model to obtain the ASR outputs. Given the noisy ASR output \(\overline{\mathbf{x}}\) paired with its corresponding clean human transcript \({\bf x}\), we minimize the contrastive objective \(\mathscr{L}_{\text{CTL}}\), which is an average of two symmetric sentence-level contrastive loss functions.

Given a batch of \(n\) examples \((X,\bar{X})=\{({\bf x}^{(1)},\overline{\mathbf{x}}^{(1)}),\ldots,({\bf x}^{( n)},\overline{\mathbf{x}}^{(n)})\}\)

\[\mathscr{L}_{\text{CTL}}=\frac{1}{2n}\sum_{i=1}^{n}\Big{(}\mathcal{L}_{c}({ \bf x}^{(i)})+\mathcal{L}_{c}(\overline{\mathbf{x}}^{(i)})\Big{)} \tag{1}\]

where

\[\mathcal{L}_{c}({\bf x})=\frac{-\exp\left(D({\bf s}_{\mathbf{x}},{\bf s}_{ \overline{\mathbf{x}}})\right)}{\exp\left(D({\bf s}_{\mathbf{x}},{\bf s}_{ \overline{\mathbf{x}}})\right)+\underset{x^{\prime}}{\sim}\exp(D({\bf s}_{ \mathbf{x}},{\bf s}_{\mathbf{x}^{\prime}}))}\]

\(D({\bf u},{\bf v})\) denotes the cosine distance between two vectors \({\bf u}\) and \({\bf v}\). \({\bf s}_{\mathbf{x}^{\prime}}\) represents a negative example

Figure 1: Proposed model and the two alternating training objectives. \(\mathscr{L}_{\text{NMT}}\) is the standard NMT negative log-likelihood loss and \(\mathscr{L}_{\text{CTL}}\) is the contrastive loss for improving robustness to ASR outputs.

 

constructed for every other sentence \({\bf x}^{\prime}\) in the batch. Given \({\bf x}^{\prime}\in(X\cup\bar{X})\setminus\{{\bf x},\overline{{\bf x}}\}\) and its sentence embedding \({\bf s}_{x^{\prime}}\)

\[\mathbf{\hat{s}_{x^{\prime}}=\begin{cases}{\bf s}_{\bf x}+\lambda_{\bf x}({\bf s }_{{\bf x}^{\prime}}-{\bf s}_{\bf x})&\text{if, $d_{\bf x}^{+}>d_{\bf x}^{-}$}\\ {\bf s}_{{\bf x}^{\prime}}&\text{otherwise}\end{cases} \tag{2}\]

where \(d_{\bf x}^{+}=\|{\bf s}_{\bf x}-{\bf s}_{\overline{{\bf x}}}\|_{2}\), \(d_{\bf x}^{-}=\|{\bf s}_{\bf x}-{\bf s}_{{\bf x}^{\prime}}\|_{2}\), and \(\lambda_{\bf x}=\frac{d_{\bf x}^{+}}{d_{\bf x}}\). The above linear interpolation in Eq 2 with exponentially decaying \(\lambda_{\bf x}\) is implemented following Wei et al. (2021) to increase the hardness of negative examples by not selecting uninformative negative examples as training progresses. \(\mathcal{L}_{c}(\overline{{\bf x}})\) follows similar formulation as \(\mathcal{L}_{c}({\bf x})\) with \({\bf x}\) replaced by \(\overline{{\bf x}}\), and vice versa.

To stabilize the NMT parameters before applying contrastive learning, we also use curriculum learning Bengio et al. (2009). We first pre-train the model with NMT loss \(\mathscr{L}_{\text{NMT}}\) only for the first \(N\) batches, followed by iteratively training alternate batches with \(\mathscr{L}_{\text{CTL}}\) and \(\mathscr{L}_{\text{NMT}}\) losses, respectively, until convergence. With this model and training regime, which alternates between training with translation and speech recognition datasets independently, we are able to build robust NMT models for cascaded ST without any requirement for speech translation data.

## 3 Experimental Setup

We experiment with English-German (En-De) and English-French (En-Fr) language directions.

**Training Data**: For parallel text translation data, we use WMT'16 En-De Bojar et al. (2016) and WMT'14 En-Fr Bojar et al. (2014) news translation task data for the corresponding language directions. To get the ASR data for contrastive learning, we use Open AI Whisper base ASR Radford et al. (2022) to transcribe the speech in the training sets from Mozilla Common Voice 12.0 Ardila et al. (2020), VoxPopuli Wang et al. (2021), and the English set of Multilingual LibrSpeech Pratap et al. (2020) for both language directions. As an additional experiment, we also try adding in-domain MuST-C training data. The full statistics of the datasets are in Appendix A.2 (Table 8).

**Evaluation**: We use MuST-C Di Gangi et al. (2019) tst-COMMON as our test set and MuST-C dev as our validation set. For MT and ST tasks, the inputs to the model are the Whisper (base) ASR outputs and the corresponding human transcripts, respectively. We evaluate against the English and French reference translations for the respective translation directions. We use case-sensitive, detokenized BLEU computed using sacrebleu1Post et al. (2018). We also compute statistical significance using paired bootstrap resampling Koehn (2004).

Footnote 1: [https://github.com/mjpost/sacrebleu](https://github.com/mjpost/sacrebleu). Signature: nrefs:1|case:mixedeff:no|tok:13a|smooth:exp

**Models**: We use a base Transformer Vaswani et al. (2017) as our baseline NMT model, which we extend to support our proposed approach, Contrastive Learning with Adversarial Data for Speech Translation (**CLAD-ST**). We also compare against two recent state-of-the-art approaches for NMT robustness which we re-implement: (1) **CONF-ASR**Wang et al. (2022) generates adversarial examples by substituting random source tokens with candidate tokens from ASR confusion set based on their embedding distances to the original token. (2) **CSA-NMT**Wei et al. (2022)2 uses a separate semantic encoder to model a semantic region around the source and target sentences. To improve NMT robustness, samples from this region are used to augment the NMT decoder representations.

Footnote 2: Code and network hyperparameters are from [https://github.com/pemywei/csannt](https://github.com/pemywei/csannt)

We tokenize all datasets using a sentencepiece model Kudo and Richardson (2018) for each language pair on the text translation datasets with a shared source-target vocabulary size of 32,000. We use OpenAI Wigner (base) ASR as our main upstream model. We also evaluate the ST performance of our model using Whisper large ASR. Other training details and hyperparameters are in Appendix A.1.

## 4 Results and Analysis

### Overall Performance

Our proposed CLAD-ST achieves 26.6 and 33.4 BLEU on the En-De and En-Fr ST tasks, significantly (\(p<0.001\)) outperforming the Transformer baseline by 3.1 BLEU points and 2.6 BLEU

\begin{table}
\begin{tabular}{l|c|c|c|c} \hline  & \multicolumn{2}{c|}{**En-De**} & \multicolumn{2}{c}{**En-Fr**} \\ \cline{2-5}
**NMT** & MT & ST & MT & ST \\ \hline Transformer-base & 28.1 & 23.5 & 36.1 & 30.8 \\ \hline CONF-ASR Wang et al. (2022) & 28.2 & 25.2 & 36.3 & 32.0 \\ CSA-NMT Wu et al. (2021) & **29.5** & 24.4 & **37.2** & 31.3 \\ \hline CLAD-ST & 28.8 & **26.6** & 36.5 & **33.4** \\ \hline \end{tabular}
\end{table}
Table 2: Performance of various models on MT and ST tasks on MuST-C st-COMMON.

 

points (see Table 2). Our approach consistently performs better than other baselines (CONF-ASR and CSA-NMT), yielding gains of 1.4 BLEU points (\(p<0.001\)) on both language directions. The improvement of our approach over CONF-ASR shows the importance of contextual modeling of the ASR errors instead of simulating them with ASR confusion lattices. CSA-NMT achieves the largest improvement in the MT task but fails to specialize to ASR errors, yielding lower improvements compared to our approach.

### Robustness

In Figure 2, we report the performance of all the EnDe models as the number of ASR errors increases in the input sequence. We group the ASR outputs based on the number of tokens different from the reference transcripts on the test set. The performance of the Transformer baseline model drastically drops as the number of ASR errors increases. CONF-ASR performs better than the Transformer baseline in all the ASR error conditions and closely with CLAD-ST on ASR outputs with one or two errors. However, CLAD-ST notably outperforms it as the number of errors increases. CSA-NMT performs slightly better than the baseline Transformer when the number of errors is few and performs slightly better than CONF-ASR when increasing the number of errors due to their semantic augmentation. Overall, our approach has shown to be robust against sequences having several ASR errors and performs consistently better than the other baselines, irrespective of the number of errors in the sequence.

### Curriculum Learning

We study the effect of the curriculum learning strategy (Section 2) and reports the results in Table 3. We find that curriculum learning strategy brings an improvement of 0.5 BLEU on En-De and 0.7 BLEU On En-Fr ST tasks, respectively.

### Out-of-distribution ASR

In Table 4, we report the performance of all the EnDe systems using Whisper base as the ASR model during training and a higher-quality ASR (Whisper large) during testing. Although training on lower-quality ASR outputs with more errors, a similar BLEU score improvement is also noticed for all the systems when fed better ASR outputs. However, CLAD-ST achieves the largest improvements over the baseline, showing that it better generalizes over different ASR-quality outputs.

### Model Size

To investigate if our results scale when we increase the size of the network, we use Transformer-large instead of Transformer-base for our baseline models as well as CLAD-ST. The results are reported in Table 5. We find that we get improvements of 1.5 and 1.3 BLEU score points for the ST task on En-De and En-Fr, respectively, compared to CONF-ASR, demonstrating its advantage even on larger NMT architectures.

### In-domain Training Data

We also experiment with adding in-domain training data from MuST-C for training the NMT models in

Figure 2: BLEU scores compared to the number of token-level errors in the ASR outputs on En-De ST task on MuST-C tst-COMMON dataset.

\begin{table}
\begin{tabular}{|l|c|c|} \hline \multirow{2}{*}{**Model**} & \multicolumn{2}{c|}{**En-De**} \\ \cline{2-3}  & **Wbase** & **Wlarge** \\  & (WER = 14.3) & (WER = 7.6) \\ \hline Transformer-base & 23.5 & 24.4 \\ \hline CONF-ASR Wang et al. [202] & 25.2 & 26.3 \\ CSA-NMT Wei et al. [2022] & 24.4 & 25.8 \\ \hline CLAD-ST & **26.6** & **27.4** \\ \hline \end{tabular}
\end{table}
Table 4: BLEU scores on MuST-C En-De tst-COMMON ST task when using Whisper-large (Wlarge) vs. Whiser-base (Wbase) ASR. Word error rate (WER) of the ASR model is provided.

 

CLAD-ST and the baselines (see Table 6). In line with our expectations, we find a notable improvement in performance of all the systems. Despite the improved baselines, we observe similar improvements for CLAD-ST of 1.4 BLEU on both En-De and En-Fr ST tasks compared to CONF-ASR, and 2.4 and 2.2 BLEU on En-De and En-Fr ST tasks, respectively, compared to CSA-NMT.

## 5 Related Work

Several prior works tried to improve the robustness of MT models to noisy ASR inputs for cascaded speech translation. Sperber et al. (2017) randomly sampled synthetic noise in the form of insertions, deletions, and substitutions and injected them into the source side of MT training data. Li et al. (2018) encoded the syllables (_Pinyin_) for Chinese ST and injected synthetic character-level substitutions to simulate ASR errors. In contrast, Martucci et al. (2021) trained a _lexical noise model_ using confusion sets extracted by aligning the actual ASR output to reference transcript to inject synthetic errors. Wang et al. (2022) picked substitution candidates from the ASR confusion set that are farthest away from the source token embeddings leading to more robust MT models. Cheng et al. (2019) also used the actual ASR output in training incorporating an additional discriminator network that tries to distinguish between human transcript and the ASR output following Cheng et al. (2018) to improve robustness. Padfield and Cherry (2021) simulated not only token-level errors but also segmentation errors and projected them on the target side as well. Bahar et al. (2021), on the other hand, proposes to jointly optimize the ASR and MT components in the cascaded system using speech-translation data. Similarly, Lam et al. (2021) also uses speech-translation data to reduce error propagation between the two components by mutually training the MT system on ASR n-best outputs and using cyclic feedback from MT outputs for self-training of the ASR model.

## 6 Conclusion

We improve the robustness of MT to ASR outputs using contrastive learning to bring the representations of clean and noisy examples closer in the semantic space. Our approach does not require any speech translation corpus. We significantly improve the translation accuracy on noisy ASR outputs without degrading translation accuracy on clean text. We also show that the approach is scalable to better-quality ASR models in the cascade other than the one used during training. The proposed approach is generic and is applicable beyond the context of speech translation alone, such as translating user-generated chat text or non-native text if paired noisy-clean data is available.

### Limitations

The limitations of our paper are:

* As in any ST cascade architecture, the performance of our MT system depends on the quality of the ASR model outputs. Since we use examples generated by the ASR to train the MT model, the quality of ASR outputs directly affects the model performance. In our work we tested two ASR systems having different quality (base: 14.3 WER and large: 7.6 WER). This is particularly relevant when English is not the source language side.
* Evaluation is done using only the BLEU score. We did not use human evaluation or COMET.
* The evaluation is limited to two language pairs having English as the source. This selection of the source language is quite important because having a no-English language as a source will expose our MT model to a) ASR models of probably lower quality and b) different and more varied linguistic challenges that might affect the work of the adversarial method.

\begin{table}
\begin{tabular}{|l|c|c|c|c|} \hline  & \multicolumn{2}{c|}{**En-De**} & \multicolumn{2}{c|}{**En-Fr**} \\ \cline{2-5}
**NMT** & MT & ST & MT & ST \\ \hline Transformer-base & 31.0 & 26.1 & 37.9 & 32.9 \\ \hline CONF-ASR [Wang et al., 2022] & 30.9 & 27.9 & 38.0 & 33.9 \\ CSA-NMT [Woi et al., 2021] & **32.1** & 26.9 & **38.8** & 33.1 \\ \hline CLAD-ST & 31.6 & **29.4** & 38.2 & **35.3** \\ \hline \end{tabular}
\end{table}
Table 6: Performance of various models on MT and ST tasks on MuST-C tst-COMMON when adding inhomian MuST-C training data.

\begin{table}
\begin{tabular}{|l|c|c|c|c|} \hline  & \multicolumn{2}{c|}{**En-De**} & \multicolumn{2}{c|}{**En-Fr**} \\ \cline{2-5}
**NMT** & MT & ST & MT & ST \\ \hline Transformer-big & 29.6 & 24.9 & 38.7 & 33.3 \\ \hline CONF-ASR [Wang et al., 2022] & 29.8 & 26.6 & 38.8 & 34.6 \\ CSA-NMT [Wu et al., 2021] & **31.1** & 25.7 & **39.9** & 33.7 \\ \hline CLAD-ST & 30.3 & **28.1** & 39.1 & **35.9** \\ \hline \end{tabular}
\end{table}
Table 5: Performance of using Transformer-large as the model architecture on MT and ST tasks on MuST-C test-COMMON.

 



## References

* Anastasopoulos et al. (2019) Antonios Anastasopoulos, Loic Barrault, Luisa Benitovogli, Marcely Zanon Boito, Ondrej Bojar, Roldano Cattoni, Anna Currey, Georgiana Dinu, Kevin Duh, Maha Elbayad, Clara Emmanuel, Yannick Esteve, Marcello Federico, Christian Federmann, Soubir Gabiche, Hongyu Gong, Roman Grundkiewicz, Barry Haddow, Benjamin Hsu, David Javorsky, Vera Kloudova, Surafel Lakew, Xutai Ma, Prashant Mathur, Paul McNamee, Kenton Murray, Maria Nadejde, Satoshi Nakamura, Matteo Negri, Jan Niehues, Xing Niu, John Ortega, Juan Pino, Elizabeth Salesky, Jiatong Shi, Matthias Sperber, Sebastian Stuker, Katsuhito Sudoh, Marco Turchi, Yogesh Virkar, Alexander Waibel, Changhan Wang, and Shinji Watanabe. 2022. Findings of the IWSLT 2022 evaluation campaign. In _Proceedings of the 19th International Conference on Spoken Language Translation_.
* Ardila et al. (2020) Rosana Ardila, Megan Branson, Kelly Davis, Michael Kohler, Josh Meyer, Michael Henretty, Reuben Morais, Lindsay Saunders, Francis Tyers, and Gregor Weber. 2020. Common voice: A massively-multilingual speech corpus. In _Proceedings of the Twelfth Language Resources and Evaluation Conference_.
* Bahar et al. (2021) Parnia Bahar, Tobias Bieschke, Ralf Schluter, and Hermann Ney. 2021. Tight integrated end-to-end training for cascaded speech translation. In _Proceedings of the 2021 IEEE Spoken Language Technology Workshop_.
* Barrault et al. (2019) Loic Barrault, Ondrej Bojar, Marta R. Costa-jussa, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Philipp Koehn, Shervin Malmasi, Christof Monz, Mathias Muller, Santanu Pal, Matt Post, and Marcos Zampieri. 2019. Findings of the 2019 conference on machine translation (WMT19). In _Proceedings of the Fourth Conference on Machine Translation_.
* Belinkov and Bisk (2018) Yonatan Belinkov and Yonatan Bisk. 2018. Synthetic and natural noise both break neural machine translation. In _International Conference on Learning Representations_.
* Bengio et al. (2009) Yoshua Bengio, Jerome Louradour, Ronan Collobert, and Jason Weston. 2009. Curriculum learning. In _Proceedings of the 26th Annual International Conference on Machine Learning_.
* Berard et al. (2016) Alexandre Berard, Olivier Pietquin, Laurent Besacier, and Christophe Servan. 2016. Listen and translate: A proof of concept for end-to-end speech-to-text translation. In _Proceedings of the NIPS Workshop on end-to-end learning for speech and audio processing_.
* Bojar et al. (2014) Ondrej Bojar, Christian Buck, Christian Federmann, Barry Haddow, Philipp Koehn, Johannes Leveling, Christof Monz, Pavel Pecina, Matt Post, Herve Saint-Amand, Radu Soricut, Lucia Specia, and Ales Tamchyna. 2014. Findings of the 2014 workshop on statistical machine translation. In _Proceedings of the Ninth Workshop on Statistical Machine Translation_.
* Bojar et al. (2016) Ondrej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Varvara Logacheva, Christof Monz, Matteo Negri, Aurelie Neveol, Mariana Neves, Martin Popel, Matt Post, Raphael Rubino, Carolina Scarton, Lucia Specia, Marco Turchi, Karin Verspoor, and Marcos Zampierl. 2016. Findings of the 2016 conference on machine translation. In _Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers_, Berlin, Germany.
* Cheng et al. (2019) Qiao Cheng, Meiyuan Fan, Yaqian Han, Jin Huang, and Yitao Duan. 2019. Breaking the data barrier: Towards robust speech translation via adversarial stability training. In _Proceedings of the 16th International Conference on Spoken Language Translation_.
* Cheng et al. (2020) Yong Cheng, Lu Jiang, Wolfgang Macherey, and Jacob Eisenstein. 2020. AdvAug: Robust adversarial augmentation for neural machine translation. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_.
* Cheng et al. (2018) Yong Cheng, Zhoapeng Tu, Fanding Meng, Junjie Zhai, and Yang Liu. 2018. Towards robust neural machine translation. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics_.
* Costa-jussa et al. (2022) Marta R Costa-jussa, James Cross, Onur Celebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elaine Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, et al. 2022. No language left behind: Scaling human-centered machine translation. _arXiv preprint arXiv:2207.04672_.
* Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_.
* Di Gangi et al. (2019) Mattia A. Di Gangi, Roldano Cattoni, Luis Bentivogli, Matteo Negri, and Marco Turchi. 2019. MuST-C: a Multilingual Speech Translation Corpus. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_.
* Hassan et al. (2018) Hang Hassan, Anthony Aue, Chang Chen, Vishal Chowdhary, Jonathan Clark, Christian Federmann, Xuedong Huang, Marcin Junczys-Dowmunt, Will Lewis, Mu Li, Shujie Liu, The-Yan Liu, Renqian Luo, Arul Menezes, Tao Qin, Frank Seide, Xu Tan, Fei Tian, Lijun Wu, Shuanghi Wu, Yingce Xia, Dongdong Zhang, Zhirui Zhang, and Ming Zhou. 2018. Achieving human parity on automatic chinese to english news translation. _arXiv preprint arXiv:1803.05567_

Philipp Koehn. 2004. Statistical significance tests for machine translation evaluation. In _Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing_.
* Kudo and Richardson (2018) Taku Kudo and John Richardson. 2018. SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_.
* Lam et al. (2021) Tsz Kin Lam, Shigehiko Schamoni, and Stefan Riezler. 2021. Cascaded models with cyclic feedback for direct speech translation. In _Proceedings of the 2021 IEEE International Conference on Acoustics, Speech and Signal Processing_.
* Li et al. (2018) Xiang Li, Haiyang Xue, Wei Chen, Yang Liu, Yang Feng, and Qun Liu. 2018. Improving the robustness of speech translation. _arXiv preprint arXiv:1811.00728_.
* Martucci et al. (2021) Giuseppe Martucci, Mauro Cettolo, Matteo Negri, and Marco Turchi. 2021. Lexical modeling of ASR errors for robust speech translation. In _Proceedings of Interspeech_.
* Padfield and Cherry (2021) Dirk Padfield and Colin Cherry. 2021. Inverted projection for robust speech translation. In _Proceedings of the 18th International Conference on Spoken Language Translation_. Association for Computational Linguistics.
* Post (2018) Matt Post. 2018. A call for clarity in reporting BLEU scores. In _Proceedings of the Third Conference on Machine Translation: Research Papers_.
* Pratap et al. (2020) Vineel Pratap, Qiantong Xu, Anuroop Sriram, Gabriel Synnaeve, and Ronan Collobert. 2020. Mls: A large-scale multilingual dataset for speech research. _arXiv preprint arXiv:2012.03411_.
* Radford et al. (2022) Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2022. Robust speech recognition via large-scale weak supervision. _arXiv preprint arXiv:2212.04356_.
* Sato et al. (2019) Motoki Sato, Jun Suzuki, and Shun Kiyono. 2019. Effective adversarial regularization for neural machine translation. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_.
* Saunshi et al. (2019) Nikunj Saunshi, Orestis Plevrakis, Sanjeev Arora, Mikhail Khodak, and Hrishikesh Khandeparkar. 2019. A theoretical analysis of contrastive unsupervised representation learning. In _International Conference on Machine Learning_.
* Sperber et al. (2017) Matthias Sperber, Jan Niehues, and Alex Waibel. 2017. Toward robust neural machine translation for noisy input sequences. In _Proceedings of the International Workshop on Spoken Language Translation_.
* Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, L ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In _Advances in Neural Information Processing Systems_, volume 30.
* Wang et al. (2021) Changhan Wang, Morgan Riviere, Ann Lee, Anne Wu, Chaitanya Talnikar, Daniel Haziza, Mary Williamson, Juan Pino, and Emmanuel Dupoux. 2021. VoxPopuli: A large-scale multilingual speech corpus for representation learning, semi-supervised learning and interpretation. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_.
* Wang et al. (2022) Shuaiibo Wang, Yufeng Chen, Songming Zhang, Deyi Xiong, and Jinan Xu. 2022. Adversarially improving NMT robustness to ASR errors with confusion sets. In _Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_.
* Wei et al. (2021) Xiangpeng Wei, Rongxiang Weng, Yue Hu, Luix Xing, Heng Yu, and Weihua Luo. 2021. On learning universal representations across languages. In _International Conference on Learning Representations_.
* Wei et al. (2022) Xiangpeng Wei, Heng Yu, Yue Hu, Rongxiang Weng, Weihua Luo, and Rong Jin. 2022. Learning to generalize to more: Continuous semantic augmentation for neural machine translation. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics_.
* Weiss et al. (2017) Ron J. Weiss, Jan Chorowski, Navdeep Jaitly, Yonghui Wu, and Zhifeng Chen. 2017. Sequence-to-sequence models can directly translate foreign speech. In _Proceedings of the Conference of the International Speech Communication Association_.

## Appendix A Implementation Details

### Training Details and Hyperparamters

We implement our approach and [20] using Fairseq3. For wang2021learning, we randomly selected 14% tokens of sentences for a replacement to generate adversarial examples based on the ASR word error rate. All experiments are performed on A100 GPU, and gradient accumulation and batch size are set to 8 iterations and 8k tokens. For CLAD-ST, our NMT model is pretrained for the first 50K batches for NMT loss in the curriculum. For evaluation, the beam size and length penalty are set to 4 and 0.6 for both tasks. Hyperparameters for all the models are shown in Table 7.

 



### Datasets

Dataset statistics can be found in Table 8. WMT'16 En-De training data4 includes Europarl, Common Crawl corpus, and News Commentary corpora. WMT'14 En-Fr5 training data includes Europarl, Common Crawl, UN corpus, News Commentary, and Fr-En Gigaword corpus.

Footnote 4: [https://www.statmt.org/wmt16/translation-task.html](https://www.statmt.org/wmt16/translation-task.html)

\begin{table}
\begin{tabular}{c|c c} \hline Hyperparameter & Baseline & CONF-ASR & CSA-NMT \\ \hline \hline encoder layers & & 6 \\ encoder embed dim & & 256 \\ encoder ffn embeded dim & & 2048 \\ encoder attention heads & & 8 \\ decoder layers & & 6 \\ decoder embedd dim & & 256 \\ decoder ffn embeded dim & & 2048 \\ decoder attention heads & & 8 \\ dropout & & 0.3 \\ optimizer & adam \\ adam-\(\beta\) & & \((0.9,0.98)\) \\ clip-norm & & 0.0 \\ lr scheduler & & invers-sqrt \\ learning rate & & Se-4 \\ warmup-updates & & 4000 \\ label-smoothing & & 0.1 \\ max tokens & 12800 & 8192 & 6144 & 8192 \\ num-initial-updates-nmt & N/A & 50000 & 50000 & 50000 \\ csannt-semantic-samples & N/A & N/A & 20 & N/A \\ \hline \end{tabular}
\end{table}
Table 7: Model hyperparameters

\begin{table}
\begin{tabular}{|c|c|c|c|} \hline
**Type** & **Name** & **Lang.** & **\#Examples** \\ \hline MT Data & \begin{tabular}{c} WMT’16 \\ WMT’14 \\ \end{tabular} & \begin{tabular}{c} En-De \\ En-Fr \\ \end{tabular} & \begin{tabular}{c} 4.SM \\ 40 M \\ \end{tabular} \\ \cline{2-4}  & \begin{tabular}{c} MuST-C \\ train \\ \end{tabular} & \begin{tabular}{c} En-De \\ En-Fr \\ \end{tabular} & \begin{tabular}{c} 251K \\ 275K \\ \end{tabular} \\ \hline ASR Data & \begin{tabular}{c} VoxPopuli \\ Commovice \\ MLS \\ \end{tabular} & \begin{tabular}{c} En \\ EN \\ \end{tabular} & \begin{tabular}{c} 0.18M \\ 0.99M \\ 11M \\ \end{tabular} \\ \hline Dev & MuST-C Dev & \begin{tabular}{c} En-De \\ En-Fr \\ \end{tabular} & \begin{tabular}{c} 1415 \\ 1412 \\ \end{tabular} \\ \hline Test & \begin{tabular}{c} MuST-C \\ test-COMMON \\ \end{tabular} & \begin{tabular}{c} En-De \\ En-Fr \\ \end{tabular} & 
\begin{tabular}{c} 2580 \\ 2632 \\ \end{tabular} \\ \hline \end{tabular}
\end{table}
Table 8: Statistics of datasets used in the experiments.