CLAD-ST: Contrastive Learning with Adversarial Data for Robust Speech Translation
Authors: Sathish Reddy Indurthi
Shamil Chollampatt
Ravi Agrawal
Marco Turchi
Venue: EMNLP 2023 Main (2023)
Keywords: robust speech translation
contrastive learning
TL;DR: designing robust speech translation models to mitigitate error propagation in cascaded speech translation

Abstract: The cascaded approach continues to be the most popular choice for speech translation (ST). This approach consists of an automatic speech recognition (ASR) model and a machine translation (MT) model that are used in a pipeline to translate speech in one language to text in another language. MT models are often trained on the well-formed text and therefore lack robustness while translating noisy ASR outputs in the cascaded approach, degrading the overall translation quality significantly. We address this robustness problem in downstream MT models by forcing the MT encoder to bring the representations of a noisy input closer to its clean version in the semantic space.  This is achieved by introducing a contrastive learning method that leverages adversarial examples in the form of ASR outputs paired with their corresponding human transcripts to optimize the network parameters. In addition, a curriculum learning strategy is then used to stabilize the training by alternating the standard MT log-likelihood loss and the contrastive losses. Our approach achieves significant gains of up to 3 BLEU scores in English-German and English-French speech translation without hurting the translation quality on clean text.

CLAD-ST:
Contrastive Learning with Adversarial Data for Robust Speech Translation
Sathish Reddy Indurthi, Shamil Chollampatt,
Ravi Agrawal, Marco Turchi
Zoom Video Communications sathishreddy. indurthi@zoom us, shamil. shollampatt@zono us, ravi. agr awal@zoom. us, marco. turchi iahhizzzomuu us
Abstract
Human Transcript: I'm sure that's wise, given the im- portance of the problem, but there's the geoengineer- ing discussion about: Should that be the back pocket case things huppen faster, this innovation goes lot slower than expect?
The cascaded approach continues be the most popular choice for speech translation (ST). This approach consists of an automatic speech recognition (ASR) model and machine trans- lation (MT) model that are used in a pipeline translate speech in one language to text in another language. MT models are often trained on well- formef text and therefore lack robust- ness while translating noisy ASR outputs in the cascaded approach, degrading the overall trans- lation quality significantly. We address this robustness problem in downstream MT models by forcing the MT encoder to bring the repre- sentations of noisy input closer to its clean version in the semantic space. This is achieved by introducing a contrastive learning method that leverages adversarial examples in the form of ASR outputs paired with their correspond- ing human transcripts to optimize the network parameters. In addition, curriculum learning strategy is then used to stabilize the training by alternating the standard MT log-likelihood loss and the contrastive losses. Our approach achieves significant gains of up to 3 BLEU scores in Binglish-Germnn and English-French speech translation without hurting the transla- tion quality on clean text.
not sure that's why it's given the portance the problem. But now that the geoengineering discussion about should that be in back pocket case things happen faster or this innovation goes lot slower than we expect.
ASR Output:
Table 1: Example human transcript and OpenAl Whis- per ASR output from the MuST-C En-De devset. The underlined text indicates erroneous phrases and itali- cized indicates punctuation errors.
been measured due to i) error propagation from the ASR and ii) the mismatch between training-testinng condition as the NMT model is typically trained on well-formed text making it weak in dealing with noisy inputs. For these reasons, there has been significant effort towards building end-to-end ST models (Bérard et al., 2016; Weiss et al., 2017). However, due to the lack of sufficient speech trans- lation datasets in many languages and the flexibility to independently optimize ASR and MT systems, cascaded systems continue to be dominant ap- proach (Anastasopoulos et al., 2022).
1 Introduction
Prior research has tried to tackle the robustness problem in NMT models independently by (1) syn- thetic noise injection (Belinkov and Bisk. 2018), including mimicking ASR errors for the ST task (Sperber et al., 2017: Li et al., 2018; Martucci al.. 2021; Wang et al., 2022) or (2) by adding noise to the continuous-space representations learned by the NMT model (Sato et al., 2019; Cheng et al., 2020; Wei et al., 2022). Some of the synthetic noise injec- tion methods replace words with probable words from ASR confusion lattices (Martucci et al., 2021: Wang et al., 2022) but fail to consider the full con- text and realistically replicate ASR errors, like the phrase-level errors and segmentation errors that ASR models often make (see Table 1). On the other
Neural machine translation (NMT) has made signif- icant advancements over the past several years with claims of achieving 'human parity' (Hassan al., 2018), and "super-human performance' (Barrault et al., 2019). However, despite making tremendous progress in quality and coverage (Costa-jussà et al., 2022), NMT models have been identified to lack robustness in dealing with noisy inputs (Belinkov and Bisk, 2018).
Robustness especially important in cascaded speech translation (ST) systems, where an NMT model works on the output of the upstream auto- matic speech recognition (ASR) system. In this sce- nario, significant MT performance degradation has

---

encoder and decoder components. Training the network parameters θ is done by mini- mizing the mean negative log- log-likelihood over a batch of n source-target text translation pairs (X, = y())))))), (x(()))))))), (x(m))((m) This ob- jective encourages producing a target sentence y with sequence of tokens . Mly] given the source sentence with sequence of tokens 21, 22...... xxx]]
soffmax
oncoder
decoder
[CTS]]
encoder
erccoder
yy( log P(y) (
[LLS]
XCLA
LNMT
+)
Figure I: Proposed model and the two alternating train- ing objectives. ΣNMT is the standard NMT negative log-likelihood loss and LCTL is the contrastive loss for improving robustness to ASR outputs.
To improve the robustness of the NMT model on noisy ASR outputs for cascaded speech translation, we use contrastive learning method (Wei et al., 2021) that is aimed to bring the representations of a noisy sentence generated by the upstream ASR closer to its clean version (the human transcript for the same audio) in the semantic space modeled by the NMT encoder.
hand, the latter approaches predominantly generate arbitrary or random error distributions and over- look the real-world error signals from the actual upstream ASR model.
To get the encoder sentence representations ef- ficiently for contrastive learning, a [CLS] token is prepended to the input sentences similar the BERT model (Devlin al., 2019). We found that using encoder output corresponding to the [CLS] token is slightly better than using the mean of en- coder outputs as the sentence representation. Sx- which also gets added to each of the decoder out- puts before softmax (see Figure 1).
To address the robustness problem, particularly in the context of cascaded ST. we propose to com- bine the best of the two approaches. This is ob- tained by training the NMT model with adversarial examples generated from the ASR outputs and en- couraging the encoder representations of both the ASR outputs and their corresponding human tran- scripts to be closer to each other. This is done via contrastive learning (Saunshi et al., 2019), in addi- tion the standard log-likelihood NMT training. and through curriculum learning (Bengio et al., 2009) by pretraining on log-likelihood loss fol- lowed by iteratively alternating between the two objectives. Our method leads to significant im- provements in the ST task on two language pairs (En-De, En-Fr) without hurting the performance on MT of well-formed texts. Our approach also follows the standard cascade approach that does not rely on any end-to-end ST data (speech aligned with their transcription and translation) unlike that Bahar et al. (2021) and Lam et al. (2021), mak- ing it easier to apply to wider range of language pairs.
Contrastive learning uses speech transcription corpora (i.e., speech paired with human transcripts).
The speech input is passed through the upstream ASR model obtain the ASR outputs. Given the noisy ASR output & paired with its corresponding clean human transcript x, we minimize the con- trastive objective LCTL. which is an average of two symmetric sentence-level contrastive loss func- tions.
Given batch of n examples (X,X) {(x(1, (1)), ...., (x(()) x(()))
LCTL. 2n
1
(x)(x() + Lc.()()) (1)
1=1
where
2 Method
exp (sxxx Sx)) Lo(x) = exp (D(sx, Sx)) + > exp D(sx: 8x))
Our NMT model is Transformer model (Vaswani et al., 2017) with several encoder and decoder layers, with self-attention modules within each block and a cross-attention module to link the
Du,, v) denotes the cosine distance between two
vectors u and v. &y represents a negative example

---

constructed for every other sentence x' in the batch. Given x' E (XU embedding Spr
En-De
En-Fr
NMT
MT
MT
U X)
xx and its sentence
Transformer-
28.1
23.5
36.1
30.8
CONF-ASR (Waig at. 2022)
28.2
25.2
36.3
32.0
CSA-NMT (Wei ct. 2021)
29.5
24.4
37.2
31.3
CLAD-ST
28.8
26.6
36.5
33.4
8x + >xx(sxx - 8x) if, df > dx otherwise
Table 2: Performance of various models on MT and ST tasks on MuST-C tst-COMMON.
8x =
5xx
where dt |sx 12 dx Sxx 12, and Ax = di
respectively. We evaluate against the English and French reference translations for the respective translation directions. We use case-sensitive, deto- kenized BLEU computed using sacrebleu' (Post, 2018). We also compute statistical significance using paired bootstrap resampling (Koehn, 2004). Models: We use a base Transformer (Vaswani et al., 2017) our baseline NMT model, which extend to support our proposed approach, Con- trastive Learning with Adversarial Data for Speech Translation (CLAD-ST). We also compare against two recent state-of-the-an ofrtt--artttttt approaches for NMT ro- bustness which we re-implement: (1) CONF-ASR (Wang et al., 2022) generates adversarial examples by substituting random source tokens with can- didate tokens from ASR confusion set based on their embedding distances to the original token. (2) CSA-NMT (Wei et al., 2022)² uses separate mantic encoder to model a semantic region around the source and target sentences. To improve NMT robustness, samples from this region are used to augment the NMT decoder representations.
The above linear interpolation in Eq with exponentially decaying Ax is implemented fol- lowing Wei et al. (2021) to increase the hardness of negative examples by not selecting uninformative negative examples training progresses. Lx(x) follows similar formulation (x) with x re- placed by x, and vice versa.
dx
To stabilize the NMT parameters before apply- ing contrastive learning, we also use curriculum learning (Bengio et al., 2009). We first pre-train the model with NMT loss .NMT only for the first N batches, followed by iteratively training alter- nate batches with LCTL and ΣNMT losses, respec- tively, until convergence. With this model and train- ing regime, which alternates between training with translation and speech recognition datasets indepen- dently, we are able to build robust NMT models for cascaded ST without any requirement for speech translation data.
3 Experimental Setup
We experiment with English-German (En-De) and English-French (En-Fr) language directions. Training Data: For parallel text translation data, we use WMT' 16 En-De (Bojar et al., 2016) and WMT' 14 En-Fr (Bojar et al., 2014) news transla- tion task data for the corresponding language direc- tions. To get the ASR data for contrastive learning, we use Open AI Whisper base ASR (Radford et al., 2022) to transcribe the speech in the training sets from Mozilla Common Voice 12.0 (Ardila al..
We tokenize all datasets using sentencepiece model (Kudo and Richardson, 2018) for each lan- guage pair on the text translation datasets with a shared source-target vocabulary size of 32,000. We use OpenAl Wispher (base) ASR as our main up- stream model. We also evaluate the ST perfor- mance of our model using Whisper large ASR. Other training details and hyperparameters are in Appendix A.I.
2020), VoxPopuli (Wang et al., 2021), and the En- glish set of Multilingual LibrSpeech (Pratap et al., 2020) for both language directions. As an addi- tional experiment, we also try adding in-domain MuST-C training data. The full statistics of the datasets are in Appendix A.2 (Table 8).
4 Results and Analysis
4.1 Overall Performance
Our proposed CLAD-ST achieves 26.6 and 33.4 BLEU the En-De and En-Fr ST tasks, sig- nificantly (p < 0.001) outperforming the Trans- former baseline by 3.1 BLEU points and 2.6 BLEU
Evaluation: We use MuST-C (Di Gangi et al., 2019) tst-COMMON as our test and MuST-C dev as our validation set. For MT and ST tasks, the inputs to the model are the Whisper (base) ASR outputs and the corresponding human transcripts,
https:/gituub.om//jjoossssssee nrefs: ccase: tok: 3al smooth:exp "Code and network hyperparameters are from https//gitthh.co//mmmycccii
Signature:

---

32
4.3 Curriculum Learning
baseline
CLAD-ST
We study the effect the curriculum learning strat- egy (Section 2) and reports the results in Table 3. We find that curriculum learning strategy brings an improvement of 0.5 BLEU on En-De and 0.7 BLEU On En-Fr ST tasks, respectively.
CONF-ASR
28
CSA-NMT
26
BLEU
24
22
Model
En-De
En-Fr
18
Transformer-base
23.5
30.8
CLAD-ST
26.6
33.4
w/o Curriculum learning
26.1
32.7
ASR Errors
Table 3: Performance on ST tasks with and without
Figure 2: BLEU scores compared to the number of token-level emrors in the ASR outputs En-De ST task on MuST-C Ist-COMMON dataset.
curriculum learning on MuST-C tst- Is--COMMON.
4.4 Out-of-distribution ASR
In Table 4, we report the performance of the En- De systems using Whisper base as the ASR model during training and a higher-quality ASR (Whisper large) during testing. Although training on lower- quality ASR outputs with more errors, similar BLEU score improvement is also noticed for all the systems when fed better ASR outputs. However, CLAD-ST achieves the largest improvements over the baseline, showing that it better generalizes over different ASR-quality outputs.
points (see Table 2). Our approach consistently per- forms better than other basclines (CONF-ASR and CSA-NMT), yielding gains of 1.4 BLEU points (p < 0.001) on both language directions. The improvement of our approach over CONF-ASR shows the importance of contextual modeling of the ASR errors instead of simulating them with ASR confusion lattices. CSA-NMT achieves the largest improvement in the MT task but fails to spe- cialize to ASR errors, yielding lower improvements compared to approach.
En-De
Model
Whase (WER 14.3)
Wlarge
(WER 7.6)
Transformer-
23.5
24.4
4.2 Robustness
CONF-ASR (Wang at. 2022)
25.2
26.3
CSA-NMT (Wei ct. 2023)
24.4
25.8
In Figure 2, we report the performance of all the Ein De models as the number of ASR errors increases in the input sequence. We group the ASR outputs based on the number of tokens different from the reference transcripts on the test set. The perfor- mance of the Transformer baseline model drasti- cally drops as the number of ASR errors increases. CONF-ASR performs better than the Transformer baseline in all the ASR error conditions and closely with CLAD-ST on ASR outputs with one or two errors. However, CLAD-ST notably outperforms as the number of errors increases. CSA-NMT per- forms slightly better than the baseline Transformer when the number of errors is few and performs slightly better than CONF-ASR when increasing the number of errors due to their semantic aug- mentation. Overall, our approach has shown to be robust against sequences having several ASR er- rors and performs consistently better than the other baselines, irrespective of the number of errors in the sequence.
CLAD-ST
26.6
27.4
Table BLEU scores on MuST-C En-De tst-
COMMON ST task when using Whisper-large (Wlarge) vs. Whisper-bass (Wbase) ASR. Word error rate (WER) of the ASR model is provided.
4.5 Model Size
To investigate our results scale when we increase the size of the network, we use Transformer-large instead of Transformer- base for our baseline mod- els as well as CLAD-ST. The results are reported in Table 5. We find that we get improvements of 1.5 and 1.3 BLEU score points for the ST task on En-De and En-Fr, respectively, compared to CONF. ASR, demonstrating its advantage even larger NMT architectures.
4.6 In-domain Training Data
We also experiment with adding in-domain training data from MuST-C for training the NMT models in

---

NMT
En-De
En-Fr
and projected them the target side as well. Bahar et al. (2021). on the other hand, proposes jointly optimize the ASR and MT components in the cas- apeech-translation caded system using speech translation data. Simi- larly, Lam et al. (2021) also uses speech-translation data to reduce error propagation between the two components by mutually training the MT system on ASR n-best outputs and using cyclic feedback from MT outputs for self-training of the ASR model.
MT
ST
MT
ST
Transformer-big
29.6
24.9
38.7
33.3
CONF- CONE-ASR (Wang at. 2023)
29.8
26.6
38.8
34.6
CSA-NMT (WeL atal. 2023)
31.1
25.7
39.9
33.7
CLAD-ST
30.3
28.1
39.1
35.9
Table Performance of using Transformer-large as the model architecture on MT and ST tasks on MuST-C Ist-COMMON.
CLAD-ST and the baselines (see Table 6). In line with our expectations, we find a notable improve- ment in performance of all the systems. Despite the improved baselines, we observe similar improve- ments for CLAD-ST of 1.4 BLEU on both En-De and En ST tasks compared to CONF-ASR, and 2.4 and 2.2 BLEU En-De and En-Fr ST tasks, respectively, compared CSA-NMT.
Conclusion
We improve the robustness MT to ASR outputs using contrastive learning to bring the represen- tations of clean and noisy examples closer in the semantic space. Our approach does not require any speech translation corpus. We significantly improve the translation accuracy on noisy ASR outputs without degrading translation accuracy on clean text. We also show that the approach is scal- able to better-quality ASR models in the cascade other than the one used during training. The pro- posed approach is generic and is applicable beyond the context of speech translation alone, such as translating user-generated chat text non-native text if paired noisy-cleat data is available.
En-De
En-Fr
NMT
M1
ST
MT
ST
Transformer-buse
31.0
37.9
32.9
CONF -SRR (Wang stal, 2022)
30.9
27.9
38.0
33.9
CSA-NMT (Wai etal. 2021)
32.1
26.9
38.8
33.1
CLAD-ST
31.6
29.4
38.2
35.3
Table 6: Performance of various models on MT and ST tasks on MuST-C Ist-COMMON when adding in- domain MuST-C training data.
Limitations
The limitations of our paper are:
5 Related Work
As in any ST cascade architecture, the per- formance of our MT system depends on the quality of the ASR model outputs. Since we use examples generated by the ASR train the MT model, the quality of ASR outputs directly affects the model performance. our work we tested two ASR systems having different quality (base: 14.3 WER and large: 7.6 WER). This is particularly relevant when English is not the source language side.
Several prior works tried improve the robustness of MT models to noisy ASR inputs for cascaded speech translation. Sperber et al. (2017) randomly sampled synthetic noise the form of insertions, deletions, and substitutions and injected them into the source side of MT training data. Li et al. (2018) encoded the syllables (Pinyin) for Chinese ST and injected synthetic character-level substitutions to simulate ASR errors. In contrast, Martucei al. (2021) trained a lexical noise model using confu- sion sets extracted by aligning the actual ASR out- put to reference transcript to inject synthetic errors. Wang et al. (2022) picked substitution candidates from the ASR confusion set that are farthest away from the source token embeddings leading to more robust MT models. Cheng et al. (2019) also used the actual ASR output in training incorporating an additional discriminator network that tries dis- tinguish between human transcript and the ASR output following Cheng et al. (2018) improve ro- bustness. Padfield and Cherry (2021) simulated not only token-level errors but also segmentation errors
Evaluation is done using only the BLEU score. We did not use human evaluation or COMET,
The evaluation is limited to two language pairs having English as the source. This selection the source language is quite important because having no-English language as a source will expose our MT model to a) ASR models of probably lower quality and b) different and more varied linguistic challenges that might affect the work of the adversarial method.

---

References
statistical machine translation. In Proceedings of the Ninth Workshop on Statistical Machine Translation.
Antonios Anastasopoulos, Loic Barrault, Luisa Ben- tivogli, Marcely Zanon Boito, Ondřej Bojar, Roldano Cattoni, Anna Currey, Georgiana Dinu, Kevin Duh, Maha Elbayad, Clara Emmanuel, Yannick Estève, Marcello Federico, Christian Federmann, Souhir Gahbiche, Hongyu Gong, Roman Grundkiewicz, Barry Haddow, Benjamin Hsu, Dávid Javorský, Vêra Kloudová, Surafel Lakew, Xutai Ma, Prashant Mathur, Paul McNamee, Kenton Murray, Maria Nädejde, Satoshi Nakamura, Matteo Negri, Jan Niehues, Xing Niu, John Ortega, Juan Pino, Eliz- abeth Salesky, Jiatong Shi, Matthias Sperber, Se- bastian Stüker, Katsuhito Sudoh, Marco Turchi, Yo- gesh Virkar. Alexander Waibel, Changhan Wang, and Shinji Watanabe. 2022. Findings of the IWSLT 2022 evaluation campaign. In Proceedings of the 19th In- ternational Conference on Spoken Language Trans- lation.
Ondfej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias Huck, An- tonio Jimeno Yepes, Philipp Kochn, Varvara Lo- gacheva, Christof Monz, Mattco Negni, Aurélie Névéol, Mariana Neves, Martin Popel, Matt Post, Raphael Rubino, Carolina Scarton, Lucia Specia, Marco Turchi, Karin Verspoor, and Marcos Zampieri. 2016. Findings of the 2016 conference on machine translation. In Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Pa- pers, Berlin, Germany.
Qiao Cheng, Meiyuan Fan. Yaqian Han, Jin Huang, and Yitao Duan. 2019. Breaking the data barrier: To- wards robust speech translation via adversarial stabil- ity training. In Proceedings of the 16th International Conference on Spoken Language Translation.
Rosana Ardila, Megan Branson, Kelly Davis, Michael Kohler, Josh Meyer, Michael Henretty, Reuben Morais, Lindsay Saunders, Francis Tyers, and Gre- gor Weber. 2020. Common voice: A massively- multilingual speech corpus. In Proceedings of the Twelfth Language Rexources and Evaluation Confer- ence.
Yong Cheng, Lu Jiang, Wolfgang Macherey, and Jacob Eisenstein. 2020. AdvAug: Robust adversarial aug- mentation for neural machine translation. In Proceed- ings of the 58th Annual Meeting of the Association for Computational Linguistics.
Yong Cheng, Zhaopeng Tu, Fandong Meng, Junjie Zhai, and Yang Liu. 2018. Towards robust neural machine translation. In Proceedings of the 56th Annual Meet- ing of the Association for Computational Linguisties.
Parnia Bahar, Tobias Bieschke, Ralf Schlüter, and Her- mann Ney. 2021. Tight integrated end-to-end training for cascaded speech translation. In Proceedings of the 2021 IEEE Spoken Language Technology Work- shop.
Marta R Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad. Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, et al. 2022. No language left behind: Scaling human-centered machine translation. arXiv preprint 22Xi:2200700467..
Loic Barrault, Ondřej Bojar, Marta R. Costa-jussä, Christian Federmann, Mark Fishel, Yvette Gra- ham, Barry Haddow, Matthias Huck, Philipp Koehn, Shervin Malmasi, Christof Monz, Mathias Müller, Santanu Pal, Matt Post, and Marcos Zampieri. 2019, Findings of the 2019 conference on machine transla- tion (WMT19). 19). In Proceedings of the Fourth Confer- ence on Machine Translation.
Jacob Devlin, Ming-War Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.
Yonatan Belinkov and Yonatan Bisk. 2018. Synthetic and natural noise both break neural machine transla- tion. In International Conference on Learning Rep- resentations.
Mattia A. Di Gangi, Roldano Cattoni, Luisa Bentivogli, Matteo Negri, and Marco Turchi. 2019. MuST CC: a Multilingual Speech Translation Corpus. In Proceed- ings of the 2019 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies, Volume (Long and Short Papers).
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. 2009. Curriculum learning. In Proceedings of the 26th Annual International Confer- ence on Machine Learning.
Alexandre Bérard, Olivier Pietquin. Laurent Besacier, and Christophe Servan. 2016. Listen and translate: proof of concept for end-to-end speech-to- to-text trans- lation. In Proceedings of the NIPS Workshop on end-to-end learning for speech and audio process- ing.
Hany Hassan, Anthony Aue, Chang Chen, Vishal Chowdhary, Jonathan Clark, Christian Federmann, Xuedong Huang. Marcin Junezys-Dowmunt Will Lewis. Mu Li, Shujie Liu, Tie- Yan Liu, Renqian Luo, Arul Menezes. Tao Qin, Frank Seide. Xu Tan, Fei Tian, Lijun Wu, Shuangzhi Wu, Yingee Xia. Dongdong Zhang, Zhirui Zhang, and Ming Zhou. 2018. Achieving human parity automatic chi- nese to english news translation. arXiv preprint 108
Ondfej Bojar, Christian Buck, Christian Federmann. Barry Haddow, Philipp Koehn, Johannes Leveling, Christof Monz, Pavel Pecina, Matt Post, Herve Saint- Amand, Radu Soricut, Lucia Specia, and Aleš Tam- chyna. 2014. Findings of the 2014 workshop on

---

Philipp Koehn. 2004, Statistical significance tests for machine translation evaluation. In Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, L ukasz Kaiser, and Illia Polosukhin. 2017, Attention is all you need. In Advances in Neural Information Pro- cessing Systems, volume 30.
Taku Kudo and John Richardson. 2018. SentencePiece:
simple and language independent subword tok- enizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference Empirical Methods in Natural Language Processing: System Demonstrations.
Changhan Wang, Morgane Riviere, Ann Lee, Anne Wu, Chaitanya Talmikar, Daniel Haziza, Mary Williamson, Juan Pino, and Emmanuel Dupoux. 2021. VoxPopuli: A large-scale multilingual speech corpus for represen- tation learning, semi-supervised learning and inter- pretation. In Proceedings of the 59th Annual Meeting of the Association for Compulational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers).
Tsz Kin Lam, Shigehiko Schamoni, and Stefan Riezler. 2021. Cascaded models with cyclic feedback for direct speech translation. Proceedings of the 2021 IEEE International Conference on Acoustics, Speech and Signal Processing.
Shuaibo Wang, Yufeng Chen, Songming Zhang, Deyi Xiong, and Jinan Xu. 2022. Adversarially improving NMT robustness to ASR errors with confusion sets. In Proceedings of the 2nd Conference of the Asia- Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Confer- ence on Natural Language Processing (Volume 2: Short Papers).
Xiang Li, Haiyang Xue, Wei Chen, Yang Liu, Yang Feng, and Qun Liu. 2018. Improving the ro- bustness of speech translation. arXiv preprint 1oXiv:1811.0072
Giuseppe Martucci, Mauro Cettolo, Matteo Negri, and Marco Turchi. 2021. Lexical modeling of ASR er- rors for robust speech translation. In Proceedings of Interspeech.
Xiangpeng Wei, Rongxiang Weng, Yue Hu, Luxi Xing. Heng Yu, and Weihua Luo. 2021. On learning univer- sal representations across languages. In International Conference on Learning Representations.
Dirk Padfield and Colin Cherry. 2021. Inverted projec- tion for robust speech translation. In Proceedings of the 18th International Conference on Spoken Lan- guage Translation. Association for Computational Linguistics.
Xiangpeng Wei, Heng Yu, Yue Hu, Rongxiang Weng, Weihua Luo, and Rong Jin. 2022. Learning to gen- eralize to more: Continuous semantic augmentation for neural machine translation. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics.
Matt Post. 2018. A call for clarity in reporting BLEU scores. Proceedings of the Third Conference on Machine Translation: Research Papers.
Ron J. Weiss, Jan Chorowski. Navdeep Jaitly, Yonghui Wu, and Zhifeng Chen. 2017. Sequence-to-sequence models can directly translate foreign speech. In Proceedings of the Conference of the International Speech Communication Association.
Vineel Pratap, Qiantong Xu, Anuroop Sriram, Gabriel Synnaeve, and Ronan Collobert. 2020. Mls: A large- scale multilingual dataset for speech research. arXiv preprint arXh::2012.0341.
A Implementation Details
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brock- man, Christine eavey, and Nlya Sutskever. 2022. Robust speech recognition via large-scale weak su- pervision. arXiv preprint 2r200:221..04356.
A.1 Training Details and Hyperparamters
We implement our approach and (Wang et al., 2022) using Fairseq". For Wang et al. (2021), we ran- domly selected 14% tokens of sentences for a placement generate adversarial examples based on the ASR word error rate. All experiments are performed on A100 GPU, and gradient accumula- tion and batch size are set to 8 iterations and 8k tokens. For CLAD-ST, our NMT model is pre- trained for the first 50K batches for NMT loss in the curriculum. For evaluation, the beam size and length penalty are set to 4 and 0.6 for both tasks. Hyperparameters for all the models are shown in Table 7.
Motoki Sato, Jun Suzuki, and Shun Kiyono. 2019. Ef- fective adversarial regularization for neural machine translation. In Proceedings of the 57th Annual Meet- ing of the Association for Computational Linguistics.
Nikunj Saunshi, Orestis Plevrakis, Sanjeev Arora, Mikhail Khodak, and Hrishikesh Khandeparkar. 2019. A theoretical analysis of contrastive unsu- pervised representation learning- In International Conference on Machine Learning.
Matthias Sperher, Jan Niehues, and Alex Waibel. 2017. Toward robust neural machine translation for noisy input sequences. Proceedings of the International Workshop on Spoken Language Translation.

---

Baseline
CONF-ASR
CSA-NMT CLAD-ST
Hyperparameter
encoder layers encoder embed dim encoder f'n embed dim encoder attention heads decoder layers dccoder embed dim decoder embed dim
256
2048
8
256
2048
decoder attention heads
dropout
0.3
optimizer
adam
adam-B
(0.9, 0.98)
clip-norm
0.0
scheduler
invers-sqrt
learning rate
5e-4
warmup-updates
4000
label- mmoothhn
0.1
max tokens
12800
8192
6144
8192
num ceen-initil--aoe caa--inttall-udates-----ss amt
N/A N/A
50000
50000
50000
semantic-sampplee
N/A
20
N/A
Table 7: Model hyperparameters
Type
Name
Lang.
#Examples
WMT 16
En-De
4.5M
MT Data
14
En-Fr
40 M
MuST-C
En-De
25TK
train
En-Fr
275K
VoxPopuli
En
0.18M
ASR Data
Commonvoice
En
0.99M
ML.S
En
11M
En-De
1415
Dev
MuST-C Dev
Fi-Fr
1412
MuST-C IsL-COMONN
En-De
2580
Test
En-Fr
2632
Table 8: Statistics of datasets used in the experiments.
A.2 Datasets
Dataset statistics can be found in Table 8. WMT' 16 En-De training data includes Europarl, Common Crawl corpus, and News Commentary corpora. WMT' 14 En-Fr training data includes Europarl, Common Crawl, UN corpus, News Commentary, and Fr-En Gigaword corpus.
"https:/www.state statmt. petattt..or//mmttt org/wmt16/ translation-tast taski
'https: / h/ww://wwssccc stannt. wtt.stt..oo///tt1..........
translation-tatt tartiinnttss.hhee html


Reviews:
[{"review_id": "Hmz8zVZ1Ln", "reviewer": "Reviewer_FsQ5", "summary": "", "questions": "- What is the performance of the cascade AST when MuST-C is used in training the ASR and the NMT model? ", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer FsQ5,\n\nWe thank you for your valuable feedback on our paper. Please find our responses below.\n\n > _**”The model size of the NMT is relatively small, i.e., a transformer-based architecture. This might raise concern on the improvement brought by the contrastive loss when scaling up the NMT training.”**_\n\nWe followed comparable work in cascaded speech translation (e.g. Wang et al, 2021) in choosing Transformer (base) as the baseline model. Nonetheless, we believe our approach will show improvement on larger Transformer baselines also since larger models tend to overfit more on the well-formed MT training data. Our proposed method reduces such overfitting via the additional loss term aimed to improve robustness on noisy ASR output. We will include a comparison in the revised paper using bigger Transformer models.\n\n >  _**“What is the performance of the cascade AST when MuST-C is used in training the ASR and the NMT model?”**_\n\nOur approach does not rely on speech translation corpus and hence using a significantly smaller resource such as MuST-C to train ASR and NMT independently is expected to yield a poorer baseline cascaded system. However, we will include additional experiments in the revised version using MuST-C.\n\n **Missing references**: We will add references to Bahar et. al 2021 and Lam et. al 2021. The key difference is that both approaches rely on speech translation data.\n\n We will fix the typos and incorporate both suggestions for improving the presentation.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Compared to research on direct model, there are less work about cascade AST. This paper presents a focused contribution to improve it. \n- The presented method is simple but effective. It does not require triplet of speech-transcription-translation nor speech-translation paired data. It does not require sophisticated changes to the NMT architecture, improving its practicality in production environment. ", "reasons_to_reject": "- The model size of the NMT is relatively small, i.e., a transformer-based architecture. This might raise concern on the improvement brought by the contrastive loss when scaling up the NMT training. \n", "questions_to_authors": "- What is the performance of the cascade AST when MuST-C is used in training the ASR and the NMT model? ", "paper_topic_and_main_contributions": "This paper presents a method, called CLAD-ST, which enhances cascade speech-to-text (AST) translation via improving the robustness of its neural machine translation (NMT) to its speech recognition (ASR) outputs. \n\nThere are two training objectives for the NMT: 1) contrastive learning and 2) cross-entropy training. In contrastive learning, it takes the ASR output and the corresponding gold-reference transcriptions as inputs. Such loss aims to reduce the distance between these two sentences while pushing them away from the other sentences, or called negative examples, resulting in stronger robustness of the NMT encoder towards the ASR output. Each sentence is approximated by the representations of a \"[CLS]\" token that is prepended at the beginning of the source sentence. The cross-entropy training is on the NMT's parallel corpora. In order to stabilise the training, the NMT is first pre-trained on the cross-entropy loss, followed by alternative training between the two losses.\n\nExperiments on En-De and En-Fr language pairs demonstrate the effectiveness of CLAD-ST over their baselines. It is worth noting that while the speech translation corpus \"MuST-C\" is used in evaluation, it is not used in training the NMT model. \n\nOverall, CLAD-ST is a simple but effective method to improve cascade AST. It is rather data-efficient as it does not require the triplet of speech-transcription-translation corpus nor paired speech-translation data. Furthermore, the training part focuses only on the NMT with minimal intervention to the architecture of the existing NMT model. However, a possible weakness of the work would be the relatively weak (compared to the Whisper-large) pre-trained NMT model, which is a shallow transformer model. \n\n\n\n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "- line 051: inflexibility <- flexibility\n- The caption of Figure 1 can be more informative. \n- Please also add the \"signature\" when reporting evaluation metrics that are implemented in sacrebleu.\n\n\n"}, "extra_text": {"Missing_References": "Since there are much recent work on cascade AST, it would be great if there is a comparison between these two methods:\n1) Bahar et. al 2021: Tight integrated end-to-end training for cascaded speech translation\n2) Lam et. al 2021: Cascaded Models With Cyclic Feedback For Direct Speech Translation"}}, {"review_id": "B5XeBZuMFb", "reviewer": "Reviewer_ZhJE", "summary": "", "questions": "A. Did you try a different approach in getting the sentence representation? Like mean or max pooling?\n\nB. What is the impact of curriculum learning in the proposed method?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer ZhJE,\n\nWe thank you for your valuable feedback on our paper. Please find our responses below.\n\n>  _**“The experimental setup is rather limited, which is partly acknowledged in the limitations”**_\n\nWe considered two English-source high-resource language pairs as they tend to produce better cascaded baselines (due to a good quality English ASR and strong NMT models) and should suffer less from error propagation compared to low-resource pairs. This makes them stronger baselines to demonstrate the effectiveness of our approach that aims to reduce this error propagation. With a non-English source low resource pair, we expect to observe similar or better improvements as they tend to suffer more from error propagation. \n\nSeveral prior published work on speech translation rely on these language pairs alone in their experimental setup (ex:  [Tran et al, EMNLP 2022](https://aclanthology.org/2022.emnlp-main.297.pdf) relies only on En-De MuST-C ST benchmark, [Ao et al, ACL 2022](https://aclanthology.org/2022.acl-long.393.pdf) report only on En-De and En-Fr ST tasks from the MuST-C benchmark similar to ours).\n\n>  _**“Using a cascade ST system eliminates the need for parallel ST data, and not the proposed method itself (as claimed)”**_\n\nOur work is on reducing the error propagation between the ASR and MT models and improving robustness of cascaded speech translation. Some prior work, including the two additional references pointed out by Reviewer 1, rely on parallel speech translation data to improve cascaded systems. Our approach does this without it. We will clarify the context of the claim in the revised version.\n\n**Answers to Questions**\n\n> **_A. “Did you try a different approach in getting the sentence representation?”_**\n\nWe tried the mean as the sentence representation. However, [CLS] performed better than the mean (~0.1-0.2 BLEU), so we decided to select [CLS] representation for our experiments. We will include the result in the revised paper. Note that using either  [CLS] or mean representation in our approach performed better than our MT baselines. \n\n> **_B. “What is the impact of curriculum learning in the proposed method?”_**\n\nThe use of curriculum learning improved the performance of our model in both language directions (+0.51 BLEU for En-De and +0.73 BLEU for En-Fr). We will add this in the revised version.\n\n**Typos Grammar Style And Presentation Improvements**\n\nWe will improve the paper by incorporating the suggestions and fixing the typos. Please note that we indeed used the “dev” portion of MuST-C for validation (thanks for pointing out the typo).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is very well written and easy to follow. The proposed method is intuitive, seems to be easily integratable to the MT model training, and yields significant improvements over the baseline and the other two tested methods (CONF-ASR, CSA-NMT) in the two tested language pairs. Furthermore is more robust to more error-prone ASR transcriptions.", "reasons_to_reject": "The experimental setup is rather limited, which is partly acknowledged in the limitations. Only two language pairs are being tested (En -> De/Fr), both of them having English source speech, and being high-resource. Additionally, the experiments are done using Whisper, which in general is a very good quality ASR system. An expanded experimental setup, using lower-quality ASR systems, pairs with non-English source, would shed more light into the usefulness of the proposed method in other non-standard scenarios.\n\nI find the claim of the introduction (L089-L094) a bit misleading. Using a cascade ST system eliminates the need for parallel ST data, and not the proposed method itself (as claimed). Furthermore, in the second part of the same claim, it is stated that \"making it easier to apply to a wider range of language pairs compared to the end-to-end approaches\", but the experiments are only done in the standard pairs that E2E systems are tested, which is a missed opportunity.", "questions_to_authors": "A. Did you try a different approach in getting the sentence representation? Like mean or max pooling?\n\nB. What is the impact of curriculum learning in the proposed method?", "paper_topic_and_main_contributions": "This paper presents a method for improving cascaded speech translation, which usually lacks in robustness since the MT model has to translate error-prone ASR outputs. The authors propose use contrastive learning to train the MT model to bring the representations of the ASR model and the ground truth source text closer in the semantic space. In their experiments using Whsiper as the ASR model, they show that their method can surpass the baseline cascaded-ST and two more similar methods from the literature in two language pairs of MuST-C. They also show that the proposed method is more robust than the others as the number of errors in the ASR output increases.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "L063-L068: which are these methods?\n\nL181-L182: Is this a typo? It seems that you use tst-COMMON both for validation and testing.\n\nL193-L194: This is the first time that the acronym is presented. Consider using the full name here.\n\nL219: EN -> En (for consistency with the rest of the paper)\n\nTable 2: Consider adding the statistical significance in the table and caption.\n\nFigure 2 & L237: number of token vs number of words\n\nFigure 2: add the tested language pair and the test set.\n\nTable 3: This table and its caption are very confusing. Consider reformulating it. Also it is not stated what the metrics are and the tested language pair (or set).\n\nTable 4: Caption says \"statistics of datasets used for training\" but also dev and test set are present."}}, {"review_id": "KwuRPawXw5", "reviewer": "Reviewer_1msj", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer 1msj,\n\nWe thank you for your valuable feedback on our paper. Please find our responses below.\n\n> **_“One of the limitations of the method is the need for human transcripts.”_**\n\nDatasets containing human transcripts are the basement to build the ASR systems and they are available in large quantities and cover a large set of languages (e.g. Common Voice, Multilingual LibriSpeech). These datasets can be used without effort.  In the case of low-resource languages with limited human transcripts, their creation is possible by leveraging the work of monolingual speakers in that particular language. This makes this process much simpler than the generation of speech-translation data that, instead, requires bilingual speakers. It is important to notice that when no human transcripts are available, the main problem becomes the creation of the ASR, making the application of our method a secondary problem.\n\n> **_“Will the performance of the method degrade when the noise in the training samples is enhanced? What is the best magnitude of the ASR noise in training data?“_**\n\nTo show how the noise level impacts the performance, we conducted two experiments using Whisper-base and Whisper-large models (Table 3). The Whisper-base has a WER 14.3 on our test set and Whisper-large has a WER 7.6. Training datasets constructed from these models differ by noise level. In both cases, the proposed approach outperforms the other methods by a  huge margin, however, the model training with less noisy transcripts (generated with a large model) achieves the highest performance.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper is well-written and easy to follow.\n- The proposed method is pretty intuitive and proved to be effective on both clean texts and noisy texts.\n- Further analyses prove that the method is applicable to various levels of noise.", "reasons_to_reject": "- One of the limitations of the method is the need for human transcripts. For the languages or scenarios where human transcripts are difficult to obtain, the application of the method may be limited.\n- As a contrastive learning method, it is uncertain whether the performance of this method will be affected by the noisy level in ASR transcripts during training. Will the performance of the method degrade when the noise in the training samples is enhanced? What is the best magnitude of the ASR noise in training data?\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper aims at improving the robustness of NMT from noisy ASR transcriptions. The authors design a simple yet effective method based on contrastive learning to bridge the discrepancy between the sentence representation of clean texts and real transcriptions. Experimental results prove the effectiveness of the proposed method.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. A missing full stop in the caption of Table 3."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Reviewers agree that the proposed method is simple and effective for improving NMT robustness in a cascade De|Fr-En speech-to-text translation scenario. The main concern that is not addressed by additional experiments promised in the rebuttals is the limited focus on only 2 high resource language pairs. While adding low resource language pairs would indeed strengthen the paper since it is likely to highlight the strengths of the proposed method, the reviewer agrees that this is beyond the scope of a short paper. However, the long paper format might have been a better fit.
Decision: Accept-Main