SlowGuess/ACL_2020_OCR huggingface dataset:
Committed triangle (CTr): Commit to an IR, then produce outputs by conditioning on both inputs and intermediate representation. Anastasopoulos and Chiang (2018), who introduce the triangle model, use it in its marginalizing form (see below). Unexplored variations include the use of discrete transcripts as IR, which interestingly could be seen as a strict generalization of the loosely coupled cascade and should therefore never perform worse than it if trained properly.
Marginalizing triangle (MTr): Produce output by conditioning on both input and IR, while marginalizing over the latter (Eq. 2). Anastasopoulos and Chiang (2018) marginalize by taking an  $n$ -best list, with  $n$  set to only 4 for computational reasons. This raises the question of whether the more computationally efficient lattices could be employed instead. Similar considerations apply to the end-to-end trainable marginalizing cascade.
Joint  $(\mathbf{Jt})$  : Changes the problem formulation to  $\hat{S},\hat{T} = \operatorname *{argmax}_{S\in \mathcal{H},T\in \mathcal{T}}Pr(S,T\mid X)$  . This is a useful optimization for many applications which display both transcripts and translations to the user, yet to our knowledge has never been explicitly addressed by researchers.
4.3 Training Strategies
This group of techniques describes the types of supervision signals applied during training.
Subtask training: Training of sub-components by pairing IRs with either the speech inputs or the output translations. Loosely coupled cascades rely on this training technique while recently proposed cascaded and triangle models often combine subtask training and end-to-end training.
Auxiliary task training: Training by pairing either model inputs or outputs with data from an arbitrary auxiliary task through multi-task training. This technique has been used in two ways in literature: (1) To incorporate ASR and MT data into direct models by using auxiliary models that share parts of the parameters with the main model (Weiss et al., 2017). Auxiliary models are introduced for training purposes only, and discarded during inference. This approach has been found
This definition subsumes pretraining, which is simply using a specific multitask training schedule.
inferior at exploiting ASR and MT data when compared to subtask training (Sperber et al., 2019a). (2) To incorporate various types of less closely related training data, such as the use of multitask training to exploit ASR data from an unrelated third language (Bansal et al., 2019; Stoian et al., 2020).
End-to-end: Supervision signal that directly pairs speech inputs and output translations. This technique is appealing because it jointly optimizes all involved parameters and may lead to better optima. The main limitation is lack of appropriate data, which can be addressed by combined training with one of the alternative supervision types, or by training on augmented data, as discussed next.
4.4 End-to-End Training Data
Manual: Speech utterances for training are translated (and possibly transcribed) by humans. This is the most desirable case, but such data is currently scarce. While we have seen growth in data sources in the past two years ( $\S 2.3$ ), collecting more data is an extremely important direction for future work.
Augmented: Data obtained by either augmenting an ASR corpus with automatic translations, or augmenting an MT corpus with synthesized speech. This has been shown more data efficient than multitask training in the context of adding large MT and ASR corpora (Jia et al., 2019a). Pino et al. (2019) find that augmented ASR corpora are more effective than augmented MT corpora. This approach allows training direct models and end-to-end models even when no end-to-end data is available. Knowledge distillation can be seen as an extension (Liu et al., 2019). An important problem that needs analysis is to what extent mismatched source-language and information loss degrade the augmented data.
Zero-Shot: Using no end-to-end data during training. While augmented data can be used in most situations in which no manual data is available, it suffers from certain biases that may harm the ST model. Similarly to how zero-shot translation enables translating between unseen combinations of source and target languages, it may be worth exploring whether some recent models, such as direct models or cascades with non-discrete IRs, can be trained without resorting to any end-to-end data for the particular language pair of interest.
5 Applications and Requirements
While we previously described the task of ST simply as the task of generating accurate text translations from speech inputs, the reality is in fact much more complicated. Future work may exploit new modeling techniques to explicitly address the aspects drawn out below.
5.1 Mode of Delivery
Batch mode: A (potentially large) piece of recorded speech is translated as a whole. Segmentation into utterances may or may not be given. This mode allows access to future context, and imposes no strict computational restrictions. Typical applications include movie subtitling (Matusov et al., 2019) and dubbing (Saboo and Baumann, 2019; Federico et al., 2020).
Consecutive: Real-time situation where inputs are provided as complete utterances or other translatable units, and outputs must be produced with low latency. A typical example is a two-way translation system on a mobile device (Hsiao et al., 2006). This is the only mode of delivery that allows interaction between speaker and translator (Ayan et al., 2013).
Simultaneous: Real-time situation where latency is crucial and outputs are produced incrementally based on incoming audio stream. Simultaneous translation is faced with an inherent delay vs. accuracy trade-off, such as in a typical lecture translation application (Fügen, 2008). In addition to computational latency, which is relevant also with consecutive translation, simultaneous translation suffers from inherent modeling latency caused by factors including reordering.
5.2 Output Medium
Text: This is a standard setting, but is nevertheless worth discussing in more detail for at least two reasons: (1) as is well-known in the subtitling industry, reading speeds can be slower than speaking and listening speeds (Romero-Fresco, 2009), implying that a recipient may not be able to follow verbatim text translations in case of fast speakers, and that summarization may be warranted. (2) Text display makes repair strategies possible that are quite distinct from spoken outputs: One can alter, highlight, or remove past outputs. One possible way of exploiting this is Niehues et al. (2018)'s strategy of simultaneous translation through re-translation.
7415
ES	:tambienengo umehestoytomandouna clase..
EN	ialso have umeh i'm taking a marketing class..
ES	porquequéva,mjayetacuera de que..
EN	because what is, mhm do you recall now that..
Table 2: Examples for faithful Spanish to English translations, taken from (Salesky et al., 2019b).
Speech: Speech outputs have been used since the early days (Lavie et al., 1996), but whether to apply text-to-speech on top of translated text has often been seen as a question to leave to user interface designers. Here, we argue that ST researchers should examine in what ways speech outputs should differ from text outputs. For example, is disfluency removal (Fitzgerald et al., 2009) beneficial for speech outputs, given that human listeners are naturally able to repair disfluencies (Lickley, 1994)? Further examples that need more exploration are prosody transfer (Aguero et al., 2006) and models that directly translate speech-to-speech (Jia et al., 2019b).
5.3 The Role of Transcripts
Mandatory transcripts: User interface displays both transcripts and translations to the user. This scenario has been implemented in many applications (Hsiao et al., 2006; Cho et al., 2013), but has received little attention in the context of end-to-end ST research. It ties together with the joint inference model ( $\S 4.3$ ). Note that with loosely coupled cascades, there is little need to consider this scenario explicitly because the application can simply display the by-product transcripts to the user. But this is not easily possible with direct models or with models using IRs other than transcripts.
Auxiliary transcripts: Transcriptions are not needed as user-facing model outputs, but may be exploited as IRs during training and possibly inference. This is the most typical formal framing of the ST task, assuming that transcribed training data is useful mainly for purposes of improving the final translation.
Transcript-free: No transcribed training data exists, so the model cannot rely on supervised transcripts as  $IR$ . The main scenario is endangered language preservation for languages without written script, where it is often easier to collect translated speech than transcribed speech (Duong et al., 2016).
5.4 Translation Method
The method of translation is an especially relevant factor in ST, which commonly includes a transfer from the spoken into the written domain. Here, we provide two reference points for the method of translation, while referring to Newmark (1988) for a more nuanced categorization.
Faithful: Keeps the contextual meaning of the original as precisely as possible within the grammatical constraints of the target language. With text as output medium, faithful translation may result in poor readability, e.g. due to the translation of disfluencies (Table 2). Arguably the most appropriate output medium for faithful ST would be speech, although user studies are needed to confirm this. Another application are high-stake political meetings in which translations must stay as close to the original sentence as possible. As we move toward more distant language pairs, the practicability of faithful translation of spoken language with disfluencies becomes increasingly questionable.
Communicative: Demonstrates the contextual meaning of the original such that both content and style are acceptable and comprehensible by the target audience. An important example for improving communicativeness is disfluency removal (Fitzgerald et al., 2009). Given that human translators and interpreters adapt their translation method depending on factors that include input and output medium (He et al., 2016), more research is needed beyond disfluency removal. Communicative translations are especially relevant in casual contexts where convenience and low cognitive effort are mandative. Arguably the closest neighbor of spoken language style in the text realm is social media, it would be interesting to attempt speech-to-text translation with social-media style outputs.
Recent works on end-to-end modeling techniques are motivated by the prospect of overcoming the loosely coupled cascade's inherent issues, yet of the issues outlined in §2.1, often only the goal of avoiding early decisions is mentioned motivationally. While early decisions and data efficiency have been recognized as central issues, empirical insights are still limited and further analysis is needed. Mismatched source-language and information loss are often not explicitly analyzed.
We conjecture that the apparent trade-off between data efficiency and modeling power may explain the mixed success in outperforming the loosely coupled cascade. In order to make progress in this regard, the involved issues (early decisions, mismatched source-language, information loss, data efficiency) need to be precisely analyzed ( $\S 3$ ), and more model variants ( $\S 4$ ) should be explored. As a possible starting point one may aim to extend, rather than alter, traditional models, e.g. applying end-to-end training as a fine-tuning step, employing a direct model for rescoring, or adding a triangle connection to a loosely coupled cascade. We further suggest that more principled solutions to the different application-specific requirements ( $\S 5$ ) should be attempted. Perhaps it is possible to get rid of segmentation as a separate step in batch delivery mode, or perhaps text as output medium can be used to visualize repairs more effectively. Several of the application-specific requirements demand user studies and will not be sufficiently solved by relying on automatic metrics only.
We started this paper with a chronological survey of three decades of ST research, focusing on carving out the key concepts. We then provided definitions of the central challenges, techniques, and requirements, motivated by the observation that recent work does not sufficiently analyze these challenges. We exposed a significant space of both modeling ideas and application-specific requirements left to be addressed in future research.
Our hope is to encourage meaningful and generalizable comparisons on our quest toward overcoming the long-standing issues found in ST models.
P. D. Aguero, Jordi Adell, and Antonio Bonafonte. 2006. Prosody Generation for Speech-to-Speech Translation. In International Conference on Acoustics, Speech, and Signal Processing (ICASSP), Toulouse, France.
Antonios Anastasopoulos and David Chiang. 2018. Tied Multitask Learning for Neural Speech Translation. In North American Chapter of the Association for Computational Linguistics (NAACL), New Orleans, USA.
Gopala Krishna Anumanchipalli, Luis C. Oliveira, and Alan W. Black. 2012. Intent transfer in speech-to-speech machine translation. In Workshop on Spoken
Language Technology (SLT), pages 153-158, Miami, USA.
Necip Fazil Ayan, Arindam Mandal, Michael Frandsen, Jing Zheng, Peter Blasco, Andreas Kathol, Frederic Bechet, Benoit Favre, Alex Marin, Tom Kwiatkowski, Mari Ostendorf, Luke Zettlemoyer, Philipp Salletmayr, Julia Hirschberg, and Svetlana Stoyanchev. 2013. 'Can you give me another word for hyperbaric?: Improving speech translation using targeted clarification questions. In International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 8391-8395, Vancouver, Canada.
Parnia Bahar, Albert Zeyer, Ralf Schlüter, and Hermann Ney. 2019. On Using SpecAugment for End-to-End Speech Translation. In International Workshop on Spoken Language Translation (IWSLT), Hongkong.
Srinivas Bangalore and Giuseppe Riccardi. 2001. A Finite-State Approach to Machine Translation. In North American Chapter of the Association for Computational Linguistics (NAACL), Pittsburgh, USA.
Sameer Bansal, Herman Kamper, Karen Livescu, Adam Lopez, and Sharon Goldwater. 2018. Low-Resource Speech-to-Text Translation. In Annual Conference of the International Speech Communication Association (InterSpeech), Hyderabad, India.
Sameer Bansal, Herman Kamper, Karen Livescu, Adam Lopez, and Sharon Goldwater. 2019. Pretraining on high-resource speech recognition improves low-resource speech-to-text translation. In North American Chapter of the Association for Computational Linguistics (NAACL), Minneapolis, USA.
Sameer Bansal, Herman Kamper, Adam Lopez, and Sharon Goldwater. 2017. Towards speech-to-text translation without speech recognition. In European Chapter of the Association for Computational Linguistics (EACL).
Daniel Beck, Trevor Cohn, and Gholamreza Haffari. 2019. Neural Speech Translation using Lattice Transformations and Graph Networks. In Thirteenth Workshop on Graph-Based Methods for Natural Language Processing (TextGraphs-13), pages 26-31, Hongkong.
Benjamin Beilharz, Xin Sun, Sariya Karimova, and Stefan Riezler. 2020. LibriVoxDeEn: A Corpus for German-to-English Speech Translation and Speech Recognition. In Language Resources and Evaluation (LREC), Marseille, France.
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. 2013. Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation. arXiv:1308.3432.
Alexandre Bérard, Laurent Besacier, Ali Can Kocabiyikoglu, and Olivier Pietquin. 2018. End-to-End Automatic Speech Translation of Audiobooks.
In International Conference on Acoustics, Speech and Signal Processing (ICASSP), Calgary, Canada.
Alexandre Berard, Olivier Pietquin, Christophe Servan, and Laurent Besacier. 2016. Listen and Translate: A Proof of Concept for End-to-End Speech-to-Text Translation. In NIPS Workshop on End-to-end Learning for Speech and Audio Processing, Barcelona, Spain.
Nicola Bertoldi and Marcello Federico. 2005. A New Decoder for Spoken Language Translation Based on Confusion Networks. In Automatic Speech Recognition & Understanding (ASRU), pages 86-91, San Juan, Puerto Rico.
Alan W. Black, Ralf D. Brown, Robert Frederking, Kevin Lenzo, John Moody, Alexander Rudnicky, Rita Singh, and Eric Steinbrecher. 2002. Rapid Development of Speech-to-Speech Translation Systems. In International Conference on Spoken Language Processing (ICSLP), Denver, USA.
Marcely Zanon Boito, William N. Havard, Mahault Garnerin, Éric Le Ferrand, and Laurent Besacier. 2020. MaSS: A Large and Clean Multilingual Corpus of Sentence-aligned Spoken Utterances Extracted from the Bible. In Language Resources and Evaluation (LREC), Marseille, France.
Francisco Casacuberta, Hermann Ney, Franz Josef Och, Enrique Vidal, J. M. Vilar, S. Barrachina, I. García-Varea, D. Llorens, C. Martínez, S. Molau, F. Nevado, M. Pastor, D. Pico, A. Sanchis, and C. Tillmann. 2004. Some approaches to statistical and finite-state speech-to-speech translation. Computer Speech and Language, 18(1):25-47.
Qiao Cheng, Meiyuan Fang, Yaqian Han, Jin Huang, and Yitao Duan. 2019. Breaking the Data Barrier: Towards Robust Speech Translation via Adversarial Stability Training. In International Workshop on Spoken Language Translation (IWSLT), Hongkong.
Yong Cheng, Zhaopeng Tu, Fandong Meng, Junjie Zhai, and Yang Liu. 2018. Towards Robust Neural Machine Translation. In Association for Computational Linguistic (ACL), Melbourne, Australia.
Eunah Cho, Christian Fügen, Teresa Hermann, Kevin Kilgour, Mohammed Mediani, Christian Mohr, Jan Niehues, Kay Rottmann, Christian Saam, Sebastian Stüker, and Alex Waibel. 2013. A real-world system for simultaneous translation of German lectures. In Annual Conference of the International Speech Communication Association (InterSpeech), pages 3473-3477, Lyon, France.
Hoang Cuong and Khalil Sima'an. 2018. A survey of domain adaptation for statistical machine translation. In Conference on Computational Linguistics (COLING), Santa Fe, USA.
Mattia Antonino Di Gangi, Viet-Nhat Nguyen, Matteo Negri, and Marco Turchi. 2020. Instance-Based
Model Adaptation For Direct Speech Translation. In International Conference on Acoustics, Speech and Signal Processing (ICASSP), Barcelona, Spain.
Paul R Dixon, Andrew Finch, H Chiori, and H Kashioka. 2011. Investigation on the Effects of ASR Tuning on Speech Translation Performance. In International Workshop on Spoken Language Translation (IWSLT), pages 167-174, San Francisco, USA.
Quoc Truong Do, Tomoki Toda, Graham Neubig, Sakriani Sakti, and Satoshi Nakamura. 2017. Preserving Word-level Emphasis in Speech-to-speech Translation. IEEE Transactions on Audio, Speech and Language Processing, 25(3):544-556.
Long Duong, Antonios Anastasopoulos, David Chiang, Steven Bird, and Trevor Cohn. 2016. An Attentional Model for Speech Translation Without Transcription. In North American Chapter of the Association for Computational Linguistics (NAACL), pages 949-959, San Diego, USA.
Marcello Federico, Robert Enyedi, and Roberto Barrachicote. 2020. From Speech-to-Speech Translation to Automatic Dubbing. arXiv:2001.06785v3.
Erin Fitzgerald, Keith Hall, and Frederick Jelinek. 2009. Reconstructing false start errors in spontaneous speech text. In European Chapter of the Association for Computational Linguistics (EACL), pages 255-263, Athens, Greece.
Christian Fügen. 2008. A System for Simultaneous Translation of Lectures and Speeches. Ph.D. thesis, University of Karlsruhe.
Antonino Mattia di Gangi, Roldano Cattoni, Luisa Bentivogli, Matteo Negri, and Marco Turchi. 2019a. MuST-C: a Multilingual Speech Translation Corpus. In North American Chapter of the Association for Computational Linguistics (NAACL), Minneapolis, USA.
Mattia A. di Gangi, Matteo Negri, Roldano Cattoni, Roberto Dessi, and Marco Turchi. 2019b. Enhancing Transformer for End-to-end Speech-to-Text Translation. In Machine Translation Summit XVII Volume 1: Research Track, pages 21-31, Dublin, Ireland.
Mattia Antonino di Gangi, Matteo Negri, and Marco Turchi. 2019c. One-to-Many Multilingual End-to-End Speech Translation. In Automatic Speech Recognition & Understanding (ASRU), December.
Pierre Godard, Gilles Adda, Martine Adda-Decker, J. Benjumea, L. Besacier, J. Cooper-Leavitt, G-N. Kouarata, L. Lamel, H. Maynard, M. Mueller, A. Rialland, S. Stueker, F. Yvon, and M. Zanon-Boito. 2018. A Very Low Resource Language Speech Corpus for Computational Language Documentation Experiments. In Language Resources and Evaluation (LREC), Miyazaki, Japan.
He He, Jordan Boyd-Graber, and Hal. 2016. Interpretese vs. Translationese : The Uniqueness of Human Strategies in Simultaneous Interpretation. In North American Chapter of the Association for Computational Linguistics (NAACL), pages 971-976, San Diego, USA.
Xiaodong He, Li Deng, and Alex Acero. 2011. Why word error rate is not a good metric for speech recognizer training for the speech translation task? In International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5632-5635, Prague, Czech Republic.
Roger Hsiao, Ashish Venugopal, Thilo Kohler, Ting Zhang, Paisarn Charoenpornsawat, Andreas Zollmann, Stephan Vogel, Alan W. Black, Tanja Schultz, and Alex Waibel. 2006. Optimizing components for handheld two-way speech translation for an English-Iraqi Arabic system. In Annual Conference of the International Speech Communication Association (InterSpeech), pages 765-768, Pittsburgh, USA.
Hirofumi Inaguma, Kevin Duh, Tatsuya Kawahara, and Shinji Watanabe. 2019. Multilingual End-to-End Speech Translation. In Automatic Speech Recognition & Understanding (ASRU), Singapore.
Sathish Indurthi, Houjeung Han, Nikhil Kumar Lakumarapu, Beomseok Lee, Insoo Chung, Sangha Kim, and Chanwoo Kim. 2020. Data Efficient Direct Speech-to-Text Translation with Modality Agnostic Meta-Learning. In Conference on Artificial Intelligence (AAAI), New York City, USA.
Javier Iranzo-Sanchez, Joan Albert Silvestre-Cerda, Javier Jorge, Nahuel Rosello, Adrià Giménez, Albert Sanchis, Jorge Civera, and Alfons Juan. 2020. Europarl-ST: A Multilingual Corpus For Speech Translation Of Parliamentary Debates. In International Conference on Acoustics, Speech and Signal Processing (ICASSP), Barcelona, Spain.
Ye Jia, Melvin Johnson, Wolfgang Macherey, Ron J Weiss, Yuan Cao, Chung-cheng Chiu Naveen, Ari Stella, and Laurenzo Yonghui. 2019a. Leveraging Weakly Supervised Data to Improve End-to-End Speech-to-Text Translation. In International Conference on Acoustics, Speech and Signal Processing (ICASSP), Brighton, United Kingdom.
Ye Jia, Ron J. Weiss, Fadi Biadsy, Wolfgang Macherey, Melvin Johnson, Zhifeng Chen, and Yonghui Wu. 2019b. Direct speech-to-speech translation with a sequence-to-sequence model. In Annual Conference of the International Speech Communication Association (InterSpeech), Graz, Austria.
Jie Jiang, Zeeshan Ahmed, Julie Carson-Berndsen, Peter Cahill, and Andy Way. 2011. Phonetic Representation-Based Speech Translation. In Machine Translation Summit (MTS), pages 81-88, Xi-amen, China.