SlowGuess/ACL_2020_OCR huggingface dataset:
Matthias Sperber
sperber@apple.com
Matthias Paulik
mpaulik@apple.com
Over its three decade history, speech translation has experienced several shifts in its primary research themes; moving from loosely coupled cascades of speech recognition and machine translation, to exploring questions of tight coupling, and finally to end-to-end models that have recently attracted much attention. This paper provides a brief survey of these developments, along with a discussion of the main challenges of traditional approaches which stem from committing to intermediate representations from the speech recognizer, and from training cascaded models separately towards different objectives.
Recent end-to-end modeling techniques promise a principled way of overcoming these issues by allowing joint training of all model components and removing the need for explicit intermediate representations. However, a closer look reveals that many end-to-end models fall short of solving these issues, due to compromises made to address data scarcity. This paper provides a unifying categorization and nomenclature that covers both traditional and recent approaches and that may help researchers by highlighting both trade-offs and open research questions.
Speech translation (ST), the task of translating acoustic speech signals into text in a foreign language, is a complex and multi-faceted task that builds upon work in automatic speech recognition (ASR) and machine translation (MT). ST applications are diverse and include travel assistants (Takezawa et al., 1998), simultaneous lecture translation (Fügen, 2008), movie dubbing/subtitling (Saboo and Baumann, 2019; Matusov et al., 2019), language documentation and crisis response (Bansal et al., 2017), and developmental efforts (Black et al., 2002).
Until recently, the only feasible approach has been the cascaded approach that applies an ASR to the speech inputs, and then passes the results on to an MT system. Progress in ST has come from two fronts: general improvements in ASR and MT models, and moving from the loosely-coupled cascade in its most basic form toward a tighter coupling. However, despite considerable efforts toward tight coupling, a large share of the progress has arguably been owed simply to general ASR and MT improvements. $^{1}$
Recently, new modeling techniques and in particular end-to-end trainable encoder-decoder models have fueled hope for addressing challenges of ST in a more principled manner. Despite these hopes, the empirical evidence indicates that the success of such efforts has so far been mixed (Weiss et al., 2017; Niehues et al., 2019).
In this paper, we will attempt to uncover potential reasons for this. We start by surveying models proposed throughout the three-decade history of ST. By contrasting the extreme points of loosely coupled cascades vs. purely end-to-end trained direct models, we identify foundational challenges: erroneous early decisions, mismatch between spoken-style ASR outputs and written-style MT inputs, and loss of speech information (e.g. prosody) on the one hand, and data scarcity on the other hand. We then show that to improve data efficiency, most end-to-end models employ techniques that re-introduce issues generally attributed to cascaded ST.
Furthermore, this paper proposes a categorization of ST research into well-defined terms for the particular challenges, requirements, and techniques that are being addressed or used. This multidimensional categorization suggests a modeling
For instance, Pham et al. (2019)'s winning system in the IWSLT 2019 shared ST task (Niehues et al., 2019) makes heavy use of recent ASR and MT modeling techniques, but is otherwise a relatively simple cascaded approach.
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7409-7421 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
space with many intermediate points, rather than a dichotomy of cascaded vs. end-to-end models, and reveals a number of trade-offs between different modeling choices. This implies that additional work to more explicitly analyze the interactions between these trade-offs, along with further model explorations, can help to determine more favorable points in the modeling space, and ultimately the most favorable model for a specific ST application.
2 Chronological Survey
This chapter surveys the historical development of ST and introduces key concepts that will be expanded upon later.2
2.1 Loosely Coupled Cascades
Early efforts to realize ST (Stentiford and Steer, 1988; Waibel et al., 1991) introduced what we will refer to as the loosely coupled cascade in which separately built ASR and MT systems are employed and the best hypothesis of the former is used as input to the latter. The possibility of speech-to-speech translation, which extends the cascade by appending a text-to-speech component, was also considered early on (Waibel et al., 1991).
These early systems were especially susceptible to errors propagated from the ASR, given the widespread use of interlingua-based MT which relied on parsers unable to handle mal-formed inputs (Woszczyna et al., 1993; Lavie et al., 1996; Liu et al., 2003). Subsequent systems Wang and Waibel (1998); Takezawa et al. (1998); Black et al. (2002); Sumita et al. (2007), relying on data driven, statistical MT, somewhat alleviated the issue, and also in part opened the path towards tighter integration.
2.2 Toward Tight Integration
Researchers soon turned to the question of how to avoid early decisions and the problem of error propagation. While the desirable solution of full integration over transcripts is intractable (Ney, 1999), approximations are possible. Vidal (1997); Bangalore and Riccardi (2001); Casacuberta et al. (2004); Pérez et al. (2007) compute a composition of FST-based ASR and MT models, which approximates the full integration up to search heuristics, but suffers from limited reordering capabilities. A much
simpler, though computationally expensive, solution is the  $n$ -best translation approach which replaces the sum over all possible transcripts by a sum over only the  $n$ -best ASR outputs (Woszczyna et al., 1993; Lavie et al., 1996). Follow-up work suggested lattices and confusion nets (Saleem et al., 2004; Zhang et al., 2005; Bertoldi and Federico, 2005) as more effective and efficient alternatives to  $n$ -best lists. Lattices proved flexible enough for integration into various translation models, from word-based translation models to phrase-based ST Matusov et al. (2005, 2008) to neural lattice-to-sequence models (Sperber et al., 2017a, 2019b; Zhang et al., 2019; Beck et al., 2019).
Another promising idea was to limit the detrimental effects of early decisions, rather than attempting to avoid early decisions. One way of achieving this is to train robust translation models by introducing synthetic ASR errors into the source side of MT corpora (Peitz et al., 2012; Tsvetkov et al., 2014; Ruiz et al., 2015; Sperber et al., 2017b; Cheng et al., 2018, 2019). A different route is taken by Dixon et al. (2011); He et al. (2011) who directly optimize ASR outputs towards translation quality.
Beyond early decisions, research moved towards tighter coupling by addressing issues arising from ASR and MT models being trained separately and on different types of corpora. Domain adaptation techniques were used by Liu et al. (2003); Fügen (2008) to adapt models to the spoken language domain. Matusov et al. (2006); Fügen (2008) propose re-segmenting the ASR output and inserting punctuation, so as to provide the translation model with well-formed text inputs. In addition, disfluency removal (Fitzgerald et al., 2009) was proposed to avoid translation errors caused by disfluencies that are often found in spoken language.
Aguero et al. (2006); Anumanchipalli et al. (2012); Do et al. (2017); Kano et al. (2018) propose prosody transfer for speech-to-speech translation by determining source-side prosody and applying transformed prosody characteristics to the aligned target words.
2.3 Speech Translation Corpora
It is important to realize that all efforts to this point had used separate ASR and MT corpora for training. This often led to a mismatch between ASR trained on data from the spoken domain, and MT trained on data from the written domain. End-to-
For a good comparison of empirical results, which are not the focus of this paper, we refer to concurrent work (Sulubacak et al., 2019). Moreover, for conciseness we do not cover the sub-topic of simultaneous translation (Fügen, 2008).
7410
end ST data (translated speech utterances) was only available in small quantities for test purposes.
Paulik (2010) proposes the use of audio recordings of interpreter-mediated communication scenarios, which is not only potentially easier to obtain, but also does not exhibit such domain mismatches. Post et al. (2013) manually translate an ASR corpus to obtain an end-to-end ST corpus, and show that training both ASR and MT on the same corpus considerably improves results compared to using out-of-domain MT data. Unfortunately, high annotation costs prevent scaling of the latter approach, so follow-up work concentrates on compiling ST corpora from available web sources (Godard et al., 2018; Kocabiyikoglu et al., 2018; Sanabria et al., 2018; di Gangi et al., 2019a; Boito et al., 2020; Beilharz et al., 2020; Iranzo-Sanchez et al., 2020; Wang et al., 2020a). Note that despite these efforts, publicly available ST corpora are currently strongly limited in terms of both size and language coverage. For practical purposes, the use of separate ASR and MT corpora is therefore currently unavoidable.
2.4 End-to-End Models
The availability of end-to-end ST corpora, along with the success of end-to-end models for MT and ASR, led researchers to explore ST models trained in an end-to-end fashion. This was fueled by a hope to solve the issues addressed by prior research in a principled and more effective way. Duong et al. (2016); Berard et al. (2016); Bansal et al. (2018) explore direct ST models that translate speech without using explicitly generated intermediate ASR output. In contrast, Kano et al. (2017); Anastasopoulos and Chiang (2018); Wang et al. (2020b) explore end-to-end trainable cascades and triangle models, i.e. models that do rely on transcripts, but are optimized in part through end-to-end training. Multi-task training and pre-training were proposed as a way to incorporate additional ASR and MT data and reduce dependency on scarce end-to-end data (Weiss et al., 2017; Berard et al., 2018; Bansal et al., 2019; Stoian et al., 2020; Wang et al., 2020b). As these techniques were not able to exploit ASR and MT data as effectively as the loosely coupled cascade, other approaches like subtask training for end-to-end-trainable cascades (Sperber et al., 2019a), data augmentation (Jia et al., 2019a; Pino et al., 2019), knowledge distillation (Liu et al., 2019), and meta-learning (Indurthi et al., 2020) were proposed. Salesky et al.
Figure 1: Illustration of inference strategies (§4.2): Committed/marginalizing cascade (CC/MC), direct (Di), committed/marginalizing triangle (CT/MT), joint (Jt). Double lines differentiate the observed variable (speech input  $X$ ) from random variables (intermediate representations  $IR$  and translations  $T$ ). Shaded circles marginalize over random variables.
(2019a) propose pre-segmenting speech frames, (Jia et al., 2019b; Tjandra et al., 2019) explore speech-to-speech translation. Sung et al. (2019); di Gangi et al. (2019b); Di Gangi et al. (2020); Bahar et al. (2019); Inaguma et al. (2019); di Gangi et al. (2019c) transfer ideas from MT and ASR fields to ST.
3 Central Challenges
Given the abundance of prior work, a clear picture on where we currently stand is needed. For purposes of identifying the key challenges in ST research, this section will contrast the extreme cases of the loosely coupled cascade (CC in Fig. 1) against the vanilla direct model (Di in Fig. 1). We emphasize that these models are only extreme points in a modeling space with many intermediate points, as we will see in §4. We assume appropriate speech features  $X$  as inputs.  $T, \hat{T} \in \mathcal{T}$  denote candidate/best translations, respectively, from the MT hypothesis space.  $S \in \mathcal{H}$  denotes a graphemic transcript from the ASR hypothesis space.
3.1 Challenges of Loosely Coupled Cascades
The loosely coupled cascade justifies its decomposition into MT model  $P_{\mathrm{MT}}(T|S)$  and ASR model  $P_{\mathrm{ASR}}(S|X)$  as follows:
<sup>3</sup>ASR and MT models trained separately on different corpora; intermediate representation is ASR 1-best output.
4Encoder-decoder model trained on speech utterances paired with translations; no intermediate representations used.
$$
\begin{array}{l} \hat {T} = \underset {T \in \mathcal {T}} {\operatorname {a r g m a x}} P (T \mid X) (1) \\ = \underset {T \in \mathcal {T}} {\operatorname {a r g m a x}} \sum_ {S \in \mathcal {H}} P (T | S, X) P (S | X) (2) \\ \approx \underset {T \in \mathcal {T}} {\operatorname {a r g m a x}} \sum_ {S \in \mathcal {H}} P _ {\mathrm {M T}} (T | S) P _ {\mathrm {A S R}} (S | X) (3) \\ \approx \underset {T \in \mathcal {T}} {\operatorname {a r g m a x}} \sum_ {S \in \mathcal {H} ^ {\prime}} P _ {\mathrm {M T}} (T | S) P _ {\mathrm {A S R}} (S | X) (4) \\ \end{array}
$$
Note that here the set  $\mathcal{H}'$  contains only a single entry, the 1-best ASR output. The approximations in these derivations directly result in the following three foundational challenges:
Erroneous early decisions: Committing to a potentially erroneous  $S$  during inference. This leads to the well-known problem of error propagation (Ruiz and Federico, 2014) and is caused by avoiding the intractable full integration over transcripts (Eq. 3) and using only the 1-best ASR output instead (Eq. 4). Typical countermeasures include increasing  $\mathcal{H}'$  to cover a larger space using lattices or confusion nets, or improving the robustness of MT models.
Mismatched source-language: ASR and MT components model the source-language (transcript) priors  $P_{\mathrm{MT}}(S)$  and  $P_{\mathrm{ASR}}(S)$  differently. Causes include both modeling assumptions, e.g. ASR modeling only unpunctuated transcripts; and mismatched training data, leading to stylistic and topical divergence. Typical countermeasures are domain adaptation techniques, disfluency removal, text normalization, and segmentation/punctuation insertion.
Information loss: Assumed conditional independence between inputs and outputs, given the transcript:  $(T \perp X) \mid S$ . This can be seen in Eq. 3 and results in any information not represented in  $S$  to be lost for the translation step. In particular, the MT model is unaware of prosody which structures and disambiguates the utterances, thus playing a role similar to punctuation in written texts; and provides ways to emphasize words or parts of the messages that the speaker think are important. Prosody also conveys information on the speaker's attitude and emotional state (Jouvet, 2019).
3.2 Challenges of the Vanilla Direct Model
Consider instead the other extreme case: an encoder-decoder model trained to directly produce translations from speech (Eq. 1). Because this model avoids the decomposition in Eq. 2-4, it is not subject to the three issues outlined in §3.1. Unfortunately, this second extreme case is often impractical due to its dependency on scarce end-to-end ST training corpora (§2.3), rendering this model unable to compete with cascaded models that are trained on abundant ASR and MT training data.
Most recent works therefore depart from this purely end-to-end trained direct model, and incorporate ASR and MT back into training, e.g. through weakly supervised training, or by exploring end-to-end trainable cascades or triangle models (CT/MT in Fig. 1). This departure raises two questions: (1) To what extent does the re-introduction of ASR and MT data cause challenges similar to those found in loosely coupled cascades? (2) Are techniques such as weakly supervised training effective enough to allow competing with the loosely coupled cascade? To address the second question, we propose the notion of data efficiency as a fourth key challenge.
Data efficiency: The increase in accuracy achievable through the addition of a certain amount of training data. To assess data efficiency, data ablations that contrast models over at least two data conditions are required. We argue that empirical evidence along these lines will help considerably in making generalizable claims about the relative performance between two ST models. Generalizable findings across data conditions are critical given that ST models are trained on at least three types of corpora (ASR, MT, and end-to-end corpora), whose availability vastly differs across languages.
3.3 Data Efficiency vs. Modeling Power - A Trade-Off?
Consider how the incorporation of MT and ASR data into ST models of any kind may inherently cause the problems as outlined in §3.1: Training on MT data may weaken the model's sensitivity to prosody; the effectiveness of training on  $\mathrm{ASR + MT}$  data may be impacted by mismatched source-language issues; even some types of end-to-end-trainable models make (non-discrete) early decisions that are potentially erroneous.
This suggests a potential trade-off between data efficiency and modeling power. In order to find
Note that our definition does not entail covariance shift and other forms of domain mismatch (Kouw and Loog, 2018) which, though relevant, are not unique to cascaded ST and are widely covered by general ASR and MT literature (Cuong and Sima'an, 2018).
English	Japanese
this is my niece, lucy	kochira wa suekko no lucy desu
ごろらは 姬rots 子 ルーナー です。
this is my niece, lucy	lucy, kono ko ga watashi no suekko desu
ルーナー、 その子が 私の 姬rots です。
will you have /cheese or /jam	chiizu toka jamu toka, dore ni shimasu ka
チ罗斯 とか 徴ム 和か、 どれ に しま�か?
will you have /cheese or \jam	chiizu ka jamu, docchi ni shimasu ka
チ罗斯 か 徴ム 和か に しま�か?
Table 1: Motivating examples for prosody-aware translation from English to Japanese. In the first example, prosody disambiguates whether the speaker is talking about Lucy as a third person or directly addressing Lucy. In the second example, prosody disambiguates whether cheese or jam is an open set or a closed set. In both cases, the surface form of the Japanese translation requires considerable changes depending on the prosody.
models that trade off advantages and disadvantages in the most favorable way, it is therefore necessary to thoroughly analyze models across the dimensions of early decisions, mismatched source-language, information loss, and data efficiency.
Analyzing early decisions: Problems due to erroneous early decisions are inference-time phenomena in which upstream ASR errors are responsible for errors in the final translation outputs. It follows that the problem disappears for hypothetical utterances for which the ASR can generate error-free intermediate representations. Thus, models that do not suffer from erroneous early decisions will expectedly exhibit an advantage over other models especially for acoustically challenging inputs, and less so for inputs with clean acoustics. This angle can provide us with strategies for isolating errors related to this particular phenomenon. Prior work in this spirit has demonstrated that lattice-to-sequence translation is in fact beneficial especially for acoustically challenging inputs (Sperber et al., 2017a), and that cascaded models with non-discrete intermediate representations are less sensitive to artificially perturbed intermediate representations than if using discrete transcripts as an intermediate representation (Sperber et al., 2019a).
Analyzing mismatched source-language: End-to-end ST corpora allow for controlled experiments in which one can switch between matched vs. mismatched (out-of-domain) MT corpora. Post et al. (2013) demonstrated that using a matched corpus can strongly improve translation quality for loosely coupled cascades. We are not aware of such analyses in more recent work.
Analyzing information loss: Prior work (Aguero et al., 2006; Anumanchipalli et al., 2012; Do et al., 2017; Kano et al., 2018) has addressed
prosody transfer in speech-to-speech translation, but to our knowledge the question of how such information should inform textual translation decisions is still unexplored. Table 1 shows examples that may motivate future work in this direction.
Analyzing data efficiency: While several prior works aim at addressing this problem, often only a single data condition is tested, limiting the generalizability of findings. We are aware of three recent works that do analyze data efficiency across several data conditions (Jia et al., 2019a; Sperber et al., 2019a; Wang et al., 2020b). Findings indicate that both pretraining and data synthesizing outperform multi-task training in terms of data efficiency, and that end-to-end trainable cascades are on par with loosely coupled cascades, while strongly outperforming multi-task training.
4 Modeling Techniques
Let us now break apart modeling techniques from prior literature into four overarching categories, with the aim of exposing the ST modeling space between the extreme points of vanilla direct models and loosely coupled cascades.
4.1 Intermediate Representations
Almost all models use intermediate representations (IRs) in some form: non-direct models to support both training and inference, and direct models to overcome data limitations. IRs are often speech transcripts, but not necessarily so. A number of factors must be considered for choosing an appropriate IR, such as availability of supervised data, inference accuracy, expected impact of erroneous early decisions, and the feasibility of backpropagation through the IR for end-to-end training. We list several possibilities below:
Transcripts: Generally used in the loosely coupled cascade. Being a discrete representation, this option prevents end-to-end training via backpropagation, although future work may experiment with work-arounds such as the straight-through gradient estimator (Bengio et al., 2013). Besides graphemic transcripts, phonetic transcripts are another option (Jiang et al., 2011).
Hidden representations: Kano et al. (2017); Anastasopoulos and Chiang (2018); Sperber et al. (2019a) propose the use of hidden representations that are the by-product of a neural decoder generating an auxiliary IR such as a transcript. Advantages of this representation are differentiability, prevention of information loss, and weakened impact of erroneous early decisions. A downside is that end-to-end ST data is required for training.
Lattices: Lattices compactly represent the space over multiple sequences, and therefore weaken the impact of erroneous early decisions. Future work may explore lattices over continuous, hidden representations, and end-to-end training for ST models with lattices as intermediate representation.
Other: Prior work further suggests presegmented speech frames (Salesky et al., 2019a) or unsupervised speech-unit clusters (Tjandra et al., 2019) as intermediate representation. Further possibilities may be explored in future work.
4.2 Inference Strategies
The conditioning graph (Fig. 1) reveals independence assumptions and use of IRs at inference time. Some strategies avoid the problem of early decisions (MC, Di, MT, Jt), while others remove the conditional independence assumption between inputs and outputs (Di, CT, MT, Jt).
Committed cascade (CC): Compute one IR, rely on it to generate outputs (Eq. 4). Includes both the loosely coupled cascade, and recent end-to-end trainable cascaded models such as by Kano et al. (2017); Sperber et al. (2019a).
Marginalizing cascade (MC): Compute outputs by relying on IRs, but marginalize over them instead of committing to one (Eq. 3). As marginalization is intractable, approximations such as  $n$ -best translation or lattice translation are generally used.
Direct (Di): Compute outputs without relying on IRs (Eq. 1). To address data limitations, techniques
such as multi-task training or data augmentation can be used, but may reintroduce certain biases.