SlowGuess/NAACL_2021_OCR huggingface dataset:
Siddharth Dalmia Brian Yan Vikas Raunak Florian Metze Shinji Watanabe
Language Technologies Institute, Carnegie Mellon University, USA
{sdalmia,byan}@cs.cmu.edu
End-to-end approaches for sequence tasks are becoming increasingly popular. Yet for complex sequence tasks, like speech translation, systems that cascade several models trained on sub-tasks have shown to be superior, suggesting that the compositionality of cascaded systems simplifies learning and enables sophisticated search capabilities. In this work, we present an end-to-end framework that exploits compositionality to learn searchable hidden representations at intermediate stages of a sequence model using decomposed sub-tasks. These hidden intermediates can be improved using beam search to enhance the overall performance and can also incorporate external models at intermediate stages of the network to re-score or adapt towards out-of-domain data. One instance of the proposed framework is a Multi-Decoder model for speech translation that extracts the searchable hidden intermediates from a speech recognition sub-task. The model demonstrates the aforementioned benefits and outperforms the previous state-of-the-art by around  $+6$  and  $+3$  BLEU on the two test sets of Fisher-CallHome and by around  $+3$  and  $+4$  BLEU on the English-German and English-French test sets of MuST-C. $^{1}$
The principle of compositionality loosely states that a complex whole is composed of its parts and the rules by which those parts are combined (Lake and Baroni, 2018). This principle is present in engineering, where task decomposition of a complex system is required to assess and optimize task allocations (Levis et al., 1994), and in natural language, where paragraph coherence and discourse analysis rely on decomposition into sentences (Johnson, 1992; Kuo, 1995) and sentence level semantics relies on decomposition into lexical units (Liu et al., 2020b).
Similarly, many sequence-to-sequence tasks that convert one sequence into another (Sutskever et al., 2014) can be decomposed to simpler sequence subtasks in order to reduce the overall complexity. For example, speech translation systems, which seek to process speech in one language and output text in another language, can be naturally decomposed into the transcription of source language audio through automatic speech recognition (ASR) and translation into the target language through machine translation (MT). Such cascaded approaches have been widely used to build practical systems for a variety of sequence tasks like hybrid ASR (Hinton et al., 2012), phrase-based MT (Koehn et al., 2007), and cascaded ASR-MT systems for speech translation (ST) (Pham et al., 2019).
End-to-end sequence models like encoder-decoder models (Bahdanau et al., 2015; Vaswani et al., 2017), are attractive in part due to their simplistic design and the reduced need for hand-crafted features. However, studies have shown mixed results compared to cascaded models particularly for complex sequence tasks like speech translation (Inaguma et al., 2020) and spoken language understanding (Coucke et al., 2018). Although direct target sequence prediction avoids the issue of error propagation from one system to another in cascaded approaches (Tzoukermann and Miller, 2018), there are many attractive properties of cascaded systems, missing in end-to-end approaches, that are useful in complex sequence tasks.
In particular, we are interested in (1) the strong search capabilities of the cascaded systems that compose the final task output from individual system predictions (Mohri et al., 2002; Kumar et al., 2006; Beck et al., 2019), (2) the ability to incorporate external models to re-score each individual system (Och and Ney, 2002; Huang and Chiang, 2007), (3) the ability to easily adapt individual components towards out-of-domain data (Koehn and Schroeder, 2007; Peddinti et al., 2015), and finally
1All code and models are released as part of the ESPnet toolkit: https://github.com/espnet/espnet.
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1882-1896 June 6-11, 2021. ©2021 Association for Computational Linguistics
(4) the ability to monitor performance of the individual systems towards the decomposed sub-task (Tillmann and Ney, 2003; Meyer et al., 2016).
In this paper, we seek to incorporate these properties of cascaded systems into end-to-end sequence models. We first propose a generic framework to learn searchable hidden intermediates using an auto-regressive encoder-decoder model for any decomposable sequence task (§3). We then apply this approach to speech translation, where the intermediate stage is the output of ASR, by passing continuous hidden representations of discrete transcript sequences from the ASR sub-net decoder to the MT sub-net encoder. By doing so, we gain the ability to use beam search with optional external model re-scoring on the hidden intermediates, while maintaining end-to-end differentiability. Next, we suggest mitigation strategies for the error propagation issues inherited from decomposition.
We show the efficacy of searchable intermediate representations in our proposed model, called the Multi-Decoder, on speech translation with a 5.4 and 2.8 BLEU score improvement over the previous state-of-the-arts for Fisher and CallHome test sets respectively (§6). We extend these improvements by an average of 0.5 BLEU score through the aforementioned benefit of re-scoring the intermediate search with external models trained on the same dataset. We also show a method for monitoring sub-net performance using oracle intermediates that are void of search errors (§6.1). Finally, we show how these models can adapt to out-of-domain speech translation datasets, how our approach can be generalized to other sequence tasks like speech recognition, and how the benefits of decomposition persist even for larger corpora like MuST-C (§6.2).
2.1 Compositionality in Sequences Models
The probabilistic space of a sequence is combinatorial in nature, such that a sentence of  $L$  words from a fixed vocabulary  $\mathcal{V}$  would have an output space  $S$  of size  $|\mathcal{V}|^{L}$ . In order to deal with this combinatorial output space, an output sentence is decomposed into labeled target tokens,  $\mathbf{y} = (y_{1}, y_{2}, \dots, y_{L})$ , where  $y_{l} \in \mathcal{V}$ .
$$
P (\mathbf {y} \mid \mathbf {x}) = \prod_ {i = 1} ^ {L} P (y _ {i} \mid \mathbf {x}, y _ {1: i - 1})
$$
An auto-regressive encoder-decoder model uses the above probabilistic decomposition in sequence-to-
sequence tasks to learn next word prediction, which outputs a distribution over the next target token  $y_{l}$  given the previous tokens  $y_{1:l-1}$  and the input sequence  $\mathbf{x} = (\mathbf{x}_1, \mathbf{x}_t, \dots, \mathbf{x}_T)$ , where  $T$  is the input sequence length. In the next sub-section we detail the training and inference of these models.
2.2 Auto-regressive Encoder-Decoder Models
Training: In an auto-regressive encoder-decoder model, the ENCODER maps the input sequence  $\mathbf{x}$  to a sequence of continuous hidden representations  $\mathbf{h}^E = (\mathbf{h}_1^E,\mathbf{h}_t^E,\dots ,\mathbf{h}_T^E)$ , where  $\mathbf{h}_t^E\in \mathbb{R}^d$ . The DECODER then auto-regressively maps  $\mathbf{h}^E$  and the preceding ground-truth output tokens,  $\hat{y}_{1:l - 1}$ , to  $\mathbf{h}_l^D$  where  $\mathbf{h}_l^D\in \mathbb{R}^d$ . The sequence of decoder hidden representations form  $\mathbf{h}^D = (\mathbf{h}_1^D,\mathbf{h}_l^D,\dots ,\mathbf{h}_L^D)$  and the likelihood of each output token  $y_{l}$  is given by SOFTMAXOUT, which denotes an affine projection of  $\mathbf{h}_l^D$  to  $\nu$  followed by a softmax function.
$$
\mathbf {h} ^ {E} = \operatorname {E N C O D E R} (\mathbf {x})
$$
$$
\hat {\mathbf {h}} _ {l} ^ {D} = \operatorname {D E C O D E R} \left(\mathbf {h} ^ {E}, \hat {y} _ {1: l - 1}\right) \tag {1}
$$
$$
P \left(y _ {l} \mid \hat {y} _ {1: l - 1}, \mathbf {h} ^ {E}\right) = \text {S O F T M A X O U T} \left(\hat {\mathbf {h}} _ {l} ^ {D}\right) \tag {2}
$$
During training, the DECODER performs token classification for next word prediction by considering only the ground truth sequences for previous tokens  $\hat{\mathbf{y}}$ . We refer to this  $\hat{\mathbf{h}}^D$  as oracle decoder representations, which will be discussed later.
Inference: During inference, we can maximize the likelihood of the entire sequence from the output space  $S$  by composing the conditional probabilities of each step for the  $L$  tokens in the sequence.
$$
\mathbf {h} _ {l} ^ {D} = \operatorname {D E C O D E R} \left(\mathbf {h} ^ {E}, y _ {1: l - 1}\right) \tag {3}
$$
$$
P \left(y _ {l} \mid \mathbf {x}, y _ {1: l - 1}\right) = \operatorname {S O F T M A X O U T} \left(\mathbf {h} _ {l} ^ {D}\right)
$$
$$
\tilde {\mathbf {y}} = \underset {\mathbf {y} \in \mathcal {S}} {\operatorname {a r g m a x}} \prod_ {i = 1} ^ {L} P \left(y _ {i} \mid \mathbf {x}, y _ {1: i - 1}\right) \tag {4}
$$
This is an intractable search problem and it can be approximated by either greedily choosing argmax at each step or using a search algorithm like beam search to approximate  $\tilde{\mathbf{y}}$ . Beam search (Reddy, 1988) generates candidates at each step and prunes the search space to a tractable beam size of  $B$  most likely sequences. As  $B \to \infty$ , the beam search result would be equivalent to equation 4.
$$
\text {G R E E D Y S E A R C H} := \underset {y _ {l}} {\operatorname {a r g m a x}} P (y _ {l} \mid \mathbf {x}, y _ {1: l - 1})
$$
$$
\operatorname {B E A M S E A R C H} := \operatorname {B E A M} \left(P \left(y _ {l} \mid \mathbf {x}, y _ {1: l - 1}\right)\right)
$$
Figure 1: The left side presents the schematics and the information flow of our proposed framework applied to ST, in a model we call the Multi-Decoder. Our model decomposes ST into ASR and MT sub-nets, each of which consists of an encoder and decoder. The right side displays a Multi-Sequence Attention variant of the  $\mathrm{DECODER}_{\mathrm{ST}}$  that is conditioned on both speech information via the  $\mathrm{ENCODER}_{\mathrm{ASR}}$  and transcription information via the  $\mathrm{ENCODER}_{\mathrm{ST}}$ .
In approximate search for auto-regressive models, like beam search, the DECODER receives alternate candidates of previous tokens to find candidates with a higher likelihood as an overall sequence. This also allows for the use of external models like Language Models (LM) or Connectionist Temporal Classification Models (CTC) for re-scoring candidates (Hori et al., 2017).
In this section, we present a general framework to exploit natural decompositions in sequence tasks which seek to predict some output  $\mathcal{C}$  from an input sequence  $\mathcal{A}$ . If there is an intermediate sequence  $\mathcal{B}$  for which  $\mathcal{A} \to \mathcal{B}$  sequence transduction followed by  $\mathcal{B} \to \mathcal{C}$  prediction achieves the original task, then the original  $\mathcal{A} \to \mathcal{C}$  task is decomposable.
In other words, if we can learn  $P(\mathcal{B} \mid \mathcal{A})$  then we can learn the overall task of  $P(\mathcal{C} \mid \mathcal{A})$  through  $\max_{\mathcal{B}} (P(\mathcal{C} \mid \mathcal{A}, \mathcal{B}) P(\mathcal{B} \mid \mathcal{A}))$ , approximated using Viterbi search. We define a first encoder-decoder  $\mathrm{SUB}_{\mathcal{A} \rightarrow \mathcal{B}} \mathrm{NET}$  to map an input sequence  $\mathcal{A}$  to a sequence of decoder hidden states,  $\mathbf{h}^{D_{\mathcal{B}}}$ . Then we define a subsequent  $\mathrm{SUB}_{\mathcal{B} \rightarrow \mathcal{C}} \mathrm{NET}$  to map  $\mathbf{h}^{D_{\mathcal{B}}}$  to the final probabilistic output space of  $\mathcal{C}$ . Therefore, we call  $\mathbf{h}^{D_{\mathcal{B}}}$  hidden intermediates. The following equations show the two sub-networks of our framework,  $\mathrm{SUB}_{\mathcal{A} \rightarrow \mathcal{B}} \mathrm{NET}$  and  $\mathrm{SUB}_{\mathcal{B} \rightarrow \mathcal{C}} \mathrm{NET}$ , which can be trained end-to-end while also exploiting compositionality in sequence tasks.
$\mathbf{SUB}_{A \to B} \mathbf{NET}$ :
$$
\mathbf {h} ^ {E} = \operatorname {E N C O D E R} _ {\mathcal {A}} (\mathcal {A})
$$
$$
\hat {\mathbf {h}} _ {l} ^ {D _ {\mathcal {B}}} = \mathrm {D E C O D E R} _ {\mathcal {B}} (\mathbf {h} ^ {E}, \hat {\mathbf {y}} _ {1: l - 1} ^ {\mathcal {B}})
$$
$$
P \left(y _ {l} ^ {\mathcal {B}} \mid \hat {\mathbf {y}} _ {1: l - 1} ^ {\mathcal {B}}, \mathbf {h} ^ {E}\right) = \text {S O F T M A X O U T} \left(\hat {\mathbf {h}} _ {l} ^ {D _ {\mathcal {B}}}\right) \tag {5}
$$
SUBB  $\rightarrow$  CCNET:
$$
P \left(\mathcal {C} \mid \hat {\mathbf {h}} _ {l} ^ {D _ {\mathcal {B}}}\right) = \operatorname {S U B} _ {\mathcal {B} \rightarrow \mathcal {C}} \operatorname {N E T} \left(\hat {\mathbf {h}} _ {l} ^ {D _ {\mathcal {B}}}\right) \tag {6}
$$
Note that the final prediction, given by equation 6, does not need to be a sequence and can be a categorical class like in spoken language understanding tasks. Next we will show how the hidden intermediates become searchable during inference.
3.1 Searchable Hidden Intermediates
As stated in section §2.2, approximate search algorithms maximize the likelihood,  $P(\mathbf{y} \mid \mathbf{x})$ , of the entire sequence by considering different candidates  $y_{l}$  at each step. Candidate-based search, particularly in auto-regressive encoder-decoder models, also affects the decoder hidden representation,  $\mathbf{h}^{D}$ , as these are directly dependent on the previous candidate (refer to equations 1 and 3). This implies that by searching for better approximations of the previous predicted tokens,  $\mathbf{y}_{l-1} = (\mathbf{y}_{\mathrm{BEAM}})_{l-1}$ , we also improve the decoder hidden representations for the next token,  $\mathbf{h}_{l}^{D} = (\mathbf{h}_{\mathrm{BEAM}}^{D})_{l}$ . As  $\mathbf{y}_{\mathrm{BEAM}} \rightarrow \hat{\mathbf{y}}$ , the decoder hidden representations tend to the oracle decoder representations that have only errors from next word prediction,  $\mathbf{h}_{\mathrm{BEAM}}^{D} \rightarrow \hat{\mathbf{h}}^{D}$ . A perfect search is analogous to choosing the ground truth  $\hat{y}$  at each step, which would yield  $\hat{\mathbf{h}}^{D}$ .
We apply this beam search of hidden intermediates, thereby approximating  $\hat{\mathbf{h}}^{D_{\mathcal{B}}}$  with  $\mathbf{h}_{\mathrm{BEAM}}^{D_{\mathcal{B}}}$ . This process is illustrated in algorithm 1, which
2Note that this framework does not use locally-normalized softmax distributions but rather the hidden representations, thereby avoiding label bias issues when combining multiple sub-systems (Bottou et al., 1997; Wiseman and Rush, 2016).
shows beam search for  $\mathbf{h}_{\mathrm{BEAM}}^{D_{\mathcal{B}}}$  that are subsequently passed to the  $\mathrm{SUB}_{\mathcal{B}\to \mathcal{C}}\mathrm{NET}$ . In line 7, we show how an external model like an LM or a CTC model can be used to generate an alternate sequence likelihood,  $P_{\mathrm{EXT}}(\mathbf{y}_l^{\mathcal{B}})$ , which can be combined with the  $\mathrm{SUB}_{\mathcal{A}\to \mathcal{B}}\mathrm{NET}$  likelihood,  $P_{\mathcal{B}}(\mathbf{y}_l^{\mathcal{B}}\mid \mathbf{x})$ , with a tunable parameter  $\lambda$ .
Algorithm 1 Beam Search for Hidden Intermediates: We perform beam search to approximate the most likely sequence for the sub-task  $\mathcal{A} \rightarrow \mathcal{B}$ ,  $\mathbf{y}_{\mathrm{BEAM}}^{\mathcal{B}}$ , while collecting the corresponding  $\mathrm{DECODER}_{\mathcal{B}}$  hidden representations,  $\mathbf{h}_{\mathrm{BEAM}}^{D_{\mathcal{B}}}$ . The output  $\mathbf{h}_{\mathrm{BEAM}}^{D_{\mathcal{B}}}$ , is passed to the final sub-network to predict final output  $\mathcal{C}$  and  $\mathbf{y}_{\mathrm{BEAM}}^{\mathcal{B}}$  is used for monitoring performance on predicting  $\mathcal{B}$ .
1: Initialize: BEAM  $\leftarrow$  {sos}; k  $\leftarrow$  beam size;
2:  $\mathbf{h}^{E_{\mathrm{A}}} \gets \operatorname{ENCODER}_{\mathcal{A}}(\mathbf{x})$
3: for  $l = 1$  to maxSTEPs do
4: for  $\mathbf{y}_{l - 1}^{\mathcal{B}}\in \mathrm{BEAM}$  do
5:  $\mathbf{h}_l^{D_{\mathcal{B}}}\gets \mathrm{DECODER}_{\mathcal{B}}(\mathbf{h}^{E_{\mathcal{A}}},\mathbf{y}_{l - 1}^{\mathcal{B}})$
6: for  $\mathbf{y}_l^B\in \mathbf{y}_{l - 1}^B +\{\mathcal{V}\}$  do
7:  $s_{l}\gets P_{\mathcal{A}\to \mathcal{B}}(\mathbf{y}_{l}^{\mathcal{B}}\mid \mathbf{x})^{1 - \lambda}P_{\mathrm{EXT}}(\mathbf{y}_{l}^{\mathcal{B}})^{\lambda}$
8:  $\mathcal{H}\gets (s_l,\mathbf{y}_l^{\mathcal{B}},\mathbf{h}_l^{D_{\mathcal{B}}})$
9: end for
10: end for
11: BEAM  $\leftarrow$  argkmax(H)
12: end for
13:  $(s^{\mathcal{B}},\mathbf{y}_{\mathrm{BEAM}}^{\mathcal{B}},\mathbf{h}_{\mathrm{BEAM}}^{D_{\mathcal{B}}})\gets \operatorname {argmax}\big(\mathrm{BEAM}\big)$
14: Return  $\mathbf{y}_{\mathrm{BEAM}}^{\mathcal{B}} \rightarrow \mathrm{SUB}_{\mathcal{A} \rightarrow \mathcal{B}} \mathrm{NET}$  Monitoring
15: Return  $\mathbf{h}_{\mathrm{BEAM}}^{D_B} \to \mathrm{Final~SUB}_{B \to C} \mathrm{NET}$
We can monitor the performance of the  $\mathrm{SUB}_{\mathcal{A}\rightarrow \mathcal{B}}\mathrm{NET}$  by comparing the decoded intermediate sequence  $\mathbf{y}_{\mathrm{BEAM}}^{\mathcal{B}}$  to the ground truth  $\hat{\mathbf{y}}^{\mathcal{B}}$ . We can also monitor the  $\mathrm{SUB}_{\mathcal{B}\rightarrow \mathcal{C}}\mathrm{NET}$  performance by using the aforementioned oracle representations of the intermediates,  $\hat{\mathbf{h}}^{D_{\mathcal{B}}}$ , which can be obtained by feeding the ground truth  $\hat{\mathbf{y}}^{\mathcal{B}}$  to  $\mathrm{DECODER}_{\mathcal{B}}$ . By passing  $\hat{\mathbf{h}}^{D_{\mathcal{B}}}$  to  $\mathrm{SUB}_{\mathcal{B}\rightarrow \mathcal{C}}\mathrm{NET}$ , we can observe its performance in a vacuum, i.e. void of search errors in the hidden intermediates.
3.2 Multi-Decoder Model
In order to show the applicability of our end-to-end framework we propose our Multi-Decoder model for speech translation. This model predicts a sequence of text translations  $\mathbf{y}^{\mathrm{ST}}$  from an input se
quence of speech  $\mathbf{x}$  and uses a sequence of text transcriptions  $\mathbf{y}^{\mathrm{ASR}}$  as an intermediate. In this case, the  $\mathrm{SUB}_{\mathcal{A} \to \mathcal{B}} \mathrm{NET}$  in equation 5 is specified as the ASR sub-net and the  $\mathrm{SUB}_{\mathcal{B} \to \mathcal{C}} \mathrm{NET}$  in equation 6 is specified as the MT sub-net. Since the MT sub-net is also a sequence prediction task, both sub-nets are encoder-decoder models in our architecture (Bahdanau et al., 2015; Vaswani et al., 2017). In Figure 1 we illustrate the schematics of our transformer based Multi-Decoder ST model which can also be summarized as follows:
$$
\mathbf {h} ^ {E _ {\mathrm {A S R}}} = \operatorname {E N C O D E R} _ {\mathrm {A S R}} (\mathbf {x}) \tag {7}
$$
$$
\hat {\mathbf {h}} _ {l} ^ {D _ {\mathrm {A S R}}} = \operatorname {D E C O D E R} _ {\mathrm {A S R}} \left(\mathbf {h} ^ {E _ {\mathrm {A S R}}}, \hat {y} _ {1: l - 1} ^ {\mathrm {A S R}}\right) \tag {8}
$$
$$
\mathbf {h} ^ {E _ {\mathrm {S T}}} = \operatorname {E N C O D E R} _ {\mathrm {S T}} \left(\hat {\mathbf {h}} ^ {D _ {\mathrm {A S R}}}\right) \tag {9}
$$
$$
\hat {\mathbf {h}} _ {l} ^ {D _ {\mathrm {S T}}} = \operatorname {D E C O D E R} _ {\mathrm {S T}} \left(\mathbf {h} ^ {E _ {\mathrm {S T}}}, \hat {y} _ {1: l - 1} ^ {\mathrm {S T}}\right) \tag {10}
$$
As we can see from Equations 9 and 10, the MT sub-network attends only to the decoder representations,  $\hat{\mathbf{h}}^{D_{\mathrm{ASR}}}$ , of the ASR sub-network, which could lead to the error propagation issues from the ASR sub-network to the MT sub-network similar to the cascade systems, as mentioned in §1. To alleviate this problem, we modify equation 10 such that  $\mathrm{DECODER}_{\mathrm{ST}}$  attends to both  $\mathbf{h}^{E_{\mathrm{ST}}}$  and  $\mathbf{h}^{E_{\mathrm{ASR}}}$ :
$$
\hat {\mathbf {h}} _ {l} ^ {D _ {\mathrm {S T}} ^ {\mathrm {S A}}} = \operatorname {D E C O D E R} _ {\mathrm {S T}} ^ {\mathrm {S A}} \left(\mathbf {h} ^ {E _ {\mathrm {S T}}}, \mathbf {h} ^ {E _ {\mathrm {A S R}}}, \hat {y} _ {1: l - 1} ^ {\mathrm {S T}}\right) \tag {11}
$$
We use the multi-sequence cross-attention discussed by Helcl et al. (2018), shown on the right side of Figure 1, to condition the final outputs generated by  $\hat{\mathbf{h}}_l^{D_{\mathrm{ST}}}$  on both speech and transcript information in an attempt to allow our network to recover from intermediate mistakes during inference. We call this model the Multi-Decoder w/ Speech-Attention.
4 Baseline Encoder-Decoder Model
For our baseline model, we use an end-to-end encoder-decoder (Enc-Dec) ST model with ASR joint training (Inaguma et al., 2020) as an auxiliarily loss to the speech encoder. In other words, the model consumes speech input using the ENCODER $_{ASR}$ , to produce  $\mathbf{h}^{E_{ASR}}$ , which is used for cross-attention by  $\text{DECODER}_{ASR}$  and the  $\text{DECODER}_{ST}$ . Using the decomposed ASR task as an auxiliary loss also helps the baseline Enc-Dec model and provide strong baseline performance, as we will see in Section 6.
5 Data and Experimental Setup
Data: We demonstrate the efficacy of our proposed approach on ST in the Fisher-CallHome cor
3The algorithm shown only considers a single top approximation of the search; however, with added time-complexity, the final task prediction improves with the n-best  $\mathbf{h}_{\mathrm{BEAM}}^{D_{\mathcal{B}}}$  for selecting the best resultant  $\mathcal{C}$ .
pus (Post et al., 2013) which contains 170 hours of Spanish conversational telephone speech, transcriptions, and English translations. All punctuations except apostrophes were removed and results are reported in terms of detokenized case-insensitive BLEU (Papineni et al., 2002; Post, 2018). We compute BLEU using the 4 references in Fisher (dev, dev2, and test) and the single reference in CallHome (dev and test) (Post et al., 2013; Kumar et al., 2014; Weiss et al., 2017). We use a joint source and target vocabulary of 1K byte pair encoding (BPE) units (Kudo and Richardson, 2018).
We prepare the corpus using the ESPnet library and we follow the standard data preparation, where inputs are globally mean-variance normalized log-mel filterbank and pitch features from up-sampled  $16\mathrm{kHz}$  audio (Watanabe et al., 2018). We also apply speed perturbations of 0.9 and 1.1 and the SS SpecAugment policy (Park et al., 2019).
Baseline Configuration: All of our models are implemented using the ESPnet library and trained on 3 NVIDIA Titan 2080Ti GPUs for  $\approx 12$  hours. For the Baseline Enc-Dec baseline, discussed in §4, we use an  $\mathrm{ENCODER}_{\mathrm{ASR}}$  consisting of a convolutional sub-sampling by a factor of 4 (Watanabe et al., 2018) and 12 transformer encoder blocks with 2048 feed-forward dimension, 256 attention dimension, and 4 attention heads. The  $\mathrm{DECODER}_{\mathrm{ASR}}$  and  $\mathrm{DECODER}_{\mathrm{ST}}$  both consist of 6 transformer decoder blocks with the same configuration as  $\mathrm{ENCODER}_{\mathrm{ASR}}$ . There are 37.9M trainable parameters. We apply dropout of 0.1 for all components, detailed in the Appendix (A.1).
We train our models using an effective batch-size of 384 utterances and use the Adam optimizer (Kingma and Ba, 2015) with inverse square root decay learning rate schedule. We set learning rate to 12.5, warmup steps to  $25\mathrm{K}$ , and epochs to 50. We use joint training with hybrid CTC/attention ASR (Watanabe et al., 2017) by setting mtl-alpha to 0.3 and asr-weight to 0.5 as defined by Watanabe et al. (2018). During inference, we perform beam search (Seki et al., 2019) on the ST sequences, using a beam size of 10, length penalty of 0.2, max length ratio of 0.3 (Watanabe et al., 2018).
Multi-Decoder Configuration: For the MultiDecoder ST model, discussed in §3, we use the same transformer configuration as the baseline for the  $\mathrm{ENCODER}_{\mathrm{ASR}}$ ,  $\mathrm{DECODER}_{\mathrm{ASR}}$ , and  $\mathrm{DECODER}_{\mathrm{ST}}$ . Additionally, the Multi-Decoder