SlowGuess/NAACL_2021_OCR huggingface dataset:
has an  $\mathrm{ENCODER}_{\mathrm{ST}}$  consisting of 2 transformer encoder blocks with the same configuration as  $\mathrm{ENCODER}_{\mathrm{ASR}}$ , giving a total of  $40.5\mathrm{M}$  trainable parameters. The training configuration is also the same as for the baseline. For the Multi-Decoder w/ Speech-Attention model (42.1M trainable parameters), we increase the attention dropout of the ST decoder to 0.4 and dropout on all other components of the ST decoder to 0.2 while keeping dropout on the remaining components at 0.1. We verified that increasing the dropout does not help the vanilla multi-decoder ST model.
During inference, we perform beam search on both the ASR and ST output sequences, as discussed in §3. The ST beam search is identical to that of the baseline. For the intermediate ASR beam search, we use a beam size of 16, length penalty of 0.2, max length ratio of 0.3. In some of our experiments, we also include fusion of a source language LM with a 0.2 weight and CTC with a 0.3 weight to re-score the intermediate ASR beam search (Watanabe et al., 2017). For the Speech-Attention variant, we increase LM weight to 0.4.
Note that the ST beam search configuration remains constant across our baseline and Multi-Decoder experiments as our focus is on improving overall performance through searchable intermediate representations. Thus, the various re-scoring techniques applied to the ASR beam search are options newly enabled by our proposed architecture and are not used in the ST beam search.
Table 1 presents the overall ST performance (BLEU) of our proposed Multi-Decoder model. Our model improves by  $+2.9 / +0.3$  (Fisher/CallHome) over the best cascaded baseline and by  $+5.6 / +1.5$  BLEU over the best published end-to-end baselines. With Speech-Attention, our model improves by  $+3.4 / +1.6$  BLEU over the cascaded baselines and  $+7.1 / +2.8$ BLEU over encoder-decoder baselines. Both the Multi-Decoder and Multi-Decoder w/ Speech-Attention on average are further improved by  $+0.9 / +0.4$  BLEU through ASR re-scoring. $^{4}$
Table 1 also includes our implementation of the Baseline Enc-Dec model discussed in §4. In this way, we are able to make a fair comparison with our framework as we control the model and inference
4We also evaluate our models using other MT metrics to supplement these results, as shown in the Appendix (A.2).
Model Type	Model Name	Uses Speech Transcripts	Fisher	CallHome
dev(↑)	dev2(↑)	test(↑)	dev(↑)	test(↑)
Cascade	Inaguma et al. (2020)	✓	41.5	43.5	42.2	19.6	19.8
Cascade	ESPnet ASR+MT (2018)	✓	50.4	51.2	50.7	19.6	19.2
Enc-Dec	Weiss et al. (2017)◇	✗	46.5	47.3	47.3	16.4	16.6
Enc-Dec	Weiss et al. (2017)◇	✓	48.3	49.1	48.7	16.8	17.4
Enc-Dec	Inaguma et al. (2020)	✓	46.6	47.6	46.5	16.8	16.8
Enc-Dec	Guo et al. (2021)	✓	48.7	49.6	47.0	18.5	18.6
Enc-Dec	Our Implementation	✓	49.6	50.9	49.5	19.1	18.2
Multi-Decoder	Our Proposed Model	✓	52.7	53.3	52.6	20.5	20.1
Multi-Decoder	+ASR Re-scoring	✓	53.3	54.2	53.7	21.1	20.8
Multi-Decoder	+Speech-Attention	✓	54.6	54.6	54.1	21.7	21.4
Multi-Decoder	+ASR Re-scoring	✓	55.2	55.2	55.0	21.7	21.5
Table 1: Results presenting the overall performance (BLEU) of our proposed multi-decoder model. Cascade and Enc-Dec results from previous papers and our own implementation of the Enc-Dec are shown for comparison. The best performing models are highlighted. Implemented with LSTM, while all others are Transformer-based.
Model	Overall ST(↑)	Sub-Net ASR(↓)	Sub-Net MT(↑)
Multi-Decoder	52.7	22.6	64.9
+Speech-Attention	54.6	22.4	66.6
Table 2: Results presenting the overall ST performance (BLEU) of our Multi-Decoder models, along with their sub-net ASR (\% WER) and MT (BLEU) performances. All results are from the Fisher dev set.
configurations to be analogous. For instance, we keep the same search parameters for the final output in the baseline and the Multi-Decoder to demonstrate impact of the intermediate beam search.
6.1 Benefits
6.1.1 Sub-network performance monitoring
An added benefit of our proposed approach over the Baseline Enc-Dec is the ability to monitor the individual performances of the ASR (\% WER) and MT (BLEU) sub-nets as shown in Table 2. The Multi-Decoder w/ Speech-Attention shows a greater MT sub-net performance than the Multi-Decoder as well as a slight improvement of the ASR sub-net, suggesting that ST can potentially help ASR.
6.1.2 Beam search for better intermediates
The overall ST performance improves when a higher beam size is used in the intermediate ASR search, and this increase can be attributed to the improved ASR sub-net performance. Figure 1 shows this trend across ASR beam sizes of 1, 4, 8, 10, 16 while fixing the ST decoding beam size to 10. A
Figure 2: Results studying the effect of the different ASR beam sizes in the intermediate representation search on the overall ST performance (BLEU) and the ASR sub-net performance (\%WER) for our multidecoder model. Beam of 1 is same as greedy search.
beam size of 1, which is a greedy search, results in lower ASR sub-net and overall ST performances. As beam sizes become larger, gains taper off as can be seen between beam sizes of 10 and 16.
6.1.3 External models for better search
External models like CTC acoustic models and language models are commonly used for re-scoring encoder-decoder models (Hori et al., 2017), due to the difference in their modeling capabilities. CTC directly models transcripts while being conditionally independent on the other outputs given the input, and LMs predict the next token in a sequence.
Both variants of the Multi-Decoder improve due to improved ASR sub-net performance using extern
Model	Overall ST(↑)	Sub-Net ASR(↓)
Multi-Decoder	52.7	22.6
+ASR Re-scoring w/ LM	53.2	22.6
+ASR Re-scoring w/ CTC	52.8	22.1
+ASR Re-scoring w/ LM	53.3	21.7
Multi-Decoder w/ Speech-Attn.	54.6	22.4
+ASR Re-scoring w/ LM	55.1	22.4
+ASR Re-scoring w/ CTC	54.7	22.0
+ASR Re-scoring w/ LM	55.2	21.9
Table 3: Results presenting the overall ST performance (BLEU) and the sub-net ASR (\%WER) of our MultiDecoder models with external CTC and LM re-scoring in the ASR intermediate representation search. All results are from the Fisher dev set.
nal CTC and LM models for re-scoring, as shown in Table 3. We use a recurrent neural network LM trained on the Fisher-CallHome Spanish transcripts with a dev perplexity of 18.8 and the CTC model from joint loss applied during training. Neither external model incorporates additional data. Although the impact of the LM-only re-scoring is not shown in the ASR % WER, it reduces substitution and deletion rates in the ASR and this is observed to help the overall ST performance.
6.1.4 Error propagation avoidance
As discussed in §3, our Multi-Decoder model inherits the error propagation issue as can be seen in Figure 3. For the easiest bucket of utterances with  $< 40\%$  WER in Multi-Decoder's ASR subnet, our model's ST performance, as measured by the corpus BLEU of the bucket, exceeds that of the Baseline Enc-Dec. The inverse is true for the more difficult bucket of  $[40,80)\%$ , showing that error propagation is limiting the performance of our model; however, we show that multi-sequence attention can alleviate this issue. For extremely difficult utterances in the  $\geq 80\%$  bucket, ST performance for all three approaches is suppressed. We also provide qualitative examples of error propagation avoidance in the Appendix (A.3).
6.2 Generalizability
In this section, we discuss the generalizability of our framework towards out-of-domain data. We also extend our Multi-Decoder model to other sequence tasks like speech recognition. Finally, we apply our ST models to a larger corpus with more language pairs and a different domain of speech.
Figure 3: Results comparing the ST performances (BLEU) of our Baseline Enc-Dec, Multi-Decoder, and Multi-Decoder w/ Speech-Attention across different ASR difficulties measured using  $\%$  WER on the Fisher dev set (1-ref). The buckets on the x-axis are determined using the utterance level  $\%$  WER using the Multi-Decoder ASR sub-net performance.
6.2.1 Robustness through Decomposition
Like cascaded systems, searchable intermediates provide our model adaptability in individual subsystems towards out-of-domain data using external in-domain language model, thereby giving access to more in-domain data. Specifically for speech translation systems, this means we can use in-domain language models in both source and target languages. We test the robustness of our Multi-Decoder model trained on Fisher-CallHome conversational speech dataset on read speech CoVost-2 dataset (Wang et al., 2020b). In Table 4 we show that re-scoring the ASR sub-net with an in-domain LM improves ASR with around  $10.0\%$  lower WER, improving the overall ST performance by around +2.5 BLEU. Compared to an in-domain ST baseline (Wang et al., 2020a), our out-of-domain Multi-Decoder with in-domain ASR re-scoring demonstrates the robustness of our approach.
6.2.2 Decomposing Speech Transcripts
We apply our generic framework to another decomposable sequence task, speech recognition, and show the results of various levels of decomposition in Table 5. We show that with phoneme, character, or byte-pair encoding (BPE) sequences as intermediates, the Multi-Decoder presents strong results on both Fisher and CallHome test sets. We also observe that the BPE intermediates perform bet
Model	Overall ST(↑)	Sub-Net ASR(↓)
IN-DOMAIN ST MODEL
Baseline (Wang et al., 2020b)	12.0	-
+ASR Pretrain (Wang et al., 2020b)◇	23.0	16.0
OUT-OF-DOMAIN ST MODEL
Multi-Decoder	11.8	46.8
+ASR Re-scoring w/ in-domain LM	14.4	36.7
Multi-Decoder w/ Speech-Attention	12.6	46.5
+ASR Re-scoring w/ in-domain LM	15.0	36.7
Table 4: Results presenting the overall ST performance (BLEU) and the sub-net ASR (\% WER) of our Multi-Decoder models when tested on out-of-domain data. All models were trained on the Fisher-CallHome  $\mathrm{Es} \rightarrow \mathrm{En}$  corpus and tested on CoVost2  $\mathrm{Es} \rightarrow \mathrm{En}$  corpus. Pretrained with 364 hours of in-domain ASR data.
Model	Intermediate	Fisher ASR(↓)	CallHome ASR(↓)
Enc-Dec◇	-	23.2	45.3
Multi-Decoder	Phoneme	20.7	40.0
Multi-Decoder	Character	20.4	39.9
Multi-Decoder	BPE100	19.7	38.9
ter than phoneme/character variants, which could be attributed to the reduced search capabilities of encoder-decoder models using beam search on longer sequences (Sountsov and Sarawagi, 2016) like in phoneme/character sequences.
6.2.3 Extending to MuST-C Language Pairs
In addition to our results using the 170 hours of the Spanish-English Fisher-CallHome corpus, in Table 6 we show that our decompositional framework is also effective on larger ST corpora. In particular, we use 400 hours of English-German and 500 hours of English-French ST from the MuST-C corpus (Di Gangi et al., 2019). Our Multi-Decoder model improves by  $+2.7$  and  $+1.5$  BLEU, in German and French respectively, over end-to-end baselines from prior works that do not use additional training data. We show that ASR re-scoring gives an additional  $+0.1$  and  $+0.4$  BLEU improvement.[5]
By extending our Multi-Decoder models to this MuST-C study, we show the generalizability of our
Table 5: Results presenting the % WER ASR performance when using the Multi-Decoder model on decomposed ASR task with phoneme, character, and BPE100 as intermediates. All results are from the Fisher-CallHome Spanish corpus. (Weiss et al., 2017)
Model	En→De ST(↑)	En→Fr ST(↑)
NeurST (Zhao et al., 2020)	22.9	33.3
Fairseq S2T (Wang et al., 2020a)	22.7	32.9
ESPnet-ST (Inaguma et al., 2020)	22.9	32.7
Dual-Decoder (Le et al., 2020)	23.6	33.5
Multi-Decoder w/ Speech-Attn. +ASR Re-scoring	26.3	37.0
26.4	37.4
Table 6: Results presenting the overall ST performance (BLEU) of our Multi-Decoder w/ Speech-Attention models with ASR re-scoring across two language-pairs, English-German  $(\mathrm{En}\rightarrow \mathrm{De})$  and English-French  $(\mathrm{En}\rightarrow \mathrm{Fr})$ . All results are from the MuST-C tst-COMMON sets. All models use speech transcripts.
approach across several dimensions of ST tasks. First, our approach consistently improves over baselines across multiple language-pairs. Second, our approach is robust to the distinct domains of telephone conversations from Fisher-CallHome and the TED-Talks from MuST-C. Finally, by scaling from 170 hours of Fisher-CallHome data to 500 hours of MuST-C data, we show that the benefits of decomposing sequence tasks with searchable hidden intermediates persist even with more data.
Furthermore, the performance of our Multi-Decoder models trained with only English-German or English-French ST data from MuST-C is comparable to other methods which incorporate larger external ASR and MT data in various ways. For instance, Zheng et al. (2021) use 4700 hours of ASR data and 2M sentences of MT data for pretraining and multi-task learning. Similarly, Bahar et al. (2021) use 2300 hours of ASR data and 27M sentences of MT data for pretraining. Our competitive performance without the use of any additional data highlights the data-efficient nature of our proposed end-to-end framework as opposed to the baseline encoder-decoder model, as pointed out by Sperber and Paulik (2020).
7 Discussion and Relation to Prior Work
Compositionality: A number of recent works have constructed composable neural network modules for tasks such as visual question answering (Andreas et al., 2016), neural MT (Raunak et al., 2019), and synthetic sequence-to-sequence tasks (Lake, 2019). Modules that are first trained separately can subsequently be tightly integrated into a single end-to-end trainable model by passing differentiable soft decisions instead of discrete decisions
Details of the MuST-C data preparation and model parameters are detailed in Appendix (A.4).
in the intermediate stage (Bahar et al., 2021). Further, even a single encoder-decoder model can be decomposed into modular components where the encoder and decoder modules have explicit functions (Dalmia et al., 2019).
Joint Training with Sub-Tasks: End-to-end sequence models been shown to benefit from introducing joint training with sub-tasks as auxiliary loss functions for a variety of tasks like ASR (Kim et al., 2017), ST (Salesky et al., 2019; Liu et al., 2020a; Dong et al., 2020; Le et al., 2020), SLU (Haghani et al., 2018). They have been shown to induce structure (Belinkov et al., 2020) and improve the model performance (Toshniwal et al., 2017), but this joint training may reduce data efficiency if some sub-nets are not included in the final end-to-end model (Sperber et al., 2019; Wang et al., 2020c). Our framework avoids this sub-net waste at the cost of computational load during inference.
Speech Translation Decoders: Prior works have used ASR/MT decoding to improve the overall ST decoding through synchronous decoding (Liu et al., 2020a), dual decoding (Le et al., 2020), and successive decoding (Dong et al., 2020). These works partially or fully decode ASR transcripts and use discrete intermediates to assist MT decoding. Tu et al. (2017) and Anastasopoulos and Chiang (2018) are closest to our multi-decoder ST model, however the benefits of our proposed framework are not entirely explored in these works.
Two-Pass Decoding: Two-pass decoding involves first predicting with one decoder and then re-evaluating with another decoder (Geng et al., 2018; Sainath et al., 2019; Hu et al., 2020; Rijhwani et al., 2020). The two decoders iterate on the same sequence, so there is no decomposition into sub-tasks in this method. On the other hand, our approach provides the subsequent decoder with a more structured representation than the input by decomposing the complexity of the overall task. Like two-pass decoding, our approach provides a sense of the future to the second decoder which allows it to correct mistakes from the previous first decoder.
Auto-Regressive Decoding: As auto-regressive decoders inherently learn a language model along with the task at hand, they tend to be domain specific (Samarakoon et al., 2018; Müller et al., 2020). This can cause generalizability issues during inference (Murray and Chiang, 2018; Yang et al., 2018),
impacting the performance of both the task at hand and any downstream tasks. Our approach alleviates these problems through intermediate search, external models for intermediate re-scoring, and multi-sequence attention.
We present searchable hidden intermediates for end-to-end models of decomposable sequence tasks. We show the efficacy of our Multi-Decoder model on the Fisher-CallHome  $\mathrm{Es} \rightarrow \mathrm{En}$  and MuST-C  $\mathrm{En} \rightarrow \mathrm{De}$  and  $\mathrm{En} \rightarrow \mathrm{Fr}$  speech translation corpora, achieving state-of-the-art results. We present various benefits in our framework, including sub-net performance monitoring, beam search for better hidden intermediates, external models for better search, and error propagation avoidance. Further, we demonstrate the flexibility of our framework towards out-of-domain tasks with the ability to adapt our sequence model at intermediate stages of decomposition. Finally, we show generalizability by training Multi-Decoder models for the speech recognition task at various levels of decomposition.
We hope insights derived from our study stimulate research on tighter integrations between the benefits of cascaded and end-to-end sequence models. Exploiting searchable intermediates through beam search is just the tip of the iceberg for search algorithms, as numerous approximate search techniques like diverse beam search (Vijayakumar et al., 2018) and best-first beam search (Meister et al., 2020) have been recently proposed to improve diversity and approximation of the most-likely sequence. Incorporating differentiable lattice based search (Hannun et al., 2020) can also allow the subsequent sub-net to digest n-best representations.
This work started while Vikas Raunak was a student at CMU, he is now working as a Research Scientist at Microsoft. We thank Pengcheng Guo, Hirofumi Inaguma, Elizabeth Salesky, Maria Ryskina, Marta Mendez Simón and Vijay Viswanathan for their helpful discussion during the course of this project. We also thank the anonymous reviewers for their valuable feedback. This work used the Extreme Science and Engineering Discovery Environment (XSEDE) (Towns et al., 2014), which is supported by National Science Foundation grant number ACI-1548562. Specifically, it used the Bridges system (Nystrom et al., 2015), which is
supported by NSF award number ACI-1445606, at the Pittsburgh Supercomputing Center (PSC). The work was supported in part by an AWS Machine Learning Research Award. This research was also supported in part the DARPA KAIROS program from the Air Force Research Laboratory under agreement number FA8750-19-2-0200. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright notation there on. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of the Air Force Research Laboratory or the U.S. Government.
Antonios Anastasopoulos and David Chiang. 2018. Tied multitask learning for neural speech translation. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 82-91, New Orleans, Louisiana. Association for Computational Linguistics.
Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. 2016. Neural module networks. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages 39-48. IEEE Computer Society.
Parnia Bahar, Tobias Bieschke, Ralf Schlüter, and Hermann Ney. 2021. Tight integrated end-to-end training for cascaded speech translation. In 2021 IEEE Spoken Language Technology Workshop (SLT), pages 950-957. IEEE.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In 3rd International Conference on Learning Representations, ICLR 2015.
Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 65-72, Ann Arbor, Michigan. Association for Computational Linguistics.
Daniel Beck, Trevor Cohn, and Gholamreza Haffari. 2019. Neural speech translation using lattice transformations and graph networks. In Proceedings of the Thirteenth Workshop on Graph-Based Methods for Natural Language Processing (TextGraphs-13), pages 26-31, Hong Kong. Association for Computational Linguistics.
Yonatan Belinkov, Nadir Durrani, Fahim Dalvi, Hassan Sajjad, and James Glass. 2020. On the linguistic representational power of neural machine translation models. Computational Linguistics, 46(1):1-52.
Léon Bottou, Yoshua Bengio, and Yann Le Cun. 1997. Global training of document processing systems using graph transformer networks. In Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pages 489-494. IEEE.
Alice Coucke, Alaa Saade, Adrien Ball, Théodore Bluche, Alexandre Caulier, David Leroy, Clément Doumouro, Thibault Gisselbrecht, Francesco Caltagirone, Thibaut Lavril, et al. 2018. Snips voice platform: an embedded spoken language understanding system for private-by-design voice interfaces. In Privacy in Machine Learning and Artificial Intelligence workshop, ICML.
Siddharth Dalmia, Abdelrahman Mohamed, Mike Lewis, Florian Metze, and Luke Zettlemoyer. 2019. Enforcing encoder-decoder modularity in sequence-to-sequence models. arXiv preprint arXiv:1911.03782.
Mattia A. Di Gangi, Roldano Cattoni, Luisa Bentivogli, Matteo Negri, and Marco Turchi. 2019. MuST-C: a Multilingual Speech Translation Corpus. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2012-2017, Minneapolis, Minnesota. Association for Computational Linguistics.
Qianqian Dong, Mingxuan Wang, Hao Zhou, Shuang Xu, Bo Xu, and Lei Li. 2020. SDST: Successive decoding for speech-to-text translation. Proceedings of the Thirty-Fifth AAAI Conference on Artificial Intelligence.
Xinwei Geng, Xiaocheng Feng, Bing Qin, and Ting Liu. 2018. Adaptive multi-pass decoder for neural machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 523-532, Brussels, Belgium. Association for Computational Linguistics.
Pengcheng Guo, Florian Boyer, Xuankai Chang, Tomoki Hayashi, Yosuke Higuchi, Hirofumi Inaguma, Naoyuki Kamo, Chenda Li, Daniel Garcia-Romero, Jiatong Shi, et al. 2021. Recent developments on ESPnet toolkit boosted by conformer. In 2021 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE.
Parisa Haghani, Arun Narayanan, Michiel Bacchiani, Galen Chuang, Neeraj Gaur, Pedro Moreno, Rohit Prabhavalkar, Zhongdi Qu, and Austin Waters. 2018. From audio to semantics: Approaches to end-to-end spoken language understanding. In 2018 IEEE Spoken Language Technology Workshop (SLT), pages 720-726. IEEE.