SlowGuess/ACL_2020_OCR huggingface dataset:
All our models are implemented based on ESPnet. We set the model dimension  $d_{model}$  to 256, the head number  $H$  to 4, the feed forward layer size  $d_{ff}$  to 2048. For LibriSpeech expanded setting,  $d_{model} = 512$  and  $H = 8$ . For all the ST models, we set the number of encoder blocks  $N_e = 12$  and the number of decoder blocks  $N_d = 6$ . Unless noted, we use  $N = 8$  encoder blocks to perform the ASR and the FMLM pre-training tasks. For MT model used in IWSLT expanded setting, we use the Transformer architecture in Vaswani et al. (2017) with  $N_e = 6$ ,  $N_d = 6$ ,  $H = 4$ ,  $d_{model} = 256$ .
We train the model with 4 Tesla P40 GPUs and batch size is set to 64 per GPU. The pre-training takes 50 and 20 epochs for each phase and the final ST task takes another 50 epochs (a total of 120 epochs). We use the Adam optimizer with warmup steps 25000 in each phase. The learning rate decays proportionally to the inverse square root of the step number after 25000 steps. We
5https://www.readbeyond.it/aeneas
<sup>6</sup>LibriSpeech En-Fr, IWSLT En-De and Fisher-CallHome Es-En
$^{7}$ https://wit3.fbk.eu/mt.php?release=2017-01-trnted
<sup>8</sup>Europarl v7, Common Crawl, News Comentary v13 and Rapid corpus of EU press releases.
Method	Enc pre-train	Dec pre-train	BLEU
MT(Berard et al., 2018)*	-	-	19.3
MT(Inaguma et al., 2019)	-	-	18.3
base setting
LSTM ST (Berard et al., 2018)*	12.9
+pre-train+multitask (Berard et al., 2018)*	✓	✓	13.4
LSTM ST+pre-train (ESPnet)	✓	✓	16.68
Transformer+pre-train (Liu et al., 2019)	✓	✓	14.30
+knowledge distillation(Liu et al., 2019)	17.02
TCEN-LSTM (Wang et al., 2019b)	✓	✓	17.05
Transformer+ASR pre-train	✓	15.97
Transformer+curriculum pre-train	✓	17.66
expanded setting
LSTM+pre-train+SpecAugment(Bahar et al., 2019)	✓(236h)	✓	17.0
Multilingual ST+pre-train (Inaguma et al., 2019)	✓(472h)	17.6
Transformer+ASR pre-train	✓(960h)	16.90
Transformer+curriculum pre-train	✓(960h)	18.01
Table 1: Comparison on LibriSpeech En-Fr test set. The size of ASR data for base setting is 100h unless labeled. Since inputs of the MT models are ground-truth text, the results of MT models can be seen as the upper-bound of ST models. *: Unknown BLEU score script.
save checkpoints every epoch and average the last 5 checkpoints as the final model. To avoid overfitting, SpecAugment strategy (Park et al., 2019) is used in ASR pre-training with frequency masking  $(\mathrm{F} = 30, \mathrm{mF} = 2)$  and time masking  $(\mathrm{T} = 40, \mathrm{mT} = 2)$ . The decoding process uses a beam size of 10 and a length penalty of 0.2.
4.4.1 Comparison with End-to-End Baselines
LibriSpeech En-Fr: The results on LibriSpeech En-Fr test set are listed in Table 1. In base setting, our method improves the "Transformer+ASR pre-train" baseline by 1.7 BLEU and beats all the previous works, even though we do not pre-train the decoder. It indicates that through a well-designed learning process, the encoder has a strong potential to incorporate large amount of knowledge. Our method beats a knowledge distillation baseline, where an MT model is utilized to teach the ST model. The reason, we believe, is that our method gives the model more training signals and makes it easier to learn. We also outperform a TCEN baseline which includes two encoders. Compared to them, our method is more flexible and incorporates all information into a single encoder, which avoids the representation gap between the two encoders.
As the ASR data size increases, the model performs better. In the expanded setting, we find the FBLT task performs poorly compared with the base setting. This is because the target word prediction task is dictionary-supervised in expanded setting rather than reference-supervised as in base setting. However, our method still outperforms the simple
pre-training method by a large margin. Besides, it is surprising to find that the end-to-end ST model is approaching the performance of an MT model, which is the upper bound of the ST model since it accepts golden source sentence without any ASR errors. This further verifies the effectiveness of our method.
IWSLT En-De: The results on IWSLT tst2013 are listed in Table 2, showing a similar trend as in LibriSpeech dataset. We find that the segmentation methods have a big influence on the final results. In the base setting, our method can improve the ASR pre-training baseline by 0.9 to 2.2 BLEU scores, depending on the segmentation methods. In the expanded setting, we find when combined with decoder pre-train, the performance is further improved and beats other expanded baselines.
4.4.2 Comparison with Cascaded Baselines
Table 3 shows comparison with cascaded ST systems. For the base setting of two tasks, our end-to-end model can achieve comparable or better results with cascaded methods. This shows the end-to-end model has powerful learning capabilities and combines the functions of two models. In the LibriSpeech expanded setting, when more ASR data is available, we also obtain a competitive performance. This indicates our method can make a good use of ASR corpus and learn valuable linguistic knowledge other than simple acoustic information. However, when additional MT data is used, there is still a gap between the end-to-end method and the cascaded method. How to utilize bilingual parallel sentences to improve the E2E ST model is worth
Method	Enc pre-train (speech data)	Dec pre-train (text data)	segment method
LIUM	aeneas
base setting
ESPnet	12.50	-
+enc pre-train	✓	13.12	-
+enc dec pre-train	✓	✓	13.54	-
Transformer+ASR pre-train	✓	15.35	17.10
Transformer+curriculum pre-train	✓	16.27	19.29
expanded setting
Multilingual ST+pre-train(Inaguma et al., 2019)	✓(472h)	14.6	-
TCEN-LSTM (Wang et al., 2019b)	✓(479h)	✓(40M)	17.65	-
CL-fast(Kano et al., 2017)(re-implemented)	✓(479h)	14.33	16.23
Transformer+curriculum pre-train+dec pre-train	✓(479h)	✓(4M)	18.15	20.35
Table 2: ST results on IWSLT En-De tst2013 set.
Method	BLEU
LibriSpeech base setting
LSTM ASR+ MT(Berard et al., 2018)	14.6
LSTM ASR+ MT(Inaguma et al., 2019)	15.8
Transformer ASR + MT(Liu et al., 2019)	17.85
Ours E2E Transformer ST	17.66
LibriSpeech expanded setting
Transformer ASR+LSTM MT*	18.05
Ours E2E Transformer ST	18.01
IWSLT base setting
LSTM ASR+ MT(Inaguma et al., 2019)	14.0
Ours E2E Transformer ST	16.27
IWSLT expanded setting
Transformer ASR+Transformer ST	22.16
Ours E2E Transformer ST	18.15
further studying.
4.5 Analysis and Discussion
Ablation Study To better understand the contribution of each component, we perform an ablation study on LibriSpeech expanded setting. The results are shown in Table 4. On the one hand, we show that both of our proposed pre-training tasks are beneficial: In “-FMLM task” and “-FBLT task”, we perform single-task pre-training for advanced course. The performance drops when we remove either one of them. On the other hand, we show the two-phases pre-training paradigm is necessary: The “- phase 2” experiment degenerates to the simple ASR pre-training baseline. In “-phase 1” setting, we find that without the ASR pre-training, the training accuracy on FMLM task and FBLT task drops a lot, which further affects the ST performance. This means the ASR task is necessary for both the advanced courses and ST. In “Multi3”
Table 3: Comparison with cascaded ST. *:we find the LSTM model outperforms Transformer model in our setting since the training data is scarce.
Method	BLEU
Our method	18.01
-FMLM task	17.62
-FBLT task	17.65
-phase 2	16.90
-phase 1	14.26
Multi3	14.82
Table 4: Ablation study on LibriSpeech expanded setting. '- indicates removing the task or phase from our method.
setting, we pre-train the model on ASR, FMLM and FBLT tasks in one phase. In this setting, we observe multi-task learning also decrease individual task performances (ASR, FMLM and FBLT) compared to curriculum learning. One reasonable explanation is that it is hard to train on the FMLM and FBLT tasks which takes masked input from randomly initialized parameters, which also leads to performance degradation on the ST task.
Hyper-parameter N During pre-training, which layer conducts ASR pre-training and FMLM loss is an important hyper-parameter. We conduct experiments on LibriSpeech base setting to explore the influence of different choices. We keep  $N_{e} = 12$  unchanged and always use the top layer to perform the FBLT task. Then we alter the hyperparameter  $N$ . We find if  $N = 6$ , the model finds it difficult to converge during ST training. That may be because the distance between the decoder and the bottom 6 encoder layers is too far so that the valuable source linguistic knowledge can not be well utilized. Moreover, the model performs undesirable if the choice is 10 or 12, which results in 16.47 and 16.14 BLEU score respectively, since the number of blocks for FBLT task is not enough. The model achieves the best performance when we choose  $N = 8$ . Thus, we use this strategy in our main experiments.
Unlabeled Speech Data In this work, we also ex
we use 12-layer encoder for ASR and FMLM pre-training for a fair comparison.
plore how to utilize the unlabeled speech data in pre-training, but only get negative results. We conduct exploratory experiments on the LibriSpeech ST task. Assume that the  $(x,y^s)$  from 100h ST corpus as labeled pre-training data and  $(x)$  from 960h LibriSpeech ASR corpus as unlabeled data. Following Jiang et al. (2019), we design an unsupervised pre-training task for elementary course, in which we randomly mask  $15\%$  of bank features and let the bottom 4 encoder layers predict the masked part. We compute the L1 loss between the prediction and groundtruth filterbanks. However, we find that this method is not helpful for the final ST task, which results in 16.85 BLEU score, lower than our base setting model (without extra data pre-training). It is still an open question about how to use unlabeled speech data.
This paper investigates the end-to-end method for ST. We propose a curriculum pre-training method, consisting of an elementary course with an AS-R loss, and two advanced courses with a frame-based masked language model loss and a bilingual lexicon translation loss, in order to teach the model syntactic and semantic knowledge in the pre-training stage. Empirical studies have demonstrated that our model significantly outperforms baselines. In the future, we will explore how to leverage unlabeled speech data and large bilingual text data to further improve the performance. Besides, we expect the idea of curriculum pre-training can be adopted on other NLP tasks.
This work was supported in part by the National Natural Science Foundation of China under Grant No.U1636116 and the Ministry of education of Humanities and Social Science project under grant 16YJC790123.
Antonios Anastasopoulos, David Chiang, and Long Duong. 2016. An unsupervised probability model for speech-to-translation alignment of low-resource languages. In EMNLP 2016, pages 1255–1263. The Association for Computational Linguistics.
Parnia Bahar, Albert Zeyer, Ralf Schlüter, and Hermann Ney. 2019. On using specaugment for end-to-end speech translation. CoRR, abs/1911.08876.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In ICLR 2015.
Sameer Bansal, Herman Kamper, Karen Livescu, Adam Lopez, and Sharon Goldwater. 2018. Low-resource speech-to-text translation. In *Interspeech* 2018, 19th Annual Conference of the International Speech Communication Association, Hyderabad, India, 2-6 September 2018, pages 1298-1302. ISCA.
Sameer Bansal, Herman Kamper, Karen Livescu, Adam Lopez, and Sharon Goldwater. 2019. Pre-training on high-resource speech recognition improves low-resource speech-to-text translation. In *NAACL-HLT* 2019, pages 58-68. Association for Computational Linguistics.
Oliver Bender, Richard Zens, Evgeny Matusov, and Hermann Ney. 2004. Alignment templates: the RWTH SMT system. In IWSLT 2004, pages 79-84.
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. 2009. Curriculum learning. In ICML 2009, pages 41-48.
Alexandre Berard, Laurent Besacier, Ali Can Kocabiyikoglu, and Olivier Pietquin. 2018. End-to-end automatic speech translation of audiobooks. In I-CASSP 2018, pages 6224-6228. IEEE.
Alexandre Berard, Olivier Pietquin, Christophe Servan, and Laurent Besacier. 2016. Listen and translate: A proof of concept for end-to-end speech-to-text translation. CoRR, abs/1612.01744.
William Chan, Navdeep Jaitly, Quoc V. Le, and Oriol Vinyals. 2016. Listen, attend and spell: A neural network for large vocabulary conversational speech recognition. In ICASSP 2016, pages 4960-4964.
Chung-Cheng Chiu, Tara N. Sainath, Yonghui Wu, Rohit Prabhavalkar, Patrick Nguyen, Zhifeng Chen, Anjuli Kannan, Ron J. Weiss, Kanishka Rao, Ekaterina Gonina, Navdeep Jaitly, Bo Li, Jan Chorowski, and Michiel Bacchiani. 2018. State-of-the-art speech recognition with sequence-to-sequence models. In ICASSP 2018, pages 4774-4778.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. In *NAACL-HLT* 2019, pages 4171–4186.
Long Duong, Antonios Anastasopoulos, David Chiang, Steven Bird, and Trevor Cohn. 2016. An attentional model for speech translation without transcription. In *NAACL HLT* 2016, pages 949–959. The Association for Computational Linguistics.
Chen Gong, Dacheng Tao, Stephen J. Maybank, Wei Liu, Guoliang Kang, and Jie Yang. 2016. Multimodal curriculum learning for semi-supervised image classification. IEEE Trans. Image Processing, 25(7):3249-3260.
3736
Alex Graves, Marc G. Bellemare, Jacob Menick, Rémi Munos, and Koray Kavukcuoglu. 2017. Automated curriculum learning for neural networks. In ICML 2017, pages 1311-1320.
Hirofumi Inaguma, Kevin Duh, Tatsuya Kawahara, and Shinji Watanabe. 2019. Multilingual end-to-end speech translation. In ASRU 2019, pages 570-577. IEEE.
Ye Jia, Melvin Johnson, Wolfgang Macherey, Ron J. Weiss, Yuan Cao, Chung-Cheng Chiu, Naveen Ari, Stella Laurenzo, and Yonghui Wu. 2019. Leveraging weakly supervised data to improve end-to-end speech-to-text translation. In ICASSP 2019, pages 7180-7184. IEEE.
Dongwei Jiang, Xiaoning Lei, Wubo Li, Ne Luo, Yuxuan Hu, Wei Zou, and Xiangang Li. 2019. Improving transformer-based speech recognition using unsupervised pre-training. CoRR, abs/1910.09932.
Takatomo Kano, Sakriani Sakti, and Satoshi Nakamura. 2017. Structured-based curriculum learning for end-to-end english-japanese speech translation. In Interspeech 2017, pages 2630-2634.
Shigeki Karita, Xiaofei Wang, Shinji Watanabe, Takenori Yoshimura, Wangyou Zhang, Nanxin Chen, Tomoki Hayashi, Takaaki Hori, Hirofumi Inaguma, Ziyan Jiang, Masao Someki, Nelson Enrique Yalta Soplin, and Ryuichi Yamamoto. 2019. A comparative study on transformer vs RNN in speech applications. In IEEE Automatic Speech Recognition and Understanding Workshop, ASRU 2019, Singapore, December 14-18, 2019, pages 449-456. IEEE.
Ali Can Kocabiyikoglu, Laurent Besacier, and Olivier Kraif. 2018. Augmenting librispeech with french translations: A multimodal corpus for direct speech translation evaluation. In LREC 2018.
Taku Kudo and John Richardson. 2018. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In EMNLP 2018: System Demonstrations, pages 66-71.
Yuchen Liu, Hao Xiong, Zhongjun He, Jiajun Zhang, Hua Wu, Haifeng Wang, and Chengqing Zong. 2019. End-to-end speech translation with knowledge distillation. In InterSpeech 2019, volume abs/1904.08075.
Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Effective approaches to attention-based neural machine translation. In EMNLP 2015, pages 1412-1421.
Lambert Mathias and William Byrne. 2006. Statistical phrase-based speech translation. In ICASSP 2006, pages 561-564.
Evgeny Matusov, Stephan Kanthak, and Hermann Ney. 2005. On the integration of speech recognition and statistical machine translation. In *INTERSPEECH* 2005, pages 3177-3180.
Michael McAuliffe, Michaela Socolof, Sarah Mihuc, Michael Wagner, and Morgan Sonderegger. 2017. Montreal forced aligner: Trainable text-speech alignment using kaldi. In Interspeech 2017, pages 498-502.
Sylvain Meignier and Teva Merlin. 2010. Lium spkdiarization: an open source toolkit for diarization. In CMU SPUD Workshot.
Hermann Ney. 1999. Speech translation: coupling of recognition and translation. In ICASSP '99, pages 517-520.
Jan Niehues, Ronaldo Cattoni, Sebastian Stüker, Mauro Cettolo, Marco Turchi, and Marcello Federico. 2018. The iwslt 2018 evaluation campaign. In Proceedings of IWSLT.
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. 2015. Librispeech: An ASR corpus based on public domain audio books. In ICASSP 2015, pages 5206-5210.
Daniel S. Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin D. Cubuk, and Quoc V. Le. 2019. Specaugment: A simple data augmentation method for automatic speech recognition. CoRR, abs/1904.08779.
Anthony Rousseau, Paul Deléglise, and Yannick Esteve. 2014. Enhancing the TED-LIUM corpus with selected data for language modeling and more TED talks. In LREC 2014, pages 3935-3939. European Language Resources Association (ELRA).
Matthias Sperber, Graham Neubig, Jan Niehues, and Alex Waibel. 2019. Attention-passing models for robust and data-efficient end-to-end speech translation. TACL, 7:313-325.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NeurIPS 2017, pages 5998-6008.
Chengyi Wang, Yu Wu, Yujiao Du, Jinyu Li, Shujie Liu, Liang Lu, Shuo Ren, Guoli Ye, Sheng Zhao, and Ming Zhou. 2019a. Semantic mask for transformer based end-to-end speech recognition. CoRR, abs/1912.03010.
Chengyi Wang, Yu Wu, Shujie Liu, Zhenglu Yang, and Ming Zhou. 2019b. Bridging the gap between pretraining and fine-tuning for end-to-end speech translation. CoRR, abs/1909.07575.
Shinji Watanabe, Takaaki Hori, Shigeki Karita, Tomo-ki Hayashi, Jiro Nishitoba, Yuya Unno, Nelson Enrique Yalta Soplin, Jahn Heymann, Matthew Wiesner, Nanxin Chen, Adithya Renduchintala, and Tsubasa Ochiai. 2018. Espnet: End-to-end speech processing toolkit. In *Interspeech* 2018, pages 2207–2211. ISCA.
Ron J. Weiss, Jan Chorowski, Navdeep Jaitly, Yonghui Wu, and Zhifeng Chen. 2017. Sequence-to-sequence models can directly translate foreign speech. In *Interspeech* 2017, pages 2625-2629. ISCA.