SlowGuess/ACL_2020_OCR huggingface dataset:
delay by using the same  $k$ . For cascaded method, we try all possible combinations of wait-  $k$  ASR and wait-  $k$  NMT models and report the best one. The accuracy of the two methods is shown in Table 3. It can be seen that 1) SimulSpeech outperforms the cascaded method when  $k < 9$  which covers most simultaneous translation scenarios. 2) Cascaded model only outperforms SimulSpeech in larger  $k^5$ . These results demonstrate the advantages of SimulSpeech specifically for simultaneous translation scenario. We further plot the BLEU scores of the two methods in Figure 6. It can be seen that SimulSpeech with wait-3 can achieve the same BLEU score with the cascaded method under wait-5. To sum up, SimulSpeech achieves higher translation accuracy than cascaded method under the same translation delay, and achieves lower translation delay with the same translation accuracy.
Figure 5: An example from the test set in  $\mathrm{En} \rightarrow \mathrm{Es}$  dataset, which demonstrates that SimulSpeech outperforms cascaded models under same delay (the delay of wait-1 for ASR plus wait-3 for NMT is equal to the delay of wait-3 for SimulSpeech). In this case, wait-1 ASR model in cascaded method does not recognize the word "classic" correctly, and results in the wrong translation in NMT model.
Model	k=1	k=3	k=5	k=7	k=9	k=inf
Cascaded	12.77	16.91	19.66	21.05	23.43	25.60
SimulSpeech	15.02	19.92	21.58	22.42	22.49	22.72
We evaluate the effectiveness of each component and show the results in Table 4. From the BLEU scores in Row 2 and Row 3, it can be seen that the translation accuracy with different wait-  $k$  can be boosted by adding auxiliary task to naive simultaneous speech to text translation model (denoted as Naive S2T).
The Effectiveness of data-level knowledge distillation We further evaluate the effectiveness of data-level knowledge distillation (Row 4 vs Row 3). The result shows that data-level knowledge distillation can achieve a large accuracy improvement.
Table 3: The comparison between two-stage cascaded method and SimulSpeech under different wait-  $k$  on  $\mathrm{{En}} \rightarrow  \mathrm{{Es}}$  dataset.
Model	k=1	k=5	k=9
Naive S2T	9.02	14.90	15.90
+Aux	12.98	19.41	20.39
+Aux+DataKD	13.77	20.98	21.52
+Aux+AttnKD	13.74	20.64	20.90
+Aux+DataKD+AttKD(SimulSpeech)	15.02	21.58	22.49
Table 4: The ablation studies on  $\mathrm{En} \rightarrow \mathrm{Es}$  dataset. The baseline model (Naive S2T) is the naive simultaneous speech to text translation model with wait- $k$  policy. We gradually add our techniques on it to evaluate their effectiveness.
The Effectiveness of attention-level knowledge distillation We further evaluate the effectiveness of attention-level knowledge distillation. We add attention-level knowledge distillation (Row 5 vs. Row 3) to the model and find that the accuracy can also be improved. As a result, we combine all the techniques together (Row 6, SimulSpeech) and obtain the best BLEU scores across different wait- $k$ , which demonstrates the effectiveness of all techniques we proposed for the training of Simul-Speech.
The Effectiveness of Speech Segmenter To evaluate the effectiveness of our segmenter, we compare the accuracy of SimulSpeech model using our segmentation method and the ground-truth segmentation, where we extract the segmentation from the ground-truth speech and corresponding transcripts using the alignment tool<sup>6</sup> and regard it as the ground-truth segmentation. As shown in Table 5, the BLEU scores of SimulSpeech using our segmentation method is close to that using ground-truth segmentation<sup>7</sup>, which demonstrates the effectiveness of our speech segmenter.
<sup>6</sup>https://github.com/lowerquality/gentle
<sup>7</sup>Note that we cannot obtain the ground-truth segmentation during inference. Therefore the accuracy gap in Table 5 is reasonable.
In a typical simultaneous translation scenario,  $k$  should be as small as possible, otherwise large delay is incurred.
Method	k=1	k=3	k=5	k=7	k=9
Ground-Truth	18.04	22.61	23.76	23.36	23.14
Our Method	15.02	19.92	21.58	22.42	22.49
Table 5: The BLEU scores of SimulSpeech on  $\mathrm{En}\rightarrow \mathrm{Es}$  using our speech segmentation method and groundtruth segmentation.
Case Analysis We further conduct case studies to demonstrate the advantages of our end-to-end translation over the previous cascaded models. As shown in Figure 5, simultaneous ASR model makes a mistake which further affects the accuracy of downstream simultaneous NMT model, while SimulSpeech is not suffered by this problem. As a result, SimulSpeech outperforms cascaded models.
Figure 6: The comparison between SimulSpeech and the cascaded method in terms of translation accuracy and delay on  $\mathrm{En} \rightarrow \mathrm{Es}$  dataset.
6.1 Speech to Text Translation
Speech to text translation has been a hot research topic in the field of artificial intelligence recently (Bérard et al., 2016; Weiss et al., 2017; Liu et al., 2019). Early works on speech to text translation rely on a two-stage method by cascaded ASR and NMT models. Bérard et al. (2016) proposed an end-to-end speech to text translation system, which does not leverage source language text during training or inference. Weiss et al. (2017) further leveraged an auxiliary ASR model with a shared encoder with the speech to text model, regarding it as a multi-task problem. Vila et al. (2018) applied Transformer (Vaswani et al., 2017b) architecture to this task and achieved good accuracy. Bansal et al. (2018) explored speech to text translation in the low-resource setting where both data and computation are limited. Sperber et al. (2019) proposed a novel attention-passing model for end-
to-end speech to text translation and achieved comparable accuracy to the cascaded models.
6.2 Simultaneous Translation
Simultaneous translation aims to translate sentences before they are finished (Fügen et al., 2007; Oda et al., 2014; Dalvi et al., 2018). Traditional speech to text simultaneous translation system usually first recognizes and segments the incoming speech stream based on an automatic speech recognition (ASR) system, and then translates it to the text in target language. And most of the previous works focus on the simultaneous machine translation part (Zheng et al., 2019): Gu et al. (2016) proposed a framework for simultaneous NMT in which an agent learns to make decisions on when to translate from the interaction with a pre-trained NMT environment. Ma et al. (2018) introduced a very simple but effective wait-  $k$  strategy for simultaneous NMT based on a prefix-to-prefix framework, which predicts the next target word conditioned on the partial source sequence the model has seen, instead of the full source sequence. The wait-  $k$  strategy will wait for the first  $k$  source words and then start to generate a target word. After that, once receiving a new source word, the decoder generates a new target word until there is no more source word, and then the translation degrades to full-sentence translation.
In this work, we developed SimulSpeech, an end-to-end simultaneous speech to text translation system that directly translates source speech into target text concurrently. SimulSpeech consists of a speech encoder, a speech segmenter, and a text decoder with wait- $k$  strategy for simultaneous translation. We further introduced several techniques including data-level and attention-level knowledge distillation to boost the accuracy of SimulSpeech. Experiments on MuST-C spoken language translation datasets demonstrate the advantages of SimulSpeech in terms of both translation accuracy and delay.
For future work, we will design more flexible policies to achieve better translation quality and lower delay in simultaneous spoken language translation. We will also investigate simultaneous translation from the speech in a source language to the speech in a target.
This work was supported in part by the National Key R&D Program of China (Grant No.2018AAA0100603), Zhejiang Natural Science Foundation (LR19F020006), National Natural Science Foundation of China (Grant No.61836002), National Natural Science Foundation of China (Grant No.U1611461), and National Natural Science Foundation of China (Grant No.61751209). This work was also partially funded by Microsoft Research Asia.
Sameer Bansal, Herman Kamper, Karen Livescu, Adam Lopez, and Sharon Goldwater. 2018. Low-resource speech-to-text translation. arXiv preprint arXiv:1803.09164.
Alexandre Bérard, Olivier Pietquin, Christophe Servan, and Laurent Besacier. 2016. Listen and translate: A proof of concept for end-to-end speech-to-text translation. arXiv preprint arXiv:1612.01744.
Kyunghyun Cho and Masha Esipova. 2016. Can neural machine translation do simultaneous translation? arXiv preprint arXiv:1606.02012.
Fahim Dalvi, Nadir Durrani, Hassan Sajjad, and Stephan Vogel. 2018. Incremental decoding and training methods for simultaneous translation in neural machine translation. arXiv preprint arXiv:1806.03661.
Mattia Antonino Di Gangi, Roldano Cattoni, Luisa Bentivogli, Matteo Negri, and Marco Turchi. 2019. MuST-C: a Multilingual Speech Translation Corpus. In NAACL-HLT, Minneapolis, MN, USA.
Christian Fügen, Alex Waibel, and Muntsin Kolss. 2007. Simultaneous translation of lectures and speeches. Machine translation, 21(4):209-252.
Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N Dauphin. 2017. Convolutional sequence to sequence learning. In ICML, pages 1243-1252. JMLR.org.
Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. 2006. Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In Proceedings of the 23rd international conference on Machine learning, pages 369-376. ACM.
Jiatao Gu, Graham Neubig, Kyunghyun Cho, and Victor OK Li. 2016. Learning to translate in real-time with neural machine translation. arXiv preprint arXiv:1610.00388.
Yoon Kim and Alexander M Rush. 2016. Sequence-level knowledge distillation. arXiv preprint arXiv:1606.07947.
Diederik Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
Yuchen Liu, Hao Xiong, Zhongjun He, Jiajun Zhang, Hua Wu, Haifeng Wang, and Chengqing Zong. 2019. End-to-end speech translation with knowledge distillation. arXiv preprint arXiv:1904.08075.
Mingbo Ma, Liang Huang, Hao Xiong, Kaibo Liu, Chuanqiang Zhang, Zhongjun He, Hairong Liu, Xing Li, and Haifeng Wang. 2018. Stacl: Simultaneous translation with integrated anticipation and controllable latency. arXiv preprint arXiv:1810.08398.
Yusuke Oda, Graham Neubig, Sakriani Sakti, Tomoki Toda, and Satoshi Nakamura. 2014. Optimizing segmentation strategies for simultaneous speech translation. In ACL, pages 551-556.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In ACL, pages 311-318.
Kanishka Rao, Hasim Sak, and Rohit Prabhavalkar. 2017. Exploring architectures, data and units for streaming end-to-end speech recognition with rnntransducer. In ASRU, pages 193-199. IEEE.
Yi Ren, Yangjun Ruan, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie-Yan Liu. 2019. *Fastspeech: Fast, robust and controllable text to speech.* arXiv preprint arXiv:1905.09263.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In ACL.
Jonathan Shen, Ruoming Pang, Ron J Weiss, Mike Schuster, Navdeep Jaitly, Zongheng Yang, Zhifeng Chen, Yu Zhang, Yuxuan Wang, Rj Skerrv-Ryan, et al. 2018. Natural tts synthesis by conditioning wavenet on mel spectrogram predictions. In ICASSP, pages 4779-4783. IEEE.
Matthias Sperber, Graham Neubig, Jan Niehues, and Alex Waibel. 2019. Attention-passing models for robust and data-efficient end-to-end speech translation. TACL, 7:313-325.
Xu Tan, Yi Ren, Di He, Tao Qin, Zhou Zhao, and TieYan Liu. 2019. Multilingual neural machine translation with knowledge distillation. arXiv preprint arXiv:1902.10461.
Ashish Vaswani, Samy Bengio, Eugene Brevdo, Francois Chollet, Aidan N. Gomez, Stephan Gouws, Llion Jones, Lukasz Kaiser, Nal Kalchbrenner, Niki Parmar, Ryan Sepassi, Noam Shazeer, and Jakob Uszkoreit. 2018. Tensor2tensor for neural machine translation. CoRR, abs/1803.07416.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017a. Attention is all you need. In NIPS 2017, pages 6000-6010.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017b. Attention is all you need. In NIPS, pages 5998-6008.
Laura Cross Vila, Carlos Escolano, José AR Fonollosa, and Marta R Costa-jussà. 2018. End-to-end speech translation with the transformer. In IberSPEECH, pages 60-63.
Ron J Weiss, Jan Chorowski, Navdeep Jaitly, Yonghui Wu, and Zhifeng Chen. 2017. Sequence-to-sequence models can directly translate foreign speech. arXiv preprint arXiv:1703.08581.
Baigong Zheng, Renjie Zheng, Mingbo Ma, and Liang Huang. 2019. Simultaneous translation with flexible policy via restricted imitation learning. arXiv preprint arXiv:1906.01135.
Chunting Zhou, Graham Neubig, and Jiatao Gu. 2019. Understanding knowledge distillation in non-autoregressive machine translation. arXiv preprint arXiv:1911.02727.