SlowGuess/ACL_2020_OCR huggingface dataset:
Chen Zhang *
zc99@zju.edu.cn
Tao Qin
taoqin@microsoft.com
In this work, we develop SimulSpeech, an end-to-end simultaneous speech to text translation system which translates speech in source language to text in target language concurrently. SimulSpeech consists of a speech encoder, a speech segmenter and a text decoder, where 1) the segmenter builds upon the encoder and leverages a connectionist temporal classification (CTC) loss to split the input streaming speech in real time, 2) the encoder-decoder attention adopts a wait- $k$  strategy for simultaneous translation. SimulSpeech is more challenging than previous cascaded systems (with simultaneous automatic speech recognition (ASR) and simultaneous neural machine translation (NMT)). We introduce two novel knowledge distillation methods to ensure the performance: 1) Attention-level knowledge distillation transfers the knowledge from the multiplication of the attention matrices of simultaneous NMT and ASR models to help the training of the attention mechanism in SimulSpeech; 2) Data-level knowledge distillation transfers the knowledge from the full-sentence NMT model and also reduces the complexity of data distribution to help on the optimization of SimulSpeech. Experiments on MuST-C English-Spanish and English-German spoken language translation datasets show that SimulSpeech achieves reasonable BLEU scores and lower delay compared to full-sentence end-to-end speech to text translation (without simultaneous translation), and better performance than the two-stage cascaded simultaneous translation model in terms of BLEU scores and translation delay.
Simultaneous speech to text translation (Fügen et al., 2007; Oda et al., 2014; Dalvi et al., 2018), which translates source-language speech into target-language text concurrently, is of great importance to the real-time understanding of spoken lectures or conversations and now widely used in many scenarios including live video streaming and international conferences. However, it is widely considered as one of the challenging tasks in machine translation domain because simultaneous speech to text translation has to understand the speech and trade off translation accuracy and delay. Conventional approaches to simultaneous speech to text translation (Fügen et al., 2007; Oda et al., 2014; Dalvi et al., 2018) divide the translation process into two stages: simultaneous automatic speech recognition (ASR) (Rao et al., 2017) and simultaneous neural machine translation (NMT) (Gu et al., 2016), which cannot be optimized jointly and result in inferior accuracy, and also incurs more translation delay due to two stages.
In this paper, we move a step further to translate the source speech to target text simultaneously, and develop SimulSpeech, an end-to-end simultaneous speech to text translation system. The SimulSpeech model consists of 1) a speech encoder where each speech frame can only see its previous frames to simulate streaming speech inputs; 2) a text decoder where the encoder-decoder attention follows the wait-  $k$  strategy (Ma et al., 2018) to decide when to listen and write on the source speech and target text respectively (see Figure 1); 3) a speech segmenter that builds upon the encoder and leverages a CTC loss to detect the word boundary, which is used to decide when to stop listening according to the
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3787-3796
Figure 1: The wait-  $k$  strategy for simultaneous speech to text translation. The model will wait for the first  $k$  source speech segments and then start to translate a target word. After that, once receiving a new source segment, the decoder generates a new target word until there is no more source word, and then the translation degrades to the full-sentence translation. The example shows the case with  $k = 2$ .
wait-  $k$  strategy.
Considering the difficulty of this task, we elaborately design two techniques to boost the performance of SimulSpeech: 1) attention-level knowledge distillation that transfers the knowledge from the multiplication of the attention matrices of simultaneous NMT and ASR model to SimulSpeech to help the training of its attention mechanism; 2) data-level knowledge distillation that transfers the knowledge from a full-sentence NMT model to SimulSpeech and also reduces the complexity of data distribution (Zhou et al., 2019) to help on the optimization of SimulSpeech model.
Compared with the cascaded pipeline that trains simultaneous ASR and NMT models separately, SimulSpeech can alleviate the error propagation problem and optimize all model parameters jointly towards the end goal, as well as reduce the delay of simultaneous translation. Experiments on MuST-C<sup>1</sup> English-Spanish and English-German spoken language translation datasets demonstrate that SimulSpeech: 1) achieves reasonable BLEU scores and lower delay compared to full-sentence end-to-end speech to text translation (without simultaneous translation), and 2) obtains better performance than the two-stage cascaded simultaneous translation model in terms of BLEU scores and translation delay.
In this section, we briefly review some basic knowledge for simultaneous speech to text translation, including speech to text translation, simultaneous translation based on wait- $k$  strategy, and CTC loss for segmentation.
Speech to Text Translation Given a set of bilingual speech-text sentence pairs  $D = \{(x,y) \in (\mathcal{X} \times \mathcal{Y})\}$ , an speech to text machine translation model learns the parameter  $\theta$  by minimizing the negative log-likelihood  $-\sum_{(x,y) \in D} \log P(y|x;\theta)$ .  $P(y|x;\theta)$  is calculated based on the chain rule  $\prod_{t=1}^{T_y} P(y_t | y_{<t}, x; \theta)$ , where  $y_{<t}$  represents the text tokens preceding position  $t$ , and  $T_y$  is the length of text sentence  $y$ . An encoder-attention-decoder framework is usually adopted to model the conditional probability  $P(y|x;\theta)$ , where the encoder maps the input audio to a set of hidden representations  $h$  and the decoder generates each target token  $y_t$  using the previously generated tokens  $y_{<t}$  as well as the speech representations  $h$ . Previous works (Bérard et al., 2016; Weiss et al., 2017; Liu et al., 2019) on speech to text translation focus on the full-sentence translation where the full source speech can be seen when predicting each target token.
Simultaneous Translation Based on Wait-  $k$  Simultaneous translation aims to translate sentences before they are finished according to certain strategies. We use wait-k strategy (Ma et al., 2018) in this work: given a set of speech and text pairs  $D = \{(x,y)\in (\mathcal{X}\times \mathcal{Y})\}$ , the model with the wait-  $k$  strategy learns the parameter  $\theta$  by minimizing the negative log-likelihood loss  $-\sum_{(x,y)\in D}\log P(y|x;k;\theta)$ , where  $k$  corresponds to the wait-  $k$  strategy.  $P(y|x;k;\theta)$  is calculated based on the chain rule
$$
P (y | x; k; \theta) = \prod_ {t = 1} ^ {T _ {y}} P \left(y _ {t} \mid y _ {<   t}, x _ {<   t + k}; \theta\right), \tag {1}
$$
where  $y_{< t}$  represents the tokens preceding position  $t$  and  $T_{y}$  is the length of target sentence  $y$ ,  $x_{< t + k}$  represents the speech segments preceding position  $t + k$ . The wait- $k$  strategy ensures that the model can see  $t + k - 1$  source segments when generating the target token  $y_{t}$ , while can see the whole sentence if there is no more source segments.
CTC for Alignment and Segmentation The connectionist temporal classification (CTC)
1 https://ict.fbk.eu/must-c/
(a) SimulSpeech model structure.
(b) The training pipeline for SimulSpeech model.
Figure 2: (a) The model structure of SimulSpeech. (b) The training pipeline for SimulSpeech model. The SimulSpeech model is shown in purple box, and the auxiliary training techniques are in other boxes.
loss (Graves et al., 2006) is widely used for alignment and segmentation, which maps the frame-level classification outputs of a speech sequence to a text sequence (with a different length from the speech sequence). For a text sequence  $y$ , CTC introduces a set of intermediate representation paths  $\phi(y)$  called CTC paths, which has a many-to-one mapping to  $y$  since multiple CTC paths can correspond to the same text sequence. For example, both the frame-level classification outputs (CTC paths) "HHELOLOO" and "HHEELLO" are mapped to text sequence "HELLO", where  $\emptyset$  is the blank symbol. The likelihood of  $y$  can thus be evaluated as a sum of the probabilities of its CTC paths:
$$
P (y | x) = \sum_ {z \in \phi (y)} P (z | x), \tag {2}
$$
where  $x$  is the utterance consisting of speech frames and  $z$  is one of the CTC path.
3 The SimulSpeech Model
Similar to many sequence to sequence generation tasks, SimulSpeech adopts the encoder-decoder framework. As shown in Figure 2a, both the encoder and decoder follow the basic network structure of Transformer (Vaswani et al., 2017a) for neural machine translation. SimulSpeech is different from Transformer in several aspects:
- To handle speech inputs, we employ a speech pre-net (Shen et al., 2018) to extract speech
features, which consists of multiple convolutional layers with the same hidden size as Transformer.
- To enable simultaneous translation, we design different attention mechanisms for the encoder and decoder. The encoder adopts masked self-attention, which masks the future frames of a speech frame when encoding it and ensures that each speech frame can only see its previous frames to simulate the real-time streaming inputs. The decoder adopts the wait- $k$  strategy (Ma et al., 2018), as shown in Equation 1, which guarantees that each target token can only see the source segments following the wait- $k$  strategy.
- As the wait- $k$  strategy requires source speech to be discrete segments, we introduce a speech segmenter to split a speech sequence into discrete segments, each representing a word or phrase. The segmenter takes the outputs of the speech encoder as inputs, passes through multiple non-linear dense layers and then a softmax linear layer to predict the character in frame level. When a word boundary token (the space character in our case) is predicted by the segmenter, SimulSpeech knows a word is ended. Multiple consecutive word boundary tokens are merged into one boundary.
Figure 3: Details of attention-level knowledge distillation.
4 Training of SimulSpeech
The training of the SimulSpeech model is more difficult than that of an NMT model or an ASR model, since SimulSpeech involves multiple modalities (i.e., speech and text) and multiple languages. In this section, we discuss how to train the SimulSpeech model. As shown in Figure 2b, we introduce the CTC loss for the training of the speech segmenter, and attention-level and data-level knowledge distillation for the training of the overall SimulSpeech model. In SimulSpeech training, the training data are provided in the format of (source speech, source text, target text) tuples.
4.1 Training Segmenter with CTC Loss
In SimulSpeech, the speech segmenter is used to detect word boundaries, and detected boundaries are used to determine when to stop listening and switch to translation, which is critical for the performance of simultaneous translation. As it is hard to find frame-level label to guide the output of the softmax linear layer in speech segmenter, we leverage connectionist temporal classification (CTC) loss to train the speech segmenter. According to Equation 2, the CTC loss is formulated as
$$
\mathcal {L} _ {\mathrm {c t c}} = - \sum_ {(x, y) \in (\mathcal {X} \times \mathcal {Y} ^ {\mathrm {s r c}})} \sum_ {z \in \phi (y)} P (z | x), \tag {3}
$$
where  $(\mathcal{X}\times \mathcal{Y}^{\mathrm{src}})$  denotes the set of source speech and source text sequence pairs, and  $\phi (y)$  denotes the set of CTC paths for  $y$
During inference, we simply use the best path decoding (Graves et al., 2006) to decide the word boundary without seeing subsequent speech frames, which is consistent with the masked self-attention in speech encoder, i.e., the output of segmenter for position  $i$  depends only on the inputs at positions preceding  $i$ .
4.2 Attention-Level Knowledge Distillation
To better train the SimulSpeech model, we propose a novel attention-level knowledge distillation that is specially designed for speech to text translation,
which transfers the knowledge from the multiplication of attention weights matrices of simultaneous ASR and NMT models, into the attention of the SimulSpeech model. In order to obtain the attention weights of simultaneous ASR and NMT, we add auxiliary simultaneous ASR and NMT tasks which share the same encoder or decoder with SimulSpeech model respectively, as shown in Figure 2b. The two auxiliary tasks both leverage a wait-  $k$  strategy similar to that used in SimulSpeech model.
Denote the sequence length of the source speech, source text and target text as  $S_{\mathrm{src}}$ ,  $T_{\mathrm{src}}$  and  $T_{\mathrm{tgt}}$  respectively. Denote the attention weights of simultaneous ASR and NMT as  $A^{T_{\mathrm{src}} \times S_{\mathrm{src}}}$  and  $A^{T_{\mathrm{tgt}} \times T_{\mathrm{src}}}$  respectively. Ideally, the attention weights of Simul-Speech  $A^{T_{\mathrm{tgt}} \times S_{\mathrm{src}}}$  should satisfy
$$
A ^ {T _ {\mathrm {t g t}} \times S _ {\mathrm {s r c}}} = A ^ {T _ {\mathrm {t g t}} \times T _ {\mathrm {s r c}}} \times A ^ {T _ {\mathrm {s r c}} \times S _ {\mathrm {s r c}}}. \tag {4}
$$
However, the attention weights are difficult to learn, and the attention weights of SimulSpeech model are more difficult to learn than that of the simultaneous ASR and NMT models since SimulSpeech is much more challenging. Therefore, we propose to distill the knowledge from the multiplication of the attention weights of the simultaneous ASR and NMT, as shown in Figure 2b and Figure 3. We first multiply the attention matrix of simultaneous NMT by that of simultaneous ASR, and then binarize the attention matrix with a threshold. We then match the attention weights that is predicted by the SimulSpeech model to the binarized attention matrix, with the loss function
$$
\mathcal {L} _ {\text {a t t} \mathrm {k d}} = - \mathcal {B} \left(A ^ {T _ {\mathrm {t g t}} \times T _ {\mathrm {s r c}}} \times A ^ {T _ {\mathrm {s r c}} \times S _ {\mathrm {s r c}}}\right) \times A ^ {T _ {\mathrm {t g t}} \times S _ {\mathrm {s r c}}}, \tag {5}
$$
where  $\mathcal{B}$  is the binarization operation which set the element of the matrix to 1 if above the threshold of 0.05, and otherwise to 0.
4.3 Data-Level Knowledge Distillation
Data-level knowledge distillation is widely used to help model training in various tasks and situations (Kim and Rush, 2016; Tan et al., 2019) and
can boost the performance of a student model. In this work, we leverage knowledge distillation to transfer the knowledge from a full-sentence NMT teacher model to a SimulSpeech model. We train a full-sentence NMT teacher model first and then generate target text  $y'$  given source text  $y$  that is paired with source speech  $x$ . Finally, we train the student SimulSpeech with the generated target text  $y'$  which is paired with the source speech  $x$ . The loss function is formulated as
$$
\mathcal {L} _ {\text {d a t a . k d}} = - \sum_ {(x, y ^ {\prime}) \in (\mathcal {X} \times \mathcal {Y} ^ {\mathrm {t g t} ^ {\prime}})} \log P (y ^ {\prime} | x), \tag {6}
$$
where  $(\mathcal{X}\times \mathcal{Y}^{\mathrm{tgt}^{\prime}})$  denotes the set of speech-text sequence pairs where text is generated by the NMT teacher model.
The total loss function to train SimulSpeech model is
$$
\mathcal {L} = \lambda_ {1} \mathcal {L} _ {\mathrm {c t c}} + \lambda_ {2} \mathcal {L} _ {\text {a t t} \cdot \mathrm {k d}} + \lambda_ {3} \mathcal {L} _ {\text {d a t a} \cdot \mathrm {k d}}, \tag {7}
$$
where  $\lambda_1, \lambda_2, \lambda_3$  are hyperparameters to trade off the three losses.
In this section, we evaluate SimulSpeech on MuST-C corpus (Di Gangi et al., 2019). First we describe experimental settings and details, then we show the experiment results, and further conduct some analyses on our model.
Datasets We use the MuST-C English-Spanish  $(\mathrm{En} \rightarrow \mathrm{Es})$  and English-German  $(\mathrm{En} \rightarrow \mathrm{De})$  speech translation corpus in our experiments. Both two datasets contain audio clips in source language, and the corresponding source-language transcripts and target-language translated text. The official data statistics and splits for train/dev/test set are shown in Table 1. For the speech data, we transform the raw audio into mel-spectrograms following Shen et al. (2018) with  $50~\mathrm{ms}$  frame size and  $12.5~\mathrm{ms}$  hop size. To simplify the model training, we remove some non-verbal annotation in the text, such as “(Laughing)”, “(Music)”. All the sentences are first tokenized with moses tokenizer² and then segmented into subword symbols using Byte Pair Encoding (BPE) (Sennrich et al., 2016), except for the label to train the speech segmenter, where we
use character sequence of source text. We learn the BPE merge operations across source and target languages. We use the speech segmenter proposed in Section 3 to split the speech mel-spectrograms into segments, where each segment is regarded as discrete tokens and represents a word or short phrase.
Task	Train	Dev	Test
En→Es	229703 (496h)	1316 (2.5h)	2502 (4h)
En→De	265625 (400h)	1423 (2.5h)	2641 (4h)
Table 1: The number of sentences and the duration of audio in MuST-C dataset.
Model Configuration We use the Transformer (Vaswani et al., 2017b) as the basic SimulSpeech model structure since it achieves state-of-the-art accuracy and becomes a popular choice for recent NMT research. The model hidden size, number of heads, number of encoder and decoder-layers are set to 384, 4, 6 and 4 respectively. Considering that the adjacent hidden states are closely related in speech task, we replace the feed-forward network in Transformer with a 2-layer 1D convolutional network (Gehring et al., 2017) with ReLU activation. Left padding is used in the 1D convolutional network in the target side (Ren et al., 2019) to avoid the output token seeing its subsequent tokens in the training stage. The kernel size and filter size of 1D convolution are set to 1536 and 9 respectively. The pre-net (bottom left in Figure 2a) is a 3-layer convolutional network with left padding, whose output dimension is same as the hidden size of the transformer encoder. The decoder of the auxiliary ASR model and the encoder of the auxiliary NMT model, as well as the encoder and decoder of the NMT teacher model share the same model structures described above.
Training and Inference SimulSpeech is trained on 2 NVIDIA Tesla V100 GPUs, with totally batch size of 64 sentences. We use the Adam optimizer with the default parameters (Kingma and Ba, 2014) and learning rate schedule in Vaswani et al. (2017a). We train the SimulSpeech with auxiliary simultaneous ASR and NMT tasks by default. We set the  $\lambda_{1}$ ,  $\lambda_{2}$ ,  $\lambda_{3}$  in Equation 7 as 1.0, 0.1, 1.0 respectively, according to the validation performance. SimulSpeech is trained and tested with the same  $k$  unless otherwise stated. The translation quality is evaluated by tokenized case sensitive BLEU (Papineni
$^{2}$ https://github.com/moses-smt/mosesdecoder/blob/master/scripts/tokenizer/tokenizer.perl
et al., 2002) with the perl scripts<sup>3</sup>. Our code is based on tensor2tensor (Vaswani et al., 2018)<sup>4</sup>.
The Metric of Translation Delay Many previous works focus on proposing the metrics of translation delay for simultaneous text to text translation, such as average proportion (AP) (Cho and Esipova, 2016) and average latency (AL) (Ma et al., 2018). The former calculates the mean absolute delay cost by each target token, while the latter measures the degree of out of sync with the speaker. In this work, we extend the AP and AL metric that are originally calculated on word sequence to speech sequence for simultaneous speech to text translation task. Our extended AP is defined as follows:
$$
A P (x, y) = \frac {1}{| x | ^ {\text {t i m e}} | y |} \sum_ {i = 1} ^ {| y |} t (i), \tag {8}
$$
where  $x$  and  $y$  are the source speech and target text,  $|x|^{\mathrm{time}}$  is the total time duration of source speech,  $|y|$  is the length of target text,  $t(i)$  is real-time delay in terms of source speech when generating the  $i$ -th word in target sequence, i.e., the duration of source speech listened by the model before writing the  $i$ -th target token. Our extended AL is defined as follows:
$$
A L (x, y) = \frac {1}{\tau (| x | ^ {\mathrm {s e g}})} \sum_ {i = 1} ^ {\tau (| x | ^ {\mathrm {s e g}})} g (i) - \frac {i - 1}{r}, \tag {9}
$$
where  $|x|^{\mathrm{seg}}$  is length of speech segments,  $g(i)$  is the delay at step  $i$ , i.e., the number of source segments listened by the model before writing the  $i$ -th target token.  $\tau(|x|^{\mathrm{seg}})$  denotes the earliest timestep where our model has consumed the entire source sequence:
$$
\tau (| x | ^ {\operatorname {s e g}}) = \arg \min  _ {t} (g (t) = | x | ^ {\operatorname {s e g}}), \tag {10}
$$
and  $r = |y| / |x|^{\mathrm{seg}}$  is the length ratio between target and source sequence.
Translation Accuracy First, we evaluate the performance of SimulSpeech model under different  $k$ . The BLEU scores of En-Es and En-De are shown in Table 2. We can see that the performance of our model does not drop a lot when  $k$  is small, compared to the full-sentence translation (training with  $k = \inf$ ).
k	1	3	5	7	9	inf
En-Es	15.02	19.92	21.58	22.42	22.49	22.72
En-Es (FS)	3.25	7.18	10.52	13.33	15.32	22.72
En-De	10.73	15.52	16.90	17.46	17.87	18.29
En-De (FS)	2.58	6.89	9.65	11.70	13.15	18.29
Table 2: The BLEU scores of SimulSpeech on the test set of the MuST-C  $\mathrm{En}\rightarrow \mathrm{Es}$  and  $\mathrm{En}\rightarrow \mathrm{De}$  dataset.  $FS$  denotes training with  $k = \inf$
Translation Delay We plot the translation quality (in terms of BLEU score) against delay metrics (AP and AL) of our SimulSpeech model and test-time wait- $k$  model (trained with full-sentence translation but only test with wait- $k$ , denoted as "train-full test-k") in Figure 4a and 4b. We can see that the BLEU scores increase as  $k$  increases, with the sacrifice of translation delay. The accuracy of SimulSpeech model is always better than the test-time wait- $k$ , which demonstrates the effectiveness of the SimulSpeech.
(a) The translation quality against the latency in terms of AP.
(b) The translation quality against the latency in terms of AL.
Figure 4: The translation quality against the latency metrics (AP and AL) on  $\mathrm{En}\rightarrow \mathrm{Es}$  dataset.
Comparison with Cascaded Models Finally, we implement the cascaded simultaneous speech to text translation pipeline and compare the accuracy of SimulSpeech with it under the same translation
$^{3}$ https://github.com/moses-smt/mosesdecoder/blob/master/scripts/generic/multi-bleu.perl  
 $^{4}$ https://github.com/tensorflow/tensor2tensor
En (source)Es (target)	1 the la	2 first primera	3 onaquí	4 here es	5 is la	6 the clásica	7 classic manzana.	8 apple.	9	10	11	12
ASR (wait-1)ASR (wait-1) + NMT (wait-3)	the	first	on pero	here la	is primera	the vez	classes	sake una	apple. manzana	motivo	de	clase.
SimulSpeech (wait-3)	la	primera es	una	manzana	clásica.