SlowGuess/NAACL_2021_OCR huggingface dataset:
Hirofumi Inaguma\* Tatsuya Kawahara\* Shinji Watanabe\*
$\spadesuit$  Kyoto University, Japan  $\spadesuit$  Johns Hopkins University, USA
inaguma@sap.ist.i.kyoto-u.ac.jp
A conventional approach to improving the performance of end-to-end speech translation (E2E-ST) models is to leverage the source transcription via pre-training and joint training with automatic speech recognition (ASR) and neural machine translation (NMT) tasks. However, since the input modalities are different, it is difficult to leverage source language text successfully. In this work, we focus on sequence-level knowledge distillation (SeqKD) from external text-based NMT models. To leverage the full potential of the source language information, we propose backward SeqKD, SeqKD from a target-to-source backward NMT model. To this end, we train a bilingual E2E-ST model to predict paraphrased transcriptions as an auxiliary task with a single decoder. The paraphrases are generated from the translations in bitext via back-translation. We further propose bidirectional SeqKD in which SeqKD from both forward and backward NMT models is combined. Experimental evaluations on both autoregressive and non-autoregressive models show that SeqKD in each direction consistently improves the translation performance, and the effectiveness is complementary regardless of the model capacity.
End-to-end speech translation (E2E-ST) (Bérard et al., 2016), which aims to convert source speech to text in another language directly, is an active research area. Because direct ST is a more difficult task than automatic speech recognition (ASR) and machine translation (MT), various techniques have been proposed to ease the training process by using source transcription. Examples include pretraining (Bérard et al., 2018; Wang et al., 2020c; Bansal et al., 2019; Wang et al., 2020d), multi-task learning (Weiss et al., 2017; Bérard et al., 2018; Bahar et al., 2019), knowledge distillation (Liu et al., 2019), meta-learning (Indurthi et al., 2020), two-pass decoding (Anastasopoulos and Chiang, 2018;
Sperber et al., 2019), and interactive decoding (Liu et al., 2020; Le et al., 2020). However, as input modalities between ST and MT tasks are different, an auxiliary MT task is not always helpful, especially when additional bitext is not available (Bahar et al., 2019). Moreover, because monotonic speech-to-transcription alignments encourage the ASR task to see surface-level local information, an auxiliary ASR task helps the E2E-ST model to extract acoustic representations, not semantic ones, from speech.
Sequence-level knowledge distillation (SeqKD) (Kim and Rush, 2016) is another approach to transferring knowledge from one model to another. Recent studies have shown that SeqKD has the effect of reducing the complexity of training data and thus eases the training of student models, e.g., non-autoregressive (NAR) models (Gu et al., 2018; Zhou et al., 2019a; Ren et al., 2020).
Paraphrasing, which represents text in a different form but with the same meaning, can also be regarded as SeqKD when using neural paraphrasing via back-translation (Mallinson et al., 2017; Wieting et al., 2017; Federmann et al., 2019). It has been studied to improve the reference diversity for MT system evaluations (Thompson and Post, 2020; Bawden et al., 2020a,b) and the performance of low-resource neural MT (NMT) models (Zhou et al., 2019b; Khayrallah et al., 2020).
In this work, due to its simplicity and effectiveness, we focus on SeqKD from text-based NMT models to improve the performance of a bilingual E2E-ST model. In order to fully leverage source language information, we propose backward SeqKD, which targets paraphrased source transcriptions generated from a target-to-source backward NMT model as an auxiliary task. Then, a single ST decoder is trained to predict both source and target language text as in a multilingual setting (Inaguma et al., 2019). This way, the decoder is biased to capture semantic representations from speech, un
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1872-1881 June 6-11, 2021. ©2021 Association for Computational Linguistics
like joint training with an auxiliary ASR task. We also propose bidirectional SeqKD, which combines SeqKD from two NMT models in both language directions. Therefore, the E2E-ST models can fully exploit the knowledge embedded in both forward and backward NMT models.
Experimental evaluations demonstrate that SeqKD from each direction consistently improves the translation performance of both autoregressive and non-autoregressive E2E-ST models. We also confirm that bidirectional SeqKD outperforms unidirectional SeqKD and that the effectiveness is maintained in large models.
In this section, we propose bidirectional SeqKD from both forward and backward NMT models that leverages machine-generated source paraphrases as another target in addition to the distilled translation to enhance the training of a bilingual E2EST model. Let  $X$  denote input speech features in a source language and  $Y^{\mathrm{s}}$  and  $Y^{\mathrm{t}}$  denote the corresponding gold transcription and translation, respectively. Let  $\mathcal{D}_{\mathrm{st}} = \{(X_i,Y_i^{\mathrm{s}},Y_i^{\mathrm{t}})\}_{i = 1}^{I}$  be an ST dataset including  $I$  samples, and  $\mathcal{D}_{\mathrm{asr}} = \{(X_i,Y_i^{\mathrm{s}})\}_{i = 1}^{I}$  and  $\mathcal{D}_{\mathrm{mt}} = \{(Y_i^{\mathrm{s}},Y_i^{\mathrm{t}})\}_{i = 1}^{I}$  denote the corresponding ASR and MT datasets, respectively. We drop the subscript  $i$  when it is obvious.
2.1 Sequence-level knowledge distillation
We first train a text-based source-to-target forward NMT model  $\mathcal{M}_{\mathrm{fwd}}$  with  $\mathcal{D}_{\mathrm{mt}}$ . Then, we perform beam search decoding with  $\mathcal{M}_{\mathrm{fwd}}$  on  $\mathcal{D}_{\mathrm{st}}$  to create a new dataset  $\mathcal{D}_{\mathrm{st}}^{\mathrm{fwd}} = \{(X_i,Y_i^{\mathrm{s}},\hat{Y}_i^{\mathrm{t}})\}_{i=1}^I$ , where  $\hat{Y}_i^{\mathrm{t}}$  is a distilled translation.  $\mathcal{D}_{\mathrm{st}}^{\mathrm{fwd}}$  is used to train the E2E-ST models, referred to as forward SeqKD (or fwd SeqKD).
2.2 Paraphrase generation
To exploit semantic information in the source language, we leverage machine-generated paraphrases of source transcriptions. We train a text-based target-to-source backward NMT model  $\mathcal{M}_{\mathrm{bwd}}$  with  $\mathcal{D}_{\mathrm{mt}}$  and then generate a new dataset  $\mathcal{D}_{\mathrm{st}}^{\mathrm{bwd}} = \{(X_i,\hat{Y}_i^{\mathrm{s}},Y_i^{\mathrm{t}})\}_{i = 1}^I$ , where  $\hat{Y}_i^{\mathrm{s}}$  is a paraphrase of  $Y_{i}^{\mathrm{s}}$ . We use  $\mathcal{D}_{\mathrm{st}}^{\mathrm{bwd}}$  for training the E2E-ST models. As neural paraphrasing can be regarded as SeqKD from  $\mathcal{M}_{\mathrm{bwd}}$ , we referred to it as backward SeqKD
(or bwd SeqKD). In this work, we do not use large paraphrase datasets (Wieting and Gimpel, 2018; Hu et al., 2019) because their availability depends on languages and domains. Moreover, neural paraphrasing is applicable to any source languages that lack a sufficient amount of paired paraphrase data.
We also propose combining forward SeqKD with backward SeqKD, referred to as bidirectional SeqKD (or bidir SeqKD), and construct a new dataset  $\mathcal{D}_{\mathrm{st}}^{\mathrm{bidir}} = \{(X_i,\hat{Y}_i^{\mathrm{s}},\hat{Y}_i^{\mathrm{t}})\}_{i = 1}^I$ . When using two references per utterance (2ref training) (Gordon and Duh, 2019), we concatenate  $\mathcal{D}_{\mathrm{st}}^{\mathrm{fwd}}$  and  $\mathcal{D}_{\mathrm{st}}^{\mathrm{bwd}}$ , and the suitable combination is analyzed in Section 4.3. This way, we can distill the knowledge of both  $\mathcal{M}_{\mathrm{fwd}}$  and  $\mathcal{M}_{\mathrm{bwd}}$  to a single E2E-ST model.
We train an E2E-ST model with a direct ST objective  $\mathcal{L}_{\mathrm{st}}(Y^{\mathrm{t}}$  or  $\hat{Y}^{\mathrm{t}}|X)$  and an auxiliary speech-to-source text objective  $\mathcal{L}_{\mathrm{src}}(Y^{\mathrm{s}}$  or  $\hat{Y}^{\mathrm{s}}|X)$ . We refer to joint training with  $\mathcal{L}_{\mathrm{src}}(Y^{\mathrm{s}}|X)$  as joint ASR and with  $\mathcal{L}_{\mathrm{src}}(\hat{Y}^{\mathrm{s}}|X)$  as backward SeqKD. Both losses are calculated from the same ST decoder. To bias the model to generate the desired target language, we add language embedding to token embedding at every token position in the decoder (Conneau and Lample, 2019).<sup>3</sup> We then apply bidirectional SeqKD to both autoregressive (AR) and non-autoregressive (NAR) E2E-ST models.
Autoregressive E2E-ST model
We use the speech Transformer architecture in (Karita et al., 2019) with an additional language embedding. The total training objective is formulated with a hyperparameter  $\lambda_{\mathrm{src}}(\geq 0)$  as
$$
\mathcal {L} _ {\text {t o t a l}} = \mathcal {L} _ {\text {s t}} + \lambda_ {\text {s r c}} \mathcal {L} _ {\text {s r c}}, \tag {1}
$$
where both  $\mathcal{L}_{\mathrm{st}}$  and  $\mathcal{L}_{\mathrm{src}}$  are defined as cross-entropy losses. The entire encoder-decoder parameters are shared in both tasks.
Non-autoregressive E2E-ST model
We adopt Orthors (Inaguma et al., 2021), in which a decoder based on a conditional masked language model (CMLM) (Ghazvininejad et al., 2019) is jointly trained with an additional AR decoder
<sup>1</sup>We focus on a complete triplet of  $(X, Y^{\mathrm{s}}, Y^{\mathrm{t}})$  only. However, the proposed method can easily be extended to a semi-supervised setting featuring additional ASR and MT pair data.
2All NMT models are autoregressive in this paper.
3We found this was more effective than replacing the start-of-sentence symbol with a language ID (Inaguma et al., 2019; Wang et al., 2020b; Le et al., 2020) as done in previous multilingual E2E-ST studies.
Language direction	BLEU (↑)	TER (↓)
De → En	43.49	38.60
Fr → En	48.55	34.30
on the shared speech encoder. The training of the NAR decoder is further enhanced with semiautoregressive training (SMART) (Ghazvininejad et al., 2020).  $\mathcal{L}_{\mathrm{st}}$  in Eq. (1) is modified as
$$
\mathcal {L} _ {\mathrm {s t}} = \mathcal {L} _ {\mathrm {c m l m}} + \lambda_ {\mathrm {a r}} \mathcal {L} _ {\mathrm {a r}} + \lambda_ {\mathrm {l p}} \mathcal {L} _ {\mathrm {l p}}, \tag {2}
$$
where  $\mathcal{L}_{\mathrm{cmlm}}$ ,  $\mathcal{L}_{\mathrm{ar}}$ , and  $\mathcal{L}_{\mathrm{lp}}$  are losses in NAR E2EST, AR E2E-ST, and length prediction tasks, respectively.  $\lambda_{*}$  is the corresponding tunable loss weight. During inference, the mask-predict algorithm is used for  $T$  iterations with a length beam width of  $l$  (Ghazvininejad et al., 2019). The best candidate at the last iteration is selected from the NAR decoder based on scores from the AR decoder (Inaguma et al., 2021). Note that we apply  $\mathcal{L}_{\mathrm{src}}$  to the NAR decoder only.
3 Experimental setting
Data We used Must-C En-De (408 hours) and En-Fr (492 hours) datasets (Di Gangi et al., 2019). Both language pairs consist of a triplet of  $(X, Y^{\mathrm{s}}, Y^{\mathrm{t}})$ . We performed the same data preprocessing as (Inaguma et al., 2020) (see details in Appendix A.1). We report case-sensitive detokenized BLEU scores (Papineni et al., 2002) on the tst-COMMON set with the multi-bleu-detok.perl script in Moses (Koehn et al., 2007).
Model configuration We used the Transformer (Vaswani et al., 2017) architecture having 12 encoder layers following two CNN blocks and six decoder layers for the ASR and E2E-ST tasks. For the MT models, we used six encoder layers. We built our models with the ESPnet-ST toolkit (Inaguma et al., 2020). See details in Appendix A.2.
Training We always initialized the encoder parameters of the E2E-ST model by those of the corresponding pre-trained ASR model (Bérard et al., 2018). We follow the same optimization strategies as in (Inaguma et al., 2021, 2020). When using joint ASR or backward SeqKD, we set  $\lambda_{\mathrm{src}}$  to 0.3. More details are described in Appendix A.3 and A.4.
Table 1: Quality of paraphrases in the training set
ID	Model	BLEU (Δ) (↑)
En-De	En-Fr
ESPnet-ST†	22.91	32.69
-	Fairseq-S2T‡	22.7	32.9
+ Multilingual°	24.5	34.9
A1	Baseline	22.77	33.51
A2	+ MT pre-training	23.12	(+0.35)	33.84	(+0.33)
A3	+ Joint ASR	22.97	(+0.20)	33.37	(-0.14)
A4	+ Bwd SeqKD	23.11	(+0.34)	33.78	(+0.23)
B1	A1 + Fwd SeqKD	24.42	(+1.65)	34.66	(+1.15)
B2	+ MT pre-training	24.68	(+1.91)	34.57	(+1.06)
B3	+ Joint ASR	24.67	(+1.90)	34.68	(+1.17)
B4	+ Original (2ref)	24.83	(+2.06)	34.92	(+1.41)
C1	A1 + Bidir SeqKD	24.83	(+2.06)	34.78	(+1.27)
C2	+ Original (2ref)	25.28	(+2.51)	35.29	(+1.78)
Table 2: BLEU scores of AR models on Must-C tst-COMMON set.  ${}^{ \dagger  }$  (Inaguma et al.,2020),  ${}^{ \ddagger  }$  (Wang et al.,2020a).  ${}^{ \circ  }$  Large model trained with eight language pairs (Wang et al.,2020a).
Inference For the AR models, we used a beam width of 4. For the NAR models, we set  $T = \{4,10\}$  and  $l = 9$  as in (Inaguma et al., 2021).
We first report the paraphrasing quality, which is shown in Table 1. As confirmed by the BLEU and translation edit rate (TER) scores (Snover et al., 2006), the paraphrased source text was not just a simple copy of the transcription (see examples in Appendix A.5).
Autoregressive models The results are shown in Table 2. Pre-training the ST decoder with the forward MT decoder (A2) improved the baseline performance (A1). Joint ASR showed a marginal improvement on En-De but a degraded performance on En-Fr (A3). We attribute this to the fact that the ASR task was more trivial than the ST task and biased the shared decoder to capture surface-level textual information. In contrast, backward SeqKD showed small but consistent improvements in both language directions (A4), and it was as effective as MT pre-training. As the encoder was already pre-trained with the ASR model, paraphrases had an additional positive effect on the BLEU improvement.
Forward SeqKD significantly improved the performance, as previously reported in (Inaguma et al., 2021). However, the gains by MT pre-training and joint ASR were diminished. Forward SeqKD was more effective than backward SeqKD solely (A4 vs. B1). However, backward SeqKD was still
Model	T	BLEU (↑)
En-De	En-Fr
Fwd SeqKD	21.93	30.46
+ Joint ASR	4	22.13	30.80
Bidir SeqKD	22.22	31.21
(Inaguma et al., 2021)	22.88	32.20
Fwd SeqKD (ours)	10	22.96	32.42
+ Joint ASR	23.31	32.41
Bidir SeqKD	23.41	32.64
beneficial on top of forward SeqKD (C1, i.e., bidirectional SeqKD) while joint ASR was less so (B3). We also augmented the target translations by concatenating  $\mathcal{D}_{\mathrm{st}}$  and  $\mathcal{D}_{\mathrm{st}}^{\mathrm{fwd}}$  (2ref training), which further improved forward SeqKD (B4). Nevertheless, a combination of 2ref training and backward SeqKD (i.e., bidirectional SeqKD with  $\mathcal{D}_{\mathrm{st}}^{\mathrm{fwd}} \cup \mathcal{D}_{\mathrm{st}}^{\mathrm{bwd}}$  ) had a complementary effect and showed the best result (C2). It even outperformed larger multilingual models (Wang et al., 2020a) without using additional data in other language pairs.
Non-autoregressive models The results are presented in Table 3. Following the standard practice in NAR models (Gu et al., 2018), we always used forward SeqKD. We did not use  $2ref$  training for the NAR models because it increases the multimodality. Joint ASR improved the performance on all NAR models, except for En-Fr with the number of iterations  $T = 10$ . However, bidirectional SeqKD with  $\mathcal{D}_{\mathrm{st}}^{\mathrm{bidir}}$  further improved the performance consistently regardless of  $T$ . Since NAR models assume conditional independence for every token, they prefer monotonic input-output alignments with lower alignment complexity in theory. However, paraphrasing collapses the monotonicity of the ASR task and increases the alignment complexity, making the auxiliary speech-to-source text task non-trivial. Nevertheless, BLEU scores were improved by adding backward SeqKD. This was probably because the complexity of transcriptions in the training data was reduced at the cost of the alignment complexity, which was more effective for the NAR models.
We analyze the performance of bidirectional SeqKD through a lens of complexity in the training data following (Zhou et al., 2019a). We aligned words in every source and target sentence pair with
Table 3: BLEU scores of NAR models on Must-C tst-COMMON set. All methods used forward SeqKD.
Condition	Entropy(↑ more complex)
En-De	En-Fr
C(Dst) (Real)	0.70	0.65
C(Dfwdst) (Fwd SeqKD)	0.52	0.47
C(Dbwdst) (Bwd SeqKD)	0.54	0.47
C(Dbidirst) (Bidir SeqKD)	0.63	0.61
C(Dst) (Real)	0.40	0.54
C(Dfwdst) (Fwd SeqKD)	0.28	0.36
C(Dbwdst) (Bwd SeqKD)	0.25	0.31
C(Dbidirst) (Bidir SeqKD)	0.37	0.49
Table 4: Corpus-level conditional entropy
Condition	Faithfulness (↓ more faithful)
En-De	En-Fr
F(Dfwdst) (Fwd SeqKD)	12.61	11.65
F(Dbwdst) (Bwd SeqKD)	9.31	8.67
F(Dbidirst) (Bidir SeqKD)	11.42	10.72
F(Dfwdst) (Fwd SeqKD)	9.58	8.48
F(Dbwdst) (Bwd SeqKD)	12.97	10.70
F(Dbidirst) (Bidir SeqKD)	11.23	9.98
Table 5: Faithfulness to training data distribution
fast_align $^4$  (Dyer et al., 2013). Then, we calculated corpus-level conditional entropy  $\mathcal{C}(\mathcal{D})$  and faithfulness  $\mathcal{F}(\mathcal{D})$  for both forward  $(\overleftarrow{\mathcal{D}})$  and backward  $(\overleftarrow{\mathcal{D}})$  language directions to evaluate the multimodality. In short, conditional entropy measures uncertainty of translation, and faithfulness is defined as Kullback-Leibler divergence and measures how close the distilled data distribution is to the real data distribution. See the mathematical definition in Appendix A.6.
The results of entropy and faithfulness are shown in Tables 4 and 5, respectively. Consistent with (Zhou et al., 2019a), the entropy of target translations was reduced by forward SeqKD, indicating target translations were converted into a more deterministic and simplified form. Interestingly, the entropy of the original translations was also reduced by backward SeqKD. In other words, backward SeqKD modified transcriptions so that the target translations can be predicted easier. This would help E2E-ST models learn relationships between source and target languages from speech because E2E-ST models are not conditioned on text in another language explicitly. Therefore, we presume that the encoder representations were enhanced by back
4https://github.com/clab/fast_align
Training data	Target1	Target2	BLEU (↑)
En-De	En-Fr
Dst ∪ Dfwd st (B4 + Joint ASR)	(YS, Yt)	(YS, Yt)	25.00	35.05
Dst ∪ Dbidir st	(YS, Yt)	(YS, Yt)	25.21	35.17
Dbwd ∪ Dbidir st	(YS, Yt)	(YS, Yt)	25.01	35.22
Dfwd st ∪ Dbwd st (C2)	(YS, Yt)	(YS, Yt)	25.28	35.29
ward SeqKD. Using machine-generated sequences in both languages increased the entropy, probably due to error accumulation. However, E2E-ST models do not suffer from it because they are conditioned on the source speech. We also confirmed similar trends in the reverse language direction.
Regarding faithfulness, distilled target sequences degraded faithfulness as expected. However, an interesting finding was that the faithfulness of bidirectional SeqKD was better than that of forward SeqKD, meaning that the former reflected the true word alignment distribution more faithfully than the latter. Although lexical choice might be degraded by targeting distilled text in both languages (Ding et al., 2021), mixing the original and distilled text by 2ref training would recover it.
We conduct an ablation study to verify the analysis in the previous section. In Table 4, we observed that it was better to have the original reference in the target sequence of either the source or target language. For example, to reduce the entropy of German text in the training set, it was best to condition the distilled German translation on the original English transcription, and vice versa. Therefore, we hypothesize that the best way to reduce the entropy in both source and target languages during 2ref training is to combine  $(\hat{Y}^{\mathrm{s}}, Y^{\mathrm{t}})$  and  $(Y^{\mathrm{s}}, \hat{Y}^{\mathrm{t}})$  for each sample. We compared four ways to leverage source text: gold transcription  $Y^{\mathrm{s}}$  only, distilled paraphrase  $\hat{Y}^{\mathrm{s}}$  only, and both. The results are shown in Table 6. We confirmed that the model trained with the original reference in either language for every target achieved the best BLEU score, which verifies our hypothesis.
4.4 Increasing model capacity
Finally, we investigate the effectiveness of bidirectional Seq-KD with 2ref training when increasing the model capacity in Table 7. The purpose of
Table 6: Ablation study of dataset concatenation on Must-C tst-COMMON set. 2ref training was used.
Model	BLEU (↑)
En-De	En-Fr
Transformer Large + Fwd SeqKD	25.19	35.47
+ Bidir SeqKD	25.62	35.74
Conformer + Fwd SeqKD	26.81	37.23
+ Bidir SeqKD	27.01	37.33
Text-based NMT (WER: 0%)†	27.56	39.09
Table 7: BLEU scores of large AR models on Must-C tst-COMMON set. 2ref training was used. † Punctuation and case information is removed on the source side.
this experiment is to verify our expectation that large models can model complex target distributions in multi-referenced training better. In addition to simply increasing the model dimensions, we also investigate Conformer (Gulati et al., 2020), a Transformer encoder augmented by a convolution module. We confirmed that bidirectional SeqKD always outperformed forward SeqKD in both language directions regardless of model configurations. We also found that the Conformer encoder significantly boosted the translation performance of forward SeqKD, but the gains of bidirectional SeqKD were transferred.
To fully leverage knowledge in both source and target language directions for bilingual E2E-ST models, we have proposed bidirectional SeqKD, in which both forward SeqKD from a source-to-target NMT model and backward SeqKD from a target-to-source NMT model are combined. Backward SeqKD is performed by targeting source paraphrases generated via back-translation from the original translations in bitext. Then, the E2E-ST model is enhanced by training to generate both source and target language text with a single decoder. We experimentally confirmed that SeqKD from each direction boosted the translation performance of both autoregressive and non-autoregressive E2EST models, and the effectiveness was additive. Multi-referenced training with the original and distilled text gave further gains. We also showed that bidirectional SeqKD was effective regardless of model sizes.
The authors thank the anonymous reviewers for useful suggestions and Siddharth Dalmia, Brian Yan, and Pengcheng Guo for helpful discussions.
Both gold translation  $Y^{\mathrm{t}}$  and distilled translation  $\hat{Y}^{\mathrm{t}}$  were always used as target sequences.