Venue: NAACL 2021

# Source and Target Bidirectional Knowledge Distillation

for End-to-end Speech Translation

 Hirofumi Inaguma\({}^{\clubsuit}\) Tatsuya Kawahara\({}^{\clubsuit}\) Shinji Watanabe\({}^{\spadesuit}\)

\({}^{\clubsuit}\) Kyoto University, Japan \({}^{\clubsuit}\) Johns Hopkins University, USA inaguma@sap.ist.i.kyoto-u.ac.jp

###### Abstract

A conventional approach to improving the performance of end-to-end speech translation (E2E-ST) models is to leverage the source transcription via pre-training and joint training with automatic speech recognition (ASR) and neural machine translation (NMT) tasks. However, since the input modalities are different, it is difficult to leverage source language text successfully. In this work, we focus on sequence-level knowledge distillation (SeqKD) from external text-based NMT models. To leverage the full potential of the source language information, we propose _backward SeqKD_, SeqKD from a target-to-source backward NMT model. To this end, we train a bilingual E2E-ST model to predict paraphrased transcriptions as an auxiliary task with a single decoder. The paraphrases are generated from the translations in bitext via back-translation. We further propose _bidirectional SeqKD_ in which SeqKD from both forward and backward NMT models is combined. Experimental evaluations on both autoregressive and non-autoregressive models show that SeqKD in each direction consistently improves the translation performance, and the effectiveness is complementary regardless of the model capacity.

## 1 Introduction

End-to-end speech translation (E2E-ST) [3], which aims to convert source speech to text in another language directly, is an active research area. Because direct ST is a more difficult task than automatic speech recognition (ASR) and machine translation (MT), various techniques have been proposed to ease the training process by using source transcription. Examples include pre-training [4, 5, 6], multi-task learning [7, 8, 9], knowledge distillation [10], meta-learning [11, 12], two-pass decoding [15, 16], and interactive decoding [10, 11]. However, as input modalities between ST and MT tasks are different, an auxiliary MT task is not always helpful, especially when additional bitext is not available [9]. Moreover, because monotonic speech-to-transcription alignments encourage the ASR task to see surface-level local information, an auxiliary ASR task helps the E2E-ST model to extract acoustic representations, not semantic ones, from speech.

Sequence-level knowledge distillation (SeqKD) [10] is another approach to transferring knowledge from one model to another. Recent studies have shown that SeqKD has the effect of reducing the complexity of training data and thus eases the training of student models, e.g., non-autoregressive (NAR) models [12, 13, 14].

Paraphrasing, which represents text in a different form but with the same meaning, can also be regarded as SeqKD when using neural paraphrasing via back-translation [16, 17, 18]. It has been studied to improve the reference diversity for MT system evaluations [15, 16] and the performance of low-resource neural MT (NMT) models [13, 12].

In this work, due to its simplicity and effectiveness, we focus on SeqKD from text-based NMT models to improve the performance of a bilingual E2E-ST model. In order to fully leverage source language information, we propose _backward SeqKD_, which targets paraphrased source transcriptions generated from a target-to-source backward NMT model as an auxiliary task. Then, a single ST decoder is trained to predict both source and target language text as in a multilingual setting [10]. This way, the decoder is biased to capture semantic representations from speech, un

like joint training with an auxiliary ASR task. We also propose _bidirectional SeqKD_, which combines SeqKD from two NMT models in both language directions. Therefore, the E2E-ST models can fully exploit the knowledge embedded in both forward and backward NMT models.

Experimental evaluations demonstrate that SeqKD from each direction consistently improves the translation performance of both autoregressive and non-autoregressive E2E-ST models. We also confirm that bidirectional SeqKD outperforms unidirectional SeqKD and that the effectiveness is maintained in large models.

## 2 Method

In this section, we propose bidirectional SeqKD from both forward and backward NMT models that leverages machine-generated source paraphrases as another target in addition to the distilled translation to enhance the training of a bilingual E2E-ST model. Let \(X\) denote input speech features in a source language and \(Y^{\rm s}\) and \(Y^{\rm t}\) denote the corresponding gold transcription and translation, respectively. Let \(\mathcal{D}_{\text{st}}=\{(X_{i},Y_{i}^{\rm s},Y_{i}^{\rm t})\}_{i=1}^{I}\) be an ST dataset including \(I\) samples, and \(\mathcal{D}_{\text{asr}}=\{(X_{i},Y_{i}^{\rm s})\}_{i=1}^{I}\) and \(\mathcal{D}_{\text{mt}}=\{(Y_{i}^{\rm s},Y_{i}^{\rm t})\}_{i=1}^{I}\) denote the corresponding ASR and MT datasets, respectively.1 We drop the subscript \(i\) when it is obvious.

Footnote 1: We focus on a complete triplet of \((X,Y^{\rm s},Y^{\rm t})\) only. However, the proposed method can easily be extended to a semi-supervised setting featuring additional ASR and MT pair data.

### Sequence-level knowledge distillation

We first train a text-based source-to-target forward NMT model \(\mathcal{M}_{\text{fwd}}\) with \(\mathcal{D}_{\text{mt}}\).2 Then, we perform beam search decoding with \(\mathcal{M}_{\text{fwd}}\) on \(\mathcal{D}_{\text{st}}\) to create a new dataset \(\mathcal{D}_{\text{st}}^{\text{fwd}}=\{(X_{i},Y_{i}^{\rm s},\hat{Y}_{i}^{\rm t })\}_{i=1}^{I}\), where \(\hat{Y}_{i}^{\rm t}\) is a distilled translation. \(\mathcal{D}_{\text{st}}^{\text{fwd}}\) is used to train the E2E-ST models, referred to as _forward SeqKD_ (or fwd SeqKD).

Footnote 2: All NMT models are autoregressive in this paper.

### Paraphrase generation

To exploit semantic information in the source language, we leverage machine-generated paraphrases of source transcriptions. We train a text-based target-to-source backward NMT model \(\mathcal{M}_{\text{bwd}}\) with \(\mathcal{D}_{\text{mt}}\) and then generate a new dataset \(\mathcal{D}_{\text{st}}^{\text{bvd}}=\{(X_{i},\hat{Y}_{i}^{\rm s},Y_{i}^{\rm t })\}_{i=1}^{I}\), where \(\hat{Y}_{i}^{\rm s}\) is a paraphrase of \(Y_{i}^{\rm s}\). We use \(\mathcal{D}_{\text{st}}^{\text{bvd}}\) for training the E2E-ST models. As neural paraphrasing can be regarded as SeqKD from \(\mathcal{M}_{\text{bwd}}\), we referred to it as _backward SeqKD_ (or bwd SeqKD). In this work, we do not use large paraphrase datasets Wieting and Gimpel (2018); Hu et al. (2019) because their availability depends on languages and domains. Moreover, neural paraphrasing is applicable to any source languages that lack a sufficient amount of paired paraphrase data.

We also propose combining forward SeqKD with backward SeqKD, referred to as _bidirectional SeqKD_ (or bidir SeqKD), and construct a new dataset \(\mathcal{D}_{\text{st}}^{\mathrm{bidir}}=\{(X_{i},\hat{Y}_{i}^{\rm s},\hat{ Y}_{i}^{\rm t})\}_{i=1}^{I}\). When using two references per utterance (_2ref_ training) Gordon and Duh (2019), we concatenate \(\mathcal{D}_{\text{st}}^{\text{fwd}}\) and \(\mathcal{D}_{\text{st}}^{\text{bvd}}\), and the suitable combination is analyzed in Section 4.3. This way, we can distill the knowledge of both \(\mathcal{M}_{\text{fwd}}\) and \(\mathcal{M}_{\text{bwd}}\) to a single E2E-ST model.

### Training

We train an E2E-ST model with a direct ST objective \(\mathcal{L}_{\text{st}}(Y^{\rm t}~{}\mathrm{or}~{}\hat{Y}^{\rm t}|X)\) and an auxiliary speech-to-source text objective \(\mathcal{L}_{\text{src}}(Y^{\rm s}~{}\mathrm{or}~{}\hat{Y}^{\rm s}|X)\). We refer to joint training with \(\mathcal{L}_{\text{src}}(Y^{\rm s}|X)\) as _joint ASR_ and with \(\mathcal{L}_{\text{src}}(\hat{Y}^{\rm s}|X)\) as _backward SeqKD_. Both losses are calculated from the same ST decoder. To bias the model to generate the desired target language, we add language embedding to token embedding at _every_ token position in the decoder Conneau and Lample (2019).3 We then apply bidirectional SeqKD to both autoregressive (AR) and non-autoregressive (NAR) E2E-ST models.

Footnote 2: All NMT models are autoregressive in this paper.

#### Autoregressive E2E-ST model

We use the speech Transformer architecture in Karita et al. (2019) with an additional language embedding. The total training objective is formulated with a hyperparameter \(\lambda_{\text{src}}(\geq 0)\) as

\[\mathcal{L}_{\text{total}}=\mathcal{L}_{\text{st}}+\lambda_{\text{src}} \mathcal{L}_{\text{src}}, \tag{1}\]

where both \(\mathcal{L}_{\text{st}}\) and \(\mathcal{L}_{\text{src}}\) are defined as cross-entropy losses. The entire encoder-decoder parameters are shared in both tasks.

#### Non-autoregressive E2E-ST model

We adopt Orthors Inaguma et al. (2021), in which a decoder based on a conditional masked language model (CMLM) Ghazvininejad et al. (2019) is jointly trained with an additional AR decoder 

on the shared speech encoder. The training of the NAR decoder is further enhanced with semi-autoregressive training (SMART) Ghazvininejad et al. (2020). \({\cal L}_{\rm st}\) in Eq. (1) is modified as

\[{\cal L}_{\rm st}={\cal L}_{\rm cmlm}+\lambda_{\rm ar}{\cal L}_{\rm ar}+\lambda_ {\rm lp}{\cal L}_{\rm lp}, \tag{2}\]

where \({\cal L}_{\rm cmlm}\), \({\cal L}_{\rm ar}\), and \({\cal L}_{\rm lp}\) are losses in NAR E2E-ST, AR E2E-ST, and length prediction tasks, respectively. \(\lambda_{*}\) is the corresponding tunable loss weight. During inference, the mask-predict algorithm is used for \(T\) iterations with a length beam width of \(l\)Ghazvininejad et al. (2019). The best candidate at the last iteration is selected from the NAR decoder based on scores from the AR decoder Inaguma et al. (2021). Note that we apply \({\cal L}_{\rm src}\) to the NAR decoder only.

## 3 Experimental setting

DataWe used Must-C En-De (\(408\) hours) and En-Fr (\(492\) hours) datasets Di Gangi et al. (2019). Both language pairs consist of a triplet of (\(X\), \(Y^{\text{s}}\), \(Y^{\text{t}}\)). We performed the same data preprocessing as Inaguma et al. (2020) (see details in Appendix A.1). We report case-sensitive detokenized BLEU scores Papineni et al. (2002) on the tst-COMMN set with the multi-bleu-detok.perl script in Moses Koehn et al. (2007).

Model configurationWe used the Transformer Vaswani et al. (2017) architecture having \(12\) encoder layers following two CNN blocks and six decoder layers for the ASR and E2E-ST tasks. For the MT models, we used six encoder layers. We built our models with the ESPnet-ST toolkit Inaguma et al. (2020). See details in Appendix A.2.

TrainingWe always initialized the encoder parameters of the E2E-ST model by those of the corresponding pre-trained ASR model Berard et al. (2018). We follow the same optimization strategies as in Inaguma et al. (2021, 2020). When using joint ASR or backward SeqKD, we set \(\lambda_{\rm src}\) to \(0.3\). More details are described in Appendix A.3 and A.4.

InferenceFor the AR models, we used a beam width of \(4\). For the NAR models, we set \(T=\{4,10\}\) and \(l=9\) as in Inaguma et al. (2021).

## 4 Results

### Main results

We first report the paraphrasing quality, which is shown in Table 1. As confirmed by the BLEU and translation edit rate (TER) scores Snover et al. (2006), the paraphrased source text was not just a simple copy of the transcription (see examples in Appendix A.5).

Autoregressive modelsThe results are shown in Table 2. Pre-training the ST decoder with the forward MT decoder (A2) improved the baseline performance (A1). Joint ASR showed a marginal improvement on En-De but a degraded performance on En-Fr (A3). We attribute this to the fact that the ASR task was more trivial than the ST task and biased the shared decoder to capture surface-level textual information. In contrast, backward SeqKD showed small but consistent improvements in both language directions (A4), and it was as effective as MT pre-training. As the encoder was already pre-trained with the ASR model, paraphrases had an additional positive effect on the BLEU improvement.

Forward SeqKD significantly improved the performance, as previously reported in Inaguma et al. (2021). However, the gains by MT pre-training and joint ASR were diminished. Forward SeqKD was more effective than backward SeqKD solely (A4 vs. B1). However, backward SeqKD was still

\begin{table}
\begin{tabular}{p{113.8pt} p{34.1pt} p{34.1pt}} \hline \hline Language direction & **BLEU** (\(\uparrow\)) & **TER** (\(\downarrow\)) \\ \hline De \(\rightarrow\) En & 43.49 & 38.60 \\ Fr \(\rightarrow\) En & 48.55 & 34.30 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Quality of paraphrases in the training set

\begin{table}
\begin{tabular}{c l c c c} \hline \hline \multirow{2}{*}{ID} & \multirow{2}{*}{Model} & \multicolumn{3}{c}{**BLEU** (\(\Delta\)) (\(\uparrow\))} \\ \cline{3-5}  & & En-De & & En-Fr \\ \hline \multirow{3}{*}{–} & ESPnet-ST† & 22.91 & 32.69 \\  & Fairseq-S2TI� & 22.7 & 32.9 \\  & + Multilingual� & 24.5 & 34.9 \\ \hline A1 & Baseline & 22.77 & 33.51 \\ A2 & + MT pre-training & **23.12** & (+0.35) & **33.84** & (+0.33) \\ A3 & + Joint ASR & 22.97 & (+0.20) & 33.37 & –(-0.14) \\ A4 & + J Bowl SeqKD & **23.11** & (+0.34) & **33.78** & (+0.23) \\ Bi & A1 + Fwd SeqKD & 24.42 & (+1.65) & 34.66 & (+1.15) \\ B2 & + MT pre-training & 24.68 & (+1.91) & 34.57 & (+1.06) \\ B3 & + Joint ASR & 24.67 & (+1.90) & 34.68 & (+1.17) \\ B4 & + Original (2ref) & **24.83** & **(+2.06)** & **34.92** & **(+1.41)** \\ C1 & Al + Bidir SeKD� & **24.83** & **(+2.06)** & **34.78** & **(+1.27)** \\ C2 & + Original (2ref) & **25.28** & **(+2.51)** & **35.29** & **(+1.78)** \\ \hline \hline \end{tabular}
\end{table}
Table 2: BLEU scores of AR models on Must-C test-COMMN set. † (Inaguma et al., 2020), ‡ Wang et al. (2020). \({}^{\circ}\)Large model trained with eight language pairs Wang et al. (2020).

 

beneficial on top of forward SeqKD (C1, i.e., bidirectional SeqKD) while joint ASR was less so (B3). We also augmented the target translations by concatenating \({\mathcal{D}_{\mathrm{st}}}\) and \({\mathcal{D}^{\mathrm{fwd}}_{\mathrm{st}}}\) (_2ref_ training), which further improved forward SeqKD (B4). Nevertheless, a combination of _2ref_ training and backward SeqKD (i.e., bidirectional SeqKD with \({\mathcal{D}^{\mathrm{fwd}}_{\mathrm{st}}}\cup{\mathcal{D}^{\mathrm{bwd}}_{ \mathrm{st}}}\)) had a complementary effect and showed the best result (C2). It even outperformed larger multilingual models Wang et al. (2020) without using additional data in other language pairs.

Non-autoregressive modelsThe results are presented in Table 3. Following the standard practice in NAR models Gu et al. (2018), we always used forward SeqKD. We did not use _2ref_ training for the NAR models because it increases the multi-modality. Joint ASR improved the performance on all NAR models, except for En-Fr with the number of iterations \(T=10\). However, bidirectional SeqKD with \({\mathcal{D}^{\mathrm{hidir}}_{\mathrm{st}}}\) further improved the performance consistently regardless of \(T\). Since NAR models assume conditional independence for every token, they prefer monotonic input-output alignments with lower alignment complexity in theory. However, paraphrasing collapses the monotonicity of the ASR task and increases the alignment complexity, making the auxiliary speech-to-source text task non-trivial. Nevertheless, BLEU scores were improved by adding backward SeqKD. This was probably because the complexity of transcriptions in the training data was reduced at the cost of the alignment complexity, which was more effective for the NAR models.

### Analysis

We analyze the performance of bidirectional SeqKD through a lens of complexity in the training data following Zhou et al. (2019). We aligned words in every source and target sentence pair with _fastalign4_Dyer et al. (2013). Then, we calculated corpus-level conditional entropy \(\mathcal{C}(\mathcal{D})\) and faithfulness \(\mathcal{F}(\mathcal{D})\) for both forward (\(\overrightarrow{\mathcal{P}}\)) and backward (\(\overleftarrow{\mathcal{D}}\)) language directions to evaluate the multi-modality. In short, conditional entropy measures uncertainty of translation, and faithfulness is defined as Kullback-Leibler divergence and measures how close the distilled data distribution is to the real data distribution. See the mathematical definition in Appendix A.6.

Footnote 4: [https://github.com/clab/fast_align](https://github.com/clab/fast_align)

The results of entropy and faithfulness are shown in Tables 4 and 5, respectively. Consistent with Zhou et al. (2019), the entropy of target translations was reduced by forward SeqKD, indicating target translations were converted into a more deterministic and simplified form. Interestingly, the entropy of the original translations was also reduced by backward SeqKD. In other words, backward SeqKD modified transcriptions so that the target translations can be predicted easier. This would help E2E-ST models learn relationships between source and target languages _from speech_ because E2E-ST models are not conditioned on text in another language explicitly. Therefore, we presume that the encoder representations were enhanced by back

\begin{table}
\begin{tabular}{l c c} \hline \hline  & \multicolumn{2}{c}{**Entropy**} \\ Condition & \multicolumn{2}{c}{(\(\uparrow\) more complex)} \\ \cline{2-3}  & En-De & En-Fr \\ \hline \(\mathcal{C}(\overrightarrow{\mathcal{D}^{\mathrm{fwd}}_{\mathrm{st}}})\) (Real) & 0.70 & 0.65 \\ \(\mathcal{C}(\overrightarrow{\mathcal{D}^{\mathrm{fwd}}_{\mathrm{st}}})\) (Fwd SeqKD) & **0.52** & **0.47** \\ \(\mathcal{C}(\overrightarrow{\mathcal{D}^{\mathrm{hvd}}_{\mathrm{st}}})\) (Bwd SeqKD) & 0.54 & 0.47 \\ \(\mathcal{C}(\overrightarrow{\mathcal{D}^{\mathrm{hidir}}_{\mathrm{st}}})\) (Bidir SeqKD) & 0.63 & 0.61 \\ \hline \(\mathcal{C}(\overleftarrow{\mathcal{D}^{\mathrm{fwd}}_{\mathrm{st}}})\) (Real) & 0.40 & 0.54 \\ \(\mathcal{C}(\overleftrightarrow{\mathcal{D}^{\mathrm{fwd}}_{\mathrm{st}}})\) (Fwd SeqKD) & 0.28 & 0.36 \\ \(\mathcal{C}(\overleftrightarrow{\mathcal{D}^{\mathrm{fwd}}_{\mathrm{st}}})\) (Bwd SeqKD) & **0.25** & **0.31** \\ \(\mathcal{C}(\overleftrightarrow{\mathcal{D}^{\mathrm{hidir}}_{\mathrm{st}}})\) (Bidir SeqKD) & 0.37 & 0.49 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Corpus-level conditional entropy

\begin{table}
\begin{tabular}{l r r} \hline \hline  & \multicolumn{2}{c}{**Faithfulness**} \\ Condition & \multicolumn{2}{c}{(\(\downarrow\) more faithful)} \\ \cline{2-3}  & En-De & En-Fr \\ \hline \(\mathcal{F}(\overrightarrow{\mathcal{D}^{\mathrm{fwd}}_{\mathrm{st}}})\) (Fwd SeqKD) & 12.61 & 11.65 \\ \(\mathcal{F}(\overrightarrow{\mathcal{D}^{\mathrm{fwd}}_{\mathrm{st}}})\) (Bwd SeqKD) & **9.31** & **8.67** \\ \(\mathcal{F}(\overrightarrow{\mathcal{D}^{\mathrm{hidir}}_{\mathrm{st}}})\) (Bidir SeqKD) & 11.42 & 10.72 \\ \hline \(\mathcal{F}(\overrightarrow{\mathcal{D}^{\mathrm{fwd}}_{\mathrm{st}}})\) (Fwd SeqKD) & **9.58** & **8.48** \\ \(\mathcal{F}(\overrightarrow{\mathcal{D}^{\mathrm{hidir}}_{\mathrm{st}}})\) (Bwd SeqKD) & 12.97 & 10.70 \\ \(\mathcal{F}(\overrightarrow{\mathcal{D}^{\mathrm{hidir}}_{\mathrm{st}}})\) (Bidir SeqKD) & 11.23 & 9.98 \\ \hline \hline \end{tabular}
\end{table}
Table 5: Faithfulness to training data distribution 

ward SeqKD. Using machine-generated sequences in both languages increased the entropy, probably due to error accumulation. However, E2E-ST models do not suffer from it because they are conditioned on the source speech. We also confirmed similar trends in the reverse language direction.

Regarding faithfulness, distilled target sequences degraded faithfulness as expected. However, an interesting finding was that the faithfulness of bidirectional SeqKD was better than that of forward SeqKD, meaning that the former reflected the true word alignment distribution more faithfully than the latter. Although lexical choice might be degraded by targeting distilled text in both languages Ding et al. (2021), mixing the original and distilled text by _2ref_ training would recover it.

### Ablation study

We conduct an ablation study to verify the analysis in the previous section. In Table 4, we observed that it was better to have the original reference in the target sequence of either the source or target language. For example, to reduce the entropy of German text in the training set, it was best to condition the distilled German translation on the original English transcription, and vice versa. Therefore, we hypothesize that the best way to reduce the entropy in both source and target languages during _2ref_ training is to combine (\(\hat{Y}^{\text{s}}\), \(Y^{\text{t}}\)) and (\(Y^{\text{s}}\), \(\hat{Y}^{\text{t}}\)) for each sample. We compared four ways to leverage source text: gold transcription \(Y^{\text{s}}\) only, distilled paraphrase \(\hat{Y}^{\text{s}}\) only, and both.5 The results are shown in Table 6. We confirmed that the model trained with the original reference in either language for every target achieved the best BLEU score, which verifies our hypothesis.

Footnote 5: Both gold translation \(Y^{\text{t}}\) and distilled translation \(\hat{Y}^{\text{t}}\) were always used as target sequences.

### Increasing model capacity

Finally, we investigate the effectiveness of bidirectional Seq-KD with _2ref_ training when increasing the model capacity in Table 7. The purpose of this experiment is to verify our expectation that large models can model complex target distributions in multi-referenced training better. In addition to simply increasing the model dimensions, we also investigate Conformer Gulati et al. (2020), a Transformer encoder augmented by a convolution module. We confirmed that bidirectional SeqKD always outperformed forward SeqKD in both language directions regardless of model configurations. We also found that the Conformer encoder significantly boosted the translation performance of forward SeqKD, but the gains of bidirectional SeqKD were transferred.

## 5 Conclusion

To fully leverage knowledge in both source and target language directions for bilingual E2E-ST models, we have proposed bidirectional SeqKD, in which both forward SeqKD from a source-to-target NMT model and backward SeqKD from a target-to-source NMT model are combined. Backward SeqKD is performed by targeting source paraphrases generated via back-translation from the original translations in bitext. Then, the E2E-ST model is enhanced by training to generate both source and target language text with a single decoder. We experimentally confirmed that SeqKD from each direction boosted the translation performance of both autoregressive and non-autoregressive E2E-ST models, and the effectiveness was additive. Multi-referenced training with the original and distilled text gave further gains. We also showed that bidirectional SeqKD was effective regardless of model sizes.

## Acknowledgement

The authors thank the anonymous reviewers for useful suggestions and Siddharth Dalmia, Brian Yan, and Pengcheng Guo for helpful discussions.

\begin{table}
\begin{tabular}{l c c c} \hline \multirow{2}{*}{Training data} & Target & Target & **BLEU** (\uparrow) \\ \cline{3-4}  & & En-De & En-Fr \\ \hline \(\mathcal{D}_{\text{st}}\cup\mathcal{D}_{\text{st}}^{\text{pred}}\) (@ at + Joint ASR) & \((\mathbf{Y}^{\text{s}},Y^{\text{r}})\) & \((\mathbf{Y}^{\text{s}},\hat{Y}^{\text{r}})\) & 25.00 & 35.05 \\ \(\mathcal{D}_{\text{st}}\cup\mathcal{D}_{\text{st}}^{\text{light}}\) & \((\mathbf{Y}^{\text{s}},\hat{Y}^{\text{r}})\) & \((\tilde{\mathbf{Y}}^{\text{s}},\hat{Y}^{\text{t}})\) & 25.21 & 35.17 \\ \(\mathcal{D}_{\text{st}}^{\text{rel}}\cup\mathcal{D}_{\text{st}}^{\text{int}}\) & \((\tilde{\mathbf{Y}}^{\text{s}},y^{\text{r}})\) & \((\tilde{\mathbf{Y}}^{\text{s}},\hat{Y}^{\text{t}})\) & 25.01 & 35.22 \\ \(\mathcal{D}_{\text{st}}^{\text{rel}}\cup\bigcup\mathcal{D}_{\text{st}}^{\text{ rel}}\) (\(\zeta_{2}\)) & \((\tilde{\mathbf{Y}}^{\text{s}},y^{\text{r}})\) & \((\mathbf{Y}^{\text{s}},\hat{Y}^{\text{t}})\) & **25.28** & **35.29** \\ \hline \end{tabular}
\end{table}
Table 6: Ablation study of dataset concatenation on Must-C tst-COMMN set. _2ref_ training was used.

\begin{table}
\begin{tabular}{l l l} \hline \multirow{2}{*}{Model} & \multicolumn{2}{c}{**BLEU** (\uparrow)} \\ \cline{2-3}  & En-De & En-Fr \\ \hline Transformer Large + Fwd SeqKD & 25.19 & 35.47 \\ + Bidir SeqKD & **25.62** & **35.74** \\ \hline Conformer + Fwd SeqKD & 26.81 & 37.23 \\ + Bidir SeqKD & **27.01** & **37.33** \\ \hline Text-based NMT (WER: 0%)† & 27.56 & 39.09 \\ \hline \end{tabular}
\end{table}
Table 7: BLEU scores of large AR models on Must-C tst-COMMN set. _2ref_ training was used. † Punctuation and case information is removed on the source side.

 



## References

* Anastasopoulos and Chiang (2018) Antonios Anastasopoulos and David Chiang. 2018. Tied multitask learning for neural speech translation. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_, pages 82-91, New Orleans, Louisiana. Association for Computational Linguistics.
* Bahar et al. (2019) Parnia Bahar, Tobias Bieschke, and Hermann Ney. 2019. A comparative study on end-to-end speech to text translation. In _Proceedings of ASRU_, pages 792-799.
* Bansal et al. (2019) Sameer Bansal, Herman Kamper, Karen Livescu, Adam Lopez, and Sharon Goldwater. 2019. Pretraining on high-resource speech recognition improves low-resource speech-to-text translation. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 58-68, Minneapolis, Minnesota. Association for Computational Linguistics.
* Bowden et al. (2020a) Rachel Dawden, Biao Zhang, Andre Tattar, and Matt Post. 2020a. ParBLEU: Augmenting metrics with automatic paraphrases for the wmt'20 metrics shared task. In _5th Conference on Machine Translation_.
* Bawden et al. (2020b) Rachel Bawden, Biao Zhang, Lisa Yankovskaya, Andre Tattar, and Matt Post. 2020b. A study in improving BLEU reference coverage with diverse automatic paraphrasing. In _Findings of the Association for Computational Linguistics: EMNLP 2020_, pages 918-932, Online. Association for Computational Linguistics.
* Berard et al. (2018) Alexandre Berard, Laurent Besacier, Ali Can Kocabiyikoglu, and Olivier Pietquin. 2018. End-to-end automatic speech translation of audiobooks. In _Proceedings of ICASSP_, pages 6224-6228.
* Berard et al. (2016) Alexandre Berard, Olivier Pietquin, Christophe Servan, and Laurent Besacier. 2016. Listen and translate: A proof of concept for end-to-end speech-to-text translation. In _Proceedings of NeurIPS 2016 End-to-end Learning for Speech and Audio Processing Workshop_.
* Conneau and Lample (2019) Alexis Conneau and Guillaume Lample. 2019. Crosslingual language model pretraining. In _Proceedings of NeurIPS_, pages 7059-7069.
* Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
* Di Gangi et al. (2019) Mattia A. Di Gangi, Roldano Cattoni, Luisa Bentivogli, Matteo Negri, and Marco Turchi. 2019. MuST-C: a Multilingual Speech Translation Corpus. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 2012-2017, Minneapolis, Minnesota. Association for Computational Linguistics.
* Ding et al. (2021) Liang Ding, Longyue Wang, Xuebo Liu, Derek F. Wong, Dacheng Tao, and Zhaopeng Tu. 2021. Understanding and improving lexical choice in non-autoregressive translation. In _Proceedings of ICLR_.
* Dyer et al. (2013) Chris Dyer, Victor Chahuneau, and Noah A. Smith. 2013. A simple, fast, and effective reparameterization of IBM model 2. In _Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 644-648, Atlanta, Georgia. Association for Computational Linguistics.
* Federmann et al. (2019) Christian Federmann, Oussama Elachqar, and Chris Quirk. 2019. Multilingual whispers: Generating paraphrases with translation. In _Proceedings of the 5th Workshop on Noisy User-generated Text (W-NUT 2019)_, pages 17-26, Hong Kong, China. Association for Computational Linguistics.
* Ghazvininejad et al. (2020) Marjan Ghazvininejad, Vladimir Karpukhin, Luke Zettlemoyer, and Omer Levy. 2020. Semi-autoregressive training improves mask-predict decoding. _arXiv preprint arXiv:2001.08785_.
* Ghazvininejad et al. (2019) Marjan Ghazvininejad, Omer Levy, Yinhan Liu, and Luke Zettlemoyer. 2019. Mask-predict: Parallel decoding of conditional masked language models. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 6112-6121, Hong Kong, China. Association for Computational Linguistics.
* Gordon and Duh (2019) Mitchell A Gordon and Kevin Duh. 2019. Explaining sequence-level knowledge distillation as data-augmentation for neural machine translation. _arXiv preprint arXiv:1912.03334_.
* Gu et al. (2018) Jiatao Gu, James Bradbury, Caiming Xiong, Victor OK Li, and Richard Socher. 2018. Non-autoregressive neural machine translation. In _Proceedings of ICLR_.
* Gulati et al. (2020) Anmol Gulati, James Qin, Chung-Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, and Ruoming Pang. 2020. Conformer: Convolution-augmented Transformer for speech recognition. In _Proceedings of Interspeech_, pages 5036-5040.
* Hu et al. (2019) J Edward Hu, Rachel Rudinger, Matt Post, and Benjamin Van Durme. 2019. ParaBank: Monolingual bitext generation and sentential paraphrasing via

lexically-constrained neural machine translation. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 33, pp. 6521-6528. Cited by: SS4.
* H. Inayama, K. Duh, T. Kawahara, and S. Watanabe (2021)Multilingual end-to-end speech translation. In Proceedings of ASRU, pp. 570-577. External Links: Link, Document Cited by: SS1, SS2.
* H. Inayama, Y. Higuchi, K. Duh, Tatsuya Kawahara, and S. Watanabe (2021)Orthros: non-autoregressive end-to-end speech translation with dual-decoder. In Proceedings of ICASSP, Cited by: SS1, SS2.
* H. Inayama, S. Kiyono, K. Duh, S. Karita, N. Yalta, T. Hayashi, and S. Watanabe (2020)ESPnet-ST: all-in-one speech translation toolkit. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, Online, pp. 302-311. External Links: Link, Document Cited by: SS1, SS2.
* S. Indurthi, H. Han, N. Kumar Lakumarapu, B. Lee, I. Chung, S. Kim, and C. Kim (2020)End-end speech-to-text translation with modality agnostic meta-learning. In Proceedings of ICASSP, pp. 7904-7908. Cited by: SS1, SS2.
* S. Karita, N. Chen, T. Hayashi, T. Hori, H. Inaguma, Z. Jiang, M. Someki, N. Enrique Yalta Soplin, R. Yamamoto, X. Wang, et al. (2019)A comparative study on transformer vs RNN in speech applications. In Proceedings of ASRU, pp. 499-456. Cited by: SS1, SS2.
* H. Khayrallah, B. Thompson, M. Post, and P. Koehn (2020)Simulated multiple reference training improves low-resource machine translation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Online, pp. 82-89. External Links: Link, Document Cited by: SS1, SS2.
* Y. Kim and A. M. Rush (2016)Sequence-level knowledge distillation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, Austin, Texas, pp. 1317-1327. External Links: Link, Document Cited by: SS1, SS2.
* D. Kingma and J. Ba (2015)Adam: a method for stochastic optimization. Proceedings of ICLR. Cited by: SS4.
* T. Ko, V. Peddinti, D. Povey, and S. Khudanpur (2015)Audio augmentation for speech recognition. In Proceedings of Interspeech, pp. 3586-3589. Cited by: SS1, SS2.
* Philipp Koehn, H. Hoang, A. Birch, C. Callison-Burch, M. Federico, N. Bertoldi, E. Cowan, W. Shen, C. Moran, R. Zens, C. Dyer, O. Bojar, A. Constantin, and E. Herbst (2007)Moses: open source toolkit for statistical machine translation. In Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics Companion Volume Proceedings of the Demo and Poster Sessions, Prague, Czech Republic, pp. 177-180. External Links: Link, Document Cited by: SS1, SS2.
* T. Kudo (2018)Subword regularization: improving neural network translation models with multiple subword candidates. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Melbourne, Australia, pp. 66-75. External Links: Link, Document Cited by: SS1, SS2.
* H. Le, J. Pino, C. Wang, J. Gu, D. Schwab, and L. Besacier (2020)Dual-decoder transformer for joint automatic speech recognition and multilingual speech translation. In Proceedings of the 28th International Conference on Computational Linguistics, Barcelona, Spain, pp. 3520-3533. External Links: Link, Document Cited by: SS1, SS2.
* Y. Liu, H. Xiong, Z. He, J. Zhang, H. Wu, H. Wang, and C. Zong (2019)End-to-end speech translation with knowledge distillation. In Proceedings of Interspeech, pp. 1128-1132. Cited by: SS1, SS2.
* Y. Liu, J. Zhang, H. Xiong, L. Zhou, Z. He, H. Wu, H. Wang, and C. Zong (2020)Synchronous speech recognition and speech-to-text translation with interactive decoding. In Proceedings of AAAI, pp. 8417-8424. Cited by: SS1, SS2.
* J. Mallinson, R. Sennrich, and M. Lapata (2017)Paraphrasing revisited with neural machine translation. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, Valencia, Spain, pp. 881-893. External Links: Link, Document Cited by: SS1, SS2.
* K. Papineni, S. Roukos, T. Ward, and W. Zhu (2002)BLEU: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, Philadelphia, Pennsylvania, USA, pp. 311-318. External Links: Link, Document Cited by: SS1, SS2.
* D. S. Park, W. Chan, Y. Zhang, C. Chiu, B. Zoph, E. D. Cubuk, and Q. V. Le (2019)SpecAugment: a simple data augmentation method for automatic speech recognition. In Proceedings of Interspeech, pp. 2613-2617. Cited by: SS1, SS2.
* Daniel Povey, A. Ghoshal, G. Boulianne, L. Burget, O. Glembek, N. Goel, M. Hannemann, P. Motlicek, Y. Qian, P. Schwarz, et al. (2011)The kaldi speech recognition toolkit. In Proceedings of ASRU, 

Yi Ren, Jinglin Liu, Xu Tan, Zhou Zhao, Sheng Zhao, and Tie-Yan Liu. 2020. A study of non-autoregressive model for sequence generation. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 149-159, Online. Association for Computational Linguistics.
* Sennrich et al. (2016) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 1715-1725, Berlin, Germany. Association for Computational Linguistics.
* Snover et al. (2006) Matthew Snover, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul. 2006. A study of translation edit rate with targeted human annotation. In _Proceedings of association for machine translation in the Americas_, volume 200. Cambridge, MA.
* Sperber et al. (2019) Matthias Sperber, Graham Neubig, Jan Niehues, and Alex Waibel. 2019. Attention-passing models for robust and data-efficient end-to-end speech translation. _Transactions of the Association for Computational Linguistics_, 7:313-325.
* Szegedy et al. (2016) Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. 2016. Rethinking the inception architecture for computer vision. In _Proc. of CVPR_, pages 2818-2826.
* Thompson and Post (2020) Brian Thompson and Matt Post. 2020. Automatic machine translation evaluation in many languages via zero-shot paraphgrasing. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 90-121, Online. Association for Computational Linguistics.
* Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In _Advances in neural information processing systems_, pages 5998-6008.
* Wang et al. (2020a) Changhan Wang, Yun Tang, Xutai Ma, Anne Wu, Dmytro Okhonko, and Juan Pino. 2020a. Fairseq S2T: Fast speech-to-text modeling with fairseq. In _Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing: System Demonstrations_, pages 33-39, Suzhou, China. Association for Computational Linguistics.
* Wang et al. (2020b) Changhan Wang, Anne Wu, and Juan Pino. 2020b. CoVoST 2: A massively multilingual speech-to-text translation corpus. _arXiv preprint arXiv:2007.10310_.
* Wang et al. (2020c) Chengyi Wang, Yu Wu, Shujie Liu, Zhenglu Yang, and Ming Zhou. 2020c. Bridging the gap between pretraining and fine-tuning for end-to-end speech translation. In _Proceedings of AAAI_, pages 9161-9168.
* Wang et al. (2020d) Chengyi Wang, Yu Wu, Shujie Liu, Ming Zhou, and Zhenglu Yang. 2020d. Curriculum pre-training for end-to-end speech translation. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 3728-3738, Online. Association for Computational Linguistics.
* Watanabe et al. (2017) Shinji Watanabe, Takeaki Hori, Suyoung Kim, John R Hershey, and Tomoki Hayashi. 2017. Hybrid CTC/attention architecture for end-to-end speech recognition. _IEEE Journal of Selected Topics in Signal Processing_, 11(8):1240-1253.
* Weiss et al. (2017) Ron J Weiss, Jan Chorowski, Navdeep Jaitly, Yonghui Wu, and Zhifeng Chen. 2017. Sequence-to-sequence models can directly translate foreign speech. In _Proceedings of Interspeech_, pages 2625-2629.
* Wieting and Gimpel (2018) John Wieting and Kevin Gimpel. 2018. ParaNMT-50M: Pushing the limits of paraphrastic sentence embeddings with millions of machine translations. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 451-462, Melbourne, Australia. Association for Computational Linguistics.
* Wieting et al. (2017) John Wieting, Jonathan Mallinson, and Kevin Gimpel. 2017. Learning paraphrastic sentence embeddings from back-translated bitext. In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_, pages 274-285, Copenhagen, Denmark. Association for Computational Linguistics.
* Zhou et al. (2019a) Chunting Zhou, Jiatao Gu, and Graham Neubig. 2019a. Understanding knowledge distillation in non-autoregressive machine translation. In _Proceedings of ICLR_.
* Zhou et al. (2019b) Zhong Zhou, Matthias Sperber, and Alexander Waibel. 2019b. Paraphrases as foreign languages in multilingual neural machine translation. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop_, pages 113-122, Florence, Italy. Association for Computational Linguistics.

 



## Appendix A Appendix

### Data preprocessing

All sentences were tokenized with the tokenizer.perl script in Moses (Koehn et al., 2007). Non-verbal speech labels such as "_(Applause)_" and "_(Laughter)_" were removed during evaluation Di Gangi et al. (2019); Inaguma et al. (2021); Le et al. (2020). We built output vocabularies based on the byte pair encoding (BPE) algorithm (Sennrich et al., 2016) with the Sentencepiece toolkit Kudo (2018)6. The joint source and target vocabularies were constructed in the ST and MT tasks, while the vocabularies in the ASR task were constructed with transcriptions only. For autoregressive models, we used \(5\text{k}\) for ASR models and \(8\text{k}\) for E2E-ST and MT models. We used \(16\text{k}\) vocabularies for non-autoregressive E2E-ST models Inaguma et al. (2021).

Footnote 6: [https://github.com/google/sentencepiece](https://github.com/google/sentencepiece)

For input speech features, we extracted \(80\)-channel log-mel filterbank coefficients computed with a \(25\)-ms window size and shifted every \(10\text{ms}\) with \(3\)-dimensional pitch features using Kaldi Povey et al. (2011). This results in \(83\)-dimensional features for every frame. The features were normalized by the mean and the standard deviation for each training set. To avoid overfitting, training data was augmented by a factor of \(3\) with speed perturbation Ko et al. (2015) and SpecAugment Park et al. (2019). We used \((m_{T},m_{F},T,F)=(2,2,40,30)\) for the hyperparameters in SpecAugment.

### Model configuration

We used the Transformer Vaswani et al. (2017) architecture implemented with the ESPnet-STD toolkit Inaguma et al. (2020) for all tasks. ASR and E2E-ST models consisted of \(12\) speech encoder blocks and six decoder blocks. The speech encoders had two CNN blocks with a kernel size of \(3\) and a channel size of \(256\) before the first Transformer encoder layer, which resulted in \(4\)-fold downsampling in the time and frequency axes. The text encoder in the MT models consisted of six Transformer blocks. The dimensions of the self-attention layer \(d_{\mathrm{model}}\) and feed-forward network \(d_{\mathrm{ff}}\) were set to \(256\) and \(2048\), respectively, and the number of attention heads \(H\) was set to \(4\). For a large Transformer model configuration, we increased \(d_{\mathrm{ff}}\) from \(256\) to \(512\) and \(H\) from \(4\) to \(8\). For a Conformer model configuration, we set \(d_{\mathrm{model}}=256\), \(d_{\mathrm{ff}}=2048\), and \(H=4\). The kernel size of depthwise separable convolution was set to \(15\). None of the other training or decoding hyperparameters were modified.

### Initialization

In addition to initializing the encoder parameters of the E2E-ST model by those of the pre-trained ASR model, the auxiliary AR decoder parameters of the NAR models were initialized by those of the corresponding pre-trained AR MT model Inaguma et al. (2021). The other decoder parameters of both the AR and NAR models were initialized as in BERT Devlin et al. (2019); Ghazvininejad et al. (2019); Inaguma et al. (2021), where weight parameters were sampled from \(\mathcal{N}(0,0.02)\), biases were set to zero, and layer normalization parameters were set to \(\beta=0\), \(\gamma=1\). Note that we did not use additional data for pre-training.

### Training

The Adam optimizer Kingma and Ba (2015) with \(\beta_{1}=0.9\), \(\beta_{2}=0.98\), and \(\epsilon=10^{-9}\) was used for training with a Noam learning rate schedule Vaswani et al. (2017). We used dropout and label smoothing Szegedy et al. (2016) with a probability of \(0.1\) and \(0.1\), respectively. The other training configurations for all tasks are summarized in Table 9. We removed utterances having more than \(3000\) input speech frames or more than \(400\) characters due to the GPU memory capacity. The last five best checkpoints based on the validation performance were used for model averaging.

\begin{table}
\begin{tabular}{p{113.8pt}|p{113.8pt}} \hline \hline Reference1 & She took our order, and then went to the couple in the booth next to us, and she lowered her voice so much, I had to really strain to hear what she was saying. \\ \multirow{2}{*}{\(\hat{\texttt{ParaphraseI (Backward NMT)}}\)} & She picked up our order, and then went to the pair in the niche next to us and lowered her voice so much that I had to really try to understand them. \\ \cline{2-3}  & And she said ”Yes, that’s former Vice President AI Gore and his wife, Tipper.” \\ \multirow{2}{*}{\(\hat{\texttt{ParaphraseI (Backward NMT)}}\)} & And the man said, “He’s come down a long way, hasn’t he?” (Laughter) \\ \multirow{2}{*}{\(\hat{\texttt{Shé said}}\)} & She said, “Yes, that’s ex-vice President AI Göre and his wife Tipper.”And thie \\  & man said, “It’s a nice gap, what?” (Laughter) \\ \hline \end{tabular}
\end{table}
Table 8: Examples of source paraphrases on the Must-C En-De training set

For the training of ASR models used for E2E-ST encoder pre-training, we removed case and punctuation information from transcriptions and then applied a joint CTC/Attention objective (Watanabe et al., 2017). However, we retained this information in the transcriptions and paraphrases used for training the E2E-ST and MT models.

### Case study

We present examples of generated paraphrases on the Must-C En-De training set in Table 8. We observed that most paraphrases kept the original meaning while some words were simplified to alternatives having a similar meaning. We also found that the first conjunction in an utterance was more likely to be omitted via paraphrasing.

### Mathematical formulation of complexity and faithfulness

In this section, we mathematically formulate the corpus-level complexity and faithfulness given \({\cal D}\in\{{\cal D}_{\rm st},{\cal D}^{\rm fwd}_{\rm st},{\cal D}^{\rm bwd}_{ \rm st},{\cal D}^{\rm bidir}_{\rm st}\}\). Our formulation follows (Zhou et al., 2019a), but we also consider the reverse language direction.

Conditional entropy (complexity)The corpus-level complexity of \({\cal D}\) in the forward language direction, \({\cal C}(\overrightarrow{{\cal D}})\), is defined as the conditional entropy \({\cal H}(Y^{\rm t}|Y^{\rm s})\) normalized over all samples. \({\cal H}(Y^{\rm t}|Y^{\rm s})\) is defined as

\[{\cal H}(Y^{\rm t}|Y^{\rm s})\] \[=-\sum_{i=1}^{I}p(Y^{\rm t}_{i}|Y^{s}_{i})\cdot\log p(Y^{\rm t}_ {i}|Y^{s}_{i})\] \[\approx-\sum_{i=1}^{I}(\prod_{k=1}^{|Y^{s}_{i}|}P(Y^{\rm t}_{i, k}|Y^{\rm s}_{i}))\cdot\sum_{k=1}^{|Y^{s}_{i}|}\log p(Y^{\rm t}_{i,k}|Y^{\rm s }_{i})\] \[\approx-\sum_{k=1}^{T^{\rm t}}\sum_{y^{\rm t}_{k}\in{\cal A}(Y^{ \prime}_{i})}p(y^{\rm t}_{k}|\text{Align}(y^{\rm t}_{k}))\cdot\log p(y^{\rm t} _{k}|\text{Align}(y^{\rm t}_{k}))\] \[=\sum_{k=1}^{T^{\rm u}}{\cal H}(y^{\rm t}|Y^{\rm s}_{i,k}),\]

where \({\cal A}\) is an external alignment model, and \(T^{\rm s}\) and \(T^{\rm t}\) are the source and target sequence lengths, respectively. We make two assumptions: (1) conditional independence of target tokens given the source text sequence, and (2) the distribution of \(p(y^{\rm t}|Y^{\rm s})\) follows the alignment model \({\cal A}\). Then, \({\cal C}(\overrightarrow{{\cal D}})\) is calculated as

\[{\cal C}(\overrightarrow{{\cal D}}) = \frac{1}{|{\cal V}^{\rm s}|}\sum_{y^{\rm t}\in{\cal V}^{\rm s}}{ \cal H}(y^{\rm t}|y^{\rm s}),\]

where \({\cal V}^{\rm s}\) is a set of all words in the source language. Division by \(|{\cal V}^{\rm s}|\) is important to normalize frequent source words.

The corpus-level complexity of \({\cal D}\) in the backward language direction, \({\cal C}(\overleftarrow{{\cal D}})\), is defined similarly as

\[{\cal C}(\overleftarrow{{\cal D}}) = \frac{1}{|{\cal V}^{\rm t}|}\sum_{y^{\rm t}\in{\cal V}^{\rm t}}{ \cal H}(y^{\rm s}|y^{\rm t}),\]

where \({\cal V}^{\rm t}\) is a set of all words in the target language.

FaithfulnessAlthough the corpus-level conditional entropy can be used to evaluate the complexity of the training data, there are also trivial solutions to generate new data with smaller complexity when target translations are not adequate. Faithfulness is a good measure to assess how close the distilled data distribution is to the real (original) data distribution. The faithfulness of \({\cal D}\) in a forward language direction \({\cal F}(\overrightarrow{{\cal D}})\) and a backward language direction \({\cal F}(\overleftarrow{{\cal D}})\) is defined as the KL-divergence of the alignment distribution between the real dataset and a distilled dataset, as

\[{\cal F}(\overrightarrow{{\cal D}}) =\frac{1}{|{\cal V}^{\rm s}|}\sum_{y^{\rm c}\in{\cal V}^{\rm s}} \sum_{y^{\rm d}\in{\cal V}^{\rm l}}p_{\rm r}(y^{\rm t}|y^{\rm s})\;\mbox{log} \frac{p_{\rm r}(y^{\rm t}|y^{\rm s})}{p_{\rm d}(y^{\rm t}|y^{\rm s})},\] \[{\cal F}(\overleftarrow{{\cal D}}) =\frac{1}{|{\cal V}^{\rm t}|}\sum_{y^{\rm t}\in{\cal V}^{\rm t}} \sum_{y^{\rm g}\in{\cal V}^{\rm s}}p_{\rm r}(y^{\rm s}|y^{\rm t})\;\mbox{log} \frac{p_{\rm r}(y^{\rm s}|y^{\rm t})}{p_{\rm d}(y^{\rm s}|y^{\rm t})},\]

where \(p_{\rm r}\) and \(p_{\rm d}\) are alignment distributions of the real and distilled data, respectively. Therefore, when \({\cal D}={\cal D}_{\rm st}\), \({\cal F}(\overrightarrow{{\cal D}})={\cal F}(\overleftarrow{{\cal D}})=0\), and it was omitted in Table 5.

\begin{table}
\begin{tabular}{l|c c c c} \hline \multirow{2}{*}{Configuration} & \multirow{2}{*}{ASR} & \multicolumn{3}{c}{E2E-ST} & \multirow{2}{*}{MT} \\ \cline{4-4}  & & AR & NAR & \\ \hline Warmup step & 25k & 25k & 25k & 8k \\ Learning rate factor & 5.0 (2.0) & 2.5 & 5.0 & 1.0 \\ Batch size \(\times\) accum & 128 & 128 & 256 & 96 \\ Epoch & 45 (30) & 30 & 50 & 100 \\ Validation metric & Accuracy & BLEU & BLEU & BLEU \\ \hline \end{tabular}
\end{table}
Table 9: Summary of training configuration. Numbers inside parentheses correspond to Conformer.