CKDST: Comprehensively and Effectively Distill Knowledge from Machine Translation to End-to-End Speech Translation
Year: 2023

Abstract:
Distilling knowledge from a high-resource task, e.g., machine translation, is an effective way to alleviate the data scarcity problem of endto-end speech translation. However, previous works simply use the classical knowledge distillation that does not allow for adequate transfer of knowledge from machine translation. In this paper, we propose a comprehensive knowledge distillation framework for speech translation, CKDST, which is capable of comprehensively and effectively distilling knowledge from machine translation to speech translation from two perspectives: cross-modal contrastive representation distillation and simultaneous decoupled knowledge distillation. In the former, we leverage a contrastive learning objective to optimize the mutual information between speech and text representations for representation distillation in the encoder. In the later, we decouple the non-target class knowledge from target class knowledge for logits distillation in the decoder. Experiments on the MuST-C benchmark dataset demonstrate that our CKDST substantially improves the baseline by 1.2 BLEU on average in all translation directions, and outperforms previous state-ofthe-art end-to-end and cascaded speech translation models. The source code is available at https://github.com/ethanyklei/CKDST.

heading: 1 Introduction
text: End-to-end (E2E) speech-to-text translation (ST), directly translating speech in one language into text in another, has recently attracted increasing attention (Duong et al., 2016; Zhang et al., 2020; Xu et al., 2021; Ye et al., 2022). Compared with traditional cascaded ST, E2E ST does not require automatic transcription, which endows itself with less error propagation and lower latency.
However, parallel ST data that consist of speech inputs and target translations, are proverbially
∗corresponding author.
limited, especially in comparison with automatic speech recognition (ASR) and machine translation (MT) data. In order to mitigate this issue, previous efforts leverage pre-training approaches (Xu et al., 2021; Ao et al., 2022) and multi-task learning (MTL) frameworks (Ye et al., 2021; Tang et al., 2021; Han et al., 2021) to transfer knowledge from ASR and/or MT to ST. Among them, knowledge distillation (KD) (Hinton et al., 2015) has proved to be an effective way to improve ST performance by transferring knowledge from MT to ST (Liu et al., 2019; Xu et al., 2021; Tang et al., 2021).
However, previous KD approaches to ST only explore the classical KD that transfers knowledge from prediction logits, which may not allow for sufficient knowledge distillation. Specifically, in classical KD (Hinton et al., 2015), two types of knowledge are encoded in prediction logits, target class knowledge from target class logits and nontarget class knowledge from non-target class logits. Each type of knowledge contributes to the success of classical logits distillation. However, Zhao et al. (2022) have found that the classical KD couples the non-target class knowledge with the target class knowledge. Such entanglement may inhibit the transfer of non-target class knowledge and limit the performance of logits knowledge distillation.
Additionally, due to the modality gap between speech and text, it might be difficult for E2E ST to sufficiently capture and translate semantic information embedded in speech inputs to target translations. Fortunately, however, in MTL-based E2E ST, a speech input is accompanied with its transcription that is used as the input fed into MT. Such speech and transcription pairs allow us to distill knowledge from transcription representations to speech representations so as to reduce the modality gap. However, such knowledge distillation has not yet been explored for end-to-end speech translation.
In order to address these two issues and efficiently distill MT knowledge to ST, we propose
3123
a Comprehensive Knowledge Distillation framework for ST (CKDST). Specifically, we propose Cross-modal Contrastive Representation Distillation (CCRD) and Simultaneous Decoupled Knowledge Distillation (SDKD) as two essential approaches for CKDST, to transfering knowledge from text representations and to performing more sufficient logits distillation.
CCRD applies a contrastive training objective to force E2E ST to learn speech representations that are closer to their corresponding textual representations. In doing so, we could increase the mutual information lower bound between speech and text representations (Tian et al., 2019). SDKD is proposed for E2E ST to mitigate the issue that the classical KD couples the target class knowledge with non-target class knowledge (Zhao et al., 2022). For more effectively transferring logits knowledge from MT to ST, we decouple these two types of knowledge in prediction logits and extend the decoupled knowledge distillation to the MTL framework where both ST and MT are fine-tuned simultaneously.
In a nutshell, our contributions are three-fold.
• We propose CKDST for end-to-end ST, which can comprehensively and effectively transfer MT knowledge to ST in both the encoder and decoder.
• We introduce CCRD and SDKD in CKDST to increase the mutual information between speech and text representations, and to decouple the non-target class knowledge from the target knowledge for more effective logits distillation, respectively.
• We conduct extensive experiments on the MuST-C benchmark dataset with four language pairs. Experiment results validate the effectiveness of the two approaches and demonstrate that our model outperforms previous best end-to-end and cascaded baselines.

heading: 2 Related Work
text: End-to-End Speech Translation. To alleviate the error propagation in cascaded ST and to ease the deployment, Bérard et al. (2016) and Weiss et al. (2017) propose to use an end-to-end architecture to directly translate speech in one language into text in another, without using the intermediate transcriptions. In recent years, increasing efforts have
been done in E2E ST (Di Gangi et al., 2019b; Liu et al., 2019; Wang et al., 2020b; Liu et al., 2020; Xu et al., 2021; Tang et al., 2021; Fang et al., 2022; Tang et al., 2022). Since the parallel speech translation data is notoriously limited, many approaches have been proposed to solve this problem, such as pre-training (Wang et al., 2020b; Xu et al., 2021; Tang et al., 2022), multi-task learning (Le et al., 2020; Zhao et al., 2021; Ye et al., 2022), and data augmentation (Bahar et al., 2019; Lam et al., 2022). Additionally, knowledge distillation from a well trained MT model to a ST model has proved effective in improving ST performance. Liu et al. (2019) leverage knowledge distillation to allow the E2E ST model to learn the same prediction distribution as the MT model. The MT model is frozen while the ST model is being trained. SATE (Xu et al., 2021) uses both pre-trained ASR model and MT model as teacher models to perform knowledge distillation. Each pre-trained model serves a different module of the ST model, and they are also frozen during training. Tang et al. (2021) propose the online-KD that simultaneously update the ST module and the MT module in a multi-task learning framework. However, these efforts only distill the knowledge from prediction logits via classical KD, and the knowledge from encoder representations is ignored. In our work, we comprehensively and efficiently distill knowledge from both encoder representations and prediction logits of MT to ST.
Knowledge Distillation. The concept of knowledge distillation has been firstly proposed by Hinton et al. (2015). KD defines a learning framework where a stronger teacher network is employed to guide the training of a student network for many tasks (Kim and Rush, 2016; Li et al., 2017; Tan et al., 2019). The subsequent works can be roughly divided into two groups, distillation from prediction logits (Furlanello et al., 2018; Cho and Hariharan, 2019; Yang et al., 2019; Mirzadeh et al., 2020) and intermediate representations (Yim et al., 2017; Huang and Wang, 2017; Heo et al., 2019; Park et al., 2019). Romero et al. (2014) explore intermediate representations for KD by using regressions to guide the feature activations of the student network. Tian et al. (2019) apply a contrastive objective to maximize the mutual information lower bound between teacher representations and student representations. In contrast, DKD (Zhao et al., 2022) decouples and amplifies student-friendly knowledge from prediction distribution to perform more
effective distillation. Our approaches are partially motivated by these previous efforts but are significantly different from them in two aspects. First, the knowledge gap in representations is due to different modalities rather than the model capability (teacher vs. student), so the inconsistency of modalities is also an important challenge we have to deal with in our distillation. Second, we don’t freeze the teacher during distillation, we argue that this may allow the teacher model to adapt to provide student-friendly knowledge.

heading: 3 CKDST
text: In this section, we first introduce the model architecture of CKDST and then elaborate the two knowledge distillation approaches in CKDST.

heading: 3.1 Model Architecture
text: CKDST adopts the encoder-decoder ST framework, as shown in Figure 1. It consists of four main components: speech encoder, text encoder, shared encoder and shared decoder, facilitating the joint training of the ST and MT task. Speech Encoder is composed of non-finetuned wav2vec 2.0 (Baevski et al., 2020) followed by two layers of 1-D CNNs. It takes speech wavforms as input to obtain low-level speech representations. Text Encoder is the normal word embedding layer, which is the same as the word embedding layer for
text translation. It takes text as input for the MT task. Shared Encoder / Decoder adopt the standard Transformer (Vaswani et al., 2017) as their backbone network. The shared encoder takes outputs from both speech and text encoder as inputs to further extract semantic information. The shared decoder generates target translations for ST and MT. And, with shared parameters, the shared encoder and decoder are expected to learn the shared knowledge between ST and MT.
A training sample for E2E ST is a (speech,transcript,translation) triplet (s, t, y). We use speech-translation pairs (s, y) as training data for ST, and transcript-translation pairs (t, y) as training data for MT. The cross-entropy loss is adopted for both ST and MT:
LST =− |y|∑
i=1
log p(yi|y<i, s)
LMT =− |y|∑
i=1
log p(yi|y<i, t) (1)

heading: 3.2 Cross-modal Constrative Representation Distillation
text: Speech inputs are usually noisier than their textual counterpart transcripts (Tang et al., 2021), which makes the extraction of semantic information from
speech difficult. Thus, we want to transfer semantic knowledge across modality, from text representations to speech representations. For this, we propose cross-modal contrastive representation distillation which employs a contrastive training objective to maximize the mutual information between text and speech representations. Due to the length difference between speech and text, we use sentencelevel representations for distillation. We apply average pooling on the output of the shared encoder in the time dimension to obtain sentence-level representations of speech and text.
Concretely, let T and S denote the sentencelevel source text representation (teacher representation) in MT and speech representation (student representation) in ST, respectively. Tian et al. (2019) show that a lower bound of the mutual information (MI) between T and S exists when we have 1 positive pair (i.e., speech-transcript pair) and N negative pairs (i.e., pairs of a speech with the rest of the transcripts in the same mini batch). The lower bound is estimated as follows:
MI(T ,S) ≥ log(N) + Eq(T ,S)[log h(T ,S)] +NEq(T )q(S)[log (1− h(T ,S))]
(2) where q(T ,S) (indicating positive pairs) and q(T )q(S) (indicating negative pairs) are the joint distribution and the product of marginal distributions of T and S, respectively. h(·) is a critic function that estimates the probability that the input pair (T ,S) is drawn from q(T ,S).
We optimize our model to maximize the expectation terms in Eq. (2). In doing so, we force our model to learn a representation for S, which is semantically close to that of T , so as to optimize the mutual information between S and T . This can be considered as a procedure that distills knowledge from the representation of T to that of S.
Given a sentence-level text-speech representation pair (T ,S), the critic function h(·) calculates a score indicating the possiblity that (T ,S) is a positive pair (drawn from q(T,S)) or negative pair (drawn from q(T )q(S)). We define the critic function h(T ,S) → [0, 1] as follows:
h(T ,S) = ecos(T ,S)/τ
1 + ecos(T ,S)/τ (3)
where cos(·) is the cosine similarity and τ is a temperature hyper-parameter. With the critic probabil-
ity h(T ,S), we can calculate the loss of CCRD:
LCCRD =− (Eq(T ,S)[log h(T ,S)] +NEq(T )q(S)[log (1− h(T ,S))]) (4)
Different from Ye et al. (2022) who use contrastive learning to bridge the modality gap between speech and text, we use NCE loss (Gutmann and Hyvärinen, 2010) as the contrastive objective, and aim to maximize the lower bound of mutual information between speech and text representations.

heading: 3.3 Simultaneous Decoupled Knowledge Distillation
text: Previous efforts (Liu et al., 2019; Xu et al., 2021; Tang et al., 2021) calculate the KL-Divergence on prediction logits between ST and MT for logits distillation. However, this classical KD loss (Hinton et al., 2015) couples target class knowledge and non-target class knowledge by the confidence of the teacher model on the target class, and suppress the non-target class knowledge transfer which limits the effectiveness of logits distillation (Zhao et al., 2022). Therefore, we propose simultaneous decoupled knowledge distillation which decouples the non-target class knowledge from target class knowledge to allow more sufficient knowledge distillation than the classical KD.
Let pTi and p S i be the probabilities of MT and ST for the i-th subword in the vocabulary V , respectively. The classical KD loss can be formulated as:
LKD = |V |∑
i=1
pTi log ( pTi pSi ) (5)
We use pt to denote the probability of the target subword. Correspondingly, the sum of the probabilities of the remaining non-target subwords is p\t = (1 − pt). Meanwhile, let p̂i be the probability of modeling on non-target subwords (i.e., without considering the target class), which can be calculated as:
p̂i = exp(zi)∑|V |
j=1,j ̸=t exp(zj) (i ̸= t) (6)
where z is the logit. Since p̂i is independent of the target subword probability pt, we assume that it represents the non-target class knowledge in prediction logits. Now, according to the above definitions,
we can reformulate Eq. (5) as:
LKD = pTt log ( pTt pSt ) + pT\t log ( pT\t pS\t )
︸ ︷︷ ︸ TCK
+ (1− pTt ) |V |∑
i=1,i ̸=t p̂Ti log ( p̂Ti p̂Si ) ︸ ︷︷ ︸ NCK
(7)
The details of the reformulation can be found in Appendix A. Obviously, the non-target class knowledge (NCK) couples with the target class knowledge (TCK) with a coupling weight (1−pTt ). Thus larger prediction scores pTt of the teacher model would lead to smaller coupling weights of NCK, which significantly suppresses the transfer of NCK. However, such suppression is not desirable since the more confident the teacher is, the more reliable and valuable knowledge it can provide, and since the contributions of NCK and TCK are from different aspects that should be considered separately (Zhao et al., 2022). Therefore, we replace (1− pTt ) with a hyper-parameter β to decouple the TCK and NCK to control the importance of the two types of knowledge, separately, and the training objective of SDKD is calculated as follows1:
LSDKD = TCK + βNCK (8)

heading: 3.4 Training and Inference
text: We train our model in a pretraining-then-finetuning manner. We first pre-train the text encoder, shared encoder and decoder with MT data. Then during the fine-tuning phase, we jointly train ST, MT, CCRD and SDKD with ST data. The overall training objective is the combination of the four task losses:
L = LST + LMT + LCCRD + LSDKD (9)
Note that we do not freeze MT parameters (i.e., we still enable the gradient propagation of MT) when distilling knowledge from representations and prediction logits. This is because we find that contiguously training MT parameters benefits ST performance in our experiments (see Appendix B).
During inference, we remove the text encoder and use the remaining modules of CKDST for speech translation.
1We regard this as decoupling because the knowledge transfer of NCK is not controlled (i.e., weighted) by the probability of the target class any more. Instead, it is controlled by β.

heading: 4 Experiments
text: We compared with state-of-the-art E2E/cascaded ST models to examine the effectiveness of the proposed CKDST.

heading: 4.1 Datasets
text: ST Dataset We conducted experiments on the MuST-C2 (Di Gangi et al., 2019a) benchmark dataset in four translation directions: English-German (En-De), English-Spanish (EnEs), English-French (En-Fr) and English-Russian (En-Ru). Each direction have around 400 hours speech. dev was used to develop and analyze our approaches, tst-common was used for testing. External MT Data We followed previous works (Tang et al., 2021; Ye et al., 2022) to use WMT3 datasets of different years as external MT data: WMT 2016 for English-German and EnglishRussian, WMT 2014 for English-French and WMT 2013 for English-Spanish.
The statistics of MuST-C and WMT datasets are shown in Table 1.

heading: 4.2 Settings
text: Pre-processing We used 16-bit 16 kHz monochannel audio wave as speech input. And we removed utterances of which the duration is longer than 30s. For text inputs, we extracted 10K unigram subwords with a shared source and target vocabulary via SentencePiece4 (Kudo and Richardson, 2018). Model Configuration We used the base version of Wav2vec 2.05 in the speech encoder, which is pretrained on audio data from LibriSpeech (Panayotov et al., 2015) without finetuning. Two layers of CNNs were stacked over Wav2vec 2.0 , where the kernel size was set to 5, stride size to 2 and hidden
2https://ict.fbk.eu/must-c/ 3https://statmt.org/ 4https://github.com/google/
sentencepiece 5https://dl.fbaipublicfiles.com/ fairseq/wav2vec/wav2vec_small.pt
size to 512. The shared encoder and decoder were configured with the base Transformer setting: 6 layers, 512 as hidden size, 8 attention heads, and 2048 as FFN hidden size. Implementation Details We implemented our model based on fairseq toolkit.6 For experiments with external data, we used external MT data for pre-training. For those without any external data, only the MT data from the ST triplet data were considered. For fine-tuning, we used the same hyperparameters for experiments with/without external MT data. Particularly, we used the Adam optimizer wiht 25K warm-up updates. The learning rate was 1e-4. The maximal number of tokens was 0.8M per batch. Both the dropout and the value of label smoothing were set to 0.1. We set the update frequency to 2. The temperature τ was 0.1 and the non-target class knowledge weight β was 4.0. We set the maximal number of updates to 200000, and used the early-stop training strategy if the performance did not improve for 10 consecutive validation runs. We trained all the models on 4 Nvidia TeslaV100 GPUs.
During inference, we averaged the checkpoints of the last 10 epochs for evaluation. We used beam
6https://github.com/facebookresearch/ fairseq
search with a beam size of 10 and length penalty was 1.0. We evaluated case-sensitive detokenized BLEU and ChrF++ by sacreBLEU7 (Post, 2018). Additionally, we also evaluated translation quality with COMET (Rei et al., 2020), which leverages pretrained language models to achieve high correlations with human quality judgments. Specifically, we used COMET-22 (wmt22-COMET-da)8. Baselines We compared our CKDST with multiple strong E2E ST baselines including: (1) Fairseq ST (Wang et al., 2020a) and (2) Espnet ST (Inaguma et al., 2020) trained only with the ST task data, (3) W-Transf (Ye et al., 2021) that uses a pretrained speech model to extract speech features, (4) XSTNet (Ye et al., 2021) that trains the ST model based on W-Transf in a multitask learning framework, (5) Chimera (Han et al., 2021) that learns a shared memory space to align speech and text, (6) STEMM (Fang et al., 2022) that mixes speech and text representations and (7) ConST (Ye et al., 2022) that applies contrastive learning to bridge the modality gap between speech and text, (8) JT-S-MT (Tang et al., 2021) that employs an online-KD method to transfer knowledge from MT
7sacreBLEU signature: BLEU+case.mixed+lang.ende+numrefs.1+smooth.exp+tok.13a+version.1.5.1
8https://huggingface.co/Unbabel/ wmt22-comet-da
to ST and (9) SATE (Xu et al., 2021) that leverages an adapter to incorporate pre-trained ASR and MT models into E2E ST, and uses classical KD for knowledge transfer. In addition to these baselines, we implemented a strong baseline "MTL baseline" that uses the same neural architecture (excluding the proposed CCRD and SDKD) as our model to jointly train ST and MT.

heading: 4.3 Main Results
text: Comparison to End-to-End Baselines. We compared our model with several strong baselines for four language pairs on the MuST-C benchmark dataset. Results are shown in Table 2. Without the external MT data, our model achieves a substantial improvement of 1.2 BLEU over the MTL baseline on average and outperforms the strongest baseline, ConST, in all translation directions. When we use the external MT data, we achieve new state-of-theart results in terms of the average BLEU score over the four translation directions and gain a 1.2 BLEU improvement over the MTL baseline. These results demonstrate that our approaches are able to effectively improve ST with knowledge distillation. Compared to previous works that explore knowledge distillation for E2E ST, we outperform JT-SMT (Tang et al., 2021) and SATE (Xu et al., 2021) by 1.4 BLEU and 0.4 BLEU on average, respectively. This suggests that our proposed knowledge distillation approaches are more effective than previous KD methods used in E2E ST. To better eval-
uate our approach, we used ChrF++ and COMET, which are more relevant to human evaluation, to assess our model. As shown in Table 3, our model achieves an average improvement of 1.08 ChrF++ and 1.03 COMET compared to the MLT baseline model. Comparison to Cascaded Baselines. We also compared our end-to-end model with cascade baselines. Espnet (Inaguma et al., 2020) and the cascaded ST system presented by Xu et al. (2021) are two strong cascaded systems trained with MuSTC and external ASR and MT data (LibriSpeech, WMT, and Opensubtitles). We implemented a strong "Cascaded baseline" using the ASR data from the ST data and the same external MT data as ours. Its ASR module is the same as our speech encoder and was trained with the CTC loss. The MT module is a standard Transformer, trained with the traditional MT loss. As shown in Table 4, our implemented Cascaded baseline is competitive to the other two cascaded baselines. Impressively, our end-to-end model outperforms all cascaded baselines in all translation directions.

heading: 4.4 Ablation Study
text: To better evaluate the contribution of our proposed knowledge distillation approaches, we progressively removed the CCRD module and the SDKD module to conduct ablation study on the MUST-C benchmark. As shown in Table 5, without CCRD, we get an average drop of 0.5 BLEU on all four translation directions. And, SDKD also contributes 0.6 BLEU on average on all translation directions. These demonstrate the effectiveness of both approaches in enhancing ST.

heading: 5 Analysis
text: Additionally, we conducted a series of in-depth analyses to further investigate how the proposed methods improve E2E ST.

heading: 5.1 Does CCRD Increase the Mutual Information?
text: The proposed CCRD distills knowledge from MT to ST by optimizing the mutual information be-
tween text and speech representations. Mutual information (MI) can be represented by the degree of overlap between two distributions. Thus, we plot the bivariate kernel density estimation (Parzen, 1962) (KDE) contour of speech and text dimreduced representations to visualize their distributions as shown in Figure 2, where t-SNE (Van der Maaten and Hinton, 2008) is used to reduce the dimension of representations into 2D. As shown in Figure 2(a), without CCRD, the overlap of the speech representation distribution and the text representation distribution is small. This shows that even with the shared encoder, the distributions of representations from the two modalities have very low MI. In contrast, when we apply CCRD, the distribution of speech representations and the distribution of text representations almost overlap. This indicates our proposed CCRD can significantly improve the MI between the two representation distributions.

heading: 5.2 Is SDKD Better than Classical KD?
text: As discussed in Section 3.3, the classical KD suppresses the knowledge of non-target classes, which limits its performance. To verify this, we conducted experiments on the MuST-C benchmark to compare the effects of SDKD and classical KD. In order to
eliminate the interference of other factors, we did not apply CCRD during training. The loss function LKD of the classical KD is estimated according to Eq. (5). During training, LKD is interpolated with the primary loss (i.e., ST loss) with weight α (Hinton et al., 2015). Therefore, the training objective for E2E ST with the classical KD is:
L = (1− α)LST + αLKD + LMT (10)
We followed Tang et al. (2021) to set α to 0.8. As shown in Table 6, SDKD outperforms the classical KD on all translation directions and achieves an average improvement of 0.4 BLEU. This indicates that separately exploring the target and non-target class knowledge is better than the coupled form.

heading: 5.3 Impact of the Non-target Class Knowledge Weight
text: For SDKD, it is important to choose an appropriate non-target class knowledge weight β. To understand the impact of β, we employed a grid search from [0, 8] to search desirable β with a stride of 2 on the MuST-C En-De dev set. Results are shown in Figure 3. The orange dashed line indicates the baseline model which uses the classical KD during training. If β = 0, it indicates that the nontarget class knowledge is ignored when distilling knowledge from prediction logits. Compared with the classical KD baseline, the model performance drops significantly if we ignore the non-target class knowledge. This suggests that the non-target class knowledge is important and useful. The curve with varying β clearly shows that the performance of the model first increases and then drops as β increases. We achieve the best BLEU score when β = 4. This indicates that appropriately increasing the importance of the non-target class knowledge is beneficial for knowledge distillation, but too large weights would undermine the performance of the model.

heading: 5.4 Impact of the Performance of the Pre-trained MT Model
text: Our proposed approaches aim to effectively distill knowledge from MT to ST, thus the pre-trained MT performance is of importance to our model. In order to study the impact of MT performance on our model, we randomly sample 1M, 2M and 3M MT data from the external MT data to pre-train the MT model so as to have different MT models with varying performance. When the size of external MT data is 0, we use the MT data from the ST triplet to pre-train the MT model. Results are shown in Figure 4. We observe that as the performance of the pre-trained MT model improves, the BLEU score of our model also keeps improving. This demonstrates that our approaches benefit from strong pre-trained MT models.

heading: 6 Conclusion
text: In this paper, we have presented CKDST, which comprehensively and effectively distills the knowledge of MT to boost the performance of E2E ST through two key approaches: CCRD and SDKD. The former leverages a contrastive objective to maximize the mutual information lower bound between speech and text representations for representation knowledge distillation. The later reformulates the classical KD loss to decouple the target class knowledge and the non-target class knowledge for more effective logits knowledge distillation. Our experiments strongly demonstrate that our approaches are able to significantly improve E2E ST and achieve new state-of-the-art results on the MUST-C benchmark dataset.
Limitations
Although the proposed CKDST distills the knowledge of MT more comprehensively and efficiently from encoder representations and prediction logits, and obtains significant improvements over previous methods, it still has limitations: (1) The batch size is not very large, limited by the memory capacity of the used hardware and the extremely long sequence length of speech inputs, which leads to a small number of negative samples used in CCRD and does not fully exploit the ability of contrastive learning. In future work, we attempt to expand the negative sample size using a mechanism like memory bank (He et al., 2020). (2) As we distill knowledge from MT to ST, the performance of the pretrained MT model has an impact on our framework.
Ethics Statement
This work presents CKDST, a knowledge distillation framework for ST to more comprehensively and effectively distill knowledge from MT to improve the performance of E2E ST. The datasets used in this study include both MuST-C and WMT. They are all public datasets and are widely used in the MT community.

heading: Acknowledgments
text: The present research was supported by the Natural Science Foundation of Xinjiang Uygur Autonomous Region (No. 2022D01D43). We would like to thank the anonymous reviewers for their insightful comments.

heading: A Reformulation Details
text: In Sec 3.3, we define the classical KD loss as follows:
LKD = |V |∑
i=1
pTi log ( pTi pSi )
= pTt log ( pTt pSt ) + |V |∑ i=1,i ̸=t pTi log ( pTi pSi
) (11)
According to the definition of p\t and p̂i in Sec 3.3, we can reformulate Eq. (11) to:
LKD =pTt log ( pTt pSt ) + |V |∑ i=1,i ̸=t pT\tp̂ T i log ( pT\tp̂ T i pS\tp̂ S i )
=pTt log ( pTt pSt )
+
|V |∑
i=1,i ̸=t pT\tp̂ T i
( log ( p̂Ti p̂Si ) + log ( pT\t pS\t ))
=pTt log ( pTt pSt ) + |V |∑ i=1,i ̸=t pT\tp̂ T i log ( p̂Ti p̂Si )
+
|V |∑
i=1,i ̸=t pT\tp̂ T i log ( pT\t pS\t )
(12) Since pT\t and p S \t are irrelevant to the class index
i, we have:
|V |∑
i=1,i̸=t
pT\tp̂ T i log ( pT\t pS\t ) = pT\t log ( pT\t pS\t ) |V |∑ i=1,i ̸=t p̂Ti
(13)
Moreover, ∑|V |
i=1,i ̸=t p̂ T i = 1, so:
|V |∑
i=1,i ̸=t pT\tp̂ T i log ( pT\t pS\t ) = pT\t log ( pT\t pS\t ) (14)
Bringing Eq. (14) back to Eq. (12), we have:
LKD =pTt log ( pTt pSt ) + pT\t log ( pT\t pS\t )
+pT\t
|V |∑
i=1,i ̸=t p̂Ti log ( p̂Ti p̂Si )
=pTt log ( pTt pSt ) + pT\t log ( pT\t pS\t ) +(1− pTt ) |V |∑
i=1,i ̸=t p̂Ti log ( p̂Ti p̂Si )
(15)

heading: B Whether to freeze MT Parameters
text: Knowledge distillation usually freezes the teacher model (i.e., the gradient propagation of the teacher model is disable). We assume that this is because the teacher model is not supervised by the primary
loss and freezing the teacher model prevents it from being degraded by the student model. Howerver, our model is trained on ST and MT, simultaneously. The teacher knowledge can be preserved by the auxiliary MT task. Moreover, we assume that not freezing teacher knowledge during knowledge distillation can make it more student-friendly. To investigate this, we conducted experiments on MustC En-De. Results are shown in Table 7. When we freeze MT in CCRD, we find the performance drops 0.3 BLEU. In SDKD, there is no difference in the performance of freezing MT or not. In general, not freezing MT when performing knowledge distillation is more suitable for our model.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: limitations

heading: 3 A2. Did you discuss any potential risks of your work?
text: limitations

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Left blank.
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? 2
3 B1. Did you cite the creators of artifacts you used? 3

heading: 3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
text: Appendix D

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? 3.2

heading: 3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any
text: information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? 3.2

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? Appendix C
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Appendix C
C 7 Did you run computational experiments? Left blank.
C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? No response.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? No response.
C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? No response.
C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? No response.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: In future work, we attempt to expand the negative sample size using a mechanism like memory bank (He et al. , 2020). (2) As we distill knowledge from MT to ST, the performance of the pretrained MT model has an impact on our framework