Pretrained Bidirectional Distillation for Machine Translation
Year: 2023

Abstract:
Knowledge transfer can boost neural machine translation (NMT), for example, by finetuning a pretrained masked language model (LM). However, it may suffer from the forgetting problem and the structural inconsistency between pretrained LMs and NMT models. Knowledge distillation (KD) may be a potential solution to alleviate these issues, but few studies have investigated language knowledge transfer from pretrained language models to NMT models through KD. In this paper, we propose Pretrained Bidirectional Distillation (PBD) for NMT, which aims to efficiently transfer bidirectional language knowledge from masked language pretraining to NMT models. Its advantages are reflected in efficiency and effectiveness through a globally defined and bidirectional context-aware distillation objective. Bidirectional language knowledge of the entire sequence is transferred to an NMT model concurrently during translation training. Specifically, we propose self-distilled masked language pretraining to obtain the PBD objective. We also design PBD losses to efficiently distill the language knowledge, in the form of token probabilities, to the encoder and decoder of an NMT model using the PBD objective. Extensive experiments reveal that pretrained bidirectional distillation can significantly improve machine translation performance and achieve competitive or even better results than previous pretrain-finetune or unified multilingual translation methods in supervised, unsupervised, and zero-shot scenarios. Empirically, it is concluded that pretrained bidirectional distillation is an effective and efficient method for transferring language knowledge from pretrained language models to NMT models.

heading: 1 Introduction
text: Initializing parameters by a pretrained masked language model (LM) (Kenton and Toutanova, 2019) is a knowledge transfer method widely applied to natural language processing tasks. Following
its success, pretrained neural machine translation (NMT) models have attracted more and more research interest (Conneau and Lample, 2019; Song et al., 2019; Liu et al., 2020; Li et al., 2022).
However, the pretrain-finetune paradigm may suffer from potential issues. As is pointed out in He et al. (2021), the finetuned model may forget some critical language generation skills learned from the pretraining phase. The catastrophic forgetting problem (Kirkpatrick et al., 2017; McCloskey and Cohen, 1989) commonly exists in transfer learning, leading to overfitting to target domains. Hu et al. (2022); Fang et al. (2022) also observe similar forgetting problems in pretrained NMT tasks. Besides, in the pretrain-finetune paradigm, model parameters are initialized by a pretrained model; this requires structure consistency (e.g., exact dimensions, layers, attention heads, etc.) between the pretrained LM and the NMT models to some extent. However, a powerful but structurally inconsistent pretrained LM may incorporate more language knowledge.
Knowledge distillation (KD) (Hinton et al., 2015) may be a potential solution to alleviate these issues, but few studies investigate language knowledge transfer from pretrained language models to NMT models by KD. Previous works use KD for model compression (Gordon and Duh, 2020), or data complexity reduction (Gu and Kong, 2021; Zhou et al., 2019), or multilingual translation (Sun et al., 2020; Tan et al., 2019). Zhou et al. (2022) utilizes confidence-based knowledge distillation to incorporate bidirectional global context into NMT models.
In this paper, we propose Pretrained Bidirectional Distillation (PBD) for NMT, which can alleviate the difference caused by pretraining (mask language modeling, perturbed sentences) and MT fine-tuning (full sentences) in the pretrain-finetune paradigm and boost large-scale translation training. In pretrained bidirectional distillation, language knowledge acquired from pretraining is continu-
1132
ously transferred to the NMT model. Knowledge transfer runs through the training process to address the forgetting problem. We deal with the pretrained language knowledge by pretrained bidirectional distillation objectives, which are the token probabilities generated by the pretrained LM about potential tokens matching a global context. The pretrained bidirectional distillation objectives are distilled to the encoder and decoder of an NMT model. Therefore, there is no need to require structure consistency between pretrained LMs and NMT models, and bidirectional distillation enriches the NMT decoder with bidirectional semantic information.
To guarantee the effectiveness and efficiency of pretrained bidirectional distillation, we propose self-distilled masked language pretraining, which can generate globally defined and bidirectional context aware token probabilities and use them as the pretrained bidirectional distillation objectives. “Globally defined” lets us obtain the full probabilities of each token in a single forward pass, guaranteeing distillation effect and execution efficiency. “Bidirectional context aware” distillation objectives incorporate bidirectional language knowledge of the whole sequence, guaranteeing effectiveness.
Extensive experiments are conducted on widely used benchmark datasets. In a supervised scenario, the proposed method achieves +2.7 and +8.5 absolute average BLEU improvement using the unified multilingual translation model and pretrainfinetune paradigm, respectively. And our model obtains 19.28 and 16.55 average BLEU in unsupervised and zero-shot scenarios, respectively, outper-
Algorithm 1 Pretrained Bidirectional Distillation for NMT Require: language model LM , NMT model TM ,
unlabeled LM data DLM , parallel data DTM 1: Initialize LM by random 2: for each X ∈ DLM do 3: Get loss L ← λLΩ + LΘ ▷ Equ 1,4 4: Update LM ← BACKPROP(L, LM) 5: end for 6: Initialize TM by random or pretraining 7: for each (X,Y ) ∈ DTM do 8: Get translation loss Lce ← TM(X,Y ) 9: Forward pass PΩ ← LM({X,Y }) 10: Get loss L ← Lce + Le + Ld ▷ Equ 8,10 11: Update TM ← BACKPROP(L, TM) 12: end for 13: return TM
forming previous models. To summarize, our contributions are as follows:
• We propose pretrained bidirectional distillation to investigate language knowledge transfer from pretrained language models to NMT models.
• We propose self-distilled masked language pretraining to support concurrently computing full token probabilities of the full sequence.
• We conduct extensive experiments to verify the effectiveness of our methods and achieve competitive or even better performance than previous pretrain-finetune or unified multilingual translation methods in supervised, unsupervised, and zero-shot scenarios.

heading: 2 Pretrained Bidirectional Distillation
text: Figure 1 and Algorithm 1 illustrate the overall flow of the proposed Pretrained Bidirectional Distillation (PBD) for machine translation. It consists of two processes: (1) Self-distilled masked language pretraining takes unlabeled LM training data as input and optimizes a token reconstruction loss and a self-distillation loss. The produced self-distilled LM has the advantage of generating the full probability prediction of all input tokens in one pass rather than only the masked tokens as in previous masked LMs. This ensures the efficiency of pretrained bidirectional distillation in the second process. (2) Translation training with PBD losses
unEncontró trabajobuen He[SEP] afound jobgood [MASK][SEP] [MASK][MASK]
ESES ESES ENES ENEN ENEN ESEN ENEN
10 32 54 76 98 310 87
Token Embeddings
Language Type Embeddings
Position Embeddings
Transformer Encoder (with contextual mask matrix)
Pcontext Pcontext PcontextPtarget Ptarget Pmask
h1h0 h3h2 h5h4 h7h6 h9h8 h3'h10 h8'h7' Hidden States
Prediction Head Ω Prediction Head Θ
𝑃𝛺(𝑤0| ሚ𝑆) 𝑃Θ(𝑤3| ሚ𝑆) 𝑃Θ(𝑤7| ሚ𝑆) 𝑃Θ(𝑤8| ሚ𝑆)𝑃𝛺(𝑤2| ሚ𝑆) 𝑃𝛺(𝑤3| ሚ𝑆) 𝑃𝛺(𝑤4| ሚ𝑆) 𝑃𝛺(𝑤5| ሚ𝑆) 𝑃𝛺(𝑤6| ሚ𝑆) 𝑃𝛺(𝑤7| ሚ𝑆) 𝑃𝛺(𝑤8| ሚ𝑆) 𝑃𝛺(𝑤9| ሚ𝑆) 𝑃𝛺(𝑤10| ሚ𝑆)𝑃𝛺(𝑤1| ሚ𝑆)
ℒ = 𝜆ℒΩ + ℒΘLoss Function
Figure 2: Overall architecture of the self-distilled masked language model. The input can be a pair of parallel sentences or a monolingual sequence; a parallel example is shown here. Pcontext, Ptarget, and Pmask denote the context, target, and masked part, respectively. S̃ = {wt|wt ∈ Pcontext} is the context.
trains a standard Encoder-Decoder NMT model using parallel data but enhances it with extra PBD losses. The PBD losses are jointly optimized with the standard translation loss, and pretrained language knowledge in the form of full token probabilities generated by the pretrained LM is distilled to the encoder and decoder of the NMT model. We will introduce these two processes in detail in the following sections.

heading: 2.1 Self-distilled Masked Language Pretraining
text: This paper proposes self-distilled masked language pretraining to obtain the pretrained bidirectional distillation objective for NMT models. Pretrained masked language models predict a token probability distribution over the vocabulary for each masked position, and these token probabilities indicate the potential tokens matching the context. Our assumption is that these token probabilities contain specific language knowledge and can be transferred to NMT models. Thus, we consider these token probabilities as the distillation objective.
However, in our preliminary experiments, we discovered that the token probabilities predicted in non-masked positions often tend to focus too much on real tokens, which fails to accurately reflect the long-tailed distribution of potential tokens.
In standard masked language pretraining, only a small percentage (typically 15%) of tokens can be masked. This limitation prevents us from efficiently achieving the full distillation objective that reflects the long-tailed distribution for each position of an input sequence in a single forward pass. To obtain a globally defined distillation objective, we adopt self-distillation, in which the token probabilities in non-masked positions are learned from the corresponding masked positions.
Figure 2 illustrates the overall architecture of the proposed self-distilled masked language model, which follows the widely used masked language model framework (Kenton and Toutanova, 2019; Conneau and Lample, 2019) with some modifications to its architecture: (1) The target tokens to be predicted have two types: masked tokens and real tokens. (2) The input sequence is partitioned into three parts to avoid exposing information between masked tokens and real tokens. (3) Masked and real tokens have different prediction heads and loss functions. The following subsections elaborate on the architecture of the self-distilled masked language model.

heading: 2.1.1 Input Representation
text: Let S denote an input sequence, and it may be a monolingual text S = {X} = {x1, · · · , xn} or
the concatenation of a pair of parallel sentences S = {X,Y } = {x1, · · · , xn, y1, · · · , ym}. According to the random masking scheme, the input sequence consists of non-masked positions and masked positions (typically 15%). Specifically, as is shown in Figure 2, a portion of positions (in this case, the 3rd, 7th, and 8th positions) have corresponding [MASK] tokens appended at the end of the sequence. Therefore, we split the complete input sequence into three parts: the context part Pcontext which is used as the known context; the masked part Pmask which is used to reconstruct the real tokens; and the target part Ptarget in which tokens are the real tokens corresponding the masked part, and they are pretended to be unknown when predicting token probabilities.
The corresponding position embeddings, language type embeddings, and a special [MASK] token embedding are summed to form the input representations in Pmask. And, the input representations in Ptarget and Pcontext are the sum of the corresponding position embeddings, language type embeddings, and the real token embeddings.

heading: 2.1.2 Contextual Mask Matrix
text: In the masked token reconstruction task, the real token should be kept unknown to the corresponding masked position. Besides, the hidden state at the masked position is also needed to be invisible to the corresponding target position in the forward pass because the predicted probability at the masked position is the learning objective of the corresponding target position (i.e., avoiding super-
vised information leaking). Since the backbone of the masked language model is an attention-based Transformer encoder, the visibility of tokens can be controlled by a contextual mask matrix. As is illustrated in Figure 3, the contextual mask matrix controls that each token can attend to itself and the tokens in Pcontext. It means that the context S̃ is set to S̃ = {wt|wt ∈ Pcontext} for all the three parts Pmask, Ptarget and Pcontext.

heading: 2.1.3 Pretraining Loss
text: We adopt different loss functions for the masked part and the target part. In the masked part, the language model learns to reconstruct the masked tokens. At each position of the target part, our model pretends not to have known the real token and predicts the potential tokens matching the context. Specifically, the probabilities of the potential tokens are learned to approximate the token reconstruction probabilities at the corresponding masked positions. This is because the token reconstruction probabilities are the predicted probabilities of potential tokens at the masked positions.
Let S̃ = {wi|wi ∈ Pcontext} denote the context token set, S̄ = {wi|wi ∈ Ptarget} denote the target token set, ti denote the token at position i. The masked token reconstruction task defines the pretraining objective LΘ as minimizing the negative log-likelihood of target tokens as below.
LΘ = − logPΘ(S̄|S̃) ≈ − ∑
wi∈S̄ logPΘ(ti = wi|S̃)
(1)
in which the token reconstruction probability PΘ is defined in the masked part and is computed by a prediction head Θ.
PΘ(ti = wi|S̃) = exp(hΘi T e(wi))∑
w∈V exp(h Θ i T e(w))
(2)
hΘi = gelu(h ′ i T WΘ + bΘ) (3)
where we use h ′ i to represent the hidden state of the last layer of a Transformer encoder at the masked position i, WΘ ∈ RD×D and bΘ ∈ RD are learnable parameters of the prediction head Ω, D is the dimension, e(w) ∈ RD denotes the embedding of token w, and V represents the vocabulary.
A self-distillation approach is adopted here to learn the potential tokens’ probabilities. The loss LΩ is defined by optimizing the KL divergence
between the probability distribution of token reconstruction and the probability distribution of potential tokens. It is equivalent to LΩ = − ∑
i∈Ptarget
∑ w∈V PΘ(ti = w|S̃) logPΩ(ti = w|S̃)
(4)
in which the probability of potential tokens PΩ is defined in the non-masked positions and is computed by a prediction head Ω.
PΩ(ti = w|S̃) = exp(hΩi
T e(w))
∑ w∈V exp(h Ω i T e(w)) (5)
hΩi = gelu(h T i WΩ + bΩ) (6)
where hi denotes the hidden state at the nonmasked position i.
The overall loss integrates LΩ and LΘ by weighted summation.
L = λLΩ + LΘ (7)
in which λ is a hyper-parameter.

heading: 2.1.4 Inference
text: In inference, there is no masked position for the input sequence S, and the probabilities of any potential token w at each position i can be computed as PΩ(ti = w|S). We consider these probabilities as the pretrained bidirectional distillation objective for NMT models.

heading: 2.2 Pretrained Bidirectional Distillation Loss
text: In this paper, the knowledge learned from the aforementioned self-distilled mask language model is transferred to an NMT model using the pretrained bidirectional distillation loss. Specifically, we concatenate the source and target sentence without masking to form an input sequence to the selfdistilled LM, and obtain the full probability prediction PΩ from the LM as the pretrained bidirectional distillation objective, which is distilled to a NMT model by optimizing the KL divergence between the pretrained bidirectional distillation objective PΩ and its corresponding predictions from an intermediate layer of the encoder or decoder.
The distillation loss of the encoder is as follows.
Le=− ∑
t
∑
w
PΩ(xt=w|X,Y ) logPe(xt=w|X)
(8) Pe = softmax(H l e ·ET ) (9)
Here, we use X and Y to denote the sentence in source and target language, respectively, and xt denotes the t-th position of X . w is a word in the vocabulary V . Hle ∈ R|X|×D represents the hidden states of an intermediate layer l of the encoder. E ∈ R|V |×D is the token embedding matrix. We reuse the token embedding matrix, therefore, the pretrained bidirectional distillation won’t add any extra parameters. The t-th row and w-th column of the probability matrix Pe is the value of Pe(xt = w|X).
Similar distillation loss is applied to the decoder.
Ld=− ∑
t
∑
w
PΩ(yt=w|X,Y) logPd(yt=w|X,Y<t)
(10) Pd = softmax(H l d ·ET ) (11)
where yt denotes the t-th position of the target sentence, and we use Hld to represent the hidden states of an intermediate layer l of the decoder. Note that these distillation losses are jointly optimized with the standard translation loss when the NMT training.
The pretrained bidirectional distillation objective is not only globally defined but also bidirectional context aware (i.e., bidirectional language knowledge of the complete source and target sentence). Therefore, it is a challenging task to approximate the pretrained bidirectional distillation objective for the encoder and decoder given only a source sentence or given the source and partial target sentence, but it is reasonable since the source sentence has complete semantics information. On the other hand, the challenging task may force the NMT model to learn global language knowledge from the self-distilled LM. It can enrich the NMT decoder with bidirectional semantic information, as using future information is important for machine translation.

heading: 3 Experiments
text: We primarily study the proposed pretrained bidirectional distillation by conducting experiments on supervised, unsupervised, and zero-shot multilingual machine translation scenarios.

heading: 3.1 Experimental Setup

heading: 3.1.1 Language Model Pretraining
text: Datasets We use the parallel dataset PC32 (Lin et al., 2020) and the monolingual dataset MC24 provided by Pan et al. (2021). PC32 contains 32
English-centric language pairs1, and MC24 consists of monolingual text in 24 languages2. We follow the original data preprocessing, data sampling, tokenization, and vocabulary by directly downloading the datasets3 released by Pan et al. (2021), thus we can have a relatively fair comparison to our primary baselines, such as mRASP (Lin et al., 2020), mRASP2 (Pan et al., 2021) and CeMAT (Li et al., 2022). When pretraining, the source and target sentences are concatenated, and substituted synonyms are not masked. The masking ratio is 20%.
Settings We adopt a 12-layer Transformer-based language model with 768 dimensions and 12 attention heads. The language model is trained on 8 Nvidia A100 GPUs for 1M steps using Adam optimizer. On each GPU, the number of tokens in each batch is at most 32K. The learning rate is set to 0.0001, and polynomial decay scheduling is used with a warm-up step of 10000. The hyperparameter λ in Equ 7 is 0.5, and the dropout rate is set to 0.1. See appendix for more details.
1En, Af, Ar, Be, Bg, Cs, De, El, Eo, Es, Et, Fi, Fr, Gu, He, Hi, It, Ja, Ka, Kk, Ko, Lt, Lv, Mn, Ms, Mt, My, Ro, Ru, Sr, Tr, Vi, Zh
2Bg, Cs, De, El, En, Es, Et, Fi, Fr, Gu, Hi, It, Ja, Kk, Lt, Lv, Ro, Ru, Sr, Tr, Zh, Nl, Pl, Pt
3https://github.com/PANXiao1994/mRASP2

heading: 3.1.2 Machine Translation Training
text: Datasets For training multilingual translation models, we reuse the parallel dataset PC32 and monolingual dataset MC24, consistent with Pan et al. (2021). We follow the experimental settings in CeMAT (Li et al., 2022) for finetuning experiments. Language pairs of various data sizes from WMT are used for finetuning, and the dataset information is shown in Table 2. For evaluating unified multilingual models, we use the evaluation datasets from WMT, IWSLT, and OPUS-100 (Zhang et al., 2020) following mRASP2 (Pan et al., 2021).
Settings We follow the model configurations used in CeMAT (Li et al., 2022) to train a Transformer-big (Vaswani et al., 2017) size NMT model, which will compare with models using the pretrain-finetune paradigm. And for a fair comparison, a larger NMT model with 12 encoder layers and 12 decoder layers is trained to compare with unified multilingual models. The contrastive loss is used in training a unified multilingual model due to its importance to zero-shot translation (Pan et al., 2021). Other training hyper-parameters are referred to from the open-source implementation of mRASP2. For pretrained bidirectional distillation losses, the intermediate layer to be distilled
is set to the antepenultimate layer of the encoder and decoder. Note that global distillation doesn’t introduce extra parameters, and our model has the same size as the major baselines.

heading: 3.2 Supervised Translation
text: We trained a unified multilingual NMT model with pretrained bidirectional distillation. As is shown in Table 1, our proposed PBD-MT clearly outperforms previously published approaches and achieves new state-of-the-art performances in most translation directions. It achieves +0.76 average BLEU improvement over mRASP2, which validates the effectiveness of the proposed pretrained bidirectional distillation.
In addition, we investigate the effect of pretrained bidirectional distillation on the pretrainfinetune paradigm. Specifically, we adopt PBD losses on the encoder and decoder when finetuning. As we can see in Table 2, PBD-MT achieves better or competitive performance compared to previous pretrain-finetune models. It is noteworthy that no matter the unified model or the pretrain-finetune model, the improvement in X→En directions is more significant than that of En→X directions. We conjecture that English sentences are much more than other languages, thus the pretrained LM has a better understanding of English language.

heading: 3.3 Unsupervised and Zero-Shot Translation
text: Table 3 summarizes the performance of unified multilingual models on a zero-shot translation scenario. Although the training data only consists of Englishcentric parallel sentences, multilingual NMT models show promising performance on zero-shot translation. Compared with mRASP2, PBD-MT further boosts the translation quality in most zero-shot di-
rections, achieving a +1.24 average gain. Besides, we evaluate the unified multilingual models in unsupervised translation directions, and the results are shown in Table 4. For PBD-MT, positive results are observed in all translation directions but one direction, and the average BLEU score increases by a +0.73 point. These results validate the positive effects of the proposed pretrained bidirectional distillation not only on supervised scenario but also zero-shot and unsupervised scenarios.

heading: 3.4 Non-autoregressive NMT
text: This section contains additional results for nonautoregressive translation (NAT) experiments. Specifically, we use a Transformer-big size fully NAT (Gu and Kong, 2021) as the base model. The model is initialized by a pretrained multilingual PBD-MT model and trained using a CTC loss as in Gu and Kong (2021). Because the decoder in the NAT model has upsampled length, for simplicity, we only adopt the encoder PBD loss when NAT training. Table 5 shows the performance of our model and other pretrained NAT models. Consistent BLEU gains are obtained by our PBD-NAT, validating its effectiveness.

heading: 3.5 Model Analysis

heading: 3.5.1 Ablation Study
text: In order to evaluate the individual contribution of model components, we conduct an ablation study. We train a self-distilled LM and Transformer-base (Vaswani et al., 2017) size bilingual NMT models on the WMT14 English-German dataset, and report the results in Table 6. Compared with the standard bilingual Transformer and confidence-based KD (Zhou et al., 2022), PBD-MT significantly improves the performance, which verifies the effectiveness of pretrained bidirectional distillation on bilingual NMT. Without the PBD loss on the encoder or decoder, the BLEU scores degrade to some extent, and the decoder PBD loss has more impact than the encoder PBD loss. The results prove the necessity of both pretrained bidirectional distillation losses.

heading: 3.5.2 Quantitative Analysis
text: To investigate the contribution of self-distillation on LM which generates globally defined distillation objectives in a single forward pass, a quantitative analysis is conducted here. Figure 4 illustrates the results. For execution efficiency, we compare marginalizing over multiple masks with the self-
distillation on LM. For example, masking 10% tokens each time results in 10 LM forward passes to generate the full distillation objectives. As we can see, the design of self-distilled LM significantly accelerates the execution speed than multiple masks. For the distillation effect, we compare distillation on partial tokens with global distillation. The red lines show that 20% is a relatively reasonable proportion for partial distillation, and as the mask ratio increases, the performance degrades. Masking too many tokens increases the uncertainty for the LM. The best performance is achieved by global distillation, verifying the superiority of globally defined distillation objectives.

heading: 3.5.3 Visualization
text: We conduct a behavior analysis to understand which tokens are considered more certain in contexts by the self-distilled language model. In this experiment, instead of softmax, we use sigmoid to compute a scalar probability in the prediction head Ω. Figure 5 visualizes the predicted self-distilled token probabilities on randomly sampled sentences. In this experiment, no token is masked; thus, the token probabilities represent the tokens’ matching degree and certainty in the complete bidirectional context. As we can see, verbs, articles, conjunctions, and prepositions are roughly of higher probabilities, while nouns, adverbs, and adjectives are harder to be predicted. It can be concluded that the syntactic structure is more regular, and meaningful words are more changeable.

heading: 4 Related Works

heading: 4.1 Masked Language Pretraining
text: Kenton and Toutanova (2019) propose BERT, a pre-trained masked language model (MLM), which succeeds in capturing the syntactic and semantic
meaning of contextualized texts by large-scale selfsupervised pretraining. Recent researches explore and strengthen BERT. XLNet (Yang et al., 2019) addresses the issue of pretrain-finetune discrepancy simultaneously considering bidirectional contexts by a permutation language modeling objective. RoBERTa (Liu et al., 2019) exhaustively explores the pretraining setup, such as data processing, training task, hyper-parameters, etc., to boost the model. ELECTRA (Clark et al., 2019) trains a discriminator to detect replaced tokens, which are substituted by an MLM generator, and improve the model’s efficiency. Due to space limitations, we can not elaborate on BERT variants. Sun et al. (2022); Naseem et al. (2021); Min et al. (2021) surveyed the pre-trained language models.

heading: 4.2 Pretrained Machine Translation
text: As far as pretrained machine translation is concerned, a lot of powerful deep learning approaches have been introduced. For instance, XLM (Conneau and Lample, 2019) introduces the crosslingual language model pretraining and get significant improvements on unsupervised and supervised NMT. MASS (Song et al., 2019) adopts the encoder-decoder framework to reconstruct a sentence fragment. mBART (Liu et al., 2020) can be directly finetuned by pretraining a complete model. mRASP (Lin et al., 2020) and mRASP2 (Pan et al., 2021) improve NMT by using code-switching strategy and contrastive learning. CeMAT (Li et al., 2022) utilizes a bidirectional decoder to improve the representation capability.

heading: 4.3 Language Knowledge Distillation
text: Knowledge distillation is an effective technique for model compression and was first proposed by Hinton et al. (2015), in which knowledge is transferred
from a teacher model to a student model. Sanh et al. (2019) distill a BERT-base model (Kenton and Toutanova, 2019) into smaller models by defining loss on the pre-trained predictions, which results in a task-agnostic pretraining distillation. Turc et al. (2019) conduct exhaustive analyses about the initialization of students in a task-specific setting, they show that students initialized by pretraining are better than that initialized from a truncated teacher (Sun et al., 2019; Sanh et al., 2019). Jiao et al. (2020); Wang et al. (2020, 2021); Choi et al. (2022) make assumptions about the student and teacher architectures and investigate aligning layer representations as well as attention matrices. Zhou et al. (2022) utilizes confidence-based knowledge distillation to incorporate bidirectional global context into NMT models.

heading: 5 Conclusion
text: In this paper, we proposed the pretrained bidirectional distillation to investigate language knowledge transfer from pretrained language models to NMT models by knowledge distillation. The proposed approach has the advantages of distillation effectiveness and efficiency, and achieves new stateof-the-art performance in supervised, unsupervised, and zero-shot multilingual translation experiments. The model analysis also shows that the proposed self-distilled language model is critical to generating globally defined distillation objectives. In the future, we will do more research on optimizing the self-distilled language model and pretrained bidirectional distillation losses.
Limitations
The pretrained bidirectional distillation transfers language knowledge through the NMT training process, a limitation of this method is that a computational overhead is introduced during training. Specifically, there is an extra language model forward pass to generate the pretrained bidirectional distillation objectives. Although we significantly reduce the computational overhead by designing a self-distilled language model, the overhead cannot be completely avoided. Fortunately, most computations stem from back-propagation when model training, and the introduced computational overhead only affects training time. Once the training is completed, the NMT has an identical inference cost as regular translation models.

heading: A LM Pretraining Details
text: We follow consistent pretraining configurations for bilingual and multilingual language models. Table 7 lists detailed hyper-parameters we used in pretraining.

heading: B NMT Training Details
text: Table 8 lists detailed hyper-parameters we used in NMT model training.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Limitations Section.
A2. Did you discuss any potential risks of your work? Not applicable. Left blank.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: At the end of the Introduction Section.

heading: 3 A4. Have you used AI writing assistants when working on this paper?
text: Grammarly. Spell checking.
B 3 Did you use or create scientific artifacts? Experimental Setup Section. We use publicly available datasets and code bases.

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Experimental Setup Section
7 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? It is free to use the data and code for research purposes, so we don’t mention it explicitly.

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Experimental Setup Section
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? Experimental Setup Section
7 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. In the Experimental Setup Section, we mention that we follow the original data preprocessing, data sampling, tokenization, and vocabulary by directly downloading the datasets released by previous papers. Thus, we give the reference and don’t repeat this information.
C 3 Did you run computational experiments? Experiments Section.

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Experimental setup Section, Appendix A, and Appendix B.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Experimental setup Section, Appendix A, and Appendix B.

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? From Section 3.2 to 3.4.2.

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Experimental setup Section, Appendix A, and Appendix B.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: In the future, we will do more research on optimizing the self-distilled language model and pretrained bidirectional distillation losses. Limitations
The pretrained bidirectional distillation transfers language knowledge through the NMT training process, a limitation of this method is that a computational overhead is introduced during training. Specifically, there is an extra language model forward pass to generate the pretrained bidirectional distillation objectives. Although we significantly reduce the computational overhead by designing a self-distilled language model, the overhead cannot be completely avoided. Fortunately, most computations stem from back-propagation when model training, and the introduced computational overhead only affects training time. Once the training is completed, the NMT has an identical inference cost as regular translation models