Venue: NAACL 2019

# Online Distilling from Checkpoints for Neural Machine Translation

Hao-Ran Wei, Shujian Huang ; Ran Wang, Xin-yu Dai and Jiajun Chen

State Key Laboratory for Novel Software Technology

Nanjing University

Nanjing 210023, China

{weihr, huangsj, wangr, daixy, chenjj}@nlp.nju.edu.cn

Corresponding Author.

###### Abstract

Current predominant neural machine translation (NMT) models often have a deep structure with large amounts of parameters, making these models hard to train and easily suffering from over-fitting. A common practice is to utilize a validation set to evaluate the training process and select the best checkpoint. Average and ensemble techniques on checkpoints can lead to further performance improvement. However, as these methods do not affect the training process, the system performance is restricted to the checkpoints generated in the original training procedure. In contrast, we propose an online knowledge distillation method. Our method on-the-fly generates a teacher model from checkpoints, guiding the training process to obtain better performance. Experiments on several datasets and language pairs show steady improvement over a strong self-attention-based baseline system. We also provide analysis on data-limited setting against over-fitting. Furthermore, our method leads to an improvement on a machine reading experiment as well.

## 1 Introduction

Neural Machine Translation (NMT) Cho et al. (2014); Sutskever et al. (2014) has been rapidly developed during the past several years. For further performance improvement, deeper and more expressive structures Johnson et al. (2017); Barone et al. (2017); Gehring et al. (2017); Vaswani et al. (2017) have been exploited. However, all of these models have more than hundreds of millions of parameters, which makes the training process more challenging. During the training of NMT models, we notice the following two problematic phenomena: First, the training process is unstable. This is evidenced by the decreasing of training loss with fluctuate performance on the validation set. Second, the performance on validation set usually begins to worsen after several epochs, while the training loss keeps decreasing, which suggests the model being at risk of over-fitting.

In order to alleviate these issues, the common practice is to periodically evaluate models on a held-out set (with each evaluated model saved as a _checkpoint_). Training is terminated when \(m\) consecutive checkpoints show no improvement and select the checkpoint with best evaluation score as the final model. Further improvement can be achieved by utilizing more checkpoints, by _smoothing_, which averages these checkpoints' parameters to generate more desirable parameters Sennrich et al. (2016a); or by _ensemble_, which averages these checkpoints' output probabilities at every step during inference Chen et al. (2017).

However, we notice that all of these methods have a limitation. Once the training process gets parameters with poor performance, selecting, smoothing or ensemble from the checkpoints in this process may have limited generalization performance as well. We impute the limitation to the "offline" property of these methods. In other words, only employing checkpoints after training cannot affect the original training process.

In this paper, we propose to utilize checkpoints to lead the training process. Our method is carried out in a _knowledge distillation_ manner. At each training step, because being evaluated on the held-out validation data, the best checkpoint up to the current training step can be seen as a model with the best generalization ability so far. Therefore, we employ this checkpoint as the teacher model, and let the current training model, as the student, learn from the output probability distributions of the teacher model, as well as truth translations in the training data. Such kind of knowledge distillation is performed on-the-fly because the teacher

model could always be updated once any latest better checkpoint is generated. We call our method **O**nline **D**istillation from Checkpoints (ODC).

We conduct experiments on four translation tasks (including two low-resource tasks), and one machine reading comprehension task. All the results demonstrate that our ODC method can achieve improvement upon strong baseline systems. ODC also outperforms checkpoint smoothing and ensemble methods, without extra cost during inference. We can achieve further improvement by combining ODC with those methods.

Major contributions of our work include:

1. In contrast to checkpoint smoothing and ensemble which do not affect the training process, we explore the way to distill knowledge from checkpoints to lead the training process in an on-the-fly manner(SS3.1, SS3.2). We obtain better performance by replacing the best checkpoint with moving average parameters at that step. (SS3.3)
2. We conduct experiments on four translation tasks, including two low resource tasks. In all the tasks our method outperforms strong baseline systems (SS4.2, SS4.3). We also conduct an experiment on machine reading comprehension task and the result shows that our method can be applied to other tasks too (SS4.4).
3. We conduct comprehensive analysis and show that our method can significantly alleviate over-fitting issue in low-resource condition (SS5.1), and help to find a wider minimum which brings better generation (SS5.2).

## 2 Background

### Neural Machine Translation

Neural Machine Translation (NMT) systems learn a conditional probability \(P(Y|X)\) for translating a source sentence \(X=(x_{1},...,x_{M})\) to a target sentence \(Y=(y_{1},...,y_{N})\), in which \(x_{i}\) and \(y_{j}\) are the \(i\)-th word and \(j\)-th word in sentence \(X\) and \(Y\), respectively. An NMT model usually consists of an encoder (parameterized by \(\theta_{\text{enc}}\)) and a decoder (parameterized by \(\theta_{\text{dec}}\)). The encoder transforms a sequence of source tokens into a sequence of hidden states:

\[H(X)=(h_{1},...,h_{M})=f_{\text{enc}}(X;\theta_{\text{enc}}). \tag{1}\]

The decoder of NMT is usually a network computing the conditional probability of each target words \(y_{j}\) based on its previous words and the source sentence:

\[p(y_{j}|y_{<j},X)\propto\exp(f_{\text{dec}}(y_{<j},s_{j},H(X);\theta_{\text{ dec}})), \tag{2}\]

where \(s_{j}\) is the hidden state of decoder at time step \(j\), \(p\) is the distribution of NMT model and \(\theta\) is all the parameters of NMT model.

The standard way to train an NMT model is to minimize the cross-entropy between the one-hot distribution of the target sentence and the NMT model's output distribution:

\[\mathcal{L}(\theta)= -\sum_{j=1}^{N}\sum_{k=1}^{|\mathcal{V}|}\mathbbm{1}\{y_{j}=k\} \tag{3}\] \[\cdot\log p(y_{j}=k|y_{<j},X;\theta),\] \[\theta^{*}= \operatorname*{arg\,min}_{\theta}\mathcal{L}(\theta), \tag{4}\]

where \(\mathbbm{1}(\cdot)\) is the indicator function and \(\mathcal{V}\) is the target vocabulary.

### Knowledge Distillation in Neural Machine Translation

Knowledge distillation is a class of methods which transfers knowledge from a pre-trained _teacher_ model \(\mathcal{T}\), to a _student_ model \(\mathcal{S}\). The teacher model can be a model with large capacity Bucila et al. (2006) or an ensemble of several models Hinton et al. (2015). In knowledge distillation, the student model learns to match the predictions of the teacher model. Concretely, assuming that we learn a classification model (parameterized by \(\theta\)) on a set of training samples in the form of \((x,y)\) with \(|\mathcal{V}|\) classes. Instead of minimizing the cross-entropy loss between one-hot label \(y\) and model's output probability \(p(y|x;\theta)\), knowledge distillation uses the teacher model's distribution \(q(\cdot|x)\) as "soft targets" and optimizes the loss:

\[\mathcal{L}_{\text{KD}}(\theta)= -\sum_{k=1}^{|\mathcal{V}|}q(y=k|x;\theta_{\mathcal{T}}) \tag{5}\] \[\log p(y=k|x;\theta),\]

where \(\theta_{\mathcal{T}}\) parameterizes the teacher model and \(p(\cdot|x)\) is the distribution of the student model.

Kim and Rush (2016) proposed that, as the loss of NMT model (Equation 4) can be factored into minimizing cross-entropy loss between the target 

words and word-level probabilities of the NMT model for every position at target side, knowledge distillation on multi-class classification can be naturally applied. They defined word-level knowledge distillation (W-KD) on a sentence as:

\[\begin{split}&\mathcal{L_{\text{W-KD}}}(\theta)=-\sum_{i=j}^{N} \sum_{k=1}^{|\mathcal{V}|}q(y_{j}=k|y_{<j},H(x);\theta_{\mathcal{T}})\\ &\cdot\log p(y_{j}=k|y_{<j},H(x);\theta),\end{split} \tag{6}\]

where \(\mathcal{V}\) is the target vocabulary.

They further proposed sequence-level knowledge distillation (S-KD), which optimizes the student model by matching the predictions of the teacher model in the probability distribution over the space of all possible target sequences:

\[\mathcal{L_{\text{S-KD}}}(\theta)=\sum_{Y\in\tau}q(Y|X;\theta_{\mathcal{T}}) \log p(Y|X;\theta), \tag{7}\]

where \(\tau\) is the space of target side sentences. As summing over exponential numbers of samples here is intractable, they proposed to train student model on samples generated by teacher model as an approximation.

## 3 Online Distillation from Checkpoints

### Online Knowledge Distillation

Traditional knowledge distillation maintains a static teacher throughout the training process, which not only requires one pre-training process to obtain the teacher model, but also limits the power of leading the training process.

In contrast, we are aiming at a more integrated process where the teacher model does not come from a separate training process, but from the current training routine itself. More specifically, we update the teacher along with the training process, so the distilled knowledge could be updated when a stronger model comes out. Figure 1 illustrates the paradigm of our method.

In generation tasks, the knowledge distillation could be performed at the word-level or sequence-level. In this paper, we focus on the word-level distillation because this distillation only needs forced teaching, which could be performed efficiently together with the training of the student model compared to generating translations from teacher model. It is more computational-friendly, especially when the NMT models are built with parallelizable convolution (Gehring et al., 2017) or self-attention structures (Vaswani et al., 2017).

### Online Distillation from Best Checkpoint

Observed from the training process of NMT models, performance on the validation set does not improve monotonically. When the performance of the training model on the validation set declines, we could always select the best checkpoint so far as the teacher, because it has the best generalization performance. Specially, when the best checkpoint

Figure 1: Illustration of online distillation from checkpoints(ODC). Darker color means better performance on validation data. In validation step \(T_{k^{\prime}}\), ODC selects the current best checkpoints as the teacher model; while in the next validation step \(T_{k^{\prime}+1}\), the training generates a better checkpoint, and use it to update the teacher model. In validation step \(T_{k^{\prime}+2}\), training model’s performance declines, and the teacher model is used to lead the training of the model. The similar situation happens during the entire training process.

 

is generated at the current time step, we only update the teacher model but perform no distillation. Figure 1 gives an illustration of this process.

The online distillation process is summarized in Algorithm 1. We use to denote the training step and to denote the parameters at time step . We denote as the time step the -th time when the model is evaluated on validation and as the validation interval, for which . Let be the time step when the best checkpoint is obtained up to , and as the teacher's parameters to lead the following training process. If the current checkpoint is the best checkpoint so far, i.e. , we update the teacher to be this new checkpoint (in Line 16 and 20). The loss for the training process at time step () () is defined as follows:

(8)

where and is defined in Equation 4 and 7, respectively (in Line 5-8).

### Integrated with Mean Teacher

Knowledge distillation usually works better when teacher models have better performance. As Tarvainen and Valpola (2017) proposed in their work, averaging model parameters over training steps tends to produce a more accurate model that using final parameters directly. They called this method as _Mean Teacher_.

Following Tarvainen and Valpola (2017), besides updating parameters, we maintain the exponential moving average (EMA) of the model parameters as:

(9)

where is the update step, is the parameters of the training model and the parameters of EMA. is the decay weight which is close to 1.0, and typically in multiple-nines range, i.e., 0.999, 0.9999. By doing so, at each timestep , parameters of NMT model has their corresponding EMA parameters .

Whenever we update teacher model with the current best checkpoint, we can use its EMA parameters instead (in Line 17-18). It can further improve the generalization ability of the teacher model, and bring a better performance of knowledge distillation. We will show in SS4.2 that using meaning teacher indeed achieves better performance.

```
1Input:validationinterval;validationcount;EMAdecayweight;initialmodelparameters0;Initialization:\(k=0\); ; 

2005, 2006 as test sets. We filter out sentence pairs whose source or target side contain more than 50 words. We use BPE Sennrich et al. (2016) with 30K merge operations on both sides.

For WMT17 Chinese-English translation task, we use the pre-processed version released by WMT4. We only use CWMT part of WMT Corpus. We use newsdev2017 as the validation set and newstest2017 s the test set. We learn a BPE model with 32K merge operations and keep all the BPE tokens in the vocabulary. We limit the maximal sentence length as 100 after BPE segmentation.

Footnote 4: [http://data.statmt.org/wmt18/translation-task/preprocessed/zh-en/](http://data.statmt.org/wmt18/translation-task/preprocessed/zh-en/)

For IWSLT15 English-Vietnamese translation task, we directly use the pre-processed data used in Luong and Manning (2015)5, which has 133K sentence pairs, with 2.70M English words and 3.31M Vietnamese words. We use the released validation and test set, which has 1553 and 1268 sentences respectively. Following the settings in Huang et al. (2017), the Vietnamese and English vocabulary size are 7,709 and 17,191, respectively.

Footnote 5: [https://github.com/efan-it/nmt-en-vi](https://github.com/efan-it/nmt-en-vi)

For WMT17 English-Thurkish translation task, We use the pre-processed data released by WMT176. It has 207K sentence pairs, with 5.21M English words and 4.63 Turkish words. We use newstest2016 as our validation set and newstst2017 as the test set. We use joint BPE segmentation Sennrich et al. (2017) to process the whole training data. The merge operations are 16K.

Footnote 6: [http://data.statmt.org/wmt17/translation-task/preprocessed/tr-en/](http://data.statmt.org/wmt17/translation-task/preprocessed/tr-en/)

Implementation DetailsWithout specific statement, we follow the _transformer_base_v1_ hyper-parameters settings 7, with 6 layers in both encoder and decoder, 512 hidden units and 8 attention heads in multi-head attention mechanism and 2048 hidden units in feed-forward layers. Parameters are optimized using AdamKingma and Ba (2014). The initial learning rate is set as 0.1 and scheduled according to the method proposed in Vaswani et al. (2017), with warm-up steps as 4000.

Footnote 7: [https://github.com/tensorflow/tensor2tensor/blob/v1.3.0/tensor2Tensor/models/transformer.py](https://github.com/tensorflow/tensor2tensor/blob/v1.3.0/tensor2Tensor/models/transformer.py)

We periodically evaluate the training model on the validation set by doing translation and compute the BLEU scores. We stop training when 50 subsequent of BLEU scores on validation set do not get improvement. We use beam search with beam size as 5.

### Evaluation on Chinese-English Translation Tasks

We first evaluate the capability of our method for improving performance when there are plenty of training data. We conduct experiments on both NIST and WMT17 Chinese-English Translation tasks.

Results on NIST DatasetWe compare our method with several ways to utilize checkpoints8:

Footnote 8: We set \(k=5\) in this experiments.

* **last-k-smoothing**: After training the baseline model, we average the parameters of the last k checkpoints as the final model.
* **best-k-smoothing**: Average the parameters of the best k checkpoints, instead of the last k,

\begin{table}
\begin{tabular}{l|c|c|c|c|c|c} \hline \multicolumn{1}{c}{} & \multicolumn{6}{c}{**NIST Chinese-English**} \\

- \\ Transformer-baseYang et al. (2018) & 42.23 & 42.17 & 41.02 &
- \\ \hline \hline baseline & 43.78 & 44.26 & 40.97 & 38.93 & 41.39 &
- \\ baseline + LKS & 44.12 & 44.87 & 41.59 & 39.22 & 41.89 & +0.50 \\ baseline + BKS & 44.23 & 44.98 & 41.62 & 39.74 & 42.11 & +0.73 \\ baseline + BKE & 44.30 & 45.01 & 41.86 & 40.05 & 42.31 & +0.92 \\ \hline \hline ODC & 45.33 & 45.18 & 42.60 & 39.67 & 42.48 & +1.10 \\ ODC + LKS & 45.05 & 45.49 & 42.99 & 40.48 & 42.99 & +1.60 \\ ODC + BKS & **45.35** & 45.49 & 43.21 & 39.96 & 42.89 & +1.50 \\ ODC + BKE & 45.34 & **45.92** & **43.35** & 40.30 & **43.19** & **+1.80** \\ \hline \hline ODC-EMA & 45.52 & 45.72 & 43.01 & **40.65** & 43.13 & +1.74 \\ \hline \end{tabular}
\end{table}
Table 1: Case-insensitive BLEU scores of Chinese-English translation on NIST datasets. “Average”means average scores on NIST04, 05 and 06.
 

as the final model. In this case, checkpoints may have better performance but higher variance which could be harmful to parameters averaging.
* **best-k-ensemble**: Do ensemble inference (average the output probabilities) with the best k checkpoints Chen et al. (2017).

As shown in Table 1, our baseline is comparable to the other two recent published results Zhang et al. (2018b); Yang et al. (2018). In consistent with Chen et al. (2017), using checkpoints for smoothing or ensemble does improve the baseline system. Using EMA parameters also improve the baseline system as well, which is in consist with Tarvainen and Valpola (2017).

Compared to the baseline, our approach ODC brings translation improvement across different test sets and achieves 42.48 BLEU scores on average(+1.09 BLEU v.s. baseline). This result confirms that using best checkpoint as teacher indeed helps improving the performance of the translation model.

Besides, ODC is comparable to the best results among smoothing and ensemble on baseline's checkpoints (achieved by best-k-ensemble). Considering that best-k-ensemble needs to decode with k models, while ODC decodes only one, our model enjoys a better efficiency. Furthermore, we can achieve further improvement by combining these methods on checkpoints generated by ODC.

Results also show that ODC-EMA (SS3.3) could achieve additional improvement from ODC itself (43.13 v.s. 42.48 BLEU), demonstrating that using EMA of the best checkpoint instead can bring better knowledge distillation performance, as it generates a better teacher model.

Results on WMT17 DatasetWe present the results on WMT17 Chinese-English translation task in Table 2. We report the results of the baseline, ODC and a recent result published by Zhang et al. (2018c). To make a fair comparison, we follow the experiment setting in Zhang et al. (2018c). The experiment results show similar trends with those on the NIST datasets. Applying ODC leads to the result of 24.22 BLEU, which is 0.85 BLEU higher compared with baseline.

### Evaluation on Low-resource Scenario

We also apply our method to two low resource translation tasks, i.e., IWSLT2015 English-Vietnamese EN2VI) and WMT17 English-Turkish (EN2TR). Due to the limited amount of training data, models are more likely to suffer from over-fitting. Therefore, we use a higher dropout rate of 0.2 and weight decay, another common technique against overfitting, with decay weight set as \(10^{-3}\) as the default setting. We implement weight decay as AdamW Loshchilov and Hutter (2017) does.

Besides, we further experiment with grid search on the validation set for optimal hyper-parameters of dropout rate and weight decay, which may lead to better results. We adopt a simple heuristic, which first searches an optimal dropout rate, and then further searches weight decay coefficients based on this dropout. We experiment with dropout as 0.2, 0.3, 0.4, and weight decay as \(10^{-1}\), \(10^{-2}\) and \(10^{-3}\).

As in Table 3, our baseline is comparable to two recent published results, respectively: EN2TR from Zhang et al. (2018c) and EN2VI from official release tensor2tensor problem9. Grid hyperparameter search does improve the baseline system. ODC leads to better results compared to the baseline, as well as the baseline with grid parameter search. ODC can achieve further improvement after searching for optimal hyper-parameters of dropout and weight decay.

Footnote 9: [https://github.com/tensorflow/tensor2tensor/pull/611](https://github.com/tensorflow/tensor2tensor/pull/611)

\begin{table}
\begin{tabular}{l|c|c} \hline
* & 23.01 \\ baseline & 21.96 & 23.37 \\ ODC & **22.24** & **24.22** \\ \hline \end{tabular}
\end{table}
Table 2: Case-sensitive BLEU scores on WMT17 Chinese-English Translation

\begin{table}
\begin{tabular}{l|c|c} \hline
* & 12.11 \\ tensor2tensor & 28.43 &
- \\ \hline \hline baseline & 28.56 & 12.20 \\ baseline w/ grid-search & 29.01 & 12.51 \\ \hline \hline ODC & **29.47** & **12.92** \\ ODC w/ grid-search & **29.59** & **13.18** \\ \hline \end{tabular}
\end{table}
Table 3: Case-sensitive BLEU scores on two low resource translation tasks.
 



### Evaluation on Machine Reading Comprehension

Although our main research is focused for the task of machine translation, the idea of ODC could be applied to other tasks as well. We experiments on the Stanford Question Answering Dataset (SQuAD) Rajpurkar et al. (2016), a machine reading comprehension task.

SQuAD contains 107,785 human-generated reading comprehension questions, with 536 Wikipedia articles. Each question is associated with a paragraph extracted from an article, and the corresponding answer is a span from this article. A machine reading comprehension model is designed to predict the start and end positions in the article of the answer.

The state-of-the-art machine reading comprehension system also employs a deep neural network structure, which is similar to NMT. We apply our ODC method on **BiDAF++**Choi et al. (2018), a multi-layer SQuAD model that augments BiDAF Seo et al. (2016) with self-attention and contextualized embeddings. We evaluate the model after each epoch and implement the knowledge distillation by teaching the student with the output distribution of answer start and end positions predicted by the best checkpoint.

For the results, ODC improves a base BiDAF++ from 76.83 to 77.40, in EM scores, showing that our method can be applied to a broader range of tasks.

## 5 Analysis

We conduct further analysis to probe into the reasons for the advantages of ODC. We first show that our method can significantly alleviate the over-fitting issue in data-limited condition. After that, we show that parameters gained from our method tend to be wider minimums, which represents better generalization.

### Effectiveness on reducing over-fitting

Taking IWSLT15 English-Vietnamese as a test-bed, we analyze whether our method could help handle the over-fitting issue. We first plot the curve of the loss on the validation set at each training step for the different models (in Figure 2, the top curve with rounds). It is easy to see that the loss curve of the baseline increases as the training goes after 50K steps, indicating a severe over-fitting. With better dropout rate and weight decay, the over-fitting is less severe; while with ODC the loss curve shows a more steady trend of decrease, and is almost always under the other two's.

The final BLEU score on the validation set (Figure 2, bottom) shows corresponding result. The grid search of hyper-parameters improves the BLEU from 26.06 to 26.42 in BLEU, while ODC achieves 26.99.

Both results indicate that our method is more effective at handling the over-fitting problem. We hold that minimizing the cross-entropy between the teacher model and the student model serves as regularization to the training of the student model, which avoids the model getting into over-fitting.

### ODC Brings Wider Minimum

In the training process in Chinese-English tasks, we do not observe obvious over-fitting issue as shown in low resource translation tasks. In this section, we analyze how ODC helps the model generalization. Keskar et al. (2016) proposed that the width

Figure 2: Loss curves (top) and final BLEU scores (bottom) on the validation set of baseline, baseline with grid-search and ODC, respectively.



of the minimum in a loss surface is related to its generalization ability. Therefore, we compare the generalization capability between baseline system and our ODC method by exploring around the parameters.

We make use of the visualization technique employed in Goodfellow and Vinyals (2014) and analyze the results on the NIST data set. Let \(\theta_{\text{base}}\) and \(\theta_{\text{ODC}}\) denote the final parameters obtained from baseline and ODC. Consider the line:

\[\theta(\alpha)=\alpha\cdot\theta_{\text{ODC}}+(1.0-\alpha)\cdot\theta_{\text{ base}}, \tag{10}\]

which connects \(\theta_{\text{base}}\) (\(\alpha=0.0\)) and \(\theta_{\text{ODC}}\) (\(\alpha=1.0\)). We plot the value of Equation 4 as a function of \(\alpha\) (normalized by count of words per sentence) with \(\theta=\theta(\alpha)\). We draw \(\alpha\) from \(-1.0\) to \(2.0\) at an interval of 0.02. In this way, the width of \(\theta_{\text{base}}\) and \(\theta_{\text{ODC}}\) can be represented as the steepness of the curve nearby. To further quantitatively represent the steepness, we compute the standard deviation of values on this curve within different distances to the two parameters, respectively. We plot them in Figure 3.

From Figure 3 we can see that the loss curve behaves steeper around the parameters of baseline than of ODC. Besides, the standard deviations of losses around the baseline model are consistently higher than ODC within all the distances. It is evident that the parameters of ODC act as a wider minimum c and explains why ODC can lead to a more generalized model.

## 6 Related Works

### Regularization in NMT

Regularization has broad applications in training NMT models to improve performance and avoid over-fitting. There are some common regularization techniques, such as \(L^{2}\) normalization and dropout Srivastava et al. (2014). These methods are simple and easy to implement but need carefully tuning on the validation set. These methods are also orthogonal to our method.

There are also some works to exploit regularization techniques in fine tuning of NMT model. Barone et al. (2017a) proposed a tuneout method which randomly replaces columns of weight matrices of out-of-domain parameter matrices. Khayralalah et al. (2018) shared similar training object with us, as they computed the KL divergence between out-of-domain and in-domain model. Both of their works request a pre-trained teacher model, while we are work on a more general training problem which does not require such kind of model.

### Online Knowledge Distillation

While traditional knowledge distillation requires a static, pre-trained teacher model, online knowledge distillation tends to overcome this problem by selecting or generating a teacher dynamically from scratch.

To the best of our knowledge, Zhang et al. (2017) is the first trial to replace the offline teacher model. They trained peer models to teach each other simultaneously. Compared to their work, our method uses the best checkpoint as the teacher, which avoids introducing extra parameters. Furlanello et al. (2018) tends to update teacher model during the training procedure iteratively, but their method needs to train the teacher model until convergence in each iteration. Instead, our method only needs one phase of training, whose overhead is relatively small.

Figure 3: The upper plot shows the validation losses curve along the line segment decided by parameters of baseline and ODC. The bottom plot shows the standard deviations within the neighborhood of baseline and ODC at different distances.

 

Lan et al. (2018) using an ensemble of several branches of the model as teacher for computer vision tasks, which only needs one-phase training as well. However, their method relies heavily on the multi-branch structures of the tasks, which are not widely applicable in neural machine translation.

## 7 Conclusion

In this paper, we propose an online knowledge distillation method with the teacher model generated from checkpoints during the training procedure. Experiments on four machine translation tasks and a machine reading task show that our method outperforms strong baseline systems. Further analysis shows that our method can effectively alleviate the over-fitting issue, and tend to find a wider minimum.

## Acknowledgement

We would like to thank the anonymous reviewers for their insightful comments. We also thank Boxing Chen from Alibaba Group for his helpful comments. This work is supported by the National Science Foundation of China (No. 61772261, 61672277) and the Jiangsu Province Research Foundation for Basic Research (No. BK20170074). Part of this work is supported by "13th Five-Yea" All-Army Common Information System Equipment Pre-Research Project (No. 31510040201).

## References

* A. M. Barone, B. Haddow, U. Germann, and R. Sennrich (2017a)Regularization techniques for fine-tuning in neural machine translation. In EMNLP 2017, pp. 1489-1494. Cited by: SS1.
* A. V. Barone, J. Helcl, R. Sennrich, B. Haddow, and A. Birch (2017b)Deep architectures for neural machine translation. In WMT 2017, Cited by: SS1.
* C. Bucila, R. Caruana, and A. Niculescu-Mizil (2006)Model compression. In KDD, Cited by: SS1.
* H. Chen, S. Lundberg, and S. Lee (2017)Checkpoint ensembles: Ensemble methods from a single training process. CoRRabs/1710.03282. External Links: Link, 1710.03282 Cited by: SS1.
* K. Cho, B. van Merrienboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y. Bengio (2014)Learning phrase representations using RNN encoder-decoder for statistical machine translation. In EMNLP 2014, Cited by: SS1.
* E. Choi, H. He, M. Iyyer, M. Yatskar, W. T. Yih, Y. Choi, P. Liang, and L. S. Zettlemoyer (2018)Quac: question answering in context. In EMNLP, Cited by: SS1.
* T. Furlanello, Z. Chase Lipton, M. Tschannen, L. Itti, and A. Anandkumar (2018)Born again neural networks. In ICML, Cited by: SS1.
* J. Gehring, M. Auli, D. Grangier, D. Yarats, and Y. N. Dauphin (2017)Convolutional sequence to sequence learning. In ICML 2017, Cited by: SS1.
* I. J. Goodfellow and O. Vinyals (2014)Qualitatively characterizing neural network optimization problems. CoRRabs/1412.6544. External Links: Link, 1412.6544 Cited by: SS1.
* Geoffrey E. Hinton, O. Vinyals, and J. Dean (2015)Distilling the knowledge in a neural network. CoRRabs/1503.02531. External Links: Link, 1503.02531 Cited by: SS1.
* Po-Sen Huang, C. Wang, D. Zhou, and L. Deng (2017)Neural phrase-based machine translation. CoRRabs/1706.05565. External Links: Link, 1706.05565 Cited by: SS1.
* M. Johnson, M. Schuster, Q. V. Le, M. Krikun, Y. Wu, Z. Chen, N. Thorrat, F. B. Viegas, M. Wattenberg, G. Corrado, M. Hughes, and J. Dean (2017)Google's multilingual neural machine translation system: enabling zero-shot translation. TACL 2017, pp. 5339-351. External Links: Link, 1706.05565 Cited by: SS1.
* N. Shirish Keskar, D. Mudigere, J. Nocedal, M. Smelyanskiy, and P. T. P. Tang (2016)On large-batch training for deep learning: generalization gap and sharp minima. CoRRabs/1609.04836. External Links: Link, 1609.04836 Cited by: SS1.
* H. Khayrallah, B. Thompson, K. Duh, and P. Koehn (2018)Regularized training objective for continued training for domain adaptation in neural machine translation. In WMT 2018, pp. 36-44. Cited by: SS1.
* Y. Kim and A. M. Rush (2016)Sequence-level knowledge distillation. In EMNLP, Cited by: SS1.
* D. P. Kingma and J. Ba (2014)Adam: a method for stochastic optimization. CoRRabs/1412.6980. External Links: Link, 1412.6980 Cited by: SS1.
* X. Lan, X. Zhu, and S. Gong (2018)Knowledge distillation by on-the-fly native ensemble. CoRRabs/1806.04606. External Links: Link, 1806.04606 Cited by: SS1.
* I. Loshchinov and F. Hutter (2017)Fixing weight decay regularization in adam. CoRRabs/1711.05101. External Links: Link, 1711.05101 Cited by: SS1.
* Minh-Thang Luong and Christopher D. Manning (2015)Stanford neural machine translation systems for spoken language domain. In IWSLT 2015, Cited by: SS1.
* P. Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang (2016)Squad: 100, 000+ questions for machine comprehension of text. In EMNLP, 

Rico Sennrich, Alexandra Birch, Anna Currey, Ulrich Germann, Barry Haddow, Kenneth Heafield, Antonio Valerio Miceli Barone, and Philip Williams. 2017. The university of Edinburgh's neural MT systems for WMT17. In _WMT 2017_.
* Sennrich et al. (2016a) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016a. Edinburgh neural machine translation systems for WMT 16. In _ACL 2016_, pages 371-376.
* Sennrich et al. (2016b) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016b. Neural machine translation of rare words with subword units. In _ACL 2016_.
* Seo et al. (2016) Min Joon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. 2016. Bidirectional attention flow for machine comprehension. _CoRR_, abs/1611.01603.
* Srivastava et al. (2014) Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: a simple way to prevent neural networks from overfitting. _The Journal of Machine Learning Research_, 15(1):1929-1958.
* Sutskever et al. (2014) Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to sequence learning with neural networks. In _NIPS 2014_, pages 3104-3112.
* Tarvainen and Valpola (2017) Antti Tarvainen and Harri Valpola. 2017. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. In _NIPS 2017_, pages 1195-1204.
* Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In _NIPS_.
* Yang et al. (2018) Zhen Yang, Wei Chen, Feng Wang, and Bo Xu. 2018. Improving neural machine translation with conditional sequence generative adversarial nets. In _NAACL-HLT_.
* Zhang et al. (2018a) Biao Zhang, Deyi Xiong, and Jinsong Su. 2018a. Accelerating neural transformer via an average attention network. In _ACL_.
* Zhang et al. (2018b) Xiangwen Zhang, Jinsong Su, Yue Qin, Yang Liu, Rongrong Ji, and Hongji Wang. 2018b. Asynchronous bidirectional decoding for neural machine translation. In _AAAI_.
* Zhang et al. (2017) Ying Zhang, Tao Xiang, Timothy M. Hospedales, and Huchuan Lu. 2017. Deep mutdual learning. _CoRR_, abs/1706.00384.
* Zhang et al. (2018c) Zhirui Zhang, Shuangzhi Wu, Shujie Liu, Mu Li, Ming Zhou, and Enhong Chen. 2018c. Regularizing neutral machine translation by target-bidirectional agreement. _CoRR_, abs/1808.04064.