Abstract: Neural machine translation (NMT) offers a novel alternative formulation of translation that is potentially simpler than statistical approaches. However to reach competitive performance, NMT models need to be exceedingly large. In this paper we consider applying knowledge distillation approaches (Bucila et al., 2006; Hinton et al., 2015) that have proven successful for reducing the size of neural models in other domains to the problem of NMT. We demonstrate that standard knowledge distillation applied to word-level prediction can be effective for NMT, and also introduce two novel sequence-level versions of knowledge distillation that further improve performance, and somewhat surprisingly, seem to eliminate the need for beam search (even when applied on the original teacher model). Our best student model runs 10 times faster than its state-of-the-art teacher with little loss in performance. It is also significantly better than a baseline model trained without knowledge distillation: by 4.2/1.7 BLEU with greedy decoding/beam search. Applying weight pruning on top of knowledge distillation results in a student model that has 13 times fewer parameters than the original teacher model, with a decrease of 0.4 BLEU.

Introduction: Neural machine translation (NMT) BIBREF2 , BIBREF3 , BIBREF4 , BIBREF5 is a deep learning-based method for translation that has recently shown promising results as an alternative to statistical approaches. NMT systems directly model the probability of the next word in the target sentence simply by conditioning a recurrent neural network on the source sentence and previously generated target words. While both simple and surprisingly accurate, NMT systems typically need to have very high capacity in order to perform well: Sutskever2014 used a 4-layer LSTM with 1000 hidden units per layer (herein INLINEFORM0 ) and Zhou2016 obtained state-of-the-art results on English INLINEFORM1 French with a 16-layer LSTM with 512 units per layer. The sheer size of the models requires cutting-edge hardware for training and makes using the models on standard setups very challenging. This issue of excessively large networks has been observed in several other domains, with much focus on fully-connected and convolutional networks for multi-class classification. Researchers have particularly noted that large networks seem to be necessary for training, but learn redundant representations in the process BIBREF6 . Therefore compressing deep models into smaller networks has been an active area of research. As deep learning systems obtain better results on NLP tasks, compression also becomes an important practical issue with applications such as running deep learning models for speech and translation locally on cell phones. Existing compression methods generally fall into two categories: (1) pruning and (2) knowledge distillation. Pruning methods BIBREF7 , BIBREF8 , BIBREF9 , zero-out weights or entire neurons based on an importance criterion: LeCun1990 use (a diagonal approximation to) the Hessian to identify weights whose removal minimally impacts the objective function, while Han2016 remove weights based on thresholding their absolute values. Knowledge distillation approaches BIBREF0 , BIBREF10 , BIBREF1 learn a smaller student network to mimic the original teacher network by minimizing the loss (typically INLINEFORM0 or cross-entropy) between the student and teacher output. In this work, we investigate knowledge distillation in the context of neural machine translation. We note that NMT differs from previous work which has mainly explored non-recurrent models in the multi-class prediction setting. For NMT, while the model is trained on multi-class prediction at the word-level, it is tasked with predicting complete sequence outputs conditioned on previous decisions. With this difference in mind, we experiment with standard knowledge distillation for NMT and also propose two new versions of the approach that attempt to approximately match the sequence-level (as opposed to word-level) distribution of the teacher network. This sequence-level approximation leads to a simple training procedure wherein the student network is trained on a newly generated dataset that is the result of running beam search with the teacher network. We run experiments to compress a large state-of-the-art INLINEFORM0 LSTM model, and find that with sequence-level knowledge distillation we are able to learn a INLINEFORM1 LSTM that roughly matches the performance of the full system. We see similar results compressing a INLINEFORM2 model down to INLINEFORM3 on a smaller data set. Furthermore, we observe that our proposed approach has other benefits, such as not requiring any beam search at test-time. As a result we are able to perform greedy decoding on the INLINEFORM4 model 10 times faster than beam search on the INLINEFORM5 model with comparable performance. Our student models can even be run efficiently on a standard smartphone. Finally, we apply weight pruning on top of the student network to obtain a model that has INLINEFORM6 fewer parameters than the original teacher model. We have released all the code for the models described in this paper.

Question: What type of weight pruning do they use?
Answer: Prune %x of the parameters by removing the weights with the lowest absolute values.