Understanding Knowledge Distillation in Non-autoregressive Machine Translation
Authors: Chunting Zhou
Jiatao Gu
Graham Neubig
Venue:  (2020)
Keywords: knowledge distillation
non-autoregressive neural machine translation

Abstract: Non-autoregressive machine translation (NAT) systems predict a sequence of output tokens in parallel, achieving substantial improvements in generation speed compared to autoregressive models. Existing NAT models usually rely on the technique of knowledge distillation, which creates the training data from a pretrained autoregressive model for better performance. Knowledge distillation is empirically useful, leading to large gains in accuracy for NAT models, but the reason for this success has, as of yet, been unclear. In this paper, we first design systematic experiments to investigate why knowledge distillation is crucial to NAT training. We find that knowledge distillation can reduce the complexity of data sets and help NAT to model the variations in the output data. Furthermore, a strong correlation is observed between the capacity of an NAT model and the optimal complexity of the distilled data for the best translation quality. Based on these findings, we further propose several approaches that can alter the complexity of data sets to improve the performance of NAT models. We achieve the state-of-the-art performance for the NAT-based models, and close the gap with the autoregressive baseline on WMT14 En-De benchmark.

Published as a conference paper at ICLR 2020
UNDERSTANDING KNOWLEDGE DISTILLATION IN NON AUTOREGRESSIVE MACHINE TRANSLATION
Chunting Zhou'; Jiatao Gu Graham Neubig!
Language Technologies Institute, Carnegie Mellon University! Facebook Al Research²
{chuntinz, gneubig] @cs. cmu.cmu.... jgue fb. com
ABSTRACT
Non-autoregressivee machine translation (NAT) systems predict a sequence of out- put tokens in parallel, achieving substantial improvements in generation speed compared to autoregressive models. Existing NAT models usually rely on the technique of knowledge distillation, which creates the training data from a pre- trained autoregressive model for better performance. Knowledge distillation is empirically useful, leading to large gains in accuracy for NAT models, but the reason for this success has, as of yet, been unclear. In this paper, we first de- sign systematic experiments to investigate why knowledge distillation is crucial in NAT training. We find that knowledge distillation can reduce the complexity of data sets and help NAT to model the variations in the output data. Furthermore, strong correlation is observed between the capacity of an NAT model and the complexity the distilled data that provides the best translation quality. Based on these findings, we further propose several approaches that can alter the complexity of data sets to improve the performance of NAT models. We achieve state-of aate-of-it of-the- art performance for NAT-based models, and close the gap with the autoregressive baseline on the WMT14 En-De benchmark!
INTRODUCTION
Traditional neural machine translation (NMT) systems (Bahdanau et al. 2015 Gehring al. 2017 Vaswani et al. 2017) generate sequences in an autoregressive fashion; each target token is predicted step-by-step by conditioning the previous generated tokens in a monotonic (e.g. left-to-right) order. While such autoregressive translation (AT) models have proven successful, the sequential de- pendence of decisions precludes taking full advantage of parallelism afforded by modern hardware (e.g. GPUs) at inference time. In contrast, non-autoregressive translation (NAT) models (Gu et al., 2018 Lee et al. 2018) predict the whole sequence or multi-token chunks of the sequence simul- taneously, alleviating this problem by trading the model's is capacity for decoding efficiency. Such a non-autoregressive factorization assumes that the output tokens are independent from each other. However, this assumption obviously does not hold in reality and as result NAT models generally perform worse than standard AT models.
One key ingredient in the training recipe for NAT models that is used in almost all existing works (Gu et al. (2018): y Lee et al. (2018); [Stern et al. (2019), inter alia) is creation training data through knowledge distillation (Hinton et al. 2015). More precisely, sequence-level knowledge distilla- tion (Kim & Rush 2016)- - a special variant of the original approach- is applied during NAT model training by replacing the target side of training samples with the outputs from pre-trained AT model trained on the same corpus with a roughly equal number parameters. It is usually assumed (Gu al. 2018) that knowledge distillation's reduction of the "modes" (alternative translations for an input) in the training data is the key reason why distillation benefits NAT training. However, this intuition has been rigorously tested, leading to three important open questions:
"Equal Contribution. Most work was done during Chunting's 's internship FAIR http:///ttrrrre//iiiieeee//eeeee
"Code is released at nonautoregresssiver weweerrrnnlatiiiiee

---

Published as a conference paper at ICLR 2020
Exactly how does distillation reduce the "modes", and how could we measure this reduction quantitativel? Why does this reduction consistently improve NAT models?
What is the relationship between the NAT model (student) and the AT model (teacher)? Are different varieties of distilled data better for different NAT models?
Due distillation, the performance of NAT models is largely bounded by the choice of AT teacher. Is there way to further close the performance gap with standard AT models?
In this paper, aim to answer the three questions above, improving understanding of knowledge distillation through empirical analysis over a variety of AT and NAT models. Specifically, our con- tributions follows:
We first visualize explicitly on a synthetic dataset how modes are reduced by distillation (3.1 Inspired by the synthetic experiments, we further propose metrics for measuring complexity and faithfulness for given training set. Specifically, our metrics are the conditional entropy and KL-divergence of word translation based on external alignment tool, and we show that these metrics are correlated with NAT model performance ($3.2).
We conduct a systematic analysis ($4) over four AT teacher models and six NAT student mod- els with various architectures on the standard WMT14 English-German German translation benchmark. These experiments find strong correlation between the capacity of an NAT model and the opti- mal dataset complexity that results in the best translation quality.
Inspired by these observations, we propose approaches to further adjust the complexity of the distilled data in order to match the model's capacity (:5). We also show that we can achieve the ofte-of-theeart performance for NAT models and largely match the performance of the AT model.
BACKGROUND
2.1 NON-AUTOREGRESSIVE NEURAL MACHINE TRANSLATION
In order to model the joint probability of the output sequence y. NMT models usually generate each output token conditioned on the previously generated ones p(y = I[==1 (ty.)eee.2 æ). This is known the autoregressive factorization. To generate translation from this model, one could predict one token at a time from left to right and greedily take arg max over each output prob- ability distribution, or use beam search to consider a fixed number of hypotheses. In this work, we study non-autores aatrarrrrreer translation (NAT), special subset of NMT models with an addi- tional restriction (the zeroth-order Markov assumption) upon the output predictions or subset thereof. The simplest formulation of an NAT model independently factors the conditional distri- bution: [zy/ = IT= p(ye (z).
Standard NAT models (Gu et al. 2018) adopt an architecture similar to the Transformer (Vaswani jet al. 2017) and make non-autoregressive predictions for the entire sequence with one forward pass of the decoder. However, because multiple translations are possible for a single input sentence (the so-called multi-modality problem; [u et al. (2018)), vanilla NAT models fail capture the dependencies between output tokens. As a result, they tend to make egregious mistakes such as outputting tokens repeatedly. To improve the model's ability to handle mulli--odalitty recent works have incorporated approaches including (I) relaxing the fully non-autoregressive restriction and adopting K decoding passes (instead of just one) to iteratively refine the generated outputs (Lee et al., 2018, Ghazvininejad et al.] [019 Wang et al.] 2018: [Stern et al.] [2018] (019 Gu et al.) 2019) (2) using latent variables (Kaiser et al. 2018 8 Ma et al.) 2019 Shu et al.) 2019) or structured information such as syntax trees (Akoury al. 2019) to capture translation variation; (3) training NAT models with objectives other than maximum likelihood (Wang et al. 2019 Wei et al.) 2019;
Shao et al. (2019) which ameliorates the effects of multi modality. However, to achieve competitive performance with the autoregressive model, almost all existing NAT models rely on training using data distilled from a pre-trained AT model instead of the real parallel training set, as described below.
2.2 SEQUENCE-LEVEI KNOWLEDGE DISTILLATION
Knowledge distillation (Liang et al. 2008 Hinton et al., 2015) was originally proposed for train- ing weaker student classifier the targets predicted from stronger teacher model. A typ-
2

---

Published as a conference paper at ICLR 2020
ical approach is using the label probabilities produced by the teacher as "soft targets" 9: exp(2./7) (pp///////, /,, exp(2)/T) for training the student model, where q, and 24 are the probability and the logit of class i respectively and T is the temperature. Prior work has shown the effectiveness of adopting knowledge distillation in adversarial defense (Papernot et al., (016)) neural network compression (Howard et al. 2017), and fast inference for speech synthesis (Oord et al. 2018).
In the context of sequence generation, Kim & Rush (2016) extend knowledge distillation to the sentence level using "hard targets" from a pretrained large teacher model to train a small sequence generation model. More precisely, the teacher distribution ((|2z)) is approximated by its mode: (t(2z)) R [tt= = arg max LET ((((2))) with the following objectives:
CDDg-KD data Σ q(t),) log ((g((t))) = Eng max g(t]m) [logp(t x/z)] RET
(I)
SET
where t E T is the space of possible target sequences. This can also be seen special case of standard distillation over the sentence space when the temperature т approaches 0, which is equiva- lent taking the arg max over all feasible translations. While the "hard target" is the most likely translation predicted by the teacher, in practice we use beam search as an approximation. As men- tioned earlier, almost all the existing literature trains NAT models using sequence-level knowledge distillation from a pre-trained AT model to achieve competitive performance. Particularly, it is com- mon to train the teacher model as a standard autoregressive Transformer (Vaswani et al. (017) with roughly equal number of trainable parameters as the desired NAT model the real data. Next, we will first study how this knowledge distillation process affects the behavior of NAT models.
3 HOW DOES DISTILLATION IMPROVE NAT?
In this section, we start from an introductory example to illustrate how NAT models fail to capture the multi-modality of data. Then propose a metric to assess the multi-modality of data set and use to test our hypothesis about how knowledge distillation affects NAT models.
3.1 SYNTHETIC EXPERIMENT FOR MULTI-MODALITY
Dataset. We start by investigating NAT's difficulties in modeling multi-modality in output data using synthetic setup where we explicitly include multiple modes in the training data. More specifically, we utilize three language pairs English-German (En-De), English-French (En-Fr), and Spaansh (En-Es) from the Europarl parallel corpus We extract sentences that have aligned sentences for all languages, and create multi-target En De/EE/FFF corpus. In this case every English imput sentence always corresponds to target sentences in three different languages, which forms three explicis output modes. Notably, this similar to the one-to-many translation setting in Johnson et al.) (2017) but in our case we do not have an explicit signal (e.g. target language tag) to tell the NMT model which target language to translate to.
Models. We train both the AT and NAT models on this concatenated data set, then compare the distributions translations with each other. We use the standard Transformerrbase) model (Vaswani et al. (017)) as the AT model, and simplified version of Gu et al. (2018) the NAT model where the decoder's 's inputs are monotonically copied from the encoder embeddings and a length predictor is learned predict the target sentence length. Both models are trained for 300, 000 steps using maximum likelihood. After training, we use both models to translate the English sentences in the validation and test sets.
Visualization of AT Outputs. The synthetic setup enables us to better understand and visualize the modes in the outputs more easily. First, we visualize the outputs from the AT model. For every translated sentence, we visualize the estimated probability distribution of language classes as a point in Fig. [] (a). This probability is calculated as the average of the posterior probability of each token, and it is estimated based on the Bayes' law:
T
1
T
ply 14 (p).))
p(lly) ~
p(ll)t)
(2)
T
T
k. p(vi (1x))
3

---

Published as a conference paper at ICLR 2020
Es
De
(a) Baseline
(b) NAT Bascline
(c) NAT Random Select
(d) NAT Distill
Figure I: Posterior distribution of language IDs for the outputs from different models. Each transla-
tion is represented as point inside the simplex A = {(Pae Pes; Per) [Px (0. 1), Pse + +Pr = 1} where pk is the estimated probability of being translated into language k E (de, es, frr.. We distin- guish the language that has the largest probability with different colors.
where I; denotes the language class and p(ue(4) is the token frequency of ye in language 4, We assume p(4:) follows uniform distribution. As shown in Fig. 1 (a), points of the AT outputs are clustered closely to each vertex of the simplex, indicating that the AT model prefers generate the whole sequence in one language. This phenomenon verifies our assumption that decoding with the AT model (distillation) is essentially selecting "modes" over the real data.
Visualization of NAT Outputs. We visualize outputs for the NAT model trained on the same data Fig. (b). In contrast to the AT results, the NAT points are scattered broadly inside the simplex, indicating that the NAT model fails to capture the mode of language types. Instead, it predicts tokens mixed with multiple languages, which corroborates our hypothesis that the NAT model has trouble consistently selecting single mode when multiple modes exist.
Next, we create two datasets that have fewer modes than the original dataset. First, we randomly select single target sentence from one of the three languages for each source sentence. Second, we perform distillation, decoding from the model trained on the combined training set. As noted in the AT results, distillation will also roughly be selecting language mode, but conjecture that this selection may be more systematic, selecting a particular language for a particular type of training sentence. As shown in Fig. (c) (d), NAT models trained both of these datasets more likely to choose one mode (language) when generating translations, showing that training with reduced modes is essential for NAT model. Furthermore, points Fig. [ (d) clearly clustered better than (c) indicating that modes selected by AT models are indeed likely more systematic and easy to capture than those generated by randomly assigning language for each sentence.
3.2 QUANTITATIVE MEASURES FOR PARALLEL DATA
To better study why distillation is crucial for NAT models, in this section, we propose quantita- tive measures for analyzing the complexity and faithfulness parallel data, two properties that we hypothesize are important for NAT training.
Measure of Complexity. Inspired by the observations in the synthetic experiments, we propose to measure of translation uncertainty, specifically operationalized as conditional entropy, the (æN, yN)}, where (æ; y)
measurement of complexity C(d) for any given dataset sentence pair instantiation of (X, Y) and X E X, Y E Y:
H(Y x²
Σ plyla) log p(yl))
wEY
p(u x))
log (z))
asm.1: conditional independence
vEy 1=1
(3)
Ty
Σ
p(v. log
asm.2: alignment model
1=1 (m))
Tb
H(vl= mi)
4

---

Published as a conference paper at ICLR 2020
d
En-De
En-Es
En-Fr
Full Real Data
Random Selection
Distillation
((d)
3.12
2.81
2.89
3.67
3.30
2.64
Table I: Complexity C(d) (t more complex) of the Europarl data set of different settings in 331
where we use æ and y to denote a word in the source and target vocabulary respectively. T. and T, denote the length of the source and target sentences. To make the computation tractable, we make two additional assumptions on the conditional distribution p(y( zr):
We assume the target tokens are independent given the source sentence. Then the conditional entropy of sentence be converted into the sum of entropy of target words conditioned on the source sentence a.
Assumption 1:
We assume the distribution of p(y (z) follows an alignment model (Dyer et al, 2013 where y/ is is generated from the word alignment distribution p(yel Allgnnyyy) This makes possible to simplify the conditional entropy to the sum of entropy of target words conditioned on the aligned source words denoted Mlyl2 zz).
Assumption 2:
The corpus level complexity C(d) is then calculated by adding up the conditional entropy H(Y|X x)) of all sentences. To prevent C(d) from being dominated by frequent words, we calculate C(d) by averaging the entropy of target words conditioned on a source word, denoted C'dd) P. Exev. H(y[z)).
To illustrate that the proposed metric is reasonable measure of complexity of parallel corpus, in Tab. [ we compute C(d) for parallel data from different language pairs, the concatenated data set, and the data distilled from the AT model described in 43.1 We observe that the conditional entropy of the distilled data much smaller than that of the concatenated or randomly selected data mentioned above. Additionally, we find that the conditional entropy of En-Es and En-Fr similar but that of En-De is relatively larger, which can also explain why the student NAT model prefers to predict the modes of Es or Fr more often than De as shown in Fig. dd..
Measure of Faithfulness. C(d) reflects the level of multi-modality of a parallel corpus, and we NAT model. However, it is not fair to assess the
have shown that simpler data set is favorable data set only by its complexity; we can trivially construct a simple data set with no variations the output, which obviously won't be useful for training. The other important measurement of the data set is its faithfulness to the real data distribution. To measure the faithfulness of parallel corpus d, we use KL-divergence of the alignment distribution between the real parallel data set r and an altered parallel data set d, denoted F(d):
(r(l)z) log (r(v)x) pa(y(x)
F(d)
Σ REV. BEV,
Σ
(4)
2
4 EMPIRICAL STUDY
In this section, perform extensive study variety of non-autoregressive (NAT) mod- els trained from different autoregressive (AT) teacher models to assess how knowledge distillation affects the performance of NAT models.
4.1 EXPERIMENTAL SETTINGS
Data. We use the data set commonly used by prior work as our evaluation benchmark: WMT14 English-Gernin German [ We use news newstest2013 as the validation set for selecting the best model, and newstest 2014 4 as the test set. We learn a byte-pair encoding (BPE, Sennrich et al. 2016 vocabulary of 37,000 on the tokenized data.
AT Models. We set up four Transformer models with different parameter sizes: Transformer- tiny/small/basc/big denoted as tiny, small, base, big respectively. We build base and big models
compute the alignment given the dataset. http::/www.sttttt.orrs.. http:///////.....a.ooooooaaaaaaa hhml
We follow
5

---

Published as a conference paper at ICLR 2020
following settings described in Vaswani et al. (2017). and reduce the model sizes for riny, small to create weaker teacher models. Details the model architectures can be found in Appendix A
All the models number steps set to 300, 000, After training, we use the resultng AT models to decode the whole training set with beam size and replace the real target sentences create new parallel corpus.
trained using the Adam optimizer (Kingma & Ba 2014) with the maximum
NAT Models. We consider the following NAT models, from vanilla to state-of-ther the-rrt. All the models using the Transformer the basic backbone and are (re- (teljimplemented based Fairsed except for FlowSeq. We briefly outline the methods and parameters here, and describe detailed settings in the Appendix A
simplified version where the decoder's inputs are directly copied from the encoder without considering latent variables.
Vanilla NAT (Gu et al. 2018): Similarly to [31]
FlowSeq (a al., 2019): FlowSeq adopts normalizing flows (Kingma & Dhariwal 2018 as the latent variables to model the mappings from source sentences to latent space.
NAT with Iterative Refinement (iNAT, Lee et al., 2018): iNAT extends the vanilla NAT by iteratively reading and refining the translation. The number of iterations is set 10 for decoding.
Insertion Transformer (InsT, Stern et al. 2019): InsT adopts similar architecture iNAT while generating the sequence by parallel insertion operations. Here, we only consider InsT trained with uniform loss as described in the original paper.
MaskPredict (MaskT, Ghazvininejad et al., 2019); MaskT adopts masked language model (Devlin et al. 2018) to progressively generate the sequence from an entirely masked input. The number of iterations is set to be 10.
Levenshtein Transformer (LevT, Gu et al. 2019): LevT uses similar architectures in InsT and MaskT while generating based on both insertion and deletion operations. We experiment with base and big LevT model (LevT and LevT-bbgg in Tab. 2
We also summarize the parameter size, performance and relative decoding speed of the NAT models introduced Tab. 2 We use the decoding time vanilla NAT to represent one unit of time, and Iters X Pass represents the relative time units used for each model.
As mentioned earlier, we analyze each model by training from both the real and 4 dis- tilled targets. We train the NAT models for
Models
Params
BLEU
Pass
Iters
AT models
the same number of steps the AT mod-
AT-tiny
16M
23.3
n
els. For a fair comparison of the actual abil- ity of each NAT-based model, we test all the models based on greedy decoding without any advanced search algorithms (e.g. length beam (Ghazvininejad et 2019), noisy par- allel decoding (Ma et al. 2219), or re-ranking from the teacher model (Gu et al., 2018)). Notably, the vanilla NAT and FlowSeq output translations with single forward pass, while the remaining models are based on the itera- tive refinement.
AT-small
37M
25.6
AT-
65M
27.1
AT-big
218M
28.2
NAT models
vanilla
71M
11.4
1
FlowSeq
73M
18.6
13
iNAT
66M
19.3
1
& n
InsT
66M
20.9
1
logz Y
MaskT
66M
23.5
1
10
LevT
66M
25.2
LevT-big
220M
26.5
R33
3k
& n
Table 2: AT and NAT models. Number of param- eters and test BLEU when trained on the real data demonstrate model capacity. Iters is number of passes used in decoding for output length n and hy- perparameter k. Pass is relative time used for one pass of decoding.
4.2 ANALYSIS OF THE DISTILLED DATA
We compare different dimensions of the data generated by the four AT models and the real data set in Fig. 3 First, Fig. (a) shows that as the capacity of the AT model increases, the complexity C(d) of the distilled data increases, which indicates that the multi-modality increases as well. At the same time, we observe that F(d) deñindd $3.2 also decreases, showing that the distilled data more faithfully represents the word-level translation distribution of the original data.
https: thub. httpp:://githubbccmm////////

---

Published as a conference paper at ICLR 2020
Source
For more than 30 years. Josef Winkler has been writing from heart, telling of the hardships of his childhood and youth.
Distlled Target
Selt mehr als 30 Jahren schreibr. Joset Winkler aus dem Herzen und erzähit von der Not seiner Kindheit und Jugend.
Target
schreibt
30
seiner
Jugend der
Figure 2: A sampled pair together with its real target from the distilled data of the base-AT model.
Chunks annotated the same colors are approximately aligned with each other.
3.2
2.75
0.475
2.50
3.1
2.25
0.425
3.0
0.400
1.75
2.9
1.50
0.350
tiny
smail
base
big
real
small
base
big
real
tin
small
base
big
real
Figure 3: Complexity C(d) (t more complex), faithfulness F(d) more faithful), training BLEU, and reordering score (t more monotonic alignment) of different distilled sets of WMT14-ENDE.
Second, we plot the BLEU score of the distilled data w.r.t to the real data set in (b) and we observe that the BLEU score of the distilled data from higher-capacity teacher model is higher, which is both intuitive and in agreement with the results on KL divergence.
We also investigate how the relative ordering of words in the source and target sentences is changed during distillation. We use the fuzzy reordering score proposed in Talbot et al. (201T). A larger fuzzy reordering score indicates the more monotonic alignments. As shown in Fig (c), the distilled data has significantly less reordering compared to the real parallel sentences, and the distilled data from a weaker AT teacher is more monotonic than a stronger AT teacher. We also show randomly sampled example in Fig. 2 where compared to the real translation, the AT distilled target is much more monotonically aligned to the source sentence. This has potential benefits that these simpler reordering patterns may be easier to learn for NAT models, but also disadvantages in that it may prevent NAT models from learning complex reordering patterns.
4.3 ANALYSIS OF DISTILLATION STRATEGIES
In 4.2. we have shown that decoding with an model reduces the conditional entropy of the parallel data set, which mitigates multi-modality in the output data. But does the decod- ing method of the AT model affect this change in the data set? We also investigate differ- ent decoding strategies when creating distilled data, using the base Transformer model the teacher and the vanilla NAT model as the student. In Tab. 3 four decoding methods are presented: sampling, sampling within the top-10 10 candidates, beam search, and greedy decod- ing. With the same AT model, the performance of the NAT model differs widely depending on the decoding approach, where distillation with beam search results in the best performance.
We can see that beam search or greedy decoding can reduce the complexity of the real data the most while maintaining high faithfulness. In contrast, sampling based decoding methods less aggressively reduce the modes in the output sequence. This finding is in con- cert with Ott et al. (2018), who demonstrate that be- cause beam search approximately selects the most probable translation, it effectively reduces diversity in the output translations compared to sampling the true distribution.
Decoding Method
C(d)
Fdd)
BLEU
sampling sampling (Top 10)
3,623
3.354
2.411
2,932
14.6
greedy beam search
1.960
2.959
18.9
1.902
2.948
19.5
Table 3: Comparisons of decoding methods on WMT14-ENDE newstest 2014 test set.
4.4 DISTILLED DATA V.S. NAT MODELS
We next examine the relationship between the NAT students and distilled training data from different
AT models. In Fig. 4 demonstrate results for the NAT models listed in $4.1 We the test

---

Published as a conference paper at ICLR 2020
28
Test BLEU
23.74
Transtomer (aanari stal. 2017)
Vanta Nsl tou 20391
22.0
21,05
20
19.3
10.31
18.55
18
Liny
small
base
big
real
Liny
smalll
base
big
real
tiny
small
base
big
real
27.24
Test BLEU
26
24:93
25.69
24.9
25x27
26.42
26.28
20.94
26.47
24
23.0
23.77
23.52
2401
25.15
22
20.90
Trandorrme mel Ivlemmm et 20171 LevT [Gu st 2019)
Trandtrmer (Vanwari 2017)
Tranffrma (Vanwant et 2037)
InsT (Stur st... 2013)
Meali (Chacttineed at 2019)
LevT.ig it 2039)
tiny
small
base
big
real
Dny
small
base
big
real
tiny
small
base
big real
Figure 4: The performance of NAT models of varying capacity trained on both the real and the distilled data from tiny, small, base and big AT models on WMT14-ENDE newstest 2014 test sets.
set performance on real data as a simple metric to measure the capacity the NAT model and arrange the subfigures in increasing order of the performance (left-to-right, top-to-bottom). The results in the figure demonstrate that, interestingly, weaker NAT students prefer distilled data with smaller complexity
measured above in $4.2 The best performance of NAT models from lower capacity ones to higher capacity ones - is achieved with distilled data of lower complexity to higher complexity, i.e. the vanilla NAT model performs best when using the distilled data from small Transformer whereas LevT achieves the best performance when training with the distilled data from a big Transformer. Third, and notably, by simply changing the distilled data set upon which the models are trained, we are able to significantly improve the state-of-the-ee results for models in particular class. For example, FlowSeq increased to 22, by simply changing from the distilled data of Transformer(base) to Transformer(small
Finally, we find that by distilling from big AT model, LevT is able to close the gap with the Transformer (base) with a similar number of parameters. Both LevT and LevT-big achieve the state-of sate--oftth-aa performance for NAT- based models.
IMPROVEMENTS TO KNOWLEDGE DISTILLATION
The previous section shows that the optimal complexity of the dataset is highly correlated with the capacity of the NAT model. In this section, we introduce three techniques that can be used to alter the distilled data match the capacity of NAT model. Specifically, these techniques can be used to simplify the data further (BANs, MoE) for lower-capacity student model or increase faithfulness of the data set (Interpolation) for a higher-capacity student model.
Born-Again Networks. We apply Born-Again neworks (BANs) to create a simplified dataset for NAT models. BANs were originally proposed as self-distillation technique (Furlanello al. 2018) that uses the output distribution of trained model to train the original model. Starting from the real data, we repeatedly train new AT models with decoded sentences from the AT model at the previous iteration. This process is repeated for l: times and yields l: distilled data sets, upon which we perform NAT training and examine how the born-again teachers affect the performance of NAT students.
We conduct experiments using the vanilla NAT model (Gu et al. 2018) (which achieved the best performance with distilled data from a small Transformer in $4.44 and the base Transformer as the AT model. As shown in Fig. [ we can make the following observations: (i) The performance of the base AT model almost remains unchanged during the reborn iterations. (ii) The performance of the vanilla NAT model can be improved by 2 BLEU when using the distilled data from reborn iteration (iii) As the reborn iterations continue, the complexity of the distilled data decreases and becomes constant eventually. Meanwhile, the quality of the distilled data compared to the real data decreases.
8

---

Published as a conference paper at ICLR 2020
28
21.31 21.47
1.88
3.15
20,97
20.77
1.66
Conditinnl Entropy
-300
26
10.5
182
3,00
3,05
25
19
180
base R.1
R2
R3 R4
R5
R6
R7
base R1
R2
R3
R4
R5
R6
R7
base
R1
R2
R3 Rebor Reralions
R4
R.
R6
R7
Robom lterations
Rebern Rerations
Figure 5: Reborn experiments: (from left to right) performance of the base AT model, performance of the vanilla NAT model, C(d) and F(d) of distilled data sets. R-i denotes the i-th reborn iteration.
71
33.04
20.00
20.27
-0
19.
19.43
1.88
Condi ona Entropy diveggence
3,00
Test
1.96
2.98
BLUU
NAT BLEU
1.84
Experts
Experts
Experts
Figure 6: MoE experiments: (from left to right) performance of the base AT model, performance of the vanilla NAT model, Cdd) and F(d) of distilled data sets w.r. the number of experts.
Mixture-of- offture-f--Experts... The mixture-of-expert model (MoE; Shen et al. (019)) learns different ex- perts for diverse machine translation, and different mixture components were shown capture sistent translation styles across examples. Inspired by this, we use one expert from the mixture model translate the training data, which supposed generate single style of translation and reduce the diversity in the original data set. Then we use the best single-expert translations as the distilled data train the vanilla NAT model. Specifically, we follow [Shen et al. (2019)s setup, using the base Transformer model and uniform hard mixture model, varying the number of experts.
In Fig. we observe that the performance the best expert of MoE tends to decrease as the number of experts increases. However, the complexity (Cld) and faithfulness (F(D) of distilled data from different MoF. models has a relatively large variance. Compared to using the distilled data from a plain base AT model, the performance of NAT model is improved by 1.21 BLEU when using the distilled data from the MoE model with the number experts of which produces the distilled data with the least complexity.
Sequence-Leve Interpolation. |4.4|show> shows stronger NAT models (e.g. MaskT, LevT) have the ability learn from the dataset that closer to the real data, and achieve bet- ter performance. We adopt the sequence-level interpolation proposed in Kim & Rush (2016) as a natural way to create better dataset. Different from distillation, interpolation picks the sentence with the highest sentence-level BLEU score w.r.t. the ground truth from T--best beam search hy- potheses. In our experiments, we first run beam search using the base Transformer model with beam size then select the sentences with the highest BLEU score from the top-3 candidates.
d
C(d)
F(d)
BLEU
base base-inter
1.902
2.948
26.94
1,908
2.916
27.32
Table 4: Results w/ and w/o sequence- level interpolation with LevT.
Tab. compares the performance of LevT trained with distilled data from the AT model with the standard distillation or interpolation. We observe that selection with BLEU score from the base AT model (base-inter) improves the performance of LevT ~ 0.4 BLEU while the dataset complexity C(d) does not increase much.
CONCLUSION
In this paper, we first systematically examine why knowledge distillation improves the performance of NAT models. We conducted extensive experiments with autoregressive teacher models of differ- ent capacity and a wide range of NAT models. Furthermore, we defined metrics that can quanti- tatively measure the complexity of parallel data Empirically, find that higher-capacity

---

Published as a conference paper at ICLR 2020
NAT model requires more complex distilled data to achieve better performance. Accordingly, propose several techniques that can adjust the complexity data set to match the capacity of an NAT model for better performance.
REFERENCES
Nader Akoury, Kalpesh Krishna, and Mohit lyyer. Syntactically supervised transformers for faster neural machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 1269-1281, Florence, Italy, July 2019. Association for Com- putational Linguistics. doi: 10. 1...8665/11/119112222 URL https: www acl web org/
anthology/BB 1122
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In International Conference on Learning Representations (ICLR), 2015.
Satanjeev Banerjee and Alon Lavie. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pp. 65-72, 2005.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre- pre-training of deep bidirectional transformers for language understanding. CoRR, abs/ 810.04805, 2018. URL
http::/arxivi Sttt:://axxiv.org/eeee abcp///b///t// 810 04805
Chris Dyer, Victor Chahuneau, and Noah Smith. A simple, fast, and effective reparameterization of IBM Model 2. In NAACI, 2013.
Tommaso Furlanello, Zachary Lipton, Michael Tschannen, Laurent Itti, and Anima Anandkumar. Bom-again neural networks. In International Conference on Machine Learning, pp. 1602-1611, 2018.
Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann Dauphin. Convolutional sequence sequence learning. In Proceedings of the 34th International Conference Machine Learning- Volume 70, pp. 1243-1252. JMLR. org, 2017.
Marjan Ghazvininejad, Omer Levy, Yinhan Liu, and Luke Zettlemoyer. Constant-time machine translation with conditional masked language models. arXiv preprint arXiv:1904.09324 2019.
Jiatao Gu, James Bradbury, Caiming Xiong, Victor O.K. Li, and Richard Socher. Non-autoregresssiv neural machine translation. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver Canada, April 30-May 2018, Conference Track Proceedings, 2018.
Jiatao Gu, Changhan Wang, and Jake Zhao. Levenshtein transformer. In Advances in Neural Infor- mation Processing Systems 33. 2019.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint AXX3::5533.0531 2015.
Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint /7X 2017.
Hideki Isozaki, Tsutomu Hirao, Kevin Duh, Katsuhito Sudoh, and Hajime Tsukada. Automatic eval- uation of translation quality for distant language pairs. In Proceedings of the 2010 Conference Empirical Methods in Natural Language Processing, pp. 944-952. Association for Computational Linguistics, 2010.
Melvin Johnson, Mike Schuster, Quoc V. Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Viégas, Martin Wattenberg, Greg Corrado, Macduff Hughes, and Jeffrey Dean. Google's multilingual neural machine translation system: Enabling zero-shot translation. Trans- actions of the Association for Computational Linguistics, 5:339-351, 2017.
10

---

Published as a conference paper at ICLR 2020
Lukasz Kaiser, Samy Bengio, Aurko Roy, Ashish Vaswani, Niki Parmar, Jakob Uszkoreit, and Noam Shazeer. Fast decoding in sequence models using discrete latent variables. In International Con- ference Machine Learning, pp. 2395-24044 2018.
Yoon Kim and Alexander M Rush. Sequence-level knowledge distillation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 1317-1327, 2016.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1x1 convolutions, In Advances in Neural Information Processing Systems, pp. 10215-10224, 2018.
Jason Lee, Elman Mansimov, and Kyunghyun Cho. Deterministic con-autoregressive neural se- quence modeling by iterative refinement. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 1173-1182, 2018.
Percy Liang, Hal Daumé III, and Dan Klein. Structure compilation: trading structure for features. In ICML, pp. 592-599, 2008
Xuezhe Ma, Pengcheng Yin, Jingzhou Liu, Graham Neubig, and Eduard Hovy. Softmax q- distribution estimation for structured prediction: A theoretical interpretation for raml. arXiv preprint arXiv: 170:. 7et.:071556. 2017.
Xuezhe Ma. Chunting Zhou, Xian Li, Graham Neubig, and Eduard Hovy. Flowseq: Non- autoregressive conditional sequence generation with generative flow. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing, Hong Kong, November 2019.
Aaron Oord, Yazhe Li, Igor Babuschkin, Karen Simonyan, Oriol Vinyals, Koray Kavukcuoglu, George Driessche, Edward Lockhart, Luis Cobo, Florian Stimberg, et Parallel wavenet: Fast high-fidelity speech synthesis. In International Conference on Machine Learning, pp. op.3915-3923, 2018.
Myle Ott, Michael Auli, David Grangier, and Mare' Aurelio Ranzato. Analyzing uncertainty in neural machine translation. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmmsssan, Stockholm, Sweden, July 10-15, 2018, pp. 3953-3962, 2018. URL http: http:///weecooooos.oppprpsss/s/s html
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng. David Grangier, and Michael Auli. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of NAACL-HLT 2019: Demonstrations, 2019.
Nicolas Papernot, Patrick McDaniel, Xi Wu, Somesh Jha, and Ananthram Swami. Distillation defense to adversarial perturbations against deep neural networks. In 2016 IEEE Symposium on Security and Privacy (SP), pp. 582-597. IEEF, 2016.
Maja Popović. chrf: character n-gram f-score for automatic mt evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pp. 392-395, 2015.
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics ( (Volmme 1: Long Papers), pp. 1715-1725, Berlin, Germany, August 2016. Association for Computational Linguistics. doi: 10.18653/v1/P16- orc//anthelogyyPI16
URL httthtt::/// www welweb.
Chenze Shao, Yang Feng, Jinchao Zhang, Fandong Meng, Xilin Chen, and Jie Zhou. Retriev- ing sequential information for non-autoregressive neural machine translation. arXiv preprint 1rXiv:1906.09444 2019.
Tianxiao Shen, Myle Ott, Michael Auli, et al. Mixture models for diverse machine translation: Tricks of the trade. In International Conference Machine Learning, pp. 9719--75522 2019.
11

---

Published as a conference paper at ICLR 2020
Raphael Shu, Jason Lee, Hideki Nakayama, and Kyunghyun Cho. autoregressive neural machine translation with deterministic inference using a delta posterior. arXiv preprint arXh::1908.07/68 2019.
Latent-variable non-
Matthew Snover, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul. A study of translation edit rate with targeted human annotation. In In Proceedings of Association for Machine Translation the Americas, pp. 223-231, 2006.
Milos Stanojevic and Khalil Simaan. Beer: Better evaluation as ranking. In Proceedings the Ninth Workshop on Statistical Machine Translation, pp. 414-419, 2014.
the
Mitchell Stern, Noam Shazeer, and Jakob Uszkoreit. Blockwise parallel decoding for deep toregressive models. In Advances in Neural Information Processing Systems, pp. 10107-10116, 2018.
Mitchell Stern, William Chan, Jamie Kiros, and Jakob Uszkoreit. Insertion transformer: Flexible sequence generation via insertion operations. arXiv preprint arXiv:1902.0322 2019.
David Talbot, Hideto Kazawa, Hiroshi Ichikawa, Jason Katz-Brown, Masakazu Seno, and Franz J Och. lightweight evaluation framework for machine translation reordering. In Proceedings of the Sixth Workshop Statistical Machine Translation, pp. 12-21. Association for Computational Linguistics, 2011.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Chunqi Wang. Ji Zhang, and Haiqing Chen. Semi-autoregressive neural machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp- 479-488, 2018.
Yiren Wang, Fei Tian, Di He, Tao Qin, ChengXiang Zhai, and Yan Liu. Non-autoregressinn machine translation with auxiliary regularization. arXiv preprint arXiv:1902.10240 2019.
Bingzhen Wei, Mingxuan Wang, Hao Zhou, Junyang Lin, and Xu Sun. Imitation learning for non- autoregressive neural machine translation. arXiv preprint arXiv: 1906, 02041, 2019.
12

---

Published as a conference paper at ICLR 2020
A EXPERIMENTAL DETAILS
A.1 AT MODELS
Model All the AT models are implemented based on the Transformer model using fairseq (Ou et al. 2019), and we basically follow the fairseq examples to train the transformers Following the notation from Vaswani al. (2017), we list the basic parameters of all the AT model we used:
Models
tiny
small
base
big
dmodel
256
512
512
1024
dhidden
1024
1024
2048
4096
Taaayers
3
6
Theaads
8
8
16
Pdropout
0.1
0.1
0.3
0.3
Table 5: Basic hyper-parameters of architecture for AT models.
Training For all experiments, we adopt the Adam optimizer (Kingma & Ba, 2014) using 81 = 0.9, B2 0.98, = le 8. The learning rate is scheduled using inverse_sqrs with a maximum learning rate 0.0005 and 4000 warmup steps. We set the label smoothing 0.1. All the models are run on GPUs for 300, 000 updates with an effective batch size 32, 000 tokens. The best model is selected based the validation loss except for FlowSeq which uses valid BLEU score.
Decoding After training, we use beam-search with a fixed beam size for all AT models to create the distilled dataset. We use length normalization without length penalty.
A.2 NAT MODELS
Model Tab. 2 also lists all the NAT models we test in this work. In general, all the NAT models ex- cept FlowSeq and LevT-big adopts a similar architecture and hyper-parameters as the Transformer- base (see Tab. 5 LevT-big is a naive extension of the original LevT model with comparable parameter setting as Transformer-big (Tab. For FlowSeq. we use the base model (FlowSeq-base, described in (Ma et al., 2019). We re-implemented the vanilla NAT simplified version of Gu cet al. (2018) where instead modeling fertility as described the original paper, we monotonically copy the encoder embeddings to the input of the decoder. All the models except InsT require the additional module to predict the length of the output sequence, or the number of placeholders to be inserted, which is implemented as standard softmax classifier over the lengths of [0, 256). For Lev'T we also have a binary classifier to predict the deletion of the incorrect tokens.
Training Similar to the AT models, all the NAT models trained using the Adam optimizer with the same learning rate scheduler, in which the warmup steps are set to 10, 000. We train the FlowSeq model on 32 GPUs with batch size as 2048 sentences, while all the other models are trained on GPUs with an effective batch size of 64, 000 tokens. Note that, the batch sizes for training NAT is typically larger than the AT model, which improves final results. There are also specialized training settings for each models:
iNAT Lee et al. 2018): following the original paper, we train the iNAT model jointly with iterations of refinement during training. For each iteration, the model has the 50% probability to learn as a denoising autoencoder, and the rest of the probability to learn from the model's own prediction.
InsT (Stern et al. 2019:
: in this work, we only consider training the Insertion Transformer (InsT) using the slot-loss based on the uniform loss function (Stern et al., 2019). That is, we assign equal probabilities to all the insertable tokens inside each slot.
MaskT (Ghazvininejad et al. 2019): following the original paper, we train the model as a typical masked language model where the ratio of masked tokens is sampled from ~ 100%.
13

---

Published as a conference paper at ICLR 2020
LevT (Gu et al. 2019): in this work, only consider sequence generation tasks, which means the training of LevT is very similar to InsT. We use sentences with randomly deleted tokens to learn insertion, and learn deletion based the model's own prediction.
Decoding For a fair comparison over all the NAT models, we use greedy decoding for all the models without considering any advanced decoding methods such searching re-ranking from a teacher model. For the vanilla NAT and FlowSeq, decoding is quite straight-forward and simply picks the arg max every position. For iNAT and MaskT, we fix the decoding steps 10. Both InsT and LevT decode in an adaptive number of iterations, and we set the maximum iterations for both models be 10. special EOS penalty that penalizes generating too short sequences is tuned based on the validation set for both InsT and LevT.
For all models, final results are calculated using tokenized BLEU score.
B REAL DATA STATISTICS
The detailed dataset split for WMT14 En-De is shown in Tab. 6 In Fig, 7 we also plot the histogram of the conditional entropy of each pair sentences H(y x: the real parallel data and different distilled data sets from the big-AT, base-AT, small-AT and tiny-AT respectively. It shows that the distribution of the sentence-level conditional entropy differs widely. The mode of H(y x)) in the real data is the highest and follows by distilled data from the big-AT, base-AT, small-AT and tin--AT. This observation aligns with the complexity value C(d) proposed in (32
Dataset
Train
Valid
Test
Vocabulary
WMT" En-De 4,500,966
3000
3003
37,009
Table 6: Dataset statisties for WMT14 En-De.
distill tiny
distill small
1.0
distill base
distill big
real data
0.8
Density
0.4
0.2
Sentence-level Conditional Entropy Hlylx)
Figure 7: Density of conditional entropy C(d) of each sentence pairs in different distilled data sets and the real data.
C ADDITIONAL METRICS
In Figure we also showed results with different metrics together with BLEU scores considering
that BLEU scores sometimes cannot fully capture the changes in the system. We considered
additional metrics in our experiments: METEOR (Banerjee & Lavie 2005). RIBES [Iszaki et al.
2010), ChrF (Popović 2015) TER (Snover al. 2006), and BEER (Stanojevic & Simaan (014). Not surprisingly, we find that all the metrics are correlated with the original BLEU scores quite well showing similar trend as discussed earlier.
14

---

Published as a conference paper at ICLR 2020
22
2400
8:377
0.34
3.155
0.140
0.33
1999
250
3.400
185
0.186
0.55
280
311U
METOON
0.18
smai
0390
tny
smail
5300
big
18y
small
base
big
0.515
0.503
0.500
0,202
0.55
0.562
0.51
0.553
0.50
0.545
6.46
0,542
0.80
0.545
12:K
18y
small
basc
b5g
5ny
small
b250
big
thn
small
base
big
Figure 8: The performance of variant measure (BLEU t. METEOR t. RIBES t. ChrF f. TER J. BEER f) for the vanilla NAT model trained on the distilled data from tiny, small, base and big AT models on WMT14-ENDE newstest 2014 test sets.
D SYNTHETIC DATA WITH ACCESS TO THE TRUE DISTRIBUTION
D.1 BACKGROUND: BAYESIAN DECISION THEORY
Bayesian decision theory is fundamental statistical approach the problem of pattern classifica- tion, which provides a principled rule of finding the optimal classification decision using probability and losses that accompany such decisions.
In the problem of structured prediction (Ma et al. 2017). let æ denote the input sequence and y denote the output label sequence. Let H. denote all the possible hypothesis functions from the input to the output space: H (h X - } Let r(yle (a) denote the conditional risk on the input æ, which is the expected loss of predicting y based on the posterior probabilities:
r xc) = E [Ly, y²)],
(5)
where L(y, 3²) the loss function that penalizes predicting the true target y' as y. The classifica- tion task aims to find a hypothesis function h that minimizes the overall risk R given by
R(h) (r(
(6)
This is known as the Bayes risk. To minimize the overall risk, obviously we need to minimize the conditional risk for each input æ. The Bayesian decision rule states that the global minimum of R(h) is achieved when the classifier make predictions that minimize each conditional risk given æ and this gives the Bayes optimal classifier:
(x)) = arg min myyl()
wEY
Let us consider two loss functions defined Eq. 5 First is the sequence-level loss Leeg (y.y') y')
1 - Il y'), then in this case the Bayes classifier is:
We (ax) = arg max P(yla:)
(8)
yEy
which is the most probable output label sequence given the input sequence æ..
15

---

Published as a conference paper at ICLR 2020
Second let us consider the token-level loss (30k(1) -Et= 1 I(ye v6), i. the sum of zero-one loss at each time step. We have:
bo
arg min Ep(y) (22
MEY
arg max E Epyy' 2)) 12 Ilyl = w/)l
DEY
arg max Σ Ep (z) [[ye 3(1
yEy
arg max
Σ t-l Ep m) [T(Ee = 3()
yEy
arg max (()]
yEy
t=1
This suggests that the Bayes classifier finds the most probable label at each time step given the input sequence.
D.2 EXPERIMENTAL SETUPS AND ANALYSIS
To study how training data affects the performance of a weaker classifier, we construct a Hidden Markov Model (HMM) by sampling the parameters of the transition and emission probabilities uniformly within (0, a) and (0. bj respectively. A higher value of and indicates an HMM model with higher uncertainty. We refer this HMM as the "true HMM" as our real data generator. Next we consider weaker classifier that uses low-dimension bidirectional-LSTI. (Bi-LSTM) to encode the input sequence and individual softmax functions at each time step to predict labels independently, which is referred the "Bi-LSTM" classifier. Obviously, the Bi-LSTM classifier is not able to model the dependencies between output labels embedded in the HMM, and equivalent to a simplified non-autoregressive generation model.
We generate the real training data Dreat {(1) (ar1,,))..... y1), (æN yN)) of size N by sampling from the joint probability of the true HMM. Similarly sample Niest data points the test data and Neatid data points as the validation data. We evaluate the classifier's token-level accuracy tace and
sequence-level accuracy sace on the test data respectively, where tace MMest ET [(M().)'=v)) T'N
and sace = Nusst I(h(a.)=y.) These two metrics correspond to the token-level loss Liok and sequence-level loss Leeg on each data point of the test data.
Noeat
First, we use (egg(a)) (x) generate the distillation labels y' from the true HMM, which corresponds to applying the Viterbi decoding to each æ; in Dreat- The training data set D Deeg is created with (æ;, v.). Next,
h (x) to generate the distillation labels ý and create the training data Diok of
(x, Ú.). To generate y. we apply the forward-backward algorithm to each x; in Dreat and obtain P(yl 22)). We take arg over the label space L: of arg max P(y ((y)( x.EC
We use these three training data (Dreals Dtok: Deeg) to train the Bi-LSTM classifier respectively. We repeat the experiment for 50 times by constructing 50 HMM models with different random seeds as the data generator. We find that when evaluating with the token-level accuracy tacc, mod- els trained with Diok yields the best performance (Bi-LSTM trained with Diok win 97.6% runs); when evaluating with the sequence-level accuracy sacc, models trained with Daeg yields the best performance (Bi-LSTM trained with Daeg win 98.5% runs). This is because the Bi-LSTM clas- sifier has difficulty modeling the true data distribution defined by an HMM. On the other hand, it casier for the Bi-LSTM classifier model the distributions of D Dseg and Dtok- Data sets Dseg and Drok define deterministic conditional distributions over the input data, which are much sim- pler than the real data distribution. By definition, Dtok created by the optimal Bayes classifier hish (æ), this means that the Bi-LSTM classifier trained with Dtok can better capture the distribu- tion of P(y:(æ) = max (uu (x). which can generalize better to the test data when evaluated with
the teken-level accuracy. Similarly, Bi-LSTM trained with D Dseg performs better on the test data with the sequence-level metric.
This corroborates our observation in machine translation task that NAT has difficulty in modeling the real conditional distribution of true sentence pairs. However, when using the distilled data translated
16

---

Published as a conference paper at ICLR 2020
from a pretrained autoregressive model with beam- decoding, it performs better the test set when evaluated with the BLEU score metric.
17


Reviews:
[{"review_id": "r1eumhbnYH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper analyses recent distillation techniques for non-autoregressive machine translation models (NAT). These models use a autoregressive teacher (AT), which typically perform better. However, AT models can not be parallelized that easily such as the NAT models. The distillation has the effect of removing modes from the dataset which helps the NAT models as they suffer from the averaging effect of maximum likelihood solutions. The authors analyze why such distillation is needed and what the effect of the complexity of the training set is and further propose 3 methods to adjust the complexity of the teacher to the complexity of the NAT model.\n\nThe paper is well written and easy to understand and the experiments are exhaustive and well done. I think the analysis using the synthetic dataset is nice, but I am not sure how surprising the results are. I think most of it has to be expected given the properties of maximum likelihood estimation. Hence, I think the contribution of the paper is a bit limited. I am, however, not an expert in the field to fully judge the contribution and will therefore put low confidence on my review. ", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "BygmuCAKjB", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors investigate non-autoregressive translation (NAT). They specifically look into how using different auto-regressive translation (AT) models for knowledge distillation impacts the quality of NAT models. The paper is well organised and the experiments are sound and interesting, shedding light on an aspect of NAT models that's been rather dismissed as secondary until now: the impact of varying AT knowledge distillation.\n\nFirst, so as to better please those out there who are more \"state-of-the-art\" inclined, I suggest the authors to better emphasise their improvements. Results obtained by their analysis can lead to improvements as stated in the last paragraph of Section 4. This could be better stressed in the introduction and it would make the main take-away messages from the paper stronger.\n\nOn a more general note, I would like to know how robust these models are. I understand this is a problem NAT and machine translation papers have in general, but I would still like to suggest the authors train each model from scratch multiple times using different random seeds, and report mean and variance of their results (i.e. BLEU). Although I understand that training each AT model 4 times and each NAT model 4x4 times (multiple times for each AT model trained) is unfeasible, you could still report mean and variance separately for AT and NAT, and simply choose one out of the 4 trained AT models per architecture to perform distillation. I recommend training each model at least 3 times and reporting mean BLEU and variance.\n\nI would also recommend using other MT metrics (e.g. chrF, METEOR, BEER, etc.), since BLEU is a comparatively weak metric in terms of correlations with human judgements. For translations into German, look into characTER, chrF and BEER. For more information on the different metrics available, how they correlate with human judgements, and which better apply to different target languages / language pairs, please refer to the WMT evaluation/metrics shared tasks. [1]\n\nI have a few general comments and suggestions:\n- In Section 3.1, when introducing the NAT model (simplified version of Gu et al., (2019)), be more specific. I would like to see an actual description of what the modifications consist of in more detail: is there a sentence length prediction step? what happens when source length is different from source length? etc.\n- In Section 3.2, right after Equation (3), the authors refer to \"x and y\" in an inverted manner, please fix it.\n- In Section 4.3, you mention that two of the decoding methods used involve \"random sampling\", which I find misleading. You probably mean sampling according to the model distribution p(y_t | y_{<t}) for all t, which is not random. I suggest you simply remove the word \"random\" and mention that you use \"sampling\" and \"sampling within the top-10 candidates\". Also, when you sampling using the top-10 candidates, do you simply re-normalise the probability mass to include only the top-10 candidate tokens?\n- As a suggestion, Tables and Figures could be more self-contained. There is always a compromise between conciseness and clarity, added by the fact that the page limit makes things even harder. However, I would recommend including more information in Table/Figure captions, especially in Figure 3 and Tables 3 and 4. Try to at least mention the training/evaluation data (dev or test?). Ideally one should understand it from reading the abstract and carefully reading the caption.\n- In Appendix A, you mention that you select models according to validation loss. Is that really the case? If so, why? I am not sure whether validation loss (i.e. word cross-entropy on the validation set) should correlate so well with translation quality (or sentence-level metrics such as BLEU).\n\n[1] http://www.statmt.org/wmt19/metrics-task.html", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "B1gjfhYBOr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "[EDIT: Thanks for the response! I am updating my score to 8 given the rebuttal to my and other reviewers' questions]\n\n--------------------------------------------\nSummary:\n--------------------------------------------\nThis paper studies knowledge distillation in the context of non-autoregressive translation. In particular, it is well known that in order to make NAT competitive with AT, one needs to train the NAT system on a distilled dataset from the teacher model. Using initial experiments on EN=>ES/FR/DE, the authors argue that this necessity arises from the overly-multimodal nature of the output distribution, and that the AT teacher model produces a less multimodal distribution that is easier to model with NAT. \n\nBased on this, the authors propose two quantities that estimate the complexity (conditional entropy) and faithfulness (cross entropy vs real data), and derive approximations to these based on independence assumptions and an alignment model.  The translations from the teacher output are indeed found to be less complex, thereby facilitating easier training for the NAT student model.\n\n--------------------------------------------\nStrengths:\n--------------------------------------------\n- Careful initial experiments to motivate the study.\n\n- The metrics under consideration (entropy/faithfulness) are carefully derived, and while some of the assumptions may be overly simplifying (e.g. conditional independence assumptions), they are reasonable enough such that the directionality is likely to be correct.\n\n- Thorough comparison of various NAT models. This will benefit the community greatly since to my knowledge, there has been no empirical head-to-head comparison of the different NAT methods that exist today.\n\n- Extensive experiments across various settings, e.g. varying the teacher model size, varying the decoding strategies, investigating BAN/MoE/Sequence-level interpolation, etc. \n\n--------------------------------------------\nWeaknesses:\n--------------------------------------------\n- It would have been interesting to consider a synthetic data setting such that one has access to the true underlying data distribution, such that approximations are not necessary.\n\n- While translation is an important application of non-autoregressive generation, it would have also been interesting to study this in other seq2seq regimes such as summarization where the conditional entropy would presumably be even higher. (However this would complicate things like calculation of alignment probabilities, etc.)\n\n--------------------------------------------\nOther Questions/Comments:\n--------------------------------------------\n- p(y_t | l_i) coming from a token frequency distribution seems a bit too simple. Do the plots change if you model p(y | l_i) with a full language model that conditions on l_i?\n\n- It's interesting to note that in Figures 1b,1c,1d, there is much more overlap between the romance languages es/fr, which are more closely related to each other. \n\n- I am not sure I agree with \"C(d) reflects the level of multi-modality of a parallel corpus\". One can certainly imagine a distribution which is unimodal but has high entropy...\n\n- It seems like that we want a teacher model with low complexity and high faithfulness. Have the authors tried training a teacher model to directly target this? The usual MLE objective obviously targets faithfulness, but perhaps one could use RL techniques to optimize for low complexity as well.\n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Main content:

Blind review #3 summarized it well, as follows:

This paper studies knowledge distillation in the context of non-autoregressive translation. In particular, it is well known that in order to make NAT competitive with AT, one needs to train the NAT system on a distilled dataset from the teacher model. Using initial experiments on EN=>ES/FR/DE, the authors argue that this necessity arises from the overly-multimodal nature of the output distribution, and that the AT teacher model produces a less multimodal distribution that is easier to model with NAT. 

Based on this, the authors propose two quantities that estimate the complexity (conditional entropy) and faithfulness (cross entropy vs real data), and derive approximations to these based on independence assumptions and an alignment model.  The translations from the teacher output are indeed found to be less complex, thereby facilitating easier training for the NAT student model.

--

Discussion:

Questions were mostly about how robust the results were on other language pairs and random starting points. Authors addressed questions reasonably.

One low review came from a reviewer who admitted not knowing the field, and I agree with the other two reviewers.

--

Recommendation and justification:

I think papers that offer empirically support for scientific insight (giving an "a-ha!" reaction), rather than massive engineering efforts to beat the state of the art, are very worthwhile in scientific conferences. This paper meets that criteria for acceptance.
Decision: Accept (Poster)