Understanding and Improving Lexical Choice in Non-Autoregressive Translation
Authors: Liang Ding
Longyue Wang
Xuebo Liu
Derek F. Wong
Dacheng Tao
Zhaopeng Tu
Venue: ICLR 2021 Poster (2021)

Abstract: Knowledge distillation (KD) is essential for training non-autoregressive translation (NAT) models by reducing the complexity of the raw data with an autoregressive teacher model. In this study, we empirically show that as a side effect of this training, the lexical choice errors on low-frequency words are propagated to the NAT model from the teacher model. To alleviate this problem, we propose to expose the raw data to NAT models to restore the useful information of low-frequency words, which are missed in the distilled data. To this end, we introduce an extra Kullback-Leibler divergence term derived by comparing the lexical choice of NAT model and that embedded in the raw data. Experimental results across language pairs and model architectures demonstrate the effectiveness and universality of the proposed approach.  Extensive analyses confirm our claim that our approach improves performance by reducing the lexical choice errors on low-frequency words.  Encouragingly, our approach pushes the SOTA NAT performance on the WMT14 English-German and WMT16 Romanian-English datasets up to 27.8 and 33.8 BLEU points, respectively. 

Published as a conference paper at ICLR 2021
UNDERSTANDING AND IMPROVING LEXICAL CHOICE IN NON- AUT-RRRRERESSSVE TRANSLATION
Liang Ding', ; Longyue Wang', Xuebo Liu", Derek F. Wong", Dacheng Tao' & Zhaopeng Tuf
'The University of Sydney Tencent AI Lab University of Macau {1din3097, dacheng tsydney au, nlp2ct
xuebo sueeoo@gaill com,
Wvinnylywang zptu etencent. com, derekfw@um edu. com
ABSTRACT
Knowledge distillation (KD) is essential for training non-autoregressive translation (NAT) models by reducing the complexity of the raw data with an autoregressive teacher model. In this study, we empirically show that side effeet of this training, the lexical choice errors on low-frequency words are propagated to the NAT model from the teacher model. To alleviate this problem, we propose expose the raw data to NAT models to restore the useful information of low fow-frequency words, which Kullback-Leibles divergence term derived by comparing the lexical choice of NAT model and that embedded in the raw data. Experimental results across language pairs and model architectures demonstrate the effectiveness and universality of the proposed approach. Extensive analyses confirm our claim that our approach improves performance by reducing the lexical choice errors on wwwfrrquency words. Encouragingly, our approach pushes the SOTA NAT performance on the WMT14 English-German and WMT16 Comanian-Englist datasets up to 27.8 and 33.8 BLEU points, respectively.
missed in the distilled data. To this end, introduce an extra
INTRODUCTION
When translating word, translation models need to spend substantial amount of its capacity in disambiguating its sense the source language and choose a lexeme in the target language which adequately express its meaning (Choi et al. 2017 Tamchyna, 2017). However, neural machine translation (NMT) has a severe problem on lexical choice, since it usually has mistranslation errors
on low-frequency words (Koehn & Knowles 2017:
Nguyen & Chiang 2018 Gu al. 2020
In recent years, there has been a grow- ing interest in non-autoregressive transla- tion (NAT, Gu et al., 2018). which im- proves decoding efficiency by predicting all tokens independently and simultane- ously. Well-performed NAT models are generally trained on synthetic data distilled by autoregressive translation (AT) teach- ers instead of the raw training data (Fig- ure a)) (Stern et al., 2019 Lee et al.
SRC
RAW-TGT
The going at Newmarket is soft Today, Newmargot's runway soft
KD-TGT
SRC
W $ M (2
RAW-TGT
The Newmarket stakes is always
KD-TGT
The Newmarquette races always
SRC
ZE T 4 # lif 45 3 HBth dp #...
RAW-TGT KD-TGT
I've ... in the 3.45 at Newmarket.
2018 Ghazvininejad et al. 2019 Gu al.
I... at 3:45 a.m. in Newmarquite.
2019 Hao et al.) 2021). Recent studies have revealed that knowledge distillation (KD) reduces the modes (i.e. multiple lex- ical choices for a source word) in the raw data by re-weighting the training exam- ples (Furlanello et al.) 20188] TTang et al.) 2020), which lowers the intrinsic uncertainty (Ott et al. 2018) and learning difficulty for NAT (Zhou ctt al. 2020 Ren et al., 2020). However, the side effect of KD has not been fully studied. In this work,
Table 1; All samples that contain the source word "HH 1/ #4 in raw and distilled training corpora, which are different in target sides (RAW-TGT vs. KD-TGT).
'Work was done when Liang Ding and Xueho Liu were interning at Tencent AT Lab.

---

Published as a conference paper at ICLR 2021
investigate this problem from the perspective of lexical choice, which is at the core of machine translation.
We argue that the lexical choice errors of AT teacher can be propagated to the NAT model via the distilled training data. To verify this hypothesis, we qualitatively compare raw and distilled training corpora. Table 1 lists all samples whose source sentences contain the place name "M 44 In the raw corpus ("RAW-TGT"), this low-frequency word totally occurs three times and corresponds to correct translation "Newmarket". However, in the KD corpus ("KD-TGT"), the word is incorrectly translated into a person name "Newmargot" (Margot Robbie is an Australian actress) or organization name "Newmarquettte" (Marquette an university Wisconsin) or even invalid one "Newmarquite".
Motivated by this finding, we explore NAT from the lexical choice perspective. We first validate our hypothesis by analyzing the lexical choice behaviors of NAT models ($3). Concretely, we propose new metric AoLC (accuracy of lexical choice) to evaluate the lexical translation accuracy of given NAT model. Experimental results across different language pairs show that NAT models trained on distilled data have higher accuracy of global lexical translation (AoLCt), which results in better sequence generation. However, fine fine-grained analyses revealed that although KD improves the accuracy on high-frequency tokens, it meanwhile harms performance on low-frequency ones (Low freq. AoLCl). And with the improvement of teacher models, this issue becomes more severe. We conclude that the lexical choice of the low-free oowffrequeccy tokens is typical kind of lost information when using knowledge distillation from AT model.
In order to rejuvenate this lost information in raw data, we propose to expose the data to the training of NAT models, which augments NAT models the ability to learn the lost knowledge by themselves. Specifically, we propose two bi-lingual lexical-level data-dependent priors (Word Aligument Distribution and Self-Distilled Distribution) extracted from raw data, which is integrated into NAT training via Kullback-Leibles divergence. Both approaches expose the lexical knowledge in the raw data to NAT, which makes it learn to restore the useful information of cow-frequency words to accomplish the translation.
We validated our approach on several datasets that widely used in previous studies (i.e. WMT14 En-De, WMT16 Ro-En, WMT17 Zh-En, and WAT17 Ja-En) and model architectures (i.e. MaskPre- dict (Ghazvininejad et al., (2019) and Levenshtein Transformer (Gu et al. 2019)). Experimental results show that the proposed method consistently improve translation performance over the standard NAT models across languages and advanced NAT architectures. The improvements come from the better lexical translation accuracy now-frequency tokens in particular) of NAT models (AoLCT), which leads to less mii-trraslltioos and low- wwwffreueeccy words prediction errors. The main contributions of this work are:
Our study reveals the side effect of NAT models' knowledge distillation on low-frequency lexicons, which makes the standard NAT training on the distilled data sub-optimal.
We demonstrate the necessity of letting NAT models learn to distill lexical choices from the raw data by themselves.
We propose an simple yet effective approach accomplish this goa which are robustly applicable to several model architectures and language pairs.
2 PRELIMINARIES
2.1 NON-AUTOOEGRRESSIVE TRANSLATION
The idea of NAT has been pioneered by Gu et al. 2018). which enables the inference process goes in parallel. Different from AT models that generate each target word conditioned on previously generated ones, NAT models break the autoregressive factorization and produce target words in parallel. Given a source sentence x, the probability of generating its target sentence y with length T is calculated as:
T
p(y v(yl))=1x = PT x: #) II 0)
(1)
Cooee is available httpp::/ggthub..coo httmm//apaad///cccc /ICCAT
2

---

Published as a conference paper at ICLR 2021
where (-) is separate conditional distribution to predict the length of target sequence. During training. the negative loglikelihood loss function of NAT is accordingly LNAT(O) = log p x). To bridge the performance gap between NAT and AT models, variety approaches have been proposed.
such as multi-turn refinement mechanism (Lee et al.) 2018; Ghazvininejad et al.
2019:
Gu et al.
2019 Kasai al. 2020), rescoring with AT models (Wei 2019 Ma et al. 2019
Sun al.
2019). adding auxiliary signals to improve model capacity (Wang et al. 2019 Ran et al.,
2019 Guo
al. 2019| Ding et al. 2020), and advanced training objective (Wei et al. 2019 Shao al. 2019] Ma et al.) 20020). Our work is complementary to theirs: while they focus on improving NAT models trained on the distilled data, we refine the NAT models by exploiting the knowledge in the raw data.
Sentence-Level Knowledge Distillation NAT models suffer from the multimodality prob- lem, in which the conditional independence assumption prevents model from properly capturing the highly multimodal distribution of target translations. For example, one En- glish source sentence "Thank you." can be accurately translated into German as any one of "Danke.". "Danke schön." " or "Vielen Dank.", all of which occur in the training data. To alleviate this problem. Gu et al. (2088) applied sequence-level KD (Kim & Rush] (2016) to construct synthetic corpus, whose target sentences are generated by an AT model trained on the raw data, as shown in Figure Ja). The NAT model only trained on distilled data with lower modes, which makes it easily acquire more deterministic knowledge (e.g. one lexical choice for each source word). While separating KD and model training makes the pipeline simple and efficient, it has one potential threat: the re-weighted samples distilled with AT model may have lost some important informa- tion. Lee et al. (2020) show that distillation benefits the sequence generation but harms the density estimation. In this study, exploit to bridge this gap by exposing the raw data to the training of NAT models, as shown in Figure bb).
Raw Data
Raw Data
AT Model
AT Model
KD Data
KD Data
NAT Model
NAT Model
(a) Separate from Raw
(b) Bridging with Raw
Figure 1: Comparison of existing two-step and our proposed NAT training scheme.
2.2 EXPERIMENTAL SETUP
Datasets Experiments were conducted on four widely-used translation datasets: WMT14 English- German DEn-De, Vaswani et al. 2017). WMT16 Romanian-English English (Ro-En, Gu et al. 2018), WMT17 Chincse-English (Zh-En, Hassan et al. 2018), and WAT17 Japanese-Englin (Ja-En, Morishita et al. 2017), which consist of 4.5M, 0.6M, 20M, and 2M sentence pairs, respectively. We use the same validation and test datasets with previous works for fair comparison. To avoid unknown words, we preprocessed data via BPE (Sennrich et al. 2016) with 32K merge operations. The GIZA++ (Och & Ney 2003) was employed to build word alignments for the training datasets. We evaluated the translation quality with BLEU (Papineni al. 2002).
NAT Models We validated our research hypotheses on SOTA NAT models:
MaskPredict (MaskT, Ghazvininejad et al. 2019) that uses the conditional mask LM (Devlin et al. 2019) to iteratively generate the target sequence from the masked input. We followed its optimal settings to keep the iteration number be 10 and length beam be 5, respectively.
Levenshtein Trunsformer (Lev'T, Gu et al. 2019) that introduces three steps: deletion, placeholder prediction and token prediction. The decoding iterations in LevT adaptively depends on certain conditions.
For regularization, we tune the dropout rate from [0.1, 0.2, 0.3] based on validation performance in each direction, and apply weight decay with 0.01 and label smoothing with = 0. We train batches of approximately 128K tokens using Adam (Kingma & Ba 2013). The learning rate warms up to 5 x 10 in the first 10K steps, and then decays with the inverse square-root schedule. We followed the common practices (Ghazvininejad et al.) 2019: Kasai et al. (2200) to evaluate the translation performance on an ensemble of top checkpoints avoid stochasticity.
3

---

Published as a conference paper at ICLR 2021
En-De
Zh-En
Ja-En
Dataset
CoD
AoLC
BLEU
CoD
AoLC
BLEU
CoD
AoLC
BLEU
Raw
3.53
74.3
24.6
5.11
68.5
22.6
3.92
73.1
27.8
KD (BASE) 1.85
75.5
26.5
3.23
71.8
23.6
2.80
74.7
28.4
KD (BIG)
1.77
76.3
27.0
3.01
72.7
24.2
2.47
75.3
28.9
Table 2: Results of different metrics the MaskT model trained on different datasets. "KD (X)" denotes the distilled data produced by the AT model with X setting. "CoD" denotes the complexity of data metric proposed by Zhou al. (2020), and "AoLC" is proposed metric to evaluate the accuracy of lexical choice in NAT models.
AT Teachers We closely followed previous works on NAT to apply sequence-level knowledge distillation (Kim & Rush 2016) to reduce the modes of the training data. More precisely, to assess the effectiveness of method under different of AT teachers, we trained three kinds of Monnformer-BAsee Transformer-Bi and Seafffomee-STRRONE The main results employ LARGE for all directions except Ro-En, which is distilled by BASE The architectures of Transformer- BIG and Transformer- STrOONG are unchanged, but STRONG utilizes large batch (458K tokens) training strategy.
Transformer (Vaswani et al., 2017) models, including
UNDERSTANDING LEXICAL CHOICE NAT MODELS
3.1 EVALUATING LEXICAL CHOICE OF NAT MODELS
Recently, Zhou et al. 2020) argue that knowledge distillation is necessary for the uncertain nature of the machine translation task. Accordingly, they propose a metric to estimate the complexity of the data (CoD), which is driven from an external word alignment model. They reveal that the distilled data indeed less complex, which facilitates easier training for the NAT model. Inspired by this, we propose a metric to measure the lexical level accuracy of model predictions.
Accuracy of Lexical Choice (AoLC) evaluates the accuracy of target lexicon chosen by a trained NAT model M for each source word. Specifically, the model M takes source word f as the input, and produce a hypothesis candidate list with their corresponding word confidence:
PM (e.
(2)
where V Vrg is the target side vocabularies over whole corpus. The AoLC score is calculated by averaging the probability of the gold target word e of each source word f:
Ejevicu PM (erls)
AoLC
(3)
V
where sre is the set of source side tokens in test set. Each gold word e is chosen with the help of the word alignment model PAA. The chosen procedure is as follows: Step 1) collecting the references of the source sentences that contains source word f. and generating the target side word bag B, with these references. Step 2) Descending PAA terms of alignment probabilities and looking up the word that first appears in By as the gold word until the By is traversed. Step 3) the gold word is still not found, let the word with the highest alignment probability in PA as the gold word. Generally, higher accuracy of lexical translation represents more confident of the predictions. We discuss the reliability of word alignment-basse based AoLC in Appendix A.I
3.2 GLOBAL EFFECT OF KNOWLEDGE DISTILLATION ON LEXICAL CHOICE
In this section, we analyze the lexical choice behaviors of NAT models with our proposed AoLC. In particular, We evaluated three MaskT models, which respectively trained on the raw data, AT-BASE and AT-BIG BIG distilled data. We compared the AoLC with other two metrics (i.e. BLEU and CoD) on three different datasets (i.e. En-De, Zh-En and Ja-En). As shown in Table 2 KD is able to
4

---

Published as a conference paper at ICLR 2021
improve translation quality of NAT models (BLEU: (D KD(BASE) > Raw) by increasing the lexical choice accuracy of data (AoLC: KD(BIG) > =D(BAsE) >Raw). As expected, NAT models trained more deterministic data (CoDJ) have lower lexical choice errors (AoLC1) globally, resulting in better model generation performance (BLFUt).
3.3 DISCREPANCY BETWEEN HIGH- AND LOW-FREQUENCY WORDS LEXICAL CHOICE
To better understand more detailed lexical change within data caused by distillation, we break down the lexicons three categories in terms of frequency. And revisit it from two angles: training data and translated data.
We first visualize the changing of training data when adopting KD in terms of words frequency density. As shown in Figure 2 we find that the kurtosis of KD data distribution is higher than that of raw, which becomes more significant when adopting stronger teacher. The side effect is obvious, that is, the original high- / low-frequency words become more fewer, making the distribution of training data more imbalance and skewed, which is problem- atic in data mining field (Chawla al.) 2004). This discrepancy may erode the translation performance of tow-frequeney words and generalization perfor- mance other domains. Here we focus low- frequency words, and generalization performance degradation will be exploited in future work.
Words Frequency Density
Raw
KD (Base)
KD (Big)
16-
14-
order to understand the detailed change dur- ing inference, we then analyze the lexical accu- racy with different frequencies in the test set. We make the comprehensive comparison cross lan- guages based on our proposed AoLC. As shown the teacher model becomes better, i.. KD(base))K((big) the lexical choice of high- frequency words becomes significantly more accurate (AoLC 1) while that of low-frequency words becomes worse (AoLC 1). Through fine-grained analysis, we uncover this interesting discrepancy between high- and low- frequency words. The same phenomena (lexical choice errors on low- frequency words propagated from teacher model) also can be found in general cases, e.g. distillation when training smaller AT models. Details can be found in Appendix A.2 To keep the accuracy of high-frequency words and compensate for the imbalanced low-frequency words caused by KD), we present a simple yet effective approach below.
Figure 2: Comparison of the token frequency density (w.r.t the sampled tokens' probability distribution) between Raw, KD (Base) and KD (Big) WMT14 En-De training data.
in Figure
High
Medium
Low
Accuracy (%)
Accuracy (%)
Accuracy (%)
70
70
70
65
60
60
60
Raw
KD (Basc) KD (Big)
Row
KD (Basc) KD (Big)
Raw
KD (Basc)
KD (Big)
(a)
En-De
(b) Zh-En
(c) Ja-En
Figure 3: Accuracy of lexical choice (AoLC) for source words of different frequency.

---

Published as a conference paper at ICLR 2021
4 IMPROVING LEXICAL CHOICE IN NAT MODELS
4.1 METHODOLOGY
Our goal is to augment NAT models learn needed lexical choices from the raw data to achieve better performance. To this end, we introduce an extra bilingual data-dependent prior objective to augment the current NAT models to distill the required lexical choices from the raw data. Specifically, Kullback-Leible divergence to guide the probability distribution of model predictions pMt (el (f) to match the prior probability distributions Q(-):
Eprior
Σ KL (L(((e()) pM cce
PM (cff)
(4)
where f the source sentence, and e the target sentence. The bilingual prior distribution Q(-) is derived from the raw data, which is independent of the model M and will be described later. The final objective for training the NAT model becomes:
L = AYCNAT + prrioo
(5)
which the imitation rate follows the logarithmic decay function:
(og(L/(((++1))) log(1/2)
i 1/2
x(1)
(6)
others
where is the current step, the total training step for distilled data. Accordingly, the NAT model merely fed with the priori knowledge derived from the raw data at beginning. Along with training, the supervision signal of the prior information is getting weaker while that the distilled data gradually prevails in the training objective. We run all models for 300K steps to ensure adequate training. thus the bilingual prior distributions will be exposed at the first 150K steps.
Choices of Prior Distribution Q(-) The goal of the prior objective is to guide the NAT models to learn to distill the lexical choices itself from the raw data. For each target word e, we use the external word alignment to select the source word with the maximum alignment probability, and (-) is rewritten as:
Q(elf) Q(elf)
Specifically, we use two types of bilingual prior distributions:
Word Alignment Distribution ( (WAD) is the distribution derived from the external word alignment pp = {PD (eilf) pD (f) where (eI..... ex) are the set of target words aligned to the source word in the training data. We follow Hinton et al. (2015) to use the softmax temperature mechanism to map Pf over the whole target vocabulary:
exp(P /T)
Q(elf) = -py Evey exp(Pp/T)
(8)
We tune the temperature from [0.5, 2, WMT14 En-De dataset and use 2 as the default setting for incorporating word alignment distribution in all datasets.
Delf--istilled Distribution (SDD) is the probability distribution for the source word f, which is produced by same NAT model pre-trained on raw data. Specifically, the model M takes source word f as input and produces probability distribution over whole words target vocabulary:
PM [PM (PM(( (ealf)..... P. (ev...
This prior distribution signal can characterized as relf-distilled lexicon level "born-again net- works" (Furlanello et al. 2018) self-knooledge distillation (Liu et al., 2020), where the teacher and student have the same neural architecture and model size, and yet surprisingly the student is able surpass the teacher's accuracy.

---

Published as a conference paper at ICLR 2021
En-De
Zh-En
Ja-En
Model
LFT
BLEU
LLL
BLEU
ALLC/LFT
BLEU
AT-TEACHER
79.0/73.0
29.2
74.7 66.2
25.3
77.1 700.8
29.8
Mas
76.3
27.0
72.7 61.5
24.2
75.3 / 66.9
28.9
+WAD
77.5/71.9
27.4
73.4 64.5
24.8
76.3 / 69.0
29.4
+SDD
77.7
27.5
73.5
/ 64.7
24.9
76.1 / 68.6
29.3
+Both
78.1
27.8
74.0 65.0
25.2
76.6 / 69.1
29.6
Table 3: Ablation Study data priors across different language pairs using the MaskT Model. "WAD" denotes word alignment distribution, and "SDD" denotes self-distilled distribution. "AoLCI LFT" denotes the lexical translation accuracies for all tokens / low-frequency tokens, respectively.
En-De
Ro-En
Model
Iter. Speed
AoLC BLEU
AoLC BLEU
AT Models
Transformer-BAst (Ro-En Teacher)
n/a n/a 0.8x
1.0x
27.3
34.1
Transtormer-Bit (En-De Teacher)
29.2
n/a
Existing NAT Models
NAT (Gu al. 2018
1.0
2.4x
19.2
31.4
Iterative NAT (Lee et al.
2018)
10.0
2,0x
21.6
30.2
DisCo (Kasai et al. 2020)
4.8
3,2x
n/a
26.8
n/a
33.3
Mask-Predics Ghazvimmejad et al. 2019)
10.0
1.5x
27.0
33.3
Levenshtein Gu ct al. 2019)
2.5
3.5x
27.3
33.3
Our NAT Models
Mask-Predict
76.3
27.0
79.2
33.3
10.0
1.5x
+Raw Data Prior
78.1
27.8t
80.6
33.7
Levenshtein +Raw Data Prior
77.0
27.2
79.8
33.2
2.5
3.5x
77.8
27.81
80.9
33.81
Table 4: Comparison with previous work on WMT14 En-De and W Ro-En datasets. "Iter." column indicate the average number of refined iterations. "*" indicates statistically significant difference (p 0.05) from baselines according to the statistical significance test (Collins et al. 2005).
4.2 EXPERIMENTAL RESULTS
Ablation Study on Raw Data Prior Table the results of our proposed two bilingual data dependent prior distributions across language pairs. The word alignment distribution (WAD) and self-distilled distribution (SDD) variants consistently improves performance over the vanilla two-step training scheme NAT model ("NAT+KD") when used individually (averagely +0.5 BLEU point), and combining them ("++oohh") ") by simply averaging the two distributions can achieve further improvement (averagely +0.9 BLEU point). The improvements on translation performance are due to increase of AoLC, especially for low-frequency tokens (averagely +3.2), which reconfirms our claim. Notably, averaging the two prior distributions could rectify each other, thus leading to further increase. We explore the complementarity of two prior schemes in Section 4.3 In the following experiments, we use the combination of WAD and SDD as the default bilingual data dependent prior.
Comparison with Previous Work Table lists the results of previously competitive studies (Gu et al. 2018 Lee et al. 2018 Kasai et al., 2020 Ghazvininejad et al.) 2019 Gu et al.) 2019) on the widely-used WMT14 En-De and WMT16 Ro-En datasets. Clearly, our bilingual data-dependent prior significantly improves translation (BLEUf) by substantially increasing the lexical choice accuracy (AoLCt). It is worth noting that our approaches merely modify the training process, thus does not increase any latency ("Speed"), maintaining the intrinsic advantages of non aonaautrrgrrssivee generation.
7

---

Published as a conference paper at ICLR 2021
Frequency
En-De
Zh-En
Ja-En
Model
En-De
Zh-En
Ja-En
High
+1.3%
+0.3%
+1.3%
Medium
+0.2%
+0.1%
+0.9%
NAT 10.3%
6.7%
9.4%
Low
+5.9%
+5.8%
+3.3%
+KD
7.6%
4.2%
6.9%
+Ours
9.8%
6.1%
8.5%
All
+2.4%
+1.8%
+1.7%
Table 8: Ratio of low-frequency target words in the MaskT model generated translations.
Table Improvement our approach over the MaskT+KD model on AoLC.
Comparison with Data Manipulation Strategies Instead using the proposed priors, we also inves- tigate effective data manipulation strategies, i.e. Data Mixing and Curriculum Learning, to force the NAT model learns from both the raw and distilled data. For data mixing, we design two settings: Mix: sim- ply combine the raw and distilled data, and then shuffle the mixed dataset. b) Tagged Mix: Inspired by suc- cesses of tagged back-translation (Caswell al. 2019 Marie et al., (020), we add tags to distinguish between KD and Raw sentences in the mixed dataset. For decay curriculum schedule, the NAT models learn more from raw data at the beginning and then learn more from KD the training goes on. The details of curriculum can be found in Appendix AA.] As in Table 5] data mixing and decay curriculum schedule improve performance on both AoLC and BLEU, which confirm the necessity of exposing raw data NAT models during training. Besides, approach still outperforms those effective strategies, demonstrating the superiority of our learning scheme.
Strategies
AoI.C
BLEU
Baseline
76.3
27.0
Mix
76.6
27.2
Tagged Mix
77.1
27.4
Decay Curriculum
77.2
27.5
Ours
78.1
27.8
Table 5: Performance of several data manip- ulation strategies En-De dataset. Base- line is the MaskT+KD model and Ours is our proposed approach.
4.3 EXPERIMENTAL ANALYSIS
In this section, conducted extensive analyses the lexical choice to better understand our approach. Unless otherwise stated, results reported the MaskPredict models in Table
Model
BLEU
AoLC
Error
MaskT +KD
22.6
68.5%
34.3%
24.2
72.7%
30.1%
Our approach improves translation performance by reducing mis-translation errors. The lexical choice ability of NAT models correlates to mis- translation errors, in which wrong lexicons are chosen to translate source words. To better understand whether our method alleviates the mis-translation problem, we assessed system output by human judgments. In particular, randomly selected 50 sentences from the Zh-En testset, and manually labelled the words with lexical choice error. We defined the lexical choice error rate E/N, where E is the number of lexical choice errors and N is the number of content words in source sentences, since such errors mainly occur in translating content words. As seen in Table 6 our approache consistently improves BLEU scores by reducing the lexical choice errors, which confirm our claim. Additionally, AoLC metric correlates well with both the automatic BLEU score and the subjective evaluation, demonstrating its reasonableness.
+RDP
25.2
74.0%
28.2%
Table Subjective evaluation of mis- translation errors on the Zh-En dataset.
Our approach significantly improves the accuracy lexical choice for low-frequency source words. As aforementioned discrepancy between high- & low-frquency words in Section 3.3 we focus on revealing the fine-grained lexical choice accuracy w.r.t our proposed AoLC. In Table majority of improvements is from the low-frequency words, confirming our hypothesis.
Our approach generates translations that contain more low-frequency words. Besides im- proving the lexical choice low-frequency words, our method results in more low-frequency words being recalled in the translation. In Table although KD improves the translation, biases the NAT
8

---

Published as a conference paper at ICLR 2021
model towards generating high-frequency tokens (Low freq. while our method can not only correct this bias (averagely +32% relative change), but also enhance translation (BLEUf in Table
Our proposed two priors complement each other by facilitating different to-
AoLC on LFT
Ratio of LFT
kens. As aforementioned in Table 3 combining two individual schemes can further increase the NAT performance. To explain how they complement each
Prior
Content
Function
Content
Function
N/A
67.7%
70. %
5.3%
2.4%
WAD
71.6%
72.9%
5.9%
2.5%
other, especially for low-frequency to- kens, we classify low-frequency tokens into categories according their lin- guistic roles: content words (e.g. noun, verb, and adjective) and function words (e.g. preposition, determiner, and punc- tuation). The results are listed Ta- ble [9] We show that WAD facilitates more on the understanding and genera- tion of content tokens, while SDD brings more gains for function (i.e. content- free) tokens. We leave more thorough exploration of this aspect for future work.
SDD
71.4%
74.3%
5.6%
3.4%
Both
71.6%
74.2%
6.2%
3.6%
Table AoL.C and Ratio of different prior schemes on Low-Frequency Tokens ("LFT"). We list the performances on different linguistic roles, i.c. content words and func- tion words. Note that Ratio of LFT means the ratio of low frequency tokens in generated translation. "N/A" means MaskT+KD baseline.
Effect of Word Alignment Quality on Model Performance. Both the proposed AoLC and priors depend heavily on the quality of word alignment, we therefore design two weaker alignment scenarios to verify the robustness of our method.
First, We adopt fast-align (Dyer et al., 2013), which is slightly weaker than GIZA++. Using fast- align, our methods can still achieve +0.6 and +0.7 improvements in terms of BLEU on En-De and Zh-En datasets, which are marginally lower than that using GIZA++ (i.e. +0.8 and +1.0 BLEU). Encouragingly, we find that the improvements in translation accuracy on uwwfrrquency words still hold (+5.5% and +5.3% vs. +5.9% and +5.8%), which demonstrates the robustmess of our approach.
In addition, we insert noises into the alignment distributions to deliberately reduce the alignment quality (Noise injection details can be found in Appendix A.44 The performances still significantly outperform the baseline, indicating that our method can tolerate alignment errors and maintain model performance some extent.
Effeet of AT Teacher To further dissect the
different effects when applying different AT teachers, we employ three teachers. Table I0 shows method der variety of teacher-student scenarios, includ- ing base, big and strong teacher-guided mod- els. Our approach obtains averagely +0.7 BLEU points, potentially complementary the major- ity of existing work on improving knowledge distillation for NAT models.
AT Teacher
NAT Model
enhance NAT models
Model
BLEU
Vanilla
+Prior
A
Base
27.3
26.5
27.2
+0.7
Big
28.4
26.8
27.5
+0.7
Strong
29.2
27.0
27.8
+0.8
Table 10: Different teachers on the En-De dataset.
RELATED WORK
Understanding Knowledge Distillation for NAT Knowledge distillation a crucial early step in the training of most NAT models. [een et al. (2020) reveal that the difficulty of NAT heavily depends on the strongness of dependency among target tokens, and knowledge distillation reduces the token dependency in target sequence and thus improves the accuracy of NAT models. In the pioneering work of NAT, Gu et al. (2018) claim that NAT suffers from the multi- modlii-mdalitty problem (i.e. multiple lexical translations for source word), and knowledge distillation can simplify the dataset, which is empirically validated by Zhou et al. (2020). We confirm and extend these results, showing that the di-distiledd dataset indeed leads to more deterministic predictions but propagates the low-frequency

---

Published as a conference paper at ICLR 2021
lexical choices errors. To this end, we enhance the NAT lexical predictions by making them learn to distill knowledge from the raw data.
Lexical Choice Problem NMT Models Benefiting from continuous representations abstracted from the training data, NMT models have advanced the state of the art in the machine translation community. However, recent studies have revealed that NMT models suffer from inadequate trans- lation (Tu et al.) 2016), in which mnis-translation error caused by the lexical choice problem is one main reason. For AT models, Arthur et al. (2016) alleviate this issue by integrating a count-base lexicon, and and Nguyen & Chiang (018)) propose additional lexical model, which is jointly trained with the AT model. The lexical choice problem is more serious for NAT models, since 1) the lexical choice errors (low-resource words in particular) of AT distillation will propagate to NAT models; and 2) NAT lacks target-side dependencies thus misses necessary target-side context. In this work, we alleviate this problem by solving the first challenge.
CONCLUSION
In this study, we investigated effects KD on lexical choice in NAT. We proposed a new metric to evaluate lexical translation accuracy of NAT models, and found that 1) KD improves global lexical predictions; and 2) KD benefits the accuracy of high-frequency words but harms the low-frequency ones. There exists discrepancy between high- and low-frequency words after adopting KD. To bridge this discrepancy, we exposed the useful information in raw data to the training of NAT models. Experiments show that our approach consistently and significantly improves translation performance across language pairs and model architectures. Extensive analyses reveal that our method reduces mistranslation errors, improves the accuracy of lexical choices for oow-freuency source words, recalling more low-frequency words in the translations as well, which confirms our claim.
ACKNOWLEDGMENTS
This work was supported by Australian Research Council Projects under grants FL- FL-117000017. 17, 19-180103424, and IC-19010003: Xuebo and Derek were supported in part by the Science and Technology Development Fund, Macau SAR (Grant No. 0101/2019/A2), and the Multi-yeat Research Grant from the University of Macau (Grant No. MYRG2020-00554F511 We also thank anonymous reviewers for their insightful comments.
REFERENCES
Philip Arthur, Graham Neubig, and Satoshi Nakamura. Incorporating discrete translation lexicons into neural machine translation. In EMNLP, 2016.
Isaac Caswell, Ciprian Chelba, and David Grangier. Tagged back-translation. WMT, 2019.
Nitesh V. Chawla, Nathalie Japkowicz, and Aleksander Kotez. Editorial: Special issue on learning from imbalanced data sets. SIGKDD Explor. News!., 2004.
Heeyoul Choi, Kyunghyun Cho, and Yoshua Bengio. Context-dependent word representation for neural machine translation. Computer Speech & Language, 45:149-160, 2017.
Michael Collins, Philipp Koehn, and Ivona Kučerová. Clause restructuring for statistical machine translation. In ACL, 2005.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL, 2019.
Liang Ding, Longyue Wang, Di Wu, Dacheng Tao, and Zhaopeng Tu. Context-aware cross-attention for non-autoregressive translation. In COLING, 2020.
Chris Dyer, Victor Chahuneau, and Noah Smith. A simple, fast, and effective reparameterization of ibm model 2. In NAACI. 2013.
10

---

Published as a conference paper at ICLR 2021
Tommaso Furlanello, Zachary Lipton, Michael Tschannen, Laurent Itti, and Anima Anandkumar. Born again neural networks. In ICML, 2018.
Marjan Ghazvininejad, Omer Levy, Yinhan Liu, and Luke Zettlemoyer. Mask-predict: Parallel decoding of conditional masked language models. In EMNLP, 2019.
Jiatao Gu, James Bradbury, Caiming Xiong. Victor OK Li, and Richard Socher. Non-autoregressive neural machine translation. In ICLR, 2018.
Jiatao Gu, Changhan Wang, and Junbo Zhao. Levenshtein transformer. In NIPS, 2019.
Shuhao Gu, Jinchao Zhang, Fandong Meng, Yang Feng, Wanying Xie, Jie Zhou, and Dong Yu. Token-level adaptive training for neural machine translation. In EMNLP, 2020.
Junliang Guo, Xu Tan, Di He, Tao Qin, Linli Xu, and Tie-Yan Liu. Non-autoregrrssive neural machine translation with enhanced decoder input. In AAAI, 2019.
Yongchang Hao, Shilin He, Wenxiang Jiao, Zhaopeng Tu, Lyu Michael, and Xing Wang. Multi-task learning with shared encoder for non-autoregressive machine translation. In NAACL, 2021.
Hany Hassan, Anthony Aue, Chang Chen, Vishal Chowdhary, Jonathan Clark, Christian Federmann, Xuedong Huang, Marcin Lunzyys-Dowmln William Lewis, Mu Li, et al. Achieving human parity on automatic chinese to english news translation. In arXiv, 2018
Geoffrey Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in a neural network. In NIPS Deep Learning and Representation Learning Workshop, 2015.
Jungo Kasai, James Cross, Marjan Ghazvininejad, and Jiatao Gu. Parallel machine translation with disentangled context transformer. In arXiv, 2020.
Yoon Kim and Alexander M Rush. Sequence-level knowledge distillation. In EMNLP. 2016.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
Philipp Koehn and Rebecca Knowles. Six challenges for neural machine translation. In WMT, 2017.
Jason Lee, Elman Mansimov, and Kyunghyun Cho. Deterministic non-autoregressive neural sequence modeling by iterative refinement. In EMNLP, 2018.
Jason Lee, Dustin Tran, Orhan Firat, and Kyunghyun Cho. On the discrepancy between density estimation and sequence generation. In ICML, 2020.
Yang Liu, Sheng Shen, and Mirella Lapata. Noisy self-knowledge distillation for text summarization, 2020.
Dabiao Ma, Zhiba Su, Wenxuan Wang, and Yu-Hao Lu. Fpets: Fully parallel end-to-end text-to- speech system. In AAAI, 2020.
Xuezhe Ma, Chunting Zhou, Xian Li, Graham Neubig, and Eduard Hovy. Flowseq: Non-
autoregressive conditional sequence generation with generative flow. EMNLP. 2019.
Benjamin Marie, Raphael Rubino, and Atsushi Fujita. Tagged back-translation revisited: Why does really work? In ACL, 2020.
Makoto Morishita, Jun Suzuki, and Masaaki Nagata. Ntt neural machine translation systems at wat 2017. In IJCNLP, 2017.
Toan Nguyen and David Chiang. Improving lexical choice in neural machine translation. In NAACL, 2018.
Franz Josef Och and Hermann Ney. A systematic comparison of various statistical alignment models. Computational Linguistics, 29(1,, 2003.
Myle Ott, Michael Auli, David Grangier, and Aurelio Ranzato. Analyzing uncertainty in neural machine translation. In /CMI., 2018.
11

---

Published as a conference paper at ICLR 2021
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. In ACL, 2002.
Qiu Ran, Yankai Lin, Peng Li, and Jie Zhou. Guiding non-autoregressive neural machine translation decoding with reordering information. In arXiv, 2019.
Yi Ren, Jinglin Liu, Xu Tan, Zhou Zhao, Sheng Zhao, and Yan Liu. A study of hon-autoregressive model for sequence generation. In ACL, 2020.
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In ACL, 2016.
Chenze Shao, Jinchao Zhang, Yang Feng, Fandong Meng, and Jie Zhou. Minimizing the bag-of- ngrams difference for non-autoregressive neural machine translation. In AAAI, 2019.
Mitchell Stern, William Chan, Jamie Kiros, and Jakob Uszkoreit. Insertion transformer: Flexible sequence generation via insertion operations. In ICML, 2019.
Zhiqing Sun, Zhuohan Li, Haoqing Wang, Di He, Zi Lin, and Zhihong Deng. Fast structured decoding for sequence models. NIPS, 2019.
Aleš Tamchyna. Lexical and morphological choices machine translation. 2017.
Jiaxi Tang, Rakesh Shivanna, Zhe Zhao, Dong Lin, Anima Singh, Ed H. Chi, and Sagar Jain. Understanding and improving knowledge distillation, 2020.
Zhaopeng Tu, Zhengdong Lu, Yang Liu, Xiaohua Liu, and Hang Li. Modeling coverage for neural machine translation. In ACL, 2016.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. In NIPS, 2017.
Yiren Wang, Fei Tian, Di He, Tao Qin, ChengXiang Zhai, and Tie- Yan Liu. Non-autoreggessive machine translation with auxiliary regularization. AAAI, 2019.
Bingzhen Wei, Mingxuan Wang. Hao Zhou, Junyang Lin, and Xu Sun. Imitation learning for non-autoregressive neural machine translation. In ACI. 2019.
Chunting Zhou, Graham Neubig, and Jiatao Gu. Understanding knowledge distillation in non- autoregressive machine translation. ICLR, 2020.
12

---

Published as a conference paper at ICLR 2021
A APPENDIX
A.1 DISCUSSION ON THE RELIABILITY OF WORD ALIGNMENT-BASER AOLC
We randomly select 20 sentence pairs from the Zh-En test set, which contains 576 source tokens. We use the trained word alignment model to produce alignments for the 20 sentence pairs, and then perform the gold word chosen procedure as described in Section 3.1 We manually evaluate these bilingual lexicons, and find that 551 out of 576 source words are aligned to reasonable equivalences (i.e. 96% accuracy). This demonstrates that reliable to calculate AoLC based on automatic word alignments.
A.2 GENERAL CASES OF THE DDEECE OF KNOWLEDGE DISTILLATION
To verify the universality of our findings that lexical choice error will propagate from teacher model, conduct the following experiments.
In particular, we experiment AT-Base and AT-Small models on the En-De data, which are distilled by the AT-Strong model. Note that the AT-Small model consists of 256 model dimensions, 4 heads, 3 encoder and 3 decoder layers. As shown in Table IT] the same phenomena can be found in AT models when distillation is used. We leave a thorough exploration of this aspect for future work.
Model
BLEU
AoLC on LFT
Ratio LFT
AT-Base
27.3
72.5%
9.2%
+KD
27.8
68.4%
7.8%
AT-Small
21.6
61.8%
10.7%
+KD
23.5
59.3%
7.1%
Table 11: Results of AT models on En-De when knowledge distillation is used. LFT denotes low- frequency tokens and Ratio LFT means the ratio of low-frequency tokens generated translation.
A.3 DECAY CURRICULUM SETUP
Specifically, the training process is divided into 5 phases, which differ at the constituent of training data. At Phase 1, all training examples are from the raw data; and at Phase 2, 75% of the training examples are from the raw data and the other 25% are from the distilled data (note that the two kinds of training examples should cover all source sentences). Similarly, the constituent ratios at the later phases are (50%, 50%), (25%. 75%), and (0%, 100%).
A.4 NOISE INJECTION SETUP
We swap the maximal probability tokens with other random tokens under the change ratio of N%. With 2% and 5% noises, our method respectively decreased by -0.1 and -0.2 BLEU scores on En-De. The improvements in translation accuracy on low- cow-frequency words are +5.7% and +5.3%, which is comparable to non-noisy one (i.c. +5.9%).
13


Reviews:
[{"review_id": "blVnXzWTH_7", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper follows up on the work (Zhou et al.) on establishing the importance of knoweldge distillation (KD) from a pretrained autoregressive translation model (AT) to train effective non-autoregresstive translation (NAT)  models. Specifically, KD is helpful because it reduces the data complexity which allows successful training of NAT models. This paper shows that KD has an undesirable effect on training of NAT models in terms of poor performance on translation into infrequent tokens and further suggests a remedy for regularizing the NAT training with an additional lexical translation loss based upon a prior translation table obtained via word alignment.\n\n-- Two approaches for obtaining the prior distribution are explored: 1) off the shelf word alignment, 2) lexical translation probabilities based on NAT pretrained on raw data. \n\n-- This approach is shown to be effective at improving performance of the underlying NAT model on different language pairs and the gains are consistent across multiple AT models used for KD.\n\n-- The comparison shows that the proposed approach significantly improves translation into rare tokens compared to a vanilla NAT model trained via KD. Also, the new NAT model also tends to produce more rare tokens. \n\n-- The proposed approach is simple to implement.\n\n-- More ablation on the kinds of gains obtained by the two different methods for prior would have strengthened the paper. How do the distributions for low frequency words change for different language pairs under different prior schemes?\n\n-- Moreover, both the metric used in the paper (AoLC) and the prior depend heavily on the quality of word alignment. An exploration into this aspect would have strengthened the paper.", "extra_scores": {}, "extra_text": {}}, {"review_id": "YdiWfQ9Iex", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In *non-autoregressive* neural machine translation (NMT), learning from the predictions of *autoregressive* teacher models through sequence-level knowledge distillation (Kim and Rush, 2016) has been an important step to improve the performance of the non-autoregressive student models. Despite the success and prevalence of this knowledge distillation (KD) technique, this paper hypothesises---and empirically validates---that this KD procedure has a detrimental side-effect of propagating **lexical choice errors** that the autoregressive teacher model makes by mistranslating **low-frequency words**, into the non-autoregressive student model. \n\nTo overcome this issue, this paper proposes a way to incorporate lexical choice **prior knowledge** from the *raw parallel text* (as opposed to the autoregressive teacher's output that may propagate lexical choice errors for low-frequency words). More concretely, this work specifies two prior distributions: (i) a word alignment distribution that specifies a list of plausible target words for each token in the source sentence, as obtained from automatic word alignment tools, and (ii) a target distribution from an identical non-autoregressive teacher model trained on the raw data (i.e. *self-distillation* or born-again network), which does not contain the same lexical choice error propagation that the autoregressive teacher's model output has. The student model is then trained to minimise KL divergence with respect to these prior distributions, in conjunction with the standard sequence-knowledge distillation loss that learns from the autoregressive teacher model's output, as determined by an interpolation coefficient with logarithmic decay (Eq. 6).\n\n**Pros:**\n1. This paper does a great job of motivating the problem of lexical choice error propagation on low-frequency words from the autoregressive teacher to the non-autoregressive student. The paper clearly states its hypothesis, provides a nice motivating example, and runs extensive preliminary experiments that convincingly confirm the existence of the lexical choice problem for low-frequency words.\n\n2.  The proposed prior knowledge approach is simple to implement, and yet provides decent improvements across all four datasets. The improvements are also consistent across different autoregressive teacher model sizes and language pairs, and the two kinds of prior knowledge can be combined to produce stronger results.\n\n3. The paper features a fairly comprehensive analysis (including in preliminary experiments) and ablation studies that help better understand where exactly the improvements are coming from. \n\n**Cons:**\nDespite the positive aspects above, I still have two major concerns regarding the paper:\n1. In Eq. 6 (page 5), the interpolation rate $\\lambda$ controls how much weight is assigned to learning from the prior knowledge vs from the autoregressive teacher. But the proposed logarithmic decay function does not make sense to me. Let $i$ be the number of steps. At the very beginning of training, $i=0$, so based on Eq. 6, $\\lambda = 1$. This makes sense since, at the beginning of training, the model only learns from the prior knowledge. But according to Eq. 6, $\\lambda$ will actually **get larger** as training progresses (up to $i \\leq I/2$). In the case where $i=I/2 - 1$, Eq. 6 will translate into $\\lambda = 2$. This does not make sense for three reasons. First, $\\lambda$ is an interpolation coefficient that therefore should be between 0 and 1. Second, $\\lambda=2$ means that the interpolation weight assigned to distilling the autoregressive teacher is -1. Third, based on the description, $\\lambda$ is designed to get smaller as more training steps are done, instead of getting larger.\n\n2. My second concern is that there is a much simpler way of injecting the prior knowledge. For instance, what if we simply provide a decaying learning schedule (i.e. a curriculum) where, at the beginning of training, the non-autoregressive student is trained to learn more from the *raw dataset*, while at the later stages of training, the non-autoregressive student is trained to learn more from the *autoregressive teacher's output*? This can potentially accomplish the exact same goal of learning the prior knowledge from the raw dataset first, and then move on to learn more from the teacher model's output. This simpler baseline should at least be compared against.\n\n3. This is a minor concern, but there are some grammatical mistakes and presentational suggestions in the paper that can be modified to improve clarity, as detailed below.\n\n**Question**\n1. In page 4, it is mentioned that: \"The chosen procedure [to get the \"gold\" lexical choice for each word] is as follows: ...\". How accurate is this procedure? Did you examine the output and double check whether the \"gold\" lexical choice corresponds well to human judgment or a dictionary entry for each word?\n\n**Presentation/Typos/Grammatical Errors:**\n1. In page 3, section 2.2, paragraph \"Datasets.\", \"... to avoid unknown *works*...\" -> \"words\".\n2. In page 4, just under Eq. 3, it is mentioned that \"$V$ is the source side vocabulary in test set\". If I understand correctly, $f$ is a token on the source sentence, so shouldn't $\\mathbf{V}^{test}_{src}$ be the list of *word tokens* (rather than source side vocabulary) on the source sentence? \n3. In page 5, \"Through fine-grained *analyzing*\" -> \"analysis\".\n4. In page 5, \"...becomes significantly accurate (AoLC)..\" -> \"significantly more accurate\".\n5. In page 6, \"...signal of the *priority* information\" -> \"prior\".\n6. In Table 3, I would suggest displaying the AoLC performance *just on rare words* (rather than overall AoLC), since that is the problem that the paper is trying to solve.\n7. In Table 5, mention what the column \"Iter.\" means.\n8. In page 7, \"... It is *worthy* noting ...\" -> \"worth\".\n9.  In page 7,  \"... by substantially *increase* the lexical choice accuracy ...\" -> \"increasing\".\n10. In section 4.3, I suggest saying a bit more about how the human judgment is collected. \n11. In page 8, \"For AT models, ...\" -> remove \"For\".\n\n-----Update after the authors' response-----\nThank you for the detailed authors' response, and for meticulously taking the feedback into account. The response has addressed most of my concerns.\n\nSome further comments:\n\n1. In Eq. 6, I think \"up to $i \\leq I/2$\" should be replaced with \"up to $i \\leq (I/2 - 1)$\", since $\\lambda$ would be negative with when $i=I/2$. Other than this, the equation looks good to me.\n2. I look forward to the addition of the results with the \"decay curriculum\" into the main paper. It is encouraging that the proposed approach outperforms this simpler \"decay curriculum\" baseline.\n\nSince the authors have addressed most of my concerns, I am therefore raising my recommendation to a \"6\".\n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "LpTgGeaiXE-", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "- Overall:\n\nThis paper analyzes the side effect of knowledge distillation in NAT where the lexical choice errors on low-frequency words are propagated to the student model from the teacher. Tackling on this, the paper then proposes to expose raw data to restore such information. In my view, the submission is well motivated and the designed experiments and results are meaningful and convincing which deserves an accept. However, as the paper focuses on analyzing a specific point (lexical choice) in a very constrained setting (NAT), the overall contribution might be incremental compared to other works in general at such a venue like ICLR.\n\n- Specific questions:\n(1) Will the same phenomena (lexical choice error propagated from teacher model) can be found in general cases when knowledge distillation is used? For instance, distillation when training small AT models. It would be nice to see a complete picture the same approach can also be used in other cases.\n(2) Is the NAT model in Eq (2) the same as the final model used for translation? If no, what it looks like?\n(3) What is the motivation of choosing the imitation rate $\\lambda$ in Eq (6)?\n(4) The improvements are still a bit marginal. It would be nice to report the statistical significance of the results. \n(5) What will be the performance by simply mixing the training data with raw data instead of using the proposed prior?", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper investigates knowledge distillation in the context of non-autoregressive machine translations. All reviewers are supportive of acceptance, especially after the thoughtful author responses. A well motivated and simple to implement approach that is giving good empirical results.
Decision: Accept (Poster)