Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: This paper follows up on the work (Zhou et al.) on establishing the importance of knoweldge distillation (KD) from a pretrained autoregressive translation model (AT) to train effective non-autoregresstive translation (NAT)  models. Specifically, KD is helpful because it reduces the data complexity which allows successful training of NAT models. This paper shows that KD has an undesirable effect on training of NAT models in terms of poor performance on translation into infrequent tokens and further suggests a remedy for regularizing the NAT training with an additional lexical translation loss based upon a prior translation table obtained via word alignment.

-- Two approaches for obtaining the prior distribution are explored: 1) off the shelf word alignment, 2) lexical translation probabilities based on NAT pretrained on raw data. 

-- This approach is shown to be effective at improving performance of the underlying NAT model on different language pairs and the gains are consistent across multiple AT models used for KD.

-- The comparison shows that the proposed approach significantly improves translation into rare tokens compared to a vanilla NAT model trained via KD. Also, the new NAT model also tends to produce more rare tokens. 

-- The proposed approach is simple to implement.

-- More ablation on the kinds of gains obtained by the two different methods for prior would have strengthened the paper. How do the distributions for low frequency words change for different language pairs under different prior schemes?

-- Moreover, both the metric used in the paper (AoLC) and the prior depend heavily on the quality of word alignment. An exploration into this aspect would have strengthened the paper.
review_title: Official review

final_score: rating: 7: Good paper, accept
review_content: - Overall:

This paper analyzes the side effect of knowledge distillation in NAT where the lexical choice errors on low-frequency words are propagated to the student model from the teacher. Tackling on this, the paper then proposes to expose raw data to restore such information. In my view, the submission is well motivated and the designed experiments and results are meaningful and convincing which deserves an accept. However, as the paper focuses on analyzing a specific point (lexical choice) in a very constrained setting (NAT), the overall contribution might be incremental compared to other works in general at such a venue like ICLR.

- Specific questions:
(1) Will the same phenomena (lexical choice error propagated from teacher model) can be found in general cases when knowledge distillation is used? For instance, distillation when training small AT models. It would be nice to see a complete picture the same approach can also be used in other cases.
(2) Is the NAT model in Eq (2) the same as the final model used for translation? If no, what it looks like?
(3) What is the motivation of choosing the imitation rate $\lambda$ in Eq (6)?
(4) The improvements are still a bit marginal. It would be nice to report the statistical significance of the results. 
(5) What will be the performance by simply mixing the training data with raw data instead of using the proposed prior?
review_title: A good analysis on knowledge distillation and lexical choice in NAT

final_score: rating: 6: Marginally above acceptance threshold
review_content: In *non-autoregressive* neural machine translation (NMT), learning from the predictions of *autoregressive* teacher models through sequence-level knowledge distillation (Kim and Rush, 2016) has been an important step to improve the performance of the non-autoregressive student models. Despite the success and prevalence of this knowledge distillation (KD) technique, this paper hypothesises---and empirically validates---that this KD procedure has a detrimental side-effect of propagating **lexical choice errors** that the autoregressive teacher model makes by mistranslating **low-frequency words**, into the non-autoregressive student model. 

To overcome this issue, this paper proposes a way to incorporate lexical choice **prior knowledge** from the *raw parallel text* (as opposed to the autoregressive teacher's output that may propagate lexical choice errors for low-frequency words). More concretely, this work specifies two prior distributions: (i) a word alignment distribution that specifies a list of plausible target words for each token in the source sentence, as obtained from automatic word alignment tools, and (ii) a target distribution from an identical non-autoregressive teacher model trained on the raw data (i.e. *self-distillation* or born-again network), which does not contain the same lexical choice error propagation that the autoregressive teacher's model output has. The student model is then trained to minimise KL divergence with respect to these prior distributions, in conjunction with the standard sequence-knowledge distillation loss that learns from the autoregressive teacher model's output, as determined by an interpolation coefficient with logarithmic decay (Eq. 6).

**Pros:**
1. This paper does a great job of motivating the problem of lexical choice error propagation on low-frequency words from the autoregressive teacher to the non-autoregressive student. The paper clearly states its hypothesis, provides a nice motivating example, and runs extensive preliminary experiments that convincingly confirm the existence of the lexical choice problem for low-frequency words.

2.  The proposed prior knowledge approach is simple to implement, and yet provides decent improvements across all four datasets. The improvements are also consistent across different autoregressive teacher model sizes and language pairs, and the two kinds of prior knowledge can be combined to produce stronger results.

3. The paper features a fairly comprehensive analysis (including in preliminary experiments) and ablation studies that help better understand where exactly the improvements are coming from. 

**Cons:**
Despite the positive aspects above, I still have two major concerns regarding the paper:
1. In Eq. 6 (page 5), the interpolation rate $\lambda$ controls how much weight is assigned to learning from the prior knowledge vs from the autoregressive teacher. But the proposed logarithmic decay function does not make sense to me. Let $i$ be the number of steps. At the very beginning of training, $i=0$, so based on Eq. 6, $\lambda = 1$. This makes sense since, at the beginning of training, the model only learns from the prior knowledge. But according to Eq. 6, $\lambda$ will actually **get larger** as training progresses (up to $i \leq I/2$). In the case where $i=I/2 - 1$, Eq. 6 will translate into $\lambda = 2$. This does not make sense for three reasons. First, $\lambda$ is an interpolation coefficient that therefore should be between 0 and 1. Second, $\lambda=2$ means that the interpolation weight assigned to distilling the autoregressive teacher is -1. Third, based on the description, $\lambda$ is designed to get smaller as more training steps are done, instead of getting larger.

2. My second concern is that there is a much simpler way of injecting the prior knowledge. For instance, what if we simply provide a decaying learning schedule (i.e. a curriculum) where, at the beginning of training, the non-autoregressive student is trained to learn more from the *raw dataset*, while at the later stages of training, the non-autoregressive student is trained to learn more from the *autoregressive teacher's output*? This can potentially accomplish the exact same goal of learning the prior knowledge from the raw dataset first, and then move on to learn more from the teacher model's output. This simpler baseline should at least be compared against.

3. This is a minor concern, but there are some grammatical mistakes and presentational suggestions in the paper that can be modified to improve clarity, as detailed below.

**Question**
1. In page 4, it is mentioned that: "The chosen procedure [to get the "gold" lexical choice for each word] is as follows: ...". How accurate is this procedure? Did you examine the output and double check whether the "gold" lexical choice corresponds well to human judgment or a dictionary entry for each word?

**Presentation/Typos/Grammatical Errors:**
1. In page 3, section 2.2, paragraph "Datasets.", "... to avoid unknown *works*..." -> "words".
2. In page 4, just under Eq. 3, it is mentioned that "$V$ is the source side vocabulary in test set". If I understand correctly, $f$ is a token on the source sentence, so shouldn't $\mathbf{V}^{test}_{src}$ be the list of *word tokens* (rather than source side vocabulary) on the source sentence? 
3. In page 5, "Through fine-grained *analyzing*" -> "analysis".
4. In page 5, "...becomes significantly accurate (AoLC).." -> "significantly more accurate".
5. In page 6, "...signal of the *priority* information" -> "prior".
6. In Table 3, I would suggest displaying the AoLC performance *just on rare words* (rather than overall AoLC), since that is the problem that the paper is trying to solve.
7. In Table 5, mention what the column "Iter." means.
8. In page 7, "... It is *worthy* noting ..." -> "worth".
9.  In page 7,  "... by substantially *increase* the lexical choice accuracy ..." -> "increasing".
10. In section 4.3, I suggest saying a bit more about how the human judgment is collected. 
11. In page 8, "For AT models, ..." -> remove "For".

-----Update after the authors' response-----
Thank you for the detailed authors' response, and for meticulously taking the feedback into account. The response has addressed most of my concerns.

Some further comments:

1. In Eq. 6, I think "up to $i \leq I/2$" should be replaced with "up to $i \leq (I/2 - 1)$", since $\lambda$ would be negative with when $i=I/2$. Other than this, the equation looks good to me.
2. I look forward to the addition of the results with the "decay curriculum" into the main paper. It is encouraging that the proposed approach outperforms this simpler "decay curriculum" baseline.

Since the authors have addressed most of my concerns, I am therefore raising my recommendation to a "6".



review_title: Official Blind Review #4 

Meta-review: This paper investigates knowledge distillation in the context of non-autoregressive machine translations. All reviewers are supportive of acceptance, especially after the thoughtful author responses. A well motivated and simple to implement approach that is giving good empirical results.
Decision: Accept (Poster)