Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 8: Accept
review_content: In this paper, the authors investigate non-autoregressive translation (NAT). They specifically look into how using different auto-regressive translation (AT) models for knowledge distillation impacts the quality of NAT models. The paper is well organised and the experiments are sound and interesting, shedding light on an aspect of NAT models that's been rather dismissed as secondary until now: the impact of varying AT knowledge distillation.

First, so as to better please those out there who are more "state-of-the-art" inclined, I suggest the authors to better emphasise their improvements. Results obtained by their analysis can lead to improvements as stated in the last paragraph of Section 4. This could be better stressed in the introduction and it would make the main take-away messages from the paper stronger.

On a more general note, I would like to know how robust these models are. I understand this is a problem NAT and machine translation papers have in general, but I would still like to suggest the authors train each model from scratch multiple times using different random seeds, and report mean and variance of their results (i.e. BLEU). Although I understand that training each AT model 4 times and each NAT model 4x4 times (multiple times for each AT model trained) is unfeasible, you could still report mean and variance separately for AT and NAT, and simply choose one out of the 4 trained AT models per architecture to perform distillation. I recommend training each model at least 3 times and reporting mean BLEU and variance.

I would also recommend using other MT metrics (e.g. chrF, METEOR, BEER, etc.), since BLEU is a comparatively weak metric in terms of correlations with human judgements. For translations into German, look into characTER, chrF and BEER. For more information on the different metrics available, how they correlate with human judgements, and which better apply to different target languages / language pairs, please refer to the WMT evaluation/metrics shared tasks. [1]

I have a few general comments and suggestions:
- In Section 3.1, when introducing the NAT model (simplified version of Gu et al., (2019)), be more specific. I would like to see an actual description of what the modifications consist of in more detail: is there a sentence length prediction step? what happens when source length is different from source length? etc.
- In Section 3.2, right after Equation (3), the authors refer to "x and y" in an inverted manner, please fix it.
- In Section 4.3, you mention that two of the decoding methods used involve "random sampling", which I find misleading. You probably mean sampling according to the model distribution p(y_t | y_{<t}) for all t, which is not random. I suggest you simply remove the word "random" and mention that you use "sampling" and "sampling within the top-10 candidates". Also, when you sampling using the top-10 candidates, do you simply re-normalise the probability mass to include only the top-10 candidate tokens?
- As a suggestion, Tables and Figures could be more self-contained. There is always a compromise between conciseness and clarity, added by the fact that the page limit makes things even harder. However, I would recommend including more information in Table/Figure captions, especially in Figure 3 and Tables 3 and 4. Try to at least mention the training/evaluation data (dev or test?). Ideally one should understand it from reading the abstract and carefully reading the caption.
- In Appendix A, you mention that you select models according to validation loss. Is that really the case? If so, why? I am not sure whether validation loss (i.e. word cross-entropy on the validation set) should correlate so well with translation quality (or sentence-level metrics such as BLEU).

[1] http://www.statmt.org/wmt19/metrics-task.html
review_title: Official Blind Review #5

final_score: rating: 3: Weak Reject
review_content: The paper analyses recent distillation techniques for non-autoregressive machine translation models (NAT). These models use a autoregressive teacher (AT), which typically perform better. However, AT models can not be parallelized that easily such as the NAT models. The distillation has the effect of removing modes from the dataset which helps the NAT models as they suffer from the averaging effect of maximum likelihood solutions. The authors analyze why such distillation is needed and what the effect of the complexity of the training set is and further propose 3 methods to adjust the complexity of the teacher to the complexity of the NAT model.

The paper is well written and easy to understand and the experiments are exhaustive and well done. I think the analysis using the synthetic dataset is nice, but I am not sure how surprising the results are. I think most of it has to be expected given the properties of maximum likelihood estimation. Hence, I think the contribution of the paper is a bit limited. I am, however, not an expert in the field to fully judge the contribution and will therefore put low confidence on my review. 
review_title: Official Blind Review #2

final_score: rating: 8: Accept
review_content: [EDIT: Thanks for the response! I am updating my score to 8 given the rebuttal to my and other reviewers' questions]

--------------------------------------------
Summary:
--------------------------------------------
This paper studies knowledge distillation in the context of non-autoregressive translation. In particular, it is well known that in order to make NAT competitive with AT, one needs to train the NAT system on a distilled dataset from the teacher model. Using initial experiments on EN=>ES/FR/DE, the authors argue that this necessity arises from the overly-multimodal nature of the output distribution, and that the AT teacher model produces a less multimodal distribution that is easier to model with NAT. 

Based on this, the authors propose two quantities that estimate the complexity (conditional entropy) and faithfulness (cross entropy vs real data), and derive approximations to these based on independence assumptions and an alignment model.  The translations from the teacher output are indeed found to be less complex, thereby facilitating easier training for the NAT student model.

--------------------------------------------
Strengths:
--------------------------------------------
- Careful initial experiments to motivate the study.

- The metrics under consideration (entropy/faithfulness) are carefully derived, and while some of the assumptions may be overly simplifying (e.g. conditional independence assumptions), they are reasonable enough such that the directionality is likely to be correct.

- Thorough comparison of various NAT models. This will benefit the community greatly since to my knowledge, there has been no empirical head-to-head comparison of the different NAT methods that exist today.

- Extensive experiments across various settings, e.g. varying the teacher model size, varying the decoding strategies, investigating BAN/MoE/Sequence-level interpolation, etc. 

--------------------------------------------
Weaknesses:
--------------------------------------------
- It would have been interesting to consider a synthetic data setting such that one has access to the true underlying data distribution, such that approximations are not necessary.

- While translation is an important application of non-autoregressive generation, it would have also been interesting to study this in other seq2seq regimes such as summarization where the conditional entropy would presumably be even higher. (However this would complicate things like calculation of alignment probabilities, etc.)

--------------------------------------------
Other Questions/Comments:
--------------------------------------------
- p(y_t | l_i) coming from a token frequency distribution seems a bit too simple. Do the plots change if you model p(y | l_i) with a full language model that conditions on l_i?

- It's interesting to note that in Figures 1b,1c,1d, there is much more overlap between the romance languages es/fr, which are more closely related to each other. 

- I am not sure I agree with "C(d) reflects the level of multi-modality of a parallel corpus". One can certainly imagine a distribution which is unimodal but has high entropy...

- It seems like that we want a teacher model with low complexity and high faithfulness. Have the authors tried training a teacher model to directly target this? The usual MLE objective obviously targets faithfulness, but perhaps one could use RL techniques to optimize for low complexity as well.


review_title: Official Blind Review #3

Meta-review: Main content:

Blind review #3 summarized it well, as follows:

This paper studies knowledge distillation in the context of non-autoregressive translation. In particular, it is well known that in order to make NAT competitive with AT, one needs to train the NAT system on a distilled dataset from the teacher model. Using initial experiments on EN=>ES/FR/DE, the authors argue that this necessity arises from the overly-multimodal nature of the output distribution, and that the AT teacher model produces a less multimodal distribution that is easier to model with NAT. 

Based on this, the authors propose two quantities that estimate the complexity (conditional entropy) and faithfulness (cross entropy vs real data), and derive approximations to these based on independence assumptions and an alignment model.  The translations from the teacher output are indeed found to be less complex, thereby facilitating easier training for the NAT student model.

--

Discussion:

Questions were mostly about how robust the results were on other language pairs and random starting points. Authors addressed questions reasonably.

One low review came from a reviewer who admitted not knowing the field, and I agree with the other two reviewers.

--

Recommendation and justification:

I think papers that offer empirically support for scientific insight (giving an "a-ha!" reaction), rather than massive engineering efforts to beat the state of the art, are very worthwhile in scientific conferences. This paper meets that criteria for acceptance.
Decision: Accept (Poster)