Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: *Paper Contributions*

The paper proposes the Glancing Transformer (GLAT), a model for non-autoregressive generative model of language (focusing on the MT domain). GLAT incorporates several changes which result in a model which is state-of-the-art in several MT categories for non-autoregressive models and without requiring iterative sampling.

*Strong points of the paper*

* The paper clearly lays out the suggested model changes, and how they related to previous work.
* Some of the results are clearly state of the art for the task.
* The model and changes are described well and clearly.

*Weak points of the paper*

* GLAT incorporates several changes from the NAT-base architecture they primarily compare against, without clear ablations to attribute cause to the improvement.
* Most of the paper discusses the impact of architectures on inference/sampling speed, but doesn't discuss training speed, which seems relevant because my understanding of the model means that each step requires an additional decoder pass.
* Some of the writing is poor and overly-colloquial.

*Clearly state your recommendation (accept or reject) with one or two key reasons for this choice.*

I believe this paper is marginally above the acceptance threshold, though I think there is a clear avenue to improve the paper and change my rating to accept.

*Supporting arguments for your recommendation.*

The proposed GLAT model essentially consists of two changes from prior work:
1) Using an unconditional forward pass of the decoder to determine which tokens to mask, and then running the decoder a second time
2) Forwarding of encoding outputs via attention instead of mask tokens in the second decoder.

My main criticism of the paper is that is it unclear how each of the two changes contribute to the overall improvement being seen by the model. It might be the case that it is exclusively one of them, and I think the paper would clearly benefit from ablations discussing the two and showing how they each contribute.

As a secondary criticism, the paper proposes using a simple random sampling strategy to determine which tokens to mask (and which to glance at). While the authors defend this as the simple choice (and it certainly is), it feels to me that there is an obvious second alternative, which is to use the probability of the tokens generated by the decoder without glancing at anything (as in mask-predict's iterative sampling) to pick which tokens to glance at and which not to. It feels to me that this alternative is substantially obvious (seeing as it is exactly how mask tokens have been predicted in the past) that it should be attempted.

*Ask questions you would like answered by the authors to help you clarify your understanding of the paper and provide the additional evidence you need to be confident in your assessment.*

I would ask that two questions be answered, both ablations trying to get at the heart of the reason for improvement:

1) What happens if you do not forward the decoder inputs to the second decoding pass, but instead set them to a unique MASK token as in prior work (but keep all other aspects of training the same)?
2) What happens if you use mask-predict's probability-based sampling criterion to pick which tokens to sample, instead of the default random strategy.

Also, I have two questions which I think also should be answered, but don't require any further experimentation:

3) What is the impact on glancing sampling to training time?
4) What is the *exact* architecture of NAT-base? You show the encoder of GLAT works better than NAT-base - is NAT-base trained with the attention (instead of UniformCopy or SoftCopy)? Could that be the reason for the improvement?

I feel that if the authors answer these questions (ideally with WMT benchmark data for the first two), along with addressing the more minor points I list below, that I would feel comfortable increasing my rating to "accept".

*Additional feedback with the aim to improve the paper.*

I had a few minor questions while reading that I think would be useful to address:
* You mention your final model is based on averaging the 5 best checkpoints, how are you measuring best?
* You make a quite strong claim that "We think GLAT is superior to AT to some extent", which I think needs either more qualification or more focus.
* I struggled with understanding the paragraph starting with "Adaptive Sampling Number", I think it could use some clean-up.

Finally, I think most of the writing is clear, but there are a few points where it becomes quite colloquial and difficult to follow, I think an additional pass cleaning up some of the structure could be quite beneficial!
review_title: Official Review of Non-iterative Parallel Text Generation via Glancing Transformer

final_score: rating: 7: Good paper, accept
review_content: - Overall Comment
This submission improves non-autoregressive translation (NAT) by proposing a non-iterative parallel text generation model called Glancing Transformer (GLAT), which includes the explicit word dependency modeling in NAT via a proposed Glancing Language Model (GLM). Compared to previous work, biggest contribution of the proposed method is that it improves the training of NAT model with a similar idea of curriculum learning, while keeping the inference time unchanged, setting a significant improvement for non-iterative NAT without reranking. It would be a good baseline for future research on non-iterative NAT models. 
However, I still have the following comments and questions:
- Methods
(1) Why wouldn't the model gets stuck at predicting only part of the words correctly? 
    As described by the algorithm, the model will sample more reference words as the inputs of the decoder if the prediction is incorrect. However, the loss is only calculated for the remaining words.
Will the model only learn to predict easy words and give up learning the difficult words? I guess the random sampling strategy might be the key but it would be good to know answers from the authors.
(2) Since the model only updates in the second pass where the inputs are always mixed with reference words and source embeddings. There will be a clear mismatch between the second pass and the first pass. Why can the first model be sure to improve while training the second pass? Only by sharing parameters between two decoding passes?
(3) Hamming distance is quite weak. Will the proposed method also apply to other distance such as Levenshtein distance or a learnable distance?
(4) What do you mean by "ratio function" $f_\text{ratio}$? What is the input to this function? Steps?
- Experiments
(1) The proposed training process is in fact very similar to Mask-Predict except the inputs (Encoder hidden states instead of [MASK]) and adjusting the number of reference during training.  It would be nice to have a fair comparison with Mask-Predict for at least two settings (and combined):
(A) Mask-Predict with encoder hidden state inputs (with uniform copy or attention)
(B) Mask-Predict with 1 decoding iteration.
(2) I did not see how the model handle the length. With another network for length prediction? Or decoding with multiple lengths? For the latter case, how to decide these lengths?
- Missing Reference
This paper had an even higher one-iteration NAT results (**25.8** with Imputer compared to **25.21** in the submission on WMT En-De). Although the methods are different and the difference is to be honest marginal, it is important to include discussions on that or combine them in future work.
 *Saharia, Chitwan, et al. "Non-autoregressive machine translation with latent alignments." arXiv preprint arXiv:2004.07437 (2020).*

review_title: Inspiring idea for training "one-iteration" NAT

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposes a non-autoregressive neural machine translation model that does not require multiple iterations to achieve a good translation quality. The key difference to previous models is that during training it uses decoding to estimate the number of words to randomly sample (proportionally to the error) as opposed to random sampling a fixed number of them and uses this sampling strategy to mask tokens. 

**Strengths** 

The idea of using the model to come up with the number of words to sample before predicting them looks new and it appears to be helping a lot the model to do better with a single iteration. 

The evaluation shows that the proposed model outperforms previous non-iterative ones and performs on par with iterative ones in some cases while being significantly faster which is promising (but it still relies on reranking based on an autoregressive model). 

Moreover, the method performs much better than the non-autoregressive baseline on longer sequences (a trait that is observed in autoregressive models) and performs better than the autoregressive baseline on very short sequences (up to 20 tokens). 

**Weaknesses** 

(1) The writing requires some more effort because it has some grammatical/spelling errors and was not clear in several parts. It would help simplify wording and make more clear statements. For instance, there is a lot of time spent describing the glancing sampling strategy while it could be described in one paragraph (random sampling strategy paragraph seems redundant). 

(2) The proposed idea has some merits and works well judging from the results but it seems somewhat incremental and not very clearly explained (see below). The connection to curriculum learning was a bit hand-wavy and hard to follow. The model does not seem to be trained on targets that are of increasing difficulty but rather the number of incorrect targets simply defines the number of *random* samples to be used for training. Was this the intended connection to this line of work? Having many errors in the prediction during training does not necessarily mean that the example is difficult. 

(3) In Section 3.3, how is the function f_{ratio} actually implemented and trained (if applicable)? It's not clear from the provided description of how this is done. How are the gradients computed through the sampling process if it's trainable? This is an essential component of the model and it hasn't been explained well. 

(4) Another setback for the reader is the lack of discussion or acknowledgment of the computational cost that is required by the method during training. Performing decoding twice during training looks interesting from the modeling perspective but what is its effect on training speed? The evaluation focus is mainly on inference time but the training speed factor should also play a role too when deciding which method to use.

(5) Related to the above, the non-autoregressive models rely on knowledge distillation and re-ranking based on a pretrained autoregressive model. This already has its own training cost, so increasing it more could lead to a situation where the benefits during inference are overshadowed by the computational cost from training.


Other comments:

Are the models trained until convergence or for a fixed number of steps?  I am wondering what is the impact of the glancing sampling strategy on the convergence. 

How did the authors come up with the number of reranking candidates for each model? It looks like the number is different for each model; this should affect the quality of the model.  

Could the authors elaborate on what do they mean by "the formulation of our proposed glancing language model *could* be maximizing the likelihood of remaining words..."? Does it actually do that? This was unclear and it leads the reader to make guesses. 

In Section 4.2, what do you mean by "BLEU and speed-up are more or less contradictory"?

In Figure 1 and 2, the replaced inputs use the notation h_2, h_3 which points to the encoded inputs but in the textual description the replaced tokens seemed to be coming from the embedding on the decoder side (Section 3.2, paragraph 2): E_{y_t \in GS(Y,\hat{Y} }(y_t). There seems to be this inconsistency between the notation in the diagrams and the notation in text, which is very confusing. 
 
review_title: Promising results, unsatisfactory delivery

final_score: rating: 5: Marginally below acceptance threshold
review_content: The authors propose Glancing Transformer for single step parallel text generation. The approach is inspired from curriculum learning i.e. the training task is adaptively controlled based on the model's current performance. Specifically, the paper proposes a glancing strategy which compares the model's generation and reference sentence, and forms a sequence which is partially masked. The number of masked tokens in this sequence depends on the similarity between model's generation and reference sentence. The model is then trained to complete the partially masked sequence. 

The model achieves strong improvements on standard non-autoregressive MT baselines while not modifying inference process, thus not compromising on inference speed over vanilla non-autoregressive models.

+ves : 
- Limitations of current non-autoregressive MT models (NAT) are well explained, and the approach is nicely motivated. The paper is well written (although there are many grammatical mistakes that can be revised by the authors) and is easy to follow. The experimental details are well documented, many ablation studies are reported apart from comparison using standard metrics.  

- The results on standard benchmarks are strong. The paper improves over vanilla NAT by approx. 5 BLEU points on average, and is only 1 BLEU point less than baseline autoregressive models while being ~7x faster in inference.

Concerns : 

- While the use of curriculum learning inspired techniques to augment NAT model training is new and interesting, their specific technique does not seem to have sufficient novelty according to me. If I understood their technique correctly, the only difference in the training algorithm between Mask-Predict (Ghazvininejad et al., 2019) and their method is the selection of "number of tokens to mask (`mask-num`)" in the decoder's input. While Mask-Predict uses a uniform distribution to sample `mask-num` , they use a glancing sampler that decides this number based on the hamming distance between model's prediction and reference sentence.

- Although the reported results and ablation studies show a significant impact of this simple change, I think more exploration of this technique is possible and should be done in the paper. E.g. A simple hamming distance may not be a good strategy to compare the model's prediction and reference sentence. A related finding has been described in detail by Ghazvininejad et al., 2020 (https://arxiv.org/abs/2004.01655). So, I believe that authors can explore more strategies to compare reference and generated sequence.

- The authors use Random Sampling like Mask-Predict as their sampling strategy. The authors argue that random sampling may not be the most efficient strategy, but it's the easiest one, and has been shown to be powerful in models like BERT. While I totally agree with their argument, I believe that there is some possibility here to exploit the fact that we have access to model's prediction and reference sentence.  E.g. one possible strategy to exploit this could be selecting tokens that the model was not able to predict correctly based on the hamming distance comparison.  

- In the Introduction section, it is mentioned that - "Second, GLM replaces the [MASK] token in the MLM with signals from the encoder, thus not only
the target word dependency, but also the dependencies between the input and target are strengthened." While this is an interesting argument, I didn't find any experiment to validate this. I think the authors should include experiments to compare use of [MASK] tokens and their approach empirically. 

Based on these concerns, I am currently inclined to recommend rejection. While I find the idea of incorporating curriculum learning very interesting (together with strong results as demonstrated by the paper), I believe that more exploration of strategies to sample number of masked tokens, and sample words from the reference sentence is necessary to make the paper publishable. I have described this point in more detail in the Concerns above. 

Minor Comments : 
- There are many grammatical errors in the current version of the paper that the authors might want to revise. (E.g. in abstract - falls -> fall, achieves -> achieve). 
- In Section 3.1 authors mention this - "Note that we use the attention mechanism to form the decoder inputs with the input X." I think it might be helpful to elaborate more on what this exactly means. 
review_title: curriculum learning based technique to improve single step parallel generation for MT

Meta-review: This work raised quite a few questions, and left the reviewers somewhat divided. The authors have done their best to answer these questions, conducting additional experiments where needed.

The close relation of this work to Mask-Predict (Ghazvininejad et al. 2019) was noted by several reviewers. Although the current version of the manuscript addresses this, the introduction still frames Mask-Predict as an iterative model, and does not explicitly make the connection between GLAT and single-iteration Mask-Predict. My impression is that this understates the relationship between these models somewhat.

Taking single-iteration Mask-Predict as a baseline, the proposed extension is fairly simple, and seemingly effective, which is a potentially impactful combination. However, the manuscript is still held back by presentation issues (including but not limited to spelling and choice of words), and I concur with Reviewer 2 that the connection with curriculum learning should be elucidated not just in words, but with supporting experimental analysis.

Regarding training cost, given that training for GLAT seems to be more costly for the same number of training iterations, a comparison where the total compute budget is held constant could be interesting -- though I appreciate that this is not a key point of the paper, as the authors point out (whereas inference cost is).

I believe the changes made by the authors in response to the reviewers' comments are substantial enough that they merit a further review cycle, and may still fall short of the reviewers' expectations in some aspects. Therefore, I will not recommend acceptance, though I want to add that this was a tough call to make. I would also like to encourage the authors to resubmit their updated manuscript.
Decision: Reject