Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This work builds on the non-autoregressive translation (NAT) by using position as a latent variable. Unlike the work by Gu et. al. 2018, where they assume the output word order to follow the word order of the input sentence, this work explores predicting word order supervision as an additional train signal. It shows that predicting the position of the words improves the performance of the translation and paraphrase task. This paper uses a heuristic that the inputs positions and output positions of the decoder with close by embeddings are more likely to represent the position mapping. 

Since word order might change across languages the idea of using position based supervision seems promising. The results from the experiments on translation and paraphrasing tasks seem promising as they beat previously established baselines. With techniques like length parallel decoding from Gu et. al. 2018, PNAT performs much better than the baselines. For the paraphrasing task, it is interesting to observe that PNAT beats the autoregressive transformer based model. 

Pros:
- This work gives a convincing argument to model word position prediction as a latent variable.
- Experiments show PNAT beats baseline models for Translation and Paraphrasing task. 
- It also shows that using position supervision increases the convergence speed of the model.

Questions:
- While position prediction seems like a good idea, I am not fully convinced of the heuristic used -  similarity between input (d_i) and output (y_j) is used to determine the position supervision. Since (d_i)s undergo many transformations to produce (y_j)s, the embedding vectors don't necessarily have to be similar for, some d_i to have greater influence on a particular y_j. It would be nice to verify this assumption using some gold data or some manual checks. 

- In addition to the previous point, is the model pertained before this heuristic is used? Since, starting with random initialization might just reinforce random position mappings based on initial conditions.

-  In describing the HSP, could you please make it more clear how the z_i are decided? Is it that the iteratively best (d_i, y_j) is selected as the z_i and then d_i & y_j are removed from the corresponding sides?

- The tables assume that the reader knows about the abbreviations. Could you please add what NPD, WT, etc. mean?

- It would be nice to see the respective results with NAR position predictor since the discussion is about building a Non Autoregressive model.

- Are the gains from NAT lost by using AR position predictor since autoregressive prediction is added indirectly to the whole model?

- In Table 2 PNAT w/HSP seems to have amazing performance compared to other models. Could the authors shed some light on why this cannot be used directly? Is it because of delays due to the iterative process in extracting z_i?



Minor:

- There are a couple of typos.


review_title: Official Blind Review #4

final_score: rating: 3: Weak Reject
review_content: This work proposes an alternative approach to non-autoregressive translation (NAT) by predicting positions in addition to the word identities, such that the word order in the final prediction doesn't matter as long as the positions are correct. The length of the translation is predicted similar to Gu et al 2017, as well as smoothly copying the source sequence to decoder input. However, since the positions are unknown, this paper employs a heuristic search method to find the nearest neighbors in the embedding space to obtain position supervision.

Experiments are conducted on a machine translation task and a paraphrase generation task. For WMT14 DE-EN and IWSLT16 DE-EN, this method gets superior performance compared to baselines. For paraphrase generation, this approach with an AR position predictor gets even better performance than a normal AR transformer. Further analysis shows that this approach degrades less without knowledge distillation, or without removing repetitions.

Pros:
1. This approach gets better performance compared to baselines.
2. The idea of modeling positions as a latent variable is interesting and might generalize to other tasks beyond NAT.

Cons:
1. This work should compare to later baselines such as FlowSeq (https://arxiv.org/pdf/1909.02480.pdf) which gets better performance with flow.
2. In table 1, although the proposed approach outperforms imitate-NAT, the speedup is lower, making it hard to judge which is actually better.
3. In table 2, why is AR predictor used? What's the performance of NAR predictor? (in general why consider NAR position predictor at all?)
4. It is not clear why heuristic search would work here. Is any pretraining required? Otherwise, since there's no gradient signal for the positions, I'm not sure how the model figures it out.

Questions:
1. How many samples are used in table 1 LPD? Or is it argmax decoding for each length?
2. Is it possible to include a few examples showing predicted positions in an appendix to help better understand the model's behavior?
3. Why do you think positions can be predicted in a NAR manner? Isn't it just shifting the burdens to the position predictor? (Since in transformers if it's able to learn positions then it should be trivial to reorder based on those positions)

Minor details:
Some typos need to be fixed.

Overall, it is an interesting idea to predict the positions and word identities separately, and with Gumbel-Softmax and VAE we might be able to optimize the true marginals instead of relying on heuristic search. However, empirically there's been better performance achieved by flow-based models, so I am inclined to reject this work.


--------updates after reading the authors' rebuttal-----
I really appreciate the substantial experiments the authors conducted post-review. However, I still have some concerns regarding the baseline flowseq. Flowseq-large gets better results but is not reported in the main paper. Besides, it is unclear how is the speedup computed since Ma et al 2019 reported that the speedup depends on batch size, and at batch size 120 the speedup is ~X6 for flowseq-base.

Minor issue: in table 1 which dataset are the speedup numbers based on?
review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: This work proposes a non-autoregressive model for conditioned text generation. The non-autoregressive decoder conditions on a sequence of discrete latent variables, which represent the generation order and can be autoregressively calculated. Instead of doing marginal inference, the paper takes the top 1 generation order that best match inputs. Experiments on machine translation and paraphrase generation show strong result in comparison to other non-autoregressive models.

The idea of delegating generation order to the latent variables seems interesting, and the paper takes a reasonable approach when fleshing it out. Most of the experiments seem solid to me. The presentation does a good job in giving a very brief description of the model to readers familiar with non-autoregressive generation; but for those who are not (like myself), much content needs to be clarified.

Details:

- How is the model trained? Eq.13--14 present the loss, does it ever include the length predictor? Is the computation graph fully-diiferentiable? If it is, how do the authors backprop through the argmax in the length? If not, is reinforcement learning used?

- Can the authors specify B around Eq.5? Also \Delta M above section 4.3.

- Is the length predictor basically a classifier? Have the authors considered doing regression, which preserves the order relation (e.g., 3 is less than 5)?

- HSP. Can the authors describe the matching algorithm in detail? If it is some well-known algo like the Hungarian algorithm, please specify. If the authors come up with their own algo, please typeset it and analyze the complexity.

- Adding onto the above: is the algorithm exact or approximate? If the later, Eq.11 should not be \argmax.

- Table 1 is really confusing: what do NPD, WT, i_dec stand for?

- 3.2 discusses two position predictors. Which of them is actually used?

- Tabl 2: what does `Searched-Position` stand for? I thought the position search is only used to derived the reference position sequence.

- It is a bit sad that the paper started with marginal inference but ends up taking the top 1. Have the authors considered taking top K of the predicted orders to have a better approximation to the likelihood?


Minor:

- The second paragraph in intro. I'm not sure why non-autoregressive model has a larger search space. Both of them search over \Sigma^\ast, with \Sigma being the vocabulary.

- Typo above section 4.4: ... not powerful enough....
review_title: Official Blind Review #1

Meta-review: This paper presents a non-autoregressive NMT model which predicts the positions of the words to be produced as a latent variable in addition to predicting the words. This is a novel idea in the field of several other papers which are trying to do similar things, and obtains good results on benchmark tasks. The major concerns are systematic comparisons with the FlowSeq paper which seems to have been published before the ICLR submission deadline. The reviewers are still not convinced by the empirical performance comparison as well as speed comparisons. With some more work this could be a good contribution. As of now, I am recommending a Rejection.
Decision: Reject