Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper proposes a generalized framework for sequence generation that can be applied to both directed and undirected sequence models. The framework generates the final label sequence through generating a sequence of steps, where each step generates a coordinate sequence and an intermediate label sequence. This procedure is probabilistically modeled by length prediction, coordinate selection and symbol replacement. For inference, instead of the intractable naive approach based on Gibbs sampling to marginalize out all generation paths, the paper proposes a heuristic approach using length-conditioned beam search to generate the most likely final sequence. With the proposed framework, the paper shows that masked language models like BERT, even though they are undirected sequence models, can be used for sequence generation, which obtains close performance to the traditional left-to-right autoregressive models on the task of machine translation.

Overall the paper has significant contributions in the following aspects:
1. It enables undirected sequence models, like BERT, to perform decoding or sequence generation directly, instead of just serving as model pre-training.
2. The proposed framework unifies directed and undirected sequence models decoding, and it can represent a few existing sequence model decoding as special cases.
3. The coordinate sequence selection function in the framework can be dependent on the intermediate label sequence. A few simple selection approaches proposed in the paper are shown to be effective. It could be further extended. 
4. The analysis of the coordinate selection order is interesting and helpful for understanding the algorithm.
5. The experiment results for decoding masked language models on machine translation are promising. It also provides the comparison to recent related work on non-autoregressive approaches.

The presentation of the paper is also clear. I am leaning towards accepting the paper.

However, there are some weaknesses:

1. It should be analyzed more why different coordinate selection approaches perform differently in linear-time decoding vs. constant-time decoding. Even in constant-time decoding, the conclusion varies in different decoding setting, easy-first is the worst for the L->1 case, but the best for the L/T case, why is that?

2. What is the motivation for "hard-first"?

3. The setting of "least2most" with L->1 is similar to Ghazvininejad et al. 2019. But Table 4 in the appendix shows the result in this paper is still worse (21.98 vs. 24.61, when both systems use 10 iterations without AR). Also, the gap from the AR baseline is larger than that in Ghazvininejad et al. 2019. Given the two systems are considered similar, it should be explained in the paper the possible reasons for these discrepancies in results.

Additional minor comments for improving the paper:

1. In the introduction, it mentions the baseline AR is (Vaswani et al. 2017), while in the experimental settings, it mentions the baseline AR is (Bahdanau et al. 2015). Please clarify which one is used.

2. In Table 1, how does T = 2L work for the "Uniform" case while the target sequence length is only T, since it is mentioned the positions are sampled without replacement. Similarly, how does T = 2L work for the "Left2Right" case? Is it just always choosing the last position when L < t <= 2L? In these two cases, it seems T > L is not needed.

3. In Table 3, the header for the 2nd column should be o_t, as defined in Section 4 - "Decoding scenarios". What is the actual value of K and K'' for the constant-time machine translation experiments in the paper?

4. "Rescoring adds minimal overhead as it is run in parallel" - it still needs to run left-to-right in sequence since it is auto-regressive. Please clarify what it means by "in parallel" here.

5. What is the range and average for the target sentence length? How is T = 20 for constant-time decoding compared to linear-time decoding in terms of speed?
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: This paper focuses on decoding/generation in neural sequence models (specifically machine translation) in a non-autoregressive manner that instead of generating in a left-to-right manner, focuses on generating sequences by picking a length first ,and then indices to replace in a deterministic or random scheme and, finally using a context sensitive distribution over vocabulary (using BERT-like masked LM scheme)  to pick the word to replace. In practice, this procedure of picking indices and words to replace is repeated T number of times and hence the final sequence is obtained by this iterative refinement procedure. This is an interesting and important research direction because not only would it result in better and context sensitive greedy/approximate-MAP decoded outputs, but also opens up opportunities for parallelization of the decoding procedure which is difficult to achieve with left-to-right decoders.
That said, the results are fairly inconclusive and the practical implementation does leave things desired for a practical decoder. As observed by the authors, different deterministic strategies for choosing T results in very different performances among the variants of the proposed approach. Besides among the variants, one clear pattern is that uniformly random picking of indices is worse than other schemes (left-to-right, least-to-most, easy-first) which is not unexpected but no conclusive empirical evidence can be found for relative differences between the performances of other 3 schemes. Moreover, the proposed decoding variants generally perform worse than or at best similarly to standard autoregressive baselines. As authors note, this is due to the mismatch between the method in which the model was trained and the decoding procedure which is not surprising, but at the same time this does not give insight into the effectiveness of the proposed decoding objective. The central question is: if the training prefers left-to-right generation then how valuable is it to device  more reasonable but incompatible decoding procedures?

Also, authors also note that index picking schemes investigated in the paper are heuristic based and a more interesting decoder could be learned if index selection procedure itself was learned with features depending on the previous index selection states, decoded states Y, and other relevant quantities. They attribute poor performance of the proposed decoder to the nature of index selection approaches investigated in the paper. I think the paper would be strengthened with results with a more sophisticated learned index selection procedure in addition to the heuristics investigated in this paper.

Overall, while the idea and motivation behind this work is exciting, the inconclusive results and the approaches for practical implementation leave open significant room for improvement. 
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: This paper presents a general framework for sentence generation using a BERT-like model. The authors decompose the problem of sentence generation into two problems. One is selecting the positions at which changes should be made, and the other is actually replacing the current word with a new word. This framework enables them to represent many decoding strategies including that of Ghazvininejas et al. (2019) in a unified manner, and they propose a new decoding strategy that considers the prediction confidence of the current and the new word. The paper also presents a heuristic algorithm for beam search decoding to find the most likely generation path. Their experimental results on the WMT14 English-German dataset suggest that the proposed approach could achieve translation quality comparable to that of the standard autoregressive approach under a constant-time translation setting.

It is nice to see existing decoding strategies represented in a generalized framework, but I was a bit disappointed that the authors do not seem to address the most critical problem in using a BERT-like model for sentence generation, namely, how to find the most likely sentence in a probabilistically sound way. It seems to me that the authors rely on at least two approximations. One is using pseudo-likelihood and the other is using the most likely generation path instead of performing marginalization. It is fine that the authors focus on empirical results of translation quality but then I would like to see more strong and extensive evidence that supports the use of such approximation.

review_title: Official Blind Review #3

Meta-review: This paper proposes a generalized way to generate sequences from undirected sequence models.

Overall, I believe a framework like this could definitely be a valuable contribution, but as Reviewer 1 and Reviewer 3 noted, the paper is a bit lacking both in theoretical analysis and strong empirical results. I don't think that this is a bad paper at all, but it feels like the paper needs a little bit of an extra push to tighten up the argumentation and/or results before warranting publication at a premier venue such as ICLR. I'd suggest the authors continue to improve the paper and aim to re-submit at revised version at a future conference. 
Decision: Reject