Non-iterative Parallel Text Generation via Glancing Transformer
Authors: Lihua Qian
Hao Zhou
Yu Bao
Mingxuan Wang
Lin Qiu
Weinan Zhang
Yong Yu
Lei Li
Venue:  (2021)

Abstract: Although non-autoregressive models with one-iteration generation achieve remarkable inference speed-up, they still fall behind their autoregressive counterparts in prediction accuracy. The non-autoregressive models with the best accuracy currently rely on multiple decoding iterations, which largely sacrifice the inference speed of non-autoregressive models.  Inspired by the way of learning word dependencies in autoregressive and iterative-decoding models, we propose Glancing Transformer (GLAT) with a glancing language model (GLM), which learns to capture the word dependency gradually. Experiments on three benchmarks demonstrate that our approach can significantly improve the accuracy of non-autoregressive models without multiple decoding iterations. In particular, GLAT achieves state-of-the-art results among non-iterative models and even outperforms top iterative counterparts in some specific benchmarks.

Under review as a conference paper at ICLR 2021
NON-ITERATIVE PARALLEL TEXT GENERATION VIA GLANCING TRANSFORMER
Anonymous authors Paper under double-blind review
ABSTRACT
Although non-autoregrresive models with one-iteration generation achieve remark- able inference speed-up, they still fall behind their autoregressive counterparts in prediction accuracy. The non-autoregressive models with the best accuracy currently rely on multiple decoding iterations, which largely sacrifice the infer- ence speed of non- aonauuorrerrssive models. Inspired by the way of learning word dependencies in autoregressive and iterative-decoding models, we propose Glanc- ing Transformer (GL.AT) with a glancing language model (GI.M, which learns to capture the word dependency gradually. Experiments three benchmarks demonstrate that our approach can significantly improve the accuracy of non- autoregressive models without multiple decoding iterations. In particular, GLAT achieves state-of-the-ar results among non-iterative models and even outperforms top iterative counterparts in some specific benchmarks.
INTRODUCTION
Non-autoregressive transformer (NAT) has attracted wide attention in neural machine translation (Gu al. (0118). which generates sentences simultaneously rather than sequentially. To enable parallel decoding, NAT imposes a conditional independence assumption among words in the output sen- tences, which leads to significantly faster inference speed (almost a dozen times speed-up) than the autoregressive Transformer (Vaswani et al., (017). However, NAT still falls behind autoregressive Transformer (AT) the quality of output sentences, such as BLEU (Papineni et al. (2002) for machine translation. We blame it for the imposed conditional independence assumption, which prevents NAT models from explicitly learning the word dependencies the output sentence. Note that such word dependency is crucial, and it is explicitly learned in the AT model through the autoregressive language models (left-to-right, see Figure Ta and Figure lb
Recently, Ghazvininejad et al. 2019); Gu et al. (0019) propose employ the Masked Language Model (MLM, (MLM,,Devlnn et al. (219)) in NAT, which includes word dependency modeling in an iterative fashion (see Figure Ic), therefore yielding quite competitive results compared to AT. Specifically, such iterative models randomly mask words in the reference and predict these masked words conditioned on unmasked ones during training. In this manner, iterative models are trained to explicitly capture the dependencies between masked words and unmasked words.
However, these iterative approaches still give poor results with one decoding iteration and have to perform multiple iterations during inference, namely iteratively refining the generated outputs of the previous iteration. Such iterative process is quite time-consuming, which partly sacrifices the speed merit of NAT. How to abandon the iterative process while enjoy the benefits of explicitly modeling word dependencies in NAT is still an open problem.
In this paper, we argue that the major culprit of the problem that mask language models have to be used together with iterative inference, the sampling strategy of masking words in MIM. In particular, MLM employs fixed uniform strategy for masking words randomly during training, which prevents the model from effectively learning word dependencies for one-iteration generation. For example, at the beginning of training, the NAT model would be poorly tuned and we should mask fewer words. If we mask too many words, it would be difficult for the NAT model to correctly predict the masked words. On the contrary, if we mask too little words at the end phase of training, the resulting NAT model is rarely trained to predict the whole sentences, and can only predict some sentence fragments. In such case, to accurately generate the whole sentence in inference, the NAT

---

Under review as a conference paper at ICLR 2021
24
2
34
31
Y4
35
Decoder
Decoder
Decoder
Chanlng Sempplag
1000 Y1 32 33
h2
R4g hgg
[MAnspreoo)
34
35
h4
y3
Encoder
Encoder
Encoder
Encoder
Xg
X3
right
(b) implicit NAT
(c) Masked LM Iterative NAT
(d) Glancing LM in GLAT
Figure 1: Different language modeling approaches of different text generation models.
model has to generate the sentence fragments iteratively. To this end, the sampling strategy is crucial for the training of NAT.
To address the above issues, we propose a simple yet effective approach called Glancing Trans- former (GLAT), which equipped with the proposed Glancing Language Model (GLM) for non- iterative parallel text generation, achieving significant improvements upon strong baselines. Intu- itively, GLM adopts a adaptive glancing sampling strategy, which glances at some fragments of the reference if the reference is too difficult to fit in the training of NAT. Correspondingly, when the model well tuned, it will adaptively reduce the percentage of glancing sampling, making sure that the resulting model could learn generate the whole sentence in one-iteration fashion.
Specifically, our proposed GLM differs from MLM two aspects. Firstly, GLM proposes an adaptive glancing sampling strategy, which enables GLAT to generate sentences in a one-iteration way, working by gradual training instead of iterative inference (see Figure Id). Generally, GLM is quite similar to curriculum learning (Bengio et al. 2009) in spirit, namely first learning generate some fragments and gradually moving to learn the whole sentences (from easy to hard). To achieve the adaptive glancing sampling. GLM performs decoding twice in training. The first decoding is the same as the vanilla NAT, and the prediction accuracy indicates whether current reference is "difficult" for fitting. In the second decoding, GLM gets words of the reference via glancing sampling according to the first decoding, and learn to predict the remaining words that are not sampled. Note that only the second decoding will update the model parameters. Secondly, instead of using the [MASK] token, GLM directly use representations from the encoder at corresponding positions, which is more natural and could enhance the interactions between sampled words and signals from the encoder.
Experimental results show that GLAT obtains significant improvements (about 5 BLEU) on standard benchmarks compared to the vanilla NAT, without losing inference speed-up. GLAT achieves competitive results against iterative approaches like Mask-Predict (Ghazvininejad et al. 2019). even outperforming the Mask-Predict model on WMT14 DE-EN and WMTT6 RO-EN. Compared the strong AT baseline, GLAT can still close the performance gap within BLEU point while keeping 7.9x speed-up. Empirically, we even find that GLAT outperforms AT when the length of the reference is less than 20 WMT14 DE-EN. We speculate this is because GLM could capture bidirectional context for generation while its left- left-to-right counterpart is only unidirectional, which indicates the potential of parallel generation approaches like GLAT.
TEXT GENERATION VIA CONDITIONAL LANGUAGE MODELING
In this section, we compare different language models used in different text generation ap- proaches. Formally, considering sequence-to-sequence model (Cho et al. 2014 Bahdanaul ct al. 2014 Vaswani et al., 2017) for predicting Y = {y11,2 Mo.,, yr. given the input sentence X IT, I2, N. In the AT model, the training objective maximizing the log-likelihood with autoregressive decomposition:
LAT
log Logg(u Doglleeeer X;0),
where the word y is conditioned on the target prefix ycr = [BOS]] ), yr,..., yt-13 and the source input X. AT models sentences from left-to-right, therefore word dependencies learned in
2

---

Under review as a conference paper at ICLR 2021
NAT Decoding
Glancing
Sampling
H
h1 91
91 V1
y1
h/2
2/2
8r _
Hamming Distunce
0.5
Replace Inputs
h2
h1
33
8:
0.7
33
N)
114
4
84
0.6
h4
hg 55 833 23
0.9
y5
Figure 2: Illustration of the procedure of glancing sampling
unidirectional way. NAT, the other hand, incorporates conditional independence assumption among words in a sentence with the aim of enabling parallel generation:
NAT
log p(u((X;))
Note that, in NAT, different y/+ in Y are predicted simultaneously, which removes the interactions among target words in the NAT modeling. Thus, to generate fluent and faithful sentences, NAT has to model the target word dependencies implicitly, which makes the learning process quite challenging.
To explicitly model the target word dependencies, previous iterative approaches, such as Mask-Prede Precc-r introduce mask language models (MLM) in NAT. MLM models word dependencies by learning to predict the masked words conditioned on the unmasked ones:
LMLM
Σ (Y)
log logp( Yel (Y (M))))
(3)
Here RY)) returns some randomly sampled words from Y, and T replaces these sampled words in Y with the [MASK] token. For example, if IM(YY) = {u2, va & (Y, RM((Y) {3/1 [MASK], [MASK], .a...... To this end, the training objective is to predict the masked words RY) given the source sentence X and the unmasked target words.
As mentioned in the Introduction, though MI.M can explicitly model target word dependencies, can hardly generate satisfactory sentences without iteration. Better language modeling approach should be explored for parallel text generation in the one-iteration way.
3 THE PROPOSED GLANCING TRANSFORMER
In this section, describe GLAT in detail. Generally, GLAT differs from vanilla NAT in that it explicitly models word dependencies via the proposed glancing language model (GLM),, which leads significant accuracy improvements. Additionally, compared to these iterative approaches that adopt MLM, the proposed GLM is also non-trivia for the employment of an adaptive glancing sampling strategy, which enables GLAT to generate promising sentences in one-iteration way. And GLAT further strengthen the decoder inputs with the source representation.
3.1 THE GLANCING LANGUAGE MODEL
Formally, given the training data D (X, y the task is predicting Y {31, U2...., yr } with the input sentence X (I1, I2, ... N. By employing GLM, the training objective of GLAT is:
LELAT = Σ
Σ
log GS(Y, X)
(XYYJED
BE GB(X,
3

---

Under review as a conference paper at ICLR 2021
Here, Y is the predicted sentences in the first decoding pass, and GS Y) is the set of sampled target words by the sampling strategy of GS (described in detail in the next section). The sampling strategy samples more words from reference Y if prediction Y is less accurate, and samples fewer words for the opposite case. Additionally, {Y) GS(Y, Y)} is the difference set, representing the remaining words except for these sampled words. Then we perform the second decoding, updating the model parameters 8, by maximizing the likelihood of these remaining words with cross-entropy.
Specifically, GLM performs decoding twice in its learning procedure. In the first decoding pass, given the encoder If eenooddr and decoder F d'ccoder. H {/1, f12, ... hr } is the encoded output sequence gathered from the input X, and Y = F dewder ( H, F enconerr ( X; x);); 0) is the predicted sentence in the first decoding pass. With the predicted sentence Y, GS(Y, Y) is the set of sampled words from reference Y, according to our adaptive sampling strategy CS that will be introduced in the next section. Note that we use the attention mechanism
form the decoder inputs with the input X.
Previous work adopt Uniform Copy (Gu et al. 2018) or SoftCopy (Wei et al. (019) instead. But empirically, we find that they produce almost the same results in our setting.
In the second decoding pass, we cover the original decoding inputs H by the embeddings of words from GS(Y, Ý) to get the new decoding inputs H' = E Foover (.. (Bt), H), where Fowerr covers according to the corresponding positions. Namely, if we have sampled word at one position, we use its word embedding to replace the original decoding input at the same position. Here the word embeddings the mix of encoding signals and reference words from glancing sampling H' as decoder inputs, the training objective of proposed glancing language model can be written Equation 4 where the probabilities of remaining words on each position p(yel GS(Y, Y), X) are computed by F doccoder H',, Coweedr (X; 0), t)
obtained from the softmax embedding matrix of the decoder. With
3.2 THE GLANCING SAMPLING STRATEGY
We adopt the glancing sampling in GLM to adaptively sample words from the reference. Intuitively, the sampling strategy of our proposed glancing sampling method will sample many words at the start of the training. when the model is not yet well tuned. As the model gets better progressively, fewer words will be sampled to enable the model to learn the simultaneous generation of the whole sentence. Note that the sampling strategy is crucial in the training of NAT, a good sampling strategy could gradually guide the model to first learn the generation of fragments and then turn to the whole sentences.
As illustrated in Figure 2 the glancing sampling could be divided into two steps: first deciding sampling number N adaptively, and then randomly selecting N words from the reference. The sampling number N will be larger when the model is poorly trained and decreases along the training process. Note that we choose to randomly select the N words from the reference. The random reference word selection is simple and yields good performance empirically.
Formally, given the input X, its predicted sentence Ý and its reference Y, the goal of glancing sampling function GS(Y, Y) is to obtain a set of sampled words S, where S is subset of Y:
GS(Y, 8) = (YY)
(5)
Here, ((Y,N(Y,)) means randomly selecting N words from Y, and N is computed as:
N(Y fratio d(Y,8)
where d(Y, Y) is a metric for measuring the differences between Y and Ý. We adopt the Hamming distance (Hamming 1950h as the metric, which is computed as d(Y, Ei=( (M + (t). With d(Y, 9), the sampling number can decided adaptively considering the training status of the model. Note that here d(Y, Y) could be other distances such as Levenshtein distance [Levenshtein 1966). but find the Hamming distance gives the best result empirically. Additionally, to better control the glancing sampling process, we include a papeer-rrraeter frati to adjust the number of sampled words more flexibly.
4

---

Under review as a conference paper at ICLR 2021
4 EXPERIMENTS
In this section, we first introduce the settings of our experiments, then report the main results compared with several strong baselines. Ablation studies and further analysis are also included to verify the effects of different components used GLAT.
4.1 EXPERIMENTAL. SETTINGS
Datasets We conduct experiments three machine translation benchmarks: WMT14 EN-DE (4.5M translation pairs), WMT16 EN-RO (610k translation pairs), and IWSLT16 DE-EN (150K translation pairs). These datasets are tokenized and segmented into subword units using BPE encodings (Sennrich et al. 2016). We preprocess WMT14 EN-DE by following the data preprocessing in Vaswani et al. (2017). 7) For WMT16 EN-RO and IWSLT16 DE-EN, we use the processed data provided in
Lee et al. (2018).
Distillation
Following previous work (Gu et al. 2018 Lee al. 2018 Wang al. 2019). also use sequence-level knowledge distillation for all datasets. We employ the transformer base Vaswani al. (017)) as the teacher for knowledge distillation. Then, train our models on distilled data for each task.
Inference GLAT only modifies the training procedure and performs one-iteration non- autoregressive generation as the vanilla NAT in Gu et al. (2018). Before decoding, GLAT first predict the target length for outputs and the length prediction is implemented as in Ghazvininejad et al. 2019). An additional [LENGTH] token is add to the source input, and the encoder output for the [LENGTH] token used to predict the length.
Besides generation with one predicted target length, we also consider the common practice of noise parallel decoding (Gu et al. 2018 Lee et al.) 2018; Guo et al., 2019a Wang et al.) 2019), which generates several decoding candidates in parallel and selects the best via re-scoring with a pre-trained autoregressive model. For GLAAT, we first predict m target length candidates, then generate output sequences with argmax decoding for each target length candidate. Then use the pre-trained transformer to rank these sequences and identify the best overall output as the final output.
Implementation We adopt the vanilla model which copies source input uniformly in Gu al. (008)) as our base model (NAT-base) and replace the UniformCopy with attention mechanism using positions. For WMT datasets, we follow the hyperparameters of the base Transformer in Vaswani ct al. (017). p. And we choose a smaller setting for IWSLT16, considering that IWSLT16 is a smaller dataset. For IWSLT16, we use layers for encoder and decoder and the model size dmadel to 256. We train the model with batches of 64k/8k tokens for WMT/IWSLT datasets, respectively. We use Adam optimizer (Kingma & Ba 2014) with (0.9, 0.999). For WMT datasets, the learning rate warms up to 5e 4 in 4k steps and gradually decays according to inverse square root schedule in Vaswani
2017). D. As for IWSLT16 16 DE-EN, we adopt linear annealing (from 3e to le 5) as in Lee et al. (2018). For the hyper parameter fratio, we adopt linear annealing from 0.5 to 0.3 for WMT datasets and fixed value of 0.5 for IWSLT16. The final model created by averaging the best checkpoints chosen by BLEU scores on the validation set.
Competitors We compare our method with strong representative baselines. including fully non- iterative models: our vanilla NAT-base model, the NAT with fertility (Gu et al.) 2018 NAT-FT), the NAT imitating AT (Wei et al. 2019 NAit-NAT), the Flow-based NAT (Ma et al., 2019 Flowseq). the NAT with hint-based training (Li et al. 2019 NAT-HINT), Imputer (Saharia et al.) 2020). the NAT with CRF (Lafferty et al. 2001 decoding (Sun et al. 2019 NAT-DCRE), and the NAT with iterative refinement NAR-IR (Lee et al. 2018), LevT (Gu et al. 2019). Predict (Ghazvininejad et al.) 2019), and JM-NAT (Guo et al. 2020). For all our tasks, we obtain other NAT models' performance by directly using the performance figures reported in their papers if they available datasets.
4.2 RESULTS
The main results on the benchmarks are presented in Table [ Obviously, GLAT significantly improves the translation quality and outperforms strong baselines by large margin. Our method
5

---

Under review as a conference paper at ICLR 2021
Table Performance on WMT14 EN-DE/DE-EN and WMTI6 EN-RO/RO-EN benchmarks. Ieec is the number of decoding iterations and m is the number of reranking candidates.
WMT14
WMT16
Models
Idec
Speed Up
EN-DE
DE-EN
EN-RO
RO-EN
Transformer (Vaswani)
27.30
/
/
AT Models
Transformer (ours)
27.48
31.27
33.70
34.05
10x
NAT-IR
10
21.61
25.48
29.32
30.19
1.5x
LevT
6+
27.27
/
33,26
4.0%
Iterative NAT
Mask-Predict
27.03
30.53
33.08
33.31
/
JM-NAT
27.31
31.02
x.7x
NAT-FT
17.69
21.47
27.29
29.06
15.6x
imit-NAT
22.44
25.67
28.61 /
28.90
18.6x
NAT-HINT
21.11
25.24
30.2x
Non-iterative NAT
Flowseq
23.72
28.39
29.73
30.72
Imputer
25.8
NAT-base (ours)
20.36
24.81
28.47
29.43
15.3
GLAT (ours)
25.21
29.84
31.19
32.04
15.3
NAT-FT (m= 100)
19.17
23.20
29.79
31.44
24x
imit-NAT (m=7)
24.15
27.28
31:45
31.81
9.7x
Non-iterative NAT Reranking
NAT-HINT (m=9)
25.20
/
17.8x
Flowseg (m=30)
25.31
30.68
32.20
32.84
NAT-DCRE (m=9)
26.07
29.68
6.1x
GLAT (m=7, ours)
26.55
31,02
32.87
33.51
7.9x
introduces explicit dependency modeling for the decoder and gradually learns simultaneous generation of whole sequences, enabling the model to better capture the underlying data structure. Compared to models with iterative decoding, our method completely maintains the inference efficiency advantage of fully non-autoregressive models, since GLAT generate with only one-iteration. Compared with the competitors, we will highlight our empirical advantages:
The performance of GLAT is surprisingly good. Compared with the vanilla NAT-base models, GLAT obtains significant improvements (about 5 BLEU) EN-DE/DE- E.DEEEE. Addi- tionally, GLAT also outperforms other fully non-autoregressive models with a substantial margin (almost +3 BLEU score average). The results are very close those of the AT model, which shows great potential.
GLAT is simple and can be applied to other NAT models flexibly, we only modify the training process by reference glancing while keeping inference unchanged. For comparison, imitate-NAT introduces additional AT models as teachers; NAT-DCRF utilizes CRF to generate sequentially; NAT-IR and Mask-Predich models need multiple decoding iterations.
Note that Imputer and GLAT use different methods to determine the best target length. Based CTC (Graves et al., (006), Imputer sets the max target length twice the length of the source input and determines the best length by removing blanks and contiguous repetitive words after generation. Thus, it is non-trivial in Imputer, while GLAT can be further improved from 25.2 to 26.5 with AT reranking on WMT14 EN-DE, which outperforms the Imputer model.
apply target length reranking
We also present scatter plot in Figure 3, displaying the trend of speed-up and BLEU scores with different NAT models. It is shown that the point of GLAT is located on the top-right of the competing methods. Obviously, GLAT outperforms our competitors in BLEU speed-up is controlled, and in speed-up, if BLEU is controlled. This indicates that GLAT outperforms previous state-of of-the-art NAT methods. Although iterative models like maintain minor speed advantages over AT. In contrast, fully non-autoregressive models remarkably improve the inference speed.
achieves competitive BLEU scores, they only
4.3 ANALYSIS
Effect of Source Input Length To analyze the effect of source input length on the models' per- formance, split the source sentences into different intervals by length after BPE and compute

---

Under review as a conference paper at ICLR 2021
GLAT Hal Transformer
17.5
14.3
Relotive Decodlo
10.0
Performacc
90-
7.5
Tranlame
NATIT
5.
NATIOCTY
1W-MAT
28
Porfomance (BLBU)
Figure 3: The trade-off between speed-up and BLEU on WMT14 DE-EN
Figure 4: Performance under different source input length on WMT14 DE-EN
20-
Faendeour Dh-ar
Dneodetuer saw TN
Trcodequar DC-tN
Encodetger 3E
50030 150000 750203
230000 250023
Decoding Heraitons
Training limatioon
Figure 5: The BLEU scores GLAT with different decoding iterations
Figure 6: Training NAT with different initial-
ized encoder
the BLEU score for each interval. The histogram of results is presented in Figure 4 NAT-base's performance drops sharply for long sentences, while the gradual learning process enables GLAT to boost the performance by a large margin, especially for long sentences. We also find that GLAT outperforms autoregressive Transformer when the source input length is smaller than 20.
GLAT Achieves Strong Results without Multiple Iterations We conduct experiments of GLAT with more than one decoding iteration inference. We adopt the inference algorithm Ghazvinine jad al. (2019) for multiple-iteration decoding. The results are shown in Figure We find that GLAT can achieve decent performances with only one decoding iteration, while further iterations only obtain minor improvements of 0.2~-033 BLEU.
To validate whether GLAT strengthens encoders, we train NAT models with different initialization for comparison. We choose NAT-base and GLAT for comparison and the only difference between them is the training procedure. Specifically, we initialize the encoders of NAT models with the parameter of trained GLAT and trained NAT-base, respectively. The parameters of encoders are fixed during training. The models are all trained using the traditional MLE loss, and the results are presented in Figure Obviously, models initialized with encoders of GLAT outperform those with NAT-base encoders, indicating that GLAT learns better encoders.
GLAT Learns Better Encoder
Training Speed of GLAT Training on V100 GPUs for WMT14 EN-DE, NAT-base takes about 45 hours and GLAT takes about 56 hours. Compared to NAT-base, the training of GLAT is about 1.2 times slower. Because we only add one forward pass of the decoder, the overhead relatively small.
4.4 ABLATION STUDY
Effectiveness of the Adaptive Sampling Number To validate the effectiveness of the adaptive sampling strategy for the sampling number NY(,, E), we also introduce two fixed approaches for comparison. The first one decides the sampling number with À T, where T is the length of Y, and is a constant ratio. The second one is relatively flexible, which sets a start ratio X, and a end ratio Ae, and linearly reduce the sampling number from X T to A T with the training process.
7

---

Under review as a conference paper at ICLR 2021
Table 2:
Performances on IWSLTI6 with Table Performances on IWSLTI6 with decreas- ing sampling ratio.
fixed sampling ratio.
Sampling Method
À
BLEU
Schedule
Sampling Method
BLEU
A
X
0.0
24.66
0.1
24.91
0.5
0
27.80
Fixed
0.2
27.12
Fixed
0.5
0.1
28,21
0.3
24.98
0.5
0.2
27.15
0.4
22.96
0.5
0.3
23.37
Adaptive
29.61
Adaptive
29.61
Table 4: Performance on WMT14 EN-DE with different reference word selection strategies.
Selection Strategy
random
Poff
-
most certain
most uncertain
GLAT
25.21 26.55
24.87 25.83
25.37 26.52
24.99 26.22
24.86
GLAT (w/ reranking m=7)
26.13
Table 5:
Ablation study on WMT14 EN-DE and WMT14 DE-EN.
WMT14 EN-DE
WMT14 DE-EN
GLAT w/ sampling strategy of Mask-Predics 19.16
23.56
GLAT w/ decoder inputs of Mask-Paa -resict GLAT
24.99
29.48
25.21
29.84
As shown in Table 2 and Table 3 clearly, our adaptive approach (Adaptive in the table) outperforms the competitors with big margins. The results confirm our intuition that the sampling schedule affects the generation performance of our NAT model. The sampling strategy, which first offers relatively easy generation problems and then turns harder, benefits the final performance. Besides, even with the simplest constant ratio, GLAT still achieves remarkable results. When set X 0.2, it even outperforms the baseline À = 0.0 by 2.5 BLEU score.
The experiments potentially support that it is beneficial to learn the generation of fragments the start and gradually transfer to the whole sequence. The flexible decreasing ratio method works better than the constant one, and our proposed adaptive approaches achieve the best results.
To analyze how the strategies of selecting reference words affect in glancing sampling, we conduct experiments with different selection strategies. By default, we assume all the words in the reference are equally important and randomly choose reference words devise four other selection methods considering the
Influence Reference Word Selection
for glancing. Besides the random strategy, prediction first decoding. For Pee and 1 - Prets the sampling probability each reference word is Piets respectively. Similar
proportional to the output probability for the reference word Piet to the word selection strategy for masking words during inference in Mask-Predict, we also add two strategies related to the prediction confidence: "most certain" and "most uncertain". We choose the positions where predictions have higher confidence for "most certain". and vise versa for "most uncertain". The results for different selection methods are listed in Table
In comparisons, the model with the selection strategy 1 P-er outperforms the one with Prefi indicating that words hard predict more important for glancing in the training process. And we find that the random strategy performs little better than the two confidence-baseee strategies. We think this indicates that introducing more randomness in the glancing sampling could enable GLAT learn to explore more dependencies among target words. We adopt the random strategy for its simplicity and good performance.
Advantages of GLAT over Mask-Predict To study the effects of sampling strategy and decoder inputs of GLAT, report the results of replacing these two modules in GLAT with the corresponding
8

---

Under review as a conference paper at ICLR 2021
Table 6: Performance on WMT14 EN-DE and WMT14 DE-EN with different distances.
WMT14 EN-DE
WMT14 DE-EN
Hamming
Levenshtein
Hamming
Levenshtein
GLAT
25.21 26.55
24.56
29.84
28.96
GLAT (w/ reranking m=7)
26.21
31.02
30.85
part in Mask-Predict, respectively in Table GLAT employs glancing sampling strategy instead of the uniform sampling strategy used in Mask-Predice and replaces the [MASK] token with source representation from the encoder. The results show that the glancing sampling strategy outperforms the uniform sampling strategy by 5-66 BLEU scores, and feeding representations from the encoder as the decoder input could still improve the strong baseline by 0.2~0.3 BLEU scores after adopting glancing sampling. To sum up, the adaptive glancing sampling approach contributes the most to the final improvement, and the use of representations from the encoder also helps a bit.
Comparison of Different Distances for Glancing Sampling We conduct experiments with two distances for comparing the predictions of first decoding and references, and the results are presented Experimental results show that both distances can be used improve the quality of
in Table g one-iteration generation, and GLAT with Hamming distance is better than GLAT with Levenshtein distance [Levenshtein] 1966). We think Hamming distance more strict than Levenshtein distance because only the same words the corresponding positions are regarded correct, which is more consistent with the training of GLAT.
5 RELATED WORK
Fully Non- Non-Autoregreesive Models A line of work introduces various forms of latent variables to reduce the model's 's burden of dealing with dependencies among output words (Gu et al., 2018: Ma
et al. 2019, Bao et al. 2019 Ran et al. 2019). Another branch of work considers transferring the ledge from autoregressive models to non-autoregressive models (Wei et al. 2019 Li et al. 2019) Guo et al. 2019b). Besides, there are also some work that apply different training objectives train non-autoregressive models (Libovický & Helel 2018; Shao et al. 2020 Ghazvininejad et al. 2020
or add regularization terms (Wang al. 2019 Guo et al. 2019a).
Non-Autoregreesivv Models with Structured Decoding To model the dependencies between words, Sun et al. (2019) introduces CRF inference module in NAT and performs additional sequential decoding after the non-autoregressive computation in inference. [eng & Rush (2020) proposes cascaded CRF decoding. Since GLAT only performs one-iteration non-autoregressive generation, approach is orthogonal to the method proposed in Sun et al. (019). We can also combine our approach with the structured decoding method.
Non-Autoregressive Models with Iterative Refinement A series of work are devotes to semi- autoregressive models that combine the strength of both types of models by iteratively refining the former outputs. Lee et al. (2018) proposed a method of iterative refinement based on denoising autoencoder. Gu et al. 2019) utilized insertion and deletion to refine the generation. Ghazvininejad ct al. (2019) trained the model in the way of the masked language model, and the model iteratively replaces the mask tokens with outputs. Despite the relatively better accuracy, the multiple decoding iterations vastly reduces the inference efficiency of non-autoregressive models.
6 CONCLUSION
In this paper, we propose Glancing Transformer with a glancing language model to improve the performance of non-iterative NAT. With the glancing language model, the model reduces learning difficulty by gradually learning with growing targets for simultaneous generation and explicitly models target word dependencies. Experimental results show that our approach significantly improves the performance of non-autoregressive machine translation with one-iteration generation.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Dzmitry Bahdanau. Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint
2014.
Yu Bao, Hao Zhou, Jiangtao Feng, Mingxuan Wang, Shujian Huang. Jiajun Chen, and Lei Li. Non-autooregressive transformer by position learning. arXiv preprint Xi:: 2019.
Yoshua Bengio, Jéröme Louradour, Ronan Collobert, and Jason Weston, Curriculum learning. In ICML, pp. 41-48, 2009.
Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Damitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. In EMNLP, pp. 1724-1734. 2014.
Yuntian Deng and Alexander M Rush. Cascaded text generation with markov transformers. arXiv preprint arXiv:2006.0111. 2020.
Jacob Devlin, Ming-Wer Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT, pp. 4171-4186, 2019.
Marjan Ghazvininejad, Omer Levy, Yinhan Liu, and Luke Zettlemoyer. Mask-predict: Parallel decoding of conditional masked language models. In EMNLP-UICNLP, pp. 61 6114-6123, 2019.
Marjan Ghazvininejad, Vladimir Karpukhin, Luke Zettlemoyer, and Omer Levy. Aligned cross entropy for non-autoregresssve machine translation. arXiv preprint arXiv:2004.01655, 2020.
Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In ICML, pp. 369-376, 2006.
Jiatao Gu, James Bradbury, Caiming Xiong. Victor O.K. Li, and Richard Socher. Non-autoreerressive neural machine translation. In ICLR, 2018.
Jiatao Gu, Changhan Wang, and Junbo Zhao. Levenshtein transformer. In NeurlPS, pp. 11179-11189, 2019.
Junliang Guo, Xu Tan, Di He, Tao Qin, Linli Xu, and Tie-Yan Liu. Non-autoregressive neural
machine translation with enhanced decoder input. In AAAI, volume 33, pp. 3723-3730, 2019a.
Junliang Guo, Xu Tan, Linli Xu, Tao Qin, Enhong Chen, and Tie- Liu. Fine-tuning by curriculum learning for non-autoregressive neural machine translation. arXiv preprint 191.:19911.07111 2019b.
Junliang Guo, Linli Xu, and Enhong Chen. Jointly masked sequence-to-sen sequeneeece model for non- autoregressive neural machine translation. In ACL, pp. 376 376-385, 2020.
Richard W Hamming. Error detecting and error correcting codes. The Bell system technical journal, 29(2):147-100, 1950.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv: 6926 2014.
John Lafferty, Andrew McCallum, and Fernando CN Pereira. Conditional random fields: Proba- bilistic models for segmenting and labeling sequence data. In ICML, pp. 282-289, 2001.
Jason Lec, Elman Mansimov, and Kyunghyun Cho. Deterministic non-autoreereessive neural sequence modeling by iterative refinement. In EMNLP, pp. 1173-1182, 2018.
Vladimir Levenshtein. Binary codes capable of correcting deletions, insertions, and reversals. In Soviet physics doklady, volume 10, pp. 707-710, 1966.
Zhuohan Li, Di He, Fei Tian, Tao Qin, Liwei Wang, and Tie- Yan Liu. Hint-based training for non-autoregressive translation. In EMNLP- 2019.
10

---

Under review as a conference paper at ICLR 2021
Jindfich Libovický and Jindfich Helel. End-to-end aon-auttreeressivve neural machine translation with connectionist temporal classification. In EMNLP, pp. 3016 3021,, 2018.
Xuezhe Ma. Chunting Zhou, Xian Li, Graham Neubig, and Eduard Hovy. FlowSeq: Non- autoregressive conditional sequence generation with generative flow. In EMNLP-UCNLP, pp. 4273-4283, Hong Kong, China, November 2019, doi: 111111////D11-----
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. BLEU: A method for automatic evaluation of machine translation. In ACL, pp. 311-318, 2002.
Qiu Ran, Yankai Lin, Peng Li, and Jie Zhou. Guiding non autoregresssvvv neural machine translation decoding with reordering information. arXiv preprint arXiv: 1911.02215, 2019.
Chitwan Saharia, William Chan, Saurabh Saxena, and Mohammad Norouzi. Non-autoregressive machine translation with latent alignments. arXiv preprint arXiv:2004.07437 2020.
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In ACL, pp. 1715-1725, 2016.
Chenze Shao, Jinchao Zhang, Yang Feng, Fandong Meng, and Jie Zhou. Minimizing the bag-of- ngrams difference for non-autoregressive neural machine translation. In AAAL pp. 198-205, 2020.
Zhiqing Sun, Zhuohan Li, Haoqing Wang. Di He, Zi Lin, and Zhihong Deng. Fast structured decoding for sequence models. In NeurIPS, pp. 3016-3026, 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS, pp. 5998-6008 2017.
Yiren Wang, Fei Tian, Di He, Tao Qin, ChengXiang Zhai, and Tie- Yan Liu. Non-autoregressive machine translation with auxiliary regularization. In AAAI, 2019.
Bingzhen Wei, Mingxuan Wang, Hao Zhou, Junyang Lin, and Xu Sun. Imitation learning for non-autoregressive neural machine translation. In ACL, 2019.


Reviews:
[{"review_id": "feTdiFsKjcF", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "- Overall Comment\nThis submission improves non-autoregressive translation (NAT) by proposing a non-iterative parallel text generation model called Glancing Transformer (GLAT), which includes the explicit word dependency modeling in NAT via a proposed Glancing Language Model (GLM). Compared to previous work, biggest contribution of the proposed method is that it improves the training of NAT model with a similar idea of curriculum learning, while keeping the inference time unchanged, setting a significant improvement for non-iterative NAT without reranking. It would be a good baseline for future research on non-iterative NAT models. \nHowever, I still have the following comments and questions:\n- Methods\n(1) Why wouldn't the model gets stuck at predicting only part of the words correctly? \n    As described by the algorithm, the model will sample more reference words as the inputs of the decoder if the prediction is incorrect. However, the loss is only calculated for the remaining words.\nWill the model only learn to predict easy words and give up learning the difficult words? I guess the random sampling strategy might be the key but it would be good to know answers from the authors.\n(2) Since the model only updates in the second pass where the inputs are always mixed with reference words and source embeddings. There will be a clear mismatch between the second pass and the first pass. Why can the first model be sure to improve while training the second pass? Only by sharing parameters between two decoding passes?\n(3) Hamming distance is quite weak. Will the proposed method also apply to other distance such as Levenshtein distance or a learnable distance?\n(4) What do you mean by \"ratio function\" $f_\\text{ratio}$? What is the input to this function? Steps?\n- Experiments\n(1) The proposed training process is in fact very similar to Mask-Predict except the inputs (Encoder hidden states instead of [MASK]) and adjusting the number of reference during training.  It would be nice to have a fair comparison with Mask-Predict for at least two settings (and combined):\n(A) Mask-Predict with encoder hidden state inputs (with uniform copy or attention)\n(B) Mask-Predict with 1 decoding iteration.\n(2) I did not see how the model handle the length. With another network for length prediction? Or decoding with multiple lengths? For the latter case, how to decide these lengths?\n- Missing Reference\nThis paper had an even higher one-iteration NAT results (**25.8** with Imputer compared to **25.21** in the submission on WMT En-De). Although the methods are different and the difference is to be honest marginal, it is important to include discussions on that or combine them in future work.\n *Saharia, Chitwan, et al. \"Non-autoregressive machine translation with latent alignments.\" arXiv preprint arXiv:2004.07437 (2020).*\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "_N8E3DFF4SK", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose Glancing Transformer for single step parallel text generation. The approach is inspired from curriculum learning i.e. the training task is adaptively controlled based on the model's current performance. Specifically, the paper proposes a glancing strategy which compares the model's generation and reference sentence, and forms a sequence which is partially masked. The number of masked tokens in this sequence depends on the similarity between model's generation and reference sentence. The model is then trained to complete the partially masked sequence. \n\nThe model achieves strong improvements on standard non-autoregressive MT baselines while not modifying inference process, thus not compromising on inference speed over vanilla non-autoregressive models.\n\n+ves : \n- Limitations of current non-autoregressive MT models (NAT) are well explained, and the approach is nicely motivated. The paper is well written (although there are many grammatical mistakes that can be revised by the authors) and is easy to follow. The experimental details are well documented, many ablation studies are reported apart from comparison using standard metrics.  \n\n- The results on standard benchmarks are strong. The paper improves over vanilla NAT by approx. 5 BLEU points on average, and is only 1 BLEU point less than baseline autoregressive models while being ~7x faster in inference.\n\nConcerns : \n\n- While the use of curriculum learning inspired techniques to augment NAT model training is new and interesting, their specific technique does not seem to have sufficient novelty according to me. If I understood their technique correctly, the only difference in the training algorithm between Mask-Predict (Ghazvininejad et al., 2019) and their method is the selection of \"number of tokens to mask (`mask-num`)\" in the decoder's input. While Mask-Predict uses a uniform distribution to sample `mask-num` , they use a glancing sampler that decides this number based on the hamming distance between model's prediction and reference sentence.\n\n- Although the reported results and ablation studies show a significant impact of this simple change, I think more exploration of this technique is possible and should be done in the paper. E.g. A simple hamming distance may not be a good strategy to compare the model's prediction and reference sentence. A related finding has been described in detail by Ghazvininejad et al., 2020 (https://arxiv.org/abs/2004.01655). So, I believe that authors can explore more strategies to compare reference and generated sequence.\n\n- The authors use Random Sampling like Mask-Predict as their sampling strategy. The authors argue that random sampling may not be the most efficient strategy, but it's the easiest one, and has been shown to be powerful in models like BERT. While I totally agree with their argument, I believe that there is some possibility here to exploit the fact that we have access to model's prediction and reference sentence.  E.g. one possible strategy to exploit this could be selecting tokens that the model was not able to predict correctly based on the hamming distance comparison.  \n\n- In the Introduction section, it is mentioned that - \"Second, GLM replaces the [MASK] token in the MLM with signals from the encoder, thus not only\nthe target word dependency, but also the dependencies between the input and target are strengthened.\" While this is an interesting argument, I didn't find any experiment to validate this. I think the authors should include experiments to compare use of [MASK] tokens and their approach empirically. \n\nBased on these concerns, I am currently inclined to recommend rejection. While I find the idea of incorporating curriculum learning very interesting (together with strong results as demonstrated by the paper), I believe that more exploration of strategies to sample number of masked tokens, and sample words from the reference sentence is necessary to make the paper publishable. I have described this point in more detail in the Concerns above. \n\nMinor Comments : \n- There are many grammatical errors in the current version of the paper that the authors might want to revise. (E.g. in abstract - falls -> fall, achieves -> achieve). \n- In Section 3.1 authors mention this - \"Note that we use the attention mechanism to form the decoder inputs with the input X.\" I think it might be helpful to elaborate more on what this exactly means. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "NRuEyxn8p5", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a non-autoregressive neural machine translation model that does not require multiple iterations to achieve a good translation quality. The key difference to previous models is that during training it uses decoding to estimate the number of words to randomly sample (proportionally to the error) as opposed to random sampling a fixed number of them and uses this sampling strategy to mask tokens. \n\n**Strengths** \n\nThe idea of using the model to come up with the number of words to sample before predicting them looks new and it appears to be helping a lot the model to do better with a single iteration. \n\nThe evaluation shows that the proposed model outperforms previous non-iterative ones and performs on par with iterative ones in some cases while being significantly faster which is promising (but it still relies on reranking based on an autoregressive model). \n\nMoreover, the method performs much better than the non-autoregressive baseline on longer sequences (a trait that is observed in autoregressive models) and performs better than the autoregressive baseline on very short sequences (up to 20 tokens). \n\n**Weaknesses** \n\n(1) The writing requires some more effort because it has some grammatical/spelling errors and was not clear in several parts. It would help simplify wording and make more clear statements. For instance, there is a lot of time spent describing the glancing sampling strategy while it could be described in one paragraph (random sampling strategy paragraph seems redundant). \n\n(2) The proposed idea has some merits and works well judging from the results but it seems somewhat incremental and not very clearly explained (see below). The connection to curriculum learning was a bit hand-wavy and hard to follow. The model does not seem to be trained on targets that are of increasing difficulty but rather the number of incorrect targets simply defines the number of *random* samples to be used for training. Was this the intended connection to this line of work? Having many errors in the prediction during training does not necessarily mean that the example is difficult. \n\n(3) In Section 3.3, how is the function f_{ratio} actually implemented and trained (if applicable)? It's not clear from the provided description of how this is done. How are the gradients computed through the sampling process if it's trainable? This is an essential component of the model and it hasn't been explained well. \n\n(4) Another setback for the reader is the lack of discussion or acknowledgment of the computational cost that is required by the method during training. Performing decoding twice during training looks interesting from the modeling perspective but what is its effect on training speed? The evaluation focus is mainly on inference time but the training speed factor should also play a role too when deciding which method to use.\n\n(5) Related to the above, the non-autoregressive models rely on knowledge distillation and re-ranking based on a pretrained autoregressive model. This already has its own training cost, so increasing it more could lead to a situation where the benefits during inference are overshadowed by the computational cost from training.\n\n\nOther comments:\n\nAre the models trained until convergence or for a fixed number of steps?  I am wondering what is the impact of the glancing sampling strategy on the convergence. \n\nHow did the authors come up with the number of reranking candidates for each model? It looks like the number is different for each model; this should affect the quality of the model.  \n\nCould the authors elaborate on what do they mean by \"the formulation of our proposed glancing language model *could* be maximizing the likelihood of remaining words...\"? Does it actually do that? This was unclear and it leads the reader to make guesses. \n\nIn Section 4.2, what do you mean by \"BLEU and speed-up are more or less contradictory\"?\n\nIn Figure 1 and 2, the replaced inputs use the notation h_2, h_3 which points to the encoded inputs but in the textual description the replaced tokens seemed to be coming from the embedding on the decoder side (Section 3.2, paragraph 2): E_{y_t \\in GS(Y,\\hat{Y} }(y_t). There seems to be this inconsistency between the notation in the diagrams and the notation in text, which is very confusing. \n ", "extra_scores": {}, "extra_text": {}}, {"review_id": "Fu91yWWeXmU", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "*Paper Contributions*\n\nThe paper proposes the Glancing Transformer (GLAT), a model for non-autoregressive generative model of language (focusing on the MT domain). GLAT incorporates several changes which result in a model which is state-of-the-art in several MT categories for non-autoregressive models and without requiring iterative sampling.\n\n*Strong points of the paper*\n\n* The paper clearly lays out the suggested model changes, and how they related to previous work.\n* Some of the results are clearly state of the art for the task.\n* The model and changes are described well and clearly.\n\n*Weak points of the paper*\n\n* GLAT incorporates several changes from the NAT-base architecture they primarily compare against, without clear ablations to attribute cause to the improvement.\n* Most of the paper discusses the impact of architectures on inference/sampling speed, but doesn't discuss training speed, which seems relevant because my understanding of the model means that each step requires an additional decoder pass.\n* Some of the writing is poor and overly-colloquial.\n\n*Clearly state your recommendation (accept or reject) with one or two key reasons for this choice.*\n\nI believe this paper is marginally above the acceptance threshold, though I think there is a clear avenue to improve the paper and change my rating to accept.\n\n*Supporting arguments for your recommendation.*\n\nThe proposed GLAT model essentially consists of two changes from prior work:\n1) Using an unconditional forward pass of the decoder to determine which tokens to mask, and then running the decoder a second time\n2) Forwarding of encoding outputs via attention instead of mask tokens in the second decoder.\n\nMy main criticism of the paper is that is it unclear how each of the two changes contribute to the overall improvement being seen by the model. It might be the case that it is exclusively one of them, and I think the paper would clearly benefit from ablations discussing the two and showing how they each contribute.\n\nAs a secondary criticism, the paper proposes using a simple random sampling strategy to determine which tokens to mask (and which to glance at). While the authors defend this as the simple choice (and it certainly is), it feels to me that there is an obvious second alternative, which is to use the probability of the tokens generated by the decoder without glancing at anything (as in mask-predict's iterative sampling) to pick which tokens to glance at and which not to. It feels to me that this alternative is substantially obvious (seeing as it is exactly how mask tokens have been predicted in the past) that it should be attempted.\n\n*Ask questions you would like answered by the authors to help you clarify your understanding of the paper and provide the additional evidence you need to be confident in your assessment.*\n\nI would ask that two questions be answered, both ablations trying to get at the heart of the reason for improvement:\n\n1) What happens if you do not forward the decoder inputs to the second decoding pass, but instead set them to a unique MASK token as in prior work (but keep all other aspects of training the same)?\n2) What happens if you use mask-predict's probability-based sampling criterion to pick which tokens to sample, instead of the default random strategy.\n\nAlso, I have two questions which I think also should be answered, but don't require any further experimentation:\n\n3) What is the impact on glancing sampling to training time?\n4) What is the *exact* architecture of NAT-base? You show the encoder of GLAT works better than NAT-base - is NAT-base trained with the attention (instead of UniformCopy or SoftCopy)? Could that be the reason for the improvement?\n\nI feel that if the authors answer these questions (ideally with WMT benchmark data for the first two), along with addressing the more minor points I list below, that I would feel comfortable increasing my rating to \"accept\".\n\n*Additional feedback with the aim to improve the paper.*\n\nI had a few minor questions while reading that I think would be useful to address:\n* You mention your final model is based on averaging the 5 best checkpoints, how are you measuring best?\n* You make a quite strong claim that \"We think GLAT is superior to AT to some extent\", which I think needs either more qualification or more focus.\n* I struggled with understanding the paragraph starting with \"Adaptive Sampling Number\", I think it could use some clean-up.\n\nFinally, I think most of the writing is clear, but there are a few points where it becomes quite colloquial and difficult to follow, I think an additional pass cleaning up some of the structure could be quite beneficial!", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work raised quite a few questions, and left the reviewers somewhat divided. The authors have done their best to answer these questions, conducting additional experiments where needed.

The close relation of this work to Mask-Predict (Ghazvininejad et al. 2019) was noted by several reviewers. Although the current version of the manuscript addresses this, the introduction still frames Mask-Predict as an iterative model, and does not explicitly make the connection between GLAT and single-iteration Mask-Predict. My impression is that this understates the relationship between these models somewhat.

Taking single-iteration Mask-Predict as a baseline, the proposed extension is fairly simple, and seemingly effective, which is a potentially impactful combination. However, the manuscript is still held back by presentation issues (including but not limited to spelling and choice of words), and I concur with Reviewer 2 that the connection with curriculum learning should be elucidated not just in words, but with supporting experimental analysis.

Regarding training cost, given that training for GLAT seems to be more costly for the same number of training iterations, a comparison where the total compute budget is held constant could be interesting -- though I appreciate that this is not a key point of the paper, as the authors point out (whereas inference cost is).

I believe the changes made by the authors in response to the reviewers' comments are substantial enough that they merit a further review cycle, and may still fall short of the reviewers' expectations in some aspects. Therefore, I will not recommend acceptance, though I want to add that this was a tough call to make. I would also like to encourage the authors to resubmit their updated manuscript.
Decision: Reject