Non-iterative Parallel Text Generation via Glancing Transformer

1 INTRODUCTION . Non-autoregressive transformer ( NAT ) has attracted wide attention in neural machine translation ( Gu et al. , 2018 ) , which generates sentences simultaneously rather than sequentially . To enable parallel decoding , NAT imposes a conditional independence assumption among words in the output sentences , which leads to significantly faster inference speed ( almost a dozen times speed-up ) than the autoregressive Transformer ( Vaswani et al. , 2017 ) . However , NAT still falls behind autoregressive Transformer ( AT ) in the quality of output sentences , such as BLEU ( Papineni et al. , 2002 ) for machine translation . We blame it for the imposed conditional independence assumption , which prevents NAT models from explicitly learning the word dependencies in the output sentence . Note that such word dependency is crucial , and it is explicitly learned in the AT model through the autoregressive language models ( left-to-right , see Figure 1a and Figure 1b ) . Recently , Ghazvininejad et al . ( 2019 ) ; Gu et al . ( 2019 ) propose to employ the Masked Language Model ( MLM , Devlin et al. , 2019 ) in NAT , which includes word dependency modeling in an iterative fashion ( see Figure 1c ) , therefore yielding quite competitive results compared to AT . Specifically , such iterative models randomly mask words in the reference and predict these masked words conditioned on unmasked ones during training . In this manner , iterative models are trained to explicitly capture the dependencies between masked words and unmasked words . However , these iterative approaches still give poor results with one decoding iteration and have to perform multiple iterations during inference , namely iteratively refining the generated outputs of the previous iteration . Such iterative process is quite time-consuming , which partly sacrifices the speed merit of NAT . How to abandon the iterative process while enjoy the benefits of explicitly modeling word dependencies in NAT is still an open problem . In this paper , we argue that the major culprit of the problem that mask language models have to be used together with iterative inference , is the sampling strategy of masking words in MLM . In particular , MLM employs a fixed uniform strategy for masking words randomly during training , which prevents the model from effectively learning word dependencies for one-iteration generation . For example , at the beginning of training , the NAT model would be poorly tuned and we should mask fewer words . If we mask too many words , it would be difficult for the NAT model to correctly predict the masked words . On the contrary , if we mask too little words at the end phase of training , the resulting NAT model is rarely trained to predict the whole sentences , and can only predict some sentence fragments . In such a case , to accurately generate the whole sentence in inference , the NAT Under review as a conference paper at ICLR 2021 ̂y1 ̂y2 ̂y4 ̂y5 ̂y3 h1 h2 h3 h4 h5 NAT DecodingH H′ Glancing Sampling Hamming Distance N ( ̂Y , Y ) = 3 y1 y2 y3 y4 y5 ̂y1 ̂y2 ̂y4 ̂y5 ̂y3 y1 y3 y5 Replace Inputs 0.8 0.5 0.7 0.6 0.9 R an do m M as ki ng h2 h4 y1 y2 y1 y2 y5 Decoder Encoder x1 x2 x3 x4 y3 y4 y3 y4 [ BOS ] Decoder Encoder x1 x2 en co de r-d ec od er at te nt io n x3 x4 y1 y4 y5 [ MASK ] [ MASK ] y2 y3 Decoder Encoder x1 x2 en co de r-d ec od er at te nt io n x3 x4 y1 h2 y5h4y3 y2 y4 y1 y2 y5 Decoder Encoder x1 x2 x3 x4 y3 y4 h2 h4h1 h3 h5 G la nc in g Sa m pl in g en co de r-d ec od er at te nt io n en co de r-d ec od er at te nt io n y1 y5y4 y3y1 y5 ( a ) Left-to-right LM in AT ̂y1 ̂y2 ̂y4 ̂y5 ̂y3 h1 h2 h3 h4 h5 NAT DecodingH H′ Glancing Sampling Hamming Distance N ( ̂Y , Y ) = 3 y1 y2 y3 y4 y5 ̂y1 ̂y2 ̂y4 ̂y5 ̂y3 y1 y3 y5 Replace Inputs 0.8 0.5 0.7 0.6 0.9 R an do m M as ki ng h2 h4 y1 y2 y1 y2 y5 Decoder Encoder x1 x2 3 4 y3 y4 y3 y4 [ BOS ] Decoder Encoder x1 x2 en co de r-d ec od er at te nt io n 3 4 y1 y4 y5 [ MASK ] [ MASK ] y2 y3 Decoder Encoder x1 x2 en co de r-d ec od er at te nt io n 3 4 y1 h2 y5h4y3 y2 y4 y1 y2 y5 Decoder Encoder x1 x2 3 4 y3 y4 h2 h4h1 h3 h5 G la nc in g Sa m pl in g en co de r-d ec od er at te nt io n en co de r-d ec od er at te nt io n y1 y5y4 y3y1 y5 ( b ) Implicit LM in NAT ̂y1 ̂y2 ̂y4 ̂y5 ̂y3 h1 h2 h3 h4 h5 NAT DecodingH H′ Glancing Sampling Hamming Distance N ( ̂Y , Y ) = 3 y1 y2 y3 y4 y5 ̂y1 ̂y2 ̂y4 ̂y5 ̂y3 y1 y3 y5 Replace Inputs 0.8 0.5 0.7 0.6 0.9 R an do m M as ki ng h2 h4 y1 y2 y1 y2 y5 Decoder Encoder x1 x2 x3 x4 y3 y4 y3 y4 [ BOS ] Decoder Encoder x1 x2 en co de r-d ec od er at te nt io n x3 x4 y1 y4 y5 [ MASK ] [ MASK ] y2 y3 Decoder Encoder x1 x2 en co de r-d ec od er at te nt io n x3 x4 y1 h2 y5h4y3 y2 y4 y1 y2 y5 Decoder Encoder x1 x2 x3 x4 y3 y4 h2 h4h1 h3 h5 G la nc in g Sa m pl in g en co de r-d ec od er at te nt io n en co de r-d ec od er at te nt io n y1 y5y4 y3y1 y5 ( c ) Masked LM in Iterative NAT ̂y1 ̂y2 ̂y4 ̂y5 ̂y3 h1 h2 h3 h4 h5 NAT DecodingH H′ Glanci g Sampling Hamming Distance N ( ̂Y , Y ) = 3 y1 y2 y3 y4 y5 ̂y1 ̂y2 ̂y4 ̂y5 ̂y3 y1 y3 y5 Replace Inputs 0.8 0.5 0.7 0.6 0.9 R an do m M as ki ng h2 h4 y1 y2 y1 y2 y5 Decoder Encoder x1 x2 x3 x4 y3 y4 y3 y4 [ BOS ] Decoder Encoder 1 x2 en co de r-d ec od er at te nt io n x3 x4 y1 y4 y5 [ MASK ] [ MASK ] y2 y3 Decoder Encoder x1 x2 en co de r-d ec od er at te nt io n x3 x4 y1 h2 y5h4y3 y2 y4 y1 y2 y5 Decoder Encoder x1 x2 x3 x4 y3 y4 h2 h4h1 h3 h5 G la nc in g Sa m pl in g en co de r-d ec od er at te nt io n en co de r-d ec od er at te nt io n 1 y5y4 y3y1 y5 ( d ) Glancing LM in GLAT Figure 1 : Different language modeling approaches of different text generation models . model has to generate the sentence fragments iteratively . To this end , the sampling strategy is crucial for the training of NAT . To address the above issues , we propose a simple yet effective approach called Glancing Transformer ( GLAT ) , which is equipped with the proposed Glancing Language Model ( GLM ) for noniterative parallel text generation , achieving significant improvements upon strong baselines . Intuitively , GLM adopts a adaptive glancing sampling strategy , which glances at some fragments of the reference if the reference is too difficult to fit in the training of NAT . Correspondingly , when the model is well tuned , it will adaptively reduce the percentage of glancing sampling , making sure that the resulting model could learn to generate the whole sentence in a one-iteration fashion . Specifically , our proposed LM differs from MLM in two aspects . Firstly , GLM proposes an adaptive glancing sampling strategy , which enables GLAT to generate sentences in a one-iteration way , working by gradual training instead of iterative inference ( see Figure 1d ) . Generally , GLM is quite similar to curriculum le rning ( Be gio et al. , 2009 ) in spirit , namely first learning to generate some fragments and gradually moving to learn the whole sentences ( from easy to hard ) . To achieve the adaptive glancing sampling , GLM performs decoding twice in training . The first decoding is the same as the vanilla NAT , and the prediction accuracy indicates whether current reference is “ difficult ” for fitting . In the second decoding , GLM gets words of the reference via glancing sampling according to the first decoding , and learn to predict the remaining words that are not sampled . Note that only the second decoding will update the model parameters . Secondly , instead of using the [ MASK ] token , GLM directly use representations from the encoder at corresponding positions , which is more natural and could enhance the interactions between sampled words and signals from the encoder . Experimental results show that GLAT obtains significant improvements ( about 5 BLEU ) on standard benchmarks compared to the vanilla NAT , without losing inference speed-up . GLAT achieves competitive results against iterative approaches like Mask-Predict ( Ghazvininejad et al. , 2019 ) , even outperforming the Mask-Predict model on WMT14 DE-EN and WMT16 RO-EN . Compared to the strong AT baseline , GLAT can still close the performance gap within 1 BLEU point while keeping 7.9× speed-up . Empirically , we even find that GLAT outperforms AT when the length of the reference is less than 20 on WMT14 DE-EN . We speculate this is because GLM could capture bidirectional context for generation while its left-to-right counterpart is only unidirectional , which indicates the potential of parallel generation approaches like GLAT . 2 TEXT GENERATION VIA CONDITIONAL LANGUAGE MODELING . In this section , we compare different language models used in different text generation approaches . Formally , considering a sequence-to-sequence model ( Cho et al. , 2014 ; Bahdanau et al. , 2014 ; Vaswani et al. , 2017 ) for predicting Y = { y1 , y2 , ... , yT } given the input sentence X = { x1 , x2 , ... , xN } . In the AT model , the training objective is maximizing the log-likelihood with autoregressive decomposition : LAT = T∑ t=1 log p ( yt|y < t , X ; θ ) , ( 1 ) where the word yt is conditioned on the target prefix y < t = { [ BOS ] , y1 , ... , yt−1 } and the source input X . AT models sentences from left-to-right , therefore word dependencies are learned in a Under review as a conference paper at ICLR 2021 y1 y2 y3 y4bos Left-to-Right Language Modeling NAT Decoder Masked Language Modeling Glancing Language Modeling y1 y2 y3 y4 y5 AT Decoder y1 mask y5 y2 NAT Decoder y1 y5 y2 h2 h4 Encoder x1 x2 x3 x4 Encoder Random Masking Encoder Glancing Sampling y3 y4mask x1 x2 x3 x4 x1 x2 x3 x4 y3 y4 en co de r-d ec od er a tte nt io n en co de r-d ec od er a tte nt io n en co de r-d ec od er a tte nt io n unidirectional way . NAT , on the other hand , incorporates conditional independence assumption among words in a sentence with the aim of enabling parallel generation : LNAT = T∑ t=1 log p ( yt|X ; θ ) . ( 2 ) Note that , in NAT , different yt in Y are predicted simultaneously , which removes the interactions among target words in the NAT modeling . Thus , to generate fluent and faithful sentences , NAT has to model the target word dependencies implicitly , which makes the learning process quite challenging . To explicitly model the target word dependencies , previous iterative approaches , such as Mask-Predict , introduce mask language models ( MLM ) in NAT . MLM models word dependencies by learning to predict the masked words conditioned on the unmasked ones : LMLM = ∑ yt∈RM ( Y ) log p ( yt|Φ ( Y , RM ( Y ) ) , X ; θ ) . ( 3 ) Here RM ( Y ) returns some randomly sampled words from Y , and Φ replaces these sampled words in Y with the [ MASK ] token . For example , if RM ( Y ) = { y2 , y3 } , Φ ( Y , RM ( Y ) ) = { y1 , [ MASK ] , [ MASK ] , y4 , ... } . To this end , the training objective is to predict the masked words RM ( Y ) given the source sentence X and the unmasked target words . As mentioned in the Introduction , though MLM can explicitly model target word dependencies , it can hardly generate satisfactory sentences without iteration . Better language modeling approach should be explored for parallel text generation in the one-iteration way .

Summary: This submission improves non-autoregressive translation (NAT) by proposing a non-iterative parallel text generation model called Glancing Transformer (GLAT), which includes the explicit word dependency modeling in NAT via a proposed Glancing Language Model (GLM). Compared to previous work, biggest contribution of the proposed method is that it improves the training of NAT model with a similar idea of curriculum learning, while keeping the inference time unchanged, setting a significant improvement for non-iterative NAT without reranking. It would be a good baseline for future research on non-iterative NAT models.