Open-ended Long Text Generation via Masked Language Modeling
Year: 2023

Abstract:
Pre-trained autoregressive (AR) language models such as BART and GPTs have dominated Open-ended Long Text Generation (OpenLTG). However, the AR nature will decrease the inference efficiency along with the increase of generation length, which hinder their application in Open-LTG. To improve inference efficiency, we alternatively explore the potential of the pre-trained masked language models (MLMs) along with a representative iterative non-autoregressive (NAR) decoding strategy for Open-LTG. Our preliminary study shows that pre-trained MLMs can merely generate short text and will collapse for long text modeling. To enhance the long text generation capability of MLMs, we introduce two simple yet effective strategies for the iterative NAR model: dynamic sliding window attention (DSWA) and linear temperature decay (LTD). It can alleviate long-distance collapse problems and achieve longer text generation with a flexible trade-off between performance and inference speedup. Experiments on the storytelling and multi-paragraph opinionated article writing tasks show that pre-trained MLMs can achieve more than 3 × → 13 × speedup with better performance than strong AR models. Our code is available at GitHub*.

heading: 1 Introduction
text: Pre-trained language models (PLMs) like BART (Lewis et al., 2020) and GPTs (Radford et al.; Radford et al.; Brown et al., 2020) have achieved remarkable progress in Open-LTG. Through modeling languages from left to right, they can autoregressively “create” fluent and grammatical content. With the further enhancement of planning strategies (Hua and Wang, 2020; Hu et al., 2022) or high-level representation learning (Guan
∗Equal Contribution †Corresponding Author
*https://github.com/dropreg/OpenLTGMLM
et al., 2021a), pre-trained AR language models can achieve promising Open-LTG. However, the low inference efficiency of AR impedes their usability in real-world applications. Table 1 presents the inference speed of a few typical AR language models. We can see that BART (Lewis et al., 2020) requires at least 1.3 seconds to generate a story with 200 tokens on the powerful NVIDIA A100 GPU, and extra planning (Hua and Wang, 2020) can make the inference process even slower (more than 30 seconds to create a 200-tokens story). In great contrast with AR models, NAR models (e.g., BERT-CRF (Su et al., 2021)) can generate more than 12 stories with the same length within one second, but their effectiveness in open-ended long text generation has not been proven yet.
The high inference efficiency of NAR models is at the sacrifice of output dependency modeling, in which each generation is executed in parallel (Xiao et al., 2022). Thus, NAR models are mainly explored and utilized for text generation tasks with adequate input information to predict each output token of different positions and extra correlations to constrain the generation process, e.g., neural machine translation (Gu et al., 2018; Huang et al., 2022), summarization (Qi et al., 2021; Agrawal and Carpuat, 2022), sentence compression (Su et al., 2021), dialogue generation (Zou et al., 2021), and constrained story-ending generation (Yang et al.,
223
2021). To the best of our knowledge, none of the existing research explores Open-LTG with NAR models, particularly based on pre-trained MLMs.
We fill this gap by first conducting a preliminary study to calibrate the potential and limitations of a pre-trained MLM, i.e., RoBERTa (Liu et al., 2019)†, on two story generation corpora, i.e., ROCStories (ROC) (Mostafazadeh et al., 2016) and WritingPrompts (WP) (Fan et al., 2018). To achieve conditional generation, we simply use RoBERTa as both the encoder and the decoder with mixed attention (He et al., 2018) to achieve encoder-decoder cross-attention. Through experiments, we found that: (1) pre-trained MLMs can achieve competitive performance in the iterative NAR fashion for open-ended short text generation (e.g., a paragraph with around 40 tokens), (2) pre-trained MLMs fail to model Open-LTG (with about 140 tokens on average), which will generate uninformative content with high-frequency and repeated tokens (e.g., “.” and “,”). Furthermore, we offer three possible reasons for the attention mechanism of MLMs and inference strategy to explain the collapse of the iterative NAR model based on pre-trained MLMs for the Open-LTG scenario.
Inspired by the above observations, we introduce two improvement strategies: Dynamic Sliding Window Attention (DSWA) and linear temperature decay strategy (LTD) to maintain more informative context content in the iterative NAR generation. As a result, iterative NAR models based on pre-trained MLMs can achieve much longer text generation than the vanilla setting. Experiments on two OpenLTG tasks (i.e., storytelling and multi-paragraph opinionated article writing) with four widely-used datasets demonstrate that the pre-trained MLM can achieve better performance (BLEU score, ROUGE score, BERT score, and Perplexity) than multiple strong AR models without extra post-training, structure modification, or using more model parameters. Importantly, our approach can speed up the inference process due to non-autoregressive properties, making the pre-trained MLM as a promising candidate for the Open-LTG community. The RoBERTa base achieves more than 3 × → 13 × with better performance to the competitive BART.
†MLMs can achieve iterative NAR generation with the mask-predict inference strategy (Ghazvininejad et al., 2019).

heading: 2 Related Work
text: Long Text Generation Text generation tasks can be classified into two categories: directed generation and open-end generation. The directed generation (Sutskever et al., 2014; Li et al., 2015; Vaswani et al., 2017) for long text scenarios has long source than the target, which is also constrained by source sequence, e.g., neural machine translation and summarization. These tasks aim to solve the quadratic growth requirement of the memory and computational of the self-attention mechanism. The openended generation task (Guo et al., 2018; Tan et al., 2020; Goldfarb-Tarrant et al., 2020; Hua and Wang, 2020; Orbach and Goldberg, 2020; Hu et al., 2022) desire to generate more freedom content and has recently become a promising research direction. Previous works have explored multiple generation strategies to generate high-quality and fluent text, e.g., planning then generating (Guo et al., 2018; Tan et al., 2020; Goldfarb-Tarrant et al., 2020; Hua and Wang, 2020; Orbach and Goldberg, 2020; Hu et al., 2022) and introducing external knowledge (Guan et al., 2020; Xu et al., 2020). Although the above strategies enable the model to achieve significant advances, time-consuming is still a critical issue that hinders their usage in real-world applications (Guan et al., 2021a; Tan et al., 2020).
Iterative Non-autoregressive Generation Nonautoregressive (NAR) model breaks the sequential dependencies from front to back for parallel text generation (Gu et al., 2018; Guo et al., 2020; Saharia et al., 2020). Furthermore, the iterative-based NAR model (Lee et al., 2018; Gu et al., 2019; Chi et al., 2021) can achieve comparable performance with the AR model. The typical CMLM model (Ghazvininejad et al., 2019) can generate fluent results conditioned on the predictions from the previous iteration instead of previous tokens:
P(Yt|X) = P(Yt|Yt−1, X) (1)
Benefiting from this, the iterative NAR model is more flexibly compared with the AR model, which can easily generate consistent and controllable text for each iteration step. To the best of our knowledge, the iterative NAR model has never been used to solve open-ended generation. Especially, we investigate its usability for the long text scenario, i.e., target lengths between 100 and 400, which is still under-explored in the directed generation tasks.

heading: 3 Preliminary Study
text: We first present the training and inference paradigm of utilizing the pre-trained MLMs for Open-LTG (§ 3.1), e.g., BERT or RoBERTa. Then, we study the significant collapse problem in a long text generation scenario by conducting preliminary experiments on two datasets with different target lengths (§ 3.2). Finally, we investigate the reason for the above issues with an exhaustive case study and exploration tests to motivate our method design (§ 3.3), where the model can generate text in nonautoregressive manner to speed up the inference.

heading: 3.1 Text Generation via Pre-trained MLMs
text: Pre-trained MLMs are typically used as the encoder to extract the representations of sentences instead of generating texts. Previous works (Dong et al., 2019; Wang et al., 2019) have indicated that the MLM encoder can support text generation tasks via attention masks or Gibbs sampling. In contrast, we introduce mixed attention and parameter sharing to the encoder-based model to solve the sequence to sequence tasks, as shown in Figure 1.
Model Training Given the parallel text generation dataset D={(X ,Y)}|D|, we can feed the source X into the MLM encoder to obtain the representation Hlsrc of l-th layer. Concretely, each layer comprises two sub-layers, including one selfattention layer and one feed-forward layer:
H̄lsrc = Self-ATTN(Hl−1src ) +Hl−1src Hlsrc = FFN(H̄lsrc) + H̄lsrc.
(2)
Then, we random mask Y = {y1, y2, · · · , y|Y|} to obtain corrupted target YM = {y1,m2, · · · ,m|Y|} (m is the symbol of mask token “<mask>”). As before, we can obtain the representation Hltgt by using
the shared parameter MLM encoder and then try to recover the masked sequence, where the mixedattention mechanism (He et al., 2018) is applied to aggregate the source HLsrc and the target Hltgt:
H̄ltgt = Mixed-ATTN(Hl−1tgt ,HLsrc) +Hl−1tgt Hltgt = FFN(H̄ltgt) + H̄ltgt.
(3)
Mixed-attention does not break the original attention mechanism, which only utilizes the target hidden states as query vector and the concatenated vector of source and target hidden states as key and value. It is worth noting that this approach is available for transformer encoder models without additional parameters.
Specifically, we uniformly mask 1 to n (target length) tokens from Y for model training. The training objective is thus to minimize the conditional MLM loss like the pre-training stage:
LMLM = − M∑
i=1
logP(yi|X ,YM)
P(yj |X ,YM) = exp(utgt/T )∑ |u′tgt| exp(u ′ tgt/T ) ,
(4)
where M is the number of masked tokens, utgt is the output logit, and T is the temperature to re-estimate the final probability.
Model Inference We use an iterative refinement strategy to generate text like CMLM (Ghazvininejad et al., 2019). In particular, We use the fully masked sequence {m1,m2, · · · ,mn} to initialize the target sequence and predict all masked tokens at the first step. Then, we iteratively regenerate the low-confidence tokens at the subsequent iteration steps to obtain better performance. For Open-LTG, we utilize the nucleus sampling (Holtzman et al., 2019) decoding strategy instead of beam search.
Length Prediction It is necessary to obtain the target length to initialize the full mask sequence as model input before inference. Specifically, we provide two strategies: 1) Fixed Length, which initializes the target length according to the average length of the validation set or human experience. 2) Prediction Module, which uses the mean-pooling layer followed by one classification layer to predict the target length by feeding HLsrc into them:
P(Ltgt|X ) = Softmax(WL(Mean-Pooling(HLsrc))), (5)
where Ltgt is the target length, and WL is the learnable parameter. Specifically, we will adjust Ltgt according to the specific offset, which is the parameter based on the validation dataset.

heading: 3.2 Extensive Trials
text: Study Settings We use Writing Prompt (WP) and ROC Stories (ROC) datasets to conduct experiments for validating whether pre-trained MLMs can work better on Open-LTG tasks. In particular, these two datasets have different lengths for target sentences, i.e., the average length of WP is 140 and ROC is 40, and more details are given in Section 5 and Appendix A. We choose RoBERTa base (Liu et al., 2019) as our backbone model and use BLEU, ROUGE, Distinct, and Lexical Repetition metrics for evaluation. During inference, we set nucleus sampling hyper-parameter top-p=0.9, temperature T =1.0, and limit the maximum iteration steps to 6 for ROC and 8 for WP.
Results As shown in Table 2, For the ROC dataset, the RoBERTa base model obtains comparable performance with BART. However, the generation quality significantly decreases for the WP dataset, which involves much longer targets. Specifically, most of the generated results are made up of duplicated function words or punctuations, e.g., “it”, “to”, “the”, and “.”, etc, which makes the
model outputs unreadable and meaningless. One intuitive question is What causes the collapse problem in Open-LTG when using pre-trained MLMs?

heading: 3.3 Analysis and Possible Improvements
text: We show typical good case and bad case in Figure 2, which are randomly selected from the ROC and WP datasets respectively to demonstrate the generation process. For each iterative refinement step of bad case, the informative tokens will be replaced by the placeholder token “<mask>” and are replaced by the function words at the subsequent steps. Thus it is unable to generate fluent results like good case. According to this observation, we try to provide some possible explanations for the aforementioned collapse issues:
1) The most intuitive reason is that the function words are often located at the front of the output distribution, which dominates the high probability region, causing the informative tokens hard to be sampled.. The output distribution trained with the ROC dataset contains more prompt-related tokens than WP, e.g., the “swim” and “water” in the top 50 candidates of ROC output, as shown in Figure 2 (distribution histogram). Worse still, the function words dominate the high probability regions (from 35% to 45%) for the bad case and lead to terrible initialization at the first iteration step.
2) The iterative refinement mechanism depends on the token confidence of generated sequences, and it is easier for the low-confidence but infor-
mative tokens to be masked. In fact, the iterative refinement mechanism is designed for directed generation tasks, e.g., neural machine translation or summarization, which usually apply the argmax operation to sample results, and the evaluation of confidence is reasonable in different iterations. Nevertheless, we use the nucleus sampling strategy for inference in Open-LTG, which leads to the lowconfidence tokens with high priority being masked.
3) The massive absent context tokens suffer a more serious multi-modality problem on long text generation in early iteration steps. As a result, the model is inclined to generate duplicated tokens due to the multi-modal output distribution. Although iterative refinement can provide additional context to alleviate this issue, the model still cannot generate the expected results. The possible explanation is that the self-attention layer needs the context token as key-value pairs to calculate the token representation. Unfortunately, the massive uninformative mask tokens (“<mask>”) in context lead to model collapse steadily worsening in the following iteration steps. Thus, we utilize the recurrent generation mechanism for model training and inference to reduce the context dependency, which can also flexibly control the maximum length of the generated sequence (please refer to the Appendix B for more details about the model architectures and experiments). The results are shown in Table 3. We can observe that the model can gradually improve its performance as the recurrent steps increase, demonstrating that informative context dependency is the implicit reason for the model collapse.
Improvements Based on the above analysis and findings, we categorize these critical factors into two types: the defects of attention mechanism and inappropriate inference strategies. In particular, we believe that each token should not pays attention to all context information, and most tokens only need the neighbor tokens’ information to represent the hidden states and predict the results. Therefore, we will change the self-attention mechanism of the pre-trained MLMs so that each tokens can attend to the restricted neighbors. Besides,
we will adjust the confidence score of the output distributions to keep the informative tokens in subsequent iteration steps instead of being masked.

heading: 4 Method
text: In this section, we propose two simple yet effective strategies for attention mechanism and inference to mitigate the model collapse problems: Dynamic Sliding Window Attention (DSWA) and Linear Temperature Decay (LTD). These designs do not break the paradigm of MLM so that it can flexibly adapt to the pre-trained models.

heading: 4.1 Dynamic Sliding Window Attention
text: We first introduced the sliding window mechanism (Beltagy et al., 2020) for the self-attention layer to adjust each token’s attention pattern, which also ensures that the top layer’s token representations can have a large receptive field, similar to CNN (Wu et al., 2018). Figure 3 illustrates the attention mask of the mixed attention layer of pretrained MLMs. It is worth noting that the key-value pairs consist of two parts: the source representation of the last layer (with green background) and the target representation of the current layer (with yellow background):
H̄ltgt = Mixed-ATTN(Win(Hl−1tgt ),HLsrc) +Hl−1tgt Hltgt = FFN(H̄ltgt) + H̄ltgt, (6)
where the operation Win(◦) employs a fixed-size window to select the neighbor token representations. Meanwhile, the query can attend all source sequence hidden states and the target sequence hidden states in the window, stemming the impact of massive absent context.
Dynamic Schedule Intuitively, it is not essential to use a fixed receptive field for each layer, e.g., the top layer may need to reduce the receptive field to perform prediction. Thus, we propose a dynamic schedule strategy for the inference stage to adjust the window size Swin of each layer:
Swin = max(αmin, L− i L ∗ αmax) ∗ Sfix, (7)
where i is the current layer number, L is the max layer number of pre-trained MLM encoder, Sfix is the fixed window size for model training, and the αmin and αmax is the lower and upper bound of coefficient hyper-parameter selected from [0, 1].
With this strategy, we can alleviate the multimodality problem by restricting the model to attend to the tokens in the window instead of the whole sequence, thus degenerating the multi-modal distribution into a uni-modal distribution. As a bonus, the top-p candidates of output distribution can contain more informative tokens.

heading: 4.2 Linear Temperature Decay
text: To further improve the effectiveness of sampling, we use the confidence-based iterative refinement by adjusting the temperature with linear schedule:
P(yi|X ,Win(YM)) = exp(ul/T )∑ l′ exp(ul′/T ) , T = β ∗ (1− t T ),
(8)
where β is hyper-parameter, t ∈ {0, · · · , T} is the current iteration step, and T is the maximum iteration step. Actually, the output distributions will be flattened when T > 1, and become sharp when T < 1. Therefore, by applying this strategy, we can penalize the distribution from peaked to flat in the former iteration steps and encourage it from flat to peaked in the later steps. The aforementioned process is shown in Figure 4.

heading: 4.3 Training and Inference
text: Given the parallel data, we use vanilla self-attention to obtain source sentence representation and sliding window mixed-attention with fixed window size to
generate the target during the training stage. During the inference, we apply DSWA to the mixedattention layer and LTD to sample the results according to the probability distributions.
Besides, the model uses the ground truth tokens as context to predict the masked tokens during the training stage and applies the randomly sampled tokens as context during the inference stage. This discrepancy makes the model only refine a fraction of the low confidence tokens, which causes the degeneration in practice. Thus, we update all target tokens according to model predictions at each iteration step by utilizing the SMART mechanism (Ghazvininejad et al., 2020).

heading: 5 Experiments

heading: 5.1 Settings
text: Datasets We conduct experiments on three OpenLTG tasks, i.e., storytelling (ROC (Mostafazadeh et al., 2016), WP (Fan et al., 2018), and WikiPlots and multi-paragraph opinionated article writing (OPINION (Hua and Wang, 2020)). For ROC datasets, we follow (Guan et al., 2021b) to mask all the names with specific placeholders to improve the generation ability. We fine-tune the model using our approach without additional corpus. More details are illustrated in Appendix A.
Implementation & Baselines We utilize the pretrained RoBERTa base‡ as our backbone model and implement all experiments with the open library Fairseq toolkit§ (Ott et al., 2019). In addition, we also compare our method with the strong baselines, e.g., the widely-used AR models like BART (Lewis et al., 2020), HINT (Guan et al., 2021b) for storytelling tasks, and PAIR (Hua and Wang, 2020) for multi-paragraph level text generation task. It is worth noting that the layer and model parameters of RoBERTa (125M) are close to BART (140M), so it can be used to compare the inference speed directly. For the inference stage, we set the max iteration step as 6 for ROC and 8 for others. We set the hyper-parameter αmin=0.125, αmax=0.75, and window size Swin equals 64. We set top-p=0.9 for all baseline models, set β=1.6 for ROC and 1.8 for WP and WikiPlots, and set β=1.5 for OPINION.
‡https://dl.fbaipublicfiles.com/ fairseq/models/roberta.base.tar.gz
§https://github.com/facebookresearch/ fairseq
Evaluation Metrics We utilize BLEU (B-n) (Papineni et al., 2002), ROUGE (R-n) (Lin, 2004), Lexical Repetition (LR-n, 4-gram repetition for n-times) (Shao et al., 2019), Semantic Repetition (SR-n, average top-n semantic similarity between any two sentences) (Guan et al., 2021b)¶, average semantic overlap (S-m, average semantic similarity of all the sentences), Distinct (D-n) (Li et al., 2016) and BERTScore (Zhang et al., 2019) for the storytelling task. As for the multi-paragraph opinionated articles writing, we utilize B-n, R-n, and METEOR (Banerjee and Lavie, 2005) to evaluate the results. The settings of n are mainly due to the length of the generated text and details are illustrated in each subsection below. We report the LR-2 and SR-1 for ROC stories and LR-5 and SR-10 for WP to reflect the lexical and semantic repetition of the generation texts. We also report the Repetition and Distinct scores of ground truth as a reference. We calculate the perplexity (PPL) using GPT2 (Radford et al.) for each model, which is the most common fluency metric.

heading: 5.2 Main Results
text: Table 4 summarize the evaluation results on each storytelling test set. We choose the appropriate checkpoint based on the repetition and distinct comparison with the ground truth of the validation set. We can observe that our approach achieves better performance on all datasets than the strong baseline model. Especially, The text generated by the RoBERTa model has high-quality and fluent results, which have high BLEU, ROUGE, BERT scores,
¶https://huggingface.co/sentencetransformers/bert-base-nli-mean-tokens
and lower perplexity, demonstrating the effectiveness of our model.
For the OPINION dataset, we use the specific plans to initialize the model input and then try to generate the missing text according to PAIRfull settings, where these special plans are extracted from the ground truth. The results are shown in Table 5. The PAIR results are based on BART, the AR model, so it has high quality even without refinement. Our model achieves better results than PAIR when using iterative refinement, demonstrating that as a masked language model, RoBERTa is more suitable to complete the planning sequence than an AR model. In addition, we found that the model works better without dynamic sliding window attention, because the additional context information provided a good initialization to the model.

heading: 5.3 Ablation Results
text: We conduct the ablation study in Table 6 to evaluate the effectiveness of each inference strategy. We can observe the performance drops when without using any strategy, and this phenomenon is significant for longer WP datasets. In particular, the results are more in tune with the current prompt benefit from the DSWA, such as better BLEU and ROUGE, and
the model generates more repetition text without LTD. Thus, the DSWA and LTD are crucial for Open-LTG, which can reduce the context dependencies to predict the output distribution better, and improve the confidence score for each iteration step to adopt the open-ended scenarios.

heading: 6 Analysis and Discussion

heading: 6.1 Speedup for Inference
text: Figure 5 illustrate the generation speed with the NVIDIA A100 GPU, which all run with the batch size equal to 1 on each test dataset. Our model can speed up the inference from 3 × to 13 × with different target lengths, i.e., from 133 token/s to 391 token/s for the ROC dataset, from 137 token/s to 882 token/s for the WP dataset, and from 132 token/s to 1753 token/s for the WikiPlots dataset. Although the smaller iteration step can further accelerate the speed, the perplexity drops significantly.

heading: 6.2 Length Prediction
text: We validate the different length prediction strategies on the WP dataset, as shown in Table 7. We initialize the full mask sequence with ground truth length to inference. For the prediction method, we select the specific offset according to the validation set, e.g., −20 for WP and −100 for WikiPlots. Besides, the prediction modules work better for short text dataset ROC with offset 0. We also found that the fixed strategy obtained comparable perfor-
mance with a slight drop, even the prediction is also a viable choice for the inference stage.

heading: 6.3 Human Evaluation
text: For human evaluation, we compare our method with strong baseline BART. We sample 100 cases from the model outputs on three different datasets in total. We hire three annotators to give their preferences (win, loss and tie) for three evaluation criteria: fluency, coherence, and relevance, which reflect the intra-sentence linguistic quality (Xu et al., 2020), inter-sentence relatedness & causal dependency and consistency of the generation results, respectively. More details are illustrated in Appendix C. We apply the Fleiss’ kappa (Fleiss, 1971) to measure the agreement among three annotators, and the results are listed in Table 8, where we report the percentage(%) of each preference when comparing with BART model. We can observe that our method can achieve better performance on three criteria when comparing with the BART model, especially for the relevance criterion, which indicates that such a NAR generation paradigm can mitigate the inconsistent issues of long text generation tasks. It is worth noting that all the inter-annotator agreements are either moderate (κ ∈ [0.4, 0.6]) or substantial (κ ∈ [0.6, 0.8]). Besides, we also plot the detailed percentage for ROC, WP, and WikiPlots on Figure 6, which can clearly exhibit the discrete distributions across three datasets. The fluency and coherence of the sentence generated by our models obviously decreased as the length increased, similar to the BART model. We will improve the text quality and overall fluency and solve the above problems for Open-LTG scenarios in future work.

heading: 7 Conclusion
text: This paper explores Open-LTG with NAR models based on pre-trained MLMs. We first examined the potential and limitations of MLMs along with the iterative NAR inference for open-ended text generation and observed that MLMs would collapse for Open-LTG. Through extensive study and analysis, we found the reason is the inappropriate attention mechanism and inference strategies, and introduced two simple strategies to alleviate such a problem, i.e., dynamic sliding window attention and linear temperature decay. Experiments demonstrate that our model achieves competitive performance and significant speedup. We hope our research can make pre-trained MLMs as new candidates for the Open-LTG community.

heading: 8 Limitation
text: Although our NAR approach can generate fluent and meaningful text, it inevitably suffers from the typical generation problems like in the AR fashion: (1) off-prompt: the provided prompt is very short, which causes the model can not focus on meaningful content and generate reasonable text. Besides, the model usually simply copy prompt text to generate results instead of planning reasonable content, such as the case 3 as shown in Table 13 in Appendix D. (2) incoherent between sentences: When the model is initialized, it does not consider the logical order between sentences, so it can only rely on the training data to learn automatically. We will consider how to generate a suitable initialization to help the model generate coherence results. Our paper’s primary concern focuses on accelerating the generation speed, and we will put how to solve these problems in future work.
Ethics Statement
Our method heavily relies on the pre-trained language models, e.g., RoBERTa, which may inherit the problematic biases (Radford et al.). We have
attempted to mitigate these issues by conducting experiments on comparatively innocuous story generation and opinion generation tasks. Furthermore, we have replaced all the names in those corpora with special placeholders. Although some measures are taken to mitigate the problematic biases, such issues cannot be solved completely. Thus, we urge the users to carefully examine the generation results and cautiously apply our method in real-world applications. Additionally, it is worth noting that all the corpora used in our experiments are only for scientific research.
As for the human evaluation process, we resort to open source web library Django|| to build our own human evaluation interface. Before releasing the human evaluation cases, we carefully check that there is no private information or other problematic biases in the cases. Besides, we did not collect personal information or ask the annotators about their private information during the annotation process. We hired three annotators and paid each of them $0.29 for each case comparison. The payment is reasonable since there are only 100 cases for annotation, and it would cost average 4 hours for one to finish all the comparisons.

heading: Acknowledgements
text: This work is supported by the National Science Foundation of China (NSFC No. 62206194), the Natural Science Foundation of Jiangsu Province, China (Grant No. BK20220488), and JSSCBS20210661. This work is also supported by Beijing Academy of Artificial Intelligence (BAAI).

heading: A Dataset and Pre-processing
text: The statistic of each dataset is shown in table 9, and we provide the download address of OPINION **, ROCStories, WritingPrompts ††, and WikiPlots ‡‡. In particular, we have to pre-process the dataset to ensure RoBERTa can handle each sample. We first use the NLTK tokenizer to split each sample into individual sentences, generally according to punctuation as a separator. Then, we collect the segment with a pre-defined segment number K to make the different pieces hold comparable lengths. Finally, we truncate the sample with a sequence length over 512 to satisfy the BERT maximum length limitation. Furthermore, we also provide the library version or link information, which is used in our paper: Transformers == v4.0.0, NLTK == v3.5, and evaluation scripts §§.

heading: B Recurrent Segment Generation
text: As shown in Figure 7, to gradually increase the context during the decoding stage, we divide the onepass parallel decoding into multiple decoding steps. Specifically, we split the target Y into multiple segments {S1,S2, · · · ,SK}, where each segment consists of multiple tokens/sentences by specifying the length of each segment. Then, the model will generate those segments incrementally, ensuring that each decoding step depends on the previously generated context to provide adequate information. In other words, we introduce NAR to generate each segment and use recurrent segment generation to keep segment-level coherence. Meanwhile, the model can obtain a flexible decoding paradigm by manipulating the length of the segments, e.g., the model can achieve one-pass decoding when setting the segment as the whole target sequence and
**https://drive.google.com/file/d/ 1gs_4fJj3U6Mrt8ekNIoDHRwSUc9WQbzp/view
††https://drive.google.com/drive/ folders/1i_2YfzpDnfuLyyctOyDabn3Br0OcK1Tj ?usp=sharing
‡‡https://github.com/markriedl/ WikiPlots
§§https://huggingface.co/evaluatemetric
achieve AR decoding (same as BART) when setting the segment as one single token.
Concretely, we feed the input text X into the BERT model to obtain the representation Hsrc. We then concatenate the hidden states of the input and previously generated context segments to feed them into the decoder mixed-attention layer and generate the k-th segment:
H̄lSk = Mixed-ATTN(Hl−1Sk ,H L src, H̃LS<k ) +Hl−1Sk HlSk = FFN(H̄lSk ) + H̄lSk , (9)
where H̃LS<k is the representation of the previous segment using the ground truth instead of the generation results HLS<k to guarantee the reliability of the context information. The model recovers the kth masked segment and calculates the cross-entropy of those masked tokens SM as the the MLM loss:
LMLM = − K∑
k=1
|SM |∑
j=1
logP(Skj |X ,S<k,Skj\M), (10)
where Skj\M is the observed tokens of k-th segment. Besides, we will select a segment number before model training and then use it to split the training data, ensuring the same number of segments for training and inference in the experiment.

heading: C Human Evaluation
text: We show the human evaluation interface in Figure 8 that was built using the python web library Django ¶¶. To test the generation ability between our method and the strong AR model (BART) in different generation tasks, we sample cases for different tasks. The statistic of sampled evaluation datasets is shown in Table 10. In each comparison, each annotator is shown with one model input (prompt) and two outputs generated from two models, namely “Text A” and “Text B”. Then, the annotators are asked to select the better text in each comparison in terms of fluency, coherence, and relevance. In case of a situation where annotators
¶¶https://www.djangoproject.com
think two texts are hard to distinguish, the “Tie” choice is allowed. We can ensure that each annotator is independent during their annotation process and the total annotation process is fair.

heading: D Case Study
text: We randomly selected some cases from different datasets to facilitate the evaluation, which was generated by the BART and our model. Table 11 illustrates the results on the ROC dataset, and we can see that our model results are close to the prompt text benefit from the NAR fashion. For example, topic case 2 is about “candy", the BART generates the sentence with fruit “grapes," instead, our model generates the “chocolate," and the whole sentence is close to the topic candy. Furthermore, our model can generate a high correlation for different sen-
tences, such as “plants, seeds and watered. finally, i had a beautiful garden." in case 11. We also provide the results of WP and WikiPlots for Table 12 and Table 13. Although these results are relatively ungrammatical and incoherent, the pre-trained MLM (RoBERTa) achieves competitive performance as BART.
Besides, the results have some grammar errors for our models, e.g., “when i got home i went to the kitchen.” in case 10. The possible explanation is that the non-autoregressive model may generate grammatically incorrect sentences during the iteration refinement procedure due to multi-modality problems. We will add grammar corrections for each iteration in future work to help the model produce better results.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: A1. Did you describe the limitations of your work?
Left blank.
A2. Did you discuss any potential risks of your work? Left blank.
A3. Do the abstract and introduction summarize the paper’s main claims? Left blank.
A4. Have you used AI writing assistants when working on this paper? Left blank.

heading: B Did you use or create scientific artifacts?
text: Left blank.
B1. Did you cite the creators of artifacts you used? Left blank.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Left blank.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Left blank.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Left blank.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Left blank.
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Left blank.

heading: C Did you run computational experiments?
text: Left blank.
C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Left blank.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Left blank.
C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Left blank.
C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Left blank.

heading: D Did you use human annotators (e.g., crowdworkers) or research with human participants?
text: Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Left blank.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Left blank.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Left blank.

heading: Future Work
text: We will add grammar corrections for each iteration in future work to help the model produce better results. ACL 2023 Responsible NLP Checklist