Teaching the Pre-trained Model to Generate Simple Texts for Text Simplification
Year: 2023

Abstract:
Randomly masking text spans in ordinary texts in the pre-training stage hardly allows models to acquire the ability to generate simple texts. It can hurt the performance of pre-trained models on text simplification tasks. In this paper, we propose a new continued pre-training strategy to teach the pre-trained model to generate simple texts. We continue pre-training BART, a representative model, to obtain SimpleBART. It consistently and significantly improves the results on lexical simplification, sentence simplification, and document-level simplification tasks over BART. At the end, we compare SimpleBART with several representative large language models (LLMs).

heading: 1 Introduction
text: Text simplification (TS) is a task in the field of natural language generation. It aims at rewriting a complex text into simple text while keeping the primary meaning intact (Laban et al., 2021).
Recently, several works have leveraged pretrained models for TS (Omelianchuk et al., 2021; Devaraj et al., 2022). However, problems arise when pre-trained models are applied to TS directly. In the pre-training stage, the model hardly acquires the ability to generate simple texts. The improvement of results on simplification tasks relies almost on the fine-tuning stage. It can hurt the performance of pre-trained models, especially for lowresource sub-tasks like lexical simplification. One reason for this shortcoming is the pre-training strategy. It randomly masks text spans in ordinary texts, teaching the model to generate ordinary texts rather than simple texts.
We are committed to adapting the pre-trained model to TS in this paper. The pre-trained model has gained the ability to generate ordinary texts, and it is costly to start pre-training from scratch. Therefore, we focus on the continued pre-training strategy (Gururangan et al., 2020). We first aim to
continue pre-training on simple texts because it contains plenty of simple words. In TS, simple texts are derived almost from SimpleWiki (Zhang and Lapata, 2017) and Newsela (Xu et al., 2015). We identify simple text spans in simple texts and dynamically replace them with <mask> tokens. Then, the pre-trained model will learn by reconstructing simple words. Meanwhile, we expect the pretrained model to learn from ordinary texts. We use a dictionary to replace complex words in ordinary texts with simple words. We also ensure the quality of the replaced sentences.
Based on BART (Lewis et al., 2020), we continue pre-training to teach it to generate simple texts and obtain SimpleBART. We then conduct experiments on three main tasks of TS: sentence simplification, lexical simplification, and documentlevel simplification. SimpleBART achieves consistent and noticeable improvements across several datasets on all three tasks over BART and several other baselines. The results illustrate that our proposed strategy helps the pre-trained model to gain the ability to generate simple texts.
To summarize, our contributions include: (1) We propose a new continued pre-training strategy to teach the pre-trained model to generate simple texts. (2) We continue pre-training BART, a representative seq2seq model, to obtain SimpleBART. It can be used for several simplification tasks and achieve consistent performance improvement. Code and SimpleBART will be released at https://github.com/RLSNLP/SimpleBART.

heading: 2 Methodology
text: As illustrated in Figure 1, our strategy is divided into two parts: learning dynamically to reconstruct simple words from simple texts and from ordinary texts where complex words are replaced with simple ones.
9345

heading: 2.1 Masking Simple Words in Simple Texts
text: We need to identify the simple words in simple texts at first. We take advantage of the DeepBlueAI model (Pan et al., 2021) that achieves state-of-theart results on the lexical complexity prediction task (Shardlow et al., 2021). A text span of length n consists of n words. The input to the DeepBlueAI model is a text span and the output is a complex value between 0 and 1. The closer this value is to 0, the simpler the text span.
Unlike the previous constant mask probability, in our strategy, the simpler a text span is, the higher its probability of being masked. This means that the mask probability is dynamic. We also set a complexity threshold of T . If the complexity c of a text span exceeds T , we will not mask this span. In our experiments, we set T to 0.25 as an empirical value. Following Lewis et al. (2020), we set the max mask probability to 0.15, and the length of a text span obeys a Poisson distribution (λ = 3). Finally, the mask probability m is calculated as:
m =    0.15× (1− 1 T · c), c ≤ T
0, c > T (1)
The function to mask the text span is denoted as g(·). Given a sentence x, the pre-trained model will learn to reconstruct x from the masked sentence:
l(x) = −logP (x|g(x)) (2)

heading: 2.2 Replacing Complex Words in Ordinary Texts
text: We also expect the pre-trained model to learn helpful information from ordinary texts. However, ordinary texts contain more complex words than simple ones, making the pre-trained model learn to reconstruct simple words much less frequently. We
introduce the dictionary SimplePPDB++ (Maddela and Xu, 2018) to address this issue. It contains millions of paraphrase rules with readability scores. Therefore, we can replace the complex words in ordinary texts with simple words. Then, the pretrained model will learn to reconstruct these simple words as in Eq.(2).
Nevertheless, a word may have different meanings in different sentences. Using a dictionary to replace complex words may change the meaning of the original sentence. Therefore, we use BERTScore (Zhang et al., 2019) to calculate the similarity between the original and replaced sentences to avoid this problem. We will discard the replaced sentences if the calculated BERTScore is lower than a similarity threshold. In our experiments, the similarity threshold is set to 0.95 as an empirical value.

heading: 3 Experimental Settings

heading: 3.1 Continued Pre-training
text: We select the BART-Large model to continue pretraining. It is a representative seq2seq model suitable for three main simplification tasks. We follow the task-adaptive pre-training method (Gururangan et al., 2020) and continue pre-training on the training set of the corresponding simplification task, ensuring that the continued pre-training texts have no intersection with the test set. We refer to the pretrained models obtained by our strategy collectively as SimpleBART.

heading: 3.2 Simplification Tasks
text: We select three representative tasks for experiments: sentence simplification, document-level simplification, and lexical simplification. For sentence simplification, we conduct experiments on Wikiauto (Jiang et al., 2020) and Newsela (Xu et al., 2015). Wikiauto is only a training set, so we use
Turkcorpus (Xu et al., 2016) as its validation and test set. Following Sun et al. (2023), we use SARI (Xu et al., 2016) and BERTScore (Zhang et al., 2019) as the evaluation metrics. BLEU and FKGL have been proven to be unsuitable for evaluating simplification (Sulem et al., 2018; Tanprasert and Kauchak, 2021). For document-level simplification, we conduct experiments on the D-Wikipedia dataset (Sun et al., 2021). We use D-SARI (Sun et al., 2021) as the evaluation metric. For lexical simplification, we conduct experiments on LexMTurk (Horn et al., 2014) and BenchLS (Paetzold and Specia, 2016). We use precision, recall, and F1 score as the evaluation metrics. For more hyperparameter setting details, please refer to Appendix B.

heading: 4 Results

heading: 4.1 Sentence Simplification
text: To demonstrate the advantages of our strategy, we develop BART-CP for a fair comparison. It continues pre-training with the same number of steps on the same data using the previous pre-training strategy from Lewis et al. (2020). In the continued pre-training stage, text spans are masked randomly.
We choose EditNTS (Dong et al., 2019), T5base (Raffel et al., 2020), and ControlTS (Maddela et al., 2021) as baselines. T5-base is close to SimpleBART in size. ControlTS achieves the state-ofthe-art result on the Newsela dataset. Following Alva-Manchego et al. (2021), BERTScoreprecision (BS) is also reported. From Table 1, the BS scores of all outputs are high enough, which means that the outputs are of high quality. According to SARI,
the most important automatic evaluation metric for sentence simplification, SimpleBART improves SARI values over BART by 1.2 points and 1.5 points, respectively. Overall, it achieves comparable results to the advanced model for the sentence simplification task. We also notice that SimpleBART outperforms BART-CP, demonstrating the effectiveness of our proposed strategy. The example outputs are given in Appendix D.

heading: 4.2 Lexical Simplification
text: We focus on generating suitable words using the pre-trained model, which is a critical step in lexical simplification. We follow Qiang et al. (2020a) and let the pre-trained models generate several candidate words. BenchLS and LexMTurk are just two test sets, so we continue pre-training on the Wikiauto training set. We choose Paetzold-NE (Paetzold and Specia, 2017a) and LSBert (Qiang et al., 2020b) as two baselines. LSBert achieves the stateof-the-art result in this task.
As shown in Table 2, SimpleBART improves the F1 scores over BART by 8.6 points and 9.7 points, respectively. It achieves comparable results to LSBert. The results also demonstrate that BART needs to gain the ability to generate simple words and the importance of introducing continued pretraining when training data is scarce.

heading: 4.3 Document-level Simplification
text: SimpleBART also performs well on the documentlevel simplification task. We choose BertSumextabs (Liu and Lapata, 2019), which achieves the state-of-the-art result on this task as a baseline. Compared with BART, SimpleBART improves the
D-SARI value by 1.8 points, making it the new state-of-the-art result.

heading: 5 Analysis

heading: 5.1 Human Evaluation
text: We hire three workers to conduct a human evaluation of the 100 randomly selected outputs of the sentence simplification task. Following Dong et al. (2019), workers rate on simplicity (Simp), fluency (Flu), and adequacy (Ade) on a 5-point Likert scale. Following Xu et al. (2016), we use simplicity gain (S+) to demonstrate how many word-level simplifications occur in sentence simplification.
Table 4 shows that SimpleBART achieves the highest Simp score among all the simplification models, close to that of the reference. SimpleBART also significantly makes more word-level simplifications compared to BART and BART-CP.

heading: 5.2 Domain Adaptation
text: Continued pre-training using our strategy on taskrelated data can improve the results. However, we still want to know if continued pre-training on more data from the same domain and different domains will improve the results. We design the following experiments. 1) Exp1: We continue pre-training on more sentences from Wikipedia and SimpleWiki, except those contained in the Wikiauto dataset. 2) Exp2: We continue pre-training on more sentences in the Newsela corpus, except those contained in the Newsela dataset. The sizes of the above texts used for continued pre-training are roughly five
times larger than the simplification training set. 3) Exp3: We continue pre-training on the Newsela training set. 4) Exp4: We continue pre-training on the Wikiauto training set.
From the results of Exp1 and Exp2 in Table 5, continued pre-training on more texts from the same domain can still enhance the simplification results. Compared to BART in Table 1, the SARI values improve by 0.6 and 1 point, respectively. From the results of Exp3 and Exp4, continued pre-training on more texts in a different domain can instead harm the results. Compared to BART, the SARI values decrease by 0.3 and 0.5 points, respectively. Thus, we suggest that future researchers use texts within the same domain (e.g., Wikiauto and Wikipedia) for continued pre-training in text simplification.

heading: 5.3 Generating Complex Texts
text: There are numerous studies dedicated to simplifying complex texts. Nevertheless, none has attempted to rewrite simple texts into complex ones. We make such an interesting attempt. We have changed our strategy to mask complex words and name the obtained model ComplexBART. When fine-tuning and testing on the Newsela dataset, we use simple texts as input and complex texts as reference.
From Table 6, ComplexBART improves the SARI value by 1.5 points over the BART model, indicating that the modified strategy can help the pre-trained model learn to generate complex texts. Thus, ComplexBART can serve as a better baseline for generating complex texts in the future.

heading: 6 Comparing SimpleBART with Large Language Models
text: Large language models (LLMs) have received widespread attention from researchers recently and have achieved state-of-the-art results on many natural language generation tasks. In this section, we select several representative large models to conduct experiments on text simplification and compare them with SimpleBART. We hope these results can serve as baselines for future research.
We choose those LLMs that provide API or model files to ensure reproducibility. We choose GPT-3.5-Turbo-03011, FLAN-T5-XL (Chung et al., 2022), and LLaMA-7B (Touvron et al., 2023) as LLM baselines and use zero-shot generation. Then, we follow the implementation2 and fine-tune FLAN-T5-base as another baseline. We collect the training sets of Wikiauto, Newsela, and DWikipedia and conduct instruction fine-tuning.

heading: 6.1 Comparison and Analysis
text: The comparison of SimpleBART results with those of the LLMs is shown in Tables 7, 8, and 9.
For the sentence-level simplification task, LLaMA and FLAN-T5-XL seem unable to understand the prompt for simplifying sentences, and they are inclined to repeat the original text. However, FLAN-T5-base, only 10% of the parameters of the above two models, performs better. It illustrates fine-tuning phase can improve performance when the model is not super large. It may be a little strange that GPT-3.5 performs worse than SimpleBART. We find that with the zero-shot setting, GPT-3.5 may not know the “degree of simplification” we want. It makes many reasonable changes to the original text, but it also keeps some of the complex parts of the original text.
For the document-level simplification task, LLaMA over-repeats sentences from the original article, and the generated text is difficult to read. The shortcomings of GPT-3.5 are similar to those of the sentence-level simplification task. Besides, limited by the number of API accesses per minute of OpenAI, we only select 1000 original documents for simplification, which takes nearly five hours.
For the lexical simplification task, neither the LLaMA nor the FLAN-T5 model could understand
1https://openai.com/blog/chatgpt 2https://github.com/philschmid/
deep-learning-pytorch-huggingface/blob/main/ training/deepseed-flan-t5-summarization.ipynb
the instruction to replace complex words with simple words. However, GPT-3.5 outperforms the other models substantially. We also find that GPT3.5 makes many sensible substitutions not included in the reference, such as replacing “acquired”with “earned”. Such results illustrate that LLMs are dominant for this task.

heading: 7 Conclusion
text: In this paper, we are committed to adapting the pre-trained model to text simplification. We propose a new pre-training strategy to allow the pretrained model to learn to generate simple texts. The adapted pre-trained model improves the results on various simplification tasks.
Limitations
The limitation of our method comes from the requirement to identify simple words in simple texts in Section 2.1. The DeepBlueAI we have used is a deep model, meaning it takes much time when inference. In our experiment, it takes 362.78 seconds to identify simple words from 10,000 sentences with an average length of 8.12. We expect that there will be methods with higher identification accuracy and higher inference speed in the future.
Due to page limitations, we have placed the related work in Appendix A and the ablation experiments in Appendix C.
Due to time constraints, we do not perform a human evaluation of the output of LLMs. We hope to conduct a more comprehensive evaluation of the performance of LLMs in the future.
Ethics Statement
The texts we have used for continued pre-training come from Wikipedia dumps and the Newsela Corpus. Using Wikipedia dumps requires following the CC-BY-SA license and GFDL. Using Newsela Corpus requires authorization, and we have received it.
This paper contains a human evaluation. We hire three experienced workers to perform it. In the recruiting process, we follow a first-come, firstserved order. We pay much more than the local minimum hourly rate.

heading: Acknowledgements
text: We thank Mounica Maddela for her help on the ControlTS baseline. This work is supported by National Key R&D Program of China (2021YFF0901502), National Science Foundation of China (No. 62161160339), State Key Laboratory of Media Convergence Production Technology and Systems and Key Laboratory of Science, Technology and Standard in Press Industry (Key Laboratory of Intelligent Press Media Technology). Wei Xu and Xiaojun Wan are the corresponding authors.

heading: A Related Work
text: A.1 Text Simplification
Text simplification contains sentence simplification, document-level simplification, and lexical simplification. Sentence simplification is rewriting a complex sentence into a more straightforward and semantically identical sentence (AlvaManchego et al., 2020). Document-level simplification is rewriting an original complex article into a simple article (Sun et al., 2021). Information not relevant to the central meaning can be removed to improve readability. Lexical simplification is to replace complex words in a sentence with more straightforward but identical meaning words (Paetzold and Specia, 2017b). It is usually framed as
a pipeline consisting of generating multiple candidate words and developing rules to select the most appropriate word from candidate words.
A.2 Lexical Complexity Prediction
The lexical complexity prediction (LCP) task is to assign a value from a continuous scale to represent the complexity of a word (Shardlow et al., 2020). Given a text and a text span in this text, the model will predict the complexity of this text span. Many studies have been devoted to improving the accuracy of model predictions (Gooding and Kochmar, 2019; Paetzold, 2021). On the latest LCP 2021 task (Shardlow et al., 2021), the DeepBlueAI model (Pan et al., 2021) achieves state-of-the-art results.
A.3 Adapting Pre-trained models
Pre-trained models have been widely used in natural language processing in recent years. However, Gururangan et al. (2020) observe the gap between the language model pre-training domain and the data distribution of the downstream task. Since then, researchers have focused on how to adapt pretrained models to downstream tasks. They have designed new methods for different tasks. Downstream tasks like machine translation (Hu et al., 2022), sentiment analysis (Gu et al., 2020), and many understanding tasks (Yu et al., 2022) can benefit from the adapted pre-trained models.

heading: B Training Parameters
text: We use the Huggingface transformers (Wolf et al., 2020) to conduct sentence and lexical simplification experiments. For document-level simplification, we follow Sun et al. (2021) and use Fairseq (Ott et al., 2019) to conduct the experiments. We choose the model that performs best on the validation set for testing. The specific parameter settings for each task are shown in Tables 10, 11, and 12. A detailed description of the dataset sizes is given in Table 13.
Here are the sources of the automatic evaluation methods we use: SARI (https://github.com/mounicam/BiSECT/ tree/main/metrics), BERTScore (https: //github.com/Tiiiger/bert_score), and D-SARI (https://github.com/RLSNLP/ Document-level-text-simplification).

heading: C Ablation Study
text: We conduct ablation experiments to explore the different contributions of replacing complex words in ordinary texts (BART-S) and masking simple words in simple texts (BART-T). We continue pretraining and fine-tuning on the Newsela dataset.
From Table 14, both methods in our proposed strategy allow the model to acquire the ability to generate simple words. Their contributions are roughly the same, but the improvement to the SARI value is less than combining them.

heading: D Example Outputs
text: ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Limitations section.

heading: 3 A2. Did you discuss any potential risks of your work?
text: Ethics Statement section.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Abstract section.
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? 2
3 B1. Did you cite the creators of artifacts you used? 2

heading: 3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
text: Ethics Statement section.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Not applicable. Left blank.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? Appendix B
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Appendix B
C 3 Did you run computational experiments? 4

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Appendix B
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Appendix B

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? 5

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? 3
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? 5
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Not applicable. Left blank.

heading: 3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students)
text: and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Ethics Statement section.

heading: 3 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to
text: crowdworkers explain how the data would be used? Ethics Statement section.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Not applicable. Left blank.

heading: Future Work
text: We expect that there will be methods with higher identification accuracy and higher inference speed in the future. Due to page limitations, we have placed the related work in Appendix A and the ablation experiments in Appendix C. Due to time constraints, we do not perform a human evaluation of the output of LLMs. We hope to conduct a more comprehensive evaluation of the performance of LLMs in the future