BLESS: Benchmarking Large Language Models on Sentence Simplification
Authors: Tannon Kew
Alison Chi
Laura Vásquez-Rodríguez
Sweta Agrawal
Dennis Aumiller
Fernando Alva-Manchego
Matthew Shardlow
Venue: EMNLP 2023 Main (2023)
Keywords: text simplification
sentence simplification
large language models
evaluation
in-context learning
TL;DR: We evaluate 44 LLMs on the task of sentence simplification in a few-shot setting across 3 different domain test sets.

Abstract: We present BLESS, a comprehensive performance benchmark of the most recent state-of-the-art Large Language Models (LLMs) on the task of text simplification (TS). We examine how well off-the-shelf LLMs can solve this challenging task, assessing a total of 44 models, differing in size, architecture, pre-training methods, and accessibility, on three test sets from different domains (Wikipedia, news, and medical) under a few-shot setting. Our analysis considers a suite of automatic metrics, as well as a large-scale quantitative investigation into the types of common edit operations performed by the different models. Furthermore, we perform a manual qualitative analysis on a subset of model outputs to better gauge the quality of the generated simplifications. Our evaluation indicates that the best LLMs, despite not being trained on TS perform comparably with state-of-the-art TS baselines. Additionally, we find that certain LLMs demonstrate a greater range and diversity of edit operations. Our performance benchmark will be available as a resource for the development of future TS methods and evaluation metrics.

BLESS: Benchmarking Large Language Models on Sentence Simplification
Tannon Kewl-i, Alison Chi Laura Vásquez-RRodrigezz22
Sweta Agrawal,, Dennis Aumiller Fernando Alva-Manchego', Matthew Shardlow UUniversity of Zurich, Switzerland National Tsing Hua University, Taiwan Idiaap Research Institute, Switzerland UUniversity of Manchester, UK University of Maryland, US CChheee, US "Cardiff University, UK Manchester Metropolitan University, UK
kewêcl. uzh. ch, achi achi@gapp.co ntwp. edu. tw, laura. www.uvaiidiaappcoom com
sweagr sweagraw@umd.r edu, al wwwaaacheooo@earrreceeo ac.uuk, m. shardlowemmu. ac uk
Abstract
nia et al., 2023), and ethics (Zhuo et al., 2023), as well as for specific NLP tasks (Li et al., 2022; Wang et al., 2023; Liu et al., 2023).
We present BLESS, a comprehensive perfor- mance benchmark of the most recent state-of- the-art large language models (LLMs) on the task of text simplification (TS). We examine how well off-the-shelf L.LMs can solve this challenging task, assessing total of 44 mod- els, differing in size, architecture, pre-training methods, and accessibility, on three test sets from different domains (Wikipedia, news, and medical) under a few-shot setting. Our anal- ysis considers a suite of automatic metrics as well a large-scale quantitative investigation into the types of common edit operations per- formed by the different models. Furthermore, we perform manual qualitative analysis on a subset model outputs to better gauge the quality of the generated simplifications. Our evaluation indicates that the best LLMss, despite not being trained on TS, perform comparably with Late-of-the-ate TS baselines. Addition- ally. we find that certain LLMs demonstrate greater range and diversity of edit operations. Our performance benchmark will be available as a resource for the development future TS methods and evaluation metrics.'
However, it remains unclear how well current LLMs can perform on the challenging task of text simplification (TS). In this paper, we focus tence simplification in English, which typically involves rephrasing part or all of sentence into language which more accessible and easier understand. While recent work has focused on evaluating TS abilities of select models, such as b-Turb (Feng et al., 2023) and mT5 (Ryan et al., 2023), there is currently no large-scale and detailed analysis of the simplification capabilities of different LLMs.
In this study, we expand both the breadth and depth of the knowledge base on TS with LLMs, evaluating a wider variety of models on three dif- ferent TS datasets: ASSET Aalaa-Manchego al., 2020a), NEWSELA (Jiang et al., 2020) and MED- EASI (Basu et al., 2023). We select these datasets to cover a variety of domains (Wikipedia, news, and medical) and a diverse set of TS operations (e.g. paraphrasing, splitting, and claboration).
1 Introduction
Specifically, we use in-context learning (ICL) and assess LLMs in few-shot setting, experi- menting with three different prompts. We select 44 widely used generative models (both open and closed-weight) and evaluate their abilities from three distinct angles. First, we rely on automatic evaluation metrics commonly used in the TS liter- ature. Second, we quantify and compare the edit operations performed by the LLMs during simplifi- cation. Finally, we perform targeted qualitative analysis to validate our findings and to better under- stand the quality of the generated simplifications. Our findings reveal that flosed-weight models pro- vide significant gains over open geinhweight alternatives under a few-shot setting, establishing them as a
Large pre-trained language models (LLMs) have demonstrated strong performance on wide range of NLP tasks without the need for task-specific fine-tuning, leading to a prevailing conventional wisdom that LLMs can solve any task. This has motivated the development of benchmarks to better understand the abilities of LLMs in specific do- mains such healthcare (Sallam, 2023), finance (Dowling and Lucey, 2023), education (Baidoo- Anu and Owusu Ansah, 2023), engineering (Soba-
'We make code and the generated system outputs available https:// //gi thub. www...c.ooo.....c cow/ htn/ZurrcchLL/BBEE.
'These authors contributed equally.
"Work done as a PhD student at the University of Manch-
ester, United Kingdom.

---

strong baseline for future work on TS. We summa- rize our contributions as follows:
Dataset
Domain
Size
Words
R TER
C
S
1. BLESS (Benchmarking Large language modEls on Sentence Simplification), per- formance evaluation benchmark of 44 LLMs in few-shot setting (Section 3).
ASSET
Wikipedia
359
22.57
18.87
10
16.79
MED-EASI
Medical
300
26.48
27.42
25.03
NEWSELA
News
256
26.44
24.82
4 23.17
Table I: Dataset Statistics. C: Complex; S: Simple; R: References. TER refers to Translation Error Rate, a measurement of the average edit distance between the source and reference texts (see https: https:///ww..c cs. umd. edu//snover/terccmm)
2. An evaluation that includes both widely used automatic metrics and an analysis of the TS edit operations performed by the models (Sec- tion 4).
3. A qualitative analysis of the results, with man- ual annotation of simplification operations and an examination of the relationships between selected evaluation metrics (Section 5).
clear how a wider spectrum of models, differing in architecture and training strategy, perform on differ- ent domains and in response to different prompts. We aim to fill this gap and study the simplification abilities of 44 LLMs in order to highlight potential
2 Related Work
Text Simplification Benchmarks Most simpli- fication work treats the task as a monolingual machine translation problem, training models on datasets containing complex-simple sentence pairs (Zhu et al., 2010). Alva-Manchege et al. (2020b) performed a standardized evaluation of general data-driven simplification systems, using Wikipedia-based datasets and NEWSELA. At the document level, Alva-Manchego et al. (2019h) con- ducted a systematic analysis of simplification op- erations to demonstrate the limitations and disrup- tions that occur when multiple sentences are in- volved. Benchmarks have also been established for more specific kinds of simplification: for example, both non-neural (Paetzold and Specia, 2016) and neural (Stajner et al., 2022; Saggion al., 2022) upproaches to lexical simplification, which aims to replace complex words with simpler alternatives.
weaknesses and determine areas for further devel- opment. To the best of our knowledge, we are the first to focus establishing the performance of recent LLMs on the task of TS.
3 BLESS: Benchmarking Large Language Models on Sentence Simplification
3.1 Datasets
Our assessment establishes the performance of cur- rent LLMs on TS according to three datasets, cov- ering different domains and styles. Table summa- rizes these datasets.
ASSET (Alva-Manchego et al., 2020a) com- prises 2,359 sentences from English Wikipedia paired with 10 simplified references. We use the official test split (359 sentences) for evaluation. These references were created by crowdworkers who were instructed to use edit operations such as replacement, splitting, and deletion.
LLM-based Simplification LLMs such as the model behind early versions of ChatGPT2, are often used out-of-the-box with- out any further training for a given domain or task. Some previous works have investigated sim- plification capabilities of select LLMs in order benchmark performance against dedicated ap- proaches (Aumiller and Gertz, 2022: Vásquez- Rodríguez et al., 2022; Ryan et al., 2023; Sun et al., 2023; Chi et al., 2023). Meanwhile, Feng et al. (2023) explored the TS abilities of the two strong- performing OpenAl models, GPT-3. 5-5Trbo and Davinci- However, despite these efforts, only have results from very limited number of LLMs and evaluation metries. Thus, remains un-
MED-EAST (Basu et al., 2023) is simplifica- tion dataset for short medical texts containing 1,979 complex (expert) simple (layman) pairs. Each text contains one or more sentences. In this dataset, simplified texts are composed using four types of operations: elaboration, replacement, deletion, and insertion. We use the released test split (300 in- stances) for our evaluation. Unlike the other two datasets, simplifications in MED-E. EASI are slightly longer than the complex source texts, due to ex- planation and decomposition of complex medical terms.
NEWSELA (Xu et al., 2015) contains 1,130 long- form news articles that have been professionally
hattps://chat.oom com

---

rewritten according to four different graded read- ability levels. For our benchmarking experiments, we opt for the Newsela-Manual test set (Jiang et al., 2020). We extract all aligned and partially aligned sentence pairs between a complex source sentence (level 0) and the four simplified article versions (levels 1-4), keeping only those sentences for which we have reference for all four simplification lev- els.3 This results in 256 test examples. Using this small subset of NEWSELA data ensures that sentence-level alignments are of high quality and capture important edit operations such as splitting.
and LLaMA models (Touvron al., 2023), and the BLOOM family (Scao et al., 2022). For closed-weight models, we focus on those developed by OpenAl. Details on each model family are provided in Ap- pendix A.
3.3 Prompts
To simplify sentences with LLMs without addi- tional fine-tuning, we use in-context learning (ICL), ICL is prompting technique that utilizes a small number of input-output examples demonstrate task (Brown al., 2020). Previous work on related tasks has demonstrated that LLMs are sensitive to which input prompts and few-shot examples are used (Zhang et al., 2022b; Lu et al., 2022; Agrawal et al., 2023). To account for this, we construct three stylistically distinct prompts that consist of task instruction and N few-shot examples (sec Figure 1). For all generation settings, we set N=3 and randomly sample complex-soure pairs from the corresponding validation sets. We leave de- tailed investigation of optimal in-context learning strategies for TS to future work.
3.2 LLM Types
We investigate total of 44 LLMs with different sizes, architectures, and training objectives. The models we consider range from 60 million to 176 billion parameters and are all based on the trans- former architecture (Vaswani et al., 2017), consist- ing of either an encoder-decoder or standalone decoder. Furthermore, all have undergone self- supervised pre-training stage. Nine of these mod- els leverage instruction-tuning, which fine-tunes a pre-trained base model on labeled instruction- response pairs from a diverse set of tasks. Finally. just three of these models have received additional training through reinforcement learning with hu- man feedback (RLHF) to better align the model's responses with human preferences (Stiennon et al., 2020; Ouyang al., 2022). Evaluating wide vari- ety of currently available models should serve broad baseline and give sufficient information on which models perform best in which domains as well as where key challenges remain.
3.4 Inference Settings
For open-weight models, we run inference on local GPUs using the Transformers library (Wolf et al., 2020). We load the models with 8-bit quantiza- tion (Dettmers al., 2022), which allows us run inference efficiently on as few as 5 A100 80GB GPUs. For closed weight models, we use the APIs provided by OpenAl. As generation hyperparam- eters, we use Nucleus Sampling (Holtzman et al., 2020) with a probability threshold of 0.9, a temper- ature of 1.0, and a maximum output length of 100 tokens. To account for the stochastic generation settings, we perform each inference run with 3 dif- ferent random seeds and aggregate the results for each metric.
We broadly distinguish between open- and closed-weigs weight models. The former pertains to mod- els for which the trained weights are accessible and thus allow for self- self-hosting. Typically, these models are considered to be "open-source." However, we note that this obfuscates specific licensing agree- ments attached to some models and whether or not the training data and code are also made available. In comparison, -eight models refer to those whose weights are kept private and can be queried only through APIs. Our open-weight models in- clude variants of the T5 family (Raffel et al., 2020), GPT-style models (Radford et al., 2019; Wang and Komatsuzaki, 2021). OPT (Zhang et al., 2022c)
3.5 Baselines
We use the MUSS (Martin et al., 2022) model as our main baseline since it has been shown to achieve state-of-the-art performance. MUSS fine-tunes BART-large (Lewis et al., 2020) model with ACCESS control tokens (Martin et al., 2020) extracted from labeled TS datasets and/or mined paraphrases train both su- pervised (MUSS-wiki-mined and unsupervised (MUSS-mined) TS systems. We use the suggested hyperparameters from the original paper to set the control tokens for simplification generation.
'These articles are simplified as whole to match the desired school grade; therefore, there is no guarantee that there will be an exact match for all the sentences in the text across all grade levels.

---

Rewrite the complex sentence with simple sentence(s). Keep the meaning same, but make simpler.
N few few-shot shot examples
3.6 Automatic Metries
To assess how well LLMs can perform TS, we evaluate all the model outputs using suite of au- tomatic metrics.* 4 We measure simplicity using SARI (Xu et al., 2016), meaning preservation using BERTScore (Zhang al., 2020), and readability using FKGL (Kincaid et al., 1975). These metrics are computed using the EASSE package (Alva- Manchego et al., 2019a). Additionally, we report LENS (Maddela et al., 2023), a recently proposed learned metric, which considers both the seman- tic similarity and the degree of simplification per- formed by the system with respect to the source sentence and references." Where possible, we also establish the performance of "gold" simplifications by evaluating available reference sentences using a 'leave-oncoout strategy. That is, in cases where multiple references are available, we select one at random and evaluate it against the remaining references.
Complex:
(complex example) Simple: (simple example]
Complex: (input)
Prompt O uses a basic instruction adapted from (Feng et al., 2023) followed by list of N few-shot examples before the input sentence be simplified.
Rewrite the complex sentence with simple sentencc(s). Keep the meaning same, but make simpler.
N few-shot examples
The sentence '(complex example]* can be simplified as follows: '(simple example)'
The sentence '(input)' can be simpli- fied as follows:
(b) Prompt uses the same basic task instruction as prompt 0, but presents few-shot examples an inline, continuous text format.
Automatic Evaluation Results
In this section, we present the results of our auto- matic evaluation of simplification outputs and sum- marize our main findings. First, we perform an ex- haustive assessment using automatic metrics (Sec- tion 3.6). For brevity, we report the results of the best- persforrfoning LLMs with SARI and BERTScore in Table 2 and provide the complete results for all 44 models and metrics in Appendix B. Then, we compute edit distance statistics to quantify the simplification operations performed by each of the LLMs (Section 4.... We begin by assessing the impact of the different prompt formats.
Please rewrite the following complex sentence in order to make it easier understand by non-native speakers of English. You can do so by replac- ing complex words with simpler syn- onyms (i.e. paraphrasing), deleting unimportant information (i.c. compres- sion), and/or splitting a long complex sentence into several simpler ones. The final simplified sentence needs to be grammatical, fluent, and retain the main ideas of its original counterpart without altering its meaning.
N few shot examples
Structured prompting improves performance. Figure 2 reveals that prompts 0 and both offer slight advantage over prompt 1, especially in re- gard to meaning preservation. This confirms that providing a structured template for few-shot exam- ples instead of embedding them within sentences is the most beneficial. Hence, we focus on prompt 2 for all our analysis, as it provides the most detailed description of the task and has also been used in prior work (Maddela et al., 2023).
Complex: (complex example)
Simple: (simple example)
Complex: (input)
(c) Prompt 2 repurposes the instructions from Alna-Mo et al., 2020a) that were provided to crowdworkers in the cre- ation of the ASSET dataset. Similarly to prompt 0, few-shot examples are presented in structured format.
Figure 1: Prompts used for LLM text simplification. The blue boxes contain the task instructions. Orange boxes show how the few-shot examples are presented to the model and yellow boxes contain the prefix for the model to continue.
Training method matters more than size. Ta- ble 2 presents the performance according to SARI
"See Appendix B.1 for details each evaluation metric. /ht thub. www/fereaaaeeeeeeeee hen//fraaveee Feralvam//e easse
"We compute LENS using its original implementation: https://githur cwm/Yao DoNS. Lon/YaoDDDu/LESS.

---

SARI
BERTScore
po
Prompl
p1
30
35
40
15
20
60
100
Figure 2: Impact of prompt selection SARI and BERTScore for all models on ASSET. Prompts 0 and achieve improved meaning preservation over prompt 1.
ASSET
MED-EASI
NEWSELA
SARI 1
BERT
SARI
BERT
SARI
BERT
Gold References
45.27
78.89
100
100
60.11
87.66
Baselines
MUSS-mined
42.29
79.86
35.15
42.55
38.40
72.14
MUSS-wiki-mined
44.90
77.71
35.12
43.07
41.24
74.1
Ada-001*
33,97
81.76
36.52
33.95
34.42
70.33
Babbage-001*
38.44
82.46
36.6
37.95
36.41
62.99
Curie-001*
39.87
82.75
38.22
39.31
37.53
69.17
Davinci-002+
42.84
85.91
36.34
43.67
40.25
73.62
Davinci-000 +
46.60
79.66
39.81
40.83
37.76
61.56
GPT-3.5- TuTbbo*
47.69
79.39
40.14
40.67
37.29
60.19
BLOOM
39.72
76.63
37.72
11.95
37.48
61.17
BLOOMZ
37.63
82.06
36.6
12.9
37.06
69.55
OPT-1.3b
33.01
75.57
34
3.82
34.76
50.78
LLMs
OPT-30b
38.04
77.22
35.08
9.96
37.58
61.79
OPT-IML-MAX-1.2 3b
36.00
79.73
37.01
11.85
37.08
62.68
OPT-IML-MAX-330 30b
42.03
79.39
35.8
11.73
39.59
66.39
F1an-T5-small
38.57
77.26
36.65
38.6
37.72
68.15
Flan-T5-bass
41.40
79.7
36.79
40.63
38.67
68.09
F1an-T5-large
42.17
80.44
35.71
41.31
39.08
70.27
Flan-T5-x1
41.07
85.06
33.21
44.12
37.51
75.5
Flan-T5-xx1
41.75
84.13
34.27
43.43
39.42
73.05
Flan-UL2
42.83
84.34
35.31
42.8
40.27
73.23
Table For brevity, we report automatic metrics for simplification (SARI) and meaning preservation (BERTScore) for select models using Prompt 2. 'x' indicates closed-weights. The full list of results is available in Tables 6, and 8 in the Appendix.
and BERTScore for the top-performing LLMs. Scaling LLMs has revealed strong benefits in few-shot settings (Brown al., 2020; Chowdh- ery et al., 2022); however, our evaluation, we observe numerous exceptions to this rule. For example, Flan-T5-large (770 million parame- ters) consistently attains higher SART scores on ASSET than Flan-T5-x1 (3 billion parameters) and Flan-T5-xx1 (11 billion parameters). Mean- while, observe that training strategies such as
instruction-tuning and RLHF help deliver greater improvements, especially for meaning preserva- tion, as measured by BERTScore. This agrees with previous findings that demonstrate the benefits instruction-based adaption strategies for improved generalization abilities (Schick and Schütze, 2021; Zhang et al., 2022a; Chung et al., 2022).
ASSET On Wikipedia-style data, OpenAl's Davinci-000 and GPT-3. 5-Turbo outperform all other tested L.LMs by considerable margin accord- ing to SARL Strikingly, these models also outper- form the ground truth references, which are closely
"We include wider comparison of selected L.LMs on ASSET in Figure 7 in the Appendix.

---

approximated by the previous state-of-the-art MUSS models. This notable since MUSS-wiki-mined was trained on the in-domain TS dataset of Wiki- Large (Zhang and Lapata, 2017). Meanwhile, for open-weight contenders, we can see in Table 2 that only a small number of models are competitive, namely -MT-IM--Max-305 Flan-T5-large. and Flan-UL2, which scores the best balance between simplicity and meaning preservation according to automatic metries.
ASSET
Cold Rafere noes
NUSS wil mined
aot
OpenAi Batbwoo-001
OpenAl Curle 001
OpanAl Devind 002
OpenAi Devinc-003
OponAi 3.5
BLOOM
BLOOMZ
1.3
0F1-306
OPT-NI--000-1.00
MED-EASI For medical-related texts, we ob- serve that the majority of the models consistently fail to preserve meaning (our qualitative analysis in Section 5 confirms this, see Table 3). The drop in meaning preservation can likely be explained by the fact that models are known to produce inade- quate generations in out-of-domain settings (Müller al., 2020; Singhal al., 2023). The models that do strike a reasonable balance with both SART and BERTScore are again OpenAl's more powerful offerings and the Flan models. Notably, we also observe that the two MUSS models are able to per- form competitively with the Flan models despite being multiple orders of magnitude smaller.
OPTINL-MMXX330
Han-
FaniTs-
Han-SSal
Fln-UL2
0.0
0.2
0.4
0.6
0.8
1.0
KEIP
INse
DELETE
REPLACE
Figure 3: Distribution of token-level edit operations
produced by the pess-performing LLMs.
references, the keep operation is by far the most prominent in this dataset. Notably, Davinci-000 perform the most diverse set of operations, with fewer additions and more re- placements than other models. Insertions are typ- ically less frequent, suggesting that the majority of the models avoid adding new and potentially irrelevant content. We observe that most LLMs are within the range of the gold references in terms of the amount of information they delete when simpli- fying.
and
NEWSELA Evaluating LLMs on professionally written simplifications from NEWSELA reveals that even the best LLMs are not able to match hu- man performance. This is observable through the clear margins of around 20 SARI points and 14 BERTScore points between the best perform- ers and the gold simplifications. On this dataset, MUSS-wiki-mines remains a strong baseline, out- performing all LLMs both metrics, while Davinci-002, Flan-UL2, and Flan-T5-xx/ show the strongest performances among the LLMs.
Qualitative Analysis
Automatic metrics are known to have blind spots and are not always entirely reliable (Alva- Manchego et al., 2021; He et al., 2022). To com- pensate for this, we perform a qualitative analysis on total of 300 system outputs.
4.1 Analysis of Edit Operations
To identify the main token-level edit operations per- formed by LLMs, we use an adaptation of the Wag- Fischrr algorithm (Wagner and Fischer, 1974), following previous work by Vásquez-Rodrrguez et al. (2021a). Specifically, we calculate the por- tion of insertion, replacement, deletion, and keep operations between the input source sentence and each of the system outputs for each dataset.
First, we check whether not each output is valid simplification and highlight common failure cases such as inappropriate changes to the meaning of the original text, ungrammatical outputs, and the occurrence of hallucinations. Then, we an- notate occurrences of common simplification edit operations such as lexical simplification, deletion, sentence splitting, reordering, and paraphrasing. For our annotations, we select model outputs from the top five systems ranked according to per-
Figure 3 shows the distribution of token-level edit operations for the best-performing LLMs on ASSET (for a more comprehensive view across all datasets and models, see Figure in the Ap- pendix). Most models perform all four operations to differing degrees; however, similar to the gold
8 Alll annotations were completed by one of the authors and validated separately by another.

---

Model outputs
%St
%MPt
%L+
%P+
%D+
%Sp+
%R+
%Hl
All
61.67
67.33
30.33
28.33
35.0
4.33
4.67
12.33
Top 5 SARI
72.0
48.0
34.66
37.33
6.67
Top 5 BERT
62.67
84.0
17.33
29.33
34.67
5.33
10.67
Top 5 FKGL
34.67
40.0
14,66
17.33
26.67
0.0
0.0
36.0
Top 5 LENS
77.33
77.33
41,33
32.0
41.33
5.33
1,33
2.67
Open-Weight
58.58
64.55
29.47
22.76
36.94
3.36
3.73
13.81
Closed Weight
Closed-Weight
87.50
90.63
37.50
75.0
18.75
12.50
12,50
On ASSET
77.0
82,0
31.0
54,0
33.0
8.0
4.0
10.0
On NEWSELA
54.0
70.0
34.0
9.0
38.0
5.0
7.0
17.0
On MED-EASI
54.0
50.0
26.0
22.0
34.0
0.0
3.0
10.0
Table 3: Results of our manual analysis. The annotation schema includes the following annotation features: S1: accepted simplificatinn, MP1: meaning preserved, L+: lexical simplification, P+: paraphrasing, R+: reordering (no changes), D+: deletion, Sp+: sentence splitting, H.J: hallucination.
formance on the individual evaluation metries of SARI, BERTScore, FKGL, and LENS. In each ranking set, randomly select five complex- simple pairs from all generation settings. To evalu- ate unique set models for greater diversity, if a system is repeated in the ranking (e.g. two different prompt types from the same model appear in the top five), we choose the next best system for analy- sis. An example of our annotated outputs is shown in Table 9 in the Appendix. Table 3 shows results from this analysis, which deseribe according to different criteria below.
By Open-Status Open-weight models most fre- quently the operations of lexical simplification, paraphrasing, and deletion, while structural opera- tions such as sentence splitting and reordering are often neglected. Many only achieve high meaning preservation by directly copying the input sentence. However, the closed-weight models investigated here behave very differently: they produce close to 10% more splitting, lexical simplification, and re-ordering than open-weight ones, while simulta- neously performing fewer deletions. This leads greater degree of paraphrasing.
By Automatic Metric Overall, we find that sim- plicity and meaning preservation fairly bal- anced. However, there is clear trade-off between these two axes when we consider the top 5 models according SARI and BERTScore. This agrees with earlier findings from Schwarzer and Kauchak (2018). Along with a higher degree of simplicity, the top 5 SART models exhibit more diverse edit op- erations than those ranked highly by BERTScoore. LENS, however, does not trade off simplicity and meaning preservation and even achieves higher simplicity score than SARI along with its in- creased level of deletion. This result in line with the previous finding that LENS achieves stronger correlations with human judgments compared to existing TS metries (Maddela et al., 2023). The top 5 models ranked by FKGL, on the other hand, produce outputs with low simplicity and meaning preservation and especially high amount of hal- lucinations. This result supports the previous find- ing that FKGL can be easily gamed by degenera- tions (Tanprasert and Kauchak, 2021) and is there- fore an unsuitable metrie for evaluating the outputs automatic TS systems.
By Domain tween different domains, we observe that all LLMs do significantly better on general encyclopedic texts in ASSET in terms of both simplicity and meaning preservation, while also exhibiting a di- verse set of edit operations. Although outputs from NEWSELA contain more hallucinations, meaning preservation is still fairly high. Outputs from MED- EASI, on the other hand, have the lowest meaning preservation by far and the least diverse set of edit operations. We find that MED-EASI outputs, along with others that do not preserve meaning, often con- tain repetitions, hallucinations, and in some cases even copy the input prompt, demonstrating ten- dency to disregard the instruction and thus fail to complete the task. These failure modes most frequently observed from the smaller T5 models, but are also exhibited by models such as LLaMA when evaluated on MED-EASI.
When comparing performance be-
Discussion
We discuss our results around the following aspects: the access level of the simplification models (open- vs. closed-wes weight),, the training strategies (general

---

ASSET
Med-EASI
Newsela
Levenshtein Similarity
0.8
0.85
X
0.8
Open Status
Open
X
0.7
Closed # Parameters
0.80
X
0.8
0.7
u greelln
10-100B
0.75
1-108
0.5
10c
under
0.70
80
85
20
40
50
60
70
BERTScore
BERTScore
BERTScore
Figure 4: BERTScore, computed between the system output and reference sentence(s), correlates strongly with Levenshtein similarity, computed between the source sentence and system outputs. This indicates that BERTScore tends reward minimally edited sentences. Levenshtein similarity is computed with the EASSE package (Alva- Manchego et al., 2019a).
pre-training vs. general fine-tuning strategies), and the utility of automatic metrics.
BLOOMZ. Therefore, we hypothesize that this drop in performance may be influenced by the multi- lingual instruction tuning setup that is unique BLOOMZ.
Access Level Among the OpenAl models, we observe that all models perform particularly well on meaning preservation according to BERTScore but exhibit considerable differences in their abil- ity to simplify, as indicated by SARI on 'weaker' models such Ada-001. Among the evaluated open-weight models, we observe that the Flan mod- els (T5 and UL2) typically perform competitively, punching well above their weight in terms of pa- rameter counts with much larger decoder-only mod- els. This is a promising finding for the category of wein-weight models, and we hope that this encour- ages future work to continue investigating different methods regardless of the model size.
Utility of Automatic Metries Overall, we find SARI and BERTScore to be useful automatic eval- uation metrics for inspecting the trade-off between simplicity and meaning preservation (see Figure 6 in the Appendix). In general, closed- weight models often strike more optimal balance. This is also supported by our qualitative analysis, which con- firmed that these models rely less on deletion, an oft-overused operation (Devaraj et al., 2022), and more on other edits (e.g. paraphrasing or splitting). Furthermore, our qualitative analysis shows that outputs with higher BERTScores tend min- imally simplified, often copying the entire input text. We validate this by studying the relation- ship between BERTScore (computed between the system output and the reference sentence(s)) and Levenshtein similarity (computed between the sys- tem output and the original input sentence). Fig- ure 4 reveals a strong positive correlation across all datasets, indicating that BERTScore tends re- ward minimally simplified responses. For some of the closed-models, which tend to perform a greater degree of paraphrasing, this leads to lower BERTScores, while models that perform more copying are rewarded. Overall, the results from our qualitative study generally showed agreement with those from our automatic evaluation metrics, particularly SARI, BERTScore, and LENS. It also enabled us to pinpoint specific operations, such as re-ordering, and identify issues, notably hallucina- tions, in system outputs.
Training Strategies Within model families, when comparing base models to their instruc- tion fine-tuned counterparts, we observe that iestruction-tuingg typically leads to better perfor- mance in our few-shot ICL setting for TS. We find this to be particularly encouraging since TS is one task often hindered by the lack of high-quality la- beled training data (Stajner, 2021).
Nevertheless, improvement not always guar- anteed, as seen when comparing BLOOM vs BLOOMZ. In this case, instruction fine-tuning leads to better- meaning preservation but reduction in the degree of simplification, indicating that the instruction- tuning method used to derive the multilingual BLOOMZ may be less suitable for English TS. This stands in stark contrast to the Flan instruction tuning method, which delivers considerable gains in both SARI and BERTScore despite sharing the same underlying instruction- tuniionntttnnnnnnni dataset as

---

7 Conclusion
achieve new state-of-the-art performance on the task of text simplification. However, very little is known about their training data, alignment strate- gies, and implementation behind paywalled APIs. Therefore, the comparison to open-source models, which contain no explicit training on the task and an extremely bare-bones implementation is poten- tially unfair.
In this paper, we provided a comprehensive as- sessment of how well out-of-the-box LLMs per- form on the task of TS with few-shot in-context learning. We found that the best LLMs outperform otate-of-the-art supervised TS baselines while also producing more diverse set of simplification op- erations. We also established that weight models perform better than open-weight ones and that general instruction-tuning often improves a model's abilities on TS. Furthermore, we empiri- cally validated the trade-off between simplicity and meaning preservation through automatic evaluation and manual analysis. Our analyses of multiple few-shot prompting strategies revealed that more structured prompting format produces better results than presenting source-target examples in continu- ous text.
Ethics Statement
This work is conducted in full awareness of and in line with the ACL Ethics Policy. Particularly, this work contributes to the transparency and fairness of evaluation methodologies in line with Sections 1.1, 1.2, 2.7, and 2.9 of the code, which innately leads avoiding seen and unseen harms (Section 1.2, 1.4). We contribute to improving expertise in the domain of text simplification (Section 2.6). All models, datasets, and compute resources are used with permission and with concern to the appropri- ate access rights and licenses (Section 2.8). Our work contributes to the professional development of the research team (Section 3.5) and more widely benefits the research community and wider society (Section 3.1) by augmenting the understanding of the capacity of LLMs on the specific task of TS.
Our performance benchmark, BLESS, provides a strong foundation for future work. For example, it remains an open question as to which expressions and instructions are optimal for prompting LLMs to simplify texts. Furthermore, this work exclusively focused few-shot in-context learning. Future work could explore the capabilities of these systems zero-shot, fine-tuned, or retrieval-based settings.
Limitations
Acknowledgements
In this section, discuss few limitations of our work. First, we only considered English TS datasets, and it still remains to be seen how these TS abilities transfer to languages other than En- glish. Additionally, we selected only a handful of output samples for manual analysis for the three test datasets considered, and all annotations were performed by one of the authors and subsequently validated by another author independently. It will be necessary to perform this at a larger scale to more accurately characterize the capabilities of each model for each domain and prompt. We fur- ther acknowledge the limits of the evaluation set itself. While we purposefully chose the test splits to cover a variety of domains, test splits for all three corpora amount to 915 samples, which could potentially limit the statistical power of results ob- tained from the assessment. Additionally, two out of the three test sets contain only sentences as input, while the third contains short multi-sentence texts, so this assessment mostly applies to the subtask of sentence simplification. Finally, our findings confirm that proprietary, closed-source models can
We would like thank Sian Gooding for her ini- tiative in motivating this project, as well as Hoang Nguyen Hung Van, Jan Trienes, and everyone the text simplification research community who joined our discussions during this journey. Thank you also to the anonymous reviewers for providing valuable feedback. This work was facilitated by the infrastructure services provided by S3IT, the Service and Support for Science IT team the University of Zurich. Laura Risquez-Rodríguee work was funded by the Kilburn Scholarship from the University of Manchester.
References
Sweta Agrawal, Chunting Zhou, Mike Lewis, Luke Zettlemoyer, and Marjan Ghazvininejad. 2023. In- context examples selection for machine translation. In Findings of the Association for Computational Linguistics: ACL 2023, pages 8857-8873, Toronto, Canada. Association for Computational Linguistics.
Fernando Alva-Manchego, Louis Martin, Antoine Bor- des. Carolina Scarton, Benoît Sagot, and Lucia Spe- cia. 2020a. ASSET: A dataset for tuning and evalua-

---

tion of sentence simplification models with multiple rewriting transformations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. pages 4668-4679, Online, Association for Computational Linguistics.
Michael Pieler. Usvsn Sai Prashanth. Shivanshu Puro- hit, Laria Reynolds, Jonathan Tow, Ben Wang, and Samuel Weinbach, 2022. GPT-NeoX-20B: An open- source autoregressive language model. In Proceed- ings of BigScience Episode #5 Workshop on Chal- lenges & Perspectives in Creating Large Lunguage Models, pages 95-136, virtual+Dublin. Association for Computational Linguistics.
Fernando Alva-Manchege Louis Martin, Carolina Scar- ton, and Lucia Specia. 2019a. EASSE: Easier auto- matic sentence simplification evaluation. In Proceed- ings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th Inter- national Joint Conference on Natural Language Pro- cessing (EMNLP-JJCNLP)): System Demonstrations, pages 49-54, Hong Kong, China. Association for Computational Linguistics.
Tom Brown, Bemjamin Mann, Nick Ryder, Melanic Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 1777-191...
Fernando Alva-Manchego, Carolina Scarton, and Lucia Specia. 2019b. Cross-sentence transformations in text simplification. In Proceedings of the 2019 Work- shop on Widening NLP, pages 181-184, Florence, Italy, Association for Computational Linguistics.
Alison Chi, Li-Kuang Chen, Yi-Chen Chang, Shu-Hui Lee, and Jason S. Chang. 2023. Learning to Para- phrase Sentences to Different Complexity Levels. arXiv preprint arXivv2308..022 02226.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham. Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311
Fernando Alva-Manchego, Carolina Scarton, and Lucia Specia. 2020b. Data-driven sentence simplitication: Survey and henchmark. Computational Linguistics, 46(1):135-11) 187.
Fernando Alva-Manchego, Carolina Scarton, and Lucia Specia. 2021. The (un)suitabiliiy of automatic evalu- ation metrics for text simplification. Computational Linguistics, 47(4):861-889.
Hyung Won Chung, Le Hou, Shayne Longpre. Bar- ret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahmma, et al. 2022. Scaling instruction-finttue language models. arXiv preprint arXiv: 2210.12416
Dennis Aumiller and Michael Gertz. 2022. UniHD at
TSAR-2022 shared task: Is compute all we need for lexical simplification? shop on Text Simplification, Accessibility, and Read- ability (TSAR-2022), pages 251-258, Abu Dhabi, United Arab Emirates (Virtual). Association for Com- putational Linguistics.
Tim Detimers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. 2022. LLM.nt80:: 8-bit Matrix Multiplication for Transformers at Scale. ArXiv:2208.03399 [es].
Proceedings of the Work-
Ashwin Devaraj. William Sheffield, Byron Wallace, and Junyi Jessy Li. 2022. Evaluating factuality in text simplification. In Proceedings of the 60th Annual Meeting of the Association for Computational Lin- guistics (Volume I: Long Papers), pages 7331-7345, Dublin, Ireland. Association for Computational Lin- guistics.
David Baidoo- Anu and Leticia Owusu Ansah. 2023, Ed- ucation in the era of generative artificial intelligence (ai): Understanding the potential benefits of chatgpt in promoting teaching and learning. Available at SSRN 4337484.
Chandrayee Basu, Rosni Vasu, Michihiro Yasunaga, and Qian Yang. 2023. Med-easi: Finely annotated dataset and models for controllable simplification of medical texts. In Proceedings of the Thirty- Seventh AAAI Conference Artificial Intelligence and Thirty-Fifth Conference on Innovative Applica- tions of Artificial Intelligence and Thirteenth Sympo- sium on Educational Advances in Artificial Intelli- AAAI Press.
Michael Dowling and Brian Lucey. 2023. Chatgpt for (finance) research: The bananarama conjecture. Fi- nance Research Letters, 1036626..
Yutao Feng, Jipeng Qiang, Yun Li, Yunhao Yuan, and Yi Zhu. 2023. Sentence Simplification via Large Language Models. ArXiv:2302. 1957 [cs].
gence,
Leo Gao, Stella Biderman, Sid Black, Laurence Gold- ing, Travis Hoppe, Charles Foster, Jason Phang, Ho- race Hc, Anish Thite, Noa Nabeshima, et al. 2020. The pile: An 800gb dataset of diverse text for lan- guage modeling. arXiv preprint arXiv:2101.0002
Jason Baumgartner, Savvas Zannettou, Brian Keegan, Megan Squire. and Jeremy Blackburn. 2020. The pushshift reddit dataset. In Proceedings of the inter- national AAAI conference on web and social media, volume 14, pages 830-839.
Tianxing He. Jingyu Zhang, Tianle Wang. Sachin Kumar, Kyunghyun Cho, James Glass, and Yulia Tsvetkov. 2022. On the Blind Spots of Model- Based Evaluation Metrics for Text Generation. ArXiv:2212.1000 [cs].
Sidney Black, Stella Biderman, Erie Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang,

---

Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The Curious Case of Neural Text Degeneration, arXiv: 1904.09751 [cs].
United Arab Emirates. Association for Computa- tional Linguistics.
Yang Liu, Dan Iter. Yichong Xu, Shuohang Wang. Ruochen Xu, and Chenguang Zhu. 2023. G-eval: Nig evaluation using gpt-4 with better human align- ment, arxiv 3663..
Srinivasan Iyer, Xi Victoria Lin, Ramakanth Pasunuru, Todor Mihaylov, Daniel Simig, Ping Yu, Kurt Shuster, Tianlu Wang, Qing Liu, Punit Singh Koura, Xian Li, Brian O'Horo, Gabriel Pereyra, Jeff Wang, Christo- pher Dewan, Asli Celikyilmaz, Luke Zettlemoyer, and Ves Stoyanov. 2023. Opt-iml: Scaling language model instruction meta learning through the lens of generalization.
Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2022. Fantastically Ordered Prompts and Where to Find Them: Overcoming Few- Shot Prompt Order Sensitivity. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume I: Long Papers), pages Dublin, Ireland. Association for Compu- tational Linguistics.
Chao Jiang, Mounica Maddela, Wuwci Lan, Yang Zhong. and Wei Xu. 2020. Neural CRF model for sentence alignment in text simplification. In Proceed- ings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7943-7960, On- line. Association for Computational Linguistics.
Mounica Maddela, Yao Dou, David Heineman, and Wei Xu. 2023, LENS: A learnable evaluation metric for text simplification. In Proceedings of the 61 An- nual Meeting of the Association for Computational Linguisties (Volume I: Long Pupers), pages 16383- 16408, Toronto, Canada. Association for Computa- tional Linguistics.
J. Peter Kincaid, Robert P. Fishburne, L Rogers, and Brad S. Chissom. 1975. Derivation of new readabil- ity formulas (automated readability index, fog count and flesch reading ease formula) for navy enlisted personnel. In Institute for Simulation and Training, pages 1-49.
Louis Martin, Éric de la Clergerie, Benoît Sagot, and Antoine Bordes. 2020. Controllable sentence sim- plification. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 4689- 4698, Marseille, France. European Language Re- sources Association.
Hugo Laurençon, Lucile Saulnier, Thomas Wang, Christopher Akiki, Albert Villanova del Moral, Teven Le Scao, Leandro Von Werra, Chenghao Mou, Ed- uardo González Ponferrada, Huu Nguyen, et al. 2022. The bigscience roots corpus: A 1.6 th composite mul- tilingual dataset. Advances in Neural Information Processing Systems, 11:31889931888
Louis Martin, Angela Fan, Éric de la Clergerie, Antoine Bordes, and Benoît Sagot. 2022. MUSS: Multilin- gual unsupervised sentence simplification by mining paraphrases. In Proceedingg of the Thirleenth Lan- guage Resources and Evaluation Conference, pages 1651-1664, Marseille, France. European Language Resources Association.
Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficiet prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3045-3059, Online and Punta Cana, Domini- can Republic. Association for Computational Lin- guistics.
Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari. Sheng Shen, Zheng-Xin Yong. Hailey Schoelkopf. et al. 2022. Crosslingual generaliza- tion through multitask finetuning. arXiv preprint arXiv:2211.01788.
Mike Lewis. Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. pre-training
BART: Denoising sequence-to-sequeeeee for natural language generation, translation, and com- prehension. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 7871-7880, Online. Association for Computa- tional Linguistics.
Mathias Müller, Annette Rios, and Rico Sennrich. 2020. Domain robustness in neural machine translation. In Proceedings of the 14th Conference of the Associa- tion for Machine Translation in the Americas (Volume 1: Research Track), pages 151-164 Virtual. Associa- tion for Machine Translation in the Americas.
Quentin Lhoest, Albert Villanova del Moral, Yacine Jemite, Abhishek Thakur, Patrick von Platen, Suraj Patil, Julien Chaumond, Mariama Drame, Julien Plu. Lewis Tunstall, et al. 2021. Datasets: A commu- nity library for natural language processing. arXiv preprint 20X:::210.028800 028402
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Car- roll L. Wainwright, Pamela Mishkinn Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray. John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback.
Yanyang Li, Jianqiao Zhao, Michael Lyu, and Li- wei Wang. 2022. Eliciting knowledge from large pre-trained models for unsupervised knowledge- grounded conversation. In Proceedings of the 2022 Conference on Empirical Methods Natural Lan- guage Processing, pages 10551-10564 Abu Dhabi,
Gustavo Paetzold and Lucia Specia. 2016. Benchmark- ing lexical simplification systems. In Proceedings

---

of the Tenth International Conference on Language Resources and Evaluation (LREC'16), pages 3074- 3080, Portorož, Slovenia. European Language Re- sources Association (ELRA).
Thomas Wolf, and Alexander M. Rush. 2022. Multi- task prompted training enables zero-shot task gener- alization. In The Tenth International Conference Learning Representations, ICLR 2022, Virtual Event, April 25-29. 2022. OpenReview.net.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: method for automatic evalu- ation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Compu- lational Linguistics, pages 311-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
Teven Le Scao, Angela Fan, Christopher Akiki, El- lie Pavlick, Suzana Ilié, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. 2022. Bloom: A 176b- parameter open-aceess multilingual language model. arXiv preprint arXiv:221..00 1.05100.
Timo Schick and Hinrich Schütze. 2021. It's not just size that matters: Small language models are also few- shot learners. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pages 2339-2353 Online. Association for Computational Linguistics.
Alec Radford, Karthik Narasimhan, Tim Salimans, Ila Sutskever, et al. 2018. Improving language under- standing by generative pre-training. Technical report, OpenAI.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, and others. 2019. Lan- guage models are unsupervised multitask learners. OpenA/ blog. 1(8):
Max Schwarzer and David Kauchak. 2018. Human Evaluation for Text Simplification: The Simplicity- Adequacy Tradeoff. Technical report, SoCal NLP Symposium.
Colin Raffel, Noam Shazeer, Adam Roberts, Kather- ine Lee. Sharan Narang, Michael Matena. Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research. 21(14)):1-67.
Prasann Singhal, Jarad Forristal, Xi Ye, and Greg Dur- rett. 2023. Assessing out-of-domain language model performance from few examples. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 2385-23997 Dubrovnik, Croatia. Association for Computational Linguistics.
Michael Ryan, Tarek Naous, and Wei Xu. 2023. Re- visiting non-English text simplification: A unified multilingual benchmark. In Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4898- 4927, Toronto, Canada. Association for Computa- tional Linguistics.
Dominik Sobania, Martin Briesch, Caril Hanna, and Justyna Petke. 2023. An analysis of the auto- matic bug fixing performance of chatgpt. In 2023 IEEE/ACM International Workshop on Automated Program Repair (APR), pages 23-30, Los Alamitos, CA, USA. IEEE Computer Society.
Horacio Saggion, Sanja Štajner, Daniel Ferrés, Kim Cheng Sheang. Matthew Shardlow, Kai North, and Marcos Zampieri. 2022. Findings of the TSAR- 2022 shared task on multilingual lexical simplifica- tion. In Proceedings of the Workshop Text Simpli- fication, Accessibility, and Readability (TSAR-2022). pages 271-283, Abu Dhabi, United Arab Emirates (Virtual). Association for Computational Linguistics.
Sanja Stajner. 2021. Automatic text simplification for social good: Progress and challenges. In Findings of the Association for Computational Linguistics: ACL- IJCNLP 2021, pages 2637-2652 Online. Association for Computational Linguistics.
Sanja Stajner. Daniel Ferrés, Matthew Shardlow, Kai North, Marcos Zampieri, and Horacio Saggion. 2022. Lexical simplification benchmarks for English, Por- tuguese, and Spanish. Frontiers in Artificial Intelli- gence. 5.
Malik Sallam. 2023. Chatgpt utility in healthcare ed- ucation, research, and practice: Systematic review on the promising perspectives and valid concerns. Healtheare, 1(6).
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. 2020. Learn- ing to summarize with human feedback. Advances in neural information processing systems, volume 33, pages 3008-3021 Curran Associates, Inc.
Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu. Urmish Thakker, Shanya Sharma Sharma, Eliza Szezechla, Taewoon Kim, Gunjan Chhablami. Nihal V. Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong. Harshit Pandey, Rachel Bawden, Thomas Wang, Tr- ishala Neeraj. Jos Rozen, Abheesht Sharma, An- drea Santilli, Thibault Févry, Jason Alan Fries, Ryan Teehan, Teven Le Scao, Stella Biderman, Leo Gao,
Renliang Sun. Wei Xu, and Xiaojun Wan. 2023, Teach- ing the pre- pre-trained model to generate simple texts for text simplification. In Findings of the Associa- tion for Computational Linguistics: ACL 2023, pages 9345 0345-9955, Toronto, Canada. Association for Com- putational Linguistics.

---

Teerapaun Tanprasert and David Kauchak. 2021. Flesch-kincaid is not a text simplification evaluation metric. In Proceedings of the 1tt Workshop on Natu- ral Language Generation, Evaluation, and Metrics (GEM 2021), pages 1-14, Online. Association for Computational Linguistics.
Dhanasekaran, Atharva Naik, David Stap. Eshaan Pathak, Giannis Karamanolakis, Haizhi Gary Lai, Is- han Purohit, Ishani Mondal, Jacob Anderson, Kirby Kuznia, Krima Doshi, Maitreya Patel. Kuntal Kumar Pal, M. Moradshahi, Mihir Parmar, Mirali Purohit, Neeraj Varshney, Phani Rohitha Kaza, Pulkit Verma, Ravsehaj Singh Puri, Rushang Karia, Shailaja Keyur Sampat, Savan Doshi, Siddharth Deepak Mishra, Su- jan Reddy, Sumanta Patro, Tanay Dixit, Xudong Shen. Chitta Baral, Yejin Choi, Hannanch Hajishirzi, Noah A. Smith, and Daniel Khashabi. 2022. Bench- marking generalization via in-context instructions on I, 600+ language tasks. ArXiv, 2204.07005
Yi Tay, Mostafa Dehghani, Vinh Q. Tran, Xavier Garcia, Jason Wei, Xuezhi Wang, Hyung Won Chung, Dara Bahri, Tal Schuster, Steven Zheng. Denny Zhou, Neil Houlsby, and Donald Metzler. 2023. UL2: Unifying language learning paradigms. In The Eleventh Inter- national Conference on Learning Representations.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux. Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and effi- cient foundation language models. arXiv preprint arXI
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V Le. 2022. Finetuned language mod- els are zero-shot learners. International Confer- ence on Learning Representations.
Laura Basquez-Rodriguez,
Nhung Nguyen, Matthew Shardlow. and Sophia Ananiadou. 2022. UMM&MMU at TSAR-2022 shared task: Prompt learning for lexical simplification. In Proceedings of the Workshop on Text Simplification, Accessibility, and Readability (TSAR-2022), pages 218-224, Abu Dhabi, United Arab Emirates (Virtual). Association for Computational Linguisties.
Thomas Wolf, Lysandre Dehut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi. Pier- ric Cistac, Tim Rault, Remi Louf, Morgan Funtow- icz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite. Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Trans- formers: State-of-the-ar natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38 38-45, Online. Association for Computational Linguistics.
Laura
Matthew Shardlow, Piotr Przybyla, and Sophia Ananiadou. 2021a. Investi- gating text simplification evaluation. In Findings of the Association for Computational Linguistics: ACL- IJCNLP 2021, pages 876-882, Online. Association for Computational Linguistics.
Wei Xu, Chris Callison- and Courtney Napoles. 2015. Problems in current text simplification re- search: New data can help. Transactions of the Asso- ciation Computational Linguistics, 5:283-297.
Laura Vásquez-Rodrrguez, Matthew Shardlow, Piotr Przybyla, and Sophia Ananiadou. 2021b. The role of text simplification operations in evaluation. In Proceedings of the First Workshop on Current Trends in Text Simplification (CTTS-202I), pages 57-69.
Wei Xu, Courtney Napoles, Ellie Pavlick, Quanze Chen, and Chris Callison-Burch. 2016. Optimizing sta- tistical machine translation for text simplification. Transactions of the Association for Computational Linguistics, 4:401-415.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, & ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Pro- cessing Systems, volume 30. Curran Associates, Inc.
Ningyu Zhang. Luoqiu Li, Xiang Chen, Shumin Deng, Zhen Bi, Chuanqi Tan, Fei Huang, and Huajun Chen. 2022a. Differentiable Prompt Makes Pre- trained Language Models Better Few-shot Learners. ArXiv:2108. 13161 [cs].
Robert A. Wagner and Michael J. Fischer. 1974. The String-to-String Correction Problem. Journal of the ACM (JACM), 21(1):168-173:
Rongzhi Zhang, Yue Yu, Pranav Shetty, Le Song, and Chao Zhang. 2022b. Prompt-based rule discovery and boosting for interactive weakly-superrvise learn ing. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 745-758, Dublin, Ireland. Association for Computational Linguistics.
Ben Wang and Aran Komatsuzaki. 2021. GPT-J- 6B: A Billion Parameter Autoregressive Lan- guage Model. https:/ httts:://giihuu...c///nooooloo com/kingoflolz/ mesh-traanffrmer-jjaa
Longyue Wang, Chenyang Lyu. Tianbo Ji, Zhirui Zhang, Dian Yu, Shuming Shi, and Zhaopeng Tu. 2023. Document-level machine translation with large lan- guage models.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher De- wan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022c. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01066
Yizhong Wang, Swaroop Mishra, Pegah Alipoor- molabashi, Yeganeh Kordi, Amirreza Mirzaei, Anjana Arunkumar, Arjun Ashok, Arut Selvan

---

Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evalu- ating text generation with BERT. In 8th International Conference on Learning Representations, ICLR 2020, Addis Abuba, Ethiopia, April 26-30, 2020. OpenRe- neew.net.
A Model Details
In this section, we describe each type of LLM we use in our experiments.
A.1 Open-weight Models
Xingxing Zhang and Mirella Lapata. 2017. Sentence simplification with deep reinforcement learning. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 584- 594, Copenhagen, Denmark. Association for Compu- tational Linguistics.
As brief disclaimer, we note that some listed mod- els are not truly "open-weight" and may require spe- cial permission to obtain weights for self-hosting. Further, in descriptions, do distinguish between different variations of the same model. We provide the details of the training data and model sizes in Table 4. We consider both encoder-decoder and decoder-only models for our evaluation as dis- cussed below.
Zhemin Zhu, Delphine Bernhard, and Tryna Gurevych. 2010. A monolingual tree-based translation model for sentence simplification. Proceedings of the 23rd International Conference on Computational Lin- guistics (Coling 2010), pages 1353-1361, Beijing, China. Coling 2010 Organizing Committee.
A.1.1 Encoder-Decoder Models
Terry Yue Zhuo, Yujin Huang. Chunyang Chen, and Zhenchang Xing. 2023. Red teaming ChatGPT via Jailbreaking: Bias, Robustness, Reliability and Toxi- city. arXiv e-prints, page arXiv:2301.1888
T5 Family We evaluate range of model variants derived from the original T5 models (Raffel al., 2020). Originally, training recipes for T5 employ pre-training with span-infilling objective and are thus not suitable for left-to-right generation tasks off the shelf. We thus use the T5-LM-adapted mod- els from (Lester et al., 2021) which have undergone continued pre-training using standard LM objec- tive.
One later derivation includes the instruction- tuned variant Flan-T5 (Chung et al., 2022), which continues training from the aforementioned T5-LM-adapted checkpoints and uses wide vari- ety of labeled data for instruction fine-tuning. No- tably, the dataset description by Chung al. (2022) does not include any reference simplification- related tasks. Similar parallel efforts lead the creation of the T models (Sanh et al., 2022).
Finally, UL (Tay et al., 2023) proposes more diverse set of pre- pre-training objectives beyond simple span corruption. Additional tasks include sequence distortion and extreme span corruption.
A.1.2 Decoder-only Models
GPT-J/GPT-X Early reproduction efforts of large- scale GPT-style models started following the surge in popularity of GPT-2 (Radford et al., 2019). For our benchmark, we include models published by EleutherAI, namely the 6 billion parameter variant of GPT-J (Wang and Komatsuzaki, 2021) and the 20 billion parameter version of GPT-NeoX (Black et al., 2022). Both models were trained with a standard LM pre-training objective and were not fine-tuned to follow instructions.

---

Models
Type
Sizes
Training Data
BLOOM
D
560M, 1b1. 3b, 7b, 175b
ROOTS (Laurengon al., 2022), Huggingface Datasets (Lhoest al., 2021)
BLOOMZ
560M, 1b1, 3h, 7h, 175b
P3 (Sanh et al., 2022), xP3
LLaMA
D
7b, 13b, 30b, 65b
CommonCrawl, C4 (Raffel et al., 2020), Github, Wikipedia, ArXiv, StackExchange
OPT
1.3b, 6.7b, 13b, 30b, 66b
(Gao et al., 2020), Reddit (Baumgartner al., 2020)
OPT-IML
D
1.3b, 30b,
OPT-IML Benchmark (lyer al., 2023)
GPT-J
6b, 20b
Pile (Gao 2020)
E-D
60m (small), 220m (base), 770m (large), 3b (xl),
C4 (Raffel al., 2020)
11b (xxl)
TO, T0pp
B-D
3b, 1lb
P3 (Sanh al., 2022)
Flan-T5
E-D
60m (small), 220m (base), 770m (large). 3b (xl),
Muffin (Wei al., 2022), P3 (Sanh al., 2022).
11b (xxl)
NIV2 (Wang et al., 2022)
UL2
R-D
20b
FLan-UL2
B-D
20b
Muffin (Wei al., 2022), P3 (Sanh al., 2022), NIV2 (Wang et al., 2022)
Table 4: Description of Open-Weight models. Model type "D" refers to decoder-only models, "E-D" for models based on an mecderr architecture.
OPT/LLaMA Reproduction efforts of large-scale decoder-only models conducted by researchers at Meta Al were released under the OPT label (Zhang al., 2022c) and more recently under the LLaMA label (Touvron al., 2023). Besides differ- ent composition in training data and some im- plementation choices relating to hardware perfor- mance, they otherwise share similar architectures and training objectives with the previously men- tioned GPT-like models. Tyer et al. (2023) experi- mented with instruction tuning the OPT models to provide OPT-IML checkpoints, which also use in BLESS.
the API inference costs of our experiments with OpenAl's models.
Model
ASSET
MED-EASI
NEWSELA
tokens
Ada-801
0,0004
0.35
0.41
0.28
Babbage -000agge-e111--
0.0005
0.44
0.51
0.35
Curie-0a1
0.002
1.76
2.01
1.41
(PT-3.5-Turde
0,002
1.75
1.95
1.37
Davinci
Davinci-98 0.02
17.52
19.90
13.96
Table 5: Pricing information for OpenAT's API models. Here we report the total costs incurred for all three inference prompts and three seeded runs, totalling nine inference per dataset. Prices listed correspond to those for the API-based models available from April through June, 2023. prices are USD.
BLOOM The result of an open collaboration, the BLOOM model family (Scao et al., 2022) represents the largest open-weight models available at the time of writing, up the full 176 billion parameter scale of GPT-3 (Brown et al., 2020). The original model was only trained with a standard LM pre- training objective. BLOOMZ models (Muennighoff et al., 2022) extend these models with instruction fine-tuning.
B Supplemental Results
Tables 6, 7, and show full results for these on ASSET, MED-EAS, and NEWSELA respectively.
B.1 Details on Evaluation Metries
A variety of automatic evaluation methods have been proposed. Commonly used automatic metrics like BLEU (Papineni et al., 2002) and SARI (Xu et al., 2016) can provide insights into how similar model's outputs to set of gold reference simplifications. However, more precisely under- stand model's strengths and weaknesses, finer- grained evaluation is often required. For example, calculating the distribution of edit simplification operations (e.g. additions and deletions) (Vásquez-
A.2 Closed Closed-Weight Models
As the current primary choice for commer- cial solutions, benchmark range of mod- els by OpenAl. Previous publications regarding the GPT family (Radford al., 2018, 2019; Brown et al., 2020) establish that these models MAda/Babbage/Cae CetCCurrriiiiii are decoder-only, with varying numbers of parameters. Table 5 shows

---

Rodríguez et al., 2021a.b) can yield more insights into the capabilities of these systems. We eval- uate model outputs according to multiple metrics. While we focus on reporting SART and BERTScore in order to relate our findings with previous work, we also compute additional evaluation metrics for more fine-grained analyses and perform qualita- tive analysis. Specifically, report:
Table 6: Simplification Results ASSET
SARI
FKGL,
BERT!
LENS
Buselines
Gold References
45.27
6.53
78.89
65.58
MUSS-mined
42.29
8.18
79.86
61.36
1. SARI (Xu al., 2016): SARI (System output Against References and against the Input sen- tence) is a holistic metric for simplification quality, computes the score for n-grams added, kept, and deleted, with respect to the input (source) and reference sentences.
MUSS-wiki-mines
44.90
5.29
77.71
69.23
LIMs
Ada-8e1 881
33.97
9.06
81.76
56.41
Babbage-001
38.44
8.65
82.46
61.39
Curie-801
39.87
8.33
82.75
63.02
Davinci-002
42.84
7.77
85,91
67.09
Davinci-003
7.74
79.66
67.39
2. BERTScore (Zhang et al., 2020): We com- pute the BERTSccoe precision, recall and Fl of the predictions against both the reference and source sentences, totaling in 6 different scores. Results reported in the paper use BERTScore F1 computed between system out- put simplifications and the gold reference sen- tence(s).
GPT-3.5-Turbo
47.69
7.51
79.39
69.17
BLOOK-560m
36.14
8.01
50.11
42,68
BL00M-1b1
8.18
68.60
51.23
BL00M-3b
37.15
7.92
72.28
54.34
BL000-7b1
36.96
8.17
77.82
57.37
BLOOK
39.72
7.78
76.63
60.37
BL 00KZ-560m
35.12
7.52
41.21
39.52
BL00M2-1b1
35.00
8.42
76.66
54,86
BLOOMZ-3b
8.73
75.86
56.78
BL0002-7b1
37.05
8.56
79.09
59.14
3. FKGL (Kincaid al., 1975): FKGL (Flesch- Kincaid Grade Level) is weighted score based on sentence length and syllable infor- mation. The lower the FKGL, the simpler the output, and the lowest possible score is -3.40. However, for given test set, we consider the best FKGL to be the score that is closest to the FKGL of the gold references.
BLOOKZ
37.63
8.27
82.06
61.07
GPT-J-6b
38.86
7.83
76.48
60.13
GPT-NeoX-20b
39.04
8.04
75.81
60.87
LLaMA-7b
40.70
7.39
75.52
62.80
LLaMA-13b
40.45
7.33
76.13
62.95
LLaMA-30b
39.14
7.32
78.74
62.73
LLaMA-65b
38.59
8.07
81.59
62.90
0PT-1.3b
33.01
8.61
75.57
$7.08
0PT-6. 7b
38.64
7.79
76.62
61.26
0PT-13b
38.78
8.03
79.08
60.51
OPT-30b
38.04
8.17
77. 77.22
60.01
4. LENS (Maddela et al., 2023): LENS (Learnalle Evaluation Metric for Text Simplification) is score between 0 and 100 estimated by model trained on complex- simple pairs annotated with human ratings. We report the average LENS score for each dataset.
DPT-66b
39.64
7.76
76.72
61.68
OPT-IML-Max-1.BB
7.66
79.73
61.31
OPT-IML-Max--20
42.03
6.62
79.39
65.29
Te-3b
35.16
8.90
54.92
50.38
T0
36.49
8.56
55.32
48.71
Têpp
35.05
8.65
47.69
44.67
TS-snal1-LM-adapt
33.89
6.61
10.27
14,56
IS-base-LM-adapt
34.70
6.80
19.63
14.27
TS-1arge-LM-adapt
31.12
6.88
37.82
15.21
T5-x1-LM-adapt
29.12
7.06
48.25
23.39
T5-xx1-LM-adapt
33.17
6.85
46.59
25.43
Flan-T5-smal1
38.57
7.58
77.26
$4,80
Flan-T5-base
41.40
7.32
79.70
62.75
Flan-T6-large
42.17
6,78
80.44
63.35
Flan-TS-x1
41.07
7.16
85.06
64.74
Flan-T5-xx1
41.75
7.27
84.13
66.08
UL2
35.65
7.65
37.01
15.99
Flan-UL2
42.83
6.85
84.34
67.36

---

Table 7: Simplification Results MED-EASI
Table 8: Simplification Results on NEWSELA
Model
SARTT
FKGIJ
BERTY
LENS
Model
SARI
FKGL, BERT!
LENS
Baselines
Baselines
Gold References
100
9.59
100
65.89
Gold References
60.11
5.88
87.66
71.02
MUSS-mines
35.15
9.29
42.55
52.48
MUSS-mined
38.40
7.86
72.14
61.49
MUSS MUSS-waki-mined
35.12
8.04
43.07
59.12
MUSS-wiki-mines
41.24
6.12
74.10
67.61
LLMs
LIMs
Ada-801
36.52
10.62
33.95
41.43
Ada-881
34.42
8.66
70.33
55.06
Babbage-001
36.60
10.49
37.95
53.91
Babbage-001
36.41
8.32
62.99
60.91
Curie-881
38.22
10.15
39.31
56.10
Curie-801
37.53
8.23
69.17
64.35
Davinci-002
36.34
10.05
43.67
57.71
Davinci-002
40.25
7.46
73.62
68.58
Davinci-003
39.81
Davinci-003
37.76
7.75
61.56
66.20
GPT-3.5-Turee
40.14
8.93
40.67
63.80
GPT-3.5-Turbo
37.29
7.80
60.19
67.97
BL00K-550m
35.37
7.58
-2.60
36.27
BL BLO0K-560m
33,41
7.76
31.85
38.58
BLO0M-1b1
35.86
7.37
1.63
BL BL00M-1b1
35.37
7.99
48.52
46.54
BLOOK-3b
35.48
7.40
5,94
42.21
BLO0K-3b
35.85
8.22
55.33
51.71
BL00M-7b1
37.47
7.23
9.53
44.17
BL BL00K-7b1
36.12
7.96
61.00
54.16
BLOOM
37.72
7.11
11.95
47.50
BLOOK
37.48
7.49
61.17
60.98
BL00MZ-560m
33.14
6.83
-3.08
38.32
00KZ-560m
28.55
7.53
17.56
34,21
BL00M2-1b1
35.65
6.40
43.69
BL00M2-1b1
35.22
7.47
54.19
53.05
BLOOM2-3b
35.68
7.17
8.56
44.79
BLO0MZ-3b
34.75
8.51
52.51
52.37
BL00M2-7b1
36.78
7.08
9.43
47.15
BL00K2-7b1
36.21
8.29
59.53
59.36
BLOOMZ
36.60
7.08
12,90
47.67
BLOOKZ
37.06
8.41
69.55
62.07
GPT-J-6b
36.20
7.01
10.53
46.67
GPT-J-6b
36.8
7.47
59.59
58.98
GPT-NeoX-206
36.02
7.07
10.62
46.46
GPT-NeoX-20b
36.87
7.62
56.85
59.71
LL.aMA-7b
36.95
6.62
10.28
48.42
LLaMA-7b
36.70
6.28
55.31
62.43
LLaMA-13b
36.98
6.73
11.43
48.63
LLaMA-13b
37.16
6.42
59.61
63.32
LLaMA-30b
37.56
6.89
12.21
47.92
LLaMA-30b
37.50
6.75
63.89
64,30
LLaMA-65b
37.86
6.85
12.20
47.45
LLaMA-65b
38.59
7.10
67.82
64.24
0PT-1.3b
34.00
7.17
3.82
43.64
0PT-1.3b
34.76
7.96
50.78
55.35
OPT-6.76
34.73
7.02
8.86
47.72
0PT-6. 7b
36.58
7.76
58.68
60.28
0PT-13b
34.69
6.96
8.73
47.16
OPT-13b
37.67
7.16
60.65
61.31
0PT-30b
35.08
7.02
9.96
46.96
OPT-30b
37.58
7.75
61.79
61.91
0PT-66b
35.72
6.96
11.42
47.28
DPT-66b
37.45
7.25
60.43
62.98
OPT-IML-MaX-1.B8
37.01
7.12
11.85
0PT-INL-Max-1.25
37.08
7.32
62.68
60.47
OPT-IML-Max-38R
35.80
6.78
11.73
49.28
-PT-IML-MAN -Max-30b
39.59
6.09
66.39
64,74
T0-3b
38.16
10.34
17.83
42.02
Te-3b
33.37
8.50
36.56
50.64
T0
35.67
10.81
15.93
42.76
T0
32.83
7.58
30.96
53.23
Töpp
35.61
10.67
11.58
36.60
Têpp
33.02
8.20
30.66
47.62
TS-small-LM-adaot
34.71
8.87
-4.15
12.68
TS-smal1-LM-adapt
30.54
6.33
4.85
16.85
T5- 15-base-LM-adapt
34.70
8.47
-0.92
16.41
IS-base-LM-adapt
32.94
6.00
12.13
17.80
TD-Large-LM-adapt
36.69
8.62
10.27
19.34
TM-large-LM-adapt
33,48
6.35
30.99
20.07
TS-xl-LM-adapt
33.65
8.83
18.91
22.59
T5-x1-LM-adapt
32.85
6.62
42.51
25.48
T5-xx1-LM-adapt
32.61
9.10
21.69
28.21
TS-xx1-LM-adapt
33.44
6.67
44.15
29.49
Flan-T5-small
36.65
8.99
38.60
45.37
Flan-T5-smal1
37.72
7.61
68.15
53.61
Flan-T5-base
36.79
9.05
40.63
51.95
Flan-T5-base
38.67
7.21
68.09
59.11
Flan-T5-large
35.71
8.70
41.31
52.59
Flan-T5-large
39.08
6.90
70.27
62.70
Flan-T5-x1
33.21
9.11
44.12
54.75
Flan-TS-x1
37.51
7.25
75.50
64.39
Flan-TS-xxl
34.27
9.13
43.43
54.70
Flan-T5-xx1
39.42
7.32
73.05
65.13
UL2
35.89
9.28
17.15
19.79
UL2
35.22
6.92
37.07
21.62
FLan-UL2
35.31
8.52
42.80
57.95
Flan-UL2
40.27
6.86
73.23
66.42

---

Model
Sentence
Annotation
Complex
They are rivaled as biological materials in toughness only by chitin.
Reference
They rivaled only by chitin in toughness.
GoT-3.5-Turbo
Chitin is the only biological material that rivals them in toughness.
St P+ I
Davinci-003
Chitin is the only biological material tougher than them.
St
Davinci-802
They are tough like chitin, which is the toughest known biological material.
MP, P+ L+
Flan-UL2
They are only second to chitin for biological materials.
MP, P+ L++
Flan-T5-large
Chitin is better than human materials in toughness.
MPJ P+ L++
Table 9: Annotation examples from a SARI-based model ranking. St Simplification, P: Paraphrasing, L: Lexical Simplification, and MP: meaning preservation. We highlight lexical simplification in bold and conflicts in meaning preservation in red.
mu
OparA/ Con
R.CO
8.00 R0
0.00P2
K0073
Figure 5: Token-level edit operations computed for all models and test sets using prompt 2. For most models, the edit operations performed ASSET and NEWSELA reflect those in the gold reference simplifications. However. the MED--EAS dataset, we observe sudden spike in insertions from all LLMs except for OpenAl and Flan models. These additions indicate the presence of potentially unrelated hallucinated tokens and endless generations. which aligns with the low BERTScore results. We regard this failure case be related the fact that MED-EAAS presents challenging domain which is out of the distribution of most general-purpose purpose models.

---

ASSET
Med-FASi
Newaela
86
X
75
70
Open Status
BERTScofe
Open
82
x
65
Parame nets
80
1003 greater
X
60
78
1-108
55
76
50
34
36
40
36
38
40
SARI
SARI
SARI
Figure 6: Adepuacy-simplictty trade-off as exhibited by a limited set of models each of the three datasets. On ASSET, higher SARI is associated with lower BERTScore. the case of MED-EASI, we can see that smaller models, which often tend copy the input sentence, are rewarded by BERTScore but punished by SARI Here, only the -lighd-weight OpenAl models exhibit a favorable balance between the two metrics. On NEWSELA, the relationship more linear. We suspect that this influenced by the fact that reference sentences are taken from multiple simplification levels (1-4) and therefore cover broader range of possible rewrites, some with more simplifying edit operations (rewarded by SARI) and some with fewer (rewarded by BERTScore).
SARI
47.5
45.0
42.5
40.0
37.5
35.0
32.5
BERTScore
86
84
82
80
78
76
74
OpenAl Ada-001
OpenAl Babbage- 001
OpenAI Curie-001
OpenAl Davinci-002
OpenAl Davinci-003
BLOOM
BLOOMZ
OPT-1.3b
OPT-IML-MAN MAX-1-X 1.3b
OPT-30b
OPT-IML-AN MAX-30b
Flan- Flan-T5-sman T5-smlll
Flan- T5-base base
Flan- T5-large
Ran- Fln-T55-l
Flan-UL2
OpenAl GPT-3.5-Turk
Flan-
Figure Visualizing LLM performance for select models, generated using prompt This visualization corresponds to the results reported in Table 2 for ASSET. Models the x-axis are ordered by model family, and within each model family, they are ordered by size (ascending).


Reviews:
[{"review_id": "6pBv2G9zcD", "reviewer": "Reviewer_WJrH", "summary": "", "questions": "Question A: Can you provide an example of the inputs and outputs for a sentence simplification task?\n\nQuestion B: Could you include the date on which you ran the experiments for closed-access models (since they are constantly being updated) and also provide pricing information for the sake of reproducibility?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to read our paper and for providing useful feedback. Unfortunately, it seems that the text in “Reasons to Accept” is cut off. We would very much appreciate it if you could kindly complete this section.\n\nBelow, we have summarised and responded to each point of concern from the review.\n\n- Existing models: Our primary goal with this paper was to understand how existing LLMs perform on the task of text simplification and provide a common evaluation benchmark. Therefore, we focused on evaluating as many LLMs as we could and did not train any new models.\n- Existing metrics and existing datasets: In order to directly compare our results with those from previous work, we chose to evaluate with widely used text simplification metrics and datasets. However, the goal of our qualitative analysis is to help fill gaps and diagnose problems with the simplification outputs that are difficult to detect with automatic metrics and likert-scale human ratings.\n- Description of the task: The comparison with paraphrasing is valid, since both tasks involve rewriting with the aim of retaining meaning. Paraphrasing’s only constraint is that the rewritten output text must retain the original text’s meaning. However, text simplification imposes an additional constraint, namely that the output should reduce the complexity of the input sentence and improve readability (e.g. avoiding rare or difficult words, reformulating challenging grammatical structures like cleft sentences into their constituent parts, reducing anaphora, deleting unimportant information, etc.). To address the point raised, we will elaborate on the task description and link it to section 4.2, which lists the general types of edit operations associated with simplification.\n\nQuestion A:\n\nHere is an example of a complex sentence with its simplification taken from the Newsela dataset:\n\nComplex: “A federally recognized tribe is considered a sovereign nation that has a government-to-government relationship with the United States and possesses the right to create and enforce laws, regulate activities within its jurisdiction and determine membership criteria.”\n\nSimple: “Recognized tribes are considered independent nations. They can make their own laws. They can govern their territory. Recognized tribes decide who is a member.”\n\nQuestion B:\n\nExperiments with the API-based models were run between late April and mid June, 2023.\nThe total inference costs for these using OpenAI models was about $115 (USD). Below is a detailed breakdown of the inference costs per model/dataset for our particular experiments. Note that we ran inference 3 times for each different prompt on each dataset, totalling 9 runs.\n\nopenai-gpt-3.5-turbo (USD 0.002 / 1k tokens): \nasset: USD 1.75, med-easi: USD 1.95, newsela: USD 1.37 \n\nopenai-text-ada-001 (USD 0.0004 / 1k tokens): \nasset: USD 0.35, med-easi: USD 0.41, newsela: USD 0.28 \n\nopenai-text-babbage-001 (USD 0.0005 / 1k tokens):\nasset: USD 0.44, med-easi: USD 0.51, newsela: USD 0.35\n\nopenai-text-curie-001 (USD 0.002 / 1k tokens):\nasset: USD 1.76, med-easi: USD 2.01, newsela: USD 1.41\n\nopenai-text-davinci-002 (USD 0.02 / 1k tokens):\nasset: USD 17.62, med-easi: USD 20.06, newsela: USD 14.10\n\nopenai-text-davinci-003 (USD 0.02 / 1k tokens):\nasset: USD 17.52, med-easi: USD 19.90, newsela: USD 13.96\n\nWe agree that these details are useful regarding reproducibility and will be sure to include them in the appendix.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The set of Large Language Models evaluated is extensive and one of the most comprehensive I've seen. The authors benchmarked 44 LLMs ranging in size from 60M to 176B parameters and include instruction-tuned models.", "reasons_to_reject": "Though the authors do their own small analysis in Section 5, the bulk of this paper uses existing metrics to evaluate existing models on existing datasets. \nThe authors do not provide an example of sentence simplification, and it is unclear from their statement of the task (“rephrasing part or all of a sentence into language which is more widely understood“) does not distinguish the task from paraphrasing. -- after rebuttal period, I am satisfied with the authors' distinction between paraphrasing and sentence simplification", "questions_to_authors": "Question A: Can you provide an example of the inputs and outputs for a sentence simplification task?\n\nQuestion B: Could you include the date on which you ran the experiments for closed-access models (since they are constantly being updated) and also provide pricing information for the sake of reproducibility?", "paper_topic_and_main_contributions": "This paper benchmarks 44 Large Language Models on the task of sentence simplification.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "g3klm8GbyU", "reviewer": "Reviewer_2xuh", "summary": "", "questions": "1.) I could be wrong and might have overlooked but did you assigns some kind of overall \"quality score(s)\" in terms of simplicity / content preservation or TS quality in general in your manual evaluation? So sth. that would be directly comparable to BERTScore or SARI? I think that would be really interesting, both for the evaluation itself but of course also for assessing the usefulness of these automatic measures in comparison with human assessments. Maybe you could elaborate a bit on the description of Table 8 in the appendix.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for taking the time to read our paper and for your feedback. We are glad to hear that you enjoyed reading our work. We have responded to the points raised below. \n\n1.1 Single quality score: No, we did not assign a single score to the outputs assessed. While this would indeed be useful to relate to the automatic metrics, we believe single value scores are generally less useful for identifying specific strengths and weaknesses of different systems. Instead, we opted for assigning binary annotations to each output sentence. For example, S↑ means that the output was simplified. Although the percentage of outputs that received S↑ is not directly comparable with SARI, it is meant to convey the same thing - the simplicity of the model outputs. The same goes for MP↑, which conveys meaning preservation like BERTScore. The primary goal of the annotation scheme is to highlight the simplification capabilities of the models at first glance and how these are reflected given the automatic metrics. We agree that in future iterations of our work, it will be valuable to score our simplification based on larger sets of system outputs, as presented by previous work where they assess how well automatic metrics correlate with human assessments (see for example Alva-Manchego, et al., 2021 (https://direct.mit.edu/coli/article/47/4/861/106930/The-Un-Suitability-of-Automatic-Evaluation-Metrics), Maddela et al., 2022 (http://arxiv.org/abs/2212.09739)). \n\n1.2 Table 8: In Table 8, we show an example of our annotation scheme over a sampled system output. We labeled the presence of a valid simplification as S↑, Paraphrasing as P↑ (rewriting a text without necessarily simplifying it), Lexical Simplification as L+ (replacing complex words or phrases with simpler ones), including highlighting this change in bold. We also highlight in red the lack of meaning preservation as MP↓. Since it’s possible that these annotations can overlap, it is difficult to reduce this to a single score.\n\nWe thank you for your suggested grammatical edits and will incorporate those accordingly. Page 4 indeed refers to Davinci-002.\n\nRegarding the reproducibility of our work, our inference settings (especially with larger models) required a significant amount of GPU compute. That aside, two out of the three test datasets (ASSET and Med-Easi) should be reproducible with our code, which we will release upon acceptance. The Newsela dataset is available under a restrictive license but is also supported in our public code release.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "I really enjoyed reading the paper and I would strongly argue to accept it! It has a number of interesting findings and overall is very complete and sound, as well as well-written and well-structured. It is very clear that the paper presents a very large work effort, which I really appreciate and deem helpful for others. Their choice of datasets from different domains, as well as the decision to use both automatic and manual evaluation methods is reasonable and fair.\nThe presentation (choice of figures/graphs, comparisons etc.) is well-thought through and makes it easy to follow the description of results, which in general, they present very well-structured. I also find the findings in the appendix very interesting.", "reasons_to_reject": "I do not see any reasons to reject the paper.", "questions_to_authors": "1.) I could be wrong and might have overlooked but did you assigns some kind of overall \"quality score(s)\" in terms of simplicity / content preservation or TS quality in general in your manual evaluation? So sth. that would be directly comparable to BERTScore or SARI? I think that would be really interesting, both for the evaluation itself but of course also for assessing the usefulness of these automatic measures in comparison with human assessments. Maybe you could elaborate a bit on the description of Table 8 in the appendix.", "paper_topic_and_main_contributions": "The paper presents an extensive performance benchmark of 44 recent state-of-the-art Large Language Models on the task of Text Simplification (TS), which they call BLESS. The evaluation is based on three different datasets with original sentences/documents and their humanly written simplifications: ASSET (Wikipedia texts), MED-EASt (short medical texts), NEWSELA (professionally rewritten news article simplifications). The 44 models cover both open and closed sourced models, a range of different LLM architecture families, pretraining objectives and parameter sizes.\n\nThe models are tested applying few-shot in-context learning, so they are given a specific prompt describing the TS task together with 3 example complex--simplification pairs sampled from the datasets.\n\nThe authors evaluate all models on automatic evaluation metrics commonly used for TS, mostly SARI (measuring simplicity) and BERTScore (measuring content/meaning preservation), but also FKGL (also roughly measuring simplicity) and LENS (predictions by a model that was trained on human ratings for TS).\nAdditionally to these general metrics of simplification quality, they inspect the portion of operations that are performed by the models, using the Wagner-Fisher algorithm that detects insertion, replacement, deletion and keep operations.\n\nIn addition to the automatic evaluation (which are known to have weaknesses), the authors present a manual qualitative analysis of a selection of different model output simplifications (300 in total), where they rate the outputs considering e.g. if it is a valid simplification, mark common errors such as meaning change, ungrammatical text and hallucinations, marking the used operations (lexical simplification, deletion, sentence splitting, reordering, paraphrasing).\n\nThey further give some interesting secondary analyses, e.g. finding mostly a trade-off between simplicity and meaning preservation or showing high correlation of BERTScore and Levenshtein distance (showing that this measure might not be the best for TS evaluation, as simply copying the source text achieves high BERTScore).\n\nThrough these two evaluation approaches they achieve a number of interesting result, including:\n* the best LLMs outperform standard TS baselines\n* LLMs show a more diverse range of operations (deletions, paraphrases, lexical simplifications)\n* the closed-weight models in general perform better than the open sourced ones\n* in perliminary experiments they tried different versions of their TS prompt, noticing that highly structured example presentation works best\n* larger models tend to be better at preserving meaning, but fail sometimes in terms of simplification\n* models that were trained with instruction-tuning or RLHF (reinforcement learning with human feedback) have slightly better quality\n* overall, all models performed best on the ASSET dataset, and worst on the medical dataset, showing the difficulty of the latter domain\n* in general, the qualitative results are in-line with the automatic ones", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "* page 4, line 313: do you mean Davinci-002 instead of Davinci-003?\n* page 8, line 506: \"trade-off\""}, "extra_text": {}}, {"review_id": "iMCSOSOSJN", "reviewer": "Reviewer_McU4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the time taken to read our paper and for providing valuable feedback. We have responded to each point of concern below.\n\n1. The main contribution of our paper lies in the evaluation of 44 LLMs both quantitatively and qualitatively for text simplification, which is a task where these models have yet to be tested comprehensively. Explicitly, we contribute 1) an investigation of a few-shot approach to text simplification, 2) an extensive automatic evaluation of the system outputs, including the analysis of edit operations and 3) a qualitative analysis that include manual annotations and correlations between the selected evaluation metrics. Our findings also reveal GPT-3.5-Turbo attains state-of-the-art performance on ASSET and Med-EASI, which establishes important baselines for future work.\n\n2. We agree that investigating more domains and languages would indeed be valuable. The domain-coverage for this study was limited by the availability of sentence-level simplification data. Our work represents the first iteration of the BLESS benchmark, and we hope that by publishing in the EMNLP conference, we will be able to build a community of Text Simplification researchers that contribute to BLESS by evaluating in the same manner, widening its scope in terms of datasets, languages and models. Since most open-weight LLMs are predominantly trained on English language data, we decided to focus solely on English in this work. Extending our approach to other domains and languages would be straightforward given appropriate test datasets and models with exposure to non-English data, and would also make a valuable contribution, which we encourage future authors to look into (as done in concurrent work (Ryan et al., 2023: https://arxiv.org/pdf/2305.15678.pdf).\n\n3. We agree that the total number of test instances from the three datasets is a limiting factor. However, investigating 44 LLMs with 3 prompts and 3 random seeds on the 915 sentences amounted to roughly 360K model outputs. Despite the relatively small number of samples annotated as part of our qualitative analysis, we were still able to identify categorical problems that we can expect to find in larger sample sizes as well, even from the best performing open-weight models.\n\n4. As part of our qualitative evaluation, we were able to identify some common failure cases (inappropriate changes to the meaning, ungrammatical outputs, and hallucinations (mentioned on lines 387-394). Rather than identifying specific failure cases for each model, we opted to measure these for each of the main metrics in order to compensate for the metrics’ blindspots. Table 3 provides a detailed summary of these phenomena.\n\nRegarding the reproducibility of our study, we intend to release our code with full details of inference parameters used and will elaborate on these in the final paper. While Newsela is indeed only available under a restrictive license, both ASSET and Med-EASI are publicly available. For these datasets, we plan to release the model outputs, which we think will be useful for future work on evaluation metrics.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1.\tThe paper makes an extensive evaluation of text simplification tasks using LLMs. It evaluates a large set of 44 diverse LLMs, covering different sizes, architectures, training methods, and access levels (open vs closed weights). This provides a comprehensive view of current LLM abilities.\n\n2.\tThe authors combine automatic metrics, edit operations, and manual evaluation to thoroughly assess model performance from different angles. The qualitative analysis gives more nuanced insights.\n\n3.\tThe paper provides a standardized benchmark that can serve as a strong baseline for future work and allow measuring progress in LLM-based simplification.\n\n4.\tThe key results highlight the promise of LLMs for simplification but also identify limitations like inadequate domain generalization. \n\n5.\tThe paper indicates which models may be most suitable for real-world applications today based on factors like accessibility, accuracy, training approach, etc.\n", "reasons_to_reject": "Some potential weaknesses or limitations of this paper that could be improved on in future work:\n\n1. It seems that this paper only evaluates multiple large language models for text simplification tasks, and is an extension of previous work, which is somewhat lacking in innovation.\n\n2. The authors only evaluated datasets in three different domains and can try to extend the data to more domains and languages.\n3. The test splits used to amount to 915 sentences total, which limits statistical power. Larger test sets could make results more robust. And, the manual evaluation only covers 300 samples, with just 5 outputs analyzed per model. A more extensive qualitative analysis could reveal more subtleties.\n4. In addition to the quantitative analysis, the paper did not analyze and categorize the common failures. If some failure sample analysis can be carried out, it seems to be more able to show the need for improvement. Moreover, does not discuss broader implications or provide deeper insight into why models behave in certain ways.\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper presents BLESS, a comprehensive performance benchmark of the most recent state-of-the-art Large Language Models (LLMs) on the task of text simplification (TS).\n\nIt evaluates 44 different LLMs on the TS task, including both open-source and commercial models with a range of sizes and architectures. The models are evaluated in a few-shot setting on 3 test datasets from different domains (Wikipedia, news, medical) using automatic metrics and analysis of simplification operations.\n\nThe study finds that closed-weight commercial models perform better than open-source models overall. Instruction tuning is shown to help improve meaning preservation.\n\nThere is a trade-off observed between simplicity and meaning preservation that varies across models and domains. The largest models tend to balance this trade-off better.\n\nAnalysis of simplification operations shows commercial models use more diverse operations compared to open models that overly rely on deletion.\n", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents an extensive performance benchmark of 44 recent state-of-the-art Large Language Models on the task of Text Simplification (TS), which they call BLESS. The evaluation is based on three different datasets with original sentences/documents and their humanly written simplifications: ASSET (Wikipedia texts), MED-EASt (short medical texts), NEWSELA (professionally rewritten news article simplifications). The 44 models cover both open and closed sourced models, a range of different LLM architecture families, pretraining objectives and parameter sizes.


Reasons To Accept:
- The set of Large Language Models evaluated is extensive. The authors benchmarked 44 LLMs ranging in size from 60M to 176B parameters and include instruction-tuned models.

Reasons To Reject:
- Though the authors do their own small analysis in Section 5, the bulk of this paper uses existing metrics to evaluate existing models on existing datasets.

The paper is a very interesting and enjoyable read and to me it seems very sound, detailed, comprehensive (they use 44 models and test on 3 different domain, using both automatic as well as manual/qualitative evaluation) and is well thought-through and structured and clearly written. The sheer number of models and evaluation methods makes it very apparent that furthermore they must have put a lot of effort in all the experiments. Moreover, the task of Text Simplification is interesting and relevant, as is - of course - the inspection of what LLMs really can do and cannot do.
Decision: Accept-Main