Dancing Between Success and Failure: Edit-level Simplification Evaluation using SALSA
Authors: David Heineman
Yao Dou
Mounica Maddela
Wei Xu
Venue: EMNLP 2023 Main (2023)
Keywords: model evaluation
text simplification
fine-grained annotation
language model analysis
human evaluation
TL;DR: SALSA, an edit-based human annotation framework for text simplification, and 19K annotations on models including GPT-3.5.

Abstract: Large language models (e.g., GPT-4) are uniquely capable of producing highly rated text simplification, yet current human evaluation methods fail to provide a clear understanding of systems' specific strengths and weaknesses. To address this limitation, we introduce SALSA, an edit-based human annotation framework that enables holistic and fine-grained text simplification evaluation. We develop twenty one linguistically grounded edit types, covering the full spectrum of success and failure across dimensions of conceptual, syntactic and lexical simplicity. Using SALSA, we collect 19K edit annotations on 840 simplifications, revealing discrepancies in the distribution of simplification strategies performed by fine-tuned models, prompted LLMs and humans, and find GPT-3.5 performs more quality edits than humans, but still exhibits frequent errors. Using our fine-grained annotations, we develop LENS-SALSA, a reference-free automatic simplification metric, trained to predict sentence- and word-level quality simultaneously. Additionally, we introduce word-level quality estimation for simplification and report promising baseline results. Our data, new metric, and annotation toolkit are available at https://salsa-eval.com.

Dancing Between Success and Failure: Edit-level Simplification Evaluation using SALSA
David Heineman, Yao Dou, Mounica Maddela, Wei Xu
School of Interactive Computing
Georgia Institute of Technology
(david. heineman, douy, mmaddela hagpllalaaggtehhhhccmmm edu; wei. caaiicggttehhhh.. edu
Abstract
Complex Sentence:
Grocery inflation in the United Kingdom reaches record high of 17.1%, according to market research group Kantar Worldpanel, amid high levels of inflation, supply chain issues and high energy costs impacting the economy.
Large language models (e.g., GPT--) uniquely capable of producing highly rated text simplification, yet current human evaluation methods fail to provide clear understanding of systems' specific strengths and weaknesses. To address this limitation, we introduce SALSA, edit-based human annotation framework that enables holistic and fine- mne-grained text simplifi- cation evaluation. We develop twenty lin- guistically grounded edit types, covering the full spectrum of success and failure across di- mensions of conceptual, syntactic and lexical simplicity. Using SALSA, we collect 19K edit annotations on 840 simplifications, revealing discrepancies in the distribution of simplifica- tion strategies performed by fine-tuned models, prompted LLMs and humans, and find GPT-3.5 performs more quality edits than humans, but still exhibits frequent errors. Using our fine- grained annotations, develop a reference-fee free automatic simplification met- ric, trained to predict sentence- and word-level quality simultaneously. Additionally, we intro- duce word-level quality estimation for simpli- fication and report promising baseline results. Our data, metric, and annotation toolkit are available at https: //salsa-//aaaa wewaeeell..ooo.. com.
Simplificaton GPT-4:
The cost of groceries in the United Kingdom has increased to record 17.1%, says market research group Kantar Worldpanel. This is due to high inflation, supply chain problems, and expensive energy affecting the economy.
Successes
Failures
reaches
Grocery inflation
Bed Sulssstution gasarakkeeenn Weneshale meaning
incressed to
inteokass wil
The cost of groceries
has
reaches
Struture Change chunger Inste
amid
Increased
due to
Gead Deletion N0y renencol
high
Paraphroge
Sccording
Paraphoww
problems
high energy
This
Sentence Spiit
expensive energy
decwine moduality
g+ levels of
Seo Mettiin
Good Delcton
Impacting
Unnecesaary
Paraphrace
affecting
Figure 1: Simplification generated by GPT-4. Our edit- level SALSA reveals LLMs succeed across many edit types, but often fail to paraphrase and generalize.
tence into multiple shorter ones (Xu et al., 2012), sentence-level scoring remains difficult to interpret since it is not reflective of detailed information about the types of edits being performed.
Fine-grained human evaluation through span se- lection has been explored for machine translation (Lommel et al., 2014) and open-ended text gen- eration (Dou et al., 2022). Yet, these evaluation methods error-driven i.e., focusing solely evaluating failure which punishes creative and diverse generations with minor errors in favor of generic ones. Additionally, machine translation and open-ended generation tasks usually retain none of the input words, while text simplification must bal- ance the editing and preservation of words in the original input (Xu et al., 2016). We thus evalu- ate simplification quality the aggregation of edit successes and failures, depicted in Figure 1.
1 Introduction
Text simplification aims to improve text's read- ability or content accessibility while preserving its fundamental meaning (Stajner, 2021; Chandrasekar et al., 1996). Traditional human evaluation for text simplification often relies on individual, shallow sentence-level ratings (Sulem et al., 8c; Alva- Manchego et al., 2021), easily affected by the an- notator's is preference or bias. Maddela et al. (2023) recently proposes more reliable and consistent human evaluation method by ranking and rating multiple simplifications altogether. However, as text simplification involves performing series of transformations, or edits, such as paraphrasing, re- moving irrelevant details, or splitting a long sen-
We introduce SALSA Success and FAilure- driven Linguistic Simplification Annotation - an

---

edit-level human evaluation framework capturing a broad range of simplification transformations.
Orig ne Sararo on Pltaann Welder
northern Italy spawns medieval asierllder, pentiamtaa songa sung by waidering bands Flagellants,
SALSA is built on a comprehensive typology ($2) containing 21 quality and error edit types. Using SALSA, we develop an interactive interface and collect 19K edit unnotations of 840 simplifications written by eleven state-of-the-ar language models and two humans. With these annotations, we con- duct large-scale analysis of model and automatic metric performance, and further introduce the au- tomatic word-level quality estimation task for text simplification. Our main findings are as follows:
Samp and Sertinse 34 mon or Veces Watter)
Goleelilod was from unrest Northorn Italy through songs sung by travaling bands Flagellants.
Selecting Edits
Seieccthe Campgey
Do e50c
Du DubuPador
Rasinder
Ralase foet sour disphited Aostoree
Seaccd
CANCRI
RAVP
AAROTAATNG AS EDIT
Fel: aulta (pomms will Poodte
Few-shot GPT-3.5 far surpasses existing models, particularly in making syntax and content edits. However, its simplifications are not aligned to the types of operations performed by human. ($4)
Compured to original cheee. phoue FApeesn::
Information Change
Baxe:
SeReCT the proct parspriane 6 the sor replity
Edit Type
Thap
No
Some fine-tuned models such as the MUSS (Mar- tin et al., 2022) produce more diverse edits than GPT-3.5, yet suffer from incredibly high errors,
Hade fhe offewey
sesinme))
Rating Efficayy
inredace
CANCEL
SAVE
while others (T5, Raffel et al., 2020) learn to minimize loss by making very few changes. (54) Open-source instruction fine-tuned models such as Alpaca (Taori et al., 2023) and Vicuna (Chiang et al., 2023) perform a similar number of edits as GPT-3.5, but at a cost of more conceptual errors due to the inherent limits of model imitation. ($4) Fine-tuned on SALSA annotations, our reference- free metric, LLESSSALSA captures the subtleties of specific simplification approaches beyond ex- isting automatic evaluation metrics. ($5)
Figure 2: The SALSA annotation process consists of (1) selecting edits, (2) identifying information change, (3) classifying edit type and (4) rating efficacy/severity.
(Du et al., 2022) collecting fine- fine-grained human feedback (Wu al., 2023)'. In the following, we describe each step of the annotation process.
2.1 Edit Selection
Annotation begins with edit selection, where an- notators identify the edits performed by the sim- plification and select the corresponding spans for each edit. We define six types of edit operations: single-operation insertion, deletion, substitution, word-Iclause-reordee and multi-operation sentence split and structure changes. An insertion deletion edit exclusively modifies content, while substi- tution either modifies or paraphrases content. Re- order, split, or structure edits perform context- free syntax transformation. As split and structure edits are multi-operation (i.e., require a combina- tion of single operations), they are defined by a set of underlying single-operation constituent edits. For example, this structure change from passive to active voice made by zero-shot GPT-3.5 involves multiple constituent edits:
Leveraging our data, we present the automatic word-level quality estimation task for text simpli- fication and establish several baseline approaches for future modeling efforts. ($6)
Our results demonstrate that SALSA provides an interpretable and exhaustive evaluation of text simplification.
2 SALSA Framework
We introduce SALSA, an edit-based human evalu- ation framework for text simplification. SALSA is defined by a typology of 21 linguiticalll-grounded edit types with the aim of capturing both successes and failures (i.e., quality changes and errors, see Figure 1). The annotation methodology of SALSA is structured decision tree and implemented via an easy-to-use interface, illustrated in Figure 2. Our interface designed with Thresh (Heineman et al., 2023), and we release our configuration to encourage adaptation to other text rewriting tasks
EXAMPLE
Zero-shot GPT-3.5
On 14 November, an interview with journalist Piers Morgan was published. where Ronaldo said... ...
On November, Piers Morgan interviewed Ronaldo, who expressed...
hhttps:/ttheeh.ttoe

---

2.2 Categorizing by Information Change
Selecting EAir
Each selected edit is then labeled with its impact the underlying sentence information: less, same, more or different information. Given the type of op- eration and change to information, we subsequently organize each edit into three linguistic families as defined by Siddharthan (2014);
Deletion
Rearder
Structure
Insertion
Substitution
Split
Information
Change
PJ Same Info.
More Information Information
Less
Same Info.
conceptual edit www.ereeeeettoooo www.eeeeettteeee....o
Lexical edits perform simple changes in "word- ing". This includes paraphrasing (i.e., substitution that keeps the same information) and inconsequen- tial trivial changes (e.g., inserting 'the').
EAT Type
sentence split word reorder reorder structure change
elaboration generalization trivial change
paraphrase trivial change
trivial change
quality adi
repotition contradiction factual error irrelevant
bad deletion coreference
sentence split bad word reorder bad component reorder bad structure change
grammar error complex wording Info, rewrite
Syntax edits capture transformations to the distri- bution of information, rather than substance. A split converts a candidate sentence to two sen- tences, a re-order edit re-arranges clauses or word- ing within clause, and a structural edit modifies the voice, tense or clausal structure. Examples of structural edit sub-types are in Appendix B.
emrrr
Rating Effieary
-3
-2
+1
+2
+3
Figure 3: The multi-stage SALSA edit evaluation frame- work. Spans are classified into twenty one success and failure types (trivial change counts as one type) using the interface shown in Figure 2.
Conceptual edits modify underlying ideas con- veyed by the text. A conceptual edit requires elab- oration to add clarifying information generaliza- tion to delete uneccssary/complicttee ideas.
Sentence splitting or reordering information may clarify a sequence of events:
EXAMPLE (component reorder)
CharGT
Poland announces the closure of a major border cross- ing with Belarus "until further notice" amid height- ened tensions hetween the two countries.
2.3 Edit Type Classification
After being categorized into lexical, syntax, or con- ceptual edit families, we further classify each edit operation into 21 fine-grained success (quality), failure (error), or trivial edit types as listed in Fig- ure 3. Successful edits simplify through diverse approaches, from paraphrasing complex spans, gen- eralization of unnecessary information, or elabora- tion to add clarity and background context. E.g.,
Poland has closed a big border crossing with Belarus due to increased tensions between the two countries. The closure will remain in effect until further notice.
Failure edits include any ablation from minor readability issues to hallucinations or deletions sentence meaning. In the following example, the coreference error captures the deleted reference be- tween the 'ICJ' and 'US' acronyms to their original definitions, useful contextual information:
Vicuna 7B can be fitted exponentially decaying curve. ... can be represented by a curve that gets smaller and smaller over time.
EXAMPLE (elaboration)
ChaGGT The International Court of Justice (ICJ) rules that the United States violated its ...
EXAMPLE (coreference error)
Often small edits, particularly to syntactic structure, can improve clarity, such as this addition of a clear subject-verb structure through the inclusion of the relative pronoun 'who':
The ICJ said the US broke ...
And often multiple edits overlap, such as this infor- mation rewrite which successfully adds clarity via reordering, but botches the author's sarcasm:
EXAMPLE (structure change) Paltrow in claims he the one crashing rather than the other way around.
GPT-4
EXAMPLE (information rewrite)
Alpaca 7B
justifics a runtime nearing 3 hours (with post- credits scene, no less), and it already opened over $100 million worldwide.
Paltrow says he the who crashed. not her.
Or this conversion of the participial phrase to relative clause to help explain significance:
takes up almost 3 hours of the movie. The movic opened to over $100 million worldwide, A post-cred-
scene completes the story.
EXAMPLE (structure change)
CharGP for the first time since 2006, ending their 17-year playoff drought...
We also separately ask annotators identify if the edit contains a grammar error. Appendix A provides an exhaustive description and examples for each edit type.
for the first time 2006, which means they have ended their 17-year playoff drought.

---

2.4 Rating Edit Efficacy/ Severity
Ralit
Sub-type
Kripp- 3 Agree Agree %
As each edit has a varying degree of impact on overall simplification quality, we finally ask anno- tators to rate the efficacy of quality edits or severity of error edits. We define three levels: - minor, 2 somewhat, and 3 major. Examples of each severity level are included in Appendix A.3.
Insertion
Information
40%
Deletion
Less Information
0.75
42%
65%
Substitution
Information
0.15
11%
Information
26%8
Reorder
Wond-level
0.12
0%
13%
Component
0.41
11%
38%
Split
Sentence Split
0.66
32%
55%
Structure
Structure
0.25
5%
25%
Subsiitation
Same Infirmaaton
0.53
21%
51%
3 Data Collection
Edit selection inter-annotator agreement sured per token. As Krippendorff's a (2018) includes unlabeled tokens, we also report the percentage an- notated tokens where at least 2 and annotators agree.
Table I:
We describe our use of SALSA to collect 19K edit annotations covering 1 11.6K spans on 840 model- generated and human-written simplifications.
3.1 Simplification Data
sisting of broad explanations of simplification con- cepts, over 100 examples covering each of the 21 SALSA edit types and interactive exercises, com- pleted two rounds of onboarding annotations and were provided continuous feedback by the authors. To concretely measure agreement for each stage of the SALSA framework, we collect annotations in three stages: (1) we have three annotators select edits, (2) fourth annotator adjudicates the edits into a single selection and (3) the initial three anno- tators classify and rate the adjudicated edits. Figure 2 illustrates our annotation interface, with further screenshots of our tutorial included in Appendix G.
Data collection is performed on an extended ver- sion of SIMPEVAL2022 (Maddela et al., 2023), in- cluding a train set covering state-of-the-ar simplifi- cation systems and held-out test set of recent LLMss We include a full description of each system in Ap- pendix C.1.
SALSA Train. We first extend the 360 simplifica- tions from SIMPEVAL2222 to 700 simplifications based on 100 complex sentences from Wikipedia articles dated between Oct 2022 and Dec 2022. The complex sentences are unseen during the training of the LLMs and were selected to be intention- ally difficult (avg. length of 37.3 words) enable an evaluation of the models' full capabilities in performing diverse simplification edits. Simpli- fications are generated by five models including fine-tuned T5-3B and T5-11B (Raffel et al., 2020), MUSS (Martin et al., 2022), controllable BART- large model trained with unsupervised, mined para- phrases, zero- and few-shot GPT-3.5 (Ouyang et al., 2022), and two human- wwrttenn references. For mod- eling experiments in $5 and $6, we divide the initial 700 simplifications by the complex sentence with a 70/30% train/dev split.
3.3 Inter-Annotator Agreement
We calculate edit selection agrecment (i.e. agree- ment prior adjudication) by each token, with Table reporting agreement per edit, further bro- ken down by their type of information change. We observe edit agrecment is highly dependent on the edit type and type of information change being performed. High agreements are seen for deletion (a=0.75), paraphrase (substitution with the same information, a=0.53), and sentence splits (a=0.66). Substitution that introduces more information, how- ever, exhibits lower agreement (a=0.15). due the subjectivity among annotators on determining whether new tokens contain 'novel' information. as was often mixed up with insertion. Reorder- ing (o=0.12) and structure edits (a=0.25) also re- port lower agreements. We fully explore the phe- nomenon of annotator disagreement in Appendix C.2, and find overlapping syntactic and content edits often have multiple correct interpretations, leading an inherent disagreement. Additionally, we find our % rates for annotator agreement are similar to fine- -oee-grrnndd evaluation frameworks other text generation tasks (Dou et al., 2022).
SALSA Test. We further gather 20 more complex sentences from Wikipedia articles published in Mar 2023 and generate 140 simplifications using re- cent LLMs including GPT-3.5, ChatGP, GPT-4, Alpaca-7B 7B (Touvron et al., 2023) and Vicuna-7B (Chiang et al., 2023), along with T5-3B and T5- 11B fine-tuned with control tokens.
3.2 Annotation
As crowd-sourced annotators have shown to have inconsistent quality (Shmueli al., 2021), we hire 6 undergraduate students from a US university. An- notators were trained with an in-depth tutorial con-

---

Conceptual Quality
Syntax Quality
Lexical Quality
2.5-
20
Spli
1.5
Struture
1.5
Component
Poroptrox
Eicboration
level Recrdar
1.0
Word-leel
0.5
0.5
Rearder
0.
0.
0.0
15 28
15 178
Zero dhat GP7-3.3
Mis5
Pew GMT-3.5
Human
15 50
Zere CPI-33 tha GPI-3.
118
MUSS
Homon
15 118
Zero CPI.3
MLSS
Fox dut GPL33.
Homee
15
Figure 4: Successful edits per-model, organized by edit type. MUSS outperforms fine-tuned T5 but fails to capture more complex simplification techniques. Compared to GPT-3.3, human written simplifications have more generalization similar distribution of syntax edits, and slightly less paraphrasing
Conceptual Error
Syntax Errr
Lexical Error
1.0
0.5
Ind
Deletion
Coreforance
Bod Spit
0.6
Illomationn Beutte
0.6
0.
Erirr
Factual
Irelavant
Costrodiction
0.2
Word-leve
Component
0.2
Orammar
Frarr
Warding
Complex
Eepestion
Zero dhat G Few sha GPT.35
178
AUSS
Humon
Zero GPT-3 3.5 Few dut GP5.3.5
HUSS
Human
75 118
Zerosshtt G.S.55 few dhst GFT.1.5
Miss
Humon
15
Figure 5: Failure edits per-model, organized by edit type. Compared to humans, both GPT-3.5 setups make more syntax and lexical errors. Although humans perform bad deletion eπrors at higher frequency than GPT-3.5, this is reflective of the inherent ambiguity in judging the relevancy of the deleted content.
4 Key Analysis
often occur among many good edits.
We use SALSA to evaluate state-of-the-art sim- plification by collecting annotations on our ex- tended version of the SIMPEVAL corpus (Maddela al., 2023), which includes fine-tuned, LLM- and simplifications. Our resulting data collection includes 19K edit annotations across 840 simplifications.
Language models elaborate, while humans gen- eralize (Fig. 4). When simplifying content, all models (excluding T5) tend elaborate a higher ratio than humans, for example, GPT-3.5 attempts to insert content 17% more often. As LLMs have shown to encode world knowledge in their parame- ters (Petroni et al., 2019; Brown et al., 2020), GPT- 3.5 elaboration is far more effective than MUSS, for example:
We present our primary results in Figures 4, 5, and 6. Figures 4 and 5 illustrate the frequency of quality and error edit types. As edits vary in length, we calculate edit coverage: the length of each edit in proportion to the total length of the simplification and report the average edit coverage for different efficacy and severity ratings in 6, showing view edit ratings adjusted for length. Additionally, we include Figure 7, which compares simplifications generated by recent instruction fine-tuned language models. The following are our key findings:
EXAMPLE
Pew-shot GPT-3.5
After defeating PSD candidate Viorica Däncilä by a landslide in 2019, his second term..
In 2019, Klaus Johannis defeated PSD candidate Vior- ica Däncilä large margin. His second term..
GPT-3.5 writes quality edits at a higher fre- quency than humans, but human edits are longer and more effective (Fig. 4, 6). Both zero- shot and few-shot GPT-3.5 produce larger number of edits, but human edits are more substantial, as demonstrated by the higher edit coverage across all efficacy levels, particularly for syntax and lexical edits. Human simplification typically deletes, para- phrases, or reorders entire clauses, while GPT-3.5 often edits single modifiers or words.
Models primarily write good edits, but still trail humans (Fig. 4, 5). We observe that 16% of model- generated edits are errors, with the best rest-performing model, shw- GPT-3.5, producing errors in only 9% of edits. We find this still trails human simplifi- cations, which have an error rate of 6%. MUSS and GPT-3.5 have median count of 1 error per simpli- fication and 63% of their simplifications contain at least one error, showing these errors are not con- centrated in a few 'bad' simplifications but instead
Fine-tuned T5-3B and T5-11B generate conser- vative simplifications (Fig. 4, 5, 6). Compared all other systems, both T5 models make minimal changes in terms of frequency and edit coverage,

---

More Information
Less Informanton
Some Inform onton
Reorder
Sentence Spitt
Structure
All
15
MUSS
Zero-let GPT:3.5
GPT-3.5
-9 -2-1 +011-249
10-1
-0-1-2-2
-3-2- 20
10-1
+2
-3-24 +2+3
Figure 6: Edit coverage of efficacy (+) and severity (-) ratings for each model, separated by simplification approach, with edit coverage defined (len(ec) ((een)))((ee((C) Ten(S)) (see $A.4). Overall, humans make the longest quality edits and most infrequent error edits. We report the distribution of each edit rating in Figure 14.
while still exhibiting high rates of error. This is likely due their training data, Wiki-Auto (Jiang et al., 2020), containing shorter sentences, usually requiring simpler simplification techniques, mak- ing it difficult for models to generalize on longer and more complex sentences. Later in Appendix D, we show using control tokens (Martin et al., 2020) during training, as done by MUSS, can improve diversity but at the expense of increasing deletion and hallucination errors.
lion constitutes 35% of error edits made by hu- mans, compared 8% by few-shot GPT-3.5. The anomaly of the bad deletion errors reveals an inher- ent subjectivity in assessing deletion:
EXAMPLE
Human written
Unlike Samuel Fuller removed...
film adaptation, which director
Ualike the first film adaptation, Samuel Fuller moved...
In this example, some annotators marked the edit as bad deletion while others consider it appropriate. As the sentence discusses a book adaptation into film, the description of 'Samuel Fuller' is helpful depending
Split edits are straightforward, Structure edits far more complex (Fig. 4, 5). Surprisingly, sentence splitting shown to be the easiest edit for all models accomplish, with a similar number made by MUSS, GPT-3.5, and humans, with even the conservative TS models making a comparable number of split edits. However, structure change and re-ordering edits are rarely seen in tne-tuned models. We speculate this may be attributed to (i) these types of edits are infrequent in the training data and (ii) GPT-3.5 has unique ability to per- form complicated syntax rewriting, echo with the findings in abstractive summarization (Goyal et al., 2022). Despite GPT-3.5's improvement, the struc- ture error rate demonstrates it has not yet reached human-level ability. Additionally, we observe zero- shot GPT-3.5 produces structure errors (see below example) at a 19% rate higher than few-shot.
the reader, which underscores the need for adaptive levels of simplification to accom- modate each reader's s needs.
Paraphrasing is a erucial, but tricky mechanism (Fig. 4, 5). MUSS, GPT-3.5, and humans all para- phrase in at least 75% of sentences. Despite low performance in conceptual and syntactic simpli- fication, MUSS paraphrases at a human-like rate likely due to its training on over one million para- phrase sentence pairs mined from web crawl data. Although zero-/few-shot GPT-3.5 paraphrases at a higher rate than humans, these edits are often are unnecessary. For instance:
Few-shot GPT-3.5 The club said on social media that customers subdued the gunman...
EXAMPLE
EXAMPLE
Zeno-shot GPT-3.5
The club reporied on social media that customers were able...
The sentence included a fine of $400...
You will receive a fine of $400...
Open-source LLMs are approaching GPT-3.5
We find human simplifications are more conserva- tive with re-ordering than models, yet attempts to simplify with re-ordering often appear arbitrary:
simplifications, or are they (Fig. 7)? Given re- cent attention to ChatGPT (OpenAI, 2022), GPT-4 (OpenAI, 2023), and the emergence of instruction fine-tuning smaller language models on outputs from proprietary LLMs, we perform supplemen- tary evaluation on these systems. The open-source Alpaca (Taori et al., 2023) and Vicuna (Chiang et al., 2023) appear to perform a similar number of
EXAMPLE
Human written
On November 2022, the British Sccretary... On November 3rd, 2022, the British Secretary...
Humans appear to produce bad deletion errors, but these are often subjective (Fig. 5). Bad dele-

---

Quality Edits
Error Edits
BERTSCORE
COMET-MQN
LENS-SALSA
Geeero ionnce
BLEU
SARI
LENS
Qualit Edite Sant.
Bror Edits Sart.
Good
1.5
Lesical
-0.167
0.126
0.025
0.120
0.407
0.443
6o
1.0
Good
Syntax
0.013
0.204
0.147
0.122
0.306
0.356
Conceptual
0.043
0.149
0.097
0.038
0.144
0.202
0.5
Faonder
Goed hod
Losical
-0.147
-0.026
-0.093
-0.068
-0.041
0,054
Synlax
-0.104
-0.013
-0.043
-0.017
0.019
0.086
Aloaco 78
Vison
OPT.35
ChaGeT
GPT.4
Alpaeo 78
0.0
Vicuno
GP CPT:355
Chergyy
GFF-4
Bron
Ofier
Conceptual
All Eroe
.0.21
0.047
0.150
0.067
0.279
0.117
0.127
0,228
0.207
0.161
0. 0.107
0.169
All Quality
-0.095
0.179
0.027
0.974
0.336
0.459
All Edits
-0.116
0.170
0.086
0.092
0.334
0.446
Figure 7: Success and failure edits on simplifications by five recent instruction fine-tuned language models.
Table 2: Pearson correlation between automatic metrics
and SALSA sub-scores ($A.4) on the SALSA test set. All refereece-based metries human-written ref- crences. Best; Second Best.
quality and error edits to GPT-3.5. However, these systems tend to write far more bad elaboration er- rors such as factual errors or contradictions:
Existing Automatic Metries. We consider five automatic metrics: BLEU (Papineni et al., 2002), SARI (Xu et al., 2016), the most widely-used text simplification metric, BERTSCORE (Zhang et al., 2020), COMET-MQM, machine translation metric (Rei et al., 2020) trained on MQM ratings (Freitag et al., 2021), and LENS (Maddela al., 2023), a recently proposed text simplification metric fine- hu-
EXAMPLE
Alpaca 7B
controversial "angel tax" provision seeking to capture some of the income entering the country from foreign investors funding India's 'ss start-ups.
controversial "angel tax" provision, which is aimed at stopping foreign investors from funneling money into India's startups.
This behavior suggests open-source instruction fine- tuned models mimic the style of their larger coun- terparts, but not their knowledge, a phenomenon observed by Gudibande et al. (2023). GPT-4 ex- hibits the best performance by making fewer con- tent errors while producing high number of qual- ity edits, but still exhibits errors particularly when paraphrasing individual spans without considering the broader sentence meaning:
tuned on SIMPEVAL that contains man ratings of simplifications from 24 systems.
LENS-SALSA. The automatic simplification met- rics mentioned above require numan-writterl refer- ences, which may not be available in every eval- uation setting. To this end, we introduce LENS- SALSA, a reference-free simplification metric en- abled by edit-level information. Based the COMETKIWI machine translation metric design (Rei al., 2022), we first pre-train LENS-SALSA on the sentence-level human ratings from SIMPE- VAL using UniTE (Wan et al., 2022), a multi-tat task learning method. Specifically, the metric is trained on the same score but from three input formats: Simp:Ref, Somplex and Somp:Com where ":"" denotes concatenation. Then, we fine- tune LENS-SALSA on SAL.SA annotations using dual-objective to predict both the sentence-level score (calculated by LENS) and a word-level qual- ity score wi E [-3,3], 3). corresponding to the ef- ficacy or severity rating ($2.4) of each word w; in the complex and simplified sentences. We use RoBERTa- large as the base model for LENS- SALSA, and 490, 210, and 140 sentence pairs for train, validation, and test, respectively. Implemen- tation details are provided in Appendix F.2.
GPT-4 Grocery intlation in the United Kingdom reaches record high of 17.1%....
EXAMPLE
The cost of groceries in the United Kingdom has increased to a record 17.1%..
While GPT-4 successfully paraphrases inflation by relating to cost, it fails to recognize the sentence is discussing inflation rate, rather than exact prices. We include further analysis, discussion, and dataset statistics in Appendix D.
5 Evaluating Metric Edit Sensitivity
While automatic metrics are traditionally evaluated using correlation with sentence-level, Likert scale human ratings on dimensions of adequacy, fluency and simplicity, this fails to understand the ability of automatic metrics to capture the subtleties of lexi- cal, syntactic, and conceptual simplification. With our SALSA annotations, we study how well cur- rent automatic metrics capture these distinct sim- plification approaches. Additionally, we introduce LENS-S LENS-SALSA, a reference- free metric fine-ture tuned on SALSA annotations.
Results. As fine-grained MQM annotations in ma- chine translation are considered a gold-standard in metric evaluation (Freitag et al., 2021), we adapt their method (detailed in $A.4) to collapse edit-

---

level ratings to single score, and calculate sub- scores by only considering certain edit types. Table 2 reports the Pearson correlation between metric scores and human sub-scores across each SALSA di- mension. LENS-SALS A achieves the highest corre- lation in nearly all edit approaches, showing its ca- pability to capture all forms of simplification. Over- all, only LENS and LENS-SA LENS-SALSA obtain substantial correlation with the overall human SALSS scores (0.33 and 0.45 respectively), while other metrics have spurious and even negative correlations with human judgments. Interestingly, COMET-MQM, intended for machine translation, performs better than BLEU and BERTScore, which further under- lines the value of span-based ratings for trained metrics. Despite strong performance, we find LENS mainly evaluates lexical and syntactic edits, rather than conceptual ones, which may be attributed to its training data consisting of shorter, paraphrase- based simplifications. Lastly, all metrics have sub- stantially higher correlation with quality than error edits. We posit this is primarily due to the sparsity and wide range of errors exhibited in the genera- tions of current nigh-performing systems.
Method
Quality
Error
Ok
Average
End-to-end
Tag
67.00
28.24
92.88
62.71
Tag-ML
70.73
30.06
93,09
64,62
Two-staaee (use worl oligner get informatioon
Tag-EI
69.09
30.37
93.04
64.17
Ec-Sep
64, 87
36.15
91.56
64.20
Ec-One
68. 77
39,50
91.91
66.73
Oracte (Ec-One)
88.
69.44
98.35
85,47
Table 3: Word-level Fl scores of different methods
SALSA test set. Oracle uses annotated edit information.
Methods. We propose two approaches: End-to- end, where a single model labels each token di- rectly; and Two-stage, where a word aligner first identifies edits, then the model labels each token us- ing the identified edit information. For end-to-end, implement the following two methods:
Tagging (Tag) is native sequence tagging model with a classification head.
Tagging with Multi fasti-taskkaa Loss (Tag-ML) is sim- ilar to the tagging method except trained with a multi-task loss function: L - Lagg + Lec- Lee is an additional objective that classifies each token into none, deletion, substitutiion, insertion.
For two-stage methods, we first apply QA- based word aligner (Nagata et al., 2020) to the sentence pair and use a set of rules to convert word alignments to edits: consecutive non-aligned words in the original sentence are labeled as a deletion edit; consecutive non-aligns -ligned words in the simpli- fied sentence are labeled as an insertion edit; and aligned words or spans that differ are labeled as substitution edit. Here are three two-stage methods: Tagging with Edit Information (Tag-EI) is a se- quence tagging model with a classification head that takes the concatenation of the hidden states of both edit type and token as the input. The hidden states of the edit type are obtained via a linear layer. Edit Classification with Separate Classifiers (Ec- Sep) contains classifier for each of the threc edit operations. Each classifier is an encoder model with feedforward neural network (FNN). The inputs to these FNNs are the hidden states of the [CLS] token and the max max-poole -poledd tokens from the edit spans (i.e.. for substitution edit, one from the original span, and one from the simplified span). Edit Classification with One Classifier (Ec-One) is one classifier with three FNNs mentioned above. The difference is the encoder is trained collectively.
6 Word-Level Quality Estimation
Word-level quality estimation (QE) is the task of predicting the quality of each token in a genera- tion, and has substantial downstream application to evaluating and refining text simplification. Despite word-level QE being a well understood task in ma- chine translation (Basu et al., 2018; Zerva et al., 2022), it has not yet been studied for text simpli- fication due to a lack of appropriately annotated data. In this section, we use SALSA annotations to demonstrate baseline approaches and highlight potential for future work.
Task. We define word-level simplification QE as classifying each token in the complex and simpli- fied sentences as quality, error, or ok. To adapt SALSA for the QE task, we label each token by the average efficacy/severity rating of its associ- ated edit: <0 as error, = as ok, and > 0 as quality. Words that are not part of any edits de- fault to the ok label. We deconstruct split and structure edits into their constituent edits, only la- bel the simplified spans for substitution edits, and exclude reorder edits due to their low frequency. The final label counts for our train, validation, test splits are: 6.8K/1.8K/27K, 2.7K//27/1 IK, and 1.7K/484/6.9 9K for quality/crror/ok respectively.
All methods (including the word aligner) use RoBERTa-large Further implementation details and results are included in Appendix F.

---

Results. Table 3 shows the test set performance for each label. Among the end-to-end methods, train- ing with multi-task loss results improvement on all three label FI scores, achieving the second-best average F1 score overall. We find edit classifica- tion approaches detect error tokens more accurately than tagging approaches. Within edit classification methods, using one classifier outperforms multiple ones due to the benefit of joint encoder training. Overall, the edit classification with one classifier method performs the best with a gain of over 11 points on error and a 4-point increase in average F1, compared to the base tagging model.
et al. (2021a) propose framework for evaluating success and failure by answering series of check- list items, with sentences given a capability score based on the number of requirements fulfilled. Ya- maguchi et al. (2023) annotates simplifications of carlier models such as DRESS (Zhang and Lapata, 2017) and SUC (Sun et al., 2020) using taxonomy of 62 emror categories, but do not analyze the SOTA, MUSS, or LLMs. Stodden and Kallmeyer (2022) proposes an interactive linguistic inspection inter- face, but this interface is not designed for human evaluation of model outputs and does not provide ratings for measuring performance.
Fine-grained Human Evaluation. Human eval- uation performed on a span-level has been pre- viously proposed for a variety of NLP tasks. translation, the Multidimensional Quality Metrics (MQM) (Lommel et al., 2014), categorizes error into accuracy and fluency sub-types and is later extended by Freitag al. (2021) to weight errors by severity and combine into a single quality score. Dou al. (2022) proposes SCARECROW to cap- ture errors appearing open-ended text generation. However, as these based evaluation schemes exclusively annotate error, they encourage generic outputs and punish interesting or diverse genera- tions. For summarization, the FRANK typology (Pagnoni et al., 2021) aggregates errors into broader categories to benchmark metrics that measure fac- tuality. Inspired by FRANK, Devaraj et al. (2022) introduces framework to evaluate factuality for text simplification.
7
Related Work
Model Evaluation. Simplification work broadly agrees some typology of simplification operations exists (Siddharthan, 2014), starting with early rule- based systems which explicitly defined specifie syn- tax operations (Dras, 1999). Past work has experi- mented with designing models to control the extent of each operation by using a pipeline to perform simplification operations independently (Maddela et al., 2021; Raffel et al., 2020), predicting edit operations (Dong et al., 2019) or augmenting fine- tuned models with learned control tokens (Martin et al., 2020, 2022). However, evaluation only con- siders sentence in its entirety rather than rating individual operations, either by automatic metrics (Kriz et al., 2020), shown to be an inadequate rep- resentation of quality (Alva-Manchego et al., 2021; Sulem et al., 2018 or by surface-level Likert rat- ings, typically asking crowd-sourced annotators to rate on scales of fluency, adequacy, and sim- plicity. These scores are difficult to interpret and capture no detail into the type of simplification be- ing written (Briakou et al., 2021; Hashimoto et al., 2019). Additionally, despite current systems' of- ten producing simplification errors (Choshen and Abend, 2018), annotating error has primarily been performed through inspection, and has not been incorporated into human or automatic evaluation (Gooding, 2022).
Conclusion
In this work, introduce SALSA, novel edit- based evaluation framework incorporating error and quality evaluation, and dimensions of lexical, syntax and conceptual simplification and demon- strate SALSA benefits in granularity, accuracy, and consistency. We employ SALSA to collect 19K edit annotation dataset and analyze the strengths and limitations of fine-tuned models, prompted LLMs, and human simplifications. Finally, we use SALSA annotations to develop a reference-free automatic metric for text simplification and demon- strate strong baselines for word-level quality esti- mation, showing promising avenues for the devel- opment of fine- me-ggrained human evaluation.
Linguistic Inspection. Manual inspection attempts to understand the behavior of simplification mod- cls datasets, characterized by detailed typolo- gies and often conducted by authors domain experts. Cardon et al. (2022) performs detailed inspection of the ASSET simplification test cor- pus (Alva-Me Manchheo et al., 2020a) to study the be- havior of automatic metrics and

---

Limitations
Acknowledgements
We thank Tarek Naous, Nghia T. Lee Fan Bai, and Yang Chen for their helpful feedback on this work. We also thank Marcus Ma, Rachel Choi, Vishnesh Ramanathan, Elizabeth Liu, Govind Ramesh, Ayush Panda, Anton Lavrouk, Vinayak Athavale, and Kelly Smith for their help with human annota- tion. This research is supported in part by the NSF awards (IS-214449) and IIS-21 12633, ODNI and IARPA via the HIATUS program (contract 2022- 22072200004). The views and conclusions con- tained herein are those of the authors and should not be interpreted as necessarily representing the of- ficial policies, either expressed or implied, of NSF, ODNI, IARPA, or the U.S. Government. The U.S. Government is authorized to reproduce and dis- tribute reprints for governmental purposes notwith- standing any copyright annotation therein.
Our annotation only represents single use case of text simplification and encourage tension of SALSA to domain-specific simplifica- tion, such as medical (Joseph et al., 2023), legal (Garimella et al., 2022), or multi-lingual text (Ryan et al., 2023), and annotations by groups of spe- cific downstream users (Stajner, 2021). The LENS- SALSA reference-free free metric is trained exclusively on Wikipedia simplification, and we do not con- sider its cross-domail generalization or its ability capture the simplification need to specific target communities. Additionally, while we demonstrate promising results on sentence-level evaluation, sim- plification is often a document-level task (Laban et al., 2021; Sun et al., 2021). Incorporating higher- level operations such as sentence fusion, paragraph compression, and reordering would require an ex- tension to SALSA and presents unique analytical challenges. Finally, detailed human evaluation in- herently requires greater resources to produce high granularity of annotations. While we show this process can be streamlined with a robust anno- tator training, SALSA requires a similar amount of resources as widely used fine-grained evaluation in other tasks such as MQM (Lommel et al., 2014) or FRANK (Pagnoni et al., 2021).
References
Fernando Alva-Manchego, Louis Martin, Antoine Bor- des. Carolina Scarton, Benoît Sagot, and Lucia Spe- cia. 2020a. ASSET: A dataset for tuning and evalua- tion of sentence simplification models with multiple rewriting transformations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguisties, pages
Online. Association for Computational Linguisties.
Ethics Statement
Fernando Alva-Manchego, Carolina Scarton, and Lucia Specia. 2020b. Data-driven sentence simplification: Survey and benchmark. Computational Linguistics.
Our annotations were performed using the SIMPE- VAL2022 corpus, originally collected from publicly available Wikipedia articles (Maddela et al., 2023) and we further extend the dataset with complex sentences collecting using the same methodology from publicly available Wikipedia articles. As dis- cussed in $3.2, we perform data collection with in-house annotators from a US university. Anno- tators were all native English speakers and paid $15-$18/hour. We took care manually review all data prior annotation as to exclude any trig- gering or sensitive material from our annotation data. Annotators were informed that any data they felt uncomfortable with was not required to anno- tate. Our interface was built using the open-source Vue.js² library, and training of our added T5-1 IB system was implemented using the open-source Hugging Face Transformers library.
46(1):135-187.
Fernando Alva-Manchego, Carolina Scarton, and Lucia Specia. 2021. The (un)suitability of automatic evalu- ation metrics for text simplification. Computational Linguistics, 47(())8611-)))
Petra Barancikova and Ondřej Bojar. 2020. COSTRA 1.0: dataset of complex sentence transformations. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 3535-3541, Mar- seille, France. European Language Resources Asso- ciation.
Chandrayee Basu, Rosni Vasu, Michihiro Yasunaga, and Qian Yang. 2023. Med-easi: Finely annotated dataset and models for controllable simplification of medical texts. arXiv preprint 002.02230..09155.
Prasenjit Basu, Santanu Pal, and Sudip Kumar Naskar. 2018. Keep it or not: Word level quality estimation for post-editing. In Proceedings of the Third Confer- ence on Machine Translation: Shared Task Papers, pages 759-764, Belgium, Brussels. Association for Computational Linguisties.
Phttps://vuejs.om hattps://huggingfacecco

---

Eleftheria Briakou, Sweta Agrawal, Ke Zhang, Joel Tetreault, and Marine Carpuat. 2021. A review of human evaluation for style transfer. In Proceedings of the 1st Workshop on Natural Language Generation, Evaluation, and Metrics (GEM 2021), pages 58-67, Online. Association for Computational Linguisties.
Melbourne, Australia, Association for Computational Linguistics.
Oscar M Humbicus-Pinedee Itziar Gonzalez-Diov and Aitor Soroa. 2021a. Linguistic capabilities for a checklist-based evaluation in automatic text simplifi- cation. In CTTS@ SEPIN.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners, Advances in neural information processing systems, 33:18 187119911
Oscar M. Cumbicus- Pineda,, Itziar Conzalez-Dioss and Aitor Soroa. 202 A syntax-aware edit-based sys- tem for text simplification. In Proceedings of the International Conference on Recent Advances in Nat- ural Language Processing (RANLP 2021). pages 324- 334, Held Online. INCOMA Ltd.
Dominique Brunato, Felice Dell'Orletta, and Giulia Venturi. 2022. Linguistically-bas comparison of different approaches to building corpora for text sim- plification: A case study on Italian. Frontiers in Psychology, 13.
Ashwin Devaraj. William Sheffield, Byron Wallace, and Junyi Jessy Li. 2022. Evaluating factuality in text simplification. In Proceedings of the 60th Annual Meeting of the Association for Computational Lin- guistics (Volume I: Long Pupers), pages 7331-7345, Dublin, Ireland. Association for Computational Lin- guistics.
Meng Cao. Yue Dong, and Jackie Cheung. 2022. Hal- lucinated but factual! inspecting the factuality of hallucinations in abstractive summarization, In Pro- ceedings of the 60th Annual Meeting of the Associa- rion for Computational Linguistics (Volume 1: Long Papers), pages 3340-333 -3544, Dublin, Ireland. Associa- tion for Computational Linguistics.
Yue Dong, Zichao Li, Mehdi Rezagholizadeh, and Jackie Chi Kit Cheung. 2019. EditNTS: An neural rerrrammm---nterrreeee
model for sentence simplifi- cation through explicit editing. In Proceedings of the 57th Annual Meeting of the Association for Computa- tional Linguisties, pages 3393-3402, Florence, Italy. Association for Computational Linguistics.
Rémi Cardon, Adrien Bibal. Rodrigo Wilkens, David Alfter, Magali Norré, Adeline Müller, Watrin Patrick, and Thomas François. 2022. Linguistic corpus an- notation for automatic text simplification evaluation. In Proceedings of the 2022 Conference on Empiri- cal Methods in Natural Language Processing, pages 1842-1866. Abu Dhabi, United Arab Emirates. Asso- ciation for Computational Linguistics.
Yao Dou, Maxwell Forbes, Rik Koncel- Kedziiosk.... Noah A. Smith, and Yejin Choi. 2022. Is GPT-3 text indistinguishable from human text? scarecrow: A framework for scrutinizing machine text. In Proceed- ings of the 60th Annual Meeting of the Association for Computational I Linguistics (Volume 1: Long Pa- pers), pages 7250-7274, Dublin, Ireland. Association for Computational Linguistics.
Mert Cemri, Tolga Cukur, and Aykut Kog. 2022. Unsu- pervised simplification of legal texts. arXiv preprint 20Xiv:200900055..
R. Chandrasekar, Christine Doran, and Srinivas. 1996. Motivations and methods for text simplification. In COLING 1996 Volume 2: The 16th International Conference on Computational Linguistics.
Mark Dras. 1999. Tree adjoining grammar and the reluctant paraphrasing of text. Macquarie University Sydney.
Wanyu Du Vipul Raheja, Dhruv Kumar, Zae Myung Kim, Melissa Lopez, and Dongyeop Kang. 2022. Understanding iterative revision from numan-writter text. In Proceedings of the 60th Anmual Meeting of the Association for Computational Linguistics ( (Vol- ume 1: Long Papers), pages 3573-3590, Dublin, Ireland, Association for Computational Linguistics.
Jiangji Chen, Rui Xu, Wenxuan Zeng. Changzhi Sun, Lei Li, and Yanghua Xiao. 2023. Converge to the truth: Factual error correction via iterative con- strained editing. In Proceedings of the Thirty-Seventl AAAI Conference on Artificial Intelligence, AAAI23.
AAAT Press.
Wei-Lin Chiang, Zhuohan Li, Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang. Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric Xing. 2023. Vicuna: An open- source chatbot impressing g with 90%* chatgpt quality.
Markus Freitag, George Foster, David Grangier, Viresh Ratnakar, Qijun Tan, and Wolfgang Macherey. 2021. Experts, errors, and context: A large-scale study of human evaluation for machine translation. Transac- tions of the Association for Computational Linguis- tics, 9:1460-1477.
Leshem Choshen and Omri Abend. 2018 Inherent bi- ases in reference-hased evaluation for grammatical error correction. In Proceedings of the 56th Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), pages 632-642,
Aparna Garimella, Abhilasha Sancheti, Vinay Aggar- wal, Ananya Ganesh, Niyati Chhaya, and Nandak- ishore Kambhatla. 2022. Text simplification for legal domain: Insights and challenges. Proceedings of the Natural Legal Language Processing Workshop

---

2022, pages 296-304, Abu Dhabi, United Arab Emi- rates (Hybrid). Association for Computational Lin- guistics.
Sebastian Joseph, Kathryn Kazanas, Keziah Reina, Vish- nesh J Ramanathan, Wei Xu, Byron C Wallace, and Junyi Jessy Li. 2023, Multilingual simplification of medical texts. arXiv preprint arXiv:2305.12552.
Sian Gooding. 2022. On the ethical considerations of text simplification. In Ninth Workshop on Speech and Language Processing for Assistive Technologies (SLPAT-20022), pages 50-57, Dublin, Ireland. Associ- ation for Computational Linguistics.
Klaus Krippendorft. 2018. Content analysis: An intro- duction to its methodology. Sage publications.
Reno Kriz, Mariaanna Apidianaki. and Chris Callison- Burch. 2020. Simple-Qr: Better automatic quality estimation for text simplification. arXiv preprint arXiv:2012.12382.
Sian Gooding and Manuel Tragut. 2022. One size does not fit all: The case for personalised word complexity models. In Findings of the Association for Compu- kational Linguistics: NAACL 2022, pages 353-365, Seattle, United States. Association for Computational Linguistics.
Dhruv Kumar, Lili Mou, Lukasz Golab, and Olga Vech- tomova. 2020. Iterative edit-based unsupervised sen- tence simplification. In Proceedings of the 58th An- nual Meeting of the Association for Computational Linguistics, pages 7918-7925 Online, Association for Computational Linguistics.
Tanya Goyal, Junyi Jessy Li, and Greg Durrett. 2022. News summarization and evaluation in the era of gpt-3, arXiv preprint arXiv:2209, 12356.
Philippe Laban, Tobias Schnabel, Paul Bennett, and Marti A. Hearst. 2021. Keep simple: Unsupervised simplification of multi- malti-paraggaph text. In Proceed- ings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 6365-6376 Online. Association for Computational Linguistics.
Arnay Gudibande, Eric Wallace, Charles Burton Snell, Xinyang Geng, Hao Liu, P. Abbeel, Sergey Levine, and Dawn Song. 2023. The false promise of imitating proprietary Ilms. ArXiv, abs/2305.15717.
Tatsunori Hashimoto, Hugh Zhang, and Percy Liang. 2019. Unifying human and statistical evaluation for natural language generation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, Volume I (Long and Short Papers), pages 1689-1701, Minneapolis, Min- nesota. Association for Computational Linguistics.
Wuwei Lan, Chao Jiang, and Wei Xu. 2021. Neural semi-Markor CRF for monolingual word alignment. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1thh International Joint Conference on Natural Lan- guage Processing (Volume I: Long Pupers), pages 68 1--688888, Online. Association for Computational Linguistics.
David Heineman, Yao Dou, and Wei Xu. 2023. Thresh: A unified, customizable and deployable platform for fine-grained text evaluation. arXiv preprint arXiv:2308.0695.
Arle Lommel, Hans Uszkoreit, and Aljoscha Burchardt. 2014. Multidimensional quality metrics (MQM): A framework for declaring and describing translation quality metrics. Revista Tradumâtica: tecnologies de la traducció, 12:455-44 463.
T Jaeger and Roger Levy. 2006. Speakers optimize information density through syntactic reduction. Ad- vances neural information processing systems, 19.
Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. Survey of halluci- nation in natural language generation. ACM Comput. Surv., 55(1)).
Ilya Loshchilov and Frank Hutter, 2019, Decoupled weight decay regularization, In International Confer- ence on Learning Representations.
Mounica Maddela, Fernando Mva-Manchego, and Wei Xu. 2021. Controllable text simplification with ex- plicit paraphrasing. ference of the North American Chapter of the Asso- ciation for Computational Linguistics: Human Lan- guage Technologies, pages 3536-3553, Online. As- sociation for Computational Linguistics.
Chao Jiang, Mounica Maddela, Wuwei Lan, Yang Zhong, and Wei Xu. 2020. Neural CRF model for sentence alignment in text simplification. In Proceed- ings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7943-7960, On- line. Association for Computational Linguistics.
Proceedings of the 2021 Con-
Mounica Maddela, Yao Dou, David Heineman, and Wei Xu. 2023, LENS: A learnabl evaluation metric for text simplification. In Proceedings of the 61st An- nual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 16383- 16408, Toronto, Canada. Association for Computa- tional Linguistics.
Xiaotong Jiang, Zhongqing Wang, and Guodong Zhou 2022. Semantic simplification for sentiment clas- sification. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Process- ing, pages 11022-11000 11022, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

---

Louis Martin, Éric de la Clergerie, Benoit Sagot, and Antoine Bordes. 2020. Controllable sentence sim- plification. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 4689- 4698, Marseille, France. European Language Re- sources Association.
Kishore Papineni. Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: a method for automatic evalu- ation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Compu- national Linguistics, pages 31 511-118, Philadelphia. Pennsylvania, USA. Association for Computational Linguistics.
Louis Martin, Angela Fan, Éric de la Clergerie, Antoine Bordes, and Benoît Sagot. 2022. MUSS: Multilin- gual unsupervised sentence simplification by mining paraphrases. In Proceedings of the Thirteenth Lan- guage Resources and Evaluation Conference, pages 1-51-,664, Marseille, France. European Language Resources Association.
Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. Language models as knowl- edge bases? In Proceedings of the 2019 Confer- ence on Empirical Methods in Natural Language Pro- cessing and the 9th International Joint Conference Natural Language Processing (EMNLP-IJCNLP), pages 2463-2473, Hong Kong, China, Association for Computational Linguistics.
Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020. On faithfulness and factu- ality in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1906-1919, On- line. Association for Computational Linguistics.
Jipeng Qiang, Yun Li, Yi Zhu, Yunhao Yuan, and Xin- dong Wu. 2020. Lexical simplification with pre- trained encoders. In Proceedings of the AAAI Con- ference on Artificial Intelligence, volume 34, pages 8649-8656.
Clara Meister, Ryan Cotterell, and Tim Vieira. 2020. beam search is the answer, what was the question? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2173-2185, Online. Association for Computa- tional Linguisties.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, al. 2020. Exploring the limits of transfer learning with a unified text-to-text trans- former. J. Mach. Learn. Res., 21(140):1-67.
Masaaki Nagata, Katsuki Chousa, and Masaaki Nishino. 2020. A supervised word alignment method based on cross-language span prediction using multilingual BERT. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 555-565, Online. Association for Computational Linguistics.
Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don't know: Unanswerable ques- tions for SQuAD. In Proceedings of the 56th Annual Meeting of the Association for Computational Lin- guisties (Volume 2: Short Papers), pages 784-789, Melbourne, Australia, Association for Computational Linguistics.
OpenAI. 2022. ChatGPT: Optimizing language mod- els for dialogue. httpp::/opent com/blog/ chatgpt.
Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. COMET: A neural framework for MT evaluation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Process- ing (EMNLP), pages 2685-2702, Online. Association for Computational Linguistics.
OpenAL. 2023. GPT-4 technical report. arXiv preprint arXiv:2303.08774.
Ricardo Rei, Marcos Treviso, Nuno M. Guerreiro, Chrysoula Zerva, Ana C Farinha, Christine Maroti, José G. C. de Souza, Taisiya Glushkova, Duarte Alves, Luisa Coheur, Alon Lavie, and André F. T. Martins. 2022. CometKiwi: IST-unbabel 2022 sub- mission for the quality estimation shared task. In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 634-645, Abu Dhabi, United Arab Emirates (Hybrid). Association for Com- putational Linguistics.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Car- roll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow in- structions with human feedback. arXiv preprint arX1v:2203.02155.
Gustavo H. Pactzold and Lucia Specia. 2013. Text simplification tree transduction. In Proceedings of the 9th Brazilian Symposium in Information and Human Language Technology.
Michael Ryan, Tarek Naous, and Wei Xu. 2023. Re- visiting non-English text simplification: A unified multilingual benchmark. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4898- 4927, Toronto, Canada. Association for Computa- tional Linguistics.
Artidoro Pagnoni, Vidhisha Balachandran, and Yulia Tsvetkov. 2021. Understanding factuality in abstrac- tive summarization with FRANK: A benchmark for factuality metries. In Proceedings of the 2021 Con- ference of the North American Chapter of the Asso- ciation for Computational Linguistics: Human Lan- guage Technologies, pages 4812 4-29, Online. As- sociation for Computational Linguistics.
Carolina Searton, Alessio Palmero Aprosio. Sara Tonelli, Tamara Martín Wanton, and Lucia Specia.

---

2017. MUSST: A multilingual syntactic simplifi- cation tool. In Proceedings of the IJCNLP 2017, System Demonstrations, pages 25-28, Tapei, Taiwan. Association for Computational Linguisties.
Renliang Sun, Hanqi Jin, and Xiaojun Wan. 2021. Document-level text simplification: Dataset, crite- ria and baseline. In Proceedings of the 2021 Confer- ence on Empirical Methods in Natural Language Pro- cessing, pages 7997-8013, Online and Punta Cana. Dominican Republic. Association for Computational Linguisties.
Boaz Shmueli, Jan Fell, Soumya Ray, and Lun-Wei Ku. 2021. Beyond fair pay: Ethical implications of NLP crowdsoureing. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pages 3758-3769, Online. Association for Computational Linguistics.
Renliang Sun, Zhe Lin, and Xiaojun Wan. 2020. On the helpfulness of document context to sentence simplifi- cation. In Proceedings of the 28th International Con- ference on Computational Linguistics, pages 1411- 1423, Barcelona, Spain (Online). International Com- mittee Computational Linguistics.
Advaith Siddharthan. 2006. Syntactic simplification and text cohesion. Research on Language and Com- putation, 4(19:77-109.
Rohan Taori, Ishaan Guirajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford Alpaca: An instruction-following LLaMA model. https:/ gi thub. www.///tttttttteoooooooooo waw.ssssuuaabsttafoorrrd stanford_aldaac.....
Advaith Siddharthan. 2014. A survey of research on text simplification. ITI-International Journal of Applied Linguistics, 165(2):259-298.
Hugo Touvron, Thibaut Lavril. Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. LLaMA: Open and ef- ficient foundation language models. arXiv preprint ar arXiv:2302.13971
Sanja Stajner: 2016. New data-driven approaches lext simplification. Ph.D. thesis, University of Wolver- hampton.
Sanja Stajner. 2021. Automatic text simplification for social good: Progress and challenges. In Findings of the Association for Computational Linguistics: ACL-
Jan Trienes, Jörg Schlötterer, Hans-Ulrich Schildhaus, and Christin Seifert. 2022. Patient-friee friendly clinical notes: Towards a new text simplification dataset. In Proceedings of the Workshop on Text Simplification, Accessibility, and Readability (TSAR-202) pages 19-27, Abu Dhabi, United Arab Emirates (Virtual). Association for Computational Linguistics.
IJCNPP 2021, pages 2637-2652, Online. Association for Computational Linguistics.
Regina Stodden and Laura Kallmeyer. 2022. TS- ANNO: An annotation tool to build, annotate and evaluate text simplification corpora. In Proceed- ings of the 60th Annual Meeting of the Association for Computational Linguistics: System Demonstra- tions, pages 145-155, Dublin, Ireland, Association for Computational Linguistics.
Yu Wan, Dayiheng Liu, Baosong Yang, Haibo Zhang. Boxing Chen, Derek Wong, and Lidia Chao. 2022. UniTE: Unified translation evaluation. In Proceed- ings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pa- pers), pages 81 177217, Dublin, Ireland. Association for Computational Linguistics.
Elior Sulem, Omri Abend, and Ari Rappoport. 2018a. BLEU is not suitable for the evaluation text simpli- fication. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 738-744, Brussels, Belgium. Association for Computational Linguistics.
Yizhong Wang. Yeganch Kordi, Swaroop Mishra, Alisa Liu. Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. Self-instruct: Aligning language models with self-generated instructions. In Proceed- ings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 13484-13508, Toronto, Canada. Association for Computational Linguistics.
Elior Sulem, Omri Abend, and Ari Rappoport. 2018b. Semantic structural evaluation for text simplification. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, Volume I (Long Papers), pages 685-696, New Or- leans, Louisiana, Association for Computational Lin- guistics.
Zeqiu Wu, Yushi Hu, Weijia Shi. Nouha Dziri, Alane Suhr, Prichviraj Ammanabrolu, Noah A Smith, Mari Ostendorf, and Hannaneh Hajishirzi. 2023. Fine-grained human feedback gives better rewards for language model training. arXiv preprint 23Xxv23306.0693
Blior Sulem, Omri Abend, and Ari Rappoport. 2018c, 8c. Simple and effective text simplification using seman- tic and neural methods. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 162-173, Melbourne, Australia. Association for Computational Linguistics.
Wei Xu, Courtney Napoles, Ellie Pavlick, Quanze Chen. and Chris Callison-Burch. 2016. Optimizing sta- tistical machine translation for text simplification. Transactions of the Association for Computational Linguistics, 4:401-415.

---

Wei Xu, Alan Ritter, Bill Dolan, Raiph Grishman, and Colin Cherry. 2012. Paraphrasing for style. In Pro- ceedings of COLIGG 2012, pages 2899-2914, Mum- bai, India. The COLING 2012 Organizing Commit- tee.
Daichi Yamaguchi, Rei Miyata, Sayuka Shimada, and Satoshi Sato. 2023. Gauging the gap between human and machine text simplification through analytical evaluation of simplification strategies and errors. In Findings of the Association for Computational Lin- guisties: EACL 2023, pages 359-375, Dubrovnik, Croatia. Association for Computational Linguistics.
Chrysoula Zerva, Frédéric Blain, Ricardo Rei, Piyawat Lertvittayakujjrn, José G. C. de Souza, Steffen Eger, Diptesh Kanojia, Duarte Alves, Constantin Oräsan, Marina Fomicheva, André T. Martins, and Lucia Specia. 2022. Findings of the WMT 2022 shared task on quality estimation. In Proceedings of the Seventh Conference on Machine Translation (WM7), pages 69-99, Abu Dhabi, United Arab Emi- rates (Hybrid). Association for Computational Lin- guistics.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q- Weinberger, and Yoav Artzi. 2020. BERTScore: Evaluating text generation with BERT. International Conference on Learning Representations.
Xingxing Zhang and Mirella Lapata. 2017. Sentence simplification with deep reinforcement learning. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 584- 594, Copenhagen, Denmark. Association for Compu- tational Linguistics.

---

A Defining the SALSA & Framework
Deleting information will, by nature, contain some amount of information and SALSA instead focuses on ensuring the deleted information is not impor- tant sentence, context or users. Consider two can- didate deletions:
We provide detail into the SALSA framework, in- cluding qualitative examples which helped guide design decisions when building the typology. Table 4 illustrates each final edit type, as organized by Figure 3. During development, we adjusted our scheme based on preliminary annotations with the final goal of SALSA's is ability to evenly represent all modes of simplification and the full space of errors.
EXAMPLE
Like so many hyped hooks before it, The Mianight Library excited me and gave pause.
Like so many hyped books before it, The Midnight Library excited me and gave me pause.
Although the deletion of Midnight is shorter, it changed the subject of the sentence, and is rated higher than the second deletion, which not cen- tral to the main idea. Generalization using para- phrase is more often preferred than deleting full clauses.
A.1 Quality Evaluation
We organize quality edits by their approach to sim- plification, as real-world application and models' capability to simplify falls into tiers of concep- tual, syntactic and lexical simplification (Stajner, 2021). An ideal simplification system demonstrates a balance of these 'tiers' and incorporates different techniques depending on the original text, context and users (Gooding and Tragut, 2022). Automatic simplification research initially focused on lexical paraphrasing (Siddharthan, 2014), but has since evolved to emphasize the importance of syntac- tic and conceptual editing (Alva-Manchego et al.. 2020b).
We observe successful conceptual edits are of- ten performed on the clause level. For example. adjunct removal via deletion:
EXAMPLE
Born into slavery in 1856, Booker T. Washington became an influential African American leader.
Booker T. Washington became an influential African American leader.
information insertion through an appositive or relative clause, although the prior is typically more common for the SIMPEVAL domain as it implies objective information:
A.1.1 Conceptual Simplification
These edits modify the underlying sentence infor- mation ideas, prerequisite for simplifying plex domains. We consider 'conceptual simplifica- tion' be interchangeable with 'semantic simpli- fication' as used in some literature (Sulem et al., 2018b; Jiang et al., 2022).
EXAMPLE
Éric Gauthier is also novella author... Éric Gauthier, famous for his soloist dancing career, is also novella author...
A.1.2 Syntactic Simplification
Elaboration. An addition of meaningful, relevant and correct information (Siddharthan, 2006), such as clarifying vague terminology, providing back- ground information on an entity subject, or ex- plicating general world knowledge unknown to the audience. Elaboration has been shown rare, but helpful mechanism in text generation (Cao al., 2022) and observe its careful use in human simplifications.
Syntax is a crucial mechanism for fluent, highly modified simplification (Stajner, 2016). Given re- cent attention in automatic simplification to syntax- aware datasets and systems (Cumbicus-Pineda et al., 2021b; Kumar et al., 2020; Alva-Manchego et al., 2020a; Scarton al., 2017), SALSA stan- dardizes the first explicit evaluation accounting for these operations.
Information Reorder. We classify two levels of reorder, word-level reorder, which reorganizes mod- ifiers within a phrase, and component-level reorder which moves clauses or content across a sentence (Siddharthan, 2006). A component-level re-order typically may be accompanied by broader struc- ture change or both re-order types may overlap, in:
Generalization. A deletion of unnecessary, irrel- evant or complicated concepts. Although we ask annotators to rate the quality of elaboration by how it improves the readability of a sentence, we ask annotators rate the quality of generalization by the relevancy of the deleted information to the main idea of the sentence. As 'relevancy" is inher- ently subjective to the user, domain and annotator, determining the threshold for 'necessary informa- tion' is crucial to standardize (Devaraj et al., 2022).
EXAMPLE
The emergence of huge radio conglomerates is rect consequence of the '96 Act.

---

Type
Description
Example
Quality Evaluation
Concepinal
Elaboration
Meaningful and correct information which enu- merales the main idea
Many volatile organic chemicals, which harm our environment, are increasing in abundance in the lower troposphere.
Generalization
Removes unnecessary, irrelevant or complicated in Information
Many volatile urganic chemicals are increasing lower posphere. (in abunulunce was removed)
Word-level Re- order
Order of words within phrase swapped
Many organic volatile chemicals increasing abundance in the lower troposphere.
Syntax
Component- lovel Reorder
Order of phrases senience swapped
the lower tropusphere, many volatile organic chemicals are increasing ahundance.
Sentence Split
Indopendent information converted to two sepa- rate sentences.
Many volatile organic chemicals are increasing, They are found in abundance in the lower troposphere.
Structure
Rewrites voice, lense or structure. See Ap- pendix for details and sub-types
The ahundance of many volatile organic chemicals is incrensing in the lower troposphere.
Change
Lexical
Paraphrase
Lexical cumplesxty of the phrase decnenses, while the meaning unchanged
Many vulatile organic chemicals are being seen more in the lower tropusphere.
Trivial Change
Adds clarity or removes verbosity, while the lexical complexity and meaning is unchanged
Many volatile organic chemicals are curreatly increasing in abun- dance in the lower troposphere.
Error Evaluation
Bad Delction
Deleted necessary and relevant content
Many chemicals are increasing abundance in the lower tropo- sphere. (volatile organic was removed) They are increasing abundance in the lower troposphere.
Conceptual
Coreference
reference named entity critical under- standing the main idea is removed
Repetition
Phrase added or changed hut fail to contain novel information insight
Many volatile organic chemicals, which are chemicals, are in- creasing in abundance in the lower tropusphere.
Contradiction
Phrase added or changed but clearly contradicts in information presented
Many volatile organic chemicals, which are decreasing in tro- posphere, are increasing in abundance in the lower troposphere. Many volatile organic chemicals are increasing abundance in the lower troposphere when they decide to.
original sentence
Factual Error
Externlly verifiable incorrect claim made by the phrase
Irrelevant
New information is introduced which unre- lated to the main idea
Many volatile organic chemicals, unlike low vapor pressure chem- icals, are incrensing in ahundance in the lower troposphere. Many volatile organic chemicals are having their abundance in- creasing the lower troposphere.
Bad Word-level Reorder
Presented new word order with less clarity within a clause
Syntox
Compu- nent Reorder
Bad
Presented new clausal order with clarity
In abundance in lower Iropusphere, many volatile organic chemicals increasing.
Bad Structure
failed attempl to modify the voice, lense or structure
Many volatile organic chemicals have been increasing in abun- dance in the lower troposphere.
Bad Split
Split at an inappropriate location or interrupted the flow of ideas
Many volatile orgunic chemicals are increasing. They are increas- ing in abundance in the lower troposphere.
Complex Word-
Lexical complexity of the phrase increases, while the meaning is retained
Many volatile organic chemicals are proliferating throughoul the lower troposphere.
ing
Lexical
Information Rewrite
All information was removed from the phrase
Many volatile organic chemicals are decreasing in abundance in the lower troposphere.
replaced with new information
Grammar
Violation of conventional grammar
Many volatile organic chemicals which are increusing in abun- dance in the lower troposphere.
Table 4: Overview of the SALSS edit-level evaluation typology. Original text for the examples: Many volatile organic chemicals are increasing in abundance in the lower troposphere.
The '96 Act had direct consequence of the emer- gence of huge radio conglomerates.
deletion, substitution, reorder). Further discussion of structure changes in $B, with examples of struc- tural change sub-types used for manual inspection in Table 5.
When faced with two equivalent phrases (e.g. 'A and B' 'B and A'), SALSA classifies the re- ordered span as the phrase more significant to the main idea of the sentence. In practice, we found this to be a helpful guideline, although annotators often simply selected the phrase appearing first in the candidate sentence.
Sentence Split. A sub-type of a structural edit. We automatically identify split changes prior to anno- tation, but annotators must first select constituent spans and then associate those spans with the corre- sponding sentence split. We find the importance of this edit is highly domain-dependedn (Figure 13).
Structural Change. As this syntax modification necessarily includes some discourse preserving ed- its (Gooding, 2022), they are defined w.r.t. some combination of constituent edits (i.e. insertion,

---

A.1.3 Lexical Simplification
Bad deletion. As the overwhelmingly most com- mon error, bad deletion removes necessary and relevant content to the main idea of the sentence. As discussed in $A.1.1, the threshold for 'rele- vancy' is ambiguous.
Paraphrase. Swapping complex spans with equiv- alent, simpler alternatives, is the most primitive. yet important, approach to simplification (Qiang et al., 2020) (also referred to as a hypernym, e-g- Stajner, 2016). These are exclusively defined by substitutions marked as same information and posi- tive impact.
Coreference. More precisely a failure in corefer- ence or anaphora resolution (Maddela et al.. 2021), this determines whether an explicit entity reference is removed. This error is only observed on a dele- tion of information.
Trivial Change. Captures any minor modifications to wording, either through a synonym replacement, or inconsequential change in wording (e.g. the, a). Trivial changes are identified as trivial insertion, trivial deletion trivial substitution. These edits differ from a content syntax modification in that they adds no new or major modification the pre- sentation of information. However, Meister al. (2020) exemplifies trivial changes should not be ignored as they may modify the information den- sity and verbosity of a sentence. An example is famously shown by Jaeger and Levy (2006):
EXAMPLE
Herbert Spencer's book makes the first...
His book makes the first...
Repetition. Some trivially additional information which simply repeats knowledge already previously contained in the candidate sentence.
EXAMPLE
the New York City Police Department is law enforcement agency... ... the New York City Police Department is a police department...
EXAMPLE
How big the family you cook for? How big the family that you cook for?
Despite successfully paraphrasing, police depart- ment, simply copies content from earlier in the sentence, instead of generating unique information.
The relativizer 'that" creates no syntactic or concep- tual simplicity, but adds clarity as to the identify of the subject. Trivial changes have previously been described with finer granularity, including subcat- egories like abbreviation, filler words, compound segmentation, anaphora (Stodden and Kallmeyer, 2022) or even changes in number/date formatting (Cardon et al., 2022) but we exclude these groups due to their sparsity and our focus on evaluating performance.
Contradiction. A negation of the meaning of the original sentence. This notably includes modify- ing an existing phrase contradict the original sentence:
EXAMPLE
the Watergate hurglars were convicted
... Watergate burglars were not convicted
generating new information making the sentence contradict itself:
A.2 Error Evaluation
EXAMPLE
We describe the SALSA етrоr typology, with ex- amples of each type in Table 4. Although despite their sparsity, errors have far greater impact on fluency and adequacy than individual quality edits (Chen al., 2023). We refined our definition of errors by focusing on minimizing the amount of eтror types while retaining the ability to capture the full possibility of simplification ablations. No- tably, we specifically exclude a hallucination due to its ambiguous definition in related work (Ji et al., 2023), and instead define our error categories to capture any possible hallucination.
Dextrose adds flavor texture dishes, although consumption known negative consequences. Dextrose adds flavor, texture and nutrition dishes, although its consumption known for negative con- sequences.
Factual Error. We asked annotators to use their commonsense knowledge and limited research to evaluate factuality in edits. Unlike contradiction, these claims introduce information which must be externally verified beyond the sentence context. Al- though factual content is an established focus for summarization evaluation (Pagnoni et al., 2021; Maynez et al., 2020), adequately retaining informa- tion (i.e. minimizing bad deletion) is a far greater concern for simplification (Devaraj et al., 2022).
A.2.1 Conceptual Errors
We identify six types of errors in content, with errors primarily being related to information inser- tion.
EXAMPLE
Hilary Clinton was born in 1947.

---

Hilary Clinton was born 1947 outside the United States.
Grammar Error. The edit violates grammatical convention. Past error analysis combines fluency and grammar into the same error type (Maddela et al., 2021) as the two are interrelated. Gram- mar errors are unique as they can co-occur with other errors, or occur alongside high quality edit, as sentence fluency is independent from adequacy (Siddharthan, 2014).
the context of work studying hallucination in LLMs, our contradiction and factual error cate- gories can be interpreted as intrinsic and extrinsic hallucination respectively (Ji et al., 2023).
Irrelevant. A sub-type of hallucination failing to insert information related to the main idea of the sentence, recognizing the threshold for 'relevancy' is ambiguous 1$1. For simplicity, we report irrelevancy alongside hallucination, as information insertion is generally a rare technique.
A.3 Edit Severity Efficacy Levels
We provide examples of each severity level, which are also included as part of annotator training:
EXAMPLE
Severity: 1 1-minor
A.2.2 Syntactic Errors
Like many hyped books before The Midnight Library excited me and gave pause
Because syntactic edits are identified by the impact of information distribution, they do not need fine- grained error typology like conceptual edits, which make diverse set of modifications. We simply observe each type failed attempted at their respective transformations.
The Midnight Library excited me and gave
The introductory clause 'Like so many hyped books before it,' situates the sentence within the context of 'hyped books.' However, it does not relate to the main idea of the sentence (the author's opinion on 'The Midnight Library").
Bad Reorder. Uses the same word-/phrase-level specification quality reorder. We also observe that phrase-level reorder errors are almost exclu- sively observed to introduce a discontinuity to the syntax tree structure (Paetzold and Specia, 2013).
EXAMPLE
Severity: 2 somewhat
Two security flaws, dubbed Meltdown and Spectre by researchers, were made public on January 2018.
Two security flaws, dubbed Meltdown and Spectre by researchers, were made public.
Bad Structure. We manually inspect structural errors according to the same sub-type specification quality edits (SB).
Although the sentence retains its core meaning without 'on 29 January 2018', the specific reference of when 'Meltdown' and 'Spectre' were 'made pub- lic' is lost.
Bad Sentence Split. Although sentence splitting is rarely rated as unhelpful, simplifications may unnecessarily segment ideas, or interrupt the flow of information.
EXAMPLE
Severity: 3 i-ajoor
If glycolysis evolved relatively late, it likely would not be as universal in organisms as is.
It likely would not be universal in organisms as it is.
A.2.3 Lexical Errors
Since the entity 'glycolysis' has been deleted, the coreference corresponding to the subject 'it' is lost.
Unrelated to information change, lexical errors evaluate primitive issues in fluency or wording. Complex Wording. An attempted paraphrase where the exact meaning retained, but the replace- ment uses more complex semantics (also referred to as hyponym, e.g. Stodden and Kallmeyer, 2022).
A.4 Overall simplification score
Similar to MQM (Lommel et al., 2014), we col- lapse edit annotations into a simplification score to allow for direct system comparison. We calculate the sentence-level score weighted sum of edit ratings:
EXAMPLE
The researchers conducted investigation.
The researchers conducted an assay.
( + (en(CC)+ +
Information Rewrite. Some substituted span whose content concerns the same subject, but fails substitute the wording correctly, either through misrepresenting or falsely interpreting the informa- tion. Although similar to a combination of infor- mation deletion and information insertion, the edit is still attempting to represent the same content.
Σ exp eEE
w(e) r(e)
where S is the simplification of complex sentence C, E the set of edits, ec and es are the parts of edit e performed on C' and S respectively, wree is the edit weight, r(e) is the edit rating (severity

---

Sub-type
Definition
Examples
Original
Simplification
Voice Change
Change the subject & receiver of
active voice
Her book makes the first thorough anal- ysis of this rural society.
The first thorough mialysis of this rural society is made by her book.
passive voice
passive voice
Elevation is not primarily considered by the system.
The system does not primarily consider elevation.
active voice
Part-of- Speech Change
Modifies words' derivation inflection
nominalisation (verb - noun)
The ability capture heen inproving...
The ability capture seen improvement...
denominalisation (adjective >> verb)
The protesters turned viokent when...
The violent protesters...
Tense Change
Modifies verb modality or tense
past perfect past simple
The guvernor told reporters he had overseen productive conversation. compute the Pearson correlation annotation quality.
The governor oversaw productive conversation.
present past
computed the Pearson correlation when assessed annotation quality.
Grammatical Number
Distinction of count changes
singular - plural
Victor scured goal against US 2011 and another in 2012. The spokesperson for the university called
Victor scured goals and 2012.
generic + specific
spokesperson the university called
Clausal Change
Modilies pred- icate structure
coordinate clause + relative clause
Donaldson allempled speak clearly and he successful.
Donaldson attempted speak clearly and fully.
subordinate clause coordinate
Although was raining outside, Jobs continued work garage.
Outside was raining and Jobs contin- garage.
clause
Table 5:
Examples of structural modification sub-types used for annotation.
efficacy), and Ien denotes character length.* For weight scheme w(e), we fit linear regression by considering the sentence-level human ratings gath- ered in SIMPE'ALL222 (Maddela et al., 2023) as gold standard. As the type of simplification de- pends the needs of each particular user group (Stajner, 2021), weights may be adjusted according to the simplification domain (Cemri et al., 2022: Basu et al., 2023; Joseph et al., 2023) or use case (Trienes et al., 2022).
a case study in Italian simplification (Brunato et al., 2022) shows this structural edit distribution may vary when adapted to the needs of other languages. Similarly, German simplification often converts genitive to dative noun cases, feature not seen in English simplification (Stodden and Kallmeyer, 2022).
C Data Collection Details
C.1 Simplification Systems
B Structural Edit Examples
Our main corpus of 700 simplifications are from the following diverse simplification approaches:
Examples of each structural edit sub-type listed in Table 5. We find training annotators label structure change sub-type improved their ability to identify structure changes. We include morpholog- ical changes (e.g., tense change) as structure edits since these typically require multiple disconnected edits to perform and impact sentence-level mean- ing. Additionally, other work (Barancikova and Bojar, 2020), specifically Stodden and Kallmeyer (2022) annotate with larger array of structural changes, notably including separate directions as distinct categories (e.g. singular - plural and plu- ral + singular) and including change in sentiment and personal/impersonal form. We exclude these types as they almost never occur in the entirety of the ASSET corpus (Cardon et al., 2022). However,
MUSS (Martin et al., 2022), BART-large model conditioned on explicit parameter tokens from Mar- tin ct al. (2020), fine-tuned on Wiki-Large (Zhang and Lapata, 2017) and mined paraphrase data. MUSS the SOTA model before GPT-3.5.
T5 (Raffel et al., 2020), an encoder-decoder trans- former pre-trained on 745 GB of web text. We use T5-3B and T5-1 IB variants and fine-tune on the aligned Wiki-Auto dataset (Jiang et al., 2020), shown to be higher quality than Wiki-Large.
GPT-3. 5, series of GPT-3 models pre-trained on text and code dated before Q4 2021. We use the best available text-davinci- InstructGPT (Ouyang et al., 2022), fine-tuned with human demonstrations and reinforcement learn- ing with human feedback. We include both zero- and sho-shot (5-shot) generation, using the same
model, based
WWo normalize the edit length and use exp to add weight for longer odits.

---

prompt setup SIMPEVAL2022 (Maddela al.. 2023).
Fleiss kappa
Agree%
Bad Deletion
0.51
64
35
Complex Wording
0.26
32
20
Humans. We ask two in-house annotators write simplifications for the 40 newly selected sentences, replicating instructions used in SIMPEVAT We average the annotations of both human simplifica- tions for dataset analysis.
Information Rewrite
0.27
26
10
Grammar
Bad Structure
0.02
6
10
Bad Reorder
0.14
19
Inrelevant
0.22
26
8
Bad Split
0.13
17
4
Repetition
0.33
30
4
Contradiction
0.19
25
1
Our test set 140 simplifications are from recent approaches, including open-source LLMs:
Coreference
Table 6: Fleiss kappa error identification agreement measured per-sentence alongside error frequencies. As errors were far more rare, we observe strong relation- ship between frequency and expected agreement.
T5 with ACCESS Tokens, we use the same train- ing setup as our fine-tuned T5 model, but prepend the input with ACCESS control tokens (Martin et al., 2020): character length ratio, dependency tree depth ratio, character-level Levenshtein sim- ilarity, and inverse frequency ratio. During infer- ence, we use 0.9 for the length ratio, and 0.75 for the other three control tokens, following the setup (Maddela et al., 2023).
complex words with simpler synonyms (i.e. para- phrasing), deleting unimportant information (i.e. compression), and/or splitting a long complex sen- tence into several simpler ones. The final simpli- fied sentence needs to be grammatical, fluent, and retain the main ideas of its original counterpart without altering its meaning."
Alpaca-7B (Taori et al., 2023), a fine-tuned LLaMA model (Touvron et al., 2023) on 52K GPT- 3.5 outputs generated using the Self-Instruct tech- nique (Wang et al., 2023). As we find the prompt used for GPT-3.5 is too complex for Alpaca, use the following prompt:
Humans. As existing automatic simplification evalu- ation metrics rely on human references, we include two human-written simplifications to use for metric evaluation, but do not collect annotations on these references.
"Rewrite the following complex sentence in or- der to make it easier to understand by non-native speakers of English."
C.2 Interpreting Annotator Agreement
As the SIMPEVAL challenge dataset contains more edits than past simplification corpora, edit anno- tation becomes significantly more challenging multiple groups of edits often overlap and simpli- fications contain more compression and sentence- level transformations. Additionally, error-prone systems like MUSS make it challenging to disam- biguate error and quality edits. Figure illustrates an example of this disagreement, showing many of the same tokens are annotated, but with different edit spans. For example, observe the last clause in the sentence, which performs a rewrite:
Vicuna-7B (Chiang et al., 2023), fine-tuned LLaMA model on 70K publicly shared ChatGPT conversations. As the training data for Vicuna in- cludes prompts that are more diverse and complex than those used by Alpaca, Vicuna can manage longer prompts, but not at the level GPT-3.5, so we use the following prompt:
"Rewrite the following complex sentence in or- der to make it easier understand by non-native speakers of English. The final simplified sentence needs to be grammatical, fluent, and retain the main ideas of its original counterpart without al- tering its meaning."
EXAMPLE
that the fort stood out for its defenders' heroic resis- tance. and the defenders of the fort gave their lives to save the city.
ChatGPT, an optimized chat variant of GPT-3.5, the model we use is gpt-3. 5-rurbo- -0301. GPT-4, large multimodal model that performs better than GPT-3.5 models. We use the version of gpt-4-0314.
We three different, but valid understandings of this phrase:
For ChatGPT and GPT-4, we use the same prompt as GPT-3.5:
1. Information was replaced The information about the defenders' resistance is inherently dif- ferent then the defenders' giving their lives save the city and is therefore an add/deletion pair.
"Rewrite the following complex sentence in or- der to make it easier to understand by non-native speakers of English. You can do so by replacing

---

Oriyinal Sentance (Himan Writtony
Repestedly besieged during the Wer of Restoration of Independence, it was during the siege of the city of Elvas, led by the Spanish commander Don Luis de Haro in 1658, that the fort slood out for its defenders' heroic resistance.
Simpitted Sentence (Hum mruann Model Wrlttan
was often attacked during the War Restoration Independence. 1 In 1658, the Spanish commander Don Luis
Haro attacked the city of Elvas, and the defenders the for: gave their lives to save the city.
Onginal Sentence (Humen Writhen"
Repestedly besieged during the War of Restoration of Independence, it during siege of the city Elvas. led by
the Spanish commaander Don Luis Haro in 1656, thai the stood for defenders' heroic resistance.
Simplited Serrence tHumnn or Al Model Wittley
was often attacked during the War of Restcration of Independence. In 1668, Spenish commander Don Luis de Haro attacked the city of Elvas, and the defenders of the fort gave their lives to save the city.
Original Sentence
Repeatedly besieged during the War of Restoration of Independence, was during the siege of the city of Elvas, led by
the Spenish comimander Don Luis de Haro 1658, that fort stood out for ta. defenders' heroic resistance.
Simpttod Sentence liaman-e-tt Mide Whnton):
often attacked during War Restoration Independence.
1658, Spanish commander Don Luis de city.
attacked City Elvas, detenders of the fort gave
Figure 8: Edit selection between three annotators on a MUSS simplification. For complex examples, multiple valid interpretations for span labeling may exist, however we find annotator's overall judgements are consistent.
Using composite edits
Using constituent edits
1. Identified the edit as a structural change, be- cause the noun siege was replaced with a verb, modifying the voice of the sentence
3502
Insetton
Sursitution
Deletion
Receder
Subeth.tizn
2. Identified paraphrase, annotating siege as a more complex word than attacked
Srrcture
3. Correctly identified both edits occurred simulta- neously
bowedon
Detellan
Subreitation
Spit
Smothe
Ieseriee
Deleraei
duterttarion
Render
We find the largest source disagreement comes from overlapping edits of multiple types, most of- ten between structural changes and other types, be- cause they often co-occur. Figure 9 demonstrates structural edits explain significant portion of dis- agrecment. Additionally, because structural edits are composite edit, the same spans are captured by the structural edits' constituent spans and re- calculating agreement using these spans, disagree- ment instead focuses on whether tokens are substi- tuted.
Figure 9: Edit-level confusion of annotated tokens. The vertical and horizontal axes represents the class of major- ity agreement and minority decision between annotators respectively. The left includes all edits, while the right calculates agreement using the underlying constituent spans selected for structure and split edits.
2. Information was retained, but paraphrased The phrase heroic resistance being equivalent in meaning gave their lives.
3. Subject was modified and information was re- placed - The subject swap between the subject of the clause being the fort to being the defend- ers. The rest being an add/deletion pair.
Within individual sentences, we often observe multiple valid interpretations for span labeling. highlighting the inherit ambiguity in the task. De- spite this, annotators still successfully communi- cated edit performance. All three annotators identi- fied both the bad deletion and hallucination errors contained in the sentence. For the full SIMPEVAL dataset, we report error identification agreement in Table 6, finding syntax errors (c.g., bad struc- ture, bad reorder) are far more difficult identify than content or lexical errors. Particularly, complex wording and grammar errors exhibit both high fre-
Varying interpretations of the same edit leads natural disagreement. However, often a clear annotation exists and not captured. For example, although we instructed annotators to create sepa- rate edits for overlapping syntax and conceptual edits, this occurred inconsistently in practice:
EXAMPLE
was during the siege of the city of Elvas
Don Luis de Haro attacked the city of Elvas

---

GP73.5
2.0
las strett
Crepto yor Ead
Enter
Bla Claction au
infermntion Pieetie
Ead sii
intert Entttion
Cansterent forr Comple
Cilatio edermsaie
erfermstion
Compue evom aaaaaa
Figure 10: Average sentence-level score across error sentences for each system.
0.38
Simptval
ASSET
250
Axg Leruhggh
5.16
0.34
6.12
Edh.ictance
200
150
0.32
100
Desemitotton
Etaoratton
Spar Srammoy
Fo
Renider
gom Pertaee glen
gar Spu tdt pevrisdo Phourta Compler
Figure 11: Average edit coverage of edit types and specific error types with 95% confidence interval. Edit coverage, the ratio of the simplification being edited, is formalized in $A.4.
Number Edits
Figure 12: Edit distance and number of annotated edits for 300 randomly sampled sentences from ASSET and SIMPEVAL. While past work found no relationship, by extending ASSET to more complex sentences we see clear correlation arise.
quency and high agreement, as the definitions of these errors unambiguous. Broadly, find that high span-level agreement is necessary for cap- turing overall, or even fine-grained sentence-level performance, a clear trade-off exists between the granularity of annotations and expected agreement.
in more than 20% of outputs (Table 6). However, eтror sizes display the same trend as their quality counterparts, with conceptual errors typically being the clause level. We also found single-word conceptual errors such as:
D Further Analysis
EXAMPLE
Zero-shot GPT-3.5
Arroyo released a statement that acted infor-
Here, we report additional findings on the SIM- PEVAL dataset and model performance, alongside observations about edit-level evaluation as a task. Figure 11 reports the average edit coverage by each edit operation and error type. We find para- phrases are typically annotated as pairs of few words, while conceptual edits typically occur on the clause level and annotated together. Sur- prisingly, structure changes often occurred as a few words:
mal concession of sorts
Arroyo released statement that was like a formal concession.
EXAMPLE
Few-shot GPT.3.5
The sentence included a fine of $400...
They imposed a fine of $400...
Were less frequent than hallucinating entirely new phrasing or idcas. This may promising for error detection as it implies error spans are often clausal and occur among many adjacent tokens.
EXAMPLE
MUSS Corbin has expanded his business include agri- tourism, using his farm host weddings... ...
Quality and Error Are Interrelated. Figure 10 displays sentence-level scores for our error typol- ogy across systems on SIMPEVAL. We find the existence of eπror to be consistent predictor of a lower quality sentence, even in human simpli- fications. However, we find some errors correlate with a higher score (e.g. bad structure, information rewrite), but this may be attributed to the multi- clause complex sentences in SIMPEVAL having a
... Corbin's business also offers agritourism and he uses his farm to host weddings...
The edit converts the beginning subordinate clause coordinate clause, yet only requires substitut- ing a single word. Errors exhibited significantly higher variance in size, which may be attributed to their sparsity, as no error except bad deletion occurs

---

Lexical
Syntax
Conceptual
Error
Quality
Overall
μ
a
μ
a
μ
a
MUSS
0.81
1.23
0.45
0.64
0.97
1.73
0.66
1.03
1.00
1.04
1.77
1.66
T5 3B
0.24
0.56
0.18
0.38
0.65
1.92
0.34
0.97
0.39
0.62
0.76
1.15
T5 11B
0.22
0.44
0.17
0.92
0.61
1.78
0.36
1.30
0.32
0.71
0.71
1.51
Zero-shot GPT-3.5
1.32
1.40
0.67
0.65
0.17
0.38
0.34
0.53
1.75
1.55
2.10
1.60
Fow-shot GPT-3.5
1.41
1.43
0.57
0.53
0.15
0.39
0.25
0.46
1.80
1.49
2.11
1.60
Human
1.25
1.85
0.60
0.86
0.32
0.83
0.25
0.62
1.64
2.04
2.16
Table 7: Mean (1) and std. deviation (a) of average sentence-level SALSA sub-scores across systems. Human simplification may he interpreted as highly simplified (μ 2.04) and highly diverse (a 16).
a far greater number of positive edits when these corresponding errors occur. Broadly, we observe an inverse relationship between error and quality, As the enror score increases (a function of the sever- ity, frequency and size of errors), the quality must decrease.
performance and that overall simplification perfor- mance is often determined by a small number of high-impase edits.
Sentence Length Impacts Edit Frequency. Pre- vious linguistic annotation of the ASSET corpus (Cardon et al., 2022) reports that the number of modifications sentence does not correlate with input size. In Figure 13, we observe the same re- lationship on ASSET, however - because ASSET only represents simplifications of simpler sentences typically containing single idea when we ex- tend the analysis to the more complex SIMPEVAL dataset, we clear relationship between the edit distance and the number of transformations in simplifications across all systems. This also best exemplified by the split edit, which often signifies too many ideas are being contained within a single sentence. Figure 12 demonstrates the proportion of simplifications which exhibit split across sen- tence lengths and edit distance. While split edits within ASSET were generally low, the much longer SIMPEVAL simplifications almost guaranteed all systems performed sentence split. These find- ings highlight that performance measures should be anngth-agnostic, as to guarantee simplifications which simply contain more transformations due to a longer original sentence length are not arbitrarily rated as higher quality.
Smpppal
0.4
0.2
0.0
0.8
WT Spit
0.6
0.1
0.2
150
200 Original Sentence Length
Edit Distance
D 1 Split
Splits
3 Spitss
Figure 13: Proportion of sentences containing at least single split. Although ASSET has much lower fre- quency sentence splits (32%), a longer input sentence implies sentence split is more likely occur.
Increased Edits Enables, But Does Not Guar- antee Performance. Table 7 reports the mean and variance of sub-scores for the sentence-level S SALSA score across each system. Edit-level scor- ing addresses the frequent evaluation concern that conservative systems may maximize their score by performing minimal number of safe edits (Alva- Manchego et al., 2021). The qualitatively conser- vative simplifications of T5 and zero-shot GPT-3.5 often score low because they fail to make many edits. SALSA distinguishes the MUSS simplifica- tions with many successes, but more failures than other systems. We find the extent of sentence edit- ing is not heuristic, but is a prerequisite for high
We report the breakdown of con- stituent edits in structure and split edits in Figure 15. Split edits typically need to rewrite the conjunc- tion through inserting & deleting discourse tokens, while structure edits are typically performed some syntax transformation the existing sentence tree, more often requiring substituted or reordered to- kens.
Composite Edits.
SALSA Test Set. Figure 16 reports the frequency of quality and error edits on the novel SAL.SA test set systems. While adding control tokens to T5 substantially improves the frequency of edits, we

---

More Information
Less Information
Some Inform onton
Reorder
Sentence Spitt
Structure
All
MUSS
GPT:3.5
GPT-3.5
-3 -2-21 +011-249
10-11-2
-5-2-1 -0-1-2-2
-2-21 +0
10-1x219
-3-2- 124
Figure 14: Distribution of ratings of edits SIMPEVAL per-model. Compared to edit coverage distributions (Fig. 6), we see the same underlying relationship, but the difference in error and GPT vs. human quality is less exaggerated. This figure does not reflect the typically much longer human simplification spans and fine-tuned models' envor spans.
delation
edits, with the exception of GPT-4 making signif- icantly less conceptual edits. Manual analysis re- veals its conceptual edits are often sentence-level operations, which are not reflected edit counts. The LLaMA- based Alpaca and Vicuna demonstrate surprisingly strong performance despite their rel- atively small size and training setup, even outper- forming the tne-tuned simplification models.
62%
9%
rearder Insertion
41%
21%
subslitution
substitution
Structure Edits
Split Edits
Figure 15: Breakdown of composite edits by the makeup of their constituent tokens.
SALSA Dataset Statistics. We report full statistics on all 840 simplifications in Table 8. Similar FRANK (Pagnoni et al., 2021), we asked annota- tors to note edits that could not be annotated, and we observe less than 0.5% of edits were not cap- tured by one of our edit types. We consider the SALSA framework complete.
Edit
Edits
Tokens
Rating
Elaboration
1947
4996
0.25
Generalization
2644
10050
0.52
Ward Reorder
67
744
0.55
Component Reorder
823
7273
0.62
Quality
Sentence Split
1605
6759
0.71
Structune Change
2013
10614
0:48
Paraphrase
4394
18278
0.7
E Further Word-level QE Results
Red Deletion
5019
-0.66
Coreference
Repetition
107
572
We include test set word-level score on words in the original sentence, simplified sentence, and both sentences (same as Table 3) in Table 9. In the orig- inal sentence, only deletion edits are labeled. Thus, the performance in the original sentence column indicates the model's ability identify quality error deletion edits. The best-performing method, Ec-One achieves over 50% in both quality and er- ror Fl. For the simplified sentence, which contains substitution and insertion edits, the model delivers better quality F1 but experiences drop in error F1. This could be due to the higher proportion of error edits in deletion compared to substitution and in- sertion. In addition, the edit classification approach significantly improves the error F1 on the simpli- fied sentence, compared the tagging approaches, which reflects that tagging methods fail to capture multiple types of edits and those spanning both sentences like substitutions.
Contradiction
22
-1.75
Factual Ervor
75
421
-1.15
Frmr Exaltation
Irreleaait
64
347
-0.67
Bad Word Reorder
18
160
-0.61
Bed Component Reorder
243
2512
-0.81
Rad Struetane Change
291
1808
-0.49
Bad Sentence Split
143
749
-094
Complex Wording
597
2412
-045
Information Rewrite
1.99
877
0065
Grummar Error
71
372
0.96
Trivial Change
2983
8876
Table 8: Basic collected statistics on our edit-level SALSA annotations of SIMPEVAL, # tokens indicates tokens highlighted within each edit's spans.
find it still underperforms both MUSS and LLMs. Additionally, T5-11B makes surprising increase in error frequency relative the increase in the number of edits it performs relative T5-3B. Lan- guage models demonstrate a smooth increase in

---

Conceptual Quality
Syntox Quality
Lexical Quality
2.5
5.0
Ovalty bals (eel
3.0
2.0
4.5
4.0
2.5
5phn
3.5
2.0
Generatration
Brrecture
Component
Forophrose
1.5
Blsboration
1.0
level Beorder
2,0
1.0
Wordilovel
fecrder
1.5
0.5
1.0
0.5
Onl Cut Alpnce 78
Viaana
OPL.2.
Charagh GPF.4
15 118 Ctd
Cer
Alpnes 78
Vecna 73
OPI.25
Chorary
OPT.4
Cut onl
Alp Alpese
Vicaa!
Charapy
OPE.4
15
15
15
Conceptual Error
Syntax Error
Lexical Error
0.6
1.0
Broe Edil Sent
Bed
0.5
0.8
1.0
Delation
Bod Spit
Information Rewrita
0.6
0.8
Enorr
Inelevant
Tochol
0.4
Bod Component. facrder
Structure
0.6
Erer
Grammer
0.4
Contradiction
0.2
Bad Word level Reardar
Warding
0.2
Repettion
0.1
0.2
38
Ctd Chl
Alpecee 78
Vicama
OPL.
..5.3.5
Charayn
GPT.4
20 CHI
118 CHi
Alpaco
Vicmna 0r133
SPLA
28 CHr
15 Ovl
Alpaco 78
Vicaa
3.535
Charopy
GPZ-4
15
15
Figure 16: Overview of success and failure edits on the SIMPEVAL test set, collected using held-out simplification models as detailed in SC.1.
Original Sentence
Simplified Sentence
Overall Error
Method
Quality
Error
Ok
Quality
Error
Ok
Quality
Ok
End-to-end
Tag
36.53
52.20
94.79
73.83
13.85
90.18
67 .0
28.24
92.88
Tag-ML.
42.17
46.88
95.22
76.86
20.61
90.03
70.73
30.06
93.09
Two-stage (use word aligner to get edit informationn)
Tag-FI
33.06
46.04
94.76
76.75
20.00
90.66
69.09
30.37
93.04
Ec-Sep
45.70
49.18
93.19
71.50
25.66
89.43
64.87
36.15
91.56
Ec-One 50.53
54.63
93.30
74.97
28.15
90.04
68.77
39.50
91.91
Oracle (Ec-One)
84.51
75.88
99.33
89.45
65.26
97.03
88.31
69.44
98.35
Table Word-level scores different methods on SALSA test set, organized by the edit belonging the original simplified sentence.
F
Implementation Details
the reference-less COMETKIWI metric for machine translation (Rei et al., 2022). We modify their task setup of predicting binary quality labels for each output word y; E {OK, BAD) to a regression task using labels ÿ E [-3, 3]. corresponding each word rating in their SALSA annotations, as we find performs better than using binary three class labels in our preliminary study. Our regression task optimizes MSE loss on the word rating objec- tive, rather than Cross Entropy Loss. The training objective can be formalized as:
F1
Generating Simplifications ($3.1)
For all prompted models, we follow the hyperpa- rameters of S1MPEvAL2022 (Maddela et al., 2023), using temperature=1. 0 and top-p=0. 95. For all T5 variants, we train them on the Wiki-Auto corpus (Jiang et al., 2020) using 8 A40 GPUs for 8 epochs with batch size of 64. We use a learning rate of 3e-4 and AdamW (Loshchilov and Hutter, 2019) as the optimizer. For MUSS, we replicate the original setup (Martin et al., 2022). We use beam search with a beam size of 10 for these fine-tuned models.
1 (v (()))
Cxeen (0)
2
F22 Automatic Metries (85)
Baseline Automatic Metries. We use RoBERTa-large as the base model for BERTSCORE and the best available -mqm as COMET-MQM.
Loword(f) Σ 2
(v: ()))))
LENS-SALSA. Our implementation is based on
L(0) = Ax + Ana

---

BERTSCORE
COMET-MQM
LENS-SALS.
2018). We RoBERTa-Large the base model. We first pre-train on monolingual word align- ment datasets MultiMWA-Wiki and MultiMWA- Newsela from (Lan et al., 2021), and then fine-tune on the SALSA annotations in the training set. During both pre-training and fine-tuning stages, we perform learning rate sweep over (le-5, 2e-5, Se- 5, 8e-5) and train for 5 epochs, and save checkpoint at the end of every epoch. The highest evaluated checkpoint (pre-train for 2 epochs and fine-tune for epochs) is selected for testing, achieving 81 .03 Fl on the validation set.
BLEU
SART
LENS
Quality
Lexical
-0.855
0.030
0.015
0,086
0.289
0.284
Syntax
-0.117
0.008
0,024
0.206
Conceprual
4.240
0..17
0.325
-0.187
0.006
0.173
broor
Lexical
40259
-0.162
-0.13
-0.004
-0.059
0.015
Syntux
0.147
-0.094
-0.136
-0.073
-0.04
00013
Concepcual
40.128
40,009
-0.293
-0.169
40,016
0.062
All Eror
0.263
0..90
0.329
0.170
00.05
0.046
All Quality
40.001
0.056
-0.018
0.033
0.304
0.318
All Edits
40.286
40,035
-0.235
-0.129
0.266
0.336
Table 10: Pearson correlation between automatic met- rics and SALSA sub-scores on the validation set, with test set performance reported in Table 2.
On a side note, for the word that tokenized into multiple tokens, we use its first token for pre- diction.
where 0 and Aw weight word- and sentence-level losses. We experimented with custom weighting for edit ratings, but did not fine performance im- provements. For fine-tuning, we Au 0.9.
G Annotation Tutorial
We include screenshots to highlight the diversity of exercises and interactive elements our detailed interface tutorial.
The COMETKIWI design aggregates hidden states using a scalar mix module, and uses two feed forward networks for sentence- and word- optimize RoBERTa-large model on the sentence-level SIM- PEVAL training data used to train LENS (Maddela al., 2023), with the training setup using only single MSE loss to predict the sentence-level score (i.e., X - 1, A, 0). We follow COMETKIWI and freeze parameter updates for the RoBERTa en- coder for the first epoch and use learning rate of le-5 and 3e-5 for pre-training and fine-tuning re- spectively. We pre-train and fine-tune for 5 epochs, using the model with the highest validation set per- formance. We report the corresponding validation performance in Table 10.
level training. For pre-training,
F.3 Edit Classification ($6).
All experiments are conducted using 2 A40 GPUs. We use the AdamW optimizer with a weight decay = 0.01, and implement our models using the Hug- ging Face Transformers. Learning rate swept over le-5, 2e-5, 5e-5, 8e-5 for each method. Each run trained for eight epochs with batch of 32, This results training times less than five min- utes per run for tagging methods and less than 20 minutes per run for the edit classification methods. We perform an evaluation of the validation set at each training step and use the model that achieved the highest validation performance the test set. For the word alignment model used in the two- stage approach, we adopt the QA-based word aligner (Nagata et al., 2020), which formulates the task a SQUAD style (Rajpurkar et al.,

---

Text Simplification Annotation Tutorial
2
4
6
8
10 Quz
Introduction
Selecting Alignments
Annotating Deletions
Annotating Insertions
Annotating Substitutions
Annotating Splits
Annotating Ordering Changes
Annotating Structure Changes
Conclusion & Reference
Introduction
Welcome! this project you simplified sentences generated by Artificial Intelligence their quality.
qualification
you perform
must to:
the changes our Al made ei.e. "selecting spans")
Evaluate quality
change
Identify errors each change
Here's big picture of what we're doing.
epfend
tiis
Hit X/Y
bokon
Orginna Sentance Pluman Witter
A
Comment
medimaal musical form Gelesterleder, perilental sung sandering bands Flagellants.
northern
Sirpliied Serdence Mode Weldens
snslence belore angiilledd la mes Almdde
Golssterladerr was created from civll unrest Northem Italy thruugh songe sung traveiing
crnciations
Flagellants.
EDITS ANNOTATIONS (5/5)
Add Edit
Add
substilute
SpRAAtE
paraphrane
delete mediaval musical
pood deletion
We've freed edit for vestonee are "poodf
Insert though
bedt erival insertion
Modly or delete aundiaiion
delete pon tandaa
good deietion
substillue
wandering bands
travelingg bands
good hroap
Figure Landing page introducing annotators to each part of the task. The 10 stages organize different concepts in the SALSA typology.

---

Examples
Observe original sentence:
Original Sentence (Human Written::
Born into slavery in Virginia in 1856, Booker T. Washington became an influential African American leader at the outset of the Progressive Era.
Simplified Sentence (Human or Al Model Wrilten):
Booker T. Washington became an influential African American leader at the outset the Progressive Era.
This sentence communicates many different facts. Here are just a few:
Booker
Washington born Virginia
Booker
Washington was born in 1856
Booker
Washington was born into slavery
Booker
Washington was an influential leader
Booker
Washington was African American
Booker
Washington influential African American leader result of being into slavery
Booker
Washington
leader the beginning the Progressive Era
Hover over each piece of information to see which part the sentence could be deleted to remove that information from the sentence, As you can see, sentences which communicate many ideas may be hard to narrow down whether a span is significant.
this case, the main idea the sentence Booker T. Washington is an influential African American leader. Deletions are necessary text simplification, we just want to ensure this main idea still being communicated
Let's put it together with few other sentences:
Original Sentence (Human Written):
Like so many hyped books before it, The Midnight Library excited me and gave me pause.
Simplitled Sentence (Human Model Written):
The Midnight Library excited me and gave me pause.
Like many hyped books before it.
deleted significant
original sentence?
1 not at all
4
somewhat
much
Figure 18: Example interactive allowing annotators different spans understand different amounts of relevancy to the main idea of the sentence.

---

Phrase vs. Syntax Edits
you phrase edits only capture individual pieces of information small set of words change. When the creates edit which modifies sentence as whole, this must captured by a structural edit. Here's some examples of overlapping phrase and syntax edits:
Original Sentence (Human Written::
The amount that Phoenix spends criminal enforcement is difficult to quantify.,
in part because: the City does not classify arrests by housing status.
Simplified Sentence (Human or Al Model Written::
Because: the City of Pheonix, in part, refuses to use an individual's housing organize arrests, it has become hard to find out how much spends on policing
structure
impact simplicity phrase?
positive
no Impact
negative
Rate the severity:
Minoor
Someehat
This edit changes the order of the clauses in the sentence. Because the sentence trying communicate cause and effect (quantifying spending difficult because lack of data), edit moves the effect before cause.
Explanation:
structure
edit the phrase?
positive
no impact
negative
Rate the efficacy:
Minor
Somewhat
3 lot
Explanation:
This edit simply moves 'because' before 'in part'. This adds clarity the order information being presented.
Figure 19: One of the 100 sentence examples provided to annotators, highlighting different types of structure edits existing within the same sentence.


Reviews:
[{"review_id": "KJ8VQZroPI", "reviewer": "Reviewer_v4Lp", "summary": "", "questions": "1. The authors mentioned that there are 21 quality and error edit types. It's hard to follow where this number comes from -- there are 6 types of edit operation, each with 4 types of information change, organized into lexical, syntax, and conceptual edits. L:161 - “After being categorized into lexical, syntax, or conceptual edit families, we further classify each edit operation into 21 fine-grained success (quality), failure (error)” -- does this mean the previous edit operations and information change are separate from the 21? Appendix A provides Table 4 which provides some insight but the main paper does not reference this in the running text. Clarity on where the 21 types would help. \n\n2. The authors use only 6 annotators for their annotation. Also for the 6 annotators, what is the significance level? It would help to justify using only 6 annotators if their significance for the annotator agreement is high. \n\n3. The overall motivation for the text simplification task: Since there is a lot of subjectivity in how edits are made, curious if the authors referred to any Cogsci literature to understand how humans abstract details. This is an existing NLP task, but the introduction does not really motivate how this might be useful\n", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you sincerely for your review and your dedicated time to this. We hope that our answers address your concerns and questions.\n\n---\n\n**Responses to your comments:**\n\n_Writing and Structure_\n\nDue to the space constraints, we decided to discuss the most interesting analysis and experiments in the main text, while moving the extensive low-level details of our proposed framework for manual quality and error categorization including the definition of each edit type to the Appendix. If published, we will use the extra page to bring these details, which we are also excited about, back to the main text and move data collection (Section 6) to precede our analysis. It is an easy fix that will greatly improve the reading flow.\n\n> “The use case of the text simplification task and the data is unclear and not addressed.”\n\nThe use case of text simplification is well-documented in existing literature [1, 2, 3], aiming to make complex text more accessible for people including but limited to children, those with cognitive disabilities, and second-language learners. Given this established understanding of the task, we chose not to reiterate its use case in our paper. However, we agree on the value of establishing the context and will elaborate on the task’s use case in the final version.\n\nAs for data’s use cases, we outline them on lines 70-74 in the introduction and elaborate in Sections 3,4, and 5, which are providing a comprehensive analysis of modern large language models and automatic metrics, training a new reference-free automatic metric, and introducing the word-level quality estimation task.\n\n[1] Carroll, John, Guido Minnen, Yvonne Canning, Siobhan Devlin, and John Tait. Practical simplification of English newspaper text to assist aphasic readers. AAAI 1998.\n\n[2] J. D. Belder and Marie-Francine Moens. Text simplification for children. SIGIR 2010.\n\n[3] Sanja Stajner. Automatic Text Simplification for Social Good: Progress and Challenges. Findings of ACL 2021.\n\n> “There was not a sufficient interpretation of the edit analysis” / “did not try to hypothesize why humans were better at some edits, why models perform the way they are, etc”\n\nOur aim is to provide an empirical analysis of the behaviors we observed in humans and models’ simplifications while avoiding making indiscriminate hypotheses. In Section 3, \"Key Analysis,\" we do discuss where the data allows for interpretation, such as the elaborative tendencies of GPT-3.5 and the conservative nature of fine-tuned T5 models. We also provide additional findings in Appendix D on specific error types and simplified techniques used by humans and models. We believe this provides a balanced and thorough analysis.\n\n---\n\n**Answers to your questions:**\n\n> “The authors mentioned that there are 21 quality and error edit types … Clarity on where the 21 types would help.”\n\nThank you for pointing this out. The full list can be found in Table 4, as well as Figure 7, in the Appendix, which we will add more explicit references at suitable places in the final version.\n\n> “The authors use only 6 annotators for their annotation. For the 6 annotators, what is the significance level?”\n\nThese annotators are in-house annotators who went through extensive training and discussion sessions, and provide higher quality annotations than the crowdsourcing workers. The number of annotators is on par with related studies [4,5].\n\nTo provide more detail to the token-level Krippendorff's Alpha agreement reported in Table 3, we perform a bootstrap analysis with 1000 iterations and report the 95% CI below:\n\n| Edit             | Sub-type           | Kripp. α 95% CI  |\n|:-----------------|:-------------------|:----------------:|\n| Insertion        | More Information   | [0.392, 0.512]     |\n| Deletion         | Less Information   | [0.722, 0.777]     |\n| Substitution     | More Information   | [0.047, 0.256]     |\n|                  | Less Information   | [0.271, 0.347]     |\n| Reorder          | Word-level         | [0.075, 0.163]     |\n|                  | Component-level    | [0.363, 0.461]     |\n| Split            | Sentence Split     | [0.625, 0.696]     |\n| Structure        | Structure          | [0.214, 0.286]     |\n| Substitution     | Same Information   | [0.500, 0.560]     |\n\nWe also calculate the inter-annotator agreement on the overall simplification score that is derived from SALSA annotation (Appendix A.3). It achieves Krippendorff’s alpha of 0.38, compared favorably to the 0.32 by rank and rate, 0.25 by direct assessment, and 0.23 by Likert scale, as reported in [6].\n\n[4] Tanya Goyal, Junyi Jessy Li, and Greg Durrett. SNaC: Coherence error detection for narrative summarization. EMNLP 2022.\n\n[5] Philippe Laban, Jesse Vig, Wojciech Kryscinski, Shafiq Joty, Caiming Xiong, Chien-Sheng Wu. SWiPE: A Dataset for Document-Level Simplification of Wikipedia Pages. ACL 2023. \n\n[6] Mounica Maddela, Yao Dou, David Heineman, Wei Xu. LENS: A Learnable Evaluation Metric for Text Simplification. ACL 2023.\n\n> “Curious if the authors referred to any Cogsci literature to understand how humans abstract details.”\n\nOur framework primarily builds upon existing linguistic analyses of text simplification [7, 8]. While there is a subfield called cognitive simplification [9, 10], we didn’t delve into that area in this paper. We appreciate this suggestion and agree that connecting our work to cognitive science would be an interesting direction for future research.\n\n[7] Sanja Štajner. New data-driven approaches to text simplification. Ph.D. thesis, University of Wolverhampton. 2016.\n\n[8] Rémi Cardon, Adrien Bibal, Rodrigo Wilkens, David Alfter, Magali Norré, Adeline Müller, Watrin Patrick, and Thomas François. Linguistic Corpus Annotation for Automatic Text Simplification Evaluation. EMNLP 2022.\n\n[9] Shira Yalon-Chamovitz, Ruth Shach, Ornit Avidan-Ziv, and Michal Tenne Rinde. The call for cognitive ramps. Work, 53(2):455–456. 2016.\n\n[10] Chamovitz, Eytan, and Omri Abend. Cognitive Simplification Operations Improve Text Simplification. CoNLL 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The authors thoroughly analyze the types of edits, and their categories, and further their quality/error analysis. It is interesting to analyze how humans make edits, what types of content are worth preserving, and what content is redundant. Further, this is a contrast against the numerous baselines compared. \n", "reasons_to_reject": "The authors collected a large-scale annotation of fine-grained edit information for the task of text simplification. However, the use case of the text simplification task and the data is unclear and not addressed. The flow of the paper is quite confusing and the content can be restructured better. The authors did analyze the types of errors by models and humans, but did not try to hypothesize why humans were better at some edits, why models perform the way they are, etc. There was not a sufficient interpretation of the edit analysis. ", "questions_to_authors": "1. The authors mentioned that there are 21 quality and error edit types. It's hard to follow where this number comes from -- there are 6 types of edit operation, each with 4 types of information change, organized into lexical, syntax, and conceptual edits. L:161 - “After being categorized into lexical, syntax, or conceptual edit families, we further classify each edit operation into 21 fine-grained success (quality), failure (error)” -- does this mean the previous edit operations and information change are separate from the 21? Appendix A provides Table 4 which provides some insight but the main paper does not reference this in the running text. Clarity on where the 21 types would help. \n\n2. The authors use only 6 annotators for their annotation. Also for the 6 annotators, what is the significance level? It would help to justify using only 6 annotators if their significance for the annotator agreement is high. \n\n3. The overall motivation for the text simplification task: Since there is a lot of subjectivity in how edits are made, curious if the authors referred to any Cogsci literature to understand how humans abstract details. This is an existing NLP task, but the introduction does not really motivate how this might be useful\n", "paper_topic_and_main_contributions": "The task addressed in this paper is the text simplification task. The authors develop an interactive interface to collect annotations of simplifications generated by humans and various models. Through this interface, they were able to collect fine-grained information about the edit operations and their quality. With this data, they trained an evaluation metric LENS-SALSA. Further they trained models to predict the word-level quality in a supervised fashion, previously missing due to lack of appropriate annotated data.  \n", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. Would move section 6 early on to get an understanding of the data used. Additionally, short blurbs on modeling details for each of the LMs (prompting and non-prompting) would help. Are the models fine-tuned? Zero-shot? Moving this earlier would help readers grasp things better. add short details on how models are prompted\n\n2. Currently, the flow of the paper is confusing to follow -- a better way to categorize and restructure to present the edit types, information changes, style of edits, and quality error would help.\n"}}, {"review_id": "6l6zZL8Gud", "reviewer": "Reviewer_AAK9", "summary": "", "questions": "- If I got it right, only spans with an edit get an efficacy or severity judgment. Then, how do you get a score (ranging from -3 to 3) for all words when training LENS-SALSA ?\n- I noticed that the SALSA interface displays the whole original and simplified sentences. In such setting, you did not discuss to which extent it is possible, for a human annotator, to assess the  efficacy or severity of a given simplification without being influenced by the rest of the sentence. In my opinion, it is not so obvious. Could you please elaborate a bit about this matter. ", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you sincerely for your review and your dedicated time to this. We hope that our answers address your concerns and questions.\n\n---\n\n**Responses to your comments:**\n\n> “The paper is really really dense. The authors have a lot to say – which is good -, but, in my opinion,it does not fit the conference format.”\n\nWe appreciate your acknowledgement of the depth of our paper. Our paper is centered around a fine-grained evaluation framework, while offering both analysis and modeling. We aim for a high-quality conference publication that provides a comprehensive understanding and answers multiple research questions, so we choose not to slice our paper into two. We will also take advantage of the extra page for publication, as detailed below.\n\n*Writing and Structure*\n\nDue to the space constraints, we made a tradeoff: discussing the most interesting analysis and experiments in the main text, while moving the extensive details of our proposed framework including the motivation and definition of each edit type to the Appendix. If published, we will use the extra page to bring these details, which we are also excited about, back to the main text and move data collection (Section 6) to precede our analysis. It is an easy fix that will greatly improve the reading flow.\n\n> “Several scholars have recently stressed that we need to consider the specific needs of the different audiences … the authors seem to overlook this fact and they propose an evaluation metric that should fit all types of texts and audiences, even though their training data has been annotated by 6 undergraduates with no experience of any specific audiences.”\n\nWe fully acknowledge the importance of audience-specific needs and have mentioned this in lines 288-290 and 1458-1463 of our paper. Our SALSA framework is designed as a flexible framework that can be adapted to various use-cases. One important thing to notice is that the complex sentences in our dataset are significantly complex, averaging 37.3 words in length. Such complexity poses challenges even for our undergraduate annotators, making our annotations and evaluation metric broadly applicable to the general public who may also find such texts difficult to understand.\n\n> “Why are morphological transformations (such as tense modification) included in the “Structure” category? Is the “Conceptual” category better than the traditional “Semantic” one? What about discourse aspects; are they overlooked?”\nWe classify morphological transformations like tense modifications under the \"Structure\" category because they often entail changes that go beyond individual words and affect sentence syntax, e.g., multiple verbs within a sentence. It also helps annotators to discern cases where a word is substituted with a simpler synonym, but in a different tense—such as changing \"anticipate\" to \"expected.\"\n\nOur terminology of “Conceptual Simplification” follows from recent work in simplification [1,2] and is interchangeable with “Semantic Simplification.”\n\nDiscourse preserving or modifying edits (as defined by Siddharth, 2006 [3]) are captured in our “Structure” edit type, as these often require either syntactic modifications or multi-edit changes. To improve the identification of these structure/discourse edits, annotators further labeled them into the 5 types listed in Table 5.\n\n[1] Sanja Štajner. New data-driven approaches to text simplification. Ph.D. thesis, University of Wolverhampton. 2016.\n\n[2] Sian Gooding. On the Ethical Considerations of Text Simplification. SLPAT. 2022.\n\n[3] Advaith Siddharthan. Syntactic Simplification and Text Cohesion. Res Lang Comput. 4, 77–109. 2006.\n\n---\n\n**Answers to your questions:**\n\n> “How do you get a score (ranging from -3 to 3) for all words when training LENS-SALSA?”\n\nThe scores for words that are part of an edit correspond to the severity (negative) or efficacy (positive) rating of that particular edit, which falls within the [-3, 3] range. Words that are not part of any edit are assigned a score of 0.\n\n> “To which extent it is possible, for a human annotator, to assess the efficacy or severity of a given simplification edit without being influenced by the rest of the sentence?”\n\nThe capability to assess each simplification edit individually stems from our edit selection process, where each edit is selected so that it can be evaluated independently of the rest of the sentence. To achieve this, we implement the following measures:\n1. We allow for overlapping edits and instruct annotators to create an edit for each individual operation (e.g., a single span could contain a re-order and substitution, yet both operations are selected and evaluated independently).\n2. Our composite edits, structure and split edits, are intentionally designed to group alike edits to be evaluated together.\n3. Our annotator adjudication step ensures edits are selected with the correct scope (i.e., each edit represents only one simplification operation).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This paper includes several innovative ideas that might help the field of ATS to progress. For instance, combining finer annotations of simplification operations (bad or good) with human judgments about the impact of those annotations is very interesting. In addition, this combined information then allows to train an evaluation metric that could pay more attention to successes and failure of ATS systems than previous metrics.\n- I believe this paper could be an influential one, if make simpler to read (with no pun intended)\n- The methodology used is solid and I have not noticed any major issue.", "reasons_to_reject": "- The paper is really really dense. The authors have a lot to say – which is good -, but, in my opinion, it does not fit to conference format. There are about 18 pages of appendices, which do not only include supplementary material, but also important elements of the paper, such as a discussion about the annotation process and inter-annotator agreement scores, parts of the description of the SALSA-LENS model that are necessary to understand its principles, etc.\n- In addition, I found the paper poorly structured. It starts by the description of the framework, which is fine, but the framework involves the description and motivation of the typology used and the data annotated. However, the typology is not motivated and its description is done in the appendices. The reader has to wait for section 6 to see the data introduced. It is also not very clear why the paper reports experimentation about SALSA adapted to QE, as this idea falls outside the proposed framework. Maybe this section could be kept for a future publication.\n- The typology used is not motivated in relation to previous work in the field of ATS. Why are morphological transformations (such as tense modification) included in the “Structure” category? Is the “Conceptual” category better than the traditional “Semantic” one? What about discourse aspects; are they overlooked?\n- Several scholars – including Stajner (2021) or Rennes (2022) ) - have recently stressed that we need to consider the specific needs of the different audiences for which ATS could be used. In this paper, the authors seem to overlook this fact and they propose an evaluation metric that should fit all types of texts and audiences, even though their training data has been annotated by 6 undergraduate with no experience of any specific audiences.   \n\nRennes, E. (2022). Automatic Adaptation of Swedish Text for Increased Inclusion (Doctoral dissertation, Linköping University Electronic Press).\nŠtajner, S. (2021). Automatic text simplification for social good: Progress and challenges. Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, 2637-2652.\n", "questions_to_authors": "- If I got it right, only spans with an edit get an efficacy or severity judgment. Then, how do you get a score (ranging from -3 to 3) for all words when training LENS-SALSA ?\n- I noticed that the SALSA interface displays the whole original and simplified sentences. In such setting, you did not discuss to which extent it is possible, for a human annotator, to assess the  efficacy or severity of a given simplification without being influenced by the rest of the sentence. In my opinion, it is not so obvious. Could you please elaborate a bit about this matter. ", "paper_topic_and_main_contributions": "This article addresses the complex challenge of evaluating automatic simplification systems. The paper is very ambitious and proposes several interesting contributions. First, it introduces a dataset which, to my knowledge, does not exist, namely a set of original and simplified sentences (produced by humans, but also by different LLMs) in which the good and bad simplifications have been annotated by 6 annotators. The annotators also judged, for each edit operation, its efficacy (for good transformations) or its severity (for bad ones). Second, the authors have integrated these assessment principles, including a typology of 21 linguistic operations, into a framework for manual evaluation of ATS, called SALSA. This framework aims at supporting more reliable human evaluation of ATS systems in the future (but this has not been tested). Finally, building on QE research on evaluation, they used the human judgments in their annotated data to train SALSA-LENS, a reference-free simplification metric that perform better, on their test set, than the other automatic metrics in the field.\n\nTo conclude, I believe this paper is very valuable for the field, as it is packed with a lot of interesting ideas and results, but it really suffers from a lack of space to expose all theses ideas in a clear way and from a poor structure. It could benefit from being rewritten and submit as a journal paper (because the authors would get more space).   ", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "Cumbicus et al. (2021) also investigated the successes and failures of ATS systems, with a very different approach. \n\nCumbicus-Pineda, O. M., Gonzalez-Dios, I., & Soroa, A. (2021). Linguistic Capabilities for a Checklist-based evaluation in Automatic Text Simplification. In CTTS@ SEPLN."}}, {"review_id": "yNlKnuEa2V", "reviewer": "Reviewer_6XdU", "summary": "", "questions": "Question A: I understand that COMET-MQM is an evaluation metric for translation that takes source, hypothesis, and reference (source and hypothesis or reference are different languages) as inputs and gives a score.  How did you adapt COMET-MQM for the simplification task?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you sincerely for your review and your dedicated time to this. We hope that our answers address your concerns and questions.\n\n---\n\n**Responses to your comments:**\n\n> “I feel that the framework has not been fully evaluated.”\n\nThe nature of a human evaluation framework in our work makes it less straightforward to be evaluated, in comparison to datasets and models that can be easily evaluated. Many existing NLP conference papers [1, 2] on human evaluation frameworks also did not and could not directly evaluate the evaluation framework itself. However, we did take the following measures to ensure our framework offers a solid and robust assessment of text simplification:\n1. Our framework builds upon existing linguistic analyses of text simplification [3, 4] and has been refined through rounds of pilot annotations.\n2. The annotation process follows the common practices for fine-grained evaluations [1, 2].\n3. We include an adjudication step for span selection to resolve disagreements.\n\n[1] Yao Dou, Maxwell Forbes, Rik Koncel-Kedziorski, Noah A. Smith, and Yejin Choi. Is GPT-3 text indistinguishable from human text? scarecrow: A framework for scrutinizing machine text. ACL 2022.\n\n[2] Tanya Goyal, Junyi Jessy Li, and Greg Durrett. SNaC: Coherence error detection for narrative summarization. EMNLP 2022.\n\n[3] Sanja Štajner. New data-driven approaches to text simplification. Ph.D. thesis, University of Wolverhampton. 2016.\n\n[4] Rémi Cardon, Adrien Bibal, Rodrigo Wilkens, David Alfter, Magali Norré, Adeline Müller, Watrin Patrick, and Thomas François. Linguistic Corpus Annotation for Automatic Text Simplification Evaluation. EMNLP 2022.\n\n> “I think there needs to be a comparison of the SALSA evaluation process and the process of annotating the 21 edit types directly”\n\nWe designed SALSA with a decision tree to reduce the cognitive load on a single annotation, as 21 linguistic types can be overwhelming when annotated on every edit. Our decision tree-based typology is also grounded in prior work [5] that organizes simplification into atomic operations. To make sure annotators understand and differentiate each edit type, we included explicit training for each type in our tutorial.\n\n[5] Regina Stodden and Laura Kallmeyer. TS-ANNO: An Annotation Tool to Build, Annotate and Evaluate Text Simplification Corpora. ACL 2022.\n\n> “The evaluation of the framework is also affected by the reliability of the data collected in this study.”\n\nAs noted in Appendix C, edit-level selection is an inherently difficult task due to the high granularity of the framework. But our token-level agreement falls within a range similar to other fine-grained evaluation framework [6]. To further improve reliability, we perform an adjudication step (L518-L523) to resolve the disagreements in span selection.\n\nTo supplement the reliability of our annotations, we calculate the inter-annotator agreement on the overall simplification score that is derived from SALSA annotation (Appendix A.3). It achieves Krippendorff’s alpha of 0.38, compared favorably to the 0.32 by rank and rate, 0.25 by direct assessment, and 0.23 by Likert scale, as reported in [7].\n\n[6] Tanya Goyal, Junyi Jessy Li, and Greg Durrett. SNaC: Coherence error detection for narrative summarization. EMNLP 2022.\n\n[7] Mounica Maddela, Yao Dou, David Heineman, Wei Xu. LENS: A Learnable Evaluation Metric for Text Simplification. ACL 2023.\n\n---\n\n**Answers to your questions:**\n\n> “How did you adapt COMET-MQM for the simplification task?”\n\nIn Table 1, We use COMET-MQM with no changes to its training or model, using the two human-written references as simplification references. We decided to include this learnable metric in our evaluation as it did surprisingly well against baselines like BLEU and SARI.\n\n*Presentation Improvements*\n\nThanks for your suggestions! Due to the space constraints, we made a tradeoff: discussing the most interesting analysis and experiments in the main text, while moving the extensive details of our proposed framework including the motivation and definition of each edit type to the Appendix. If published, we will use the extra page to bring these details, which we are also excited about, back to the main text and move data collection (Section 6) to precede our analysis. It is an easy fix that will greatly improve the reading flow.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This paper proposes a useful framework for evaluation for simplification tasks. \n- It offers a comprehensive analysis of the collected data by SALSA.\n- In addition to analyzing simplification models, they also demonstrate the analysis of automatic evaluation metrics and their applied use in QE tasks.\n", "reasons_to_reject": "While this paper shows many ways to use the data collected by the proposed framework, I feel that the framework has not been fully evaluated. For example, I think there needs to be a comparison of the SALSA evaluation process and the process of annotating the 21 edit types directly. Consequently, the evaluation of the framework is also affected by the reliability of the data collected in this study. In fact, some edit types have a low agreement (Table 3), which also affects the reliability of the various analyses based on that data (Section 3,4,5).", "questions_to_authors": "Question A: I understand that COMET-MQM is an evaluation metric for translation that takes source, hypothesis, and reference (source and hypothesis or reference are different languages) as inputs and gives a score.  How did you adapt COMET-MQM for the simplification task?", "paper_topic_and_main_contributions": "This paper proposes an annotation framework for evaluation of simplification tasks, called SALSA. Using SALSA, 19K annotations are collected. The authors demonstrate the usefulness of the framework by showing how some data can be used. First, they analyze several simplification models and human edits. Second, they used annotations to evaluate automatic evaluation metrics such as BLEU. Moreover, they propose a new automatic evaluation metric, Lens-SALSA, which is trained on the annotation data. Finally, they show that the data can be used for word-level quality estimation tasks.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "I think there is room for improvement in the structure of the paper. For example, the data collection process (Section 6)  is presented later in the paper but should be explained before the key analysis (Section 3). Also, there is an explanation of the 21 edit types in the Appendix, but this information should also be before Section 3, otherwise, it is not easy to understand the details of the analysis."}}, {"review_id": "9nHRxHaxGr", "reviewer": "Reviewer_nj2X", "summary": "", "questions": "1. It is unclear why it is claimed in Figure 3 that MUSS performs paraphrasing at a human rate.\n2. It might be helpful to clarify whether the MUSS model is supervised or unsupervised in the main text.\n3. L188: Unclear whether and how this definition of edit coverage differs from the equation presented in Figure 5 and it is also unclear how to read Figure 5. Could you explain why for example in more information, muss has a higher overall proportion than other models?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you sincerely for your review and your dedicated time to this.\n\n---\n\n**Responses to your comments:**\n\n> “It might be useful to define what is considered a successful or a failed edit as observed in the data or as was considered by the authors more directly.”\n\nThank you for the suggestion. Our definition of success and failure is based on the general notion of simplicity, which is whether an edit makes the sentence simpler or more complex. Our 21 specific categories of successful or failed edits are refined through initial pilot rounds of annotations.\n\n---\n\n**Answers to your questions:**\n\n> “Why it is claimed in Figure 3 that MUSS performs paraphrasing at a human rate.”\n\nThank you for pointing this out. This point was intended to contextualize Figure 3 w.r.t Figure 5, which shows the edit coverage for paraphrasing to be similar to humans. We will clarify this statement in our final version.\n\n> “It might be helpful to clarify whether the MUSS model is supervised or unsupervised in the main text.”\n\nWe will clarify this in our final version.\n\n> “How the definition of edit coverage at L188 differs with the equation in Figure 5 and how to read Figure 5. Could you explain why for example in more information, muss has a higher overall proportion than other models?\n\nL188 should be _the length of each edit in proportion to the total length of the complex sentence and simplification_ as shown in the equation. We will correct it in the final version.\n\nRegarding Figure 5, each column should be considered independently, as they are rescaled to facilitate comparison between different models.\n\nMUSS having a higher overall proportion in the \"more information\" category is because it makes longer and more error edits. We observe that MUSS often generates its own interpretations that are not well-grounded in the context.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper proposes a new evaluation framework and a dataset as well as a word-level QE task for text simplification which could help significantly advance research in TS evaluation beyond coarse-grained Likert-scale rating.\n2. The proposed reference-free metric achieves better correlation with human judgments on both sentence and word-level quality estimation tasks on the collected benchmark,\n3. A thorough evaluation is conducted to study how human-based edits differ from model-generated edits.", "reasons_to_reject": "1. While the evaluation focuses on the objective aspect of what constitutes a successful or a failed edit, the framework still does not give much information about who are these simplifications useful for as what constitutes simplified really depends upon the target audience. This has been acknowledged in L288 but it might be useful to define what is considered a successful or a failed edit as observed in the data or as was considered by the authors more directly.\n2. Not a reason for rejection and really appreciate all the hard work put into this work but that also makes it difficult to follow the core contributions, breaking this paper into two would greatly benefit the readers as a lot of interesting and relevant content is currently in the appendix material. For example Figure 7 really helps ground success and failure edits which if it is indeed the core contribution should ideally be in the main text. For word-level QE results, table 9 is in appendix material.  ", "questions_to_authors": "1. It is unclear why it is claimed in Figure 3 that MUSS performs paraphrasing at a human rate.\n2. It might be helpful to clarify whether the MUSS model is supervised or unsupervised in the main text.\n3. L188: Unclear whether and how this definition of edit coverage differs from the equation presented in Figure 5 and it is also unclear how to read Figure 5. Could you explain why for example in more information, muss has a higher overall proportion than other models?", "paper_topic_and_main_contributions": "The paper proposes a new evaluation framework for evaluating simplified sentences using 21 linguistically grounded edit types. The authors collect 19k high-quality annotations on 840 simplified outputs from multiple systems where the quality is ensured by training sessions of 6 native U.S. undergraduate students. A new word-level quality estimation task with baseline approaches is proposed where the proposed reference-free metric LENS-SAMSA is shown to outperform other TS evaluation metrics. Their findings show that GPT generates more diverse edits than human-based simplification but also introduces more errors compared to human reference simplification.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Section 6 should come earlier as it helps ground the nature of the document simplified, without this information it was hard to follow the results discussed before."}, "extra_text": {"Missing_References": "A reference-free metric for TS has been proposed in prior work. Please correct L624 and add a citation to Kriz, Reno, Marianna Apidianaki, and Chris Callison-Burch. \"Simple-qe: Better automatic quality estimation for text simplification.\" arXiv preprint arXiv:2012.12382 (2020)."}}]

Author rebuttal:

Meta-review: The authors built an interactive interface to collect annotations of simplifications made by humans and models. Using this data, they developed an evaluation metric called LENS-SALSA and trained models to predict word-level quality. All the reviewers unanimously commend the authors for their meticulous examination of edit types, their categorization, and the extensive analysis of the data collected by SALSA. 

However, the major flaw of the paper, as pointed out by the reviewers, is the excessive volume of information it presents while pushing substantial and pertinent content to the appendix due to space constraints. Reviewers express dissatisfaction with the paper's intricate structure and concur, as humorously noted by reviewer AAK9, that it requires simplification.
I am confident that the authors have the capacity to implement revisions to the paper within the provided timeframe, including the extra page. My strong recommendation, however, is for them to earnestly consider the invaluable guidance from the reviewers regarding the paper's restructuring.
Decision: Accept-Main