LENS : A Learnable Evaluation Metric for Text Simplification
Year: 2023

Abstract:
Training learnable metrics using modern language models has recently emerged as a promising method for the automatic evaluation of machine translation. However, existing human evaluation datasets for text simplification have limited annotations that are based on unitary or outdated models, making them unsuitable for this approach. To address these issues, we introduce the SIMPEVAL corpus that contains: SIMPEVALPAST, comprising 12K human ratings on 2.4K simplifications of 24 past systems, and SIMPEVAL2022, a challenging simplification benchmark consisting of over 1K human ratings of 360 simplifications including GPT3.5 generated text. Training on SIMPEVAL, we present LENS, a Learnable Evaluation Metric for Text Simplification. Extensive empirical results show that LENS correlates much better with human judgment than existing metrics, paving the way for future progress in the evaluation of text simplification. We also introduce RANK & RATE, a human evaluation framework that rates simplifications from several models in a list-wise manner using an interactive interface, which ensures both consistency and accuracy in the evaluation process and is used to create the SIMPEVAL datasets.

heading: 1 Introduction
text: Text simplification is a text-to-text generation task that aims to make a text easier to read while preserving its original meaning (Saggion, 2017). Automatic evaluation of text simplification is challenging because a sentence can be simplified in many ways, such as paraphrasing complex words, deleting insignificant information, and splitting long sentences into shorter ones. An ideal automatic metric should accommodate these diverse choices while capturing semantic similarity and fluency. However, existing metrics such as SARI (Xu et al., 2016) and BERTScore (Zhang et al., 2020) struggle
∗Equal contribution.
Original Sentence from our SimpEvaL2022: The book is illustrated with more than 200 newly commissioned color photographs by the British architectural photographer Morley von Sternberg, as well as older images where needed to illustrate buildings that have been significantly altered or demolished.
New Ben chmark! New Ben chmark!
to capture all the aspects and achieve a high correlation with human evaluation (Alva-Manchego et al., 2021) (see Figure 1). These metrics fail even more when evaluating high-quality systems that have close performance, calling for a more robust and accurate metric for text simplification.
Prior work on machine translation (Rei et al., 2020; Sellam et al., 2020) has seen the success of using language models as an automatic metric by training them on human judgments, such as Direct Assessment (DA) that rates generations on a 0-100 scale. However, existing human evaluation datasets (Alva-Manchego et al., 2021; Sulem et al., 2018) for text simplification are not suitable for training metrics because they include a limited number of annotations or systems. Besides, these datasets do not include state-of-the-art generation models such
16383
as GPT-3.5 (Ouyang et al., 2022) that have been shown to generate human-like quality text (Dou et al., 2022; Goyal et al., 2022).
In this work, we introduce LENS, a Learnable Evaluation Metric for Text Simplification. LENS is the first supervised metric for the task and uses an adaptive ranking loss to promote fair comparison of simplifications that have undergone different edits (i.e., splitting, paraphrasing, and deletion). To train LENS, we collect 12K human judgments on 2.4K simplifications by 24 simplification systems from the literature, which we name as the SIMPEVALPAST dataset. We also create SIMPEVAL2022 to evaluate LENS and other metrics in a realistic and challenging setting of assessing simplifications by state-of-the-art language models on the more recent, complex, and longer sentences published on Wikipedia after Oct 22nd, 2022. SIMPEVAL2022 contains over 1K human ratings on 360 simplifications generated by 4 SOTA models, including GPT-3.5, and 2 humans.
Empirical experiments show that LENS achieves a higher correlation of 0.331 with human ratings on SIMPEVAL2022, which is more than twice as high as the correlation scores of 0.112 and 0.149 by BERTScore and SARI, respectively. We further demonstrate that incorporating LENS into decoding process, using minimum Bayes risk framework (Fernandes et al., 2022), can directly improve the automatic text simplification system’s performance. We expect that our data collection method, including RANK & RATE, a list-wise human evaluation interface, can be easily adapted to other text generation tasks.

heading: 2 Background
text: Issues of Existing Automatic Metrics. SARI (Xu et al., 2016) is the most commonly used metric for text simplification that computes F1/precision scores of the n-grams inserted, deleted, and kept when compared to human references. As SARI measures n-gram string overlap, it penalizes simplifications that are synonymous to the reference but uses different words (see first two examples in Figure 1). Alva-Manchego et al. (2021) showed that BERTScore (Zhang et al., 2020), which measures similarity based on BERT (Devlin et al., 2019) embeddings, is better at capturing semantic similarity between the simplification system’s outputs and the references. However, it fails to penalize conservative systems that make trivial or no changes to the
input as illustrated in Figure 2. This figure shows average metric scores by SARI and BERTScore for a conservative system that copies the input (Copy), an oracle system of human simplification (Human)1, and three disfluent corrupted systems that drop 10% of the input words (Drop), scramble 5% of the input words (Scramble), or insert a period in the middle (Split). BERTScore ranks some conservative or corrupted systems above human simplifications. SARI penalizes the conservative system because it focuses on the edits performed by the system but ranks the disfluent systems above the conservative system. An ideal automatic metric for text simplification should capture both semantic similarity and the edits performed by the system.
Lack of Human Evaluation Data. There is a lack of suitable human evaluation datasets for finetuning pre-trained language models for simplification evaluation. Alva-Manchego et al. (2021) released 0-100 continuous scale ratings on fluency, meaning preservation, and simplicity for only 600 simplifications sampled from six systems. Sulem et al. (2018) released 5-point Likert scale ratings on the same three dimensions for 1,960 simplifications generated by different configurations of six systems. Both datasets are relatively small and cover too few different system designs. For example, they do not include the newer state-of-the-art systems (Sheang and Saggion, 2021; Martin et al., 2022) that are based on T5 (Raffel et al., 2020) and other large pre-trained language models.
1We use SIMPEVAL2022 (§3.1) for the analysis. Out of the two human simplifications, we randomly selected one as the oracle and the other as the reference to compute metric scores.

heading: 3 Automatic Evaluation Metric
text: To tackle the challenge of limited human evaluation data, we curate SIMPEVAL, a corpus containing over 13K human judgements on 2.8K simplification from 26 systems. This facilitates the training and evaluation of LENS (A Learnable Evaluation Metric for Text Simplification), the first supervised automatic metric for text simplification evaluation. In this section, we first describe the creation of SIMPEVAL datasets in §3.1 and then LENS in §3.2.

heading: 3.1 Collecting Human Judgements
text: We collect SIMPEVALPAST, containing 12K human ratings on 2.4K simplifications from 24 systems on sentences from TurkCorpus (Xu et al., 2016), to train LENS. For evaluating LENS and other simplification metrics, we create SIMPEVAL2022 that consists of 1,080 human ratings on 360 simplifications from both humans and SOTA models, including GPT-3.5. It features more complex sentences from Wikipedia written after Oct 22nd, 2022, very recent to the time we conduct the experiments, to reduce the risk of “data contamination” (i.e., appearing in the training data of LLMs) and serve as a more challenging test bed for large language models. Table 1 shows the summary of both datasets.
A Diverse Set of Simplification Systems. We consider the following systems (further details in Appendix C): (i) two GPT-3.52 outputs under zero-shot and 5-shot settings; (ii) eight fine-tuned Transformer-based systems of varied sizes and parameter settings (Sheang and Saggion, 2021; Raffel et al., 2020; Martin et al., 2020; Maddela et al., 2021); (iii) three supervised BiLSTM-based systems that use vanilla RNN, reinforcement learning (Zhang and Lapata, 2017), or explicit editing (Dong et al., 2019); (iv) one unsupervised and one semisupervised system utilizing auto-encoders (Surya et al., 2019); (v) two systems that apply statistical machine translation approaches to simplification (Wubben et al., 2012; Xu et al., 2016); (vi) a rule-based system (Kumar et al., 2020); (vii) three hybrid systems that combine linguistic rules with data-driven methods (Narayan and Gardent, 2014; Sulem, 2018; Maddela et al., 2021); (viii) two naive baselines that copy the input or scramble 5% of the input words; and (ix) six human-written simplifications, including two from ASSET (Alva-Manchego
2Specifically, we use the text-davinci-003 model which is the most recent variant with 175B parameters. We provide the prompts we used in Appendix D.2.
et al., 2020), one from TurkCorpus (Xu et al., 2016), one from Simple Wikipedia that were automatically aligned (Kauchak, 2013), and two newly written by our trained in-house annotators.
Complex Sentences Selection. For SIMPEVALPAST, we sample 100 complex sentences from the test set of the widely-used TurkCorpus (Xu et al., 2016) and ASSET (Alva-Manchego et al., 2020) evaluation benchmarks, which share the same set of complex sentences but have different human simplifications in terms of edit operations. ASSET contains 10 human references for each complex sentence that are used to train LENS. GPT is pre-trained on a vast amount of web text, and
the sentences in TurkCorpus/ASSET are derived from the 12-year-old Parallel Wikipedia Simplification corpus (Zhu et al., 2010), which may have already been encountered by GPT-3.5. To address this “data contamination” issue and provide a more challenging benchmark for state-of-the-art models, we manually select 60 long, complex sentences covering recent world events such as the Twitter acquisition and World Cup for SIMPEVAL2022.3 These sentences are from new revisions or articles added to Wikipedia between October 22nd and November 24th, 2022. They have an average length of 38.5 tokens, much longer than the 19.7-token average of TurkCorpus and ASSET. Data Annotation. We ask in-house annotators to rate each simplification on a single 0-100 overall quality scale using our RANK & RATE framework (more details in §6). We provided an extensive tutorial and two training sessions to the annotators, which involved rating system outputs for five input sentences (screenshots in Appendix F). We periodically inspected the annotations to prevent the deterioration of quality over time. All annotators are English speakers who are university students. Each simplification in SIMPEVALPAST receives 5 ratings, while each simplification in SIMPEVAL2022 is rated by 3 annotators. We follow WMT (Akhbardeh et al., 2021) to normalize the raw scores by the mean and standard deviation of each rater, also known as the z-score, which are later used to train LENS. The inter-annotator agreement for SIMPEVALPAST is 0.70 using the interval Krippendorff’s α on z-scores, and 0.32 for SIMPEVAL2022, partly because it contains GPT-3.5 outputs that are quite competitive with human. Both are considered fair to good agreement (Krippendorff, 2004).

heading: 3.2 A New Learnable Metric – LENS
text: Given an input text c, the corresponding system output s, and a set of n references R = {r1, r2, . . . , rn}, LENS produces a real-valued score zmax = max1≤i≤n(zi) that maximizes over the quality scores zi of s in regards to each reference ri. Our model encodes all texts into vectors (c, s, ri) using Transformer-based encoders such as RoBERTa (Liu et al., 2019), then combines them into an intermediate representation H = [ s; ri; s ⊙ c; s ⊙ ri; |s − c|; |s − ri| ]
3For example, "Musk stated that Twitter Blue’s pricing would be raised to around US$8.00 per-month, and include reduced advertising on the Twitter service, the ability to post longer audio and video files, and verified account status."
by concatenation ([; ]) and element-wise product (⊙), which is then fed to a feedforward network to predict zi.
For training, besides considering all references equally (i.e., LENSall when k = n in Eq. (1)), we also adopt a reference-adaptive loss that selects a subset of references closer to s in terms of edit operations rather than the entire set R. It encourages the metric to consider that different simplifications (e.g., paraphrasing-focused, deletion-focused, with or without splitting) can be acceptable, as long as they are close to some (not necessarily all) of the human references. We compute this loss (Ladapt) as:
Ladapt = 1
km
m∑
j=1
∑
zl∈Z′ (hj − zl)2 (1)
where h is human rating and m is the training data size. We compute the set of predicted scores Z = {z1, zi, . . . , zn} corresponding to references in R and then choose top k (k ≤ n) values from Z to form a subset Z ′ ⊆ Z. Finally, we calculate the mean squared error (MSE) between the human rating h and each score in Z ′. By selecting top k scores in Z, we focus the training of metric on references similar to s in terms of writing style or editing operations. This loss also aligns the training step with the inference step, where we select the best-predicted score zmax corresponding to R. Although multiple references are ideal, the proposed loss can also use a single reference, similar to the standard MSE loss. We train LENS on our SIMPEVALPAST dataset (details in §3.1). We provide further implementation details in Appendix A.
Similar to the COMET (Rei et al., 2020) and BLEURT (Sellam et al., 2020) metrics for MT evaluation, LENS is trained on human ratings after zscore normalization, which are real values predominantly lie between [-3, 3]. To make LENS more interpretable, we rescale the predicted scores to the range between [0, 100] as the percentage of area under a normal curve of mean 0 and standard deviation 1 corresponding to zi. We present the rescaled scores for experiments and analyses in this paper. Besides, we use RoBERTa-large as the underlying model of LENS throughout the paper, unless otherwise specified (§5).

heading: 4 Experiments
text: We benchmark LENS metric against the existing automatic metrics (i) for evaluating text simplification system outputs and (ii) for training better
automatic simplification models when used as alternative reward functions.

heading: 4.1 Correlation with Human Evaluation
text: We demonstrate that LENS correlates better with human judgments than the existing metrics.
Evaluation Datasets. We compare LENS to the existing metrics, namely SARI (Xu et al., 2016), BERTScore (Zhang et al., 2020),4 BLEU, and Flesch-Kincaid Grade Level readability (FKGL) (Kincaid, 1975) on three datasets: SIMPEVAL2022 (§3.1), WIKI-DA released by AlvaManchego et al. (2021) with 0-100 continuous scale ratings on fluency, meaning preservation, and simplicity for 600 simplifications across six systems, and NEWSELA-LIKERT collected by Maddela et al. (2021) with 5-point Likert scale ratings on the same dimensions for 500 simplifications across five systems. While SIMPEVAL2022 and WIKI-DA are derived from Wikipedia, NEWSELALIKERT is derived from news articles in Newsela (Xu et al., 2015), a widely used corpus for text
4We use precision score of BERTScore as it has been shown to perform better for simplification evaluation (AlvaManchego et al., 2021).
simplification. When calculating metric scores for SIMPEVAL2022 that contains two human simplifications, we use one as the reference and the other as the oracle simplification system. We remove the complex sentences in WIKI-DA that overlap with SIMPEVALPAST, the training dataset for LENS.
Evaluation Setup. We report Kendall Tau-like correlation for SIMPEVAL2022 to capture the ability of metrics to distinguish two systems, which is close to the real-world scenario. Kendall Taulike correlation is predominantly used in machine translation metric evaluation at WMT (Bojar et al., 2017; Ma et al., 2018) for the same reason as it focuses on the relative ranking of the outputs. Given an input c and its simplifications from N systems S = {s1, . . . , sm, . . . , sn, . . . , sN}, we extract (sm, sn) pairs, where 1 ≤ m < n ≤ N , and calculate Kendall Tau-like coefficient τ :
τ = |Concordant| − |Discordant| |Concordant|+ |Discordant| (2)
where Concordant is the set of pairs where the metric ranked (sm, sn) in the same order as humans ranked and Discordant is the set of the pairs where the metric and humans disagreed. We report separate coefficients for paraphrase- (τpara),
and split-focused (τspl) simplifications, along with (τall) for all of them. Following the literature, we only used the (sm, sn) pairs for which the difference in ratings is more than 5 out of the 100 points, and all the annotators agreed on the ranking order. To make results comparable to existing work (Alva-Manchego et al., 2021, 2020) that evaluated on WIKI-DA and NEWSELA-LIKERT, we report Pearson correlation (ρ) between the metric scores and the human ratings.
Results. Table 2 shows that LENS outperforms the existing metrics on SIMPEVAL2022 and WIKIDA belonging to the Wikipedia domain, and NEWSELA-LIKERT based on newswire domain. The difference is more substantial on SIMPEVAL2022 consisting of similar performing SOTA systems, where the τall of LENSk=3 exceeds the τall of BERTScore by 0.22 points. Training using top k references (LENSk=3) has improved τall on SIMPEVAL2022 and ρ along the simplicity dimension on the rest when compared to using all the references (LENSall). Figure 3 shows the 95% confidence intervals for τall on SIMPEVAL2022 and ρ for simplicity, which is deemed to be the most important dimension in prior work (Alva-Manchego et al., 2021; Xu et al., 2016), on WIKI-DA and NEWSELA-LIKERT calculated using bootstrapping methods by Deutsch et al. (2021). A smaller interval indicates that the metric exhibits lower variance and higher reliability. LENS exhibits smaller intervals than the other metrics.

heading: 4.2 LENS as Training or Decoding Objectives
text: We also incorporate LENS into training as an alternative reward function in minimum risk training framework (Kumar and Byrne, 2004; Smith and Eisner, 2006) and into decoding as a reranking objective in minimum Bayes risk decoding (Fernandes et al., 2022; Freitag et al., 2022) to improve the quality of generated simplifications.
Minimum Risk Training (MRT). Given the input c and reference r, we generate a set of candidates S for each training step and calculate the expected risk (Lrisk) as follows:
Lrisk = ∑
s∈S cost(c, r, s) P (s|c)∑ s′∈S P (s ′|c) (3)
P (s|c) = T∏
t=1
P (st|c, s<t; θ)
cost(c, r, s) = 1−Metric(c, r, s)
where Metric(c, r, s) can be any evaluation metric, θ are the model parameters, and s<t = s1, . . . st−1 is a partial generation of s.
Following previous work (Shen et al., 2016; Wieting et al., 2019), we first train the seq-to-seq generation model with maximum likelihood estimation (MLE) for a few epochs and then we change the loss to a combination of LMLE and expected risk:
LMRT = γLMLE + (1− γ)Lrisk. (4)
We choose γ = 0.3 and |S| = 10 for our experiments.
Minimum Bayes Risk Decoding (MBR). We adopt the MBR framework proposed by Fernandes et al. (2022), where we first generate a set of candidates S for input c during inference then rerank them by comparing each candidate s ∈ S to all the other candidates in the set:
ûMBR = argmax s∈S
1 |S| ∑
s′∈S Metric(c, s′, s). (5)
For our experiments, we generate the candidates using beam search with beam size = |S|.
Experiment Setup. We fine-tune a T5 model that prepends control tokens to the input (Sheang and Saggion, 2021) to control various aspects of the generated simplification and has shown state-ofthe-art performance for the task. We use WIKIAUTO (Jiang et al., 2020) for training, ASSET
for validation, and the challenging, complex sentences in SIMPEVAL2022 for testing. WIKIAUTO consists of 400k complex-simple sentence pairs extracted from Normal and Simple Wikipedia document pairs. For our experiments, we trained T5 towards LENS, BLEU, SARI, and BERTScore. We provide more implementation details in Appendix B. We report the same metrics on the test set along with self-BLEU to capture the diversity of the outputs. We also ask 3 annotators to rate the system outputs from SIMPEVAL2022 using our evaluation interface (§6) and report the averaged ratings.
Results. Table 3 shows that LENS integrated into minimal Bayes risk decoding (MBR-LENS) achieves the best human evaluation results while it makes the most number of edits to the input (less copying) as indicated by its lowest self-BLEU score. Although MRT-LENS has slightly lower average human ratings than MRT-BERTScore and MRT-SARI, the pairwise comparison shows that
they are very comparable: MRT-LENS generates 28% better, 28% worse, and 44% equal quality simplifications compared to MRT-SARI. Additionally, BERTScore and SARI show a decrease in quality when used in decoding than standard maximum likelihood estimation (MLE). It is noteworthy that we use a beam size of 10 for MLE rather than a large search space of beam size 100 because generation quality degrades with increased beam size as shown in Table 4 as well as in existing literature (Stahlberg and Byrne, 2019; Meister et al., 2020).
Given the success of using LENS as utility function of MBR decoding on T5-base, we further apply it to larger models, including T5-3B and T5-11B. As displayed in Table 4, MBR-LENS with 100 candidates improves over standard beam search by an increase of over 11 points of LENS score across all model sizes. Although MBR may inflate the results of the utility function it uses (Fernandes et al., 2022), our human evaluations solidify the assertion that T5-11B with MBR-LENS decoding exceeds standard beam search, thereby establishing state-of-the-art (SOTA) performance among opensource models. When compared to close-source large language models, T5-11B with MBR-LENS achieves on-par performance with GPT-3.5.

heading: 5 LENS Analysis
text: In this section, we delve into the impact of the underlying model architecture and the number of references on the performance of LENS.
Model Architecture. Table 6 shows the Kendall Tau-like correlation (τpara, τspl, and τall) of LENS metric trained on various encoders on SIMPEVAL2022. LENS metrics trained on RoBERTa encoders (Liu et al., 2019) perform better than their respective LENS metrics trained on the BERT encoders (Devlin et al., 2019). Among all models, LENS trained on RoBERTa-large achieves the highest overall correlation. It has a substantial improvement in τspl with a trade-off in τpara, in comparison to RoBERTa-base.
Number of References. Figure 4 shows the Pearson correlation of LENS metric on WIKI-DA for a varied number of references used during inference. Although the metric performs the best with 10 references, we see only a slight drop with one reference, demonstrating that LENS is capable of evaluating with single or multiple references.

heading: 6 RANK & RATE Framework
text: To facilitate consistent evaluation and enable the training of LENS, we develop RANK & RATE , a human evaluation framework to assist annotators in efficiently comparing and rating many (>20) system outputs at once in a list-wise ranking manner.

heading: 6.1 Methodology
text: We describe the three-step annotation methodology for RANK & RATE (Figure 5) as follows:
Step 1 - Categorizing System Outputs. As there are different acceptable ways to simplify the input text, we first display the system outputs in groups as split-, deletion-, or paraphrase-focused based on the following criteria: (i) outputs with multiple sentences are split-focused, (ii) outputs with a compression ratio less than 0.5 or generated by deleting words from the original sentence are deletion-focused, (iii) the rest are paraphrasefocused. Annotators can adjust the initial automatic categorization during the annotation process.
Step 2 - Highlighting Edits. To help annotators notice the changes made by each system and subsequently improve the accuracy of their ratings, we use a state-of-the-art word alignment model by Lan et al. (2021) to highlight the edits performed by the system and classify them into three types: (i) Deletion-edits are phrases or words with no match in the modified output, (ii) Paraphrase-edits are new phrases or words added or modified, and (iii) Splitedits are any added periods (“.”). Deletion-edits are marked with a red caret (“∧”), paraphrase-edits with bolded text, and split-edits with two vertical bars (“ || ”) (see Figure 5). We ask annotators to correct the misclassified edits using an interactive pop-up window (see Appendix F).
Step 3 - Ranking and Rating System Outputs. Following the machine translation evaluation at WMT (Ma et al., 2018, 2019; Barrault et al., 2020; Akhbardeh et al., 2021), we ask annotators to rate the quality of system outputs on a 0-100 continuous scale instead of the 5-point Likert scale because the former was shown to provide higher levels of interannotator consistency (Novikova et al., 2018) and
is more suitable to apply on many statistical models (Graham et al., 2013). Our interactive interface allows the annotators to move the system outputs up and down to rank them and compare similar quality outputs more easily by placing them together.
We provide the annotation instructions and screenshots of the interface in Appendix F.

heading: 6.2 Human Evaluation Comparison
text: We compare RANK & RATE with the existing human evaluation methods: 5-point Likert (Sulem et al., 2018) and Direct Assessment with a continuous scale of 0 to 100 (Alva-Manchego et al., 2021), which were both conducted on three dimensions: fluency, adequacy, and simplicity. For a fair comparison, we annotate the same set of simplifications using each method, resulting in SIMPLIKERT2022 and SIMPDA2022. Table 7 shows similar rankings of the systems by the three methods with very slight differences. We also calculate the interval Krippendorff’s α (Krippendorff, 2011) for interannotator agreement. RANK & RATE achieves an α of 0.32, which is higher than the 0.23 α by Likert and 0.25 α by DA. All values are considered fair (Krippendorff, 2004).

heading: 7 Other Related Work
text: Text-to-text Generation Metrics. There is currently no single automatic metric to evaluate all the text-to-text generation tasks that revise text. SARI (Xu et al., 2016) is the main metric for text simplification and has been used by other generation tasks such as sentence splitting and fusion (Rothe et al., 2020; Kim et al., 2021), decontextualization (Choi et al., 2021), and scientific rewriting (Du et al., 2022). Style transfer tasks (Hu et al., 2017; Rao and Tetreault, 2018; Prabhumoye et al., 2018; Krishna et al., 2020; Xu et al., 2012; Ma et al., 2020) use different automatic metrics to measure each aspect of
text: (i) text similarity metrics such as BLEU, METEOR (Lavie and Agarwal, 2007), and BERTScore to measure content preservation, (ii) text classification models (Sennrich et al., 2016; Luo et al., 2019; Krishna et al., 2022b) or embedding-based edit distance metrics such as MoverScore (Zhao et al., 2019; Mir et al., 2019) to evaluate target style, and (iii) perplexity or pseudo log likelihood (Salazar et al., 2020) to measure fluency.
Incorporating Evaluation Metrics into Training and Inference. Prior studies have improved MLE-trained generation systems with alternative training approaches that integrate evaluation metrics based on reinforcement learning (Ranzato et al., 2015; Li et al., 2016; Gong et al., 2019), minimum risk (Smith and Eisner, 2006; Shen et al., 2016), and ranking (Hopkins and May, 2011; Xu et al., 2016; Krishna et al., 2022a). Incorporating metrics into decoding has also been explored by reranking the generations using discriminative rankers (Shen et al., 2004; Lee et al., 2021), energy-based rankers (Bhattacharyya et al., 2021), and minimum risk (Kumar and Byrne, 2004; Freitag et al., 2022). We use minimum risk as it has been shown to help machine translation systems (Wieting et al., 2019; Fernandes et al., 2022; Amrhein and Sennrich, 2022).

heading: 8 Conclusion
text: We introduce LENS, the first supervised automatic metric for text simplification. We show that LENS exhibits higher human correlation than other automatic metrics. We also introduce RANK & RATE framework, which allows annotators to evaluate multiple systems’ outputs at once in a list-wise manner. Using it, we create SIMPEVALPAST to train LENS, and SIMPEVAL2022 as a new metric evaluation benchmark for text simplification. We hope our metric, data, and framework will facilitate future research in text simplification evaluation.
Limitations
In this paper, we show that LENS shows better human correlation than other metrics on Wikipedia and news domains. Future research can further experiment and extend LENS to other domains, such as medical and children’s books, as the preference for different simplification operations can vary depending on the domain and user. Additionally, our work focuses on sentence-level simplification, and future work can extend LENS to evaluating paragraph- and document-level simplification. SIMPEVAL dataset and LENS are also limited to the English language.
Ethics Statement
We used in-house annotators to collect human ratings in SIMPEVAL datasets and write simplifications in SIMPEVAL2022. The annotators are university-level undergraduate and graduate students, including both native and non-native speakers of English. We did not collect any personal information from the annotators. We paid each annotator $15 per hour, which is above the US federal minimum wage. We ensured that the content shown to the annotators was not upsetting and let them know that they could skip the task if they felt uncomfortable at any point. We also let the annotators know the purpose of the collected data.
The original complex sentences in the SIMPEVAL datasets are from the publicly available Wikipedia. The simplifications are either from the existing work or human simplifications collected from our annotators. We used the authorreleased simplification outputs if they are available. For T5 (base, large, and 3B) systems, we trained our own simplification models using open-sourced code from the Hugging Face Transformers5 library.

heading: Acknowledgments
text: We thank Nghia T. Le, Tarek Naous, Yang Chen, and Chao Jiang as well as three anonymous reviewers for their helpful feedback on this work. We also thank Marcus Ma, Rachel Choi, Vishnesh J. Ramanathan, Elizabeth Liu, Alex Soong, Govind Ramesh, Ayush Panda, Anton Lavrouk, Vinayak Athavale, and Kelly Smith for their help with human evaluation. This research is supported in part by the NSF awards IIS-2144493 and IIS-2112633,
5https://huggingface.co/
ODNI and IARPA via the BETTER program (contract 2019-19051600004) and the HIATUS program (contract 2022-22072200004). The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of NSF, ODNI, IARPA, or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein.

heading: A LENS Metric - Implementation Details
text: We leverage Transformers and PyTorch Lightning libraries to implement the metric. We used RoBERTaLarge model (Liu et al., 2019) as the encoder. We fine-tuned the metric with the styleadaptive loss for 20 epochs and selected the checkpoint with the best validation loss. We used the ten human references from ASSET to calculate metric scores while training LENS using SIMPEVALPAST. We used a batch size of 8, Adam optimizer with a learning rate of 3e-05 for the encoder and 1e-05 for other layers. We use dropout of 0.15 and freeze the encoding layers for 1 epoch. We train the model on one Quadro RTX 6000 GPU with 25GB RAM, which takes around 3 hours.
B Incorporating Evaluation Metrics into Training and Inference - Implementation Details
We implement the controllable sentence simplification system proposed by Sheang and Saggion (2021), a T5-base model. The input sentence is prepended with four control tokens, namely character length ratio, dependency tree depth ratio, character-level Levenshtein similarity, and inverse frequency ratio, to control various aspects of the generated simplification. During training, each control value is calculated using the corresponding training pair and discretized into bins of width 0.05. During inference, we set the control tokens to the average value of the training set. We used 0.9 for the length ratio and 0.75 for the rest. We also finetune the controllable T5-3B and T5-11B versions using the same approach.
We use the Hugging Face Transformers library6
for implementing the base model and the MRT framework. We first train the model for 5 epochs using MLE loss and then fine-tune it for 5 epochs using MRT. We use Adam optimizer with a learning rate of 1e-4, linear learning rate warmup of 4k steps, weight decay of 0.1, epsilon of 1e-8, and batch size of 16. During MRT, we use a beam size of 8 to generate candidates. The rest of the parameters are left with default values from the library. During inference, we use a beam size of 10. Our models are trained on 2 A40 GPUs with 45GB RAM for 48 hours.
We used the code released by Fernandes et al. (2022)7 for the MBR framework. We generated
6https://huggingface.co/ 7https://github.com/deep-spin/qaware-decode
candidates using beam search of size 100 and selected the top 100 candidates for reranking. We used the above T5 model fine-tuned using MLE to generate the candidates. The rest of the parameters are left with default values from the library.

heading: C Systems in SIMPEVAL Datasets
text: Data-driven Neural Models. We use ten supervised models: (i) three fine-tuned T5 (Raffel et al., 2020) models of various sizes, namely T5-base, T5-large, and T5-3B, (ii) a controllable T5-base model that prepends tokens to the input to control the lexical and syntactic complexity of the output (Sheang and Saggion, 2021), (iii) two BART (Lewis et al., 2020) models that also use control tokens where one is trained on Wikipedia (Martin et al., 2020), and the other is fine-tuned on a combination of Wikipedia and web-mined paraphrases (Martin et al., 2022), (iv) a BERT-initialized Transformer (Maddela et al., 2021), (v) a BiLSTM editbased approach that first generates the edit operations, and then the simplification (Dong et al., 2019), (vi) a BiLSTM that directly generates simplifications using reinforcement learning (Zhang and Lapata, 2017), and (vii) a vanilla BiLSTM model. In addition, we also include one unsupervised model and one semi-supervised model by Surya et al. (2019) that uses an auto-encoder with adversarial and denoising losses.
Few-shot Methods. We include simplifications generated by GPT-3.58 under zero-shot and 5-shot settings (prompts are provided in Appendix D.2).
Data-driven Statistical Methods. We incorporate two systems that applied statistical machine translation (MT) approaches to text simplification: (i) a phrase-based MT system that reranks the outputs based on their dissimilarity with the input (Wubben et al., 2012), and (ii) a syntactic MT system that uses paraphrase rules for lexical simplification (Xu et al., 2016).
Rule-based Methods. Kumar et al. (2020) iteratively generates candidates using rules and ranks them with a linguistic scoring function.
Hybrid Methods. We utilize three hybrid systems that combine linguistic rules with data-driven methods: (i) Narayan and Gardent (2014) uses semantic structure to predict sentence splitting and
8Specifically, we use the text-davinci-003 model which is the most recent variant with 175B parameters.
paraphrases with a phrase-based MT system, (ii) Sulem et al. (2018) performs splitting and deletion using linguistic rules and paraphrasing using a BiLSTM, and (iii) Maddela et al. (2021) generates candidates with different amounts of splitting and deletion and then paraphrases the best candidate with a BERT-initialized Transformer.
Naive Baseline Methods. Existing metrics are biased towards conservative systems because their outputs are generally fluent and exhibit high lexical overlap with the input/reference (Pang and Gimpel, 2019; Krishna et al., 2020). We add two conservative systems that are challenging for automatic metrics: (i) a system that always copies the input and (ii) a content-preserving but nonsensical system that scrambles 5% of the input words.
Humans. We also add human-written simplifications using different instructions from ASSET (Alva-Manchego et al., 2020) and TurkCorpus (Xu et al., 2016), two widely used evaluation benchmarks for sentence simplification, and an autoaligned one from the SimpleWiki (Kauchak, 2013).
D Implementation Details for Simplification Systems
D.1 T5 Setup
We use the Hugging Face Transformers library9. We fine-tune T5-base, T5-large, and T5-3B on 4 A40 GPUs of a total batch size of 64 for 20 epochs (10.4K steps), 16 epochs, and 8 epochs, respectively. We use a learning rate of 3e-4. We save checkpoints every 5K steps and select the best one by performing a manual inspection on a set of 60 simplifications from the development set, resulting in 80K, 60K, and 25K steps for T5-base, T5large, and T5-3B respectively, which are in a similar range to FLAN-T5 (Chung et al., 2022). We use AdamW (Loshchilov and Hutter, 2017) as the optimizer.
D.2 GPT-3.5 Setup
Hyperparameters. We use the text-davinci-003 GPT-3.5 model from OpenAI API10. To generate simplification, we use the following hyperparameters: temperature=1 and top-p=0.95.
Prompts. We use the instruction from ASSET (Alva-Manchego et al., 2021) to prompt GPT-3.5.
9https://huggingface.co/ 10https://beta.openai.com/
Zero-shot setting: Please rewrite the following complex sentence in order to make it easier to understand by non-native speakers of English. You can do so by replacing complex words with simpler synonyms (i.e. paraphrasing), deleting unimportant information (i.e. compression), and/or splitting a long complex sentence into several simpler ones. The final simplified sentence needs to be grammatical, fluent, and retain the main ideas of its original counterpart without altering its meaning.
Input: {input} Output:
5-shot setting: Please rewrite the following complex sentence in order to make it easier to understand by non-native speakers of English. You can do so by replacing complex words with simpler synonyms (i.e. paraphrasing), deleting unimportant information (i.e. compression), and/or splitting a long complex sentence into several simpler ones. The final simplified sentence needs to be grammatical, fluent, and retain the main ideas of its original counterpart without altering its meaning.
Examples: Input: {random sampled from ASSET} Output: {random sampled human reference of the Input from ASSET}
Input: {random sampled from ASSET} Output: {random sampled human reference of the Input from ASSET}
Input: {random sampled from ASSET} Output: {random sampled human reference of the Input from ASSET}
Input: {random sampled from ASSET} Output: {random sampled human reference of the Input from ASSET}
Input: {random sampled from ASSET} Output: {random sampled human reference of the Input from ASSET}
Please rewrite the following complex sentence in order to make it easier to understand by non-native speakers of English. You can do so by
replacing complex words with simpler synonyms (i.e. paraphrasing), deleting unimportant information (i.e. compression), and/or splitting a long complex sentence into several simpler ones. The final simplified sentence needs to be grammatical, fluent, and retain the main ideas of its original counterpart without altering its meaning.
Input: {input} Output:

heading: E LENS metric - Additional Results

heading: F Annotation Interface
text: F.1 Step - 1: System Output Categorization.
F.2 Step - 2: Highlighting System Edits.
F.3 Step - 3: Rating and Ranking System Outputs.
F.4 Annotation Instructions
Figure 10: Instructions for the overall task.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Limitations section

heading: 3 A2. Did you discuss any potential risks of your work?
text: Limitations section Ethics Statement section

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Abstract and Section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 3

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Section 3, Appendix sections A - D.

heading: 3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
text: Sections 3 Ethics Statement section

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Sections 3 Ethics Statement section

heading: 3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any
text: information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Ethics Statement section

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? Sections 3 Ethics Statement section
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 3
C 3 Did you run computational experiments? Section 4

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Appendix Sections A - D
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Section 3 Appendix Sections A - D

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 4

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Appendix Sections A - D
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? Section 3

heading: 3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots,
text: disclaimers of any risks to participants or annotators, etc.? Appendix section F

heading: 3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students)
text: and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Sections 3 Ethics Statement section

heading: 3 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to
text: crowdworkers explain how the data would be used? Ethics Statement section
7 D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No. This study does not meet the definition of "human subjects" research by our institutional IRB review board, as we did not: "(1) Interact with, intervene with, or obtain/access private, identifiable information or data about, a living individual (includes online surveys) or (2) Conduct research on a drug, biologic, or medical device". The annotations are linguistic judgements provided by hired in-house employees.

heading: 3 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data?
text: Ethics Statement section

heading: Future Work
text: We hope our metric, data, and framework will facilitate future research in text simplification evaluation. Limitations
In this paper, we show that LENS shows better human correlation than other metrics on Wikipedia and news domains. Future research can further experiment and extend LENS to other domains, such as medical and children’s books, as the preference for different simplification operations can vary depending on the domain and user. Additionally, our work focuses on sentence-level simplification, and future work can extend LENS to evaluating paragraph- and document-level simplification. SIMPEVAL dataset and LENS are also limited to the English language