Simplicity Level Estimate (SLE): A Learned Reference-Less Metric for Sentence Simpliﬁcation
Authors: Liam Cripwell
Joël Legrand
Claire Gardent
Venue: EMNLP 2023 Main (2023)
Keywords: simplification
evaluation
quality estimation

Abstract: Automatic evaluation for sentence simplification remains a challenging problem. Most popular evaluation metrics require multiple high-quality references -- something not readily available for simplification -- which makes it difficult to test performance on unseen domains. Furthermore, most existing metrics conflate simplicity with correlated attributes such as fluency or meaning preservation. We propose a new learned evaluation metric --- SLE --- which focuses on simplicity, outperforming almost all existing metrics in terms of correlation with human judgements.

Simplicity Level Estimate (SLE):
A Learned Reference-Less Metric for Sentence Simplification
Liam Cripwell
Joël Legrand
Claire Gardent CNRS/LORIA
Université de Lorraine
Université de Lorraine
CNRS/LORIA liam.cripwell@e loriia
Centrale Supélec
Université de Lorraine
CNRS/LORIA
Laire.gardent @ loria.fr
joel.legrand inn.iergrand@nii..fr
Abstract
Metric
Simplification
Semantic
Ref-less
BLEU
X
Automatic evaluation for sentence simplifica- remains challenging problem. Most popular evaluation metrics require multiple high-quality references something not read- ily available for simplification which makes it difficult to test performance unseen de- mains. Furthermore, most existing metrics conflate simplicity with correlated attributes such as fluency meaning preservation. We propose a new learned evaluation metric (SLF) which focuses simplicity, outperforming al- most all existing metrics in terms of correla- with human judgements.
BERTScore
/
X
QUESTEVAL
/
SARI
X
FKGL
X
LENS
/
SLE
Table 1: Desirable attributes of popular simplification evaluation metrics whether they are designed with simplification in mind, use semantic representations, do not require references.
2021b). Table describes how popular metrics con- form to various desirability standards.
1 Introduction
We propose SLE (Simplicity Level Estimate), a learned reference-less metric that trained to esti- mate the simplicity of a sentence.' Different from pierence-bbssed metrics (which estimate simplicity with respect to reference), SLE can be used absolute measure of simplicity, relative mea- sure of simplicity gain compared to the input, to measure error with respect to a target simplicity level. this short paper, we focus on simplic- ity gain with respect to the input and show that SLE is highly correlated with human judgements of simplicity, competitive with the best performing reference-based metric. We also show that, when controlling for meaning preservation and fluency, many existing metries used to assess simplifica- tions do correlate well with human ratings of simplicity.
Text simplification involves the rewriting of a text to make it easier to read and understand by wider audience, while still expressing the same core meaning. This has potential benefits for dis- advantaged end-users (Gooding, 2022), while also showing promise preprocessing step for down- stream NLP tasks (Miwa al., 2010; Mishra et al., 2014; Stajner and Popovic, 2016; Niklaus et al., 2016). Although some recent work considers sim- plification of entire documents (Sun et al., 2021; Cripwell et al., 2023a,b) the majority of work fo- cuses on individual sentences, given the lack of high-quality resources (Nisioi et al., 2017; Martin al., 2020, 2021).
major limitation evaluating sentence simpli- fication is that most popular metries require high- quality references, which are rare and expensive to produce. This also makes it difficult to assess models on new domains where labeled data is un- available. Another limitation is that many metrics evaluate simplification quality by combining mul- tiple criteria (fluency, adequacy, simplicity) which makes difficult determine where exactly sys- tems succeed and fail, as these criteria are often highly correlated - meaning that high scores could be spurious indications of simplicity (Scialom et al.,
2
A Metric for Simplicity
The SLE Metric. We propose SLE, a learned metric which predicts a real-valued simplicity level for given sentence without the need for references. Given some sentence t, the system predicts SLE(!) R. with high values indicating higher
'odde and resources are available at httt:::://// h/t.........ttt////gg///./.. thub. com/ hom/liane llammcriii ppee wiwelllllseeee 1/sle/.

---

simplicity. This can only be used as an absolute measure of simplicity for system output y. but also to measure the simplicity gain relative to input æ:
where fL, is the revised FKGL score of sen- tence zy. Intuitively, this inverts FKGL scores (so that higher values higher simplicity) and rescales them to be E 20. The [0, 2] scaling is used order for the distribution of final scores in each reading level to have #1 variance and overlap with adjacent groups (see Figure I for a visual rep- resentation).
(I) In this paper we primarily focus on ASLE, as it is the most applicable variant under common sentence simplification standards.
ASLE( = SLE(O) SLE(2)
From this, derive the final revised labels:
Model. As the basis for the metric, we fine-tune pretrained RoBERTa modef2 perform regression over simplicity levels given sentence inputs, using a batch size of 32 and = 1e-5, We ran training experiments on a computing grid with a Nvidia A40 GPU.
- -- L1,t
(3)
where f is the mean of f' LL is the reading level for the ith sentence of L, and 1',, is its revised soft version.
Data. We use Newsela (Xu et al., 2015), which consists of 1,130 news articles manually rewritten at five discrete reading levels (0-4), each increasing in simplicity. Existing works often assume sen- tences have the same reading level the document they are from (Lee and Vajjala, 2022; Yanamoto et al., 2022); however, we expect there to be a lot of variation in the simplicity of sentences within doc- uments and overlap across levels. As such, merely training to minimize eтror with respect to these la- bels would likely result in mode collapse within levels (peaky, low-entropy distribution) and strong overfitting to the Newsela corpus. To address this mismatch between document- and sentence-level simplicity, we take the following two mitigating steps to allow the model to better differentiate be- tween sentences from the same reading level.
For example, if the original document has read- ing level 3, and one of the sentences has revised FKGL (Equation 2) of 1.5, then the softened label for that sentence will therefore be 3.5 (Equation 3). Figure shows the distributional differences be- tween the original reading levels and the resulting softened versions for the training data.
We report results for a model using softened labels (SLE) as well as variant using the original quantized labels (SLEz).
Document-Level Optimization. Given that Newsela reading levels are assigned at the document level, the labels of individual sentences are likely noisy, but approach the document label average. We therefore observe and perform early stopping with respect to the lecument-level validation MAE (Mean Absolute Error) and use a train/dev/test split (90/5/5) that keeps sentences from all versions of a given article together. The size of each data split is illustrated in Table 2.
Label Softening. We attempt to mitigate peak- iness in the output distribution by softening the quantized reading levels assigned to each sentence in the training data. Specifically, we interpolate regression labels throughout overlapping class re- grade
3 Experiments
gions 1+ according to their level (FKGL) (Kincaid et al., 1975). FKGL is a readability metric often used education as a means to judge the suitability of books for students (high values = high complexity).
3.1 Similarity Metries
We compare SLE with four reference-based and two reference-less metrics previously used to sess the output of simplification models. Table summarizes their main features.
If L is the set of sentences belonging to some reading level. we define an intra-level ranking ac- cording re-scaled, negative FKGLs:
The most commonly used evaluation met- ric is SARI (Xu al., 2016), which compares n- gram edits between the output, input and references. Despite its widespread usage, SARI has known lim- itations. The small set of operations it considers (2)
SARI
ft {- {-ffkgg fxgl(2;)) x3 E L}
fL,A min ft.
fr, max fI min ft
"e fine-tune the pretrained roberta-base model from ht attp:: hww.gggggaaac..eeeeoooo co/ roberta base with added regression head.
"Note that we also account for extreme FKGL values by excluding sentences with FKGL scores in the top or hottom percentile.

---

reading level
_eveel
7000
60000
50000
5000
3
Count
40000
Count
4000
30000
reading level
3000
20000
2000
10000
1000
0.0
0.5
1.0
1.5
2.0
2.5
3.0
3.5
4.0
reading_level
reading_leve [eggg__ol____oohhh
(a) Original
(b) Softened
Figure I: Distribution of (a) original quantized and (b) softened labels for sentences the SLE training data.
Total
Train
52,488
63,993
61,248
55,620
60,416
293,765
Dev
3069
3,595
3,348
2,976
3,228
16,216
Test
2901
3,710
3,348
3,024
3,244
16,227
All
58,458
71,298
67,944
61,620
66,888
326,208
Table Number of sentences sourced from documents of each quantized reading level.
makes much more focused towards lexical sim- plifications, showing very low correlations with human ratings in cases where structural changes (e.g. sentence splitting) have occurred (Sulem et al., 2018). As is token-based, it is totally reliant the references, without any robustness towards syn- onomy.
optimally.
FKGL. The Kinsch-Kincaid grade level (FKGL) (Kincaid al., 1975) is a document-level metric used measure text readability without any references. based on basic surface-level features like word/sentence lengths. It has seen some success in evaluating simplification (Scialom et al., 2021b). Unlike most other metrics, it does not explicitly consider the adequacy and fluency dimensions, it reference-less and assumes the text already well-formed (Xu al., 2016).
BERTScore. Zhang et al. (2019) present BERTScore, which overcomes some of these short- comings given its use of embeddings to compute similarities. It has been found to correlate highly with human ratings of simplicity (Alha-Manchego et al., 2021), but still requires references. It is reportedly worse than SARI at differentiating conservative edits (Maddela et al., 2022) and its high correlation with simplicity ratings may be spurious (Scialom et al., 2021b).
QUESTEVAL. Scialom et al. (2021a) propose QUESTEVAL, reference-less metric that com- pares two texts by generating and answering ques- tions between them. Although originally intended for summarization, it has shown some promise as a potential meaning preservation metric for simplifi- cation (Scialom al., 2021b).
LENS. Recently, Maddela et al. (2022) propose LENS. a learnable metric specifically designed for simplification, which aims to better account for different operation types. It is trained to focus semantic similarity without respect writing style via enference-adaptiiee loss. simplification quality score is predicted given an output and set of references. L.ENS shows higher correlations to human quality judgements than any previous metric, but still requires multiple references to work
3.2 Evaluation
We evaluate SLE both in terms of its ability to per- form the regression task and how well correlates with human judgements of simplicity. For the lat- ter we consider ASLE, as this conforms with what human evaluators were asked when giving ratings (to measure simplicity gain vs. the input).

---

Regression. To evaluate regression models we consider (i) the MAE with respect the original quantized reading levels, (ii) the document-level error when averaging all sentence estimates from a given document (Doc-MAE), and (iii) the Fl score as if performing classification task, after round- ing estimates. We expect the best model for our purposes to achieve lower Doc-MAE as it should better approximate true document-level simplicity labels in aggregate.
Model
MAE 1 Doe-MAE
SLE; (quantized)
0.825
0.544
0,401
SLE (softened)
0.924
0.448
0.402
Table 3: Accuracy results for reading level estimators.
Erors are calculated according the original quan- tized reading level labels.
Metric
Human Likert
Simplicity
-Dv DAV
LENS SARI
0,531
0,429e*
0,395
0.109
BERT
0,389**
0.142
Correlation with Human Simplicity Judgments. We test the effectiveness of the metric by com- paring its correlation with two datasets of hu- man simplicity ratings: Simplicity-DA (Alva- Manchego al., 2021) and Human-Likert (Scialom al.. 2021b). Simplicity-DA contains 600 sys- tem outputs, each with 15 ratings and 22 refer- ences, whereas Human-Liker contains 100 human- written sentence simplifications, each with ~60 simplicity ratings and 10 references. We use all references when computing the reference- based metrics and consider the average human simplicity rating for each item.
BLEU
0.333**
0.084
ASLE
0,516**
0.381
ASLEs
0.479**
0.328
FKGL
0,354
0.260*
QUESTEVAL
0.133
0.090
Table 4: Absolute Pearson correlations with human judgements of simplicity. The top tier contains reference-based metries and the bottom reference-less. * indicates significance with p-value 0.01 and ** 0.001.
viously worsens MAE with respect to the origi- nal reading levels, the document-level MAE is im- proved, suggesting that quantized labels lead to more extreme false negatives under uncertainty, scores are drawn towards integer values. When treated as classification task (by rounding predic- tions) both systems show similar performance (F1). This shows us that SLE is better able to approxi- mate document-level simplicity ratings on average, with little to no drawback at the sentence level (as- suming quantized labels were correct).
As Simplicity-DA consists of system output sim- plifications, it naturally contains some sentences that are not fluent or semantically adequate. In such cases, humans would likely give low scores to the simplicity dimension as well (e.g. it is not simple to understand non non-fluent text) - this is re- flected in the inter-correlation between simplicity and the two other dimensions (Pearsons' r of 0.771 for fluency and 0.758 for adequacy). Thus, only consider a subset (Simplicity-DA/> contain- ing those system outputs with both human fluency and meaning preservation ratings at least 0.3 std. devs above the mean (top ~30%)) which allows us to more appropriately consider how well metrics identify simplicity alone. For Human-LLkerrt, the inter-correlation with fluency and meaning preser- vation are less pronounced, but do still exist (0.736 and 0.370).5 As such, metric with high cor- relation on Human-Likert but low correlation on meany Simplicity-DAA means it is likely measuring one the other aspects rather than simplicity itself.
Correlations with human ratings of simplicity are shown in Table 4. The best metric on Human- Likert is LENS, closely followed by ASLE, with other metrics lagging quite far behind. This clearly shows the effectiveness of ASLE as it is able to outperform all existing metrics but for LENS, with- out requiring any references and using smaller network architecture than LENS and BERTScore. On Simplicity-DA. metries follow similar rank order except for certain metries dropping substan- tially (SARI, BERTScore, BLEU). 6 As Human- Likert still has moderate inter-correlation between evaluation dimensions, the large drops in perfor- mance can likely be attributed to these mostly mea-
4 Results
Results on the regression task can be seen in Ta- ble We see that although using soft labels ob-
"Scialom et al. (2021b) report very poor reference-bused metric correlations on Human-Likert substantially lower than our results. When discussed with the authors, they were no longer in possession of code that could reproduce their origi- nally reported results.
WWe also exclude 166 examples that exist within the LENS training data. We do not perform any filtering on Human-Likeet.

---

suring semantic similarity with references rather than the actual simplicity. Accounting for the inter- correlation between dimensions has less impact on metrics like ASLE and FKGL, confirming the validity of readability-basse metrics as potential measures of pure simplicity.
in mind, SLE could also possibly be used evalu- ale document simplification either via averaging sentence scores or using some other aggregation method.
7 Conclusion
In this paper presented SLE reference-less evaluation metric for sentence simplificatinn that is competitive with or better than the best performing reference-based metrics in terms of correlation to human judgements of simplicity. We reconsider the ability of popular metrics to accurately gauge sim- plicity when controlling for other factors such as fluency and semantic adequacy, confirming suspi- cions that many do not measure simplicity directly. We hope this work motivates further investigation into the efficacy of standard simplification evalu- ation techniques and the proposal of new method- ologies.
5 Related Work
Štajner al. (2014) attempt assess each quality dimension of simplifications by training classifiers of two (good, bad) or three (good, medium, bad) classes using existing evaluation metrics as fea- tures. However, when the simplicity dimension is considered, performance was poor (Štajner et al., 2016). Later, Martin et al. (2018) were able to slightly improve this after exploring a wide range of features. However, these works do not predict real-valued estimates of simplicity nor have been adopted as evaluation metrics.
Some studies from the automatic readability as- sessment (ARA) literature use quantized Newsela reading levels as labels to train regression mod- els. Lee and Vajjala (2022) do so in order to pre- dict the readability of full documents, which does not extend to sentence simplification. Yanamoto et al. (2022) predict a reading level accuracy within an RL reward for sentence simplification, but do so using the reading levels that were assigned to each document. This too does not transfer well to sentence-level evaluation, given the imprecision and noise introduced by the use of quantized ratings that were assigned at the document level. These approaches have not been applied the actual eval- uation of sentence simplification systems.
Acknowledgements
We thank the anonymous reviewers for their feed- back. We gratefully acknowledge the support of the French National Research Agency (Gar- dent; award ANR-20-CHIA-0003, XNLG "Multi- lingual, Multi-Source Text Generation").
Experiments presented this paper were car- ried out using the Grid 5000 testbed, supported by scientific interest group hosted by Inria and including CNRS, RENATER and several Univer- sities as well other organizations (see https: hwwp/wwwgggg www.wgrid500..o fr).
Limitations
The SLE metric model is trained entirely on English-language data and therefore will not be effective for evaluating simplification in other lan- guages. Producing multilingual version of the metric is likely possible by using either different datasets or adapting other methods from the multi- lingual NLP literature, but we leave this to future work.
6 Future Directions
In this paper we explore the efficacy of SLE as a measure of raw simplicity or relative simplicity gain (ASLE). However, given the flexibility of not relying on references, SLE can potentially be used in other ways. For example, one could measure an error with respect target simplicity level, 1*:
Furthermore, as SLE has been primarily trained on news articles, it could exhibit a drop in perfor- mance quality when used to evaluate text from spe- cialized domains that use vocabularies likely not encountered during training (e.g. medical, legal do- mains). In such cases, producing domain-specific version of SLE via specialized pretraining or fine- tuning should be feasible, given sufficient data.
eSLE (g) (S) l1]
This could be useful in the evaluation of control- lable simplification systems, which should be able satisfy simplification requirements of specific user groups or reading levels (Martin et al., 2020; Cripwell et al., 2022; Yanamoto et al., 2022). As is trained with aggregate document-level accuracy

---

References
Benoft Sagot. 2018. Reference-less quality estima- tion of text simplification systems. In Proceedings of the 1st Workshop on Automatic Text Adaptation (ATA), pages 29-38, Tilburg, the Netherlands, Asso- ciation for Computational Linguisties.
Fernando Alva-Manchego, Carolina Scarton, and Lu- cia Specia. 2021. The (Un)Suitability of Automatic Evaluation Metries for Text Simplification. Compu- lational Linguistics, pages 1-29.
Kshitij Mishra, Ankush Soni, Rahul Sharma, and Dipti Sharma. 2014. Exploring the effects of sentence simplification on Hindi to English machine transla- tion system. In Proceedings of the Workshop on Automatic Text Simplification Methods and Appli- cations in the Multilingual Society (ATS-MA 2014). pages 21 21-29, Dublin, Ireland. Association for Com- putational Linguistics and Dublin City University.
Liam Cripwell, Joël Legrand, and Claire Gardent. 2022. Controllable sentence simplification via operation classification. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 2091-2103, Seattle, United States. Association for Computational Linguistics.
Liam Cripwell, Johl Legrand, and Claire Gardent. 2023a. Document- planning for text simplifi- cation. In Proceedings of the 17th Conference of the European Chapter of the Association for Com- putational Linguistics, pages 993-1006, Dubrovnik, Croatia. Association for Computational Linguistics.
Makoto Miwa, Rune Setre. Yusuke Miyao, and Jun iicii Tsujii. 2010. Entity-focused sentence sim- plification for relation extraction. In Proceedings of the 23rd International Conference Compula- tional Linguistics (Coling 2010), pages 788-796, Beijing, China. Coling 2010 Organizing Committee.
Liam Cripwell, Johl Legrand, and Claire Gardent. 2023b. Context-aware document simplification. In to appear in Findings of the Association for Com- putational Linguistics: ACI. 2023, Toronto, Canada. Association for Computational Linguistics.
Christina Niklaus, Bernhard Bermeitinger, Siegfried Handschuh, and André Freitas. 2016. A sentence simplification system for improving relation extrac- tion. In Proceedings of COLING 2016, the 26th In- ternational Conference on Computational Linguis- tics: System Demonstrations, pages 170-174, Os- aka, Japan. The COLING 2016 Organizing Commit- tec.
Sian Gooding. 2022. On the ethical considerations of text simplification. In Ninth Workshop on Speech and Language Processing for Assistive Technologies (SLPAT-2022), pages 50-57, Dublin, Ireland. Asso- ciation for Computational Linguistics.
Sergiu Nisioi, Sanja Stajner, Simone Paolo Ponzetto, and Liviu P. Dinu. 2017. Exploring neural text sim- plification models. In Proceedings of the 55th An- nual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 85 85-91, Vancouver, Canada. Association for Computational Linguistics.
Peter Kincaid, Robert P Fishburne Jr, Richard L Rogers, and Brad S Chissom. 1975. Derivation of new readability formulas (automated readability in- dex, fog count and flesch reading ease formula) for navy enlisted personnel. Technical report, Naval Technical Training Command Millington TN Re- search Branch.
Thomas Scialom, Paul-Alexis Dray, Sylvain Lamprier, Benjamin Piwowarski, Jacopo Staiano, Alex Wang, and Patrick Gallinari. 2021a. QuestEval: Summa- rization asks for fact- evaluation. In Proceed- ings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6594-6604, Online and Punta Cana, Dominican Republic. Asso- ciation for Computational Linguistics.
Justin Lee and Sowmya Vajjala. 2022. A neural pair-
wise ranking model for readability assessment. In Findings of the Association for Computational Lin- guistics: ACI. 2022, pages 3802-3813, Dublin, Ire- land. Association for Computational Linguistics.
Mounica Maddela, Yao Dou, David Heineman, and Wei Xu. 2022. Lens: A learnable evaluation metric for text simplification.
Thomas Scialom, Louis Martin, Jacope Staiano, Éric Villemonte de la Clergerie, and Benoît Sagot. 2021b. Rethinking automatic evaluation in sentence simpli- fication.
Louis Martin, Éric de la Clergerie, Benoît Sagot, and Antoine Bordes. 2020. Controllable sentence sim- plification. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 4689- 4698, Marseille, France. European Language Re- sources Association.
Sanja Štajner, Ruslan Mitkov, and Horacio Saggion. 2014. One step closer to automatic evaluation of text simplification systems. In Proceedings of the 3rd Workshop on Predicting and Improving Text Read- ability for Target Reader Populations (PITR), pages 1-10, Gothenburg, Sweden. Association for Compu- tational Linguistics.
Louis Martin, Angela Fan, Éric de la Clergerie, An- toine Bordes, and Benoît Sagot. 2021. Muss: Multi- lingual unsupervised sentence simplification by min- ing paraphrases. arXiv preprint arXiv:2005.0033 00352.
Sanja Stajner and Maja Popovic. 2016. Can text simpli- fication help machine translation? In Proceedings of the 19th Annual Conference of the European Associ- ation for Machine Translation, pages 230-242.
Louis Martin, Samuel Humeau, Piere--Emmaneel Mazaré, Eric de La Clergerie, Antoine Bordes, and

---

Model
MAE
Doe-MAE
Fl T
B Regression Entropy
SLEg (quantized)
0.820
0.543
0.404
Figure shows the distributional difference be- tween the predictions from the SLE model with softened vs. quantized labels. Using the soft- ened labels leads to predictions following much smoother distribution.
SLE (softened)
0.931
0.457
0.405
Table Accuracy results reading level estimators the validation set. Errors are calculated according the original quantized reading level labels.
Quantized Labels
Sanja Stajner, Maja Popovic, Horacio Saggion, Lucia Specia, and Mark Fishel. 2016. Shared task on quality assessment for text simplification. Training, 218(95):192.
1750
1500
Count
Elior Sulem, Omri Abend, and Ani Rappoport. 2018. Semantic structural evaluation for text simplifica- tion. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Com- putational Linguisties: Human Language Technolo- gies, Volume (Long Papers), pages 685-696, New Orleans, Louisiana. Association for Computational Linguistics.
1000
750
500
250
0
0.0
0.5
1.0
1.5
2.0
2.5
3.0
3.5
SLE2
Renliang Sun, Hanqi Jin, and Xiaojun Wan. 2021. Document-level text simplification: Dataset, crite- ria and baseline. In Proceedings of the 2021 Con- ference Empirical Methods in Natural Language Processing, pages 7997-8013, Online and Punta Cana, Dominican Republic. Association for Compu- tational Linguistics.
Softened Labels
1200
800
Count
Xu, Chris Callison- -urrhh, and Courtney Napoles. 2015. Prohlems in current text simplification re- search: New data can help. Transactions of the Asso- ciation for Computational Linguistics, 3:283-297.
600
200
Wei Xu, Courtney Napoles, Ellie Pavlick, Quanze Chen, and Chris Callison-Burch 2016. Optimizing statistical machine translation for simplification. Transactions of the Association for Computational Linguistics, 4:401-415.
11
3
SLE
Figure Distribution of test set predictions from SLE models trained on quantized vs. softened labels.
Daiki Yanamoto, Tomoki Ikawa, Tomoyuki Kajiwara, Takashi Ninomiya, Satoru Uchida, and Yuki Arase. 2022. Controllable text simplification with deep re- inforcement learning. In Proceedings the 2nd Conference of the Asia-Pacifee Chapter of Associ- ation for Computational Linguistics and the 12th In- ternational Joint Conference Natural Language Processing (Volume 2: Short Papers), pages 398- 404, Online only. Association for Computational Linguistics.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi 2019. Bertscore: Evaluating generation with BERT. CoRR, abs/1994.09665.
A Validation Set Results
Table 5 lists the model performances on the valida- tion set. Notice they follow very similar pattern to what is seen in the test set results Table 3.


Reviews:
[{"review_id": "uA9u6ZAuBe", "reviewer": "Reviewer_bXBN", "summary": "", "questions": "Question A: Why do you multiply by 2 in the FKGL scores transformation to reach the [0, 2] range instead of [0, 1]? I infer this is related to the subtraction of the mean to approximate the range of simplicity level labels, but it is unclear.\n\nQuestion B: What is the motivation behind the sum of transformed FKGL scores and simplicity levels in eq. (3)? Some brief insights would help in understanding why SLE correlates well with human evaluations.\n\nQuestion C: Do you plan to release the code and data in addition to the metric itself?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for taking the time to review our paper.\n\nRe reasons to reject:\n\n1.\n\nThe [0,2] interval is used so that there is an approximate +-1 rescaling (see lns 97-100) around the original document simplicity level (Equation 3) and this was chosen by design (we want the different reading level groups to overlap with each other). In Equation 3 you can see that this is subtracted by the mean, $\\bar{f^\\prime_L}$ (~1), which then obviously yields the +-1 distribution. These values are then summed with the original simplicity level (0-4) to have them rescaled to be +-1 of that value. E.g. if the original document has a reading level of 3, and one of the sentences has a revised FKGL (Equation 2) of 1.5, then the softened label for that sentence will therefore be 3.5 (i.e., 1.5 - 1 + 3). Fig 1 (App, p7) illustrates this visually by showing the distributions of the softened labels.\n\n2.\n\nThe correlations of all metrics (except SLE) for the full SimplicityDA dataset can be seen in the original SimplicityDA paper. We feel like trying to include this in our short paper would be redundant and take up an unnecessary amount of space, especially considering our point is that these results are flawed anyway. SLE unsurprisingly has very low correlations there because it is impossible for it to consider meaning preservation (it doesn’t see references or inputs) and is only trained with fluent human-written data, so obviously isn’t equipped to determine disfluency either. See lines 214-230 for our rationale behind this move (also the thorough analysis in Scialom et al, 2021b).\n\nWe feel like this point about robustness doesn’t really apply as the inclusion of those results would mean we would effectively be evaluating SLE on tasks that it is not trained to perform (meaning preservation and fluency) and also architecturally unable to handle the inputs that would be necessary to do so (meaning preservation). We also include results on another dataset (Human-Likert) which contains higher quality texts while still having some inter-correlation between the dimensions, so this should provide a good indication of robustness without using completely unsuitable data. \n\n3.\n\nWe do mention our training parameters in the paper (lines 73-76). As for the hardware setup, we didn’t mention this in this version of the paper as we thought it could lead to anonymity issues, but it was done using a fairly modest GPU. We will also be releasing our full training code so reproducibility shouldn’t be a problem (the training setup is not much more complicated than a standard regression model).\n\nThe scores on the validation data are listed below. As you can see, they follow a very similar pattern to the test set results.\n\n```bash\nModel   MAE    Doc-MAE  F1\nSLE_Z   0.820  0.543    0.404\nSLE     0.931  0.457    0.405\n```\n\nQuestion A: \n\nThis is more or less answered above in point 1. Using an interval of [0,1] would mean scores are softened by a +-0.5 factor. This would yield a much peakier final distribution that would have very little representation for values on the borders between reading levels (see Figure 1 for visual context). We use the [0,2] scaling so that the distribution of scores in each reading level has a wider variance and overlap with adjacent groups (+-1.0).\n\nQuestion B: See response to point 1 above.\n\nQuestion C: Yes, we plan to release the full code, model, and data (and data generation code). An anonymized version of much of this is also already included in the supplementary materials, but a polished and enhanced version will be put on Github by the time of publication.\n\nRe typos/presentation and missing references: Thanks for pointing those out, we will make sure to address them in the final version of the manuscript. \n\nThank you again for your review of our paper and we hope we have sufficiently clarified any concerns you may have had.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper introduces a new state-of-the-art metric sentence simplicity metric that better correlates with human evaluations, using a novel technique to leverage document-level annotations for attributing sentence-level simplicity. The derivations are sound, and the experiments show that the novel smoothing technique allows for better performance than regressing over discrete levels. This technique could be further generalized to other tasks with per-level annotations. Additionally, the experiments support the claim that SLE correlates more closely with human evaluations than other methods. The paper is very well written, easy to read, and the motivation, arguments, experiments, and conclusions are communicated effectively.", "reasons_to_reject": "The specific solutions to some problems lack technical motivation. Examples include: (i) scaling FKGL scores to [0, 2], and (ii) summing FKGL scores and simplicity levels. This makes it difficult to understand whether the novel ideas were obtained by design or discovered through trial.\n\nRegarding the experimentation, the authors argue that one of the evaluation datasets for human correlations, Simplicity-DA, includes undesirable qualities. These are removed for evaluation, but it would have been interesting to see SLE’s (and the other metrics’) performance including sentences with these qualities to study its robustness. \n\nHardware set-up, training parameters, and validation set results are not reported: the paper may lack sufficient information to effectively reproduce the reported results, especially since one of the novel ideas includes early stopping with the validation set results.", "questions_to_authors": "Question A: Why do you multiply by 2 in the FKGL scores transformation to reach the [0, 2] range instead of [0, 1]? I infer this is related to the subtraction of the mean to approximate the range of simplicity level labels, but it is unclear.\n\nQuestion B: What is the motivation behind the sum of transformed FKGL scores and simplicity levels in eq. (3)? Some brief insights would help in understanding why SLE correlates well with human evaluations.\n\nQuestion C: Do you plan to release the code and data in addition to the metric itself?", "paper_topic_and_main_contributions": "This paper proposes SLE, an automatic metric for evaluating sentence simplification in a reference-less manner. It argues that existing sentence simplification metrics combine various criteria instead of focusing on simplicity, and describes how SLE does this by design while also providing a way to measure simplicity gain. The key ideas are: (i) regressing over simplicity levels, (ii) obtaining soft labels by mixing simplicity levels with Flesch-Kincaid readability scores, and (iii) document-level optimization due to noisy sentence simplicity labels. The experiments show that SLE better correlates with human evaluations than existing metrics, and the authors explain that it will be publicly released once the paper is published.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "There are no missing references. Avoid citing published works from arXiv: Martin et. al., 2021 is cited from arXiv, the LREC version should be cited instead."}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Throughout the paper: remove spaces before footnotes. Additionally, “referenceless” can be used, without hyphen.\n\nThe notation can be made more consistent. Consider changing “t” to “x” in line 064, and ensure that the output co-domain is correctly defined in line 065 to avoid confusion, otherwise one might understand SLE to output scores in (-inf, inf), which I understood to not be the case given the existing sentence simplicity levels it was trained on.\n\nFurthermore, line 104 defines L as the set of sentences belonging to some reading level, but then lowercase L is used in eq (3) to denote a sentence’s level. I would opt for a different letter, e.g. S, for the set of sentences, and introduce the lowercase L as a restriction in eq (2) when defining the set f_L. Overall, the Label Softening section was difficult to parse. The paper would greatly benefit from a better presentation of its technical aspects."}}, {"review_id": "ppyWr689vx", "reviewer": "Reviewer_SwXe", "summary": "", "questions": "In continuation with the discussion on SARI (lines 135-146), to what extent does SLE capture structural simplicity? Or, in other words, to what extent is structural simplicity present in the Newsela dataset on which SLE learns and on the Simplicity-DA and Human-Likert Benchmarks on which it is evaluated?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for taking the time and effort to review our paper\n\nYes, you raise a good point re interpretability of e.g. structural vs lexical simplicity. As it stands SLE doesn’t really provide any clear indication of this as it is just a single score. However, we are fairly confident that it has learned to consider both of these factors as Newsela contains quite a diverse range of transformations (~24% of sentences are split). Also the data evaluated on does contain instances that include sentence splitting as they are derived from ASSET, which was specifically built to cover a wide array of possible transformation types. In SimplicityDA, 25% of references contain a sentence split and 73% of items have at least one reference with a split, however, splits are only in ~7.3% of outputs. In Human-Likert, there are a lot more with ~18% outputs being splits.\n\nRe presentation improvements: including example outputs and scores for references is a good idea that we will aim to take into account for the final version. Using some of the extra space to extend discussion of future directions would also be valuable.\n\nThank you again for taking the time to review our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper is very well-written and well-presented.\n- It represents an interesting contribution to sentence simplification evaluation, following the ideas of the use of a separate measure for the simplicity dimension of sentence simplification and of the ability to evaluate simplicity without using references. \n- The connection between the sentence-level and document-level simplicity as done in the computation of the measure seems very useful.\n      ", "reasons_to_reject": "A drawback of the approach is that it is opaque in the sense that the measure does not tell us in which way the output is simpler than the input, for example in terms of operations performed or in terms of lexical vs. structural simplicity. I think it would be useful to discuss it in the paper.", "questions_to_authors": "In continuation with the discussion on SARI (lines 135-146), to what extent does SLE capture structural simplicity? Or, in other words, to what extent is structural simplicity present in the Newsela dataset on which SLE learns and on the Simplicity-DA and Human-Likert Benchmarks on which it is evaluated?", "paper_topic_and_main_contributions": "This paper presents a new evaluation measure for simplicity in the context of sentence simplification evaluation. The measure is learnable, using the Newsela labels together with label softening as a basis for training a RoBERTa-based regression model and computes the simplicity relative to the input (but can also address absolute simplicity) without using references.\nThe new measure called SLE achieves the highest correlation with human evaluation on two benchmarks, among the reference-less measures. The results also show the difficulty of existing measures that achieve high overall correlations to capture the simplicity dimension of the task.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Presentation:\n\n- It would be useful to present example of outputs that get high scores and low scores by SLE, also comparing with other metrics.\n- I think the differences between LENS and SLE, beyond the use of references, should be described in more details.\n- It would be interesting to also report the scores for the references.\n- It would be useful to extend the part on future directions (the adaptation of the measure to a target simplicity level) to a discussion about reference-less measures, also mentioning a recent work on the limitations of reference-less measures for text generation:\n\"On the Limitations of Reference-Free Evaluations of Generated Text\", Daniel Deutsch, Rotem Dror and Dan Roth, Proceedings of EACL 2023, pages 10960 - 10977.\n\nMinor:\n\n- Footnote 4: \"the\" should be removed\n\n- line 282: it would be useful to explicitly write Reinforcement Learning.\n"}}, {"review_id": "TO31zlknwW", "reviewer": "Reviewer_eCtA", "summary": "", "questions": "#Q1) Equation 2. Why do we need scalar multiplier 2 that results in rescaled value of [0,2]. Why not [0, 1] and hence no scalar multiplier (Occam’s razar)?\n\n#Q2)Seems like roberta-base model is used. Did you try roberta-large as pretrained model? Considering that GPUs are getting faster over years, large maybe the new base, and some users may be able to afford extra cost for higher performance. \n", "limitations": "", "rating": 4, "confidence": 1, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for taking the time to review our work.\n\nRegarding your point on the low correlation for some of the metrics on SimplicityDA: we view this as a being a result of those metrics (SARI, BertScore, Bleu, QuestEval) not actually measuring simplicity, but only the inter-correlated attributes (e.g. meaning preservation) that we are removed from the SimplicityDA data (cf Sec. 3.2) but which still exist in the  Human-Likert dataset as the correlation coefficients for those metrics suddenly increase a lot on the Human-Likert data. In contrast, the fact that Pearson and significance remains high on both datasets for these metrics that are more specifically focused on simplicity (SLE, LENS, FKGL) gives us confidence that these are quite reliable metrics as they have good correlation on both datasets.\n\nQuestion 1.\n\nUsing an interval of [0,1] would mean scores are softened by a +-0.5 factor. This would yield a much peakier final distribution that would have very little representation for values on the borders before reading levels (see Figure 1 for visual context). We use the [0,2] scaling so that the distribution of scores in each reading level has a wider variance and overlap with adjacent groups (+-1.0).\n\nQuestion 2.\n\nYes, we initially considered using roberta-large (this is what LENS and BERTScore are based on) but switched to roberta-base due to us having  restrictive GPU access. Training larger versions of the model would definitely be something we’d encourage or even try to do ourselves in the future and we will be releasing all of our training code to allow for this.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The proposed metric appears to have strong correlations.  Paper is easy to read and follow. The approach is simple and elegant. See Contributions in the above paragraph (which seems sufficient for a short paper). Assuming authors would provide finetuned models for use, it would be a useful addition to community. ", "reasons_to_reject": "There seem no major flaws in methodology and analysis. However, a nitpick; in Table 3, results are based on pearson correlation, which is known to suffer from outliers in the sample set. While the authors give ranking on two datasets, nearly half of metrics don't have a statistically significant correlation on Simplicity-DA, so we not thoroughly convinced about the usefulness of the dataset. Having Kendal’s Tau in addition to pearson r would be one way of boosted our confidence in the results, or another dataset where most commonly used nmetrics have statistically sound correlations. ", "questions_to_authors": "#Q1) Equation 2. Why do we need scalar multiplier 2 that results in rescaled value of [0,2]. Why not [0, 1] and hence no scalar multiplier (Occam’s razar)?\n\n#Q2)Seems like roberta-base model is used. Did you try roberta-large as pretrained model? Considering that GPUs are getting faster over years, large maybe the new base, and some users may be able to afford extra cost for higher performance. \n", "paper_topic_and_main_contributions": "Evaluation metric for Sentence Simplication task. Named SLE, is a model based (i.e. learned) metric. Finetune Roberta on a datatset (Newsela) that has 1130 new articles rewritten to 5 reading levels. There are some challenges,e.g., rewritten docs have mixture of levels among sentences. Authors propose a method to tackle this challenge: label softening and doc-level optimization.\n\nContributions:\n*  They develop a (learned) metric for simplicity evaluation (SLE). They describe methods to tackle the challenge of mismatch in sentence-level and document-level simplicity annotation. _Note: proposed SLE metric is reference less._\n* They conduct (meta) evaluation. The results show that SLE is the best reference-free metric, a runner up only to the reference based LENS metric. It has stronger correlations with human scores than many other commonly used reference based metrics such as SARI, BERTScore etc.\n", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "\nFigures in Page 7 maybe possible to move into the whitespace in Page6. (Option 1: subfigures one above the other. Option 2: `\\usepackage{balance}`)\n"}, "extra_text": {}}, {"review_id": "qKn5FhMJC7", "reviewer": "Reviewer_1Hnr", "summary": "", "questions": "A. Can SLE generalize to out-of-distribution data? As it is trained on only a thousand news articles, would it be hard to generalize to more oral text like twitter texts?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for taking the time to review our paper.\n\nThe datasets we evaluate the metrics on (Human-Likert and SimplicityDA) are actually not from news data, so the evaluation in the paper does capture the generalizability of SLE. Both of those datasets use simplifications of sentences from the ASSET and TurkCorpus datasets, both of which are sourced from Wikipedia articles. This could actually be considered as another potential strength of SLE over other learned metrics like LENS (which was trained on the same Wikipedia domain we evaluate on). \n\nAs you suggest, if the domain shifts further towards twitter comments or transcribed speech, it is likely that the performance will diminish, but this would probably be true for other learned metrics as well. However, we will make our training code publicly available so that newer versions could be trained on wider or more specialized data in the future.\n\nWe hope that this clarifies the concern you raised in your reason to reject. Thank you once again for taking the time to review our work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The real valued simplicity measure SLE is intuitive and performs very well as a referenceless measure, even competitive with reference based measures (outperforming all except LENS).\n2. The proposed metric SLE is less affected when accounting for inter-correlation between dimensions. The authors have good analysis on cases when simplicity dimensions are correlated.", "reasons_to_reject": "1. Unsure how well SLE generalizes if trained on news data.", "questions_to_authors": "A. Can SLE generalize to out-of-distribution data? As it is trained on only a thousand news articles, would it be hard to generalize to more oral text like twitter texts?", "paper_topic_and_main_contributions": "The authors propose Simplicity Level Estimate (SLE), a referenceless metric for sentence simplification. SLE is finetuned on a pretrained RoBERTa model for regression on simplicity levels given sentence inputs on Newsela. Label softening is used to soften the reading levels assigned to each sentence, and document-level optimization is performed by early stop with respect to the document-level validation MSE. The metric considers both simplification and semantics, outperforms all other referenceless metrics, and is competitive with state-of-the-arts reference-based metrics.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose a new learned evaluation metric for sentence simplification, SLE, which focuses on simplicity and outperforms almost all existing metrics in terms of correlation with human judgments.
Decision: Accept-Main