Introduction
Text simplification (hereafter TS) has received increasing interest by the scientific community in recent years. It aims at producing a simpler version of a source text that is both easier to read and to understand, thus improving the accessibility of text for people suffering from a range of disabilities such as aphasia BIBREF0 or dyslexia BIBREF1 , as well as for second language learners BIBREF2 and people with low literacy BIBREF3 . This topic has been researched for a variety of languages such as English BIBREF4 , BIBREF5 , BIBREF6 , BIBREF7 , French BIBREF8 , Spanish BIBREF9 , Portuguese BIBREF10 , Italian BIBREF11 and Japanese BIBREF12 .
One of the main challenges in TS is finding an adequate automatic evaluation metric, which is necessary to avoid the time-consuming human evaluation. Any TS evaluation metric should take into account three properties expected from the output of a TS system, namely:
TS is often reduced to a sentence-level problem, whereby one sentence is transformed into a simpler version containing one or more sentences. In this paper, we shall make use of the terms source (sentence) and (TS system) output to respectively denote a sentence given as an input to a TS system and the simplified, single or multi-sentence output produced by the system.
TS, seen as a sentence-level problem, is often viewed as a monolingual variant of (sentence-level) MT. The standard approach to automatic TS evaluation is therefore to view the task as a translation problem and to use machine translation (MT) evaluation metrics such as BLEU BIBREF15 . However, MT evaluation metrics rely on the existence of parallel corpora of source sentences and manually produced reference translations, which are available on a large scale for many language pairs BIBREF16 . TS datasets are less numerous and smaller. Moreover, they are often automatically extracted from comparable corpora rather than strictly parallel corpora, which results in noisier reference data. For example, the PWKP dataset BIBREF4 consists of 100,000 sentences from the English Wikipedia automatically aligned with sentences from the Simple English Wikipedia based on term-based similarity metrics. It has been shown by BIBREF7 that many of PWKP's “simplified” sentences are in fact not simpler or even not related to their corresponding source sentence. Even if better quality corpora such as Newsela do exist BIBREF7 , they are costly to create, often of limited size, and not necessarily open-access.
This creates a challenge for the use of reference-based MT metrics for TS evaluation. However, TS has the advantage of being a monolingual translation-like task, the source being in the same language as the output. This allows for new, non-conventional ways to use MT evaluation metrics, namely by using them to compare the output of a TS system with the source sentence, thus avoiding the need for reference data. However, such an evaluation method can only capture at most two of the three above-mentioned dimensions, namely meaning preservation and, to a lesser extent, grammaticality.
Previous works on reference-less TS evaluation include BIBREF17 , who compare the behaviour of six different MT metrics when used between the source sentence and the corresponding simplified output. They evaluate these metrics with respect to meaning preservation and grammaticality. We extend their work in two directions. Firstly, we extend the comparison to include the degree of simplicity achieved by the system. Secondly, we compare additional features, including those used by BIBREF18 , both individually, as elementary metrics, and within multi-feature metrics. To our knowledge, no previous work has provided as thorough a comparison across such a wide range and combination of features for the reference-less evaluation of TS.
First we review available text simplification evaluation methods and traditional quality estimation features. We then present the QATS shared task and the associated dataset, which we use for our experiments. Finally we compare all methods in a reference-less setting and analyze the results.
Using MT metrics to compare the output and a reference
TS can be considered as a monolingual translation task. As a result, MT metrics such as BLEU BIBREF15 , which compare the output of an MT system to a reference translation, have been extensively used for TS BIBREF6 , BIBREF19 , BIBREF20 . Other successful MT metrics include TER BIBREF21 , ROUGE BIBREF22 and METEOR BIBREF23 , but they have not gained much traction in the TS literature.
These metrics rely on good quality references, something which is often not available in TS, as discussed by BIBREF7 . Moreover, BIBREF19 and BIBREF24 showed that using BLEU to compare the system output with a reference is not a good way to perform TS evaluation, even when good quality references are available. This is especially true when the TS system produces more than one sentence for a single source sentence.
Using MT metrics to compare the output and the source sentence
As mentioned in the Introduction, the fact that TS is a monolingual task means that MT metrics can also be used to compare a system output with its corresponding source sentence, thus avoiding the need for reference data. Following this idea, BIBREF17 found encouraging correlations between 6 widely used MT metrics and human assessments of grammaticality and meaning preservation. However MT metrics are not relevant for the evaluation of simplicity, which is why they did not take this dimension into account. BIBREF20 also explored the idea of comparing the TS system output with its corresponding source sentence, but their metric, SARI, also requires to compare the output with a reference. In fact, this metric is designed to take advantage of more than one reference. It can be applied when only one reference is available for each source sentence, but its results are better when multiple references are available.
Attempts to perform Quality Estimation on the output of TS systems, without using references, include the 2016 Quality Assessment for Text Simplification (QATS) shared task BIBREF25 , to which we shall come back in section SECREF3 . BIBREF26 introduce another approach, named SAMSA. The idea is to evaluate the structural simplicity of a TS system output given the corresponding source sentence. SAMSA is maximized when the simplified text is a sequence of short and simple sentences, each accounting for one semantic event in the original sentence. It relies on an in-depth analysis of the source sentence and the corresponding output, based on a semantic parser and a word aligner. A drawback of this approach is that good quality semantic parsers are only available for a handful of languages. The intuition that sentence splitting is an important sub-task for producing simplified text motivated BIBREF27 to organize the Split and Rephrase shared task, which was dedicated to this problem.
Other metrics
One can also estimate the quality of a TS system output based on simple features extracted from it.
For instance, the QuEst framework for quality estimation in MT gives a number of useful baseline features for evaluating an output sentence BIBREF28 . These features range from simple statistics, such as the number of words in the sentence, to more sophisticated features, such as the probability of the sentence according to a language model. Several teams who participated in the QATS shared task used metrics based on this framework, namely SMH BIBREF18 , UoLGP BIBREF29 and UoW BIBREF30 .
Readability metrics such as Flesch-Kincaid Grade Level (FKGL) and Flesch Reading Ease (FRE) BIBREF31 have been extensively used for evaluating simplicity. These two metrics, which were shown experimentally to give good results, are linear combinations of the number of words per sentence and the number of syllables per word, using carefully adjusted weights.
Methodology
Our goal is to compare a large number of ways to perform TS evaluation without a reference. To this end, we use the dataset provided in the QATS shared task. We first compare the behaviour of elementary metrics, which range from commonly used metrics such as BLEU to basic metrics based on a single low-level feature such as sentence length. We then compare the effect of aggregating these elementary metrics into more complex ones and compare our results with the state of the art, based on the QATS shared task data and results.
The QATS shared task
The data from the QATS shared task BIBREF25 consists of a collection of 631 pairs of english sentences composed of a source sentence extracted from an online corpus and a simplified version thereof, which can contain one or more sentences. This collection is split into a training set (505 sentence pairs) and a test set (126 sentence pairs). Simplified versions were produced automatically using one of several TS systems trained by the shared task organizers. Human annotators labelled each sentence pair using one of the three labels Good, OK and Bad on each of the three dimensions: grammaticality, meaning preservation and simplicity. An overall quality label was then automatically assigned to each sentence pair based on its three manually assigned labels using a method detailed in BIBREF25 . Distribution of the labels and examples are presented in FIGURE FIGREF10 and TABLE TABREF12 .
The goal of the shared task is, for each sentence in the test set, to either produce a label (Good, OK, Bad) or a raw score estimating the overall quality of the simplification for each of the three dimensions. Raw score predictions are evaluated using the Pearson correlation with the ground truth labels, while actual label prediction are evaluated using the weighted F1-score. The shared task is described in further details on the QATS website.
Features
In our experiments, we compared about 60 elementary metrics, which can be organised as follows:
MT metrics
BLEU, ROUGE, METEOR, TERp
Variants of BLEU: BLEU_1gram, BLEU_2gram, BLEU_3gram, BLEU_4gram and seven smoothing methods from NLTK BIBREF32 .
Intermediate components of TERp inspired by BIBREF18 : e.g. number of insertions, deletions, shifts...
Readability metrics and other sentence-level features: FKGL and FRE, numbers of words, characters, syllables...
Metrics based on the baseline QuEst features (17 features) BIBREF28 , such as statistics on the number of words, word lengths, language model probability and INLINEFORM0 -gram frequency.
Metrics based on other features: frequency table position, concreteness as extracted from BIBREF33 's BIBREF33 list, language model probability of words using a convolutional sequence to sequence model from BIBREF34 , comparison methods using pre-trained fastText word embeddings BIBREF35 or Skip-thought sentence embeddings BIBREF36 .
TABLE TABREF30 lists 30 of the elementary metrics that we compared, which are those that we found to correlate the most with human judgments on one or more of the three dimensions (grammaticality, meaning preservation, simplicity).
Experimental setup
We rank all features by comparing their behaviour with human judgments on the training set. We first compute for each elementary metric the Pearson correlation between its results and the manually assigned labels for each of the three dimensions. We then rank our elementary metrics according to the absolute value of the Pearson correlation.
We use our elementary metrics as features to train classifiers on the training set, and evaluate their performance on the test set. We therefore scale them and reduce the dimensionality with a 25-component PCA, then train several regression algorithms and classification algorithms using scikit-learn BIBREF37 . For each dimension, we keep the two models performing best on the test set and add them in the leaderboard of the QATS shared task (TABLE TABREF39 ), naming them with the name of the regression algorithm they were built with.
Comparing elementary metrics
FIGURE TABREF32 ranks all elementary metrics given their absolute Pearson correlation on each of the three dimensions.
INLINEFORM0 -gram based MT metrics have the highest correlation with human grammaticality judgments. METEOR seems to be the best, probably because of its robustness to synonymy, followed by smoothed BLEU (BLEUSmoothed in TABREF30 ). This indicates that relevant grammaticality information can be derived from the source sentence. We were expecting that information contained in a language model would help achieving better results (AvgLMProbsOutput), but MT metrics correlate better with human judgments. We deduce that the grammaticality information contained in the source is more specific and more helpful for evaluation than what is learned by the language model.
It is not surprising that meaning preservation is best evaluated using MT metrics that compare the source sentence to the output sentence, with in particular smoothed BLEU, BLEU_3gram and METEOR. Very simple features such as the percentage of words in common between source and output also rank high. Surprisingly, word embedding comparison methods do not perform as well for meaning preservation, even when using word alignment.
Methods that give the best results are the most straightforward for assessing simplicity, namely word, character and syllable counts in the output, averaged over the number of output sentences. These simple features even outperform the traditional, more complex metrics FKGL and FRE. As could be expected, we find that metrics with the highest correlation to human simplicity judgments only take the output into account. Exceptions are the NBSourceWords and NBSourcePunct features. Indeed, if the source sentence has a lot of words and punctuation, and is therefore likely to be particularly complex, then the output will most likely be less simple as well. We also expected word concreteness ratings and position in the frequency table to be good indicators of simplicity, but it does not seem to be the case here. Structural simplicity might simply be more important than such more sophisticated components of the human intuition of simple text.
Even if counting the number of words or comparing INLINEFORM0 -grams are good proxies for the simplification quality, they are still very superficial features and might miss some deeper and more complex information. Moreover the fact that grammaticality and meaning preservation are best evaluated using INLINEFORM1 -gram-based comparison metrics might bias the TS models towards copying the source sentence and applying fewer modifications.
Syntactic parsing or language modelling might capture more insightful grammatical information and allow for more flexibility in the simplification model. Regarding meaning preservation, semantic analysis or paraphrase detection models would also be good candidates for a deeper analysis.
We should be careful when interpreting these results as the QATS dataset is relatively small. We compute confidence intervals on our results, and find them to be non-negligible, yet without putting our general observations into question. For instance, METEOR, which performs best on grammaticality, has a INLINEFORM0 confidence interval of INLINEFORM1 on the training set. These results are therefore preliminary and should be validated on other datasets.
Combination of all features with trained models
We also combine all elementary metrics and train an evaluation models for each of the three dimensions. TABLE TABREF39 presents our two best regressors in validation for each of the dimensions and TABLE TABREF39 for classifiers.
Combining the features does not bring a clear advantage over the elementary metrics METEOR and NBOutputSyllablesPerSent. Indeed our best models score respectively on grammaticality, meaning preservation and simplicity: 0.33 (Lasso), 0.58 (Ridge) and 0.49 (Ridge) versus 0.39 (METEOR), 0.58 (METEOR) and 0.49 (NBOutputSyllablesPerSent).
It is surprising to us that the aggregation of multiple elementary features would score worse than the features themselves. However, we observe a strong discrepancy between the scores obtained on the train and test set, as illustrated by TABLE TABREF32 . We also observed very large confidence intervals in terms of Pearson correlation. For instance our lasso model scores INLINEFORM0 on the test set for grammaticality. This should observe caution when interpreting Pearson scores on QATS.
On the classification task, our models seem to score best for meaning preservation, simplicity and overall, and third for grammaticality. This seems to confirm the importance of considering a large ensemble of elementary features including length-based metrics to evaluate simplicity.
Conclusion
Finding accurate ways to evaluate text simplification (TS) without the need for reference data is a key challenge for TS, both for exploring new approaches and for optimizing current models, in particular those relying on unsupervised, often MT-inspired models.
We explore multiple reference-less quality evaluation methods for automatic TS systems, based on data from the 2016 QATS shared task. We rely on the three key dimensions of the quality of a TS system: grammaticality, meaning preservation and simplicity.
Our results show that grammaticality and meaning preservation are best assessed using INLINEFORM0 -gram-based MT metrics evaluated between the output and the source sentence. In particular, METEOR and smoothed BLEU achieve the highest correlation with human judgments. These approaches even outperform metrics that make an extensive use of external data, such as language models. This shows that a lot of useful information can be obtained from the source sentence itself.
Regarding simplicity, we observe that counting the number of characters, syllables and words provides the best results. In other words, given the currently available metrics, the length of a sentence seems to remain the best available proxy for its simplicity.
However, given the small size of the QATS dataset and the high variance observed in our experiments, these results must be taken with a pinch of salt and will need to be confirmed on a larger dataset. Creating a larger annotated dataset as well as averaging multiple human annotations for each pair of sentences would help reducing the variance of the experiments and confirming our findings.
In future work, we shall explore richer and more complex features extracted using syntactic and semantic analyzers, such as those used by the SAMSA metric, and paraphrase detection models.
Finally, it remains to be understood how we can optimize the trade-off between grammaticality, meaning preservation and simplicity, in order to build the best possible comprehensive TS metric in terms of correlation with human judgments. Unsurprisingly, optimizing one of these dimensions often leads to lower results on other dimensions BIBREF38 . For instance, the best way to guarantee grammaticality and meaning preservation is to leave the source sentence unchanged, thus resulting in no simplification at all. Improving TS systems will require better global TS evaluation metrics. This is especially true when considering that TS is in fact a multiply defined task, as there are many different ways of simplifying a text, depending on the different categories of people and applications at whom TS is aimed.
Acknowledgments
We would like to thank our anonymous reviewers for their insightful comments.

Question: what approaches are compared?
Answer: MT metrics, Readability metrics and other sentence-level features, Metrics based on the baseline QuEst features, Metrics based on other features