Hybrid Models for Sentence Readability Assessment
Year: 2023

Abstract:
Automatic readability assessment (ARA) predicts how difficult it is for the reader to understand a text. While ARA has traditionally been performed at the passage level, there has been increasing interest in ARA at the sentence level, given its applications in downstream tasks such as text simplification and language exercise generation. Recent research has suggested the effectiveness of hybrid approaches for ARA, but they have yet to be applied on the sentence level. We present the first study that compares neural and hybrid models for sentence-level ARA. We conducted experiments on graded sentences from the Wall Street Journal (WSJ) and a dataset derived from the OneStopEnglish corpus. Experimental results show that both neural and hybrid models outperform traditional classifiers trained on linguistic features. Hybrid models obtained the best accuracy on both datasets, surpassing the previous best result reported on the WSJ dataset by almost 13% absolute.

heading: 1 Introduction
text: Text readability is defined as the cognitive load of a reader to comprehend a text (Martinc et al., 2021). Research on automatic readability assessment (ARA) has traditionally aimed at passages (Azpiazu and Pera, 2019), e.g., labeling a passage with its difficulty level.
There has been growing interest in assessing the difficulty of individual sentences (Štajner et al., 2017; Brunato et al., 2018; Lu et al., 2020; Schicchi et al., 2020), given its application in various downstream tasks in natural language processing (NLP). It is essential to generation tasks that are sensitive to language difficulty, such as pedagogical material and exercises (Pilán et al., 2014). It also facilitates explainable text simplification (Gârbacea et al., 2021) by identifying which sentences require simplification. Sentence-level ARA is a task in its own right since a substantial drop in performance
has been observed when passage-level ARA models are applied on individual sentences (Kilgarriff et al., 2008; Pilán et al., 2016).
Similar to many other NLP tasks, passage-level ARA has benefited from the advent of neural approaches (Filighera et al., 2019; Tseng et al., 2019; Martinc et al., 2021). Recent research has also applied ‘hybrid’ models, which leverage both linguistically motivated features and neural models (Deutsch et al., 2020; Lee et al., 2021; Lim et al., 2022). For sentence-level ARA, although neural models have been evaluated (Schicchi et al., 2020; Arase et al., 2022), there has not been any attempt to integrate linguistic features.
This paper applies neural models and hybrid models on sentence-level ARA and compares their performance with a non-neural classifier trained on linguistic features. To our knowledge, this is the first study on hybrid models for sentencelevel ARA. Experimental results show that a hybrid model offers the best performance, and surpasses the previous best result reported on the Wall Street Journal dataset (Brunato et al., 2018). 1

heading: 2 Previous work

heading: 2.1 Neural and hybrid approaches
text: Readability formulas (Kincaid et al., 1975) and traditional approaches for readability assessment have mostly relied on one-hot linguistic features and language models (Collins-Thompson, 2008; Sung et al., 2015). More recent studies have shown that neural approaches can improve assessment performance (Azpiazu and Pera, 2019; Martinc et al., 2021). An active area of ARA research is to investigate how to incorporate linguistic features into neural models. On passage-level assessment, some studies observed no effect (Deutsch et al., 2020) or only marginal improvement (Filighera et al., 2019)
1All data and code are publicly released at https:// github.com/ffliu6/Hybrid4SentenceARA.
448
from linguistic features, while others reported significant improvement, e.g. by combining Random Forest and RoBERTa (Lee et al., 2021), and concatenating linguistic features with sentence embeddings from BERT hidden layers (Imperial, 2021). However, there has not yet been any study on hybrid models on sentence-level ARA.

heading: 2.2 Sentence readability assessment
text: Most previous research on sentence readability pursued binary classification or pairwise difficulty prediction (Ambati et al., 2016; Schumacher et al., 2016). An algorithm combining rule-based and statistical classifiers yielded 71% accuracy on binary classification of texts for learning Swedish as a foreign language (Pilán et al., 2014). Statistical classifiers achieved 66% accuracy on an English dataset based on Wikipedia and Simple Wikipedia (Vajjala and Meurers, 2014) and between 78.9% and 83.7% on an Italian dataset (Dell’Orletta et al., 2014).
There have also been a few studies on sentence-level ARA involving multi-way classifiers trained with traditional machine learning methods. Brunato et al. (2018) developed an SVM linear regression model with a variety of surface, morphological and syntactic features. The model achieved 59.1% and 60% accuracy on an Italian and an English dataset of sentences graded on a 7-point scale. Sentence length and nominal modification were found to correlate significantly with sentence difficulty. A Bayesian Ridge Regression Model, trained on a variety of linguistic features including syntax, lexical, morphology and cohesion, has been shown to achieve high correlation with human judgment on German sentence difficulty (Weiss and Meurers, 2022). A classifier has also been trained on features derived from the phrase complexity level of n-grams (Štajner et al., 2017). It attained 0.66 weighted F-score on an English dataset on a 5-point scale. A classifier for Chinese sentences, based on vocabulary and grammar points, reached 31.92% accuracy on 10-way classification (Lu et al., 2020).
Two studies have applied neural models on sentence-level ARA. Schicchi et al. (2020) showed that an RNN-based architecture outperformed Vec2Read (Mikolov et al., 2013). Arase et al. (2022) found that the BERT-base model outperformed traditional machine learning classifiers on their annotated CEFR-based sentence difficulty dataset. However, they did not attempt to incorporate any linguistic features. This paper aims to
fill in this gap with a comparison of neural models, hybrid models and traditional classifiers.

heading: 3 Data
text: We used the following two datasets in our experiments. Detailed statistics are shown in Table 3 and Table 4 (see Appendix A).

heading: 3.1 Wall Street Journal (WSJ)
text: This corpus (Brunato et al., 2018) consists of 1,200 sentences drawn from the Wall Street Journal (Nivre et al., 2007) and graded on a difficult scale from 1 to 7. Each sentence was rated by 20 native speakers on a difficult scale from 1 (“very easy”) to 7 (“very difficult”). Our evaluation is based on the set of 650 sentences whose grade was agreed upon by at least 14 of the 20 annotators. While it is possible to restrict the evaluation to sentences with an even higher rate of agreement, it would lead to a substantially smaller dataset, whose size is already much smaller than other datasets. 2

heading: 3.2 OneStopEnglish (OSE)
text: This corpus (Vajjala and Lučić, 2018) consists of aligned texts graded at three reading grades: beginner, intermediate, and advanced. Each of the 189 texts has three versions corresponding to these grades, with a total of 19,904 sentences in the 567 texts. 3
Instead of assigning the grade of the text to all sentences in that text (Pilán et al., 2014), we determined the difficulty of each individual sentence based on the human revision. Among the sentences in intermediate texts, 10.21% appear verbatim in the beginner version; among those in the advanced texts, 18.76% appear verbatim in one of the lower versions. These sentences are labeled with the lowest grade at which they appear. All other sentences are labeled with the grade of the text — the fact the human editors revised them implies that their grade could not be lower.

heading: 4 Approach

heading: 4.1 Baseline: Linguistic Model
text: We used the scikit-learn implementation of Random Forests (RF) and XGBoost (XGB) (Pedregosa
2No sentence in this subset was graded at 6 or 7. 3Sentence segmentation was performed with NLTK (Bird
et al., 2009).
et al., 2011). We extracted 255 linguistic features with LingFeat4 for each sentence. We performed feature selection with the Variance Threshold in scikit-learn on the dev set.5 Similar to Lu et al. (2020), we trained these classifiers with linguistic features as well as bag-of-word features.

heading: 4.2 Baseline: Neural Model
text: Transformer-based neural models have achieved impressive performance in many natural language processing tasks.
We fine-tuned BERT (Devlin et al., 2019), BART (Lewis et al., 2019), RoBERTa (Liu et al., 2019), XLNet (Yang et al., 2019) and ELECTRA (Clark et al., 2020) on our datasets (Section 3) into an ARA classifier6 , using the pre-trained versions released by Huggingface (Wolf et al., 2019). We used the base versions of all of the above, as well as the large versions of BART, RoBERTa and ELECTRA.

heading: 4.3 Hybrid Models
text: We implemented three hybrid models. The following model incorporates linguistic features into a neural model:
Concatenated Model Similar to Song et al. (2021), the input to model consists of the input sentence w1w2 . . . wn concatenated with the linguistic features f1, f2 . . . fn, in the format “[CLS] w1w2 . . . wn [SEP] f1f2 . . . fn”.
The following two models wrap the linguistic features and neural model output in a non-neural statistical classifier:
Hard Label Following Deutsch et al. (2020), the grade of the sentence, as predicted by the Neural Model (Section 4.2), serves as an additional feature in the statistical classifier (Section 4.1).
Soft Labels Following Lee et al. (2021), the probability of each grade, as predicted by the Neural Model (Section 4.2), serve as additional features alongside the linguistic features in the statistical classifier (Section 4.1).
4https://github.com/brucewlee/lingfeat 5The threshold set to 0.8. 6We used the Adam algorithm (Kingma and Ba, 2015) for optimization. The epoch for each training is 10, and set the maximum word embedding size as 128.

heading: 5 Experiments

heading: 5.1 Set-up
text: We report results in terms of accuracy (Acc.), F1score, Precision, Recall and QWK scores.
We used stratified ten-fold cross validation in WSJ and OSE experiments, with a 8:1:1 split for training, development and testing.7 For the OSE dataset, all sentences from the same text are placed in the same fold, so that the entities and topics mentioned in the test sentences would not be seen during training.

heading: 5.2 Results
text: Linguistic Model. XGBoost (XGB) outperformed Random Forest (RF) and Linear Regression (LR) on all datasets. On OSE and WSJ, it achieved 0.451 and 0.618 accuracy, respectively, compared to 0.412 and 0.551 for RF, and 0.374 and 0.413 for LR. We will therefore present results based on XGB in the remainder of this section.
Neural Model. Table 1 presents the performance of neural models on the WSJ and OSE datasets. On the WSJ dataset, RoBERTa obtained the best performance among base versions, at a 0.668 accuracy. Large models were found to outperform base versions on the WSJ dataset, in which BART-large produced the highest accuracy at 0.679. On the OSE dataset, BART obtained the best performance among base versions, at a 0.571 accuracy. Large models were also found to outperform base versions on the OSE dataset, in which BART-large produced the highest accuracy at 0.571. Generally, BART-large model achieved the best performance on all datasets, at 0.679 and 0.571 accuracy for the WSJ and OSE datasets, respectively. We will therefore use its predictions for hybrid models.
The results for OSE and WSJ in Table 2 are based on the BART-large model, which obtained the best performance on both datasets. Consistent with results from passage-level ARA, the Neural Model achieved better performance over the Linguistic Model on both datasets in all metrics. Despite the relatively small amount of training data in the WSJ datasets, the Neural Model still offered competitive performance.
Hybrid Models. The previous best published result for the WSJ dataset 0.600, obtained with an
7The hyperparameters for learning rate, dropout and batch size are tuned on the dev set. We found best performance with learning rate at 1 · e−5, dropout at 0.2, and set batch size as 32.
SVM model (Brunato et al., 2018). The Hybrid Model with Hard Label surpassed this result by almost 13% absolute to achieve state-of-the-art result, at 0.729 accuracy. The Soft Labels Model produced the second best performance, followed by the Neural Model. The Concatenated Model did not outperform the Neural Model, which may because long complex sequences and the size of dataset easily lead to overfit on the transformerbased models. The improvement of the Hard Label Model over the Neural Model8 was statistically significant.
On the OSE dataset, the Soft Labels Model obtained the best performance in accuracy, though at a lower accuracy (0.581) than on the WSJ dataset. This likely reflects more fuzzy boundaries between the categories in the OSE corpus, where all sentences in the original texts were used. The Hard Label Model produced the second best performance as OSE dataset, followed by the Neural Model also. The Concatenated Model obtained worse perfor-
8At p < 3.6 · e−6 according to McNemar’s Test.
mance than Neural Model also. The improvement of the Soft Label Model over the Neural Model9 was statistically significant.

heading: 6 Conclusion
text: We have presented the first study on hybrid models on automatic readability assessment (ARA) at the sentence level. Our contribution is two-fold. First, we demonstrated that hybrid models outperform neural models, suggesting that linguistic features can capture salient properties that indicate sentence difficulty. Second, we compared three types of hybrid model, and showed that using the neural model’s predictions as features in a traditional classifier yielded the best result, surpassing the previous best published result on the WSJ dataset by almost 13% absolute. These experimental results are expected to help inform future research on sentence-level ARA.
9At p < 1.4 · e−4 according to McNemar’s Test.

heading: 7 Limitation
text: Our experimental results should be interpreted with the following limitations in mind. First, our experiments involved relatively small datasets in English only. The performance of the model should also be evaluated on other languages and larger datasets. Second, the improvement observed in our best models depends on both the efficacy of the linguistic features and on the strength of the neural model itself. As neural models continue to improve and effective linguistic features are identified, the best methods for combining may also need to be updated.

heading: Acknowledgement
text: This work was partly supported by the Language Fund from the Standing Committee on Language Education and Research (project EDB(LE)/PR/EL/203/14) and by the General Research Fund (project 11207320).

heading: A Appendix: Corpus statistics

heading: Future Work
text: These experimental results are expected to help inform future research on sentence-level ARA. 9At p < 1. 4 · e−4 according to McNemar’s Test