# Readability Classification with Wikipedia Data and All-MiniLM Embeddings

Elena Vergou, Ioanna Pagouni, Marios Nanos, Katia Lida Kermanidis

► To cite this version:

Elena Vergou, Ioanna Pagouni, Marios Nanos, Katia Lida Kermanidis. Readability Classification with Wikipedia Data and All-MiniLM Embeddings. 19th IFIP International Conference on Artificial Intelligence Applications and Innovations (AIAI), Jun 2023, León, Spain. pp.369-380, 10.1007/978-3-031-34171-7_30. [https://doi.org/10.1007/978-3-031-34171-7_30](https://doi.org/10.1007/978-3-031-34171-7_30)

HAL Id: hal-05419408

https://inria.hal.science/hal-05419408v1

Submitted on 16 Dec 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
<figure><img src="image_2.png" /></figure>

ifip

International Federation for Information Processing

This document is the original author manuscript of a paper submitted to an IFIP
conference proceedings or other IFIP publication by Springer Nature. As such, there
may be some differences in the official published version of the paper. Such
differences, if any, are usually due to reformatting during preparation for publication or
minor corrections made by the author(s) during final proofreading of the publication
manuscript.
<figure><img src="image_3.png" /></figure>

# Readability Classification with Wikipedia Data and All-MiniLM Embeddings

Elena Vergou, Ioanna Pagouni, Marios Nanos, and Katia Lida Kermanidis(✉)

Department of Informatics, Ionian University, Corfu, Greece
{elenberg,p19 pago,p12 nano,kerman}@ionio.gr

**Abstract.** Evaluating the readability of text has been a critical step in several applications, ranging from text simplification, learning new languages, providing school children with appropriate reading material to conveying important medical information in an easily understandable way. A lot of research has been dedicated to evaluating readability on larger bodies of texts, like articles and paragraphs, but the application on single sentences has received less attention. In this paper, we explore several machine learning techniques - logistic regression, random forest, Naive Bayes, KNN, MLP, XGBoost - on a corpus of sentences from the English and simple English Wikipedia. We build and compare a series of binary readability classifiers using extracted features as well as generated all-MiniLM-L6-v2-based embeddings, and evaluate them against standard classification evaluation metrics. To the authors' knowledge, this is the first time this sentence transformer is used in the task of readability assessment. Overall, we found that the MLP models, with and without embeddings, as well as the Random Forest, outperformed the other machine learning algorithms.

**Keywords:** Readability classification · Text simplification · Embeddings

## 1 Introduction

Text simplification refers to the process of converting a given text, whether that is a sentence or a paragraph or even longer text, to a simpler version with higher readability while retaining the original meaning. There are many domains and applications where it's very important to have highly readable material available, as for example in the case of reading material provided to students from lower grades, for students studying foreign languages, where complicated literature text needs to be presented in a clear and simple format so that students can slowly learn and, become familiar with, the language. Another application is the medical field, where it is very important that clear and simple language is used to convey to patients critical, medical information that often contains a lot of medical terms and complicated linguistic items. Another indicative application is culture, where text describing cultural artifacts (e.g. maritime goods or regulations as handled in the ENIRISST + project<fnref n="1" />) is processed for a more intelligent transfer of

<footnote n="1">¹ https://enirisst-plus.gr/.</footnote>
cultural axioms. Text simplification is usually accomplished through a mixture of lexical and syntactical changes to the original text. On the lexical side, more complicated words are substituted by simpler, more understandable ones, while on the syntactical side the focus is on simplifying the grammatical and syntactic structure of the text, for example by eliminating subordinate or relative clauses.

A very critical step towards text simplification is the identification of the readability level of a given text. Originally, readability measures in the form of formulae were created based on a few basic metrics, for example number of words, syllables or sentences and their ratios, without any regard towards how word order factors in, or the fact that shorter sentences or words with fewer syllables are not always simpler to understand. More recent approaches, on the other hand, focus on regression or classification algorithms, leveraging several features and exploring non-linearities in the relationship, and use manually classified or graded text corpora. Employing machine learning schemata, these texts are usually either classified as simple or not, or graded along an ordinal scale. In general, typical learning features that have been considered for readability identification fall under a few high-level categories - lexical features (relative word frequencies, text probability based on language models, etc.), syntactic features (sentence length, etc.), discourse features (measures of text cohesion) and semantic features (use of idioms, type of text, etc.).

Unlike previous work, we used a corpus of Wikipedia and Simple Wikipedia sentences, cleaned them, and engineered features and generated embeddings. For the engineered features, multiple surface and part-of-speech (POS) features as well as more traditional readability measures were produced and then fed through a feature selection process to reduce the number of total features and pick the most informative ones for the readability task. Grid search cross validation was used to optimally pick hyperparameters for the models and trained an array of different ML models in order to evaluate comparative performance of different machine learning algorithms and different sources of features - engineered features as well as embeddings based on a recently published pre-trained encoder [15], that offers speed and performance and is tuned for multiple different use-cases. Unlike previous work, the state-of-the-art sentence transformer all-MiniLM-L6-v2 was used to generate embeddings for the sentences which were then used as features with several different machine learning models. Also, to the best of our knowledge, the proposed training pipeline and methodology with feature selection and grid search cross validation for the proposed feature set has not been previously used.

## 2 Related Work

Originally the task of readability assessment was based on readability formulae, with the most popular and widely used ones being the Flesch Reading Ease Score (FRES) [7] and the Flesch-Kincaid readability formula [8]. The FRES uses the average sentence length and the average number of syllables per word in a linear combination and produced an output score between 1 and 100 to evaluate readability, while the Flesch-Kincaid formula uses the same input to produce a grade level. Although these formulae, along with other readability scores, have been considered for the evaluation of text complexity in recent academic research [9], it should be noted that they were designed primarily to be applied at the document rather than the sentence level.
Another approach to readability assessment comes with the use of statistical language models. Instead of feature engineering and creating surface features based on sentence characteristics, like word or syllable counts, a language model is built based on text level distributions where counts of events act as estimates for the probabilities. In an interesting example, the probability of a grade level given a document is estimated based on the convex combination of a sentence distribution model and a unigram language model in [10] with an accuracy of 75%.

With the advances in the NLP field and the gains in computing power, the domain of readability assessment has flourished in academic research in the last two decades with many contributions on both the types of models and algorithms tested as well as the types of features used in modeling readability. In [1], the focus was on the combination of model aspect and embeddings generation, as they did not engineer any features themselves but instead chose to explore different neural network architectures - Recurrent Neural Networks (RNN), Hierarchical Attention Network (HAN), Multi-Head Attention (MHA) and Attention over Bidirectional Context (BCA) - to map the text to a vector representation which was then used in an ordinal regression to predict text complexity, and used pre-trained Glove embeddings in all models. The methodology was applied to graded textbooks and the WeeBit corpus and, based on the Spearman correlation evaluation metric, different models perform better under different circumstances; for example the HAN fared better for entire documents while the RNN did better at the paragraph level. It was also shown that a well trained neural network can generalize well outside the genres of corpora it was trained with.

In [2], the focus was on engineering a set of complex features to use in a Support Vector Machine (SVM) classifier for different grades of text complexity. The text corpus was based on a series of educational articles from the Weekly Reader with 4 different readability grades, augmented with texts from Encyclopedia Britannica and Britannica Elementary as well as CNN articles along with their simplified versions. Statistical language models were leveraged to measure the probability of sequences of co-occurring words and, based on those, new features were engineered, like perplexity, which was then used, along with more traditional features like Flesch-Kincaid score, Average number of syllables per word, Average sentence length, out-of-vocabulary (OOV) rate scores and Parse features, to estimate the complexity of the text with the SVM classifier. The best results were achieved by leveraging together language model-based and more traditional features.

An interesting contribution on the modeling side and the types of models considered, comes from [3], where the authors focused on the exploration of a series of different machine learning models - Random Forests, Decision Trees, Linear Regression, Naïve Bayes, K-Nearest Neighbors (KNN) and Support Vector Machines (SVM) - and their performance as binary classifiers along with the creation of engineered features, like Word and Character counts, Part of Speech counts etc. The methodology was applied to pairs of sentences coming from the English Wikipedia and the Simple English Wikipedia, the best performing model was the Random Forest, and the most important features were specificity and ambiguity. In [4], the authors used a corpus of both, documents and sentences, compared the performance and focused on the syntactic parsing of the sentences to engineer features. They also used lexical and POS features among others.
They observed that when moving from the document-based binary classification to the sentence-based one, accuracy dropped from 90% to 66%.

Most of the research in the last 20 years or so has concentrated on the document or paragraph level, rather than the sentence level. For example, in [11], the authors cast the problem of predicting readability of entire text passages and web pages as a multinomial Naïve Bayes classification task, leveraging multiple language models. In [12] readability assessment is applied to web queries and their resulting summaries and a gradient boosted tree estimates readability. In [13], the authors combine news articles and pairs of sentences from Wikipedia and Simple wikipedia. They break up the text simplification task into its component sub-tasks, the prediction of complexity itself and the complexity explanation, and use adversarial networks to show that leveraging complexity prediction as a first step in a text simplification pipeline reduces their overall error rates. [14] experimented with traditional readability features and more novel ones along with different machine learning approaches, like ranking, classification and regression and reported that the classification model performed best.

# 3 Data

## 3.1 Source Data and Data Cleaning

The data used, is based on the Version 2.0 of the dataset made available by D. Kauchak [16], and is made up of a corpus of approximately 167K aligned sentences sourced from the English Wikipedia and the Simple English Wikipedia, downloaded in May 2011. While these aligned sentences are meant to capture the same semantic information, the Simple English Wikipedia articles are a modified, simpler version of the English-language edition of Wikipedia with easier to understand vocabulary, grammar and syntactic structures. The labels used in the binary classification models were 1 for the sentences originating from the Simple English Wikipedia and 0 for the sentences from the English Wikipedia. After combining the two sets of sentences the total number of rows amounted to approximately 334k.

For pre-processing, all sentences of length less than 4 words were discarded, as they represent mostly templates and other assets that are mistakenly included in the corpus. Furthermore, such short sentences cannot meaningfully allow the models to learn patterns of readability. Sentences where the strings.css,.js or MediaWiki appear in the articles title were also dropped, as they were mistakenly included in the corpus and they don't represent real pairs of aligned sentences. After discarding these sentences the dataset contains approximately 331k sentences.

## 3.2 Feature Engineering and Preparation

With regard to feature preparation and engineering, the Python Natural Language Tool Kit (NLTK) was used to parse the sentences first [17]. A word tokenizer was used based on the Treebank tokenizer, which uses regular expressions, splits contractions, commas and single quotes, and treats punctuation marks as individual tokens. All tokens were then transformed to lower case. The total number of tokens as well as the total number
of words (which is equal to the initial number of tokens after removing the number of punctuation marks) were produced as features. The total number of characters, as well as their average and standard deviation per word, were calculated and used as features. The total number of syllables was also computed, as well as the average and standard deviation of the number of syllables per word. A measure of lexical diversity, the Type Token Ratio (TTR), was produced by taking the ratio of the number of unique words to the total number of words, and the same calculation was performed with tokens instead of words.

One of the most widely known readability metrics was used as well - the Flesch-Kincaid reading ease score. The score was calculated based on the weighted average number of words per sentence and the average number of syllables per word with low scores indicating complicated text. Another readability metric was also calculated - the Gunning Fog Score - which is based on the weighted average of the number of words per sentence and the ratio of the number of complex words to the total number of words, where a complex word is defined as a word with 3 or more syllables. In this particular case, since the corpus is made up of sentences, there is only one sentence per observation point. For POS tagging, the NLTK package was leveraged, which uses the Penn Treebank tagset on the entire list of tokens. This generated a list of tuples of tokens and their tags, and then the total number of nouns, adjectives, verbs, pronouns, prepositions and adverbs in the sentence was taken as an individual feature, while grouping all other part-of-speech categories in another feature. To normalize for the different sentence lengths, percentages were calculated and used as features.

In total, the features that were initially calculated were the total number of words, total number of tokens, total number of characters, average number of characters, standard deviation of the total number of characters, total number of syllables, average number of syllables per word, standard deviation of the total number of syllables per word, lexical diversity, Flesch Reading Score, number of number words, the Gunning Fog Score, percentage of verbs, adverbs, nouns, adjectives, pronouns, prepositions, the rest, percentage of verbs, percentage of adverbs, percentage of nouns, percentage of adjectives, percentage of pronouns, percentage of prepositions, percentage of remaining tokens, number of remaining tokens, number of unique tokens, number of unique words, the TTR ratio calculated based on tokens and on words. Since in the feature engineering stage, a lot of similarly calculated features were created, once with tokens and once with words, it is important to add a stage of feature selection so as to reduce the total amount of features used for modeling to a subset that involves the most informative for this task. This reduction in the size of features should help to speed up the computations required for training and reduce the amount of irrelevant or not very helpful features present in the modeling pipelines.

Outside of the engineered features, the Sentence Transformer Python package was used to generate sentence embeddings based on the input sentences that allow the retention of the semantic information in the encodings. The recently pre-trained sentence-transformers/all-MiniLM-L6-v2 transformer model was used, which has been trained on over 1 billion training pairs. It maps sentences to a 384 dimensional dense vector space and it was chosen because of its good balance between average evaluation performance and runtime performance in previously reported work [18].
## 3.3 Data Splits and Cross Validation

The data was split into train and test using 80% and 20% of the data respectively. Grid search cross validation was used, for estimating the parameters of each model. This technique searches through the space of the cross-product of the set of parameters and, at each iteration, part of the training dataset is held out for validation, while the rest is used to fit the model. Then the average performance over the different folds is used to compare the different configurations of parameters to select the best. Parameter tuning is performed while keeping the test data separate from the training and tuning process. This provides insight on the actual generalization ability of the models on unseen data, and on their ability to avoid over fitting.

## 3.4 Feature Selection

For feature selection, an XGBoost model was built on the training dataset and the resulting feature weights were used to select a feature subset. Specifically, a binary XGBoost Classifier was trained with maximum depth 4 and 200 estimators (trees), which were selected after trying out different hyperparameters. The resulting weight indicates how useful a feature was in the construction of the boosted decision trees and the more a feature is used to make decisions, the higher is its relative importance. The weight of each feature is calculated as the number of times it appears in the model trees, and then the values are normalized to sum to one. All features with weights over 3% are picked, and are shown in the chart below (Fig. 1).

<figure><img src="image_4.png" /><figcaption>Fig. 1. Feature importances by weight.</figcaption></figure>
# 4 Model Training and Results

An array of different machine learning models were trained so that the comparative performance can be evaluated for better fit with regard to this particular learning task. The specific modeling algorithms used were chosen based on an examination of other academic papers and literature on the topic of text simplification and readability assessment along with a more general examination of accurate and scalable learning techniques used in industry and academia. All models were run in two different versions - once with embeddings created by the all-MiniLM-L6-v2 encoder, which has not been used before in this task based on our search, and once with the subset of engineered features selected as a result of the feature selection process. The features were also standardized to avoid scale bias as some models are sensitive to it. For the models without embeddings, cross validation was used to pick the best model parameters, and then said parameters were used to train a model on the entire training dataset. The resulting trained model was used to produce inferences on the test dataset and report the evaluation metrics. Four typical classification evaluation metrics were produced - ROC-AUC, accuracy, recall and precision for both classes for both versions, i.e. with and without embeddings.

## 4.1 Logistic Regression

Logistic regression was used to estimate the probability of a sentence being simple or not. The logistic regression algorithm is based on the logistic function and produces outputs that represent the probability of a sentence belonging to a particular class. Grid search cross validation with precision as the scoring function was used with grid configurations with or without an intercept, and either the L1 or L2 regularization penalties. The L1 and L2 penalties can be leveraged to prevent overfitting of the model to the training data and thus improve the generalization performance to previously unseen data. This can be particularly important when the model has been trained on a domain-specific corpus and is asked to confront a different domain. Grid search cross-validation was used with the 'liblinear' solver and 100 max iterations. The best performance in cross-validation was achieved with the L1 penalty and with an intercept. When using the embeddings as features, cross-validation couldn't be used due to lack of computing power because of large data dimensionality. Maximum iterations of 200 were used along with an intercept, no penalty and the 'sag' solver as it tends to converge faster for high dimensionality data (Fig. 2).

<table><caption>Fig. 2. Logistic regression results</caption><thead><tr><th></th><th>roc_auc_score</th><th>accuracy_score</th><th>recall_score_1</th><th>precision_score_1</th><th>recall_score_0</th><th>precision_score_0</th></tr></thead><tbody><tr><td>LogisticRegression_train</td><td>0.60</td><td>0.57</td><td>0.59</td><td>0.57</td><td>0.55</td><td>0.58</td></tr><tr><td>LogisticRegression_test</td><td>0.60</td><td>0.57</td><td>0.59</td><td>0.57</td><td>0.55</td><td>0.58</td></tr><tr><td>LogisticRegression_embeddings_train_embeddings</td><td>0.61</td><td>0.58</td><td>0.57</td><td>0.58</td><td>0.55</td><td>0.58</td></tr><tr><td>LogisticRegression_embeddings_test_embeddings</td><td>0.60</td><td>0.58</td><td>0.57</td><td>0.58</td><td>0.55</td><td>0.58</td></tr></tbody></table>
## 4.2 Random Forest

A random forest classifier is an ensemble learning technique that makes use of the output of multiple decision trees to make a classification decision. The engineered features were used and experiments were run with various tree depth limits (4,6,10), estimators, i.e. how many trees we have in the random forest (100,150,200), and maximum number of features considered when looking for the best split - in ('sqrt', 'log2', 3). The best performance was achieved with max depth 6, 200 estimators and max features 'sqrt'. For the embeddings model we also built a model with max depth 6, 200 estimators and max features 'sqrt' (Fig. 3).

<table><caption>Fig. 3. Random Forest results</caption><thead><tr><th></th><th>rocord score</th><th>accuracy score</th><th>recall score</th><th>precision score</th><th>recall score</th><th>precision score</th></tr></thead><tbody><tr><td>RandomForestClassifier train</td><td>0.62</td><td>0.58</td><td>0.60</td><td>0.58</td><td>0.56</td><td>0.58</td></tr><tr><td>RandomForestClassifier test</td><td>0.62</td><td>0.58</td><td>0.60</td><td>0.58</td><td>0.56</td><td>0.59</td></tr><tr><td>RandomForestClassifier embeddings train embeddings</td><td>0.61</td><td>0.57</td><td>0.59</td><td>0.57</td><td>0.56</td><td>0.58</td></tr><tr><td>RandomForestClassifier embeddings test embeddings</td><td>0.54</td><td>0.53</td><td>0.54</td><td>0.53</td><td>0.51</td><td>0.53</td></tr></tbody></table>

## 4.3 Naïve Bayes

The Naïve Bayes classifier makes use of the Bayes' theorem to produce probabilistic estimates of class membership of an instance. Grid search cross-validation was not used for this trainer for both versions with the engineered features and the embeddings (Fig. 4).

<table><caption>Fig. 4. Naïve Bayes results</caption><thead><tr><th></th><th>rocord score</th><th>accuracy score</th><th>recall score</th><th>precision score</th><th>recall score</th><th>precision score</th></tr></thead><tbody><tr><td>GaussianNB_train</td><td>0.59</td><td>0.56</td><td>0.60</td><td>0.56</td><td>0.55</td><td>0.58</td></tr><tr><td>GaussianNB_test</td><td>0.59</td><td>0.57</td><td>0.60</td><td>0.58</td><td>0.55</td><td>0.58</td></tr><tr><td>GaussianNB_embeddings_train_embeddings</td><td>0.57</td><td>0.55</td><td>0.55</td><td>0.55</td><td>0.55</td><td>0.58</td></tr><tr><td>GaussianNB_embeddings_test_embeddings</td><td>0.56</td><td>0.54</td><td>0.55</td><td>0.54</td><td>0.55</td><td>0.58</td></tr></tbody></table>

## 4.4 K-Nearest Neighbors

The KNN classifier is a non-parametric machine learning technique that produces inferences by examining the most similar training samples nearest to the inference instance. For the KNN model, grid search cross-validation was used with the engineered features and experiments were run with the number of neighbors between 3 and 5 inclusive. The best performance was achieved with four 4 neighbors and a KNN model was also fit with the embeddings and four neighbors (Fig. 5).
<table><caption>Fig. 5. KNN results</caption><thead><tr><th></th><th>roc_auc_score</th><th>accuracy_score</th><th>recall_score_1</th><th>precision_score_1</th><th>recall_score_0</th><th>precision_score_0</th></tr></thead><tbody><tr><td>knn_train</td><td>0.75</td><td>0.67</td><td>0.46</td><td>0.79</td><td>0.55</td><td>0.58</td></tr><tr><td>knn_test</td><td>0.49</td><td>0.49</td><td>0.30</td><td>0.48</td><td>0.55</td><td>0.58</td></tr><tr><td>knn_embeddings_train_embeddings</td><td>0.62</td><td>0.58</td><td>0.30</td><td>0.68</td><td>0.55</td><td>0.58</td></tr><tr><td>knn_embeddings_test_embeddings</td><td>0.28</td><td>0.34</td><td>0.14</td><td>0.23</td><td>0.55</td><td>0.58</td></tr></tbody></table>

## 4.5 MLP (Multi-layer Perceptron)

An MLP model is a type of feedforward neural network with at least one hidden layer. In our experiments, grid search cross-validation was used with the engineered features and while using one hidden layer and the adam solver, due to the small dimensionality of the dataset and to avoid an overcomplicated model that could lead to overfit, the size of the hidden layer was used in the grid search, and 50 or 100 neurons were tried. Also, the logistic and tanh activation functions were tried along with different learning rates (0.001, 0.01). As a result of the cross-validation 50 was picked as the number of neurons in the hidden layer along with the logistic activation function and an adapting learning rate starting at 0.001. For the embeddings model, given the higher dimensionality a more complicated architecture was tried with two hidden layers and number of neurons 100 for the first hidden layer and 50 for the second hidden layer The remaining parameters remained the same as in the model with the engineered features (Fig. 6).

<table><caption>Fig. 6. MLP results</caption><thead><tr><th></th><th>roc_auc_score</th><th>accuracy_score</th><th>recall_score_1</th><th>precision_score_1</th><th>recall_score_0</th><th>precision_score_0</th></tr></thead><tbody><tr><td>MLP_train</td><td>0.62</td><td>0.58</td><td>0.52</td><td>0.80</td><td>0.65</td><td>0.57</td></tr><tr><td>MLP_test</td><td>0.62</td><td>0.58</td><td>0.52</td><td>0.59</td><td>0.64</td><td>0.57</td></tr><tr><td>mlp_embeddings_train_embeddings</td><td>0.68</td><td>0.62</td><td>0.55</td><td>0.64</td><td>0.69</td><td>0.60</td></tr><tr><td>mlp_embeddings_testCD36</td><td>0.63</td><td>0.58</td><td>0.52</td><td>0.60</td><td>0.65</td><td>0.57</td></tr></tbody></table>

## 4.6 XGBoost

The XGBoost algorithm is a gradient boosting algorithm that iteratively adds weaker decision trees with a goal to produce a stronger model. For the XGBoost model, cross-validation experiments were run with different learning rates of (0.3, 0.5, 0.7, 1, 1.2), maximum depth of (3, 4, 5, 6) and number of trees in (200, 250, 300). The optimal configuration of parameters as a result of cross validation was a depth of 5, number of trees at 200 and learning rate of 0.3. The same parameters were used for the embeddings model as grid cross validation could not be used with the embeddings model due to lack of computation power (Fig. 7).
<table><caption>Fig. 7. XGBoost results</caption><thead><tr><th></th><th>roc_auc_score</th><th>accuracy_score</th><th>recall_score_1</th><th>precision_score_1</th><th>recall_score_0</th><th>precision_score_0</th></tr></thead><tbody><tr><td>XGBoost_train</td><td>0.68</td><td>0.62</td><td>0.60</td><td>0.62</td><td>0.63</td><td>0.61</td></tr><tr><td>XGBoost_test</td><td>0.60</td><td>0.57</td><td>0.56</td><td>0.57</td><td>0.58</td><td>0.57</td></tr><tr><td>XGBoost_embeddings_train_embeddings</td><td>0.74</td><td>0.67</td><td>0.66</td><td>0.67</td><td>0.68</td><td>0.67</td></tr><tr><td>XGBoost_embeddings_test_embeddings</td><td>0.52</td><td>0.51</td><td>0.50</td><td>0.51</td><td>0.51</td><td>0.51</td></tr></tbody></table>

## 5 Evaluation

In general, the Logistic Regression, Naive Bayes and MLP models had consistent performance between training and test dataset in the evaluation metrics and for both model versions with and without embeddings, which meant that they didn't overfit the training data and are able to generalize well to unseen data. Overall, the KNN models had the worst generalization ability on unseen data, as evaluation metrics make evident that they tend to overfit the training data.

All model performances were compared also on the test dataset with all dataset versions, both with and without embeddings. In terms of the precision score the highest performance was achieved with the MLP with and without embeddings for the positive class - sentences which come from the Simple English Wikipedia - while for the other class the Random Forest with the manually created features was the top performer. In terms of recall, for the positive class, Naive Bayes and Random Forest with manually created features performed at the top, while for the other class the MLP models with and without embeddings were top performers. In terms of overall accuracy, the two MLP models with and without embeddings as well as the Random Forest model with manually created features were top performers. In terms of ROC AUC the two MLP models with and without embeddings were the best performing models.

Although not exactly comparable due to different data size and features, the academic paper with the closest set-up to ours, [16], reports an accuracy score of 84.14% for their Random Forest model while in this case it's close to 58%, their linear regression model has accuracy of 74.62% while in our case the logistic regression has accuracy of approximately 57%. One of the possible reasons behind the difference in performance is that while the raw source data was the same, the data cleaning process and the resulting number of sentences were different, the features used were different, including the embeddings, the modeling methodology and parameter tuning was different. In our case, even very short sentences were kept - of size 5 or 6 - which could make it harder for the models to pick up on differences, trends and patterns. Also, while in a pair of sentences - one from the English Wikipedia and the other from the Simple English Wikipedia - it's fair to say that one is simpler than the other as this is the starting assumption, over the entire dataset of sentences it might not be the case that each one of the complex sentences is harder than each of the simple sentences (Fig. 8).
<table><caption>Fig. 8. Consolidation and comparison of evaluation metrics by model</caption><thead><tr><th></th><th>roc_auc_score</th><th>accuracy_score</th><th>recall_score_1</th><th>precision_score_1</th><th>recall_score_0</th><th>precision_score_0</th></tr></thead><tbody><tr><td>mlp_embeddings_test_embeddings</td><td>0.629</td><td>0.583</td><td>0.515</td><td>0.595</td><td>0.651</td><td>0.575</td></tr><tr><td>MLP_test</td><td>0.621</td><td>0.582</td><td>0.520</td><td>0.592</td><td>0.644</td><td>0.574</td></tr><tr><td>RandomForestClassifier_test</td><td>0.619</td><td>0.582</td><td>0.601</td><td>0.578</td><td>0.563</td><td>0.587</td></tr><tr><td>LogisticRegression_embeddings_test_embeddings</td><td>0.605</td><td>0.577</td><td>0.587</td><td>0.577</td><td>0.553</td><td>0.578</td></tr><tr><td>XGBoost_test</td><td>0.604</td><td>0.589</td><td>0.557</td><td>0.569</td><td>0.580</td><td>0.569</td></tr><tr><td>LogisticRegression_test</td><td>0.601</td><td>0.573</td><td>0.544</td><td>0.568</td><td>0.600</td><td>0.577</td></tr><tr><td>GaussianNB_test</td><td>0.594</td><td>0.589</td><td>0.604</td><td>0.563</td><td>0.553</td><td>0.578</td></tr><tr><td>GaussianNB_embeddings_test_embeddings</td><td>0.563</td><td>0.540</td><td>0.547</td><td>0.538</td><td>0.553</td><td>0.578</td></tr><tr><td>RandomForestClassifier_embeddings_test_embeddings</td><td>0.541</td><td>0.528</td><td>0.543</td><td>0.526</td><td>0.513</td><td>0.530</td></tr><tr><td>XGBoost_embeddings_test_embeddings</td><td>0.519</td><td>0.508</td><td>0.505</td><td>0.506</td><td>0.510</td><td>0.509</td></tr><tr><td>knn_test</td><td>0.493</td><td>0.487</td><td>0.295</td><td>0.476</td><td>0.553</td><td>0.578</td></tr><tr><td>knn_embeddings_test_embeddings</td><td>0.282</td><td>0.343</td><td>0.138</td><td>0.232</td><td>0.553</td><td>0.578</td></tr></tbody></table>

## 6 Conclusions

In this paper several engineered features were explored along with generated embeddings on the task of readability assessment applied on a corpus of English and simple English Wikipedia sentences. A series of different widely used machine learning binary classifiers were separately trained with the engineered features and the all-MiniLM-L6-v2 based embeddings. As a conclusion, it was found that the MLP models with and without embeddings as well as the Random Forest models outperformed the other machine learning algorithms. To the best of our knowledge, this was the first use of the state-of-the-art sentence transformer all-MiniLM-L6-v2 in the field of text simplification to generate embeddings for the sentences to be used as features with several different machine learning models. Also to the best of our knowledge and research this exact training pipeline and methodology with feature selection and grid search cross validation with these features has not been previously used.

As a potential future direction, it would be interesting to continue experimenting with different and more complex neural network architectures as they appeared the most promising. It would also be interesting to try different pre-trained encoders to generate embeddings from the sentences to train classifiers. Another potential future direction would be to experiment with different sentence lengths as in this paper almost all sentences were included and it would be reasonable to assume that smaller sentences might have very few discernable differences between their versions, which would make it harder for a model to learn patterns to discern the two. It would also be interesting to try more experiments with parameter tuning the model trainers. It would be challenging also to expand this research with different datasets sources from various different domains (for example, scientific text, medical text, etc.) and see how the performance changes for the different domains. Also, another idea would be to experiment with an n-gram language model as input to the classifier.

**Acknowledgments.** This research was co-financed by the European Union and Greek national funds through the "Competitiveness, Entrepreneurship and Innovation" Operational Programme
2014–2020, under the Call “Support for regional excellence”; project title: “Intelligent Research Infrastructure for Shipping, Transport and Supply Chain - ENIRISST+”; MIS code: 5047041.

## References

1. Nadeem, F., Ostendorf, M.: Estimating Linguistic Complexity for Science Texts. In: 57th Annual Meeting of the Association for Computational Linguistics. Florence, Italy, pp. 4541–4551 (2019)

2. Schwarm, S., Ostendorf, M.: Reading Level Assessment Using Support Vector Machines and Statistical Language Models. In: 43rd Annual Meeting of the Association for Computational Linguistics (ACL), Michigan, USA, pp. 497–504 (2005)

3. Kauchak, D., Mouradi, O., Pentoney, C., Leroy, G.: Text Simplification Tools: Using Machine Learning to Discover Features that Identify Difficult Text. IEEE Trans. Learn. Technol. 7(3), 276–288 (2014)

4. Vajjala, S., Meurers, D.: Assessing the Relative Reading Level of Sentence Pairs for Text Simplification. In: 13th Conference of the European Chapter of the Association for Computational Linguistics (EACL), Avignon, France, pp. 482–492 (2012)

5. Nisioi, S., Štajner, S., Ponzetto, S.P., Dinu, L. P.: Exploring Neural Text Simplification Models. In: 55th Annual Meeting of the Association for Computational Linguistics (ACL), Vancouver, Canada, pp. 1083–1092 (2017)

6. Saggion, H.: Automatic Text Simplification. Cham, Springer. Switzerland (2017)

7. Flesch, R.: The Art of Readable Writing. Harper, New York (1949)

8. Kincaid, P., Robert P., Fishburne, R., Rogers, L., Chissom, B.S.: Derivation of new readability formulas (Automated Readability Index, Fog count and Flesch Reading Ease Formula) for Navy enlisted personnel. Technical report, Naval Technical Training Command. (1975) https://doi.org/10.1007/978-3-031-02166-4

9. Sander Wubben, S., van den Bosch, A., Krahmer, E.: Sentence simplification by monolingual machine translation. Long Papers. In: 50th Annual Meeting of the Association for Computational Linguistics 1, 1015–1024 (2012)

10. Si, L., Callan, J.: A statistical model for scientific readability. In: 10th International Conference on Information and Knowledge Management, CIKM, pp. 574–576, New York. ACM (2001)

11. Collins-Thompson, K., Callan, J.P.: A language modeling approach to predicting reading difficulty. HLT-NAACL, 193–200 (2004)

12. Kanungo, T., Orr, D.: Predicting the readability of short web summaries. Second ACM International Conference on Web Search and Data Mining, pp. 202–211. ACM (2009)

13. Garbacea, C., Guo, M., Carton, S., Mei, Q.: Explainable Prediction of Text Complexity: The Missing Preliminaries for Text Simplification. In: 57th Annual Meeting of the Association for Computational Linguistics, pp. 2254–2264 (2019)

14. Aluisio, S., Specia, L., Gasperin, C., and Scarton, C.: Readability Assessment for Text Simplification. In: 27th International Conference on Computational Linguistics, pp. 1246–1257 (2018)

15. Hugging Face. (n.d.). Sentence Transformers: all-MiniLM-L6-v2. Retrieved from https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2

16. Kauchak, D.: Data and Code for Automatic Text Simplification. Retrieved from https://cs.pomona.edu/~dkauchak/simplification/

17. Bird, S., Loper, E., Klein, E.: Natural Language Processing with Python. O’Reilly Media Inc. (2009)

18. Sentence Transformers: pre-trained models evaluation https://www.sbert.net/docs/pretained_models.html