Strategies for Arabic Readability Modeling (2024)

Abstract: Automatic readability assessment is relevant to building NLP applications for education, content analysis, and accessibility. However, Arabic readability assessment is a challenging task due to Arabic’s morphological richness and limited readability resources. In this paper, we present a set of experimental results on Arabic readability assessment using a diverse range of approaches, from rule-based methods to Arabic pretrained language models. We report our results on a newly created corpus at different textual granularity levels (words and sentence fragments). Our results show that combining different techniques yields the best results, achieving an overall macro F1 score of 86.7 at the word level and 87.9 at the fragment level on a blind test set. We make our code, data, and pretrained models publicly available.1

heading: 1 Introduction
text: The task of automatic readability assessment aims at modeling the reading and comprehension difficulty of a given piece of text for a particular target audience. This is relevant to building and enhancing pedagogical natural language processing (NLP) applications, which aid students in language learning (Xia et al., 2016; Vajjala and Meurers, 2012), help teachers with designing curricula and writing assessments (Collins-Thompson and Callan, 2004b), and enable the personalization of NLP systems’ output to target users with different readability levels (Marchisio et al., 2019; Agrawal and Carpuat, 2019). Research on English automatic readability assessment have garnered substantial interest in terms of dataset creation (Heilman et al., 2007; Vajjala and Meurers, 2013; Xia et al., 2016; Vajjala and Lučić, 2018) and modeling advancements (Deutsch et al., 2020; Martinc et al., 2021; Lee and Vajjala, 2022). In contrast, other languages such as Arabic have not received as much attention.
1https://github.com/CAMeL-Lab/ samer-arabic-readability
Arabic is a morphologically rich and orthographically ambiguous language. Words have many inflected forms varying in terms of gender, number, person, case, aspect, mood, voice, as well as a large number of attachable clitics, such as pronominal objects and prepositions (Habash, 2010). Arabic’s high level of complexity poses a significant challenge for new learners. Furthermore, while Modern Standard Arabic (MSA) is used in education and the media, modern-day Arabs natively speak a variety of Arabic dialects that differ from MSA, making MSA readability a relevant issue for them too. There are growing research efforts on Arabic readability assessment (Al-Khalifa and Al-Ajlan, 2010; Al Tamimi et al., 2014; El-Haj and Rayson, 2016; Saddiki et al., 2018). However, we are not aware of any work that systematically explores modeling approaches for Arabic readability at different textual granularity levels. In this paper, we present Arabic readability assessment results using diverse approaches relying on frequency and rule-based models as well as pretrained language models (PLMs). We use the newly created SAMER Arabic Text Simplification Corpus (Alhafni et al., 2024) and report on word-level and fragment-level readability. Our contributions are as follows:
• We systematically explore different modeling approaches to report on the task of Arabic readability assessment, ranging from rulebased methods to Arabic PLMs.
• We benchmark our models on a new corpus with different readability levels.
• We show that combining different modeling techniques yields optimal results: 86.7 wordlevel macro F1 and 87.9 fragment-level macro F1 on a blind test set.
We discuss related work in §2, provide an overview of our dataset in §3, describe our models for Arabic readability assessment in §4, and discuss results in §5.
55

heading: 2 Related Work

heading: 2.1 Readability Assessment Datasets
text: Automatic readability assessment has received considerable attention, leading to the development of many resources (Collins-Thompson and Callan, 2004a; Pitler and Nenkova, 2008; Feng et al., 2010; Vajjala and Meurers, 2012; Xu et al., 2015; Xia et al., 2016; Nadeem and Ostendorf, 2018; Vajjala and Lučić, 2018; Deutsch et al., 2020; Lee et al., 2021). Most of the English datasets were initially derived from textbooks as they are considered to be naturally suited for readability assessment research, given that the linguistic characteristics of texts become more complex as school grade increases (Vajjala, 2022). However, many textbooks are under copyright restrictions and may not be accessible in a digitized form. This led to relying on crowd sourcing to annotate data collected from the web (Vajjala and Meurers, 2012; Vajjala and Lučić, 2018) or from English assessment exams targeting second-language (L2) learners (Xia et al., 2016), where the Common European Framework of Reference (CEFR) (Council of Europe, 2001) is used.
When it comes to Arabic, specifically Modern Standard Arabic (MSA), early work on readability assessment relied mainly on academic curricula (Al-Khalifa and Al-Ajlan, 2010; Al Tamimi et al., 2014; Forsyth, 2014; Khalil et al., 2018). More recently, there have been more efforts to create Arabic readability assessment resources. Khallaf and Sharoff (2021) consolidated multiple annotated L2 datasets and mapped their readability levels to CEFR. Habash and Palfreyman (2022) created the ZAEBUC dataset that contains essays written by native Arabic speakers, which were manually corrected and annotated for writing proficiency using the CEFR levels. Naous et al. (2023) introduced a manually annotated multi-domain multilingual dataset for readability assessment. In our work, we use the newly introduced publicly available SAMER Arabic Text Simplification Corpus (Alhafni et al., 2024), which was manually annotated for readability leveling. We discuss the corpus in more detail in §3. It is noteworthy that this corpus is one of the publicly available resources created by the Simplification of Arabic Masterpieces for Extensive Reading (SAMER) project which includes a readability leveled lexicon (Al Khalil et al., 2020a; Jiang et al., 2020), and a Google Doc add-on (Hazim et al., 2022).

heading: 2.2 Approaches to Readability Assessment
text: Early approaches for automatic readability assessment relied on surface-level features that could be extracted from raw text such as the average number words per sentence and the average number of characters per word. Such approaches include commonly used readability measures such as the DaleChall Readability Score (Dale and Chall, 1948) and the Flesch-Kincaid Grade Level (FKGL) (Flesch, 1948). With the emergence of machine learning and data driven methods, approaches were extended to leverage statisical language models (Si and Callan, 2001) and linguistic features (Heilman et al., 2007; Petersen and Ostendorf, 2009; Ambati et al., 2016). More recently, deep learning approaches were explored (Cha et al., 2017; Jiang et al., 2018; Azpiazu and Pera, 2019), including the use of Transfomerbased PLMs (Deutsch et al., 2020; Lee and Vajjala, 2022; Naous et al., 2023; Imperial and Kochmar, 2023).
Although a lot of this research evolved on English, approaches to modeling Arabic readability assessment witnessed a similar trend. Inspired by English readability formulas, Al-Tamimi et al. (2014) developed the Arabic Automatic Readability Index (AARI). Similarly, El-Haj and Rayson (2016) introduced OSMAN, an adaptation of conventional readability formulas such as FKGL to Arabic. When it comes to machine learning models, the majority were based on linguistic features such as perplexity scores from statistical language models (Al-Khalifa and Al-Ajlan, 2010), morphological information (e.g., lemmas, morphemes, part-ofspeech tags) (Cavalli-Sforza et al., 2014; Forsyth, 2014; Saddiki et al., 2015; Nassiri et al., 2017), and syntactic features (Saddiki et al., 2018). Despite the various efforts on modeling Arabic readability assessment, only few attempts were made to explore deep learning approaches. Khallaf and Sharoff (2021) and Naous et al. (2023) presented results on using BERT (Devlin et al., 2019; Antoun et al., 2020) for readability assessment. Moreover, it is worth noting that the majority of research on Arabic readability assessment report results at either the document or sentence levels.
In our work, we draw inspiration from previous efforts to explore various modeling approaches for Arabic readability assessment at both the word and fragment levels, encompassing a spectrum from rule-based models to PLMs and their combinations.

heading: 3 Data

heading: 3.1 SAMER Corpus
text: We extensively use the SAMER Arabic Text Simplification Corpus (Alhafni et al., 2024). The corpus consists of original texts selected from 15 publicly available Arabic fiction novels. It includes two simplified parallel versions for each text targeting learners at two readability levels (Level 4 and Level 3). The levels are based on Al Khalil et al. (2020a)’s five-level lexical readability scale which ranges from Level 1 (Low Difficulty/Easy Readability) to Level 5 (High Difficulty/Hard Readability). The SAMER Corpus simplification guidelines consider the readability level of a text to be equal to the highest readability level found among the words in the text. So, a Level 4 text cannot have any Level 5 words, but must have at least one Level 4 word. As part of the manual simplification process, the human annotators simplified the original text to Level 4, and then to Level 3. This was done by first automatically obtaining the word-level readability of the original text using the SAMER Google Doc add-on (Hazim et al., 2022) and then manually performing minimal replacements, insertions, and deletions to simplify the text from a higher to lower readability levels. In some cases the annotators minimally modified some words to maintain gram-
matical agreement without changing their lexical readability levels. The add-on was used to confirm the target levels were reached; however, the annotators were allowed to overwrite incorrect automatically assigned word readability levels. The SAMER Corpus release includes the original paragraphs and their simplified counterparts segmented into smaller parallel sentence fragments using punctuation marks. Our readability assessment experiments only use the original text fragments. The release also includes the final word readability levels; however, we do not use them as we opted to employ a more generic solution for word-level readability assignment, which we discuss next.

heading: 3.2 Fragments, Words, and Readability Levels
text: We model readability assessment on three levels: Levels 5, 4, and 3, at the word level and fragment level. To assign a readability label to each word in the original fragments, we first obtain word-level alignments between the original fragments and the simplified parallels using an edit distance word alignment tool (Alhafni et al., 2023; Khalifa et al., 2021). We then derive the readability labels based on whether the words in the original fragments were changed in the simplified Levels 4 and 3 texts. See example in Figure 1. Similar to the SAMER Corpus, we consider the readability level of the
fragment to be equal to the highest word readability level found among the words in the fragment. Tables 1 and 2 present the statistics of the corpus at the fragment and word levels, respectively.
While this alignment-based approach is applicable to any parallel original-simplified text, it struggles to distinguish between lexical readability changes and grammatical agreement changes. Nevertheless, this holistic approach is valuable for text simplification tasks that require altering both words and their grammatical dependents. Importantly, this limitation does not affect the readability level of the text fragments.

heading: 4 Approach
text: We present below the set of models we use for word-level and fragment-level readability labeling.

heading: 4.1 Word-Level Readability Labeling
text: We investigate four models to label words according to their readability levels: Maximum Likelihood Estimation (MLE), Lexicon lookup, Frequency-based labeling, and BERT-based token classification. Each model relies on different resources as summarized in Table 3. Moreover, each model has a different set of parameters, which were tuned to optimize the performance on the Dev set. We further investigate combining the models in a cascaded setup, leveraging their complementary strengths to address the limitations of each model individually.

heading: 4.1.1 Maximum Likelihood Estimation
text: The MLE model assigns the readability level R that maximizes the conditional probability P (R|W ), where R is the readability level of word W as estimated over the training data. Among in-vocabulary words, 97.2% appear with one readability level in the training, and 0.2% appear with
all three. For out-of-vocabulary (OOV) words, we back-off to a default readability level or to one of other models we discuss below.

heading: 4.1.2 Lexicon
text: Our second model, Lex, leverages the SAMER Readability Lexicon (Al Khalil et al., 2020a), which consists of over 40K lemmas manually annotated with their readability levels (1 to 5). For the purposes of our task, we consider lemmas of Levels 1 and 2 to be included under Level 3. During inference, we use Inoue et al. (2022)’s morphological disambiguator as implemented in CAMel Tools (Obeid et al., 2020) to identify the lemma and part-of-speech tag for each word. We infer the readability level of the word using its lemma’s readability level in the SAMER lexicon. In cases where the morphological disambiguator returns multiple top lemma analyses, we select the lowest readability associated with these lemmas. For OOV words, we back-off to a default readability level or to one of other models we discuss below.

heading: 4.1.3 Frequency-Based Models
text: Given the limited vocabulary seen the SAMER Corpus, we explore different approaches to derive readability levels from frequency data building on the known observation about the inverse correlation between frequency and readability levels (Al Khalil et al., 2020b): more frequent words have easier/lower readability levels. We leverage our SAMER Corpus training data and link it with type frequency data from a corpus of 12.6B word tokens (11.4M types) used to pretrain the CAMeLBERT models (Inoue et al., 2021).2 We sort the 11.4M word types by frequency and divide them into adjacent bins for which we assign readability levels using one of two methods:
Distribution-based Labeling (Dist-Freq) In this method, we divide the word types into three bins that mirror the distribution of readability levels in our training data (as seen in Table 2). More concretely, the most frequent words that account for 86.5% of the total distribution mass are assigned to Level 3, followed by 9.0% assigned to Level 4, and the remaining tail assigned to Level 5.
Example-based Labeling (Ex-Freq) Based on the assumption that types of a certain readability level tend to have similar frequencies in a large
2https://github.com/CAMeL-Lab/Camel_Arabic_ Frequency_Lists
corpus, we divide the frequency-sorted types into equally sized bins based on cumulative frequency. Utilizing our training data as training examples, we assign a readability level to all the types within each bin according to the majority readability level of training words found in that bin. During inference, if words are not observed in any bin, we default to assigning a readability level of 5, reflecting the expectation that rare words are typically harder to read. We empirically experiment with different numbers of bins and found that 10,000 bins yield the highest performance in terms of macro F1 score.

heading: 4.1.4 BERT Token Classification
text: We build a word-level classifier by leveraging a Transformer-based PLM. There are many Arabic monolingual BERT (Devlin et al., 2019) models available such as AraBERT (Antoun et al., 2020), ARBERT (Abdul-Mageed et al., 2021), and JABER (Ghaddar et al., 2022). However, we chose to use CAMeLBERT MSA (Inoue et al., 2021) as it was pretrained on the largest MSA dataset to date, and following the recommendations of Inoue et al. (2021) to use it for tasks on MSA. We finetune CAMeLBERT MSA using Hugging Face’s Transformers (Wolf et al., 2020) by adding a fullyconnected linear layer with a softmax on top of its architecture. Given that BERT operates at the subword-level (i.e., wordpieces), we assign to each subword the readability level of the word it belongs to. During inference, we label each word according to the highest readability level among its subwords. We fine-tune our model on a single GPU for 10 epochs with a learning rate of 1e-5, a batch size of 32, and a maximum sequence length of 30.

heading: 4.1.5 System Combination
text: In addition to evaluating the various models discussed above, we consider their combinations to exploit their complementarities. Our approach to combining the systems runs the Lex and MLE models first (independently and together in different orders – four combinations) followed by one of the following six models: default level 3, 4, or 5, Dist-Freq, Ex-Freq, or BERT. The total is 24 combinations layered in two or three steps. See Table 9 in Appendix A. Since the early layers, Lex and MLE, do not handle unknown words, the later layers resolve these cases. We evaluate all these system combinations on both word and fragment leveling in terms of accuracy and macro F1 score.

heading: 4.2 Fragment-Level Readability Labeling
text: We consider two approaches to fragment-level labeling: a direct BERT-based approach and an aggregation of the word-level predictions.

heading: 4.2.1 BERT Fragment Classification
text: We train a fragment-level classifier by fine-tuning CAMeLBERT MSA. We add a fully connected linear layer on top of the representation of the whole fragment. We experiment with different ways of obtaining the fragment representation from the BERT model: using the [CLS] token, mean-pooling, and max-pooling, and found mean-pooling to outperform the other representations in all evaluation metrics. We fine-tune our model using Hugging Face’s Transformers (Wolf et al., 2019) on a single GPU for 10 epochs with a learning rate of 5e-5, a batch size of 32, and a maximum sequence length of 20.

heading: 4.2.2 Aggregating Word-Level Predictions
text: Finally, we aggregate the word-level labels produced by the various models discussed in §4.1 above to assign fragment-level labels: the fragment label equals the highest readability level found among its words.

heading: 5 Results
text: We present and discuss the results of our evaluation below. The complete set of results for word-level and fragment-level labeling across all experimental setups is available in Appendix A.

heading: 5.1 Word-Level Labeling Results
text: Table 4 presents the results on the Dev set. We start off with the results of the standalone models in Table 4(a). The frequency-based approaches (Dist-Freq and Ex-Freq) improve over the majority class baseline (Default Level 3). However, they are outperformed by BERT. This improvement is attributed to the significant increase in the F1 scores for Level 4 and Level 5 words. In Table 4(b) we show that the results improve further when combining the MLE model with BERT as a back-off system.
Results in Table 4(c) show that using the frequency-based and BERT models as back-off systems to Lex improve the results compared to defaulting to Level 3, with Lex → BERT being the best performer. However, the improvements when using a back-off model to Lex are not as large as the ones observed when using the MLE model (Table 4(b)). This is due to the larger coverage the
Lexicon has on the Dev set (96.4% of all tokens) compared to the MLE system (79.0%).
Finally, in Table 4(d) we present the maximal combination results. We find that using the MLE model, followed by Lex and then BERT yields the best results. We further tune this combination by considering different probability thresholds at which to back-off from MLE. We found 85% MLE minimum probability to give the best results on the Dev set. Our best model combination is thus Tuned-MLE → Lex → BERT with 87.9 F1.

heading: 5.2 Fragment-Level Labeling Results
text: Table 5 presents the fragment-level results on the Dev set. We find that, although the fragment-level
BERT classifier does better than its word-level counterpart, the aggregated word-level models perform better on the fragment-level. We obtain the best results using the (Tuned-MLE → Lex → BERT) model, achieving an F1 score of 87.9. It is interesting to note that the best system coincidentally achieves the same overall F1 macro at the word and fragment levels. Our best system is better at predicting Level 3 words compared to Level 3 fragments (98.2 v.s. 92.1). Conversely, the system is better at predicting Level 4 and Level 5 fragments compared to the words. This makes sense given that Level 3 fragments are exclusively composed of Level 3 words, any word-level error on a Level 3 fragment leads to fragment error. In sum, the 3.7%
accuracy errors at the word level lead to 11.4% accuracy errors at the fragment level. Table 6 presents a detailed breakdown of the combinations of word and fragment errors.
Finally, we revisit our best model combination Tuned-MLE → Lex → BERT in Table 7, where we give a summary of the decisions and mistakes made by each of its three components and their effect on word-level and fragment-level performance. We notice that most of the decisions were taken by the MLE model, which had the lowest error rate, and the lowest rate of error propagation to the fragment level. However, when errors at the word level happen, there is a large chance a fragment error will follow suit in all three models. Moreover, we note that the performance is highly degraded by the last model (BERT) decisions, with 35.6% word-level and 67.7% fragment-level errors.

heading: 5.3 Blind Test Results
text: Table 8 presents the results on the Test set. We observe consistent conclusions to the Dev results. Our best system (Tuned-MLE → Lex → BERT) achieves an overall F1 score of 86.7 at the word level and 87.9 at the fragment level.

heading: 5.4 Manual Error Analysis
text: We manually classified 100 cases of word readability errors from the Dev set (out of 814 or 3.7% of all words) into seven distinct error types. We provide a brief description of each error type below, with its percentage of occurrence. The errors are presented in order of precedence, so if there is an Input error, we do not consider any other error below it, and so on.
Input Error: 3% The word is malformed in terms of spelling; e.g., é ̄PAªÓ mςArfh̄ instead of é ̄PAªÓ mςArfh ‘his features’.
Gold Reference Annotation Error: 18% The human annotator made a mistake of undersimplification or over-simplification, e.g., rewriting K Q¢Ë@ Q » @ Aª¢ ® ̄ ‘they crossed most of the road’
as K Q¢Ë@ Q » @ AJ Ô ̄ ‘they walked most of the road’
(L4) is unnecessary since the original is not L5.
Gold Reference Determination Error: 8% As discussed in §3.2, our process to determine the word-level readability confused grammatical agreement changes with lexical simplification changes, e.g., the phrase èYªj. JÖÏ @ ém×CÓ mlAmHh Almtjςdh̄ ‘his wrinkled features’ is simplified correctly to Yªj. JÖÏ @ éêk. ð wjhh Almtjςd ‘his wrinkled face’ by changing the first word’s lemma and only changing the gender agreement of the second word; however both are considered changed and thus assigned a higher level.
MLE Error: 22% The MLE model misclassified a word, e.g., confusing YêÓ mahd ‘cradle’ (L5) with the verb mah∼ad ‘he paved’ (L4).
Disambiguation Error: 31% The Lex model misclassified a word whose lemma is in the lexicon, because of morphosyntactic or lemmatization choice errors, e.g.,
Y ® JÓ manfað ‘outlet’ (L4) is incorrectly identified as munaf∼ið ‘executor’ (L3).
Lexicon Error: 11% The correct lemma is not in the lexicon, and an incorrect lemma is chosen, e.g., for the word A ÓB lAmsA the system chose the verbal analysis laAmas ‘touched’ instead of the nominal active participle laAmis ‘touching’.
BERT Error: 7% The word is OOV in the lexicon, and BERT misclassified it, e.g., the lemma Õç 'Ag HAŷm ‘hovering’ (annotators assigned L5) is not in the lexicon, and BERT misclassified it as L4.
The lexicon and disambiguation errors take a significant share of all errors and direct us towards working on improving these resources in the future; as better generalizing models are developed, we would rely less on the MLE model. The rate of gold errors is low and within reason given the complexity of the task.

heading: 6 Conclusions and Future Work
text: We explored the problem of Arabic readability assessment using a diverse set of approaches relying on frequency and rule-based models as well as Arabic pretrained language models (PLMs). We reported results using a newly manually created corpus at both the word and fragment levels. We further highlighted the strengths and weaknesses of each approach and underscored the importance of employing different strategies to address Arabic readability assessment effectively. Our findings demonstrate that combining different modeling techniques yields the best results, achieving an overall macro F1 score of 86.7 at the word level and 87.9 at the fragment level.
In future work, we plan to explore the effect of various linguistic features in enhancing machine learning models for Arabic readability assessment. We plan to continue to improve basic enabling technologies such as morphological disambiguation and lemmatization and study their effect on readability models. We further plan to employ our best results in the development of online tools to support pedagogical NLP applications.

heading: Acknowledgements
text: We acknowledge the support of the High Performance Computing Center at New York University Abu Dhabi.
Limitations
We acknowledge the following limitations.
• By focusing on lexical readability, the approach used to create the SAMER corpus ignores many readability related phenomena such as phonological, morphological and syntactic complexity.
• The SAMER corpus does not cover all variations of Arabic text genres, which limits the robustness of the results.
• The assessment at three readability levels might not capture the full complexity of text readability at wider age and education level ranges.
• The study lacks human evaluation to corroborate the automatic readability assessments, which is crucial for validating the practical effectiveness of the models.

heading: A All Word-level Model Combination Results

heading: Future Work
text: In future work, we plan to explore the effect of various linguistic features in enhancing machine learning models for Arabic readability assessment. We plan to continue to improve basic enabling technologies such as morphological disambiguation and lemmatization and study their effect on readability models. We further plan to employ our best results in the development of online tools to support pedagogical NLP applications