A Large and Balanced Corpus for Fine-grained Arabic Readability Assessment
Authors: Khalid Elmadani, Nizar Habash, Hanada Taha
Accepted at: acl25

A Large and Balanced Corpus for Fine-grained Arabic Readability Assessment

Khalid N. Elmadani,† Nizar Habash,† Hanada Taha-Thomure‡
†Computational Approaches to Modeling Language Lab, New York University Abu Dhabi
‡Zai Arabic Language Research Centre, Zayed University
{khalid.nabigh,nizar.habash}@nyu.edu, Hanada.Thomure@zu.ac.ae

Abstract

This paper introduces the Balanced Arabic Readability Evaluation Corpus (BAREC),1 a large-scale, fine-grained dataset for Arabic readability assessment. BAREC consists of 69,441 sentences spanning 1+ million words, carefully curated to cover 19 readability levels, from kindergarten to postgraduate comprehension. The corpus balances genre diversity, topical coverage, and target audiences, offering a comprehensive resource for evaluating Arabic text complexity. The corpus was fully manually annotated by a large team of annotators. The average pairwise inter-annotator agreement, measured by Quadratic Weighted Kappa, is 81.8%, reflecting a high level of substantial agreement. Beyond presenting the corpus, we benchmark automatic readability assessment across different granularity levels, comparing a range of techniques. Our results highlight the challenges and opportunities in Arabic readability modeling, demonstrating competitive performance across various methods. To support research and education, we make BAREC openly available, along with detailed annotation guidelines and benchmark results.2

1 Introduction

Text readability impacts understanding, retention, reading speed, and engagement (DuBay, 2004). Texts above a student’s readability level can lead to disengagement (Klare, 1963). Nassiri et al. (2023) highlighted that readability and legibility depend on both external features (e.g., production, fonts) and content. Text leveling in classrooms helps match books to students’ reading levels, promoting independent reading and comprehension (Allington et al., 2015). Developing readability models is crucial for improving literacy, language learning, and academic performance.

Readability levels have long been a key component of literacy teaching and learning. One of the most widely used systems in English literacy is Fountas and Pinnell (Fountas and Pinnell, 2006), which employs qualitative measures to classify texts into 27 levels (A to Z+), spanning from kindergarten to adult proficiency. Similarly, Taha-Thomure (2017)’s system for Arabic has 19 levels from Arabic letters ١ A to ٢٩ Q. These fine-grained levels are designed for pedagogical effectiveness, ensuring young readers experience gradual, measurable progress, particularly in early education (K–6) (Barber and Klauda, 2020). A key advantage is that they can be easily mapped to coarser levels with fewer categories, which may be more efficient for broader applications in readability research and automated assessments.

In this paper we present the Balanced Arabic Readability Evaluation Corpus (BAREC) – a large-scale fine-grained readability assessment corpus across a broad space of genres and readability levels. Inspired by the Taha/Arabi21 readability reference (Taha-Thomure, 2017), which has been instrumental in tagging over 9,000 children’s books, BAREC seeks to establish a standardized framework for evaluating sentence-level3 Arabic text readability across 19 distinct levels, ranging from kindergarten to postgraduate comprehension.

Our contributions are: (a) a large-scale curated corpus with 69K+ sentences (1M+ words) spanning diverse genres; and (b) benchmarking of automatic readability assessment models across multiple granularities, including both fine-grained (19 levels) and collapsed tiered systems (e.g., five-level and three-level scales) to support various research and application needs, aligning with previous Arabic readability frameworks (Al Khalil et al., 2018; Al-Khalifa and Al-Ajlan, 2010).

1 بارق bAriq is Arabic for ‘very bright and glittering’.
2 http://barec.camel-lab.com
3 We use sentence to refer to any standalone text segment, including phrases and single words (e.g., book titles).
2 Related Work

Automatic Readability Assessment  Automatic readability assessment has been widely studied, resulting in numerous datasets and resources (Collins-Thompson and Callan, 2004; Pitler and Nenkova, 2008; Feng et al., 2010; Vajjala and Meurers, 2012; Xu et al., 2015; Xia et al., 2016; Nadeem and Osten-dorf, 2018; Vajjala and Lučić, 2018; Deutsch et al., 2020; Lee et al., 2021). Early English datasets were often derived from textbooks, as their graded content naturally aligns with readability assessment (Vajjala, 2022). However, copyright restrictions and limited digitization have driven researchers to crowdsource readability annotations from online sources (Vajjala and Meurers, 2012; Vajjala and Lučić, 2018) or leverage CEFR-based L2 assessment exams (Xia et al., 2016).

Arabic Readability Efforts  Arabic readability research has focused on text leveling and assessment across various frameworks. Taha-Thomure (2017) proposed a 19-level system for children’s books based on qualitative and quantitative criteria. Other efforts applied CEFR leveling to Arabic, including the KELLY project’s frequency-based word lists, manually annotated corpora (Habash and Palfreyman, 2022; Naous et al., 2024), and vocabulary profiling (Soliman and Familiar, 2024). El-Haj et al. (2024) introduced DARES, a readability assessment dataset collected from Saudi school materials. The SAMER project (Al Khalil et al., 2020) developed a lexicon with a five-level readability scale, leading to the first manually annotated Arabic parallel corpus for text simplification (Al-hafni et al., 2024). Automated readability assessment has also been explored through rule-based and machine learning approaches. Early models relied on surface-level features like word and sentence length (Al-Dawsari, 2004; Al-Khalifa and Al-Ajlan, 2010), while later work incorporated POS-based and morphological features (Forsyth, 2014; Saddiki et al., 2018). The OSMAN metric (El-Haj and Rayson, 2016) leveraged script markers and diacritization, and recent efforts (Liberato et al., 2024) achieved strong results using pretrained models on the SAMER corpus.

Building on these efforts, we curated the BAREC corpus across genres and readability levels, and manually annotated it at the sentence-level based on an adaptation of Taha/Arabi21 guidelines (Taha-Thomure, 2017), offering finer-grained control and a more objective assessment of textual variation.

3 BAREC Corpus Annotation

In this section, we summarize the guidelines and annotation process. For more details, see Habash et al. (2025). In the next section, we discuss corpus selection and statistics.

3.1 BAREC Guidelines

We present below a summarized version of the BAREC annotation guidelines. A detailed account of the adaptation process from Taha-Thomure (2017)’s guidelines is in Habash et al. (2025).

Readability Levels  The readability level system of Taha-Thomure (2017) uses the Abjad order of Arabic letters for 19 levels: 1-alif, 2-ba, 3-jim, through to 19-qaf. This system emphasizes a finer distinction in the lower levels, where readability is more varied. The BAREC pyramid (Figure 1) illustrates the scaffolding of these levels and their mapping to, guidelines components, school grades, and three collapsed versions of level size 7, 5, and 3. All four level types (19-7-5-3) are fully aligned to easy mapping from fine-grained to coarse-grained levels. We present results for these levels in Section 6.

Readability Annotation Principles  The guidelines focus on readability and comprehension, considering the ease of reading and understanding for independent readers. The evaluation does not depend on grammatical analysis or rhetorical depth but rather on understanding basic, literal meanings. Larger texts may contain sentences at different readability levels, but we focus on sentence-level evaluation, ignoring context and author intent.

Textual Features  Levels are assessed in six key dimensions. Each of these specify numerous linguistic phenomena that are needed to qualify for being ranked in a harder level. Annotators assign each sentence a readability level based on its most difficult linguistic phenomenon. The Cheat Sheet used by the annotators in Arabic and its translation in English are included in Appendix A.

1. Spelling: Word length and syllable count affect difficulty.
2. Word Count: The number of unique words determines the highest level for easier levels.
3. Morphology: We distinguish between simple and complex morphological forms including the use of clitics and infrequent inflectional features, such as the dual.
Figure 1: The BAREC Pyramid illustrates the relationship across BAREC levels and linguistic dimensions, three collapsed variants, and education grades.

<table>
  <tr>
    <th>RL</th>
    <th>Arabic Sentence/Phrase</th>
    <th>Translation</th>
  </tr>
  <tr>
    <td>1-alif</td>
    <td>ألف</td>
    <td>Rabbit</td>
  </tr>
  <tr>
    <td>3-jim</td>
    <td>أنا أحب اللون الأحمر.</td>
    <td>I love the color red.</td>
  </tr>
  <tr>
    <td>5-ha</td>
    <td>القطة تشرب على السرير وتشم رائحة الشمس الدافئة</td>
    <td>The cat rests on the bed and enjoys the warm sunshine.</td>
  </tr>
  <tr>
    <td>7-zay</td>
    <td>الأصدقاء يحتفلون بعيد ميلاد صديقهم بكعكة وهدايا رائعة</td>
    <td>Friends celebrate their friend's birthday with cake and amazing gifts.</td>
  </tr>
  <tr>
    <td>10-ya</td>
    <td>سألت هل قلت شيءًا بالكذب قال إن يقول ما قال كذبًا أن لا</td>
    <td>I asked you whether you were accusing him of lying before he said what he said, and you said no.</td>
  </tr>
  <tr>
    <td>12-lam</td>
    <td>لا أحد يجمع هذه الزهرة ما في باقة، فهي منتشرة جدًا حتى الله كان من المعروف عنها أنها تنمو بين الحجارة الرصف، وتنتشر في كل مكان مثل الحشائش everywhere like weeds—and they have the very unsightly name of “dog-flowers” or “dandelions.”</td>
    <td>No one puts these flowers together in a bouquet, they are so common—they have even been known to grow between paving stones, and spring up everywhere like weeds—and they have the very unsightly name of “dog-flowers” or “dandelions.”</td>
  </tr>
  <tr>
    <td>14-nun</td>
    <td>حين جاءت هذه الزراعة في الصخور المشعهة تثير إلى خروج المركبة من نطاق التأثير المحدود للجاذبية الشمسية</td>
    <td>This increase in charged particles indicates the spacecraft’s departure from the influence of the solar wind which is called the heliosphere (which, according to some definitions, is the border of the solar system).</td>
  </tr>
  <tr>
    <td>16-ayn</td>
    <td>ويذهب المؤرخون إلى أن النبلاء الدينيين كان من الممكنين، نظرًا له في هذه الأسواق في يذهب إليها الشعراء ليعرضوا شعرهم، فنشأ بهذا ذاع صيته، وانتشر شعره الرقيق</td>
    <td>Historians assert that Al-Nabigha Al-Dhubyani was one of the arbiters. In these markets, a dome is erected for him where poets go to present their poetry. Whomever he praised, his fame spread, and his poetry circulated among the caravans.</td>
  </tr>
  <tr>
    <td>19-qaf</td>
    <td>كان كان حجور المالكيين عرباً خالياً سفين بالتوافق من دد</td>
    <td>As if the camel saddles of the Malikiya caravan leaving the Dadi valley were great ships</td>
  </tr>
</table>

Table 1: Representative subset of examples of the 19 BAREC readability levels, with English translations, and readability level reasoning. Underlining is used to highlight the main keys that determined the level.

4. Syntax: Specific sentence structure and syntactic relation constructions are identified as pivotal for certain levels.

5. Vocabulary: The complexity of word choices is key, with higher levels introducing more technical and classical literature vocabulary.

6. Content: The required prior knowledge and abstraction levels are considered for higher levels.

The BAREC pyramid (Figure 1) illustrates which aspects are used (broadly) for which levels. For example, spelling criteria are only used up to level 7-zay, while syntax is used until level 15-sin, and word count is not used beyond level 11-kaf.

Problems and Difficulties Annotators are encouraged to report any issues like spelling errors, colloquial language, or problematic topics. Difficulty is noted when annotations cannot be made due to conflicting guidelines.

A few representative examples for each level are provided in Table 1. A full set of examples with explanations of leveling choices is in Appendix A.3.

3.2 Annotation Team and Process

Annotation Team The BAREC annotation team comprised six native Arabic speakers, all of whom are experienced Arabic language educators. Among the team members, one individual (A0) brought prior experience in computational linguistic annotation projects, while the remaining five (A1-5) possessed extensive expertise in readability leveling, gained through their involvement in the Taha/Arabi21 project.

Annotation Process The annotation process began with A0, who led sentence-level segmentation and initial text flagging and selection. We followed the Arabic sentence segmentation guidelines by Habash et al. (2022a). Subsequently, A1-5 were tasked with assigning readability labels to the indi-
<table>
  <tr>
    <th>Domain</th>
    <th>Readership Group</th>
    <th>#Documents</th>
    <th>#Sentences</th>
    <th>#Words</th>
  </tr>
  <tr>
    <td>Arts & Humanities</td>
    <td>Foundational</td>
    <td>562 29%</td>
    <td>24,978 36%</td>
    <td>274,497 26%</td>
  </tr>
  <tr>
    <td>Arts & Humanities</td>
    <td>Advanced</td>
    <td>478 25%</td>
    <td>15,285 22%</td>
    <td>222,933 21%</td>
  </tr>
  <tr>
    <td>Arts & Humanities</td>
    <td>Specialized</td>
    <td>327 17%</td>
    <td>10,179 15%</td>
    <td>155,565 15%</td>
  </tr>
  <tr>
    <td>STEM</td>
    <td>Foundational</td>
    <td>27 1%</td>
    <td>533 1%</td>
    <td>12,879 1%</td>
  </tr>
  <tr>
    <td>STEM</td>
    <td>Advanced</td>
    <td>85 4%</td>
    <td>1,948 3%</td>
    <td>48,501 5%</td>
  </tr>
  <tr>
    <td>STEM</td>
    <td>Specialized</td>
    <td>68 4%</td>
    <td>2,199 3%</td>
    <td>49,265 5%</td>
  </tr>
  <tr>
    <td>Social Sciences</td>
    <td>Foundational</td>
    <td>44 2%</td>
    <td>2,270 3%</td>
    <td>26,692 3%</td>
  </tr>
  <tr>
    <td>Social Sciences</td>
    <td>Advanced</td>
    <td>168 9%</td>
    <td>5,463 8%</td>
    <td>110,226 11%</td>
  </tr>
  <tr>
    <td>Social Sciences</td>
    <td>Specialized</td>
    <td>163 8%</td>
    <td>6,586 9%</td>
    <td>138,813 13%</td>
  </tr>
  <tr>
    <td>Arts & Humanities</td>
    <td></td>
    <td>1,367 71%</td>
    <td>50,442 73%</td>
    <td>652,995 63%</td>
  </tr>
  <tr>
    <td>STEM</td>
    <td></td>
    <td>180 9%</td>
    <td>4,680 7%</td>
    <td>110,645 11%</td>
  </tr>
  <tr>
    <td>Social Sciences</td>
    <td></td>
    <td>375 20%</td>
    <td>14,319 21%</td>
    <td>275,731 27%</td>
  </tr>
  <tr>
    <td></td>
    <td>Foundational</td>
    <td>633 33%</td>
    <td>27,781 40%</td>
    <td>314,068 30%</td>
  </tr>
  <tr>
    <td></td>
    <td>Advanced</td>
    <td>731 38%</td>
    <td>22,696 33%</td>
    <td>381,660 37%</td>
  </tr>
  <tr>
    <td></td>
    <td>Specialized</td>
    <td>558 29%</td>
    <td>18,964 27%</td>
    <td>343,643 33%</td>
  </tr>
  <tr>
    <td colspan="2"></td>
    <td>1,922 100%</td>
    <td>69,441 100%</td>
    <td>1,039,371 100%</td>
  </tr>
</table>

Table 2: Summary statistics of the BAREC Corpus.

ividually segmented texts. The annotation was done through a simple Google Sheet interface. A1-5 received folders containing annotation sets, comprising 100 randomly selected sentences each. The average annotation speed was around 2.5 hours per batch (1.5 minutes/sentence).

Before starting the annotation, all annotators received rigorous training, including three pilot rounds. These rounds provided opportunities for detailed discussions of the guidelines, helping to identify and address any issues. 19 shared annotation sets (100 sentence each) were included covertly to ensure quality and measure inter-annotator agreement (IAA). Finally, we conducted a thorough second review of the corpus data, resulting in every sentence being checked twice for the first phase (10,658 sentences) before continuing to finish the 69,441 sentences (1M words).

In total, the annotators annotated 92.6K sentences, 25% of which is not in the final corpus: 3.3% were deemed problematic (typos and offensive topics); 11.5% were part of the second round of first phase annotation; and 10.3% were part of the IAA efforts, not including their unification. We report on IAA in Section 6.1.

4 BAREC Corpus

Corpus Selection In the process of corpus selection, we aimed to cover a wide educational span as well as different domains and topics. We collected the corpus from 1,922 documents, which we manually categorized into three domains: Arts & Humanities, Social Sciences, and STEM (details in Appendix C.2) and three readership groups: Foundational, Advanced, and Specialized (details in Appendix C.3). Table 2 shows the distribution of the documents, sentences, and words across domains and groups. The distribution across readership levels aligns with the corpus’s educational focus, with a higher-than-usual proportion at foundational levels. Variations across domains reflect differences in the availability of texts and reader interest (more Arts & Humanities, less STEM). The corpus uses documents from 30 different resources. All selected texts are either out of copyright, within the fair-use limit, or obtained in agreement with publishers. The decision of selecting some of these resources is influenced by the fact that other annotations exist for them. Around 25% of all sentences came from completely new sources that were manually typed to make them digitally usable. All details about the resources are available in Appendix C.
Figure 2: The distribution of the readership groups across BAREC levels.

<table>
  <tr>
    <th>#Documents</th>
    <th>#Sentences</th>
    <th>#Words</th>
  </tr>
  <tr>
    <td>Train</td>
    <td>1,518 79%</td>
    <td>54,845 79%</td>
    <td>832,743 80%</td>
  </tr>
  <tr>
    <td>Dev</td>
    <td>194 10%</td>
    <td>7,310 11%</td>
    <td>101,364 10%</td>
  </tr>
  <tr>
    <td>Test</td>
    <td>210 11%</td>
    <td>7,286 10%</td>
    <td>105,264 10%</td>
  </tr>
  <tr>
    <td>All</td>
    <td><b>1,922 100%</b></td>
    <td><b>69,441 100%</b></td>
    <td><b>1,039,371 100%</b></td>
  </tr>
</table>

Table 3: BAREC Corpus splits.

Readability Statistics Figure 2 shows the distribution of the three readership groups across all readability levels. As expected, foundational texts strictly dominate the lower levels up to 9-ta, then the presence of advanced and specialized texts starts increasing gradually till the highest level. Specialized texts dominate the highest levels, while the middle levels (10-ya to 14-nun) include a mix of the three groups with a slight advantage for advanced texts.

Corpus Splits We split the corpus into Train (\( \simeq 80\% \)), Dev (\( \simeq 10\% \)), and Test (\( \simeq 10\% \)) at the document level. Sentences from IAA studies are divided between all splits. However, We will release the IAA studies as a special set as they provide multiple references from different annotators for each example.\(^{2}\) Also, if other annotations exist for a resource (e.g., CamelTB (Habash et al., 2022b) and ReadMe++ (Naous et al., 2024)), we follow the existing splits. Table 3 shows the corpus splits in the level of documents, sentences, and words. More details about the splits across readability levels, domains, and readership groups are available in Appendix B.

5 Experiments

5.1 Metrics

In this paper, we define the task of Readability Assessment as an ordinal classification task. We use the following metrics for evaluation.

Accuracy (Acc\(^{19}\)) The percentage of cases where reference and prediction classes match in the 19-level scheme. We addition consider three variants, Acc\(^{7}\), Acc\(^{5}\), Acc\(^{3}\), that respectively collapse the 19-levels into the 7, 5, and 3-level schemes discussed in Section 3.

Adjacent Accuracy (\( \pm 1 \) Acc\(^{19}\)) Also known as off-by-1 accuracy. It allows some tolerance for predictions that are close to the true labels. It measures the proportion of predictions that are either exactly correct or off by at most one level.

Average Distance (Dist) Also known as Mean Absolute Error (MAE), it measures the average absolute difference between predicted and true labels.

Quadratic Weighted Kappa (QWK) An extension of Cohen’s Kappa (Cohen, 1968; Doewes et al., 2023) measuring the agreement between predicted and true labels, but applies a quadratic penalty to larger misclassifications, meaning that predictions farther from the true label are penalized more heavily.

We consider Quadratic Weighted Kappa as the primary metrics for selecting the best system.
<table>
  <tr>
    <th>Input Variant</th>
    <th>Example</th>
  </tr>
  <tr>
    <td>Original</td>
    <td>فألي شرق البحر يرفع مرمٌ الذهب والفضة.</td>
  </tr>
  <tr>
    <td>Word</td>
    <td>فألي شرق البحر يرفع مرمٌ الذهب والفضة.</td>
  </tr>
  <tr>
    <td>Lex</td>
    <td>الي شرق البحر يرفع مرمٌ ذهب وفضة.</td>
  </tr>
  <tr>
    <td>D3Tok</td>
    <td>فأ+ لي شرق+ البحر+ يرفع+ مرمٌ+ الذهب+ و+ ال+ فضة.</td>
  </tr>
  <tr>
    <td>D3Lex</td>
    <td>فأ+ لي شرق+ البحر+ يرفع+ مرمٌ+ الذهب+ و+ ال+ فضة.</td>
  </tr>
</table>

Table 4: Example sentence in different input variants.

5.2 Input Variants

In morphologically rich languages, affixation, compounding, and inflection convey key linguistic information that influences readability. Human annotators consider morphological complexity when assessing readability, but standard tokenization may obscure these cues. Segmenting sentences into morphological units helps preserve structural patterns relevant to readability prediction.

We generate four input variants using CamelTools morphological disambiguation to identify top choice analysis in context (Obeid et al., 2020).4 For the Word variant, we simply tokenize the sentences and remove diacritics and kashida using CAmeL Tools (Obeid et al., 2020). For Lex, we replace each word with its predicted Lemma. For D3Tok, we tokenize the word into its base and clitics form; and for D3Lex, we replace the base form in D3Tok with the lemma. All variants are dediacritized. Table 4 shows an example of a sentence and the corresponding input variants.

5.3 Fine-Tuning

We fine-tuned the top three Arabic BERT-based models according to Inoue et al. (2021) (AraBERTv02 (Antoun et al., 2020), MARBERTv2 (Abdul-Mageed et al., 2021), CamelBERT-msa (Inoue et al., 2021)). We also added AraBERTv2 to our experiments due to the possible matching between its pre-training data (morphologically segmented sentences by Farasa (Darwish and Mubarak, 2016)) and the different input variants.

5.4 Loss Functions

Since readability levels exhibit a natural ordering, we explore loss functions that account for the distance between predicted and true labels (Heilman et al., 2008). In addition to standard cross-entropy loss (CE), we experiment with Ordinal Log Loss (OLL) (Castagnos et al., 2022), Soft Labels Loss (SOFT) (Bertinetto et al., 2020), Earth Mover’s Distance-based loss (EMD) (L. Hou, 2017), and Regression using Mean Squared Error (Reg) as these have been previously used for ordinal classification tasks. OLL, SOFT, and EMD incorporate a distance matrix \( D \) into their formulations to penalize predictions proportionally to their distance from the true label. For simplicity, we define the distance between any two adjacent levels as one, setting \( D(i, j) = |i - j| \) for labels i and j. For regression, we round the final output to the nearest readability level to ensure predictions align with the 19 levels.

5.5 Hyper-parameters

For all experiments, we use a learning rate of \( 5 \times 10^{-5} \), a batch size of 64, and train for six epochs on an NVIDIA V100 GPU. After training, we select the best-performing epoch based on evaluation loss. For Ordinal Log Loss (OLL), we experiment with different values of the weighting parameter \( \alpha \), choosing from {0.5, 1, 1.5, 2}. Similarly, for Soft Labels Loss (SOFT), we evaluate different values of the smoothing parameter \( \beta \), selecting from {2, 3, 4, 5}. The training of the models in this paper took approximately 20 hours.

5.6 Procedure

Our experiments involve three main variables: the pretrained model, the input variant, and the loss function. Our goal is to determine the optimal combination of these three factors. Due to the large number of experiments required, we divide the process into two stages. In Stage 1, we train all combinations of pretrained models and input variants using cross-entropy loss. We then select the best combination based on a majority vote from our primary evaluation metrics (Acc, Acc ±1, Dist, and QWK). In Stage 2, we take the best combination of pretrained model and input variant from the first stage and train models using all the different loss functions.

6 Results

6.1 Inter-Annotator Agreement (IAA)

In this section, we report on 16 IAA studies, excluding the three pilots and first two IAAAs, which overlapped with annotator training.

Pairwise Agreement The average pairwise exact-match over 19 Barec levels between any two annotators is only 61.1%, which reflects the
task’s complexity. Allowing a fuzzy match distance of up to one level raises the match to 74.4%. The overall average pairwise level difference is 0.94 levels. The average pairwise Quadratic Weighted Kappa 81.8% (substantial agreement) confirms most disagreements are minor (Cohen, 1968; Doewes et al., 2023).

Unification Agreement After each IAA study, the annotators discussed and agreed on a unified readability level for each sentence. On average, the exact match between the annotators and the unified level (Acc^{19}) was 71.7%, reflecting the difficulty of the task. However, the high average ±1 Acc^{19} (82.3%), low Distance (0.65), and strong Quadratic Weighted Kappa (88.1%) suggest that most disagreements between annotators and the unified labels were minor. For more detailed results on IAA, see (Habash et al., 2025).

6.2 Stage 1 Results

Table 5 presents the results of stage 1, where we evaluate different combinations of pretrained models and input variants using cross-entropy loss. Based on the all metrics, we observe that the AraBERTv02 and AraBERTv2 models generally achieve higher performance across multiple input variants.

Among input variants, the Word and D3Tok representations tend to yield better results compared to Lex and D3Lex. Specifically, AraBERTv2 with the D3Tok input achieves the best scores in all metrics. Notably, AraBERTv2 is the only model that benefits from the D3Tok and D3Lex inputs compared to the Word input, showing an improvement across all metrics. We argue that this occurs because AraBERTv2 is the only model in this set that was pretrained on segmented data, making it more compatible with morphologically segmented input. These results suggest that both the choice of input variant and the pretrained model significantly impact performance.

Based on all metrics, we select AraBERTv2 with the D3Tok input as the best-performing combination. In stage 2, we evaluate it with different loss functions. The confusion matrix for this model is available in the Appendix D.1.

6.3 Stage 2 Results

Table 6 presents the results of stage 2, where we use the best model from stage 1 to evaluate different loss functions. Among all the loss functions evaluated, Cross-Entropy (CE) achieves the highest exact accuracy (Acc^{19}) at 56.6%, indicating that it performs best when predicting the exact readability level. In contrast, other loss functions show stronger performance on metrics that consider the ordinal nature of readability levels. Notably, Regression achieves the highest ±1 accuracy at 73.1% and the best Quadratic Weighted Kappa (QWK) at 84.0%, suggesting it excels at predicting levels close to the gold label, despite being the worst in terms of exact accuracy. These findings support that loss functions designed for ordinal or continuous labels—such as EMD, OLL, and Regression—are more effective on evaluation metrics that reward proximity to the correct label, even if they underperform on strict accuracy. More results for other loss functions are in Appendix D.2.

6.4 Ensemble Results

Table 7 presents results from Stage 1, where AraBERTv2 is evaluated with four different input variants, and Stage 2, where it is trained using the two best-performing loss functions. It also includes results from two ensemble strategies applied across all six models to assess whether combining predictions can further improve performance. We also include an oracle combination, which represents an upper bound on performance. This allows us to estimate the maximum potential gain achievable through ensembling.

Ensemble To further improve performance, we experiment with ensemble methods. We define the Average ensemble, where the final prediction is the rounded average of the levels predicted by the six models, and the Most Common ensemble, where the final prediction is the predicted levels’ mode.

The results show that the Average ensemble performs better in terms of Distance, indicating that it tends to stay closer to the correct label. However, it struggles with exact accuracy (Acc), as averaging can blur distinctions between classes. On the other hand, the Most Common ensemble achieves higher Acc but can sometimes be misled by an incorrect majority, leading to greater deviation from the correct label.

Oracle We also report an Oracle Combination, where we assume access to the best possible prediction from the six models for each sample. This serves as an upper bound on model performance. The Oracle results are significantly higher than those of individual models and are comparable to
<table>
  <tr>
    <th>Input</th>
    <th>Model</th>
    <th>Acc<sup>19</sup></th>
    <th>±1 Acc<sup>19</sup></th>
    <th>Dist</th>
    <th>QWK</th>
  </tr>
  <tr>
    <td rowspan="4">Word</td>
    <td>CamelBERT-msa</td>
    <td>54.4%</td>
    <td>68.7%</td>
    <td>1.20</td>
    <td>79.1%</td>
  </tr>
  <tr>
    <td>MARBERTv2</td>
    <td>53.3%</td>
    <td>68.0%</td>
    <td>1.20</td>
    <td>79.1%</td>
  </tr>
  <tr>
    <td>AraBERTv02</td>
    <td>55.8%</td>
    <td>69.2%</td>
    <td>1.17</td>
    <td>79.2%</td>
  </tr>
  <tr>
    <td>AraBERTv2</td>
    <td>51.6%</td>
    <td>65.9%</td>
    <td>1.32</td>
    <td>76.3%</td>
  </tr>
  <tr>
    <td rowspan="4">Lex</td>
    <td>CamelBERT-msa</td>
    <td>48.3%</td>
    <td>64.4%</td>
    <td>1.34</td>
    <td>77.1%</td>
  </tr>
  <tr>
    <td>MARBERTv2</td>
    <td>50.1%</td>
    <td>64.9%</td>
    <td>1.31</td>
    <td>77.0%</td>
  </tr>
  <tr>
    <td>AraBERTv02</td>
    <td>48.8%</td>
    <td>65.4%</td>
    <td>1.30</td>
    <td>78.5%</td>
  </tr>
  <tr>
    <td>AraBERTv2</td>
    <td>50.1%</td>
    <td>65.4%</td>
    <td>1.29</td>
    <td>77.7%</td>
  </tr>
  <tr>
    <td rowspan="4">D3Tok</td>
    <td>CamelBERT-msa</td>
    <td>54.8%</td>
    <td>68.2%</td>
    <td>1.21</td>
    <td>78.2%</td>
  </tr>
  <tr>
    <td>MARBERTv2</td>
    <td>54.0%</td>
    <td>68.5%</td>
    <td>1.20</td>
    <td>78.9%</td>
  </tr>
  <tr>
    <td>AraBERTv02</td>
    <td>54.8%</td>
    <td>68.1%</td>
    <td>1.22</td>
    <td>78.2%</td>
  </tr>
  <tr>
    <td>AraBERTv2</td>
    <td><b>56.6%</b></td>
    <td><b>69.9%</b></td>
    <td><b>1.14</b></td>
    <td><b>80.0%</b></td>
  </tr>
  <tr>
    <td rowspan="4">D3Lex</td>
    <td>CamelBERT-msa</td>
    <td>51.1%</td>
    <td>65.5%</td>
    <td>1.29</td>
    <td>78.0%</td>
  </tr>
  <tr>
    <td>MARBERTv2</td>
    <td>51.6%</td>
    <td>65.7%</td>
    <td>1.28</td>
    <td>78.0%</td>
  </tr>
  <tr>
    <td>AraBERTv02</td>
    <td>53.3%</td>
    <td>68.1%</td>
    <td>1.24</td>
    <td>78.2%</td>
  </tr>
  <tr>
    <td>AraBERTv2</td>
    <td>53.2%</td>
    <td>67.1%</td>
    <td>1.24</td>
    <td>78.6%</td>
  </tr>
</table>

Table 5: Results comparing different combinations of models and input variants on BAREC Dev set. Bold are the best results on each metric.

<table>
  <tr>
    <th>Loss</th>
    <th>Acc<sup>19</sup></th>
    <th>±1 Acc<sup>19</sup></th>
    <th>Dist</th>
    <th>QWK</th>
  </tr>
  <tr>
    <td>CE</td>
    <td><b>56.6%</b></td>
    <td>69.9%</td>
    <td>1.14</td>
    <td>80.0%</td>
  </tr>
  <tr>
    <td>EMD</td>
    <td>55.3%</td>
    <td>70.3%</td>
    <td><b>1.11</b></td>
    <td>81.2%</td>
  </tr>
  <tr>
    <td>OLL2</td>
    <td>35.2%</td>
    <td>70.3%</td>
    <td>1.25</td>
    <td>82.0%</td>
  </tr>
  <tr>
    <td>OLL15</td>
    <td>47.3%</td>
    <td>71.1%</td>
    <td>1.13</td>
    <td>82.8%</td>
  </tr>
  <tr>
    <td>OLL1</td>
    <td>50.8%</td>
    <td>71.5%</td>
    <td>1.12</td>
    <td>81.7%</td>
  </tr>
  <tr>
    <td>OLL05</td>
    <td>53.1%</td>
    <td>68.8%</td>
    <td>1.18</td>
    <td>79.7%</td>
  </tr>
  <tr>
    <td>SOFT2</td>
    <td>55.8%</td>
    <td>69.8%</td>
    <td>1.15</td>
    <td>80.0%</td>
  </tr>
  <tr>
    <td>SOFT3</td>
    <td>56.4%</td>
    <td>69.9%</td>
    <td>1.14</td>
    <td>80.1%</td>
  </tr>
  <tr>
    <td>SOFT4</td>
    <td>56.4%</td>
    <td>69.9%</td>
    <td>1.15</td>
    <td>79.6%</td>
  </tr>
  <tr>
    <td>SOFT5</td>
    <td>56.2%</td>
    <td>69.5%</td>
    <td>1.17</td>
    <td>79.3%</td>
  </tr>
  <tr>
    <td>Reg</td>
    <td>43.1%</td>
    <td><b>73.1%</b></td>
    <td>1.13</td>
    <td><b>84.0%</b></td>
  </tr>
</table>

Table 6: Loss functions comparisons on BAREC Dev set. We use AraBERTv2 model and D3Tok input with all loss function. Bold are the best results on each metric.

human annotators’ agreement with the unified labels (see section 6.1). This suggests that while individual models are still far from human-level performance, ensembling has the potential to push results closer to human agreement. More oracle combinations are provided in Appendix D.4. We also include more results on the impact of training granularity on readability level prediction in Appendix D.3

Finally, table 8 shows the results on the test set. We note that the trends observed in the development set persist in the test set, further validating our findings.

6.5 Error Analysis

To assess the errors in our best-performing model, we analyzed error patterns in the inter-annotator portion of the development (DEV) set. Each sentence in this subset had five human annotations, which we compared to the model’s prediction.

We grouped sentences by the level of annotator agreement, from full agreement (5 out of 5 annotators) down to minimal agreement (1 out of 5). Full 5-way agreement accounts for 25% of the data. With each reduction in agreement – to 4, 3, 2, and finally 1 annotator – the cumulative coverage increases to 50%, 61%, 72%, and 87%, respectively. In other words, in 87% of the cases, the model prediction can be meaningfully compared to at least some level of human consensus.

The remaining 13% fall outside this range. In 1% of these, the model’s prediction was within the span of human annotations but did not exactly match any of them. In 3%, the prediction was above the maximum annotation, and in 9%, it was below the minimum. We manually reviewed these out-of-range cases and found that the annotators were generally correct. We speculate that the model’s errors arise from limited training data, lack of con-
<table>
  <tr>
    <th>Input</th>
    <th>Loss</th>
    <th>Acc<sup>19</sup></th>
    <th>±1 Acc<sup>19</sup></th>
    <th>Dist</th>
    <th>QWK</th>
    <th>Acc<sup>7</sup></th>
    <th>Acc<sup>5</sup></th>
    <th>Acc<sup>3</sup></th>
  </tr>
  <tr>
    <td>Word</td>
    <td>CE</td>
    <td>51.6%</td>
    <td>65.9%</td>
    <td>1.32</td>
    <td>76.3%</td>
    <td>61.6%</td>
    <td>67.2%</td>
    <td>74.0%</td>
  </tr>
  <tr>
    <td>Lex</td>
    <td>CE</td>
    <td>50.1%</td>
    <td>65.4%</td>
    <td>1.29</td>
    <td>77.7%</td>
    <td>60.6%</td>
    <td>66.3%</td>
    <td>74.9%</td>
  </tr>
  <tr>
    <td>D3Tok</td>
    <td>CE</td>
    <td><b>56.6%</b></td>
    <td>69.9%</td>
    <td>1.14</td>
    <td>80.0%</td>
    <td>65.9%</td>
    <td>70.3%</td>
    <td>76.5%</td>
  </tr>
  <tr>
    <td>D3Lex</td>
    <td>CE</td>
    <td>53.2%</td>
    <td>67.1%</td>
    <td>1.24</td>
    <td>78.6%</td>
    <td>63.6%</td>
    <td>69.0%</td>
    <td>75.3%</td>
  </tr>
  <tr>
    <td>D3Tok</td>
    <td>EMD</td>
    <td>55.3%</td>
    <td>70.3%</td>
    <td><b>1.11</b></td>
    <td>81.2%</td>
    <td>65.2%</td>
    <td>70.0%</td>
    <td>76.4%</td>
  </tr>
  <tr>
    <td>D3Tok</td>
    <td>Reg</td>
    <td>43.1%</td>
    <td><b>73.1%</b></td>
    <td>1.13</td>
    <td><b>84.0%</b></td>
    <td>61.1%</td>
    <td>67.8%</td>
    <td>75.9%</td>
  </tr>
  <tr>
    <td colspan="2">Average</td>
    <td>46.9%</td>
    <td>72.5%</td>
    <td><b>1.11</b></td>
    <td>83.4%</td>
    <td>64.0%</td>
    <td>70.3%</td>
    <td><b>77.2%</b></td>
  </tr>
  <tr>
    <td colspan="2">Most Common</td>
    <td>56.3%</td>
    <td>70.0%</td>
    <td>1.13</td>
    <td>80.4%</td>
    <td><b>66.3%</b></td>
    <td><b>70.9%</b></td>
    <td>76.9%</td>
  </tr>
  <tr>
    <td colspan="2">Oracle Combo</td>
    <td>75.2%</td>
    <td>87.4%</td>
    <td>0.50</td>
    <td>93.8%</td>
    <td>83.2%</td>
    <td>85.7%</td>
    <td>89.1%</td>
  </tr>
</table>

Table 7: Results comparing different loss function, ensemble methods, and oracle performance on BAREC Dev set. Bold are the best results across individual models and across ensembles.

<table>
  <tr>
    <th>Input</th>
    <th>Loss</th>
    <th>Acc<sup>19</sup></th>
    <th>±1 Acc<sup>19</sup></th>
    <th>Dist</th>
    <th>QWK</th>
    <th>Acc<sup>7</sup></th>
    <th>Acc<sup>5</sup></th>
    <th>Acc<sup>3</sup></th>
  </tr>
  <tr>
    <td>Word</td>
    <td>CE</td>
    <td>51.1%</td>
    <td>65.1%</td>
    <td>1.31</td>
    <td>76.2%</td>
    <td>60.7%</td>
    <td>65.6%</td>
    <td>72.2%</td>
  </tr>
  <tr>
    <td>Lex</td>
    <td>CE</td>
    <td>51.2%</td>
    <td>66.2%</td>
    <td>1.23</td>
    <td>78.5%</td>
    <td>61.1%</td>
    <td>66.2%</td>
    <td>74.4%</td>
  </tr>
  <tr>
    <td>D3Tok</td>
    <td>CE</td>
    <td>55.9%</td>
    <td>70.0%</td>
    <td>1.12</td>
    <td>80.2%</td>
    <td>65.1%</td>
    <td>69.4%</td>
    <td>75.2%</td>
  </tr>
  <tr>
    <td>D3Lex</td>
    <td>CE</td>
    <td>53.7%</td>
    <td>67.9%</td>
    <td>1.17</td>
    <td>79.5%</td>
    <td>63.8%</td>
    <td>69.1%</td>
    <td>74.8%</td>
  </tr>
  <tr>
    <td>D3Tok</td>
    <td>EMD</td>
    <td>54.9%</td>
    <td>71.4%</td>
    <td><b>1.02</b></td>
    <td>83.7%</td>
    <td>64.9%</td>
    <td>69.0%</td>
    <td>75.2%</td>
  </tr>
  <tr>
    <td>D3Tok</td>
    <td>Reg</td>
    <td>41.4%</td>
    <td><b>73.5%</b></td>
    <td>1.11</td>
    <td>84.4%</td>
    <td>59.4%</td>
    <td>65.3%</td>
    <td>72.8%</td>
  </tr>
  <tr>
    <td colspan="2">Average</td>
    <td>46.0%</td>
    <td>73.4%</td>
    <td>1.06</td>
    <td><b>84.5%</b></td>
    <td>63.6%</td>
    <td>69.4%</td>
    <td><b>75.8%</b></td>
  </tr>
  <tr>
    <td colspan="2">Most Common</td>
    <td><b>56.2%</b></td>
    <td>70.4%</td>
    <td>1.07</td>
    <td>81.3%</td>
    <td><b>65.9%</b></td>
    <td><b>70.0%</b></td>
    <td>75.6%</td>
  </tr>
  <tr>
    <td colspan="2">Oracle Combo</td>
    <td>75.9%</td>
    <td>87.8%</td>
    <td>0.46</td>
    <td>94.7%</td>
    <td>83.5%</td>
    <td>85.7%</td>
    <td>88.9%</td>
  </tr>
</table>

Table 8: Results comparing different loss function, ensemble methods, and oracle performance on BAREC Test set. Bold are the best results across individual models and across ensembles.

textual understanding, or insufficient modeling of linguistic features. For example, the obscure word عصامة SAmh ‘tightly wound head dress’ may be misinterpreted as the feminine form of the proper name عصام SAm ‘Esam’, much like connecting كريم krym ‘Kareem’ with كريمة krymh ‘Kareema’. However, عصامة SAmh is not a plausible proper name. This remains speculative, as our model is not inherently interpretable.

7 Conclusions and Future Work

This paper presented the Balanced Arabic Readability Evaluation Corpus (BAREC), a large-scale, finely annotated dataset for assessing Arabic text readability across 19 levels. With over 69K sentences and 1 million words, it is the largest Arabic corpus for readability assessment, covering diverse genres, topics, and audiences, to our knowledge. High inter-annotator agreement ensures reliable annotations. Through benchmarking various readability assessment techniques, we highlighted both the challenges and opportunities in Arabic readability modeling, demonstrating promising performance across different methods.

Looking ahead, we plan to expand the corpus, enhancing its size and diversity to cover additional genres and topics. We also aim to add annotations related to vocabulary leveling and syntactic treebanks to study less-explored genres in syntax. Future work will include analyzing readability differences across genres and topics. Additionally, the tools we have developed will be integrated into a system to help children’s story writers target specific reading levels.

The BAREC dataset, its annotation guidelines, and benchmark results, will be made publicly available to support future research and educational applications in Arabic readability assessment.
Acknowledgments

The BAREC project is supported by the Abu Dhabi Arabic Language Centre (ALC) / Department of Culture and Tourism, UAE. We acknowledge the support of the High Performance Computing Center at New York University Abu Dhabi. We are deeply grateful to our outstanding annotation team: Mirvat Dawi, Reem Faraj, Rita Raad, Sawsan Tannir, and Adel Wizani, Samar Zeino, and Zeina Zeino. Special thanks go to Abdallah Abushmaes, Karin Aghadjanian, and Omar Al Ayyoubi of the ALC for their continued support. We would also like to thank the Zayed University ZAI Arabic Language Research Center team, in particular Hamda Al-Hadhrami, Maha Fatha, and Metha Talhak, for their valuable contributions to typing materials for the project. We also acknowledge Ali Gomaa and his team for their additional support in this area. Finally, we thank our colleagues at the New York University Abu Dhabi Computational Approaches to Modeling Language (CAMeL) Lab, Muhammed Abu Odeh, Bashar Alhafni, Ossama Obeid, and Mostafa Saeed, as well as Nour Rabih (Mohamed bin Zayed University of Artificial Intelligence) for their helpful conversations and feedback.

Limitations

One notable limitation is the inherent subjectivity associated with readability assessment, which may introduce variability in annotation decisions despite our best efforts to maintain consistency. Additionally, the current version of the corpus may not fully capture the diverse linguistic landscape of the Arab world. Finally, while our methodology strives for inclusivity, there may be biases or gaps in the corpus due to factors such as selection bias in the source materials or limitations in the annotation process. We acknowledge that readability measures can be used with malicious intent to profile people; this is not our intention, and we discourage it.

Ethics Statement

All data used in the corpus curation process are sourced responsibly and legally. The annotation process is conducted with transparency and fairness, with multiple annotators involved to mitigate biases and ensure reliability. All annotators are paid fair wages for their contribution. The corpus and associated guidelines are made openly accessible to promote transparency, reproducibility, and collaboration in Arabic language research.

References

Ahmed Abdelali, Kareem Darwish, Nadir Durrani, and Hamdy Mubarak. 2016. Farasa: A fast and furious segmenter for Arabic. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), pages 11–16, San Diego, California.

Muhammad Abdul-Mageed, AbdelRahim Elmadany, and El Moatez Billah Nagoudi. 2021. ARBERT & MARBERT: Deep bidirectional transformers for Arabic. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 7088–7105, Online. Association for Computational Linguistics.

Abbas Mahmoud Al-Akkad. 1938. Sarah. Hindawi.

Imam Muhammad al Bukhari. 846. Sahih al-Bukhari. Dar Ibn Kathir.

M Al-Dawsari. 2004. The assessment of readability books content (boys-girls) of the first grade of intermediate school according to readability standards. Sultan Qaboos University, Muscat.

Hend S Al-Khalifa and Amani A Al-Ajlan. 2010. Automatic readability measurements of the Arabic text: An exploratory study. Arabian Journal for Science and Engineering, 35(2 C):103–124.

Muhamed Al Khalil, Nizar Habash, and Zhengyang Jiang. 2020. A large-scale leveled readability lexicon for Standard Arabic. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 3053–3062, Marseille, France. European Language Resources Association.

Muhamed Al Khalil, Hind Saddiki, Nizar Habash, and Latifa Alfalasi. 2018. A Leveled Reading Corpus of Modern Standard Arabic. In Proceedings of the Language Resources and Evaluation Conference (LREC), Miyazaki, Japan.

Bayan Al-Safadi. 2005. Al-Kashkoul: selection of poetry and prose for children (الكشكول: مختارات من الشعر والنص للأطفال). Al-Sa’ih Library (مكتبة السايع).

A. Alfaifi. 2015. Building the Arabic Learner Corpus and a System for Arabic Error Annotation. Ph.D. thesis, University of Leeds.

Bashar Alhafni, Reem Hazim, Juan David Pineros Liberato, Muhamed Al Khalil, and Nizar Habash. 2024. The SAMEr Arabic text simplification corpus. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 16079–16093, Torino, Italia. ELRA and ICCL.

Richard L Allington, Kimberly McCuiston, and Monica Billen. 2015. What research says about text complexity and learning to read. The Reading Teacher, 68(7):491–501.
Shatha Altamamami, Eric Atwell, and Ammar Alsalka. 2019. The arabic–english parallel corpus of authentic hadith. International Journal on Islamic Applications in Computer Science And Technology-IJASAT.

Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. AraBERT: Transformer-based model for Arabic language understanding. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 9–15, Marseille, France. European Language Resource Association.

Amelia T. Barber and Susan L. Klauda. 2020. How reading motivation and engagement enable reading achievement: Policy implications. Policy Insights from the Behavioral and Brain Sciences, 7(1):27–34.

Luca Bertinetto, Romain Mueller, Konstantinos Terakis, Sina Samangooei, and Nicholas A. Lord. 2020. Making Better Mistakes: Leveraging Class Hierarchies With Deep Networks . In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12503–12512, Los Alamitos, CA, USA. IEEE Computer Society.

Houda Bouamor, Nizar Habash, Mohammad Salameh, Wajdi Zaghouani, Owen Rambow, Dana Abdurahim, Ossama Obeid, Salam Khalifa, Fadhl Eryani, Alexander Erdmann, and Kemal Ofazer. 2018. The MADAR Arabic dialect corpus and lexicon. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA).

François Castagnos, Martin Mihelich, and Charles Dognin. 2022. A simple log-based loss function for ordinal text classification. In Proceedings of the 29th International Conference on Computational Linguistics, pages 4604–4609, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.

Jacob Cohen. 1968. Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit. Psychological bulletin, 70(4):213.

Kevyn Collins-Thompson and James P. Callan. 2004. A language modeling approach to predicting reading difficulty. In Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics: HLT-NAACL 2004, pages 193–200, Boston, Massachusetts, USA. Association for Computational Linguistics.

Kareem Darwish and Hamdy Mubarak. 2016. Farasa: A new fast and accurate Arabic word segmenter. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 1070–1074, Portorož, Slovenia. European Language Resources Association (ELRA).

Tovly Deutsch, Masoud Jasbi, and Stuart Shieber. 2020. Linguistic features for readability assessment. In Proceedings of the Fifteenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 1–17, Seattle, WA, USA → Online. Association for Computational Linguistics.

Afrizal Doewes, Nughthoh Arfawi Kurdhi, and Akarti Saxena. 2023. Evaluating quadratic weighted kappa as the standard performance metric for automated essay scoring. In Proceedings of the 16th International Conference on Educational Data Mining, pages 103–113, Bengaluru, India. International Educational Data Mining Society.

William H DuBay. 2004. The principles of readability. Online Submission.

Kais Dukes, Eric Atwell, and Nizar Habash. 2013. Supervised collaboration for syntactic annotation of quranic arabic. Language resources and evaluation, 47(1):33–62.

Matthias Eck and Chiiori Hori. 2005. Overview of the IWSLT 2005 evaluation campaign. In Proceedings of the Second International Workshop on Spoken Language Translation, Pittsburgh, Pennsylvania, USA.

Mahmoud El-Haj and Paul Rayson. 2016. OSMAN — a novel Arabic readability metric. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 250–255, Portorož, Slovenia. European Language Resources Association (ELRA).

Mo El-Haj, Sultan Almjaiwel, Damith Premasiri, Tharindu Rasaninghe, and Ruslan Mitkov. 2024. DARES: Dataset for Arabic readability estimation of school materials. In Proceedings of the Workshop on DeTermIt! Evaluating Text Difficulty in a Multilingual Context @ LREC-COLING 2024, pages 103–113, Torino, Italia. ELRA and ICCL.

Mo El-Haj and Saad Ezzini. 2024. The multilingual corpus of world’s constitutions (MCWC). In Proceedings of the 6th Workshop on Open-Source Arabic Corpora and Processing Tools (OSACT) with Shared Tasks on Arabic LLMs Hallucination and Dialect to MSA Machine Translation @ LREC-COLING 2024, pages 57–66, Torino, Italia. ELRA and ICCL.

Lijun Feng, Martin Jansche, Matt Huenerfauth, and Noémie Elhadad. 2010. A comparison of features for automatic readability assessment. In Coling 2010: Posters, pages 276–284, Beijing, China. Coling 2010 Organizing Committee.

Jonathan Forsyth. 2014. Automatic readability prediction for modern standard Arabic. In Proceedings of the Workshop on Open-Source Arabic Corpora and Processing Tools (OSACT).

Irene C Fountas and Gay Su Pinnell. 2006. Leveled books (k-8): Matching texts to readers for effective teaching. Heinemann Educational Books.

Nizar Habash, Muhammed AbuOdeh, Dima Tajji, Reem Faraj, Jamila El Gizuli, and Omar Kallas. 2022a. Camel treebank: An open multi-genre Arabic dependency treebank. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 2672–2681, Marseille, France. European Language Resources Association.

Nizar Habash, Muhammed AbuOdeh, Dima Tajji, Reem Faraj, Jamila El Gizuli, and Omar Kallas. 2022b. Camel treebank: An open multi-genre Arabic dependency treebank. In Proceedings of the Thirteenth
Language Resources and Evaluation Conference, pages 2672–2681, Marseille, France.

Nizar Habash and David Paffreymon. 2022. ZAEBUC: An annotated Arabic-English bilingual writer corpus. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 79–88, Marseille, France. European Language Resources Association.

Nizar Habash, Hanada Taha-Thomure, Khalid N. Elmadani, Zeina Zeino, and Abdallah Abushmaes. 2025. Guidelines for fine-grained sentence-level Arabic readability annotation. In Proceedings of the 19th Linguistic Annotation Workshop (LAW-XIX), Vienna, Austria. Association for Computational Linguistics.

Michael Heilman, Kevyn Collins-Thompson, and Maxine Eskenazi. 2008. An analysis of statistical models and features for reading difficulty prediction. In Proceedings of the Third Workshop on Innovative Use of NLP for Building Educational Applications, pages 71–79, Columbus, Ohio. Association for Computational Linguistics.

Go Inoue, Bashar Alhafni, Nurpeis Baimukan, Houda Bouamor, and Nizar Habash. 2021. The interplay of variant, size, and task type in Arabic pre-trained language models. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 92–104, Kyiv, Ukraine (Virtual). Association for Computational Linguistics.

Muhamed Al Khalili, Hind Saddiki, Nizar Habash, and Latifa Alfalasi. 2018. A Leveled Reading Corpus of Modern Standard Arabic. In Proceedings of the Language Resources and Evaluation Conference (LREC), Miyazaki, Japan.

G.R. Klare. 1963. The Measurement of Readability. Iowa State University Press.

Fajri Koto, Haanon Li, Sara Shatnawi, Jad Doughman, Abdelrahman Sadallah, Aisha Alraeesi, Khalid Al-mubarak, Zaid Alyafeai, Neha Sengupta, Shady Shehata, Nizar Habash, Preslav Nakov, and Timothy Baldwin. 2024. ArabicMMLU: Assessing massive multitask language understanding in Arabic. In Findings of the Association for Computational Linguistics: ACL 2024, pages 5622–5640, Bangkok, Thailand. Association for Computational Linguistics.

D. Samaras L. Hou, C.P. Yu. 2017. Squared earth mover’s distance loss for training deep neural networks on ordered-classes. In NIPS workshop on Learning on Distributions, Functions, Graphs and Groups.

Bruce W. Lee, Yoo Sung Jang, and Jason Lee. 2021. Pushing on text readability assessment: A transformer meets handcrafted linguistic features. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10669–10686, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.

Juan Liberato, Bashar Alhafni, Muhamed Khalil, and Nizar Habash. 2024. Strategies for Arabic readability modeling. In Proceedings Of The Second Arabic Natural Language Processing Conference, pages 55–66, Bangkok, Thailand. Association for Computational Linguistics.

Pierre Lison and Jörg Tiedemann. 2016. OpenSubtitles2016: Extracting Large Parallel Corpora from Movie and TV Subtitles. In Proceedings of the Language Resources and Evaluation Conference (LREC), Portorož, Slovenia.

Farah Nadeem and Mari Ostendorf. 2018. Estimating linguistic complexity for science texts. In Proceedings of the Thirteenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 45–55, New Orleans, Louisiana. Association for Computational Linguistics.

Tarek Naous, Michael J Ryan, Anton Lavrouk, Mohit Chandra, and Wei Xu. 2024. ReadMe++: Benchmarking multilingual language models for multi-domain readability assessment. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 12230–12266, Miami, Florida, USA. Association for Computational Linguistics.

Naoual Nassiri, Violetta Cavalli-Sforza, and Abdelhak Lakhouaja. 2023. Approaches, methods, and resources for assessing the readability of arabic texts. ACM Trans. Asian Low-Resour. Lang. Inf. Process., 22(4).

Ossama Obeid, Nasser Zalmout, Salam Khalifa, Dima Tajji, Mai Oudah, Bashar Alhafni, Go Inoue, Fadhl Eryani, Alexander Erdmann, and Nizar Habash. 2020. CAMeL tools: An open source python toolkit for Arabic natural language processing. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 7022–7032, Marseille, France. European Language Resources Association.

Emily Pitler and Ani Nenkova. 2008. Revisiting readability: A unified framework for predicting text quality. In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, pages 186–195, Honolulu, Hawaii. Association for Computational Linguistics.

Hind Saddiki, Nizar Habash, Violetta Cavalli-Sforza, and Muhamed Al Khalil. 2018. Feature optimization for predicting readability of arabic 11 and 12. In Proceedings of the 5th Workshop on Natural Language Processing Techniques for Educational Applications, pages 20–29.

Eli Smith and Cornelius Van Dyck. 1860. New Testament (Arabic Translation).

Eli Smith and Cornelius Van Dyck. 1865. Old Testament (Arabic Translation).

Rasha Soliman and Laila Familiar. 2024. Creating a CEFR Arabic vocabulary profile: A frequency-based multi-dialectal approach. Critical Multilingualism Studies, 11(1):266–286.

Hanada Taha-Thomure. 2007. Poems and News (أشعار وأخبار). Educational Book House (دار الكتاب التربوي للنشر والتوزيع).

Hanada Taha-Thomure. 2017. Arabic Language Text Leveling (معايير هادئه لتصنيف مستويات النصوص العربية). Educational Book House (دار الكتاب التربوي للنشر والتوزيع).
Toshiyuki Takezawa, Genichiro Kikui, Masahide Mizushima, and Eiichiro Sumita. 2007. Multilingual spoken language corpus development for communication research. In International Journal of Computational Linguistics & Chinese Language Processing, Volume 12, Number 3, September 2007: Special Issue on Invited Papers from ISCSLP 2006, pages 303–324.

Ibn Tufail. 1150. Hayy ibn Yaqdhan. Hindawi.

Unknown. 12th century. One Thousand and One Nights.

Sowmya Vajjala. 2022. Trends, limitations and open challenges in automatic readability assessment research. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 5366–5377, Marseille, France. European Language Resources Association.

Sowmya Vajjala and Ivana Lučić. 2018. On-eStopEnglish corpus: A new corpus for automatic readability assessment and text simplification. In Proceedings of the Thirteenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 297–304, New Orleans, Louisiana. Association for Computational Linguistics.

Sowmya Vajjala and Detmar Meurers. 2012. On improving the accuracy of readability classification using insights from second language acquisition. In Proceedings of the Seventh Workshop on Building Educational Applications Using NLP, pages 163–173, Montréal, Canada. Association for Computational Linguistics.

Menglin Xia, Ekaterina Kochmar, and Ted Briscoe. 2016. Text readability assessment for second language learners. In Proceedings of the 11th Workshop on Innovative Use of NLP for Building Educational Applications, pages 12–22, San Diego, CA. Association for Computational Linguistics.

Wei Xu, Chris Callison-Burch, and Courtney Napoles. 2015. Problems in current text simplification research: New data can help. Transactions of the Association for Computational Linguistics, 3:283–297.
A  BAREC Annotation Guidelines Cheat Sheet and Examples

A.1 Arabic Original
<table>
  <tr>
    <th>BAREC Level</th>
    <th>Grade</th>
    <th>ACTFL Word Count</th>
    <th>Spelling/Phonunciation</th>
    <th>Morphology</th>
    <th>Syntax</th>
    <th>Vocabulary</th>
    <th>Idea/Content</th>
  </tr>
  <tr>
    <td>1-alf</td>
    <td>Pre1-1</td>
    <td>Novice Low</td>
    <td>1</td>
    <td>+ One-syllable<br>and two-syllable words</td>
    <td>+ Singular imperfective verb</td>
    <td>+ One word</td>
    <td>+ Common noun<br>+ Proper noun (frequent and simple)<br>+ Personal pronouns (non-clitics)<br>+ Numerical identical to dialectal form - SAMER I<br>+ Numbers (Arabic or Indo-Arabic) 1-10</td>
    <td>+ Direct, explicit, and concrete ideas.<br>+ No symbolism in the text.</td>
  </tr>
  <tr>
    <td>2-ba</td>
    <td></td>
    <td>Novice Low</td>
    <td>≤2</td>
    <td>+ Three-syllable words</td>
    <td></td>
    <td>+ Verb<br>+ Adjective<br>+ Vocabulary similar to dialectal form - SAMER I<br>+ Specific cardinal numbers<br>+ The five nouns: Abe (father), Aw (brother)</td>
    <td></td>
  </tr>
  <tr>
    <td>3-jim</td>
    <td>1</td>
    <td>Novice Mid</td>
    <td>≤4</td>
    <td>+ Prisitcile: Definite article: Al-<br>+ Prisitcile: Conjunction wa+<br>+ Enclitic: First Person Singular pronoun</td>
    <td>+ Apposition (full)<br>+ Demonstratives</td>
    <td>+ Common MSA vocabulary - SAMER I<br>+ Singular demonstrative pronoun<br>+ Numbers: 11-100</td>
    <td></td>
  </tr>
  <tr>
    <td>4-dil</td>
    <td></td>
    <td>Novice Mid</td>
    <td>≤6</td>
    <td>+ Words with an elongated Alif (e.g. 'Bilad')</td>
    <td>+ Imperfective verb<br>+ Prepositional predicates<br>+ Nouned adverbials</td>
    <td>+ Verbal sentence w/o direct object<br>+ Preposition and object</td>
    <td>+ Prepositions</td>
    <td></td>
  </tr>
  <tr>
    <td>5-ha</td>
    <td></td>
    <td>Novice High</td>
    <td>≤8</td>
    <td>+ Four-syllable words<br>+ Enclitic: Singular and Plural pronouns<br>+ Enclitic (in nouns and adjectives)<br>+ Sound feminine plural</td>
    <td>+ Verbal sentence with one nominal direct object<br>+ Conjoined sentences<br>+ Basic interrogative particles: what, where, when, whether<br>+ Exclamatory form: how=comparison adjective</td>
    <td>+ Ordinal numbers<br>+ Numbers: 101-1,000<br>+ Dual and plural demonstrative pronoun</td>
    <td>+ Content is from the reader's life.<br>+ No symbolism in the text.</td>
  </tr>
  <tr>
    <td>6-waw</td>
    <td>2</td>
    <td>Novice High</td>
    <td>≤9</td>
    <td>+ Five-syllable words<br>+ Singular and plural perfective verb<br>+ Sound masculine plural</td>
    <td>+ Sequence with two verbs (e.g., a verbal sentence a clausal direct object introduced with Masdar 'an + sth/that')</td>
    <td>+ MSA vocabulary - SAMER I</td>
    <td></td>
  </tr>
  <tr>
    <td>7-zay</td>
    <td>Intermediate Low</td>
    <td>≤10</td>
    <td>+ Six-syllable or more words<br>+ Verb/sious nouns with weak final letters</td>
    <td>+ Dual imperative verb<br>+ Dual imperfective verb<br>+ Singular imperative verb<br>+ Enclitics: dual pronoun<br>+ Enclitic: dual article<br>+ Way of oath</td>
    <td>+ Adverbial adjective (time and place adverb)<br>+ Circumstantial adjective<br>+ Interrogative particle ha'</td>
    <td>+ High frequency MSA vocabulary - SAMER II</td>
    <td>+ Some symbolism, but not everything is stated directly in the sentence.</td>
  </tr>
  <tr>
    <td>8-ha</td>
    <td>Intermediate Low</td>
    <td>≤11</td>
    <td></td>
    <td>+ Plural imperative verb<br>+ Feminine plural suffix (nun) in nouns<br>+ Other pronomics: future sa+, conjunction fa-, conjunction fa', conjunction fa', then, until, or, whether, but, as, for example</td>
    <td>+ Absolute object (emphasizing the verb)<br>+ Object of purpose<br>+ Object of accompaniment<br>+ Verbal sentence with two direct objects</td>
    <td>+ MSA vocabulary - SAMER I and II<br>+ Negation particles<br>+ Numbers: 1,001-1,000,000</td>
    <td>+ Some symbolism that requires the reader's help to understand the idea.</td>
  </tr>
  <tr>
    <td>9-ta</td>
    <td>Intermediate Mid</td>
    <td>≤12</td>
    <td></td>
    <td>+ Dual imperative verb<br>+ Enclitic Hamza<br>+ Bo of oath<br>+ Oath: The particle of oath, the object of the oath, and the answer to the oath</td>
    <td>+ Vocative</td>
    <td>+ Vocabulary describing positive and negative emotional and mood states like joy, happiness, anger, regret, sorrow</td>
    <td>+ Some symbolism at the event level in the sentence that the reader understands through prior knowledge</td>
  </tr>
  <tr>
    <td>10-ya</td>
    <td>Intermediate Mid</td>
    <td>≤15</td>
    <td></td>
    <td>+ Passive voice</td>
    <td>+ Jussa and its sisters (particles introducing a subject)<br>+ Kausu and its sisters (past tense verb)<br>+ Predicative predicate, postponed subject<br>+ Chain of narration<br>+ rabba preposition construction<br>+ Relative pronouns<br>+ Circumstantial and object classes</td>
    <td>+ Singular relative pronouns<br>+ Verbal particles qad and laqad<br>+ Preposition-Conjunctions: mimma, fimma...</td>
    <td></td>
  </tr>
  <tr>
    <td>11-kaf</td>
    <td>Intermediate High</td>
    <td>≥20</td>
    <td></td>
    <td>+ Acting derivatives (e.g., the active participle)</td>
    <td>+ Nominal sentence with a nominal predicate<br>+ El-kafl data (full in statute)</td>
    <td>+ Dual and plural relative pronouns</td>
    <td>+ A degree of symbolism and a need for prior knowledge to understand the meaning of the sentence.</td>
  </tr>
  <tr>
    <td>12-lam</td>
    <td>5</td>
    <td>Advanced Low</td>
    <td></td>
    <td>+ Diminutive form</td>
    <td>+ Parentheticals (explanation, blessing)<br>+ Exception<br>+ Elision<br>+ Apposition (e.g., partitive or containing)<br>+ Specification (masnaze construction)</td>
    <td>+ MSA vocabulary - Samer III<br>+ Frozen Verbs (e.g., dama' Amen)<br>+ Numbers: 1,001-1,000,000<br>+ Five Nouns: Dhu (possession nominal)<br>+ Interjections: hula', fadl, etc.</td>
    <td></td>
  </tr>
  <tr>
    <td>13-mim</td>
    <td>6-7</td>
    <td>Advanced Mid</td>
    <td></td>
    <td>+ Energetic mood (emphatic nun)<br>+ Ta of oath</td>
    <td>+ Conditional sentences<br>+ Inactive particle lamma (not yet)</td>
    <td>+ Words describing deep psychological states and deep emotions, psychological ailments<br>+ Use of God, incantatory words (abhaqatun) (e.g., LLLC)</td>
    <td>+ Symbolic, abstract, and deeper meanings, especially in terms of the psychological illnesses/character events.<br>+ Local cultural expressions that may not be understood by those outside the culture.</td>
  </tr>
  <tr>
    <td>14-nun</td>
    <td>8-9</td>
    <td>Advanced High</td>
    <td></td>
    <td>+ Semantic emphasis<br>+ Praise and dispraise<br>+ Masdar 'an clause as a subject<br>+ Enclitic: dual comparative adjective: fa'an</td>
    <td>+ MSA vocabulary - SAMER IV<br>+ General legal, scientific, religious, political vocabulary, etc.<br>+ Five Nouns: fa', fimma...</td>
    <td></td>
    <td>+ Symbolic, abstract, and deeper symbolic ideas that require prior linguistic and cognitive knowledge to understand.</td>
  </tr>
  <tr>
    <td>15-sin</td>
    <td>10-11</td>
    <td>Superior Low</td>
    <td></td>
    <td></td>
    <td>+ Uncommon constructions that are ambiguous and need disambiguation for clarification</td>
    <td>+ Specialized vocabulary that requires the concept/idea to comprehend it<br>+ Shortening in proper names (e.g., /fa'an/ instead of fa'an)</td>
    <td></td>
  </tr>
  <tr>
    <td>16-ayn</td>
    <td>12</td>
    <td>Superior Mid</td>
    <td></td>
    <td></td>
    <td></td>
    <td>+ MSA vocabulary - SAMER V<br>+ Specialized and highly elevated Arabic vocabulary<br>+ Vocabulary mostly distant from dialects</td>
    <td></td>
  </tr>
  <tr>
    <td>17-fa</td>
    <td>University Year 1-2</td>
    <td>Superior High</td>
    <td></td>
    <td></td>
    <td></td>
    <td>+ Scientific and heritage vocabulary not in use today, but familiar to a novice specialist</td>
    <td>+ Scientific and heritage vocabulary not in use today, but familiar to the advanced researcher specialist</td>
  </tr>
  <tr>
    <td>18-sad</td>
    <td>University Year 3-4</td>
    <td>Distinguished</td>
    <td></td>
    <td></td>
    <td></td>
    <td>+ Scientific and heritage vocabulary not in use today, but familiar to the advanced researcher specialist</td>
    <td></td>
  </tr>
  <tr>
    <td>19-qaf</td>
    <td>Specialist</td>
    <td>Distinguished*</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
</table>

Difficulty Problem
Generally, we use this tag for instances containing:
• Spelling mistakes (e.g., Hamza, Ta Marbutha, Ash maqra'a/Ya')
• Errors in diacritics
• Linguistic awkwardness (illiteracy, colloquialism, poor translation)
• Role reversal (e.g., the subject is the object)
• Inappropriate topics (racism, bias, bullying, pornography, etc.)
• Sentences and phrases mostly written in languages other than Arabic or in non-Arabic script

*However, in the following cases, we provide the level and add a note in the comments column:
• Error in Hamza al-Wasl/Hamza al-Qaf
  ≥1
• Offensive words
  ≥2 (C)
• Dotted Yaa missing at the beginning of the sentence
  ≥4 (Q)
• Dotted Yaa missing at the end of the word
  ≥4 (Q)
A.3 Annotation Examples
Representative examples of the 19 BAREC readability levels, with English translations, and readability level reasoning. Underlining is used to highlight the main keys that determined the level.

<table>
  <tr>
    <th>RL</th>
    <th>Arabic Sentence/Phrase</th>
    <th>Translation</th>
    <th>Reasoning</th>
  </tr>
  <tr>
    <td>1-alif</td>
    <td>أرنب Rabbit</td>
    <td></td>
    <td><i>One bisyllabic familiar noun</i></td>
  </tr>
  <tr>
    <td>2-ba</td>
    <td>ملعب أطفال A large playground</td>
    <td></td>
    <td>Noun-adjective</td>
  </tr>
  <tr>
    <td>3-jim</td>
    <td>أنا أحب اللون الأحمر I love the color red.</td>
    <td></td>
    <td>Definite article</td>
  </tr>
  <tr>
    <td>4-dal</td>
    <td>الشمس تشرق في الصباح الباكر The sun rises early in the morning</td>
    <td></td>
    <td>Prepositional phrase</td>
  </tr>
  <tr>
    <td>5-ha</td>
    <td>القطة تستريح على السرير وتستمتع بالشمس The cat rests on the bed and enjoys the warm sunshine</td>
    <td></td>
    <td>A conjoined sentence</td>
  </tr>
  <tr>
    <td>6-waw</td>
    <td>سأكون مسئوليتي My behavior is my responsibility</td>
    <td></td>
    <td>Five syllable word</td>
  </tr>
  <tr>
    <td>7-zay</td>
    <td>الأصدقاء يحتفلون بعيد ميلاد صديقهم بكمية رائعة Friends celebrate their friend's birthday with cake and amazing gifts.</td>
    <td></td>
    <td>Broken plural</td>
  </tr>
  <tr>
    <td>8-ha</td>
    <td>أسمع إلى كل فقرة من الفقرتين الآتتين، ثم أجب: I listen to each of the following two paragraphs, then I answer:</td>
    <td></td>
    <td>٢ (then) is in level 8-ha ٢</td>
  </tr>
  <tr>
    <td>9-ta</td>
    <td>فقل لي كم صحيح معك يا سمك هل fish, do you abide by the old promise He said in annoying, eloquent words: Oh fish, oh fish, do you abide by the old promise</td>
    <td></td>
    <td>Vocative construction</td>
  </tr>
  <tr>
    <td>10-ya</td>
    <td>أنت أعلم العهد القديم ميم asked you whether you were accusing him of lying before he said what he said, and you said no.</td>
    <td></td>
    <td>Auxiliary Kaana</td>
  </tr>
  <tr>
    <td>11-kaf</td>
    <td>حسام سعيد قلبه بسبب فوز فريقه Hossam, his heart is happy because of his team's victory.</td>
    <td></td>
    <td>Acting derivative (happy is predicative)</td>
  </tr>
  <tr>
    <td>12-lam</td>
    <td>لا أحد يجمع هذه الزهر معًا في باقة، فهي منتشرة في حجرات كأنها من المعروف عنها أنها تتواجد بين الحجارة الرصف، وتنبت في كل مكان كمثل المشاكل الصغيرة، وتتحمل اسمًا أجنبيًا جدًا وهو «زهور الكلاب» أو «نضيجات البرية» No one puts these flowers together in a bouquet, they are so common—they have even been known to grow between paving stones, and spring up everywhere like weeds—and they have the very unsightly name of “dog-flowers” or “dandelions.”</td>
    <td></td>
    <td>Parenthetical phrase</td>
  </tr>
  <tr>
    <td>13-mim</td>
    <td>ومن يفعل المعرفة مع غير أهلها يجاز كما جاز من يجوز أم عارض And whoever offers good deeds to someone undeserving will be rewarded like he who gave shelter to a hyena</td>
    <td></td>
    <td>Conditional phrase</td>
  </tr>
  <tr>
    <td>14-nun</td>
    <td>حيث إن هذه الزيادة في الجسيمات المشحونة This increase in charged particles indicates the spacecraft's departure from the influence of the solar wind, which is called the heliosphere (which, according to some definitions, is the border of the solar system).</td>
    <td></td>
    <td>General geography vocabulary</td>
  </tr>
  <tr>
    <td>15-sin</td>
    <td>وكان من عادةها أن تقارن نفسها وبين بطلة الرواية إذا أحسست بضعف إعجابها بها أو نادرا عليها، وتسأل في ذلك لسان ذكيًا تسمى المقالطة في جوابها، إلا على سبيل المزاح والذكاء: It was her habit to compare herself with the heroine of the novel when she felt his admiration or praise for her, asking him smart and tricky questions that did not allow answering deceptively, except by joking and teasing.</td>
    <td></td>
    <td>Specialized vocabulary that requires understanding the concept to comprehend its use</td>
  </tr>
  <tr>
    <td>16-ayn</td>
    <td>ويذهب المؤرخون إلى أن النابغة الذويباني كان من المتكبرين، فقام به في هذه الأسواق فيه one of the arbiters. In these markets, a dome is erected for him where poets go to present their poetry. Whomever he praised, his fame spread, and his poetry circulated among the caravans.</td>
    <td></td>
    <td>Historians assert that Al-Nabigha Al-Dhubyani was one of the arbiters. In these markets, a dome is erected for him where poets go to present their poetry. Whomever he praised, his fame spread, and his poetry circulated among the caravans.</td>
    <td>Specialized and uncommon vocabulary</td>
  </tr>
  <tr>
    <td>17-fa</td>
    <td>بين ضم القل وخف النبض Between the thrusts of lances and the fluttering of ensigns</td>
    <td></td>
    <td>Heritage vocabulary familiar to a novice specialist</td>
  </tr>
  <tr>
    <td>18-sad</td>
    <td>أنا لا أرى إلا ما أريته وأتوقى ما أتخوف I wasn't able to see except with extreme effort and difficulty like a water basin in solid undrillable land</td>
    <td></td>
    <td>Specialist vocabulary, symbolic poetic ideas requiring prior knowledge</td>
  </tr>
  <tr>
    <td>19-qaf</td>
    <td>كان جنود الملكية غوا خلا سفين As if the camel saddles of the Malikyya caravan leaving the Dadi valley were great ships</td>
    <td></td>
    <td>Advanced specialist vocabulary, symbolic poetic ideas requiring prior knowledge</td>
  </tr>
</table>
B  BAREC Corpus Splits

B.1  Sentence-level splits across readability levels

<table>
  <tr>
    <th>Level</th>
    <th>All</th>
    <th>Train</th>
    <th>Dev</th>
    <th>Test</th>
  </tr>
  <tr>
    <td>1-alif</td>
    <td>409 1%</td>
    <td>333 1%</td>
    <td>44 1%</td>
    <td>32 0%</td>
  </tr>
  <tr>
    <td>2-ba</td>
    <td>437 1%</td>
    <td>333 1%</td>
    <td>68 1%</td>
    <td>36 0%</td>
  </tr>
  <tr>
    <td>3-jim</td>
    <td>1,462 2%</td>
    <td>1,139 2%</td>
    <td>182 2%</td>
    <td>141 2%</td>
  </tr>
  <tr>
    <td>4-dal</td>
    <td>751 1%</td>
    <td>587 1%</td>
    <td>78 1%</td>
    <td>86 1%</td>
  </tr>
  <tr>
    <td>5-ha</td>
    <td>3,443 5%</td>
    <td>2,646 5%</td>
    <td>417 6%</td>
    <td>380 5%</td>
  </tr>
  <tr>
    <td>6-waw</td>
    <td>1,534 2%</td>
    <td>1,206 2%</td>
    <td>189 3%</td>
    <td>139 2%</td>
  </tr>
  <tr>
    <td>7-zay</td>
    <td>5,438 8%</td>
    <td>4,152 8%</td>
    <td>701 10%</td>
    <td>585 8%</td>
  </tr>
  <tr>
    <td>8-Ha</td>
    <td>5,683 8%</td>
    <td>4,529 8%</td>
    <td>613 8%</td>
    <td>541 7%</td>
  </tr>
  <tr>
    <td>9-ta</td>
    <td>2,023 3%</td>
    <td>1,597 3%</td>
    <td>236 3%</td>
    <td>190 3%</td>
  </tr>
  <tr>
    <td>10-ya</td>
    <td>9,763 14%</td>
    <td>7,741 14%</td>
    <td>1,012 14%</td>
    <td>1,010 14%</td>
  </tr>
  <tr>
    <td>11-kaf</td>
    <td>4,914 7%</td>
    <td>4,041 7%</td>
    <td>409 6%</td>
    <td>464 6%</td>
  </tr>
  <tr>
    <td>12-lam</td>
    <td>14,471 21%</td>
    <td>11,318 21%</td>
    <td>1,491 20%</td>
    <td>1,662 23%</td>
  </tr>
  <tr>
    <td>13-mim</td>
    <td>4,039 6%</td>
    <td>3,252 6%</td>
    <td>349 5%</td>
    <td>438 6%</td>
  </tr>
  <tr>
    <td>14-nun</td>
    <td>10,687 15%</td>
    <td>8,573 16%</td>
    <td>1,072 15%</td>
    <td>1,042 14%</td>
  </tr>
  <tr>
    <td>15-sin</td>
    <td>2,547 4%</td>
    <td>2,016 4%</td>
    <td>258 4%</td>
    <td>273 4%</td>
  </tr>
  <tr>
    <td>16-ayn</td>
    <td>1,141 2%</td>
    <td>866 2%</td>
    <td>114 2%</td>
    <td>161 2%</td>
  </tr>
  <tr>
    <td>17-fa</td>
    <td>480 1%</td>
    <td>364 1%</td>
    <td>49 1%</td>
    <td>67 1%</td>
  </tr>
  <tr>
    <td>18-sad</td>
    <td>103 0%</td>
    <td>67 0%</td>
    <td>13 0%</td>
    <td>23 0%</td>
  </tr>
  <tr>
    <td>19-qaf</td>
    <td>116 0%</td>
    <td>85 0%</td>
    <td>15 0%</td>
    <td>16 0%</td>
  </tr>
  <tr>
    <th>Total</th>
    <th>69,441 100%</th>
    <th>54,845 100%</th>
    <th>7,310 100%</th>
    <th>7,286 100%</th>
  </tr>
</table>
B.2 Sentence-level splits across domains and readership groups

<table>
  <tr>
    <th>Domain</th>
    <th>Readership Group</th>
    <th>All</th>
    <th>Train</th>
    <th>Dev</th>
    <th>Test</th>
  </tr>
  <tr>
    <td rowspan="3">Arts & Humanities</td>
    <td>Foundational</td>
    <td>24,978 36%</td>
    <td>20,161 37%</td>
    <td>2,397 33%</td>
    <td>2,420 33%</td>
  </tr>
  <tr>
    <td>Advanced</td>
    <td>15,285 22%</td>
    <td>11,982 22%</td>
    <td>1,653 23%</td>
    <td>1,650 23%</td>
  </tr>
  <tr>
    <td>Specialized</td>
    <td>10,179 15%</td>
    <td>7,755 14%</td>
    <td>1,090 15%</td>
    <td>1,334 18%</td>
  </tr>
  <tr>
    <td rowspan="3">STEM</td>
    <td>Foundational</td>
    <td>533 1%</td>
    <td>453 1%</td>
    <td>80 1%</td>
    <td>0 0%</td>
  </tr>
  <tr>
    <td>Advanced</td>
    <td>1,948 3%</td>
    <td>1,741 3%</td>
    <td>137 2%</td>
    <td>70 1%</td>
  </tr>
  <tr>
    <td>Specialized</td>
    <td>2,199 3%</td>
    <td>1,600 3%</td>
    <td>258 4%</td>
    <td>341 5%</td>
  </tr>
  <tr>
    <td rowspan="3">Social Sciences</td>
    <td>Foundational</td>
    <td>2,270 3%</td>
    <td>1,355 2%</td>
    <td>600 8%</td>
    <td>315 4%</td>
  </tr>
  <tr>
    <td>Advanced</td>
    <td>5,463 8%</td>
    <td>4,394 8%</td>
    <td>514 7%</td>
    <td>555 8%</td>
  </tr>
  <tr>
    <td>Specialized</td>
    <td>6,586 9%</td>
    <td>5,404 10%</td>
    <td>581 8%</td>
    <td>601 8%</td>
  </tr>
  <tr>
    <td rowspan="3">Arts & Humanities</td>
    <td></td>
    <td>50,442 73%</td>
    <td>39,898 73%</td>
    <td>5,140 70%</td>
    <td>5,404 74%</td>
  </tr>
  <tr>
    <td>STEM</td>
    <td>4,680 7%</td>
    <td>3,794 7%</td>
    <td>475 6%</td>
    <td>411 6%</td>
  </tr>
  <tr>
    <td>Social Sciences</td>
    <td>14,319 21%</td>
    <td>11,153 20%</td>
    <td>1,695 23%</td>
    <td>1,471 20%</td>
  </tr>
  <tr>
    <td rowspan="3"></td>
    <td>Foundational</td>
    <td>27,781 40%</td>
    <td>21,969 40%</td>
    <td>3,077 42%</td>
    <td>2,735 38%</td>
  </tr>
  <tr>
    <td>Advanced</td>
    <td>22,696 33%</td>
    <td>18,117 33%</td>
    <td>2,304 32%</td>
    <td>2,275 31%</td>
  </tr>
  <tr>
    <td>Specialized</td>
    <td>18,964 27%</td>
    <td>14,759 27%</td>
    <td>1,929 26%</td>
    <td>2,276 31%</td>
  </tr>
  <tr>
    <td colspan="2"></td>
    <td>69,441 100%</td>
    <td>54,845 100%</td>
    <td>7,310 100%</td>
    <td>7,286 100%</td>
  </tr>
</table>
C  BAREC Corpus Details

C.1 Resources
We present the corpus sources in groups of their general intended purpose.

C.1.1 Education
Emarati Curriculum  The first five units of the UAE curriculum textbooks for the 12 grades in three subjects: Arabic language, social studies, Islamic studies (Khalil et al., 2018).

ArabicMMLU  6,205 question and answer pairs from the ArabicMMLU benchmark dataset (Koto et al., 2024).

Zayed Arabic-English Bilingual Undergraduate Corpus (ZAEBUC)  100 student-written articles from the Zayed University Arabic-English Bilingual Undergraduate Corpus (Habash and Palfreyman, 2022).

Arabic Learner Corpus (ALC)  16 L2 articles from the Arabic Learner Corpus (Alfaifi, 2015).

Basic Travel Expressions Corpus (BTEC)  20 documents from the MSA translation of the Basic Traveling Expression Corpus (Eck and Hori, 2005; Takezawa et al., 2007; Bouamor et al., 2018).

Collection of Children poems  Example of the included poems: My language sings (اللغة تغني), and Poetry and news (أشعار وأخبار) (Al-Safadi, 2005; Taha-Thomure, 2007).

ChatGPT  To add more children’s materials, we ask Chatgpt to generate 200 sentences ranging from 2 to 4 words per sentence, 150 sentences ranging from 5 to 7 words per sentence and 100 sentences ranging from 8 to 10 words per sentence.5 Not all sentences generated by ChatGPT were correct. We discarded some sentences that were flagged by the annotators. Table 9 shows the prompts and the percentage of discarded sentences for each prompt.

C.1.2 Literature
Hindawi  A subset of 264 books extracted from the Hindawi Foundation website across different different genres.6

Kalima  The first 500 words of 62 books from Kalima project.7

Green Library  58 manually typed books from the Green Library.8

Arabian Nights  The openings and endings of the opening narrative and the first eight nights from the Arabian Nights (Unknown, 12th century). We extracted the text from an online forum.9

Hayy ibn Yaqdhan  A subset of the philosophical novel and allegorical tale written by Ibn Tufail (Tufail, 1150). We extracted the text from the Hindawi Foundation website.10

Sara  The first 1000 words of Sara, a novel by Al-Akkad first published in 1938 (Al-Akkad, 1938). We extracted the text from the Hindawi Foundation website.11

The Suspended Odes (Odes)  The ten most celebrated poems from Pre-Islamic Arabia (المقطات Mu’allaqat). All texts were extracted from Wikipedia.12

C.1.3 Media
Majed  10 manually typed editions of Majed magazine for children from 1983 to 2019.13

ReadMe++  The Arabic split of the ReadMe++ dataset (Naous et al., 2024).

Spacetoon Songs  The opening songs of 53 animated children series from Spacetoon channel.

Subtitles  A subset of the Arabic side of the OpenSubtitles dataset (Lison and Tiedemann, 2016).

WikiNews  62 Arabic WikiNews articles covering politics, economics, health, science and technology, sports, arts, and culture (Abdelali et al., 2016).

C.1.4 References
Wikipedia  A subset of 168 Arabic wikipedia articles covering Culture, Figures, Geography, History, Mathematics, Sciences, Society, Philosophy, Religions and Technologies.14

5https://chatgpt.com/
6https://www.hindawi.org/books/categories/
7https://alc.ae/publications/kalima/
8https://archive.org/details/201409_201409
9http://al-nada.eb2a.com/1000ela&ela/
10https://www.hindawi.org/books/90463596/
11https://www.hindawi.org/books/72707304/
12https://ar.wikipedia.org/wiki/المقطات
13https://archive.org/details/majid_magazine
14https://ar.wikipedia.org/
<table>
  <tr>
    <th>Prompt</th>
    <th>#Words per Sentence</th>
    <th>Prompt Text</th>
    <th>% Discarded</th>
  </tr>
  <tr>
    <td>Prompt 1</td>
    <td>2-4</td>
    <td>I am creating a children's textbook to practice reading in Arabic. I need short sentences containing 2 to 4 words that are limited to children's vocabulary. Give me 200 sentences in Standard Arabic -- no need to include English.</td>
    <td>1.5%</td>
  </tr>
  <tr>
    <td colspan="4">Examples</td>
  </tr>
  <tr>
    <td>Prompt 2</td>
    <td>5-7</td>
    <td>I am creating a children's textbook to practice reading in Arabic. I need 5-word, 6-word, and 7-word sentences that are limited to children's vocabulary. Give me 150 sentences in Standard Arabic -- no need to include English.</td>
    <td>1.3%</td>
  </tr>
  <tr>
    <td colspan="4">Examples</td>
  </tr>
  <tr>
    <td>Prompt 3</td>
    <td>8-10</td>
    <td>I am creating a children's textbook to practice reading in Arabic. I need long sentences (8-word, 9-word, and 10-word sentences) that are limited to children's vocabulary. Give me 100 sentences in Standard Arabic -- no need to include English.</td>
    <td>1.0%</td>
  </tr>
  <tr>
    <td colspan="4">Examples</td>
  </tr>
</table>

Table 9: ChatGPT Prompts. % Discarded is the percentage of discarded sentences due to grammatical errors.

Constitutions The first 2000 words of the Arabic constitutions from 16 Arabic speaking countries, collected from MCWC dataset (El-Haj and Ezzini, 2024).

UN The Arabic translation of the Universal Declaration of Human Rights.15

C.1.5 Religion

Old Testament The first 20 chapters of the Book of Genesis (Smith and Van Dyck, 1865).16

New Testament The first 16 chapters of the Book of Matthew (Smith and Van Dyck, 1860).16

Quran The first three Surahs and the last 14 Surahs from the Holy Quran. We selected the text from the Quran Corpus Project (Dukes et al., 2013).17

Hadith The first 75 Hadiths from Sahih Bukhari (al Bukhari, 846). We selected the text from the LK Hadith Corpus18 (Altammami et al., 2019).

Some datasets are chosen because they already have annotations available for other tasks. For example, dependency treebank annotations exist for Odes, Quran, Hadith, 1001, Hayy, OT, NT, Sara,WikiNews, ALC, BTEC, and ZAEBUC (Habash et al., 2022a).

15https://www.un.org/ar/about-us/universal-declaration-of-human-rights
16https://www.arabicbible.com/
17https://corpus.quran.com/
18https://github.com/ShathaTm/LK-Hadith-Corpus

C.2 Domains

Arts & Humanities The Arts and Humanities domain comprised the following subdomains.
• Literature and Fiction: Encompasses novels, short stories, poetry, and other creative writing forms that emphasize narrative and artistic expression.
• Religion and Philosophy: Contains religious texts, philosophical works, and related writings that explore spiritual beliefs, ethics, and metaphysical ideas.
• Education and Academic Texts (on Arts and Humanities): Includes textbooks, scholarly articles, and educational materials that are often structured for learning and academic purposes.
• General Knowledge and Encyclopedic Content (on Arts and Humanities): Covers reference materials such as encyclopedias, almanacs, and general knowledge articles that provide broad information on various topics.
• News and Current Affairs (on Arts and Humanities): Includes newspapers, magazines, and online news sources that report on current events and issues affecting society.

Social Sciences The Social Sciences domain comprised the following subdomains.
• Business and Law: Encompasses legal texts, business strategies, financial reports, and corporate documentation relevant to professional and legal contexts.
• Social Sciences and Humanities: Covers disciplines like sociology, anthropology, history, and cultural studies, which explore human society and culture.

• Education and Academic Texts (on Social Sciences): Includes textbooks, scholarly articles, and educational materials that are often structured for learning and academic purposes.

• General Knowledge and Encyclopedic Content (on Social Sciences): Covers reference materials such as encyclopedias, almanacs, and general knowledge articles that provide broad information on various topics.

• News and Current Affairs (on Social Sciences): Includes newspapers, magazines, and online news sources that report on current events and issues affecting society.

Specialized  Represents readers with advanced skills, typically starting in 9th grade or above in specialized topics, who can comprehend and engage with complex, domain-specific texts in specialized fields.

STEM  The Science, Technology, Engineering and Mathematics domain comprised the following subdomains.

• Science and Technology: Includes scientific research papers, technology articles, and technical manuals that focus on advancements and knowledge in science and tech fields.

• Education and Academic Texts (on STEM): Includes textbooks, scholarly articles, and educational materials that are often structured for learning and academic purposes.

• General Knowledge and Encyclopedic Content (on STEM): Covers reference materials such as encyclopedias, almanacs, and general knowledge articles that provide broad information on various topics.

• News and Current Affairs (on STEM): Includes newspapers, magazines, and online news sources that report on current events and issues affecting society.

C.3 Readership Groups

Foundational  This level includes learners, typically up to 4th grade or age 10, who are building basic literacy skills, such as decoding words and understanding simple sentences.

Advanced  Refers to individuals with average adult reading abilities, capable of understanding a variety of texts with moderate complexity, handling everyday reading tasks with ease.
<table>
  <tr>
    <th>Resource</th>
    <th>#Documents</th>
    <th>#Sentences</th>
    <th>#Words</th>
  </tr>
  <tr>
    <td>al-Kashkuul</td>
    <td>17</td>
    <td>330</td>
    <td>2,306</td>
  </tr>
  <tr>
    <td>Arabian Nights</td>
    <td>24</td>
    <td>669</td>
    <td>6,835</td>
  </tr>
  <tr>
    <td>ALC</td>
    <td>16</td>
    <td>676</td>
    <td>8,395</td>
  </tr>
  <tr>
    <td>ArabicMMLU</td>
    <td>344</td>
    <td>6,205</td>
    <td>187,604</td>
  </tr>
  <tr>
    <td>BTEC</td>
    <td>20</td>
    <td>1,865</td>
    <td>14,663</td>
  </tr>
  <tr>
    <td>chatGPT</td>
    <td>3</td>
    <td>443</td>
    <td>2,502</td>
  </tr>
  <tr>
    <td>Constitutions</td>
    <td>16</td>
    <td>1,490</td>
    <td>30,370</td>
  </tr>
  <tr>
    <td>Emarati Curriculum</td>
    <td>126</td>
    <td>13,365</td>
    <td>113,952</td>
  </tr>
  <tr>
    <td>Green Library</td>
    <td>58</td>
    <td>2,809</td>
    <td>45,078</td>
  </tr>
  <tr>
    <td>Hadith</td>
    <td>75</td>
    <td>672</td>
    <td>7,057</td>
  </tr>
  <tr>
    <td>Hanging Odes</td>
    <td>10</td>
    <td>764</td>
    <td>7,269</td>
  </tr>
  <tr>
    <td>Hayy ibn Yaqdhan</td>
    <td>1</td>
    <td>65</td>
    <td>1,038</td>
  </tr>
  <tr>
    <td>Hindawi</td>
    <td>275</td>
    <td>13,195</td>
    <td>227,677</td>
  </tr>
  <tr>
    <td>Kalima</td>
    <td>62</td>
    <td>2,767</td>
    <td>43,423</td>
  </tr>
  <tr>
    <td>Majed</td>
    <td>294</td>
    <td>11,490</td>
    <td>121,126</td>
  </tr>
  <tr>
    <td>Mama Makes Bread</td>
    <td>1</td>
    <td>39</td>
    <td>468</td>
  </tr>
  <tr>
    <td>My Language Sings</td>
    <td>16</td>
    <td>362</td>
    <td>1,897</td>
  </tr>
  <tr>
    <td>New Testament</td>
    <td>16</td>
    <td>566</td>
    <td>9,471</td>
  </tr>
  <tr>
    <td>Old Testament</td>
    <td>20</td>
    <td>525</td>
    <td>8,874</td>
  </tr>
  <tr>
    <td>Poems and News</td>
    <td>1</td>
    <td>391</td>
    <td>1,239</td>
  </tr>
  <tr>
    <td>Poems of Suleiman Al-Issa</td>
    <td>1</td>
    <td>97</td>
    <td>336</td>
  </tr>
  <tr>
    <td>Quran</td>
    <td>42</td>
    <td>405</td>
    <td>7,744</td>
  </tr>
  <tr>
    <td>ReadMe++</td>
    <td>88</td>
    <td>1,371</td>
    <td>32,131</td>
  </tr>
  <tr>
    <td>Sara</td>
    <td>1</td>
    <td>57</td>
    <td>1,169</td>
  </tr>
  <tr>
    <td>Spacetoon Songs</td>
    <td>53</td>
    <td>870</td>
    <td>3,836</td>
  </tr>
  <tr>
    <td>Subtitles</td>
    <td>11</td>
    <td>502</td>
    <td>3,207</td>
  </tr>
  <tr>
    <td>Universal Declaration of Human Rights</td>
    <td>1</td>
    <td>88</td>
    <td>1,276</td>
  </tr>
  <tr>
    <td>WikiNews</td>
    <td>62</td>
    <td>875</td>
    <td>15,967</td>
  </tr>
  <tr>
    <td>Wikipedia</td>
    <td>168</td>
    <td>5,402</td>
    <td>117,100</td>
  </tr>
  <tr>
    <td>ZAEBUC</td>
    <td>100</td>
    <td>1,086</td>
    <td>15,361</td>
  </tr>
  <tr>
    <th>Totals</th>
    <th>1,922</th>
    <th>69,441</th>
    <th>1,039,371</th>
  </tr>
</table>

Table 10: BAREC Corpus Details: the texts used to build the dataset, and the number of documents, sentences, and words extracted from each text.
D Additional Results

D.1 Confusion Matrix

Figure 3 shows the confusion matrix for the best-performing model from Stage 1: the AraBERTv2 model trained on D3Tok sentences with Cross-Entropy (CE) loss. The matrix uses F-scores to account for the unbalanced distribution of readability levels. The strong diagonal indicates a high rate of exact matches between predicted and gold labels. However, the model exhibits more disagreement at the higher, more difficult levels—likely due to the scarcity of training examples in those levels. Additionally, the model shows a tendency to under-estimate readability levels, favoring lower labels. This aligns with the patterns observed in the error analysis discussed in Section 6.5.

![Confusion matrix of F-score across the different readability levels for the best model from stage 1.](page_374_682_1002_496.png)

Figure 3: Confusion matrix of F-score across the different readability levels for the best model from stage 1.
D.2 All Loss Functions

<table>
  <tr>
    <th>Input</th>
    <th>Model</th>
    <th>Acc<sup>19</sup></th>
    <th>±1 Acc<sup>19</sup></th>
    <th>Dist</th>
    <th>QWK</th>
  </tr>
  <tr>
    <td><b>Word</b></td>
    <td>SVM</td>
    <td>36.2%</td>
    <td>47.9%</td>
    <td>2.03</td>
    <td>53.4%</td>
  </tr>
  <tr>
    <td><b>D3Tok</b></td>
    <td>SVM</td>
    <td>37.2%</td>
    <td>49.3%</td>
    <td>1.92</td>
    <td>56.5%</td>
  </tr>
  <tr>
    <td><b>Word</b></td>
    <td>DecisionTree</td>
    <td>27.2%</td>
    <td>41.2%</td>
    <td>2.50</td>
    <td>44.2%</td>
  </tr>
  <tr>
    <td><b>D3Tok</b></td>
    <td>DecisionTree</td>
    <td>29.9%</td>
    <td>44.2%</td>
    <td>2.33</td>
    <td>52.5%</td>
  </tr>
  <tr>
    <td colspan="6"><b>D3Tok</b> AraBERTv2</td>
  </tr>
  <tr>
    <td>+CE</td>
    <td></td>
    <td><b>56.6%</b></td>
    <td>69.9%</td>
    <td>1.14</td>
    <td>80.0%</td>
  </tr>
  <tr>
    <td>+EMD</td>
    <td></td>
    <td>55.3%</td>
    <td>70.3%</td>
    <td><b>1.11</b></td>
    <td>81.2%</td>
  </tr>
  <tr>
    <td>+OLL2</td>
    <td></td>
    <td>35.2%</td>
    <td>70.3%</td>
    <td>1.25</td>
    <td>82.0%</td>
  </tr>
  <tr>
    <td>+OLL15</td>
    <td></td>
    <td>47.3%</td>
    <td>71.1%</td>
    <td>1.13</td>
    <td>82.8%</td>
  </tr>
  <tr>
    <td>+OLL1</td>
    <td></td>
    <td>50.8%</td>
    <td>71.5%</td>
    <td>1.12</td>
    <td>81.7%</td>
  </tr>
  <tr>
    <td>+OLL05</td>
    <td></td>
    <td>53.1%</td>
    <td>68.8%</td>
    <td>1.18</td>
    <td>79.7%</td>
  </tr>
  <tr>
    <td>+SOFT2</td>
    <td></td>
    <td>55.8%</td>
    <td>69.8%</td>
    <td>1.15</td>
    <td>80.0%</td>
  </tr>
  <tr>
    <td>+SOFT3</td>
    <td></td>
    <td>56.4%</td>
    <td>69.9%</td>
    <td>1.14</td>
    <td>80.1%</td>
  </tr>
  <tr>
    <td>+SOFT4</td>
    <td></td>
    <td>56.4%</td>
    <td>69.9%</td>
    <td>1.15</td>
    <td>79.6%</td>
  </tr>
  <tr>
    <td>+SOFT5</td>
    <td></td>
    <td>56.2%</td>
    <td>69.5%</td>
    <td>1.17</td>
    <td>79.3%</td>
  </tr>
  <tr>
    <td>+Reg</td>
    <td></td>
    <td>43.1%</td>
    <td><b>73.1%</b></td>
    <td>1.13</td>
    <td><b>84.0%</b></td>
  </tr>
</table>

Table 11: Loss functions comparisons on **BAREC** Dev set. For SVM and Decision Tree classifiers, we used count vectorizer.

D.3 Impact of Training Granularity on Readability Level Prediction

To analyze the effect of training granularity on readability level prediction, we compare two approaches: (1) training on all 19 levels and then mapping predictions to lower levels (7, 5, or 3), and (2) training directly on the target granularity.

Table 12 presents the results of this comparison. Overall, training on 19 levels and then mapping achieves slightly better performance across for 5-level and 3-level granularities compared to direct training. Moreover, the performance gap between the two approaches widens as the target granularity becomes coarser, suggesting that finer-grained supervision during training provides more informative learning signals, which translate into improved generalization when predictions are mapped into broader scales.

<table>
  <tr>
    <th>Train Gran</th>
    <th>Dev Gran</th>
    <th>Input</th>
    <th>Model</th>
    <th>Acc</th>
    <th>±1 Acc</th>
    <th>Dist</th>
    <th>QWK</th>
  </tr>
  <tr>
    <td>19</td>
    <td>7</td>
    <td>D3Tok</td>
    <td>CE</td>
    <td><b>65.9%</b></td>
    <td>88.9%</td>
    <td>0.51</td>
    <td>79.9%</td>
  </tr>
  <tr>
    <td>7</td>
    <td>7</td>
    <td>D3Tok</td>
    <td>CE</td>
    <td>65.2%</td>
    <td><b>89.5%</b></td>
    <td><b>0.50</b></td>
    <td><b>81.0%</b></td>
  </tr>
  <tr>
    <td>19</td>
    <td>5</td>
    <td>D3Tok</td>
    <td>CE</td>
    <td><b>70.3%</b></td>
    <td>93.5%</td>
    <td><b>0.37</b></td>
    <td><b>78.3%</b></td>
  </tr>
  <tr>
    <td>5</td>
    <td>5</td>
    <td>D3Tok</td>
    <td>CE</td>
    <td>67.8%</td>
    <td><b>93.7%</b></td>
    <td>0.39</td>
    <td>77.3%</td>
  </tr>
  <tr>
    <td>19</td>
    <td>3</td>
    <td>D3Tok</td>
    <td>CE</td>
    <td><b>76.5%</b></td>
    <td><b>97.6%</b></td>
    <td><b>0.26</b></td>
    <td><b>74.7%</b></td>
  </tr>
  <tr>
    <td>3</td>
    <td>3</td>
    <td>D3Tok</td>
    <td>CE</td>
    <td>74.4%</td>
    <td>96.9%</td>
    <td>0.29</td>
    <td>74.0%</td>
  </tr>
</table>

Table 12: Comparison between training on 19 levels then mapping to the target granularity vs. training directly on the target granularity.
D.4 Ensembles & Oracles

<table>
  <tr>
    <th rowspan="2">Word</th>
    <th colspan="5">CE</th>
    <th>EMD</th>
    <th>Reg</th>
    <th colspan="5">Metrics</th>
  </tr>
  <tr>
    <th>Lex</th>
    <th>D3Tok</th>
    <th>D3Lex</th>
    <th>D3Tok</th>
    <th>D3Tok</th>
    <th></th>
    <th></th>
    <th>Acc<sup>19</sup></th>
    <th>±1 Acc<sup>19</sup></th>
    <th>Dist</th>
    <th>QWK</th>
  </tr>
  <tr>
    <td>✓</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td>51.6%</td>
    <td>65.9%</td>
    <td>1.32</td>
    <td>76.3%</td>
  </tr>
  <tr>
    <td></td>
    <td>✓</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td>50.1%</td>
    <td>65.4%</td>
    <td>1.29</td>
    <td>77.7%</td>
  </tr>
  <tr>
    <td></td>
    <td></td>
    <td>✓</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td><b>56.6%</b></td>
    <td>69.9%</td>
    <td>1.14</td>
    <td>80.0%</td>
  </tr>
  <tr>
    <td></td>
    <td></td>
    <td></td>
    <td>✓</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td>53.2%</td>
    <td>67.1%</td>
    <td>1.24</td>
    <td>78.6%</td>
  </tr>
  <tr>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td>✓</td>
    <td></td>
    <td></td>
    <td></td>
    <td>55.3%</td>
    <td>70.3%</td>
    <td><b>1.11</b></td>
    <td>81.2%</td>
  </tr>
  <tr>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td>✓</td>
    <td></td>
    <td></td>
    <td>43.1%</td>
    <td><b>73.1%</b></td>
    <td>1.13</td>
    <td><b>84.0%</b></td>
  </tr>
  <tr>
    <td>Average</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td>46.9%</td>
    <td>72.5%</td>
    <td><b>1.11</b></td>
    <td>83.4%</td>
  </tr>
  <tr>
    <td>Most Common</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td>56.3%</td>
    <td>70.0%</td>
    <td>1.13</td>
    <td>80.4%</td>
  </tr>
  <tr>
    <th colspan="8">Oracle Combinations</th>
    <th colspan="5"></th>
  </tr>
  <tr>
    <td>✓</td>
    <td>✓</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td>62.4%</td>
    <td>76.6%</td>
    <td>0.88</td>
    <td>88.4%</td>
  </tr>
  <tr>
    <td>✓</td>
    <td></td>
    <td>✓</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td>63.5%</td>
    <td>76.7%</td>
    <td>0.89</td>
    <td>87.7%</td>
  </tr>
  <tr>
    <td>✓</td>
    <td></td>
    <td></td>
    <td>✓</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td>63.2%</td>
    <td>76.6%</td>
    <td>0.88</td>
    <td>88.2%</td>
  </tr>
  <tr>
    <td>✓</td>
    <td></td>
    <td></td>
    <td></td>
    <td>✓</td>
    <td></td>
    <td></td>
    <td></td>
    <td>63.3%</td>
    <td>77.9%</td>
    <td>0.83</td>
    <td>89.2%</td>
  </tr>
  <tr>
    <td>✓</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td>✓</td>
    <td></td>
    <td></td>
    <td>62.0%</td>
    <td>80.7%</td>
    <td>0.77</td>
    <td>90.8%</td>
  </tr>
  <tr>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td>69.5%</td>
    <td>82.3%</td>
    <td>0.67</td>
    <td>91.4%</td>
  </tr>
  <tr>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td></td>
    <td></td>
    <td></td>
    <td>72.0%</td>
    <td>84.5%</td>
    <td>0.59</td>
    <td>92.6%</td>
  </tr>
  <tr>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td></td>
    <td></td>
    <td>73.6%</td>
    <td>86.6%</td>
    <td>0.53</td>
    <td>93.4%</td>
  </tr>
  <tr>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td></td>
    <td>75.2%</td>
    <td>87.4%</td>
    <td>0.50</td>
    <td>93.8%</td>
  </tr>
</table>

Table 13: Comparison between individual models, ensembles and oracles on BAREC Dev set.

Reviewer paper summaries: This paper presented the BAREC Arabic corpus for sentence readability assessment. The corpus is composed of 68K Arabic sentences annotated for readability level according to a 19-level readability scheme. The corpus is created by combining sentences from a variety of Arabic resources and then manually annotated by humans on the 19-level scale. The paper then fine-tunes 3 different Arabic BERT models on the corpus and compares their performance for different combinations of input segmentations.
Summary of strengths: The size of the corpus is pretty large (68k sentences) and all sentences are fully annotated by humans for their readability level. The corpus also draws data from multiple sources, covering three main domains (e.g. STEM, humanities, etc.)
Summary of weaknesses: - The annotation process for the sentences is unclear. From section 3, it can be concluded that the sentences are labeled according to a 19-level scheme based on previous literature of Taha-Thomure (2017). However, little to no details are provided about what these levels mean, how to differentiate between them, and why this particular scheme was chosen for the corpus. This makes it difficult to understand why this choice is better for readability annotation compared to labeling levels used in previous datasets. The pyramid presented in Figure 1 is also confusing with little explanation of what BAREC-3/5/7 levels mean and how the mapping to the 19-levels work. Further, the process that the annotators follow to decide the label of the sentence is described properly; a large table is provided in Appendix A but it’s not clear how they make a decision for each textual feature.

- The experiments run on the corpus are limited to just a comparison of 3 Arabic BERT variants for different types of segmentation of the input sentences and different choices loss functions in fine-tuning, which do not provide new insights.
Comments, suggestions and typos: The organization and writing of the paper is confusing: it first starts with the annotation process in section 3, then discusses the data sentences were obtained from in section 4, and the annotator agreement is later discussed in section 6 which is about experiments. I suggest revising the organization to tie all these points together.
Overall assessment scores: 3
Metareview score: 3
Metareview: This paper introduces BAREC, a large-scale, fine-grained Arabic readability corpus containing over 68,000 manually annotated sentences across 19 readability levels. The dataset is constructed with strong attention to genre diversity and pedagogical granularity. The authors benchmark multiple Arabic BERT models on this resource and commit to releasing the dataset and accompanying guidelines.
Summary of reasons to publish: The reviewers recognize BAREC as a valuable contribution to the Arabic NLP community. It addresses a gap in fine-grained readability assessment resources, particularly in low-resource languages like Arabic. The dataset's scale, annotation quality (QWK = 81.3%), and broad topical coverage support its utility. The authors' detailed documentation and experimental benchmarks further enhance the dataset’s usability.