QAES: First Publicly-Available Trait-Specific Annotations for Automated Scoring of Arabic Essays (2024)

Abstract: Automated Essay Scoring (AES) has emerged as a significant research problem within natural language processing, providing valuable support for educators in assessing student writing skills. In this paper, we introduce QAES, the first publicly available trait-specific annotations for Arabic AES, built on the Qatari Corpus of Argumentative Writing (QCAW). QAES includes a diverse collection of essays in Arabic, each of them annotated with holistic and trait-specific scores, including relevance, organization, vocabulary, style, development, mechanics, and grammar. In total, it comprises 195 Arabic essays (with lengths ranging from 239 to 806 words) across two distinct argumentative writing tasks. We benchmark our dataset against the state-of-the-art English baselines and a feature-based approach. In addition, we discuss the adopted guidelines and the challenges encountered during the annotation process. Finally, we provide insights into potential areas for improvement and future directions in Arabic AES research.

text: Proceedings of The Second Arabic Natural Language Processing Conference, pages 337–351 August 16, 2024 ©2024 Association for Computational Linguistics
1 Introduction
Automated Essay Scoring (AES) is used to automatically evaluate essays, eliminating the need for human intervention. AES has gained significant importance in educational assessment, offering an efficient way to evaluate written essays promptly. Traditionally, AES focused on assigning a single holistic score to an essay, reflecting its overall quality (Xie et al., 2022; Yang et al., 2020). This approach simplifies the evaluation by summarizing the performance of the essay. However, relying solely on a uni-dimensional score is insufficient for guiding students on how to improve areas of weakness. Consequently, recent research has shifted
towards trait-specific scoring, which assigns scores to distinct traits of the essay (Kumar et al., 2022; Ormerod, 2022). By assessing each trait separately, educators and students gain deeper insight into areas for improvement. Although significant progress has been made in AES for languages such as English (Klebanov and Madnani, 2022), the assessment of Arabic essays remains understudied. This is primarily due to the lack of publicly available annotated datasets tailored for Arabic essay scoring.
To address this challenge, we introduce the QCAW annotations for Automated Essay Scoring (QAES),1 the first freely available annotated Arabic essay corpus for trait-specific scoring. We build QAES by annotating the Qatari Corpus of Argumentative Writing (QCAW) (Ahmed et al., 2024) across seven traits: Relevance, Organization, Vocabulary, Style, Development, Mechanics, and Grammar. We release these annotations to the research community, aiming to foster advancements in Arabic AES.
We also evaluate the performance of featurebased and state-of-the-art English baselines on the dataset. Our efforts help bridge the resource gap and empower educators and researchers with valuable insights into the nuances of Arabic writing.
The contributions of this paper are four-fold: (1) Annotating the Arabic essays of the QCAW dataset, providing both holistic and trait-specific scores, (2) Releasing QAES,2 the first publiclyavailable trait-specific annotations for Arabic AES, (3) Providing insights into the annotation challenges and areas for improvement, and (4) Evaluating the performance of feature-based and stateof-the-art baselines on QAES.
1Pronounced in Arabic as “ ̄” (as in “hñÊÖÏ @ áK. ̄”). 2https://sites.google.com/view/bigir/datasets
337
The rest of the paper is organized as follows: Section 2 reviews related work on Arabic essay datasets. Section 3 presents the QCAW dataset considered for annotating. Section 4 outlines the annotation process. Section 5 provides a detailed analysis of the annotations. Section 6 discusses the experimental setup and results. Finally, Section 7 concludes with future directions.
2 Related Work
In Automated Essay Scoring (AES), a significant contrast exists between English and Arabic datasets. While English AES benefits from well-established large-scale benchmark datasets, such as the Automated Student Assessment Prize (ASAP)3 dataset, there is a lack of publicly available annotated datasets for Arabic AES. This could be attributed to the limited research focus on the Arabic language, coupled with its high ambiguity, rich morphology, and complex morpho-syntactic rules. Despite these challenges, some initiatives have developed modest Arabic essay corpora for preliminary studies. These datasets, summarized in Table 1, although limited, are essential for advancing Arabic AES.
Publicly-Available Arabic Essay Datasets When exploring Arabic AES datasets, three stand out: (1) The Zayed Arabic English Bilingual Undergraduate Corpus (ZAEBUC) (Habash and Palfreyman, 2022), consisting of essays by first-year university students in the UAE, enriched with linguistic annotations such as POS tagging, grammar and spelling corrections, and lemmatization; (2) The Arabic Learner Corpus (ALC),4 a collection of Arabic texts written by non-native learners of Arabic in Saudi Arabia also annotated with linguistic features, yet, they are not publicly disclosed; and (3) The Qatari Corpus of Argumentative Writing (QCAW) (Ahmed et al., 2024), containing long essays with publicly available POS tags but non-public holistic score annotations. While these datasets provide valuable linguistic insights, they lack publicly available holistic or trait-specific content quality annotations, limiting their usability in developing AES systems.
Non-public Arabic Essay Datasets Several datasets are not publicly available for research. For example the "Abbir" dataset (Alghamdi et al.,
3https://www.kaggle.com/c/asap-aes 4https://www.arabiclearnercorpus.com 5The essays are publicly available, but not the annotations.
2014), comprising essays from college students in Saudi Arabia and including holistic scores ranging from 1 to 6 (best score). Another example is the "AAEE" dataset (Azmi et al., 2019), which scores essays written by students in grades 7 to 12, based on criteria such as semantic analysis, writing style, and spelling accuracy.
Short Answer Datasets Several short answer datasets have been collected for Arabic AES. For instance, Abdeljaber (2021) introduced an accessible dataset with 330 answers for 10 questions. Similarly, Ouahrani and Bennouar (2020) presents the freely available AR-ASAG dataset, which includes 2,133 student answers to 48 questions from a cybercrimes course. Other datasets are not publicly available, such as the "eJaya-NN" dataset (Gaheen et al., 2020) with 240 answers to one quiz question, and the "Philosophy" dataset (Gomaa, 2014), an expansion of their initial Arabic short answer benchmark set (Gomaa and Fahmy, 2014) containing 600 responses to 50 questions scored from 0 to 10. Shehab et al. (2018) collected 210 short answers from secondary students in a sociology course. Additionally, Nael et al. (2022) translated the English ASAP Short Answer Scoring dataset into Arabic using Google Translate. Several other datasets exist, but they are typically close-domain and small-scale with limited applicability.
The absence of publicly available annotations for Arabic essay scoring obstructs the development and validation of AES models tailored to Arabic essays. Most existing Arabic essay datasets are small, often confined to specific domains, very limited in terms of prompts, primarily focus on short answers, and only consider holistic scoring, oversimplifying the multifaceted nature of writing assessment. Furthermore, these datasets are often proprietary or not publicly accessible, hindering reproducibility and collaboration. In response to
this challenge, we extend the Qatari Corpus of Argumentative Writing (QCAW) to present QAES, the first public trait-specific annotations for Arabic AES. Augmenting the QCAW dataset with adequate annotations would unlock its full potential and drive the evolution of AES in Arabic, echoing the advancements seen in English AES.
3 The QCAW Corpus
The Qatari Corpus of Argumentative Writing (QCAW) (Ahmed et al., 2024) is a publicly available corpus comprising 195 argumentative essays written in both Arabic and English by first-year Arabic native university students as part of a compulsory university-level course. It provides a valuable resource for AES as its lengthy essays offer insights into Arabic linguistic styles and different writing proficiency levels. Furthermore, the essays were collected from a diverse student sample, considering factors such as gender, year of study, and major. In our work, we only consider the Arabic essays. These essays constitute responses to two distinct argumentative writing tasks, originally derived from TOEFL writing prompts: Task 1: ‘Telephones and emails have made communication between people less personal"; and Task 2: ‘Technology has enabled students nowadays to learn more information quickly". Table 2 provides a breakdown of the tasks featured in the QCAW dataset, showing a skew towards Task 1 in terms of both the number of essays and their length. On average, after tokenizing with the NLTK tokenizer and removing punctuation, essays are approximately 489 words long, ranging from 239 to 806 words. The total number of tokens across all essays is 95,342.
4 Constructing QAES
To create QAES, annotators were hired to annotate the Arabic essays of the QCAW dataset across seven traits: Relevance (REL), Organization (ORG), Vocabulary (VOC), Style (STY), Development (DEV), Mechanics (MEC), and Grammar (GRA) in addition to a Holistic (HOL) score of overall quality. The process entails the deployment of trusted assessment guidelines/rubrics, the selec-
tion of specialist annotators, and the delivery of indepth training sessions to ensure consistency and understanding throughout the annotation process.
4.1 Annotation Guidelines To score student responses, we provided the annotators with the rubrics used in the Core Academic Skills Test (CAST) developed by the Qatar University Testing Center (QUTC).6 These rubrics were designed to assess students’ ability to write persuasive/argumentative essays, which matches perfectly with the tasks covered in the QCAW corpus. Additionally, CAST’s rubrics underwent extensive measures to ensure that they were aligned with the guidelines established by subject matter experts and reviewed by independent specialists.
The rubrics evaluate the seven traits considered in our work. The REL trait is assessed on threelevel scale: 0 (not relevant), 1 (partially relevant), and 2 (completely relevant). The other six traits are evaluated on a five-level scale: 1 (lowest) to 5 (highest). Each level includes a detailed description summarizing the characteristics of the text classified at that level, as shown in Table 3. For example, a score of 1 in the ORG trait is assigned if the introduction and conclusion are absent, and there is no organization or sequence between paragraphs. In contrast, a score of 5 indicates a well-organized text with a clear introduction, two to three coherent body paragraphs, and a strong conclusion. Similarly, the STY trait is assigned a score of 1 if the text employs very basic linear connecting words such as "and" and "then". The highest score (5) is given when the discourse is well-developed, with good inclusion of subtopics and details, a strong conclusion, appropriate use of a variety of organizational patterns, and a wide range of structural cohesion devices. Overall, a score of zero is given to all traits if the response was solely memorized or copied from the Internet, or if the student failed to attempt the task. In addition to the trait scores, an overall holistic score is computed by summing the seven trait scores.
4.2 Annotation Process Two Arabic language specialists (main annotators) were selected to evaluate student essays. Both possess teaching and assessment experience with essay questions for a similar age group. One of the annotators is a CAST-certified annotator with prior
6https://www.qu.edu.qa/sites/en_US/ testing-center/TestDevelopment/cast
experience employing the CAST rubrics and international standardized writing assessments. Additionally, a third annotator, a language assessment expert with extensive teaching and scoring expertise for CAST and other language evaluations, was employed to help resolve disagreements between the two primary annotators.
Before starting the scoring process, the two main annotators received training, which included moderation and norming sessions to ensure they fully understand the assessment rubric and maintain consistent annotation procedures. The third annotator led the initial moderation session, providing an overview of the process and QCAW background, and presenting examples of QCAW essays. The rubrics were then thoroughly discussed with an emphasis on the differences between each level within each trait. Next, the norming process took place, where the two main annotators independently scored four essays of varying levels. After submitting their scores, a discussion followed about the rationale behind each score, culminating in a group discussion to address any discrepancies. After revising some essays, a second moderation session was held between the two main annotators to reach agreements on specific decisions. The third annotator was subsequently briefed to ensure all decisions were mutually agreed upon by all three annotators. In cases where there is a difference of 11 points or more out of a total of 32 points in the overall holistic score between the main annotators, the third annotator reviews the responses and the scores provided by the main annotators to determine the final score for each trait.
Each writing task was then completely assessed independently by the main annotators. The final score per trait was calculated as the rounded integer mean of the two assessments unless the response was reviewed by the third annotator in the case of
score discrepancy as described earlier.
5 Annotation Analysis
In this section, we analyze the annotations via two essay examples explaining score assignments, examine the inter-annotator agreement, and study trait distributions across tasks.
5.1 Example Analysis of Graded Essays We analyzed two sample essay responses, one from each task, to clarify the scoring rationale employed by the annotators. A detailed description of the two examples is provided in Appendix B.
The first essay, 1-16A, deviated from the given prompt, which called for reflections on the impact of e-mail and telephones on human relations. Instead, the response briefly discussed the pros and cons of social media without substantively engaging with a specified topic or advocating for a particular point of view, leading to a REL score of 1. Despite including an introduction, three paragraphs, and a conclusion, the ORG received a score of 3 due to inadequate coherence. The VOC, though commendable in range, suffered from numerous lexical errors and inappropriate word choices, as they were significantly off-topic and clearly deviated from the main subject, such as “ éK PðQÖÏ @ HX@ñmÌ'@” (traffic accidents), “ HCK. A¾Ë@ ð ¼C B@ ” (wires and cables), and “YK Yî DË @ ð QÒ J JË @ ” (bullying and threatening), leading to a score of 2. The STY scored 3 points for using various connecting words, albeit with limited use of cause-and-effect indicators and occasional lapses in coherence, as evidenced by simplistic linking words such as “½Ë Xð” (and that) and sequential sentences lacking cohesion. The DEV of ideas, which received a score of 2, only superficially addressed the question’s theme without employing persuasive elements, such as citations or examples, resulting in a disjointed narrative lack-
ing coherence. For MEC, the student demonstrated proficiency in punctuation and spelling, earning a score of 3, with no significant errors that detract from readability. The GRA category also received a score of 3 despite pervasive inaccuracies, including errors in derivation and parsing, which could potentially lead to comprehension difficulties upon initial reading. Despite these issues, the essay remained readable, though compromised by lexical and grammatical errors such as “ú æ
” and “ éËñÒm× h. @”.
A higher-level essay example, 2-54A, demonstrated strong REL to the prompt and earned a score of 2, as the ideas directly and accurately addressed the topic without deviation. Its cohesive ORG was commended with a score of 4, showcasing an introductory, followed by four interconnected paragraphs and a concluding summary of the writer’s stance. The VOC trait, which gained a score of 5, highlighted the student’s adept use of a wide range of conventional vocabulary related to the topic, such as “ éJ Ò J K ð QK ñ¢ “ (development and growth), ” é K @YmÌ'@ ð Pñ¢ JÊË” (for development and modernity), and “Õæ Êª
JË @ èXñk. ” (quality of education). In terms of STY, the essay got a score of 4 for effectively employing appropriate linking devices to convey explanations, interpretations, and presentations, ensuring seamless transitions between ideas throughout the paragraphs, such as “ÈC g áÓ ½Ë Xð“ (and that is through), “½Ë X ú Î« BA
JÓ” (as an example of that), “ áºËð” (but), “ IJ k” (where), and “ AÒJ Bð” (especially). The DEV trait scored 4 for the comprehensive exploration of the impact of technology on learning speed and information accessibility, supported by well-reasoned arguments and examples. The MEC got a score of 4, as the essay proficiently used punctuation and spelling, with minor errors attributable to carelessness, such as “Ȳ Aî E @” (it is) and “ZAÖ ß @ ” (but). The GRA achieved the highest possible score of 5 due to correct and versatile sentence structures, with few observed grammatical errors, showing the student’s exceptional writing proficiency. Overall, the essay’s strong coherence, vocabulary richness, stylistic finesse, and grammatical accuracy underscore its effectiveness in addressing the given prompt.
5.2 Inter-Annotator Agreement To analyze the quality of the annotations, we measure the Inter-Annotator Agreement (IAA) using Quadratic Weighted Kappa (QWK), which is widely recognized as the standard metric for AES.
ment as Moderate , Substantial , and Almost Perfect .
QWK measures the agreement between human and automated scores (Williamson et al., 2012). Particularly, it is well-suited for dealing with ordinal scales, as it takes into account the magnitude of the differences between the ratings, making it a suitable metric for computing the inter-annotator reliability in the context of essay scoring (Doewes et al., 2023b). Table 4 presents the IAA for all traits of Task 1 and 2, along with their corresponding strength of agreement according to the scale outlined by Landis and Koch (1977). In particular, Task 1 poses a greater challenge for grading compared to Task 2, as evidenced by its relatively lower QWK values. This disparity in performance could be attributed to the misinterpretation of Task 1 among some students, which was a challenge encountered during the annotation process. Specifically, Task 1 prompt was about mobile phones and emails and their impact on interpersonal relationships. However, some students instead discussed the effects of social media use on communication. Furthermore, a significant portion of the students did not grasp the nature of an argumentative essay, resulting in them transforming the topic into a discussion essay. These issues likely introduce inconsistencies in the grading process.
For the traits, GRA exhibited the lowest agreement for both tasks, indicating a potential source of ambiguity or subjectivity in the scoring. One possible explanation is the difficulty of establishing a consistent quantifiable grading scale of mistakes in essays of varying lengths and qualities. This variability allows annotators considerable discretion in determining the appropriate number of mistakes associated with each score level, complicating the task of consistently identifying and assessing grammatical errors. In contrast, the REL and DEV traits show higher IAA scores for both tasks, with QWK
values exceeding 0.75 for Task 1 and 0.8 for Task 2. The higher agreement in REL can be attributed to its three-level scoring rubric, which simplifies the evaluation process. Similarly, the DEV trait benefits from the most detailed rubric among all traits, minimizing the influence of annotator interpretation, and ensuring more consistent scoring. This difference in IAA scores highlights the importance of clear and detailed rubrics in reducing subjectivity and improving inter-annotator reliability.
5.3 Annotation Statistics Figure 1 depicts the distribution of trait scores across QAES tasks. Overall, the distributions resemble a normal distribution, indicating a positive trend. Notably, MEC and GRA traits exhibit consistency across tasks. This suggests that these traits are task-independent, showing the student’s language proficiency rather than knowledge of a specific topic. Also, there is a similarity between the distributions of MEC and GRA traits, which suggests a strong correlation between those traits.
An important observation drawn from the distribution of REL scores reveals that Task 2 was notably easier to address than Task 1, as evidenced by the skew towards score 2. This is further supported by Figure 2, which reveals a distribution of holistic scores skewed toward higher values for Task 2. Interestingly, we also observed that the rest of the traits are task-dependent. Consequently, there are more students receiving scores of 4 and 5 for Task 2 than for Task 1, in these traits.
This analysis indicates a clear distinction in writing skills across the two tasks, which highlights the importance of using diverse traits to comprehensively evaluate essay writing skills, offering valuable insights for educators to enhance student performance in essay writing.
6 Arabic AES: Preliminary Experiments
To explore the potential of QAES to develop Arabic AES models, we conducted several preliminary experiments. This section details the baseline models and discusses their performance.
6.1 Experimental Setup In this section, we review the baseline models covering their implementation details, the evaluation measures, and the dataset splits used for analysis.
6.1.1 Baselines We consider two types of baselines: feature-based and English state-of-the-art (SOTA). The featurebased baseline involves feature engineering to train a regression model. For the English SOTA baseline, we selected models that performed best on the ASAP dataset, including one achieving the best performance at the holistic level (Xie et al., 2022) and another at the traits level (Kumar et al., 2022).
Feature-based (LR) It uses traditional features to capture different aspects of writing proficiency and trains a scoring model with Linear Regression (LR) for its interpretability and simplicity. The features include (i) surface features (e.g., text length in words and characters), (ii) syntactic features (e.g., sentence structure, POS, spelling errors), (iii) lexical features (e.g., vocabulary richness, lexical density), (iv) semantic features (e.g., coherence, perplexity scores), and (v) N-gram features (e.g., frequency of word sequences). Table 9 presents the full list of features used in our model.
Holistic SOTA (NPCR) We implemented the model introduced by Xie et al. (2022) for English
holistic scoring, optimizing regression and ranking. It predicts the difference between representations of a reference essay and the input essay, then adds the reference essay’s score to the final prediction. BERT embeddings (Devlin et al., 2019) were originally used for essay representations; we replaced BERT with AraBERT (Antoun et al., 2020) and followed the same setup.
Traits SOTA (STL & MTL) We implemented the multi-task approach by Kumar et al. (2022), which achieved impressive results in English trait assessment by treating individual traits as the primary task. Their model combines CNNs and RNNs, with initial essay representations obtained from GloVe embeddings (Pennington et al., 2014). We tested both their single-task learning (STL) and multi-task learning (MTL) models for both holistic and trait scoring using AraBERT (Antoun et al., 2020) embeddings (instead of GloVe) while retaining their original approach.
6.1.2 Implementation Details To implement the LR model, we used Scikit-learn’s regression models and preprocessing tasks and NLTK for text tokenization and stop-word removal. We used Farasa toolkit (Abdelali et al., 2016) for POS tagging and Spellchecker for spelling error detection. For semantic analysis, we leveraged AraBERT embeddings from the publicly available checkpoints on Hugging Face.7
To implement the English SOTA models, we used the publicly available code by Xie et al. (2022).8 We built the Kumar et al. (2022) model using PyTorch, adopting the hyperparameters in Kumar et al. (2022).
6.1.3 Evaluation Measures To evaluate our models, we use QWK, the most common metric for assessing agreement between human annotators and systems. However, it has limitations, including a need for a large sample size, and its sensitivity to the score scale (Doewes et al., 2023a). Hence, for all traits except REL, we need at least 50 predictions to calculate QWK and obtain reliable results (Cicchetti, 1981). To address these limitations, we also use the Root Mean Square Error (RMSE), a metric commonly used for ordinal classification tasks (Esuli et al., 2009) similar to predicting score levels in AES tasks. Report-
7https://huggingface.co/aubmindlab/ bert-base-arabertv02
8https://github.com/CarryCKW/AES-NPCR
ing both QWK and RMSE aims to more reliably evaluate the quality of the tested models.
6.1.4 Dataset Splits As the dataset is small, cross-validation was essential. However, QWK’s reliability depends on the sample size, requiring a minimum of 50 essays per trait (except for relevance). Hence, we used a 2- fold cross-validation split: 40% for training, 10% for development, and 50% for testing. This unconventional split maximizes test examples for more reliable results. We have made the splits available for reproducibility.
6.2 Results and Discussions In this section, we evaluate the baseline performance across various scoring dimensions using QWK and RMSE. We compare the 4 baseline models: LR, STL, MTL, and NPCR, across the two tasks. In addition, we conducted an ablation study on the LR model to assess the impact and determine the significance of each feature category for various traits.
Holistic Scoring Table 5 presents the holistic scoring results. The LR model achieves the highest QWK scores in both tasks, indicating better agreement, while the NPCR model records the lowest QWK scores for both tasks. STL and MTL models show similar agreement levels, though MTL’s QWK score drops notably in Task 2. For RMSE, STL and MTL models score lower in Task 1, reflecting more precise predictions. In Task 2, MTL has the lowest RMSE, while LR consistently has the highest RMSE in both tasks. Overall, the performance is weak, indicated by a very low correlation and an error of about 4-7 points.
We note that a single significant disagreement between raters impacts QWK more than RMSE. This implies that LR model, despite having higher QWK values in both tasks, has a higher RMSE because it captures overall patterns but lacks precise scores. Conversely, other models might more ac-
curately predict most scores but severely misjudge some, leading to lower RMSE and QWK scores.
Trait Scoring Subfigures (a) and (b) in Figure 3 display the QWK and RMSE performance for Task 1, respectively, while subfigures (c) and (d) show the QWK and RMSE performance for Task 2, respectively. This provides a comparative view of the model’s performance across various scoring traits. For both tasks, LR consistently achieved the highest QWK across nearly all dimensions, indicating better agreement with human annotators. For example, in Task 1, LR’s QWK score for ORG was 0.184, outperforming STL’s 0.142 and MTL’s 0.090. Conversely, MTL’s struggle is evident, with negative scores in several traits, such as -0.052 for GRA in Task 1 and -0.055 for ORG in Task 2. STL exhibited intermediate performance, generally outperforming MTL but not reaching the efficacy of LR. Task 2 showed higher QWK measures across all models and traits, aligning with our prior observation that it is easier for students to comprehend. The superior performance of LR can be attributed to the dataset’s small size, as LR is less dependent on large training examples for effective training. In contrast, STL and MTL, with 439,211 and 411,266,6 parameters respectively, being neuralbased methods, typically require more substantial training data to achieve optimal performance.
Regarding RMSE, for Task 1, STL consistently achieved the lowest RMSE values across most dimensions, indicating higher accuracy. For instance, STL recorded an RMSE of 0.698 for REL, compared to LR’s 0.951 and MTL’s 0.716. However, for Task 2, the trend is not consistent across all traits. While STL still outperforms LR in many cases, MTL tends to have the lowest average RMSE values. This analysis indicates that when considering RMSE measure, both MTL and STL are more effective methods than LR, showcasing their potential as robust models for AES.
Typically, a lower RMSE indicates a higher QWK, reflecting a good fit and agreement. However, the LR model showed the highest QWK but also the highest RMSE, suggesting lower accuracy. This discrepancy may arise from QWK’s sensitivity to sample range and quantity.
Conversely, RMSE showed minimal error suggesting good model performance. Despite possible underfitting in STL and MTL neural models, due to their large parameter spaces, their superior RMSE performance indicates that RMSE might be a better measure of model accuracy than QWK for smallscale datasets like QAES.
Ablation study We conducted an ablation study to analyze the impact of different feature categories on LR model performance for various traits. A
detailed representation of the drop in QWK when training LR model with all feature categories but one is provided in Table 6. Results show varying effects on QWK scores across traits and tasks. In Task 1, excluding N-gram features significantly reduces performance across all traits except REL, showing their crucial role. Across both tasks, omitting surface features boosts VOC and GRA scores, while removing semantic features diminishes DEV performance as expected, given their role in measuring text coherence and complexity. While lexical and syntactic features are vital for text understanding, their impact varies for each trait and task. This study shows the critical need for a tailored evaluation of feature importance, taking into account the unique characteristics of each trait and the specific requirements of each task.
7 Conclusion and Future work
In conclusion, QAES represents a pioneering step in the field of Arabic AES by introducing the first publicly available trait-specific annotations for Arabic essays. Our work addresses a critical gap in Arabic AES research, providing a valuable resource for exploring and developing advanced essay scoring systems based on multiple traits including relevance, organization, vocabulary, style, development, mechanics, and grammar.
Benchmarking against English state-of-the-art baselines and a feature-based approach has demonstrated QAES’s potential to serve as a platform for future studies. Despite the challenges encountered during the annotation process, the insights gained pave the way for enhancing the reliability of Arabic AES systems. Moving forward, it is imperative to continue refining the annotation guidelines,
expanding the dataset, and exploring innovative methodologies to further improve the progress of Arabic AES. We believe that the journey towards improving AES for Arabic is just beginning, and QAES is a promising step forward in this ongoing quest for automated essay scoring proficiency.
8 Limitations
Although this work represents a significant step forward in the application of AES for Arabic, several limitations should be acknowledged. First, the dataset size remains relatively small compared to AES datasets available for other languages. This presents a challenge for the application of machine learning, particularly deep learning models, as evidenced by our preliminary results.
Furthermore, while the reported average agreement between the two annotators is substantial across the two tasks, additional moderation sessions could have potentially resolved more disagreements and clarified ambiguous points.
Lastly, this study did not develop any model specifically tailored for Arabic language assessment. Instead, it focused on adapting existing AES methodologies to the Arabic context. We encourage further research to explore the development of specialized models for Arabic AES, considering the unique linguistic characteristics and challenges of the language.
Acknowledgment
This work was supported by NPRP grant# NPRP14S-0402-210127 from the Qatar National Research Fund (a member of Qatar Foundation). The statements made herein are solely the responsibility of the authors.
References Ahmed Abdelali, Kareem Darwish, Nadir Durrani, and
Hamdy Mubarak. 2016. Farasa: A fast and furious segmenter for arabic. In Proceedings of the 2016 conference of the North American chapter of the association for computational linguistics: Demonstrations, pages 11–16.
Hikmat A. Abdeljaber. 2021. Automatic arabic short answers scoring using longest common subsequence and arabic wordnet. IEEE Access, 9:76433–76445.
Abdelhamid M. Ahmed, Xiao Zhang, Lameya M. Rezk, and Wajdi Zaghouani. 2024. Building an Annotated L1 Arabic/L2 English Bilingual Writer Corpus: The Qatari Corpus of Argumentative Writing (QCAW). Corpus-based Studies across Humanities, 1(1):183– 215.
Mansour Alghamdi, Mohamed Alkanhal, Mohamed AlBadrashiny, Abdulaziz Al-Qabbany, Ali Areshey, and Abdulaziz Alharbi. 2014. A hybrid automatic scoring system for Arabic essays. AI Communications, 27(2):103–111.
Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. AraBERT: Transformer-based model for Arabic language understanding. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 9–15, Marseille, France. European Language Resource Association.
Aqil M. Azmi, Maram F. Al-Jouie, and Muhammad Hussain. 2019. Aaee – automated evaluation of students’ essays in arabic language. Information Processing Management, 56(5):1736–1752.
Domenic V Cicchetti. 1981. Testing the normal approximation and minimal sample size requirements of weighted kappa when the number of categories is large. Applied psychological measurement, 5(1):101– 104.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Afrizal Doewes, Nughthoh Kurdhi, and Akrati Saxena. 2023a. Evaluating quadratic weighted kappa as the standard performance metric for automated essay scoring. In 16th International Conference on Educational Data Mining, EDM 2023, pages 103– 113. International Educational Data Mining Society (IEDMS).
Afrizal Doewes, Nughthoh Arfawi Kurdhi, and Akrati Saxena. 2023b. Evaluating Quadratic Weighted
Kappa as the Standard Performance Metric for Automated Essay Scoring. In Proceedings of the 16th International Conference on Educational Data Mining, pages 103–113. International Educational Data Mining Society.
A. Esuli, S. Baccianella, and F. Sebastiani. 2009. Evaluation measures for ordinal regression. In Intelligent Systems Design and Applications, International Conference on, pages 283–287, Los Alamitos, CA, USA. IEEE Computer Society.
Marwa M. Gaheen, Rania M. ElEraky, and Ahmed A. Ewees. 2020. Automated students arabic essay scoring using trained neural network by e-jaya optimization to support personalized system of instruction. Education and Information Technologies, 26(1):1165–1181.
Wael Hassan Gomaa. 2014. Arabic Short Answer Scoring with Effective Feedback for Students Arabic Short Answer Scoring with Effective Feedback for Students. (January).
Wael Hassan Gomaa and Aly Aly Fahmy. 2014. Automatic scoring for answers to arabic test questions. Computer Speech Language, 28(4):833–857.
Nizar Habash and David Palfreyman. 2022. ZAEBUC: An annotated Arabic-English bilingual writer corpus. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 79–88, Marseille, France. European Language Resources Association.
Beata Beigman Klebanov and Nitin Madnani. 2022. Automated essay scoring. Springer Nature.
Rahul Kumar, Sandeep Mathias, Sriparna Saha, and Pushpak Bhattacharyya. 2022. Many hands make light work: Using essay traits to automatically score essays. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1485–1495, Seattle, United States. Association for Computational Linguistics.
J. Richard Landis and Gary G. Koch. 1977. The measurement of observer agreement for categorical data. Biometrics, 33(1):159–174.
Omar Nael, Youssef ELmanyalawy, and Nada Sharaf. 2022. Arascore: A deep learning-based system for arabic short answer scoring. Array, 13:100109.
Christopher Michael Ormerod. 2022. Mapping between hidden states and features to validate automated essay scoring using deberta models. Psychological Test and Assessment Modeling, 64(4):495–526.
Leila Ouahrani and Djamal Bennouar. 2020. AR-ASAG an ARabic dataset for automatic short answer grading evaluation. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 2634–2643, Marseille, France. European Language Resources Association.
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543, Doha, Qatar. Association for Computational Linguistics.
Abdulaziz Shehab, Mahmoud Faroun, and Magdi Rashad. 2018. An automatic arabic essay grading system based on text similarity algorithms. International Journal of Advanced Computer Science and Applications, 9.
David M. Williamson, Xiaoming Xi, and F. Jay Breyer. 2012. A framework for evaluation and use of automated scoring. Educational Measurement: Issues and Practice, 31(1):2–13.
Jiayi Xie, Kaiwei Cai, Li Kong, Junsheng Zhou, and Weiguang Qu. 2022. Automated essay scoring via pairwise contrastive regression. In Proceedings of the 29th International Conference on Computational Linguistics, pages 2724–2733, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
Ruosong Yang, Jiannong Cao, Zhiyuan Wen, Youzheng Wu, and Xiaodong He. 2020. Enhancing automated essay scoring performance via fine-tuning pre-trained language models with combination of regression and ranking. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1560–1569, Online. Association for Computational Linguistics.
A Grading Rubric
For annotating QAES, we employed the rubric from the Core Academic Skills Test (CAST) developed by the Qatar University Testing Center (QUTC). This rubric was used to score 7 traits: REL, ORG, VOC, STY, DEV, MEC, and GRA. Table 7 presents a translated version of CAST grading rubric for each considered trait.
B Detailed Example Analysis of Graded Essays
In this section, we closely examine two comprehensive examples from QAES, one from each task, and thoroughly discuss the rationale behind the scores assigned to the responses by the annotators for the seven different traits: relevance (REL), organization (ORG), vocabulary (VOC), style (STY), development (DEV), mechanics (MEC), and grammar (GRA). Table 8 displays the complete text for the two selected examples, using three different colors to highlight various traits: blue for MEC and GRA, red for STY, and green for VOC. This color-coding provides clear examples for each trait, facilitating
a better understanding of how the responses were evaluated.
Example (16A) of Task 1 had 500 words. Unfortunately, the provided response was not directly related to the prompt. Specifically, the prompt requested insight into the effects of email and telephones on human relationships, but the response briefly touched upon the advantages and disadvantages of social media without delving into the primary topic or attempting to persuade readers to adopt a particular stance. Accordingly, a score of 1 was assigned to the REL trait of this response.
The ORG trait received 3 points, as the response had an introduction, three paragraphs, and a conclusion, although coherence could have been better. The VOC category was awarded 2 points, as the writing contained a good range of words; however, many lexical errors and inappropriate word choices that could impede comprehension were found. Phrases and words were used that do not fit the topic and do not convey the intended meaning. Refer to Table 8, where such instances are colored in green. The STY was awarded 3 points for employing a range of standard transitional phrases, though it demonstrated a limited use of cause-andeffect connectors. Additionally, several sentences were missing transitional elements entirely, which are essential for linking them to preceding content and demonstrating the coherence of the paragraph’s ideas. For clarity, examples of these linking words are highlighted in red in Table 8.
The Dev of ideas was awarded only 2 points because the response barely addressed the question, providing only a superficial treatment of the relevant ideas. The expected means of persuasion, such as examples, were notably absent. The text began by introducing the concept of social networking sites and proceeded to discuss their benefits in the second paragraph. However, it then shifted to their drawbacks in the following paragraph, and subsequently to issues related to mobile phones. This approach was fragmented and lacked coherence, resulting in an incomplete and disjointed discussion.
The MEC category received 3 points as the student used punctuation and spelling correctly throughout the response, without any frequent or gross errors. Finally, the GRA received 3 points, as there were various grammatical errors, such as derivation and parsing marks, which indicated inaccuracies in sentence construction. The text is full of many spelling and grammatical errors in most paragraphs, which may sometimes cause misun-
derstanding when reading the sentence for the first time. Examples of these errors are colored in blue in Table 8.
The chosen sample (54A) for Task 2 comprised 500 words. It earned 2 marks for REL. Specifically, the prompt requested insights into how technology has enabled students nowadays to learn more information quickly and effectively. The student’s response was entirely pertinent to the subject matter of the question, with all the ideas presented being directly related to the topic and no ideas deviating from the content of the required question. The essay began with an introduction, followed by four successive and well-connected paragraphs, culminating in a conclusion that summarizes the writer’s stance on the issue at hand. This ORG was awarded 4 marks. For VOC, which got 5 marks, the student utilized an extensive range of conventional vocabulary with connotations, with the majority of their vocabulary stemming from the semantic field of the topic. Examples of such word choices are colored in green in Table 8.
The use of appropriate linking devices that indicated explanation, interpretation, and presentation according to the ideas presented earned the essay 4 points in the STY category. Examples of such devices are shown in red in Table 8. The topic was conveyed through sequential and interconnected paragraphs, with each idea seamlessly leading to the next one. The DEV trait was awarded 4 marks due to the student’s well-reasoned response on the topic of technology and its impact on the speed of learning and access to information. The writer’s stance was clearly articulated and supported by comprehensive examples and explanations that demonstrated the validity of the content presented. All the ideas, both main and subsidiary, were related to the content of the question. Examples of ideas presented to enrich the topic: in the first paragraph, “facilitating the teaching method” and “developed traditional methods into modern ones”; in the second paragraph “Facilitating the delivery of information to students”, “facilitating students’ understanding”, and “taking into account learning styles”; in the third paragraph “delivering information to students”, and “helps take into account different learning styles”. In the fourth paragraph, “Help take care of people with special needs” and in the fifth paragraph, “Qatar’s role in developing the technology factor”, and “Qatar Vision 2030”.
The student demonstrated strong proficiency in
punctuation and spelling throughout the essay, with only minor errors resulting from carelessness or haste. 4 marks were awarded to the student in this MEC area. The GRA score was the highest possible (5 points) because of the students’ correct and versatile use of sentence structures and writing forms. The writing contained various grammatical structures with rare errors. Generally speaking, the errors in mechanics and grammar did not exceed 10 errors in 500 words. This is excellent - especially when the errors are lapses, such as the words colored in blue shown in Table 8.
C Extracted Features
We utilized five categories of features in the development of our feature-based model. Table 9 presents all the extracted features across these categories, accompanied by brief descriptions. The feature list comprises 23 Surface features, 12 Syntactic features, 5 Lexical features, 2 Semantic features, and 500 unigram and bigram features.

heading: Future Work
text: Finally, Section 7 concludes with future directions. 2 Related Work
In Automated Essay Scoring (AES), a significant contrast exists between English and Arabic datasets. While English AES benefits from well-established large-scale benchmark datasets, such as the Automated Student Assessment Prize (ASAP)3 dataset, there is a lack of publicly available annotated datasets for Arabic AES. This could be attributed to the limited research focus on the Arabic language, coupled with its high ambiguity, rich morphology, and complex morpho-syntactic rules. Despite these challenges, some initiatives have developed modest Arabic essay corpora for preliminary studies. These datasets, summarized in Table 1, although limited, are essential for advancing Arabic AES. Publicly-Available Arabic Essay Datasets When exploring Arabic AES datasets, three stand out: (1) The Zayed Arabic English Bilingual Undergraduate Corpus (ZAEBUC) (Habash and Palfreyman, 2022), consisting of essays by first-year university students in the UAE, enriched with linguistic annotations such as POS tagging, grammar and spelling corrections, and lemmatization; (2) The Arabic Learner Corpus (ALC),4 a collection of Arabic texts written by non-native learners of Arabic in Saudi Arabia also annotated with linguistic features, yet, they are not publicly disclosed; and (3) The Qatari Corpus of Argumentative Writing (QCAW) (Ahmed et al. , 2024), containing long essays with publicly available POS tags but non-public holistic score annotations. While these datasets provide valuable linguistic insights, they lack publicly available holistic or trait-specific content quality annotations, limiting their usability in developing AES systems. Non-public Arabic Essay Datasets Several datasets are not publicly available for research. For example the "Abbir" dataset (Alghamdi et al. ,
3https://www. kaggle. com/c/asap-aes 4https://www. arabiclearnercorpus. com 5The essays are publicly available, but not the annotations. 2014), comprising essays from college students in Saudi Arabia and including holistic scores ranging from 1 to 6 (best score). Another example is the "AAEE" dataset (Azmi et al. , 2019), which scores essays written by students in grades 7 to 12, based on criteria such as semantic analysis, writing style, and spelling accuracy. Short Answer Datasets Several short answer datasets have been collected for Arabic AES. For instance, Abdeljaber (2021) introduced an accessible dataset with 330 answers for 10 questions. Similarly, Ouahrani and Bennouar (2020) presents the freely available AR-ASAG dataset, which includes 2,133 student answers to 48 questions from a cybercrimes course. Other datasets are not publicly available, such as the "eJaya-NN" dataset (Gaheen et al. , 2020) with 240 answers to one quiz question, and the "Philosophy" dataset (Gomaa, 2014), an expansion of their initial Arabic short answer benchmark set (Gomaa and Fahmy, 2014) containing 600 responses to 50 questions scored from 0 to 10. Shehab et al. (2018) collected 210 short answers from secondary students in a sociology course. Additionally, Nael et al. (2022) translated the English ASAP Short Answer Scoring dataset into Arabic using Google Translate. Several other datasets exist, but they are typically close-domain and small-scale with limited applicability. The absence of publicly available annotations for Arabic essay scoring obstructs the development and validation of AES models tailored to Arabic essays. Most existing Arabic essay datasets are small, often confined to specific domains, very limited in terms of prompts, primarily focus on short answers, and only consider holistic scoring, oversimplifying the multifaceted nature of writing assessment. Furthermore, these datasets are often proprietary or not publicly accessible, hindering reproducibility and collaboration. In response to
this challenge, we extend the Qatari Corpus of Argumentative Writing (QCAW) to present QAES, the first public trait-specific annotations for Arabic AES. Augmenting the QCAW dataset with adequate annotations would unlock its full potential and drive the evolution of AES in Arabic, echoing the advancements seen in English AES. 3 The QCAW Corpus
The Qatari Corpus of Argumentative Writing (QCAW) (Ahmed et al. , 2024) is a publicly available corpus comprising 195 argumentative essays written in both Arabic and English by first-year Arabic native university students as part of a compulsory university-level course. It provides a valuable resource for AES as its lengthy essays offer insights into Arabic linguistic styles and different writing proficiency levels. Furthermore, the essays were collected from a diverse student sample, considering factors such as gender, year of study, and major. In our work, we only consider the Arabic essays. These essays constitute responses to two distinct argumentative writing tasks, originally derived from TOEFL writing prompts: Task 1: ‘Telephones and emails have made communication between people less personal"; and Task 2: ‘Technology has enabled students nowadays to learn more information quickly". Table 2 provides a breakdown of the tasks featured in the QCAW dataset, showing a skew towards Task 1 in terms of both the number of essays and their length. On average, after tokenizing with the NLTK tokenizer and removing punctuation, essays are approximately 489 words long, ranging from 239 to 806 words. The total number of tokens across all essays is 95,342. 4 Constructing QAES
To create QAES, annotators were hired to annotate the Arabic essays of the QCAW dataset across seven traits: Relevance (REL), Organization (ORG), Vocabulary (VOC), Style (STY), Development (DEV), Mechanics (MEC), and Grammar (GRA) in addition to a Holistic (HOL) score of overall quality. The process entails the deployment of trusted assessment guidelines/rubrics, the selec-
tion of specialist annotators, and the delivery of indepth training sessions to ensure consistency and understanding throughout the annotation process. 4. 1 Annotation Guidelines To score student responses, we provided the annotators with the rubrics used in the Core Academic Skills Test (CAST) developed by the Qatar University Testing Center (QUTC). 6 These rubrics were designed to assess students’ ability to write persuasive/argumentative essays, which matches perfectly with the tasks covered in the QCAW corpus. Additionally, CAST’s rubrics underwent extensive measures to ensure that they were aligned with the guidelines established by subject matter experts and reviewed by independent specialists. The rubrics evaluate the seven traits considered in our work. The REL trait is assessed on threelevel scale: 0 (not relevant), 1 (partially relevant), and 2 (completely relevant). The other six traits are evaluated on a five-level scale: 1 (lowest) to 5 (highest). Each level includes a detailed description summarizing the characteristics of the text classified at that level, as shown in Table 3