Advancements in Arabic Grammatical Error Detection and Correction: An Empirical Investigation
Authors: Bashar Alhafni
Go Inoue
Christian Khairallah
Nizar Habash
Venue: EMNLP 2023 Main (2023)
Keywords: Grammatical Error Correction
Grammatical Error Detection
Arabic

Abstract: Grammatical error correction (GEC) is a well-explored problem in English with many existing models and datasets. However, research on GEC in morphologically rich languages has been limited due to challenges such as data scarcity and language complexity.
In this paper, we present the first results on Arabic GEC using two newly developed Transformer-based pretrained sequence-to-sequence models. We also define the task of multi-class Arabic grammatical error detection (GED) and present the first results on multi-class Arabic GED. We show that using GED information as auxiliary input in GEC models improves GEC performance across three datasets spanning different genres. Moreover, we also investigate the use of contextual morphological preprocessing in aiding GEC systems.
Our models achieve SOTA results on two Arabic GEC shared task datasets and establish a strong benchmark on a recently created dataset. We make our code, data, and pretrained models publicly available.

Advancements in Arabic Grammatical Error Detection and Correction: An Empirical Investigation
Bashar Alhafni, Go Inoue', Christian Khairallah, Nizar Habash
Computational Approaches to Modeling Language Lab
New York University Abu Dhabi
'Mohamed bin Zayed University of Artificial Intelligence Calhafni,christian christian wttpssiiiianaaaalaaaaaaaaaa khairallah, nirar. habash habashJ@nyu Jnnu.. edu go. inoue@mbzuai. ac.
Abstract
attention, largely due to the lack of datasets and standardized error type annotations. Specifically for Arabic, the focus on GEC started with the QALB 2014 (Mohit et al., 2014) and QALB-2015 (Rozovskaya et al., 2015) shared tasks; however, recent sequence-to-sqquenee (Seq2Seq) modeling advances have not been explored much in Arabic GEC. Moreover, multi-class Arabic GED has not been investigated due to the lack of error type in- formation in Arabic GEC datasets. In this paper, we try to address these challenges. Our main con- tributions are as follows:
Grammatical error correction (GEC) well- explored problem in English with many exist- ing models and datascts. However, research GEC in morphologically rich languages has been limited due to challenges such as data scarcity and language complexity. this paper, we present the first results Arabic GEC using two newly developed Transformer-based pre- trained sequence-to-sete toseuunnce models. We also define the task of multi-class Arabic grammati- cal error detection (GED) and present the first results multi-class Arabic GED, We show that using GED information as an auxiliary in- put in GEC models improves GEC performance across three datasets spanning different genres. Moreover, we also investigate the use of con- textual morphological preprocessing aiding GEC systems. Our models achieve SOTA re- sults on two Arabic GEC shared task datasets and establish strong benchmark created dataset. We make our code, data, and pretrained models publicly available.'
We are the first to benchmark newly developed pretrained Seq2Seq models on Arabic GEC.
2.
We tackle the task of Arabic GED by introduc- ing word-level GED labels for existing Arabic GEC datasets, and present the first results on multi-class Arabic GED.
recently
We systematically show that using GED infor- mation in GEC models improves performance across GEC datasets in different domains.
1 Introduction
4. We leverage contextual morphological prepro- cessing in improving GEC performance.
English grammatical error correction (GEC) has witnessed significant progress in recent years due to increased research efforts and the organization of several shared tasks (Ng et al., 2013, 2014; Bryant et al., 2019). Most state-of-the-art (SOTA) GEC systems borrow modeling ideas from neural ma- chine translation (MT) to translate from erroneous to corrected texts. In contrast, grammatical error detection (GED), which focuses locating and identifying errors in text, is usually treated se- quence labeling task. Both tasks have evident ped- agogical benefits to native (L1) and foreign (L22 language teachers and students. Also, modeling GED information explicitly within GEC systems yields better results in English (Yuan et al., 2021). )). When it comes to morphologicalyy rich lan- guages, GEC and GED have not received as much Chttps://githher waap:ttttw.com/MMMeLLaaaaa/rreee
5. We achieve SOTA results on two (L1 and L2) previously published Arabic GEC datasets. We also establish a strong benchmark a recently created L1 Arabic GEC dataset.
2 Related Work
GEC Approaches Early efforts focused on build- ing feature- machine learning (ML) classifiers to fix common error types (Chodorow et al., 2007; Tetreault and Chodorow, 2008; Dahlmeier and Ng, 201 I; Kochmar et al., 2012; Rozovskaya and Roth, 2013; Farra al., 2014). Such models required fea- ture engineering and lacked the ability correct all error types simultaneously.
Reformulating GEC as a monolingual MT task alleviated these issues, first with statistical MT ap-

---

proaches (Felice et al., 2014; Junczys-Dowmin and Grundkiewiez. 2014, 2016) and then neural MT approaches (Yuan and Briscoe, 2016; Xie et al., 2016; Junczys-Dowmunt et al., 2018; Watson et al., 2018), with Transformer- based models being the most dominant (Yuan et al., 2019; Zhao et al.,
Dataset
Split
Words
Err.
Type
Domain
Train-LI1
1M
30%
L1]
Comments
QALB-2014
Dev-L1
54K
31%
L1
Comments
Test-Li
51K
32%
L1
Comments
Train-L2
43K
30%
L2
Essays
Dev-1.2
25K
29%
1.2
Essays
QALB-2015
Test-1.2
23K
29%
1.2
Essays
Test-LI
49K
27%
LI
Comments
2019; Grundkiewicz et al., 2019; Katsumata and Komachi, 2020; Yuan and Bryant, 2021).
Train-LI
25K
24%
L1
Essays
ZAEBUC
Dev-L1
5K
25%
L1
Essays
Test-L1
5K
26%
L.1
Essays
More recently, edit-based models have been pro- posed to solve GEC (Awasthi et al., 2019; Malmi et al., 2019; Stahlberg and Kumar, 2020; Mallinson et al., 2020; Omelianchuk et al., 2020; Straka et al., 2021; Mallinson al., 2022; Mesham et al., 2023), While Seq2Seq models generate corrections to erro- neous input, edit-based models generate a sequence of corrective edit operations. Edit-based models add explainability to GEC and improve inference time efficiency. However, they generally require human engineering to define the size and scope of the edit operations (Bryant et al., 2023).
Table 1: Corpus statistics of Arabic GEC datasets.
model and achieved SOTA results on the LI Arabic GEC data used in the QALB-2014 and 2015 shared tasks. Recently, vanilla Transformers (Vaswani et al., 2017) were explored for synthetic data gen- cration to improve LI Arabic GEC and were tested on the L1 data of the QALB-2014 4 and 2015 shared tasks (Solyman et al., 2021, 2022, 2023). To the best of our knowledge, the last QALB-2015 I.2 reported results were presented in the shared task itself. We compare systems against the best previously developed models whenever feasible. A number of researchers reported on Arabic bi- nary GED. Habash and Roth (2011) used feature- engineered SVM classifiers to detect Arabic hand- writing recognition errors. Alkhatib et al. (2020) and Madi and Al- Ahaliaa (2020) used LSTM-based classifiers. None of them used any of the publicly available GEC datasets mentioned above train and test their systems. In our work, we explore multi-class GED by obtaining error type annota- tions from ARETA (Belkebir and Habash, 2021), an automatic error type annotation tool for MSA. To our knowledge, we are the first to report on Arabic multi-class GED. We report on publicly available data to enable future comparisons.
GED Approaches Rei and Yannakoudakis (2016) presented the first GED results using a neural approach framing GED as a binary (cor- rect/incorrect) sequence tagging problem. Others used pretrained language models (PLMs) such as BERT (Devlin et al., 2019), ELECTRA (Clark et al., 2020), and XLNeT (Yang et al., 2019) to improve binary GED (Bell al., 2019; Kaneko and Komachi, 2019; Yuan et al., 2021; Rothe et al., 2021). Zhao al. (2019) and Yuan et al. (2019) demonstrated that combining GED and GEC yields improved results: they used multi-task learning to add token-level and sentence-level GED as auxil- iary tasks when training for GEC. Similarly, Yuan et al. (2021) showed that binary and multi-class GED improves GEC.
Arabic GEC and GED The Qatar Arabic Lan- guage Bank (QALB) project (Zaghouani et al., 2014, 2015) organized the first Arabic GEC shared tasks: QALB-2014 (L.) (Mohit al., 2014) and QALB-2015 (L1 and L2) (Rozovskaya et al., 2015). Recently, Habash and Palfreyman (2022) created the ZAEBUC corpus, a new L1 Arabic GEC corpus of essays written by university students. We report on all of these sets.
3 Background
3.1 Arabic Linguistic Facts
Modern Standard Arabic (MSA) is the official form of Arabic primarily used in education and media across the Arab world. MSA coexists in a diglos- sic (Ferguson, 1959) relationship with local Arabic dialects that are used for daily interactions. When native speakers write in MSA, there is frequent code-mixing with the dialects in terms of phonolog- ical, morphological, and lexical choices (Habash et al., 2008). In this paper, we focus on MSA GEC. While its orthography is standardized, written Ara- bic suffers many orthographic inconsistencies even
Arabic GEC modeling efforts ranged from feature-based ML classifiers statistical MT models (Rozovskaya et al., 2014; Bougares and Bouamor, 2015; Nawar, 2015). Watson et al. (2018) introduced the first character-level Seq2Seq

---

in professionally written news articles (Buckwalter, 2004; Habash et al., 2012). For example, hamzated Alifs (, A are commonly confused with the un-hamzated letter (1 A), and the word-final letters و and و are often used interchangeably. These errors affect 11% of all words (4.5 errors per sen- tence) in the Penn Arabic Treebank (Habash, 2010). Additionally, the use of punctuation in Arabic is very inconsistent, and omitting punctuation marks is very firquuent (Awad, 2013; Zaghouani and Awad, 2016). Punctuation errors constitute ~40% of er- rors in the QALB-2014 GEC shared task. This is ten times higher than punctuation errors found in the English data used in the CoNLL-2013 GEC shared task (Ng et al., 2013). Arabic has a large vo- cabulary size resulting from its rich morphology. which inflects for gender, number, person, case, state, mood, voice, and aspect, and cliticizes nu- merous particles and pronouns. Arabic's diglossia, orthographic inconsistencies, and morphological richness pose major challenges to GEC models.
presents automatic error type distributions over the training portions of the three datasets. Ortho- graphic errors are more common in the LI datasets (QALB-22) 2-14 and ZAEBUC) compared to the L. dataset (QALB-2015)) In contrast, morphological, syntactic, and semantic errors are more common in QALB-201 Punctuation errors are more common in QALB-2014 and QALB-2015 compared with ZAEBUC.
3.3 Metries for GEC and GED
GEC systems are most commonly evaluated us- ing reference-based metrics such as the Max.Match (M²) scorer (Dahlmeier and Ng, 2012), ERRANT (Bryant et al., 2017), and GLEU (Napoles et al., 2015), among other reference-based and reference- less metrics (Felice and Briscoe, 2015; Napoles et al., 2016; Asano et al., 2017; Choshen et al., 2020; Maeda et al., 2022). In this work, we use the M scorer because it is language agnostic and was the main evaluation metric used in previous work on Arabic GEC. The M scorer compares hypothesis edits made by GEC system against notated reference edits and calculates the precision (P), recall (R), and Fu.5- In terms of GED, we fol- low previous work (Bell et al., 2019; Kaneko and Komachi, 2019; Yuan et al., 2021) and use macro precision (P), recall (R), and Fo.s for evaluation. We also report accuracy.
3.2 Arabic GEC Data
We report three publicly available Arabic GEC datasets. The first two come from the QALB- 2014 (Mohit et al., 2014) and QALB-2015 (Ro- zovskaya et al., 2015) shared tasks. The third is the newly created ZAEBUC dataset (Habash and Palfreyman, 2022). None of them were manually annotated for specific error types. Table 1 presents summary of the dataset statistics. Detailed dataset statisties are presented in Appendix B.
Arabic Grammatical Error Detection
Most of the work on GED has focused on En- glish ($2), where error type annotations are provided manually (Yannakoudakis et al., 2011; Dahlmeier et al., 2013) or obtained automatically using an eтror type annotation tool such as ER- RANT (Bryant et al., 2017). However, when it comes to morphologically rich languages such as Arabic, GED remains a challenge. This is largely due to the lack of manually annotated data and stan- dardized error type frameworks. In this work, we treat GED mutli-class sequence labeling task. We present method to automatically obtain error type annotations by extracting edits from parallel erroneous and corrected sentences and then passing them to an Arabic error type annotation tool. To the best of our knowledge, this is the first work that explores multi-class GED in Arabic.
QALB-2014 consists of native/L1 user com- ments from the Aljazeera news website, whereas QALB-2015 consists of essays written by Arabic L2 learners with various levels of proficiency. Both datasets have publicly available training (Train), development (Dev), and test (Test) splits. The ZAEBUC dataset comprises essays written by na- tive Arabic speakers, which were manually cor- rected and annotated for writing proficiency using the Common European Framework of Reference (CEFR) (Council of Europe, 2001). Since the ZAE- BUC dataset did not have standard splits, we ran- domly split it into Train (70%), Dev (15%), and Test (15%), while keeping a balanced distribution of CEFR levels.
The three sets vary in a number of dimensions: domain, level, number of words, percentage of er- roneous words, and types of errors. Appendix C
4.1 Edit Extraction
Before automatically labeling each erroncous sen- tence token, we need to align the erroneous and
"Arabic HSB translitcration (Habash et al., 2007).

---

13
12
11
10
9
8
5
4
3
2
1
Erroncous
4:
4
jbl
L
بحم ااامممي ا
الرااس
دسال
إسامممم
3
Alignments
inyyjAtyh
albyh
motiA
f
bilkkmh
AlAjimAgy
AfrwASi
wadjji
AstxdAm
14:
13
12 سلية
10
6
5
Corrected
2 4
jbi
مي الاوناو Y
املل
الم
سلcدددl
+
y
wAyjAbyb altyhh
A0Ar
fibA
ALAjemAcy
AltwASi
wsdjt
AstxdAm
Ad
M
R
Lev.
K
K
R
K
$
ARETA
R
K
D
Ours
R
K
D
M
43-Class
PM
O1:OT
Delete
OH
Merge
PM
OT
OH
OHf
Split
13-Class
O
Delete
Merge
0
Split
2-Class
E
E
E
E
E
Figure 1: An example showing the differences between the alignments of the M2 scorer, a standard Levenshtein distance, ARETA, and our proposed algorithm. The edit operations keep (K), replace (R), insert (I), delete (D), merge (M), and split (S). Dotted lines between the erroneous and corrected sentences represent gold alignment. The last three rows present different granularities of ARETA error types based on our aligmment. The sentence in the figure can be translated "Social media must be used wisely, it has both negative and positive effects".
corrected sentence pairs to locate the positions of all edits so as to map errors to corrections. This step is usually referred to as edit extraction in GEC literature (Bryant et al., 2017).
QALB-2014
QALB-2015
P
R
AER
P1
R
AER L
M
92.5
87.1
0.10
90.8
83.3
0.13
Lev.
86.8
84.3
0.14
84.5
84.2
0.16
ARETA
84.3
82.9
0.16
84.1
84.7
0.16
We first obtain character-level alignment be- tween the erroneous and corrected sentence pair by computing the weighted Levenshtein edit dis- tance (Levenshtein, 1966) for each pair of tokens in the two sentences. The output of this alignment a sequence token-level edit operations represent- ing the minimum number of insertions, deletions, and replacements needed to transform one token into another. Each of these operations involves one token at most belonging to either sentence. How- ever, some errors may involve more than single edit operation. To capture multi-token edits, extend the alignment cover merges and splits by implementing an iterative algorithm that greed- ily merges or splits adjacent tokens such that the overall cumulative edit distance is minimized.
Ours
99.6
99.7
98.0
0.02
Table Evaluation of different alignment algorithms.
et al., 2013) which defines seven error classes cov- cring orthography (O), morphology (M), syntax (X), semantics (S), punctuation (P), merges, and splits. The error classes are further differentiated into 32 error tags that can be assigned individually or in combination.
ARETA comes with its own alignment algorithm that extracts edits, however, it does not handle many-to-one and many-to-many edit operations (Belkebir and Habash, 2021). We replace ARETA's internal alignment algorithm with ours to increase the coverage error typing. Using our edit ex- traction algorithm with ARETA enables us to au- tomatically annotate single-token and multi-token edits with various error types. Appendix C presents the error types obtained from ARETA by using our alignment over the three GEC datasets use.
4.2 Error Type Annotation
Next, we pass the extracted edits to an automatic annotation tool to label them with specific error types. We use ARETA, an automatic error type an- notation tool for MSA (Belkebir and Habash, 2021). Internally, ARETA is built using combination of rule-based components and an Arabic morphologi- cal analyzer (Taji al., 2018; Obeid et al., 2020). It uses the error taxonomy the Arabic Learner Corpus (ALC) (Alfaifi and Atwell, 2012; Alfaifi
To demonstrate the effectiveness of our align- ment algorithm, we compare our algorithm the alignments generated by the M scorer, a standard Levenshtein edit distance, and ARETA. Table 2 presents the evaluation results of the alignment al- gorithms against the manual gold alignments of the

---

QALB-2014 and QALB-2015 Dev sets in terms of precision (P), recall (R), and alignment error rate (AER) (Mihalcea and Pedersen, 2003; Och and Ney, 2003). Results show that our alignment algorithm is superior across all metrics.
maximum likelihood estimator over the training data: P(cyju, e;); where w; and w;-1 are the erroneous word (or phrases in case of a merge error) and its bigram context, e, is the error type of Wi and Gj is the correction of Wi- During inference, we pick the correction that maximizes the MLE probability. If the bigram context (w; and w;-1) was not observed during training, we backoff to unigram. If the erroneous input word was not observed training, we pass it to the output.
Figure presents an example of the different alignments generated by the algorithms we evalu- ated. The M scorer's alignment over-clusters mul- tiple edits into a single edit (words 6-13). This is not ideal, particularly because the M scorer does not count partial matches during the evaluation, which leads to underestimating the models' per- formances (Felice and Briscoe, 2015). A standard Levenshtein alignment does not handle merges cor- rectly, e, words 8 and 9 in the erroneous sentence are aligned to words 9 and 10 in the corrected ver- sion. Among the drawbacks of ARETA's alignment is that it does not handle merges, e.g., erroneous words 8 and 9 are aligned with corrected words and 10, respectively.
ChatGPT Given the rising interest in using large language models (LLMs) for variety of NLP tasks, we benchmark ChatGPT (GPT-3.5) on the task of Arabic GEC. We follow the setup presented by Fang al. (2023) on English GEC. To the best of our knowledge, we are the first present Chat- GPT results on Arabic GEC. The experimental setup along with the used prompts are presented in Appendix A.
Seq2Seq with GED Models We experiment with two newly developed pretrained Arabic Transformerbbased Seq2Seq models: AraBART (Kamal Eddine et al., 2022) (pretrained on 24GB of MSA data mostly in the news domain), and AraT5
5 Arabic Grammatical Error Correction
Recently developed GEC models rely on Transformer-based architectures, from standard Seq2Seq models to edit-based systems built on top of Transformer encoders. Given Arabic's morphological richness and the relatively small size of available data, we explore different GEC models, from morphological analyzers and rule-based systems pretrained Seq2Seq models. Primarily, we are interested exploring modeling approaches to address the following two questions:
(Nagoudi et al., 2022) (pretrained on 256GB of both MSA and Twitter data).
We extend the Seq2Seq models porate token-level GED information during train- ing and inference. Specifically, we feed predicted GED tags as auxiliary input to the Seq2Seq mod- cls. We add an embedding layer to the encoders of AraBART and AraT5 right after their correspond- ing token embedding layers, allowing us to learn representations for the auxiliary GED input. The GED embeddings have the same dimensions the positional and token embeddings, so all three em- beddings can be summed before they are passed the multi-head attention layers in the encoders.
incor-
RQ1: Does morphological preprocessing im- prove GEC in Arabic?
RQ2: Does modeling GED explicitly improve GEC in Arabic?
Morphological Disambiguation (Morph) We use the current SOTA MSA morphological ana- lyzer and disambiguator from CAMeL Tools (In- oue et al., 2022; Obeid al., 2020). Given an input sentence, the analyzer generates a set of potential analyses for each word and the disambiguator se- lects the optimal analysis in context. The analyses include minimal spelling corrections for common errors, diacritizations, POS tags, and lemmas. We use the dediacritized spellings as the corrections.
Our approach is similar to what was done by Yuan et al. (2021), but it is much simpler as it re- duces the model's size and complexity by not intro- ducing an additional encoder to process GED input. Since the training data we use is relatively small, not drastically increasing the size of AraBART and AraT5 becomes important not hinder training.
Experiments
Maximum Likelihood Estimation (MLE) We exploit our alignment algorithm to build a simple lookup model to map erroncous words to their cor- rections. We implement this model as a bigram
6.1 Arabic Grammatical Error Detection
We build word-level GED classifiers using Transformer-basedd PLMs. From the many avail-

---

43-Class
13-
13-Class
2-Class
P
Fas
Acc.
P
Fas
Acc.
Fas
Acc.
Dev-LI
56.7
48.4
53.3
94.1
69.0
58.7
65.3
94.7
95.8
92.7
95.1
96.1
QALB-2014
Test-LI
55.0
45.5
50.6
93.6
58.1
54.2
56.8
94.1
95.4
91.5
94.5
95.5
Dev-1.2
39.0
35.0
36,9
84.5
55.1
47.3
51.7
85.3
87.0
80.4
85.2
88.9
QALB-2015
Test-L.
51.8
45.3
49.4
94.9
66.5
56.2
60.7
95.6
96.2
93.9
95.7
96.7
Test-1.2
37.0
35.4
35.8
85.5
52.8
48.6
51.0
86.5
88.6
81.3
86.6
89.9
Dev-Li
50.9
43.7
47.5
92.6
57.1
52.9
55.7
93.3
95.7
92.8
95.1
95.5
ZAEBUC
Test-L1
54.9
43.3
49.8
91.9
69.2
56.6
62.4
92.6
95.5
92.5
94.8
95.2
Table 3: GED results on the Dev and Test in terms of macro precision, recall, Fas, and accuracy.
able Arabic monolingual BERT models (Antoun al., 2020; Abdul-Mageed al., 2021: Lan et al..
numbers of errors despite the number of classes, but the verbose systems struggle with detecting the specific class labels.
2020; Safaya al., 2020; Abdelali al., 2021), we chose to use CAMeLBERT MSA (Inoue et al., 2021), as was pretrained on the largest MSA dataset to date.
6.2 Arabic Grammatical Error Correction
We explore different variants of the above- mentioned Seq2Seq models. For each model, we study the effects of applying morphological prepro- cessing (+Morph), providing GED tags as auxil- iary imput (+GED), or both (+Morph+GED), Ap- plying morphological preprocessing simply means correcting the erroncous input using the morpholog- ical disambiguator before training and inference. To increase the robustness of the models that take GED tags as auxiliary input, we use predicted (not gold) GED tags when we train the GEC sys- tems. For each dataset, we run its respective GED model on the same training data was trained on and we pick the predictions of the worst check- point. During inference, we resolve merge and delete errors before feeding erroneous sentences to the model. This experimental setup yields the best performance across all GEC models.
In our GED modeling experiments, we project multi-token error type annotations to single-token labels. In the case of a Merge error (many-to-one), we label the first token as Merge-B (Merge begin- ning) and all subsequent tokens as Merge-l (Merge inside). For all other multi-token error types, we repeat the same label for each token. We further label all deletion errors with a single Delete tag. To reduce the output space of the error tags, we only model the 14 most frequent error combinations (ap- pearing more than 100 times). We ignore unknown errors when we compute the loss during training; however, we penalize the models for missing them in the evaluation. Since the majority of insertion er- rors are related to missing punctuation marks rather than missing words (see Appendix C), and due to inconsistent punctuation error annotations (Mohit al., 2014), exclude insertion errors from GED modeling and evaluation. We leave the inves- tigation of insertion errors future work. The full GED output space we model consists of 43 error tags (43-Class).
To ensure fair comparison to previous work on Arabic GEC, we follow the same constraints that were introduced in the QALB-2014 and QALB- 2015 shared tasks: systems tested on QALB-2014 are only allowed to use the QALB-200 training data, whereas systems tested on QALB-2015 are allowed to use the QALB-2014 and QALB-201 training data. For ZAEBUC, train our systems on the combinations of the three training datasets. We report our results in terms of precision (P), re- call (R), F,, and Fo.s- F, was the official metric used in the QALB-2014 and QALB-2015 shared tasks. However, we follow the most recent work on GEC and use Fo.5 (weighing precision twice much as recall) as main evaluation metric.
We take advantage of the modularity of the ARETA error tags conduct multi-class GED ex- periments, reducing the 43 error tags to their cor- responding 13 main error categories as well as to a binary space (correct/incorrect). The statistics of the error tags we model across all datasets are in Appendix D. Figure/ shows an example of er- ror types at different granularity levels. Table 3 presents the GED granularity results. Unsurpris- ingly, all numbers go up when we model fewer error types. However, modeling more error types does not significantly worsen the performance in terms of error detection accuracy. It seems that all systems are capable of detecting comparable
We use Hugging Face's Transformers (Wolf et al., 2019) to build our GED and GEC models. The hyperparameters we used are detailed in Ap- pendix A.

---

QALB-2014
QALB-2015
ZAEBUC
Avg.
Fas
R
Fi
Fus
Fi
FSs
Fo.
B&B (2015)
56.7
34.8
45.1
50.4
W+ (2018)
80.0
62.5
70.2
75.8
Morph
76.4
30.4
43.5
58.7
9.4
28.2
78.0
36,9
50.1
50.2
MLE
89.2
41.3
56.5
72.4
73.7
20.1
48.0
90.1
55.6
68.8
80.1
66.9
+Morph
88.5
44.9
59.6
74.1
22.0
33.2
48.0
89.1
61.8
73.0
68.0
ChatGPT
67.7
60.6
63.9
66.1
54.9
36.9
44.1
50.0
68.1
52.1
59.1
64.2
60.1
AraT5
82.5
66.3
73.5
78.6
69.3
39.4
50.2
60.2
84.1
67.4
74.8
80.1
73.0
+Morph
83.1
65.8
73.4
78.9
40.6
51.3
60.9
85.0
71.3
77.5
81.8
73.9
+GRDD4
82.6
67.1
74.1
79.0
69.5
41.9
52.3
61.4
85.7
66.7
75.0
81.0
73.8
+Morph +GED*
83.1
67.9
74.7
79.6
68.4
41.5
51.7
60.6
85.2
71.2
77.6
82.0
74.0
AraBART
83.2
64.9
72.9
78.7
68.6
42.6
52.6
61.2
87.3
70.6
78.1
83,4
74.4
+Morph
82.4
67.2
74.0
78.8
68.5
44.3
53.8
61.7
87.2
71.6
78.7
83.6
74.7
+GED
83.3
65.9
73.6
79.1
68.2
45.3
54.4
61.9
87.2
72.9
79.4
83.9
75.0
+Morph +GED
83.4
66.3
73.9
79.3
68.2
46.6
55.4
62.4
87.3
73.6
79.9
84.2
75.3
Table 4: GEC results on the Dev sets of QALB-2014, QALB-2015, and ZAEBUC. B&B (2015) and W+ (2018)
refer to Bougares and Bouamor (2015) and Watson et al. (2018), respectively. The best results are bold.
DALB-20144
QAL D-1552
ZAEBUC
Avg.
F₁
Fas
P
F1
Fo.5
P
R
Fi
Fo.
Fas
43-Class
[Oracle]
85.5
73.3
79.0
82.8
73.9
57.2
64.5
69.8
89.8
82.0
85.7
88.1
80.2
13-Class
[Oracle]
85.4
73.2
78.8
82.6
73.5
55.9
63.5
69.2
89.4
82.2
85.7
87.9
79.9
2-C 2lCsass
[Oracle]
84.2
72.1
77.7
81.4
71.6
54.5
61.9
67.4
86.6
80.0
83.2
85.2
78.0
Class
"3.3""
"73.9
79:3"
68.2
"46.6"
"55.4"
62.4""
"873"
73.6
779,9
"84.2
75:3"
13-Class
83.9
65.7
73.7
79.5
68.0
46.6
55.3
62.3
87.6
73.9
80.2
84.5
75,4
2-Class
82.5
67.3
74.2
79.0
68.3
45.0
54.3
61.9
86.0
72.3
78.6
82.9
74.6
Table GED granularity results when used within the best GEC system (AraBART+Mooph--GE))) on the Dev sets of DALB-20144 QALB-2015, and ZAEBUC. The best results are in bold.
7 Results
Does GED help Arabic GEC? We start off by using the most fine-grained GED model (43- Class) to exploit the full effect of the ARETA GED tags and to guide our choice between AraBART and AraT5. Using GED as an auxiliary input in both AraT5 and AraBART improves the results across all three Dev sets, with AraBART+GED demonstrating superior performance compared the other models, on average. Applying mor- phological preprocessing as well as using GED as an auxiliary input yields the best performance across the three Dev sets, except for QALB- 2015 in the case of AraT5+MMrph+GED. Overall, AraBBRRT+Mopph-GED is the best performer on average in terms of Fo.s. The improvements using GED with GEC systems are mostly due to call. An comparison between AraBART and the AraBART+Morph+GEN model (Appendix E) shows improved performance on the majority of the error types.
Table 4 presents the results on the Dev sets.
Baselines The Morph system which did not use any waining data constitutes solid baseline for mostly addressing the noise in Arabic spelling. The MLE system claims the highest precision of all compared systems, but suffers from low recall as expected. ChatGPT has the highest recall among the baselines, but with lower precision. A sample of 100 ChatGPT mismatches reveals that 37% are due to mostly acceptable punctuation choices and 25% are valid paraphrases or re-orderings; however, 38% are grammatically or lexically incorrect.
Seq2Seq Models AraT5 and AraBART outper- form previous work on QALB-2011 and QALB- 2015, with AraBART being the better model on average.
Does morphological preprocessing improve Ara- bic GEC? Across all models (MLE, AraT5, and AraBART), training and testing on morphologically preprocessed text improves the performance, ex- cept for MLE+Morph on DALB-2011. where there is no change in Fo.5-
To study the effect of GED granularity on GEC, train two additional AraBART+Morph+GER models with 13-Class and 2-Class GED tags. The results in Table show that 13-Class GED was best in QALB-22 2012 4 and ZAEBUC, whereas 43-Class

---

QALB-2014
QALB-201S-LI
QALB-2015-L2
ZAEBUC
Avg.
R
Fas
R
Fe.
R
F
Fas
P
R
F1
Fas
Fa.
B&B (2015)
54.1
33.3
41.2
48.1
W+ (2018)
70.4
73.2
S+ (2022)
79.1 65.8
71.8
76.0
78.4
70.4
74.2
76.6
AraBART
84.0 64.7
73.1
79.3
82.0
71.7
76.5
79.7
69.6
43.5
53.5
62.1
86.0
71.6
78.2
82.7
75.9
+Morph
83.3 67.4
74.5
79.5
81.7
73.0
77.1
79.8
68.7
43.6
53.3
61.6
85.3
71.8
78.0
82.3
75.8
+GED
84,2 65.4
73.6
79.6
81.2
72.4
76.5
79.3
69.0
45.4
54.7
62.5
85.4
72.6
78.5
82.5
76.0
+Morph+GED*
83.9 65.7
73.7
79.5
82.6
72.1
77.0
80.3
67.6
45.2
54.2
61.5
85.4
73.7
79.1
82.7
76.0
+GED 15
65.0
73.3
79.4
81.3
72.7
76.8
79.5
69.3
44.9
34.3
62.5
85.9
73.4
79.2
83.1
76.1
"Morph+GED"
83,9 65.3
73.4
79.4
81.1
73.4
77.1
79.5
68.2
44.8
54.1
61.8
85.2
73.7
79.0
82.6
75.8
+GED=
83.8
64.5
72.9
79.1
81.4
71.5
76.2
79.2
69.1
44.9
$4.4
62.4
85.7
71.5
78.0
82.4
75.8
GEDD
83.0 67.0
74.1
79.2
81.3
73.8
77.4
79.7
68.1
45.3
54.4
61.9
85.7
72.4
78.5
82.7
75.9
Table 6: GED granularity results when used within GEC on the Test sets of QALB-2014, QALB-2015 and ZAEBUC. B&B (2015), W++ (2018), and S+ (2022) refer to Bougares and Bouamor (2015), Watson et al. (2018). and Solyman et al. (2022), respectively. The best results in bold.
QALB-2014
QALB- 2-1--001-LL1
QALB-200 2-15-201--22
ZAEBUC
Avg.
R
Fus
R
F65
R
Fus
P
R
F₁
Fas
Fa.
AraBART
89.5
77.3
83.0
86.8
90.1
81.4
85.5
88.2
71.8
40.7
52.0
62.3
89.5
76.9
82.7
86.6
81.0
+Morph
88.4
78.9
83.4
86.3
89.9
83.1
86.4
88.5
70.2
41.8
52.4
61.8
88.4
76.3
81.9
85.7
80.6
+GED*
89.7
78.9
84.0
87.3
89.8
81.8
85.6
88.1
70.7
43,6
53,9
62.9
89.2
77.0
82.7
86.5
81.2
+Morph+GED**
88.8
80.1
84.2
86.9
90.0
83.8
86.8
88.7
69.0
43.6
53.4
61.8
88.7
78.4
83.2
86.4
80.9
*GED"
89.8
78.9
84.0
87.3
89.8
82.2
85.8
88.2
71.0
42.8
53.4
62.7
89.9
77.8
83.4
87.2
81.4
+Morph+GED"
88.6
80.0
84.1
86.7
89.5
84,1
86.7
88.3
68.9
43.5
53.3
61.7
88.9
78.4
83.3
86.5
80.8
+GED"
77.6
83.0
86.7
89.4
81.8
85.5
87.8
70.6
42.4
53.0
62.3
89.0
77.0
82.6
86.3
80.8
+Morph+GED*
87.8
79.8
83.6
86.1
89.9
83.0
86.3 88.5
69.5
43.5
53.5
62.1
89.2
77.7
83.1
86.6
80.8
Table 7: No punctuation GED granularity results when used within GEC on the Test sets of QALB-2014, QALB- 2015, and ZAEBUC. The best results are in bold.
GED was best in QALB-2015 in terms of Fo.s. However, in terms of precision and recall, GED models with different granularity behave differently across the three Dev sets. On average, using any GED granularity improves over AraBART, with 13-Class GED yielding the best results, although it is only 0.1 higher than 43-Class GED in terms of Fu.s- For completeness, we further estimate an oracle upper bound by using gold GED tags with different granularity. The results (in Table 5) show that using GED with different granularity improves the results considerably. This indicates that GED is providing the GEC system with additional infor- mation; however, the main bottleneck the GED prediction reliability as opposed to GED granular- ity. Improving GED predictions will most likely lead to better GEC results.
Morph, GED, or both improves the results over AraBART, except for 2-Class GED, AraBART+43- Class GED is the best performer (0.3 increase in Fo.s, although not statistically significant)." It is worth noting that AraBART+Morph achieves the highest recall on QALB-2014 (2.7 increase over AraBART and statistically significant at p < 0.05). For QALB-2015-LI, using GED by itself across all granularity did not improve over AraBART, but when combined with Morph, the 43-Class GED model yields the best performance in Fo.s (0.6 in- crease statistically significant at p < 0.05). When comes to QALB-2015-L2, Morph does not help. but using GED alone improves the results over AraBART, with 43-Class and 13-Class GED being the best (0.4 increase). Lastly, in ZAEBUC, Morph does not help, but using 13-Class GED by itself improves over AraBART (0.4 increase). Overall, all the improvements we observe are attributed to recall, which is consistent with the Dev results. Following the QALB-2015 shared task (Ro- zovskaya et al., 2015) reporting of no-punctuation
Since the best- best-performing models the three Dev sets benefit from different GED granularity when used with Morph, we present the results on the Test sets using all dif- ferent GED granularity models. The results of using AraBART and its variants on the Test sets are presented in Table 6. On QALB-201 2014, using
Test Results
"Statistical significance was done using two-sided ap- proximate randomization test.

---

results due to observed inconsistencies in the ref- erences (Mohit et al., 2014), we present results on the Test sets without punctuation errors in Table The results are consistent with those with punc- tuation, indicating that GED and morphological preprocessing yield improvements compared to us- ing AraBART by itself across all Test sets. The score increase among all reported metrics when removing punctuation, specifically in the 1.1 data, indicates that punctuation presents challenge for GEC models and needs further investigation both in terms of data creation and modeling approaches.
P
R
+Morph
0.4
0.8
+GED43
0.2
0.7
+GED
-0.2
0.7
+GED2
0.3
0.5
+GED"
0.2
0.6
+Morph+GED*
-0.5
1.3
+Morph+GEDis
-0.8
1.4
+Morph+GED2
-0.8
1.8
+Morph+GED"
-0.7 1.5
Table 8: Average absolute changes in precision (P) and recall (R) when introducing Morph, GED, or both to AraBART and its variants on the Test sets. GED" indi- cates the average absolute changes of all models using GED. Bolding highlights the best performance across Morph, GED* and Morph+GED*.
Analyzing the Test Results Table 8 presents the average absolute changes in precision and recall over the Test sets when introducing Morph, GED, or both. Adding Morph alone GED alone im- proves recall (up to 0.8 in the case of Morph) and slightly hurts precision. When using both Morph and GED, we observe significant improvements in recall with an average of 1.5 but with higher drops of precision with an average of 0.7.
the limitation of excluding insertion errors when modeling GED. Furthermore, our GEC systems could benefit from employing a copying mecha- nism (Zhao et al., 2019; Yuan et al., 2019), par- ticularly because of the limited training data avail- able in Arabic GEC. Moreover, the dataset sizes of
8 Conclusion and Future Work
and ZAEBUC are too small to allow to test for statistical significance.
We presented the first results Arabic GEC using Transformer-based pretrained Seq2Seq models. We also presented the first results on multi-class Arabic GED. We showed that using GED information as an auxiliary input in GEC models improves GEC performance across three datasets. Further, we investigated the use of contextual morphological preprocessing in aiding GEC systems. Our models achieve SOTA results two Arabic GEC shared tasks datasets and establish strong benchmark on recently created dataset.
Acknowledgements
We thank Ted Briscoe for helpful discussions and constructive feedback. We acknowledge the sup- port of the High Performance Computing Center at New York University Abu Dhabi. Finally, we wish to thank the anonymous reviewers at EMNLP 2023 for their feedback.
In future work, we plan to explore other GED and GEC modeling approaches, including the use of syntactic models (Li et al., 2022; Zhang et al., 2022). We plan to work more insertions, punctu- ation, and infrequent eиror combinations. We also plan to work GEC for Arabic dialects, i.e., the conventional orthography of dialectal Arabic nor- malization (Habash et al., 2018; Eskander al., 2013; Eryani et al., 2020).
References
Ahmed Abdelali, Sabit Hassan, Hamdy Mubarak, Ka- reem Darwish, and Younes Samih. 2021. Pre-training bert on arabic tweets: Practical considerations.
Muhammad Abdul- Mageed, AbdelRahim Elmadany, and El Moatez Billah Nagoudi. 2021. ARBERT & MARBERT: Deep bidirectional transformers for Ara- bic. In Proceedings of the 59th Annual Meeting the Association for Computational Linguistics and the 11th International Joint Conference on Natural Lan- guage Processing (Volume 1: Long Papers), pages 7088-7105, Online. Association for Computational Linguistics.
Limitations
Although using GED information an auxiliary in- put improves GEC performance, our GED systems are limited as they can only predict error types for up to 512 subwords since they are built by fine-tuning CAMeLBERT. We also acknowledge
Abdullah Alfaifi and Eric Atwell. 2012. Arabic learner corpora (alc): a taxonomy of coding errors. In The 8th International Computing Conference in Arabic.
Abdullah Alfaifi, Eric Atwell, and Ghazi Abuhakema. 2013. Error annotation of the Arabic learner corpus.

---

In Language Processing and Knowledge the Web, pages 14-22. Springer.
types for grammatical error correction. In Proceed- ings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 793-805, Vancouver, Canada. Association for Computational Linguistics.
Manar Alkhatib, Azza Abdel Monem, and Khaled Shaalan. 2020. Deep learning for Arabic error detec- tion and correction. ACM Trans. Asian Row-Resour. Lang. Inf. Process., 19(5).
Christopher Bryant, Zheng Yuan, Muhammad Reza Qorib, Hannan Cao, Hwee Tou Ng. and Ted Briscoe. 2023. Grammatical error correction: A survey of the state of the art.
Wissam Antoun, Fady Baly, and Hazem Hajj- 2020. AraBERT: Transeormer-based model for Arabic lan- guage understanding. In Proceedings of the 4th Work- shop on Open-Source Arabic Corpora and Process- ing Tools, with Shared Task on Offensive Language Detection, pages 9-15, Marseille, France. European Language Resource Association.
Tim Buckwalter. 2004. Issues in Arabic orthogra- phy and morphology analysis. In Proceedings of the Workshop on Computational Approaches to Ara- bic Script- Languages, pages 31-34, Geneva, Switzerland. COLING.
Hiroki Asano, Tomoya Mizumoto, and Kentaro Inui. 2017. Reference- metrics can be replaced with reference-less metrics in evaluating grammatical er- ror correction systems. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 343 348, Taipei, Taiwan, Asian Federation of Natural Language Processing.
Martin Chodorow, Joel Tetreault, and Na-Rae Han. 2007. Detection of grammatical errors involving prepositions. In Proceedings of the Fourth ACL- SIGSEM Workshop on Prepositions, pages 25-30, Prague. Czech Republic. Association for Computa- tional Linguisties.
Leshem Choshen, Dmitry Nikolaev, Yevgeni Berzak, and Omri Abend. 2020. Classifying syntactic errors in learner language. In Proceedings of the 24th Con- ference on Computational Natural Language Learn- ing, pages 97-107, Online. Association for Computa- tional Linguistics.
Dana Awad. 2013. La ponctuation en Arabe: histoire et règles. etude constrative avec le français et I l'anglais.
Abhijeet Awasthi, Sunita Sarawagi, Rasna Goyal, Sabyasachi Ghosh, and Vihari Piratla. 2019. Par- allel iterative edit models for local sequence trans- duction. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference Natu- ral Language Processing DOMNLP-IJCNLPP pages 4260 4270, Hong Kong, China. Association for Com- putational Linguisties.
Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020. Electra: Pre-training text encoders as discriminators rather than generators. In International Conference on Learning Representa- tions.
C. o. E. Council of Europe. 2001. Common euro- pean framework of reference for languages: learning. teaching, assessment. Cambridge University Press.
Riadh Belkebir and Nizar Habash. 2021. Automatic error type annotation for Arabic. In Proceedings of the 25th Conference on Computational Natural Lan- guage Learning. pages 6666606,60 Online. Association for Computational I Linguistics.
Daniel Dahlmeier and Hwee Tou Ng. 2011. Grammat- ical error correction with alternating structure opti- mization. In Proceedings of the 49th Annual Meet- ing of the Association for Computational Linguistics: Human Language Technologies, pages land, Oregon, USA. Association for Computational Linguistics.
Samuel Bell, Helen Yannakoudakis, and Marek Rei. 2019. Context is key: Grammatical error detection with contextual word representations. In Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educatiinnl Applications, pages 103- 115, Florence, Italy. Association for Computational Linguistics.
Port-
Daniel Dahlmeier and Hwee Tou Ng. 2012. Better evaluation for grammatical error correction. In Pro- ceedings of the 2012 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 568-572 Montréal, Canada. Association for Compu- tational Linguistics.
Fethi Bougares and Houda Bouamor. 2015. UMMU@QA QAAB@-AAB2222 shared task: Character and word level SMT pipeline for automatic error correction of Arabic text. In Proceedings of the Second Workshop on Arabic Natural Language Pro- cessing, pages 166-172, Beijing, China. Association for Computational Linguistics.
Daniel Dahlmeier, Hwee Tou Ng, and Siew Mei Wu. 2013. Building large annotated corpus of learner English: The NUS corpus of learner English. In Pro- ceedings of the Eighth Workshop on Innovative Use of NLP for Building Educational Applications, pages 22-31, Atlanta, Georgia. Association for Computa- tional Linguistics.
Christopher Bryant, Mariano Felice. Øistein E. Ander- sen, and Ted Briscoe. 2019, The BEA-2019 shared task on grammatical error correction. In Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications. pages 52-75, Florence, Italy. Association for Computational Lin- guistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for
Christopher Bryant, Mariano Felice, and Ted Briscoe. 2017. Automatic annotation and evaluation of error

---

Computational Linguisties: Human Language Tech- nologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Zalmout, Sara Hassan, Faisal Al shargi. Sakhar Alkhereyf, Basma Abdulkareem, Ramy Eskander. Mohammad Salameh, and Hind Saddiki. 2018. Uni- fied Guidelines and Resources for Arabic Dialect Orthography. In Proceedings of the International Conference on Language Resources and Evaluation (I.REC 2018), Miyazaki, Japan.
Fadhl Eryani, Nizar Habash, Houda Bouamor, and Salam Khalifa. 2020. A spelling correction corpus for multiple Arabic dialects. In Proceedings of the Twelfth Language Resources and Evaluation Confer- ence, pages 41 4130-4138, Marseille, France. European Language Resources Association.
Nizar Habash and David Palfreyman. 2022. ZAEBUC: An annotated Arabic-English bilingual writer corpus. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 79-88, Marseille, France. European Language Resources Association.
Ramy Eskander, Nizar Habash, Owen Rambow, and Nadi Tomeh. 2013. Processing spontaneous orthog- raphy. In Proceedings of the Conference of the North American Chapter of the Association for Compula- tional Linguistics (NAACL), pages 585-595, Atlanta, Georgia.
Nizar Habash. Owen Rambow, Mona Diab, and Reem Kanjawi-Faraj. 2008. Guidelines for Annotation of Arabic Dialectness. In Proceedings of the Workshop on HLT & NLP within the Arabic World, Marrakech, Morocco.
Tao Fang, Shu Yang, Kaixin Lan, Derek F. Wong. Jin- peng Hu, Lidia S. Chao, and Yue Zhang. 2023. Is chatgpt highly fluent grammatical error correction system? a comprehensive evaluation.
Nizar Habash and Ryan Roth. 2011. Using deep mor- phology to improve automatic error detection in Ara- bic handwriting recognition. In Proceedings of the Conference of the Association for Computational Lin- guistics (ACL), pages 875-884, Portland, Oregon, USA.
Noura Farra, Nadi Tomeh. Alla Rozovskaya, and Nizar Habash. 2014. Generalized character level spelling error correction. In Proceedings of the 52nd Annual Meeting of the Association for Computational Lin- guistics (Volume 2: Short Papers), pages 161-167, Baltimore, Maryland. Association for Computational Linguistics.
Nizar Habash, Abdelhadi Soudi, and Tim Buckwalter. 2007. On Arabic Transliteration. In A. van den Bosch and A. Soudi, editors, Arabic Computational Morphology: Showledge-based and Empirical Meth- ods, pages 15-22. Springer, Netherlands.
Mariano Felice and Ted Briscoe. 2015. Towards a stan- dard evaluation method for grammatical error de- tection and correction. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 578-587, Denver, Col- orado. Association for Computational Linguistics.
Nizar Y Habash. 2010. Introduction to Arabic natural language processing, volume 3. Morgan & Claypool Publishers.
Go Inoue, Bashar Alhafni, Nurpeiis Baimukan, Houda Bouamor, and Nizar Habash. 2021. The interplay of variant, size, and task type in Arabic pre-trained language models. In Proceedings of the Sixth Arabic Natural Language Processing Workshop. pages 92- 104, Kyiv, Ukraine (Virtual). Association for Compu- tational Linguistics.
Mariano Felice, Zheng Yuan, Øistein E. Andersen. He- len Yannakoudakis, and Ekaterina Kochmar. 2014. Grammatical error correction using hybrid systems and type filtering. In Proceedings of the Eighteenth Conference on Computational Natural Language Learning: Shared Task, pages 15-24, Baltimore, Maryland. Association for Computational Linguis- tics.
Go Inoue, Salam Khalifa, and Nizar Habash. 2022. Mor- phosyntactic tagging with are-trained language mod- els for Arabic and its dialects. In Findings of the As- sociation for Computational Linguistics: ACL 2022, pages
Charles F Ferguson. 1959. Diglossia. Word, 15(2):325- 340.
Dublin, Ireland. Association for Computational Linguistics.
Roman Grundkiewiez, Marcin Junczys-Dowmunt,, and Kenneth Heafield. 2019. Neural grammatical error correction systems with unsupervised pre- pre-trinnig on synthetic data. In Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 252-263, Florence, Italy, Association for Computational Linguistics.
Marcin Junczys-Doww Dowmunt and Roman Grundkiewicz. 2014. The AMU system in the CoNLL-2014 shared task: Grammatical error correction by data-intensive and feature-rich statistical machine translation. In Proceedings of the Eighteenth Conference on Com- putational Natural Language Learning: Shared Task, pages 25-33, Baltimore, Maryland. Association for Computational Linguistics.
Nizar Habash, Mona Diab, and Owen Rambow. 2012. Conventional orthography for dialectal Arabic. In Proceedings of the Fighth International Conference on Language Resources and Evaluation (LREC'12), pages 711-718, Istanbul, Turkey. European Lan- guage Resources Association (ELRA).
Marcin Manezys-Dowminn and Roman Grundkiewicz. 2016. Phrase-based based machine translation is state-of- the-art for automatic grammatical error correction. In Proceedings of the 2016 Conference on Empiri- cal Methods in Natural Language Processing, pages 1546-1556, Austin, Texas. Association for Computa- tional Linguistics.
Nizar Habash, Salam Khalifa, Fadhl Eryani, Owen Ram- bow, Dana Abdulrahim, Alexander Erdmann. Reem Faraj, Wajdi Zaghouani, Houda Bouamor, Nasser

---

Marcin Junezys-Dowmunt, Dowmunt, Roman Grundkiewiez, Shubha Guha, and Kenneth Heafield. 2018. Ap- proaching neural grammatical error correction as low-resource machine translation task. In Proceed- ings of the 2018 Conference of the North Ameri- can Chapter of the Association for Computational Linguistics: Human Language Technologies, Vol- ume I (Long Papers), pages 595-606, New Orleans, Louisiana. Association for Computational Linguis- tics.
Jonathan Mallinson, Jakub Adamek, Eric Malmi, and Aliaksei Severyn. 2022. EdiT5: Semi- auttoregrreeesi text editing with 15 warm-start. In Findings of the Association for Computational Linguisties: EMNLP 2022, pages 2126-2138, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Jonathan Mallinson, Aliaksei Severyn, Eric Malmi, and Guillermo Garrido. 2020. FELIX: Flexible text edit- ing through tagging and insertion. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1244-1255, Online. Association for Computational Linguistics.
Moussa Kamal Eddine, Nadi Tomeh, Nizar Habash, Joseph Le Roux, and Michalis Vazirgiannis. 2022. AraBART: pretrained Arabic sequence-to-sequeee model for abstractive summarization. ings of the The Seventh Arabic Natural Language Processing Workshop (WANLP), pages 31-42, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational I Linguistics.
Eric Malmi, Sebastian Krause, Sascha Rothe, Daniil Mirylenka, and Aliaksei Severyn. 2019. Encode, tag, realize:
Proceed-
text editing. In Proceedings of the 2019 Conference Empirical Methods in Nat- ural Language Procexsing and the 9th International Joint Conference on Natural Language Processing (EMNLP- pages 5054 5065, Hong Kong, China. Association for Computational Linguistics.
Masahiro Kaneko and Mamoru Komachi. 2019. Multi- head multi- attention to deep language represen- tations for grammatical error detection.
Stuart Mesham, Christopher Bryant, Marek Rei, and Zheng Yuan. 2023. An extended sequence tagging vocabulary for grammatical eπrоr correction.
Satoru Katsumata and Mamoru Komachi. 2020. Stronger baselines for grammatical error correction model. In Pro- ceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Lin- guistics and the 10th International Joint Conference on Natural Language Processing, pages -22-832, Suzhou, China. Association for Computational Lin- guistics.
using a pretrained mncoder-decodee
Rada Mihalcea and Ted Pedersen. 2003. An evaluation exercise for word alignment. In Proceedings of the HLT-NAACL 2003 Workshop on Building and Using Parallel Texts: Data Driven Machine Translation and Beyond, pages 1-10.
Behrang Mohit, Alla Rozovskaya, Nizar Habash, Wa- jdi Zaghouani, and Ossama Obeid. 2014. The first QALB shared task on automatic text correction for Arabic. In Proceedings of the EMNLP 2014 Workshop Arabic Natural Language Processing (ANLP), pages 39 47, Doha, Qatar. Association for Computational Linguisties.
Ekaterina Kochmar, Øistein Andersen, and Ted Briscoe. 2012. HOO 2012 eиror recognition and correction shared task: Cambridge University submission re- port. Proceedings of the Seventh Workshop on Building Educational Applications Using NLP, pages 242-250, Montréal, Canada. Association for Compu- tational Linguistics.
El Moatez Billah Nagoudi, AbdelRahim Elmadany, and Muhammad Abdul- 2022. AraT5: Text-to- text transformers for Arabic language generation. In Proceedings of the 60th Annual Meeting of the As- sociation for Computational Linguistics (Volume I: Long Papers). pages 628-647, Dublin, Ireland. Asso- ciation for Computational Linguistics.
Wuwei Lan, Yang Chen, Wei Xu, and Alan Ritter. 2020. An empirical study of pre- trained transformers for Arabic information extraction. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4727-4734, Online. Association for Computational Linguistics.
Courtney Napoles, Keisuke Sakaguchi, Matt Post, and Joel Tetreault. 2015. Ground truth for grammatical eтror correction metrics. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Confer- ence on Natural Language Processing (Volume 2: Short Papers), pages 588-593, Beijing, China. Asso- ciation for Computational Linguistics.
V. I. Levenshtein. 1966. Binary Codes Capable of Cor- recting Deletions, Insertions and Reversals. Soviet Physics Doklady, 10:707.
Zuchao Li, Kevin Parnow, and Hai Zhao. 2022. Incor- porating rich syntax information in grammatical error correction. Information Processing & Management, 1928):102891.
Courtney Napoles, Keisuke Sakaguchi, and Joel Tetreault. 2016. There's s no comparison: Reference- less evaluation metrics in grammatical error correc- tion. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Process- ing, pages 1109-2115,, Austin, Texas. Association for Computational Linguistics.
Nora Madi and Hend AI-Khalifa. 2020. Error detec- tion for Arabic text using neural sequence labeling. Applied Sciences, 10(15).
Koki Maeda, Masahiro Kaneko, and Naoaki Okazaki. 2022. IMPARA: Impact- metric for GEC us- ing parallel data. In Proceedings of the 29th Inter- national Conference on Computational Linguistics, pages 3578-35588 37888 Gyeongju, Republic of Korea. In- ternational Committee on Computational Linguistics.
Michael Nawar. 2015. CUFE@QALB-200 shared task: Arabic error correction system. In Proceedings
of the Second Workshop on A Arabic Natural Language

---

Processing, pages 133-137, Beijing, China. Associa- tion for Computational Linguistics.
Workshop on Arabic Natural Language Processing (ANLP), pages 160-164, Doha, Qatar. Association for Computational Linguistics.
Hwee Tou Ng, Siew Mei Wu, Ted Briscoe, Christian Hadiwinoto, Raymond Hendy Susanto, and Christo- pher Bryant. 2014. The CoNLL-2014 shared task on grammatical error correction. In Proceedings of the Eighteenth Conference on Computational Natu- ral Language Learning: Shared Task, pages 1-14, Baltimore, Maryland, Association for Computational Linguistics.
Alla Rozovskaya and Dan Roth. 2013. Joint learning and inference for grammatical error correction. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 791- 802, Seattle, Washington, USA. Association for Com- putational Linguistics.
Ali Safaya, Moutasem Abdullatif, and Deniz Yuret. 2020. KUISAIL SemEval-22020 task 12: BERT- CNN for offensive speech identification in social me- dia. In Proceedings of the Fourteenth Workshop on Semantic Evaluation, pages 2054-2059, Barcelona (online), International Committee for Computational Linguistics.
Hwee Tou Ng. Siew Mci Wu, Yuanbin Wu, Christian Hadiwinoto, and Joel Tetreault. 2013. The CoNLL- 2013 shared task on grammatical error correction. In Proceedings of the Seventeenth Conference on Computational Natural Language Learning: Shared Task, pages 1-12, Sofia, Bulgaria. Association for Computational Linguistics.
Aiman Solyman, Zhenyu Wang. Qian Tao, Arafat Abdulgader Mohammed Elhag, Rui Zhang, and Zeinab Mahmoud. 2022. Automatic Arabic gram- matical error correction based on expectation- maximization routing and target-bidirectional agree- ment. Knowledge-Basee Based Systems, 241:108180.
Ossama Obeid, Nasser Zalmout, Salam Khalifa, Dima Taji, Mai Oudah, Bashar Alhafni, Go Inoue, Fadhl Fryani, Alexander Tirdmann, and Nizar Hahash. 2020. CAMeL tools: An open source python toolkit for Arabic natural language processing. In Proceedings of the Twelfth Language Resources and Evaluation Conference. pages 7022-7032. Marseille, France. Eu- ropean Language Resources Association.
Aiman Solyman, Marco Zappatore, Wang Zhenyu, Zeinab Mahmoud, Ali Alfatemi, Ashraf Osman Ibrahim, and Lubna Abdelkareim Gabralla. 2023. Optimizing the impact of data augmentation for low- resource grammatical error correction. Journal of King Saud University- - Computer and Information Sciences, 3516):00157..
Franz Josef Och and Hermann Ney. 2003. A systematic comparison of various statistical alignment models. Computational Linguistics, 19-151
Kostiantyn Omelianchuk, Vitaliy Atrasevych, Artem Chernodub, and Oleksandr Skurzhanskyi. 2020. GECToR grammatical eтror correction: Tag, not rewrite. In Proceedings of the Fifieenth Workshop on Innovative Use of NL.P for Building Educational Applications, pages 163-170, Seattle, WA, USA- + Online. Association for Computational Linguistics.
Aiman Solyman, Wang Zhenyu, Tao Qian, Arafat Ab- dulgader Mohammed Elhagg Muhammad Toseef and Zeinab Alcibeid. 2021. Synthetic data with neural machine translation for automatic correction in Arabic grammar. Egyptiaan Informatics Journal, 22(3):303-315.
Marek Rei and Helen Yannakoudakis. 2016. Composi- tional sequence labeling models for eтror detection in learner writing. In Proceedings of the 54th Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), pages 1181-1191, Berlin, Germany. Association for Computational Lin- guistics.
Felix Stahlberg and Shankar Kumar. 2020. Seq2Edits: Sequence transduction using span-level edit opera- tions. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5147-5159, Online. Association for Computational Linguistics.
Milan Straka, Jakub Náplava, and Jana Straková. 2021. Character transformations for non-autoregressive GEC tagging. In Proceedings of the Seventh Work- shop on Noisy bher-generated Text (W-NUT 2021). pages 417-422, Online. Association for Computa- tional Linguistics.
Sascha Rothe, Jonathan Mallinson, Eric Malmi, Sebas- tian Krause, and Aliaksei Severyn. 2021. A simple recipe for multilingual grammatical error correction. In Proceedings of the 59th Annual Meeting of the As- sociation for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers). ). pages 702-707, Online. Association for Computational Linguistics.
Dima Taji. Jamila EI Gizuli, and Nizar Habash. 2018. An Arabic dependency treebank in the travel do- main. In Proceedings of the Workshop on Open- Source Arabic Corpora and Processing Tools (OS- ACT), Miyazaki, Japan.
Alla Rozovskaya, Houda Bouamor, Nizar Habash, Wa- jdi Zaghouani, Ossama Obeid, and Behrang Mohit. 2015. The second QALB shared task on automatic text correction for Arabic. In Proceedings of the Second Workshop on Arabic Natural Language Pro- cessing, pages 26-35, Beijing, China. Association for Computational Linguistics.
Joel R. Tetreault and Martin Chodorow, 2008. The ups and downs of preposition error detection in ESL writing. In Proceedings of the 22nd International Conference on Computational Linguistics (Coling 2008), pages 865-872, Manchester, UK. Coling 2008 Organizing Committee.
Alla Rozovskaya, Nizar Habash, Ramy Eskander, Noura Farra, and Wael Salloum. 2014. The Columbia sys- tem the QALB-2014 shared task on Arabic er- ror correction. In Proceedings of the EMNLP 2014
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz

---

Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Pro- cessing Systems, volume 30. Curran Associates, Inc.
evaluation. In The 2nd Workshop on Arabic Corpora and Processing Tools 2016 Theme: Social Media. page 22.
Daniel Watson, Nasser Zalmout, and Nizar Habash. 2018. Utilizing character and word embeddings for mod-
Wajdi 7aghouani, Nizar Habash, Houda Bouamor, Alla Rozovskaya, Behrang Mohit, Abeer Heider, and Kemal Oflazer. 2015. Correction annotation for non-native arabic texts: Guidelines and corpus, In Proceedings of the Linguistic Anmotation Workshop (LAW), pages 129-139.
text normalization with qquuence-t--squeeee cls. In Proceedings of the 2018 Conference on Em- pirical Methods in Natural Language Processing. pages 837-843, Brussels, Belgium. Association for Computational Linguistics.
Wajdi Zaghouani, Behrang Mohit, Nizar Habash, Os- sama Obeid, Nadi Tomeh, Alla Rozovskaya, Noura Farra, Sarah Alkuhlani, and Kemal Oflazer. 2014. Large scale Arabic error annotation: Guidelines and framework. In Proceedings of the Ninth International Conference on Language Resources and Evaluation 14), Reykjavik, Iceland. European Language Resources Association (ELRA).
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R' t'm Louf, Morgan Funtowiez, and Jamie Brew. 2019. Huggingface's transformers: State-of-the natural language processing, ArXiv, abs/1910.03771.
Ziang Xie, Anand Avati, Naveen Arivazhagan, Dan Jurafsky, and Andrew Y. Ng. 2016, Neural language correction with character- based attention.
Yue Zhang, Bo Zhang. Zhenghua Li, Zuyi Bao, Chen Li, and Min Zhang. 2022. SynGEC: eehhax-enhanced grammatical error correction with a tailored GEC- oriented parser. In Proceedings of the 2022 Con- ference on Empirical Methods in Natural Language Processing, pages 2518-2531, Abu Dhabi, United Arab Emirates. Association for Computational Lin- guistics.
Zhilin Yang. Zihang Dai, Yiming Yang, Jaime Car- bonell, Russ R Salakhutdinov, and Quoc V Le. 2019. Xinet: Generalized autoregressive pretraining for lan- guage understanding. In Advances in Newral Infor- mation Processing Systems, volume 32. Curran Asso- ciates, Inc.
Wei Zhao, Liang Wang, Kewei Shen, Ruoyu Jia, and Jingming Liu. 2019. Improving grammatical error correction via pre-training copy-augmented archi- tecture with unlabeled data. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, Volume (Long and Short Papers), pages 156-165, Minneapolis, Min- nesota. Association for Computational Linguistics.
Helen Yannakoudakis, Ted Briscoe, and Ben Medlock. 2011. new dataset and method for automatically grading ESOL texts. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 180-189, Portland, Oregon, USA. Association for Computational Linguistics.
Zheng Yuan and Ted Briscoe. 2016. Grammatical er- ror correction using neural machine translation. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pages 380-386, San Diego, California. Association for Computational Linguistics.
Zheng Yuan and Christopher Bryant. 2021. Document- level grammatical error correction. In Proceedings of the 16th Workshop on Innovative Use of NLP for Building Educational Applications, pages 75-84, On- line. Association for Computational Linguistics.
Zheng Yuan, Felix Stahlberg, Marek Rei, Bill Byrne, and Helen Yannakoudakis. 2019. Neural and FST- based approaches to grammatical error correction. In Proceedings the Fourteenth Workshop on Innova- rive Use of NLP for Building Educational Applica- tions, pages 228-239, Florence, Italy. Association for Computational Linguistics.
Zheng Yuan, Shiva Taslimipoor, Christopher Davis, and Christopher Bryant 2021. Multi-class grammatical error detection for correction: A tale two systems. In Proceedings of the 2021 Conference on Empiri- cal Methods in Natural Language Processing, pages 8722-8736, Online and Punta Cana, Dominican Re- public. Association for Computational Linguistics.
Wajdi Zaghouani and Dana Awad. 2016. Toward an Arabic punctuated corpus: Annotation guidelines and

---

A Detailed Experimental Setup
Grammatical Error Detection Our GED mod- els were fine-tuned for 10 epochs using a learning rate of 5e-5, a batch size of 32, and seed of 42. At the end of the fine-tuning, we pick the best check- point based on the performance on the Dev sets.
Grammatical Error Correction When using AraBART, we fine-tune the models for 10 epochs by using a learning rate of 5e-5, a batch size 32, a maximum sequence length of 1024, and a seed of 42. For AraT5, we fine-tune the models for 30 epochs by using learning rate of le-4 and the rest of the hyperparameters are the same the ones used in AraBART. During inference, search with beam width of for all models. At the end of the fine-tuning, we pick the best check- point based on the performance on the Dev sets by using the M² scorer. The M scorer suffers from extreme running times in cases where the generated outputs differ significantly from the input. To mit- igate this bottleneck, we extend the M² scorer by introducing time limit for each sentence during evaluation. If the evaluation of a single generated sentence surpasses this limit, we pass the input sen- tence to the output without modifications. We use this extended version of the M scorer when report- ing our results on the Dev sets. When reporting our results on the Test sets, we use the M scorer release that is provided by the QALB shared task. We make our extended version of the 2 scorer publicly available.
beam
ChatGPT We start with prompting ChatGPT with a 3-shot prompt. Our exact prompt is the following:
"Please identify and correct any spelling and grammar mistakes the following sentence indicated by <input> IN- PUT <finpu>> tag. You need comprehend the sentence whole before gradually identifying and correcting any errors while keeping the original sentence structure unchanged much as possible.
Afterward, output the corrected version dircetly without any explanations. Here
in-context examples:
(1), <input> SRC-1 </inputz: <output> TGT-1 </outputs.
(2), <input> SRC-2 </input>: <output> TGT-2 </output>.
(3), <input> SRC-3 </inputz: <output> TGT-3 </output>.
Please feel free to refer these examples. Remember to format your corrected output results with tag cout- put> Your Corrected Version </oulputs. Please start: put> INPUT </input>"

---

B Datasets Statistics
Dataset
Split
Lines
Words
Err. %
Level
Domain
Train-L1
19,411
1,021,165
30%
Native
Comments
QALB-2014
Dev-L1
1,017
53,737
31%
Native
Comments
Test-L1
968
51,285
32%
Native
Comments
Train-L2
310
43,353
30%
L2
Essays
Dev-L2
154
24,742
29%
L2
Essays
QALB-2015
Test-L.2
158
22,808
29%
L.2
Essays
Test-L1
920
48,547
27%
Native
Comments
Train-L1
150
25,127
24%
Native
Essays
ZAEBUC
Dev-L1
33
5,276
25%
Native
Essays
Test-L1
31
5,118
26%
Native
Essays
Table 9: Corpus statistics of Arabic GEC datasets.

---

C Error Types Statistics
Tag
Error Description
Example
QALB-2014
QALB-2015
ZAEBUC
OA
Alif, Ya & Alif-Maqsura
عي عي
7,627
3%
290
2%
27
0%
OC
Char Order
li e- شري
466
0%
45
0%
30
0%
OD
Additional Char
4,086
1%
283
2%
103
2%
0G
Lengthening short vowels
تقم شور
0
0%
0%
0
0%
OH
Hamza errors
-
90,579
30%
1,076
8%
1,905
32%
Orthography
OM
Missing char(s)
سلااان - سننن
4,062
1%
361
3%
123
2%
(0)
ON
Tanwin Confusion
0%
0%
OR
Char Replacement
رصااا ممان
8,350
3%
762
162
3%
OS
Shortening long vowels
أرللت زرشت
0%
0%
0%
OT
Hfa/Ta/T Ta-Marbuta Confusion
Rs مشاك كم مشار
14,688
5%
54
0%
408
7%
OW
Confusion in Alif Fariqa
وكلزا وكلر
1,885
1%
32
0%
12
0%
00
Other orthographic errors
1,632
1%
38
0%
148
2%
MI
Word inflection
عزيف - مررف
1,360
0%
400
3%
127
2%
Morphology
MT
Verb tense
حقت أنر - ترحقي
76
0%
136
1%
4
0%
(M)
MO
Other morphological errors
15
0%
0%
3
0%
Case
ال , - راع
5,980
2%
279
2%
201
3%
Definiteness
سن السن
0%
835
1%
XG
Gender
بي الغيبب
0%
317
1%
Syntax (X)
XM
Missing word
عيي Null
1,375
0%
763
6%
68
1%
XN
Number
5. أككز شككب
1,107
0%
210
2%
30
0%
XT
Unnecessary word
Null- علي
1,047
0%
418
3%
116
2%
XO
Other syntuctic errors
3,270
1%
122
1%
57
1%
SF
Conjunction error
فيجان - سبحل
96
0%
46
0%
4
0%
Semantics
SW
Word selection error
Ge - im
4,711
2%
865
7%
120
2%
SO
Other semantic errors
380
0%
114
1%
27
0%
PC
Punctuation confusion
: - .b
11,361
4%
7%
237
4%
Ponctuation
PM
Missing punctuation
المااوي العقبمم
97,271
32%
2,915
22%
479
8%
(P)
Unnecessary punctuation
العام المنن
5,553
2%
213
204
3%
PO
Other
punctuation
0%
0%
0%
Merge
MG
Words merged
j/ز للالز
15,063
5%
377
3%
849
14%
Split
SP
Words split
رفال قلل
7,828
3%
80
1%
49
1%
Unknown
UNK
Unkown Lirors
فللرر الشبن الغامونن
2,053
1%
303
2%
93
2%
Comb.
Error Combinations
امدايةة e- اتاايي
11,304
4%
848
7%
314
5%
304,886
13,043
6,037
Table 10: The statistics of the types in the Train of QALB-2014, QALB-2015, and ZAEBUC. The emor types are based on the extended ALC (Alfaifi et al., 2013) taxonomy as used by Belkebir and Habash (2021).

---

D GED Granularity Data Statistics
QALB-2014
QALB-2015
ZAEBUC
2-Class
13-Class
43-Class
Train
Dev
Test
Train
Test-L1
Test-L2
Train
Dev
Test
Delete
Delete
6,442
346
540
584
339
250
309
Merge-B
Merge-B
15,063
797
795
377
231
625
199
849
180
133
Merge-I
Merge-1
15,296
812
807
390
241
629
200
851
180
133
M
2
MI
1,360
69
400
169
MT
76
4
136
72
2
40
4
0
1
M+0
MI-OH
243
17
15
7
8
O
3,255
166
164
75
144
70
296
OA 7,627
313
252
290
514
138
27
OC 466
27
19
45
23
17
26
30
7
7
OD 4,086
207
204
283
167
146
166
103
24
OH 90,579
4,785
4,632
1,076
4,499
1,905
401
451
OM 4,062
228
217
361
215
188
OR 8,358
425
446
763
415
369
362
162
32
36
OT 14,688
758
623
54
37
733
26
408
101
138
OW 1,885
149
2
OA 480
19
12
23
OA
215
6
4
11
O
0
OD+OG
573
32
32
22
15
23
I1
4
2
OD+OH 317
OD 104
12
ODIOR 675
33
26
61
32
22
32
8
2
2
OH+OM
2,339
134
123
231
106
114
109
54
15
13
Off 1,468
OM
382
15
19
62
27
23
15
17
OR+OT
193
10
7
4
0
O+X
OH+XC
323
24
18
155
20
4
11,379
S
536
41
19
188
125
26
103
44
14
SF
96
4
46
33
21
4
0
2
SW
4,804
201
229
887
502
186
422
121
22
3,668
XC 5,980
373
369
279
180
289
141
201
31
XC
XG
296
23
40
3
0
0
XC
XN
500
18
29
13
23
24
3
XF
XG
809
38
30
317
175
35
158
86
20
24
XM
225
15
6
151
91
12
68
14
3
XN
1,107
47
210
115
47
84
30
2
155
9
24
Split
Split
7,828
432
399
80
42
382
34
49
10
10
UNK
UNK
6,835
331
300
969
454
257
416
361
78
61
C
795,510
41,875
39,690
33,007
19,004
38,063
17,651
18,411
3,839
3,683
1,021,165
53,737
51,285
43,353
24,742
48,547
22,808
25,127
5,276
5,118
Table 11: The statistics of the different GED granularity error types we model across the three datasets. The description of the labels in the 13-Class and 43-Class categories are in Appendix C. For the 2-Class labels, E refers to erroneous words and refers to correct words.

---

E
Error Analysis on Error Types
QALB-2014
QALB-2015
ZAEBUC
AraBART
Best System
AraBART
Best System
AraBART
Best System
Delete
40.1
40.8
40.5
45.3
47.5
51.9
Merge-B
91.2
93.0
82.4
86.6
96.7
96.7
Merge-I
91.0
93.0
81.7
86.4
96.7
96.7
M
24.8
27.6
37.0
40.8
48.9
48.6
M+0
54.8
37.7
17.2
13.2
100.0
55.6
O
94.0
94.3
80.3
80.2
94.0
94,4
O+X
67.7
73.9
0.0
0.0
0,0
0.0
P
76.4
77.4
64.5
63.7
66.8
62.8
43.3
44.5
33.8
34.1
36.1
40.4
X
58.5
61.1
59.6
63.9
69.5
72.9
Split
87.6
87.1
78.0
78.9
88.2
88.2
UNK
50.2
57.2
37.9
35.2
57.1
63.1
C
96.2
96.8
89.9
91.4
95.4
96.1
Macro Avg
67.4
68.0
54.1
55.5
69.0
66.7
Table 12: Specific error type performance of AraBART and our best system (AmBAARI-Mophh-GED on average the Dev sets of QALB-2014, QALB-2015, and ZAEBUC. Results reported in terms of Fo.s- The hest results are in bold.


Reviews:
[{"review_id": "qK9BghaIps", "reviewer": "Reviewer_51Ai", "summary": "", "questions": "The problem of Arabic dialects is not really discussed in the paper. The large pretrained models are probably trained also on dialectal data. Are dialectal words considered as errors in MSA texts? Is there a special error type for them? \n\nPlease, provide better description of the alignment algorithm. It is not clear, whether the Levenshtein algorithm is applied to tokens or to the whole sentences.  The greedy part of the algorithm is not explained well. \n\nLine 565 refers to a statistical test of significance. Please, provide details on the test results also for other experiments (mainly presented in Table 6). \n\n", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your comments and for pointing out the strengths of our work. We address all the points you mentioned in the reasons to reject and the questions you have below.\n\n**Reasons to Reject**:\n\n  * **The effect of including the morphological analysis and error classification is rather small (measured on the test sets, Table 6). Most of the results are reported as not statistically significant**: While the improvements on the test sets are relatively small when using morphological preprocessing, GED, or both, this does not necessarily indicate that using morphology or GED does not help Arabic GEC significantly. In fact, we show on the development sets that when using an Oracle GED system with morphological preprocessing (Table 5), we are able to get an improvement of up to ~5 points on average F0.5.\n\n\n* **Only the overall (average) improvements of the models combining GED and morphological analysis is provided**: We provide results of all systems we develop across all datasets in terms of precision, recall, and F0.5. We only use the average F0.5 in Tables 4 and 5 when we report results on the development sets of QALB-2014, QALB-2015, and ZAEBUC. Our intention of using average F0.5 was merely to select a single best system (AraBART+Morph+GED) when we report results on the test sets individually. That’s why Table 6 does not include average F0.5 results. We will add the averages for the F0.5 test results in the camera-ready for completeness. \n\n* **It is not clear, whether there are some error types where the models can benefit from knowing the explicit information of the type of the error:** In Section 7 (**Lines 534-537**), we mention that we provide in **Appendix E** a specific error type performance by comparing AraBART to our best system that uses morphological preprocessing and GED tags as auxiliary input (AraBART+Morph+GED) on the dev sets. Table 11 in **Appendix E** indicates the specific error types that benefit from modeling GED explicitly.\n\n* **It is not clear, from the paper, how accurate is the automatic method for error type classification. At least a small qualitative study should have been performed to analyse whether the errors are well classified. The authors report the results on the automatically obtained annotations, but the performance measured w.r.t. ground truth is not discussed:** The automatic error type classification tool we use, ARETA ([Belkebir and Habash, 2021](https://aclanthology.org/2021.conll-1.47.pdf)), was evaluated against the manual annotation of the Arabic Learner Corpus (ALC). [Belkebir and Habash, 2021](https://aclanthology.org/2021.conll-1.47.pdf) report that ARETA matches the ALC annotation with 89.2% (F1 micro) on the development test, and 85.8% (F1 micro) on a blind test set. We will make sure to reference the performance of ARETA in the camera-ready of our paper if it’s accepted.\n\n\n**Questions:**\n\n * **The problem of Arabic dialects is not really discussed in the paper. The large pretrained models are probably trained also on dialectal data. Are dialectal words considered as errors in MSA texts?** Regarding the pretrained Seq2Seq models (i.e., AraBART and AraT5), we mention in **Lines 386-390** that AraBART was pretrained on MSA text while AraT5 was pretrained on a mix of MSA text and Twitter data, which contains dialectal text. As mentioned in Section 3.1, Arabic is diglossic, and borrowing from the dialects is frequent when native speakers write in MSA. The two L1 datasets we use to train and evaluate our models, QALB-2014 and ZAEBUC, contain errors that are almost always in MSA. However, they have errors that are related to dialectal usage. The creators of QALB 2014 intentionally excluded dialectal sentences; and focused on MSA, which may include some dialectal effects. When QALB-2014 was originally annotated, only three types of dialectal errors were manually corrected: morphological choices, phonological choices, and closed-class dialectal words ([Zaghouani et al., 2014](http://www.lrec-conf.org/proceedings/lrec2014/pdf/956_Paper.pdf)). However, these errors were not marked as dialectal errors (in fact no errors are marked at all in QALB data beyond correcting them). The error annotation tool we use was built based on an MSA-only error taxonomy (**Line 288**), and therefore, we do not mark dialectal words using error types outside of the seven error types that are provided by ARETA (**Lines 293-298**). We are aware of other existing Arabic dialectal GEC datasets ([Khalifa et al., 2018](https://aclanthology.org/L18-1607.pdf), [Eryani et al., 2020](https://aclanthology.org/2020.lrec-1.508.pdf)), but our work is mainly intended to address the problem of GEC in MSA. Arabic GEC has a long way to go, and we definitely plan to explore GEC in other Arabic dialectal varieties in future work. We will make sure to mention this in the future work section of the camera-ready version if our paper is accepted.\n\n\n * **Please, provide better description of the alignment algorithm. It is not clear, whether the Levenshtein algorithm is applied to tokens or to the whole sentences. The greedy part of the algorithm is not explained well**: Our alignment algorithm is similar to a standard word edit-distance alignment algorithm applied at the sentence level (over tokens) with backtracking, except that we have two design enhancements to arrive at multi-token alignments and use character-level information to make optimized word alignment decisions. Our algorithm has 3 main components:\n\n      1.  We modify the standard Levenshtein algorithm such that the token replacement costs are obtained with a secondary application of the Levenshtein edit distance at the token level (over characters).\n      2.  Once we obtain the edit distance table from step 1, we obtain an initial token level alignment by backtracking. Each edit in this alignment (replacement, insertion, deletion, keep) involves one token at most. To obtain multi-token edits, we implement an iterative greedy algorithm, which we explain in 3.\n      3. We apply an iterative greedy algorithm to convert adjacent sequences of replacements, insertions, and deletions into singular operations of word merge or word split such that the overall cost is minimized (based on Levenshtein character-level edit distance).\n    \n      We will make sure to expand the description of our algorithm in the camera-ready version if our paper is accepted.\n\n  * **Line 565 refers to a statistical test of significance. Please, provide details on the test results also for other experiments (mainly presented in Table 6):** We did not run standard statistical significance checks for the test sets results of QALB-2015-L2 and ZAEBUC because they are too small to allow us to test for statistical significance (**Lines 621-623**). We will consider Oracle segmentation (based on the corrected gold target) of QALB-2015-L2 and ZAEBUC. This allows us to obtain smaller segments and increase the number of examples in each dataset so we can run statistical significance.\n\n\n\nThank you again for your comments and suggestions. We hope that our response clarifies your concerns, and we will make sure to address all of them in the extra page we are allowed to add in the camera-ready version if our paper is accepted.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Nicely written paper on an interesting topic. Showing how explicit GED and Morphological analysis can help in GEC in Arabic.\nStrong evaluation on three diverse datasets, outperforming the current SOTA. Comparison with several baselines, also including ChatGPT. \nThe list of references is extremely rich and can well serve as a source of relevant references on this topic\n", "reasons_to_reject": "While the authors outperform the current SOTA on all the datasets, the main reason for that is the large pretrained models - AraT5 and AraBART (the latter showed better perfomrance). The effect of including the morhological analysis and error classification is rather small (measured on the test sets, Table 6), most of the results are reported as not statistically significant. \n\nOnly the overall (average) improvements of the models combining GAD and Morphological analysis is provided. It is not clear, whether there are some error types where the models can benefit from knowing the explicit information of the type of the error.  \n\nIt is not clear, from the paper, how acurrat is the automatic method for error type classification. At least a small qualitative study should have been performed to analyse, whether the errors are well classified. The authors report the results on the automatically obtained annotations, but the performance measured w.r.t. ground truth is not discussed. \n\n\n", "questions_to_authors": "The problem of Arabic dialects is not really discussed in the paper. The large pretrained models are probably trained also on dialectal data. Are dialectal words considered as errors in MSA texts? Is there a special error type for them? \n\nPlease, provide better description of the alignment algorithm. It is not clear, whether the Levenshtein algorithm is applied to tokens or to the whole sentences.  The greedy part of the algorithm is not explained well. \n\nLine 565 refers to a statistical test of significance. Please, provide details on the test results also for other experiments (mainly presented in Table 6). \n\n", "paper_topic_and_main_contributions": "The paper addresses the problem of grammatical error correction (GEC) and grammatical error detection (GED) in Arabic, more specifically on Modern Standard Arabic (no dialectal Arabic). Arabic is a morphologically rich language therefore the tasks is more difficult compared to e.g. English. \n\nThe authors work with three existing datasets for GEC in MSA, for all of them the erroneous text is provided with the corrected one. \n\nFor GED, the authors extend the three existing datasets with automatic annotation of errors types. This mainly requires alignment of the two texts. The authors propose a novel method for obtaining automatic word alignment of erroneous and correct text in Arabic, achieving almost 100% Precision and Recall. After alignment, the errors are automatically classified into 7 error classes and 32 subtypes (assigned individually or in a combination). An Arabic PLM CAMeLBERT is finetuned for the classification tasks with varying number of classes (43/13/2) and evaluated on the three data sets. \n\nFor GEC, where each error must be detected and a replacement produced.  The authors proposes two models, one based on AraBART, one on AraT5, each in three configurations, one includes morphological analysis of each Arabic word, one detecting error type, the last one combining both. The performance of those models are compared with other solutions and a few baselines (including e.g. ChatGPT). The results show a superior performance of the models combining both GED and Morphological analysis.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "QZRng7EQYl", "reviewer": "Reviewer_2Jvn", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your comments and for pointing out the strengths of our work. We will definitely add a paragraph at the end of the introduction outlining the structure of the paper in the camera-ready version if our paper is accepted.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Grammatical error correction (GEC) is still problematic for morphologically rich languages, such as Arabic, due to the complexity and nature of such languages. Having a paper that focuses on GEC for Arabic, helps the NLP community in developing GEC systems for Arabic. \n", "reasons_to_reject": "No reasons", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper focuses on investigating the effectiveness of Grammatical Error Detection (GED) on Arabic Grammatical Error Correction (GEC). For the investigation process, they depend on three datasets (QALB-2014, QALB-2015, and ZAEBUC). They claim that they were the first to benchmark newly developed pretrained Seq2Seq models on Arabic Grammatical Error Correction. As for the GED task, they used their own alignment algorithm which shows that they were superior across all metrics. They could show that using GED information in GEC models improves the performance across GEC datasets. In addition, they discussed the importance of leveraging contextual morphological preprocessing in improving the GEC performance. Their experiments achieved SOTA results on the two (L1 and L2) datasets of (QALB-2014 and QALB-21015).", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "It would be better if you add the structure of the paper in the last part of the introduction."}, "extra_text": {}}, {"review_id": "YvLrlQHDzR", "reviewer": "Reviewer_XEzF", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your comments and for pointing out the strengths of our work. Previous work on self-supervised learning in Arabic showed that pretraining BERT-based models on more data does not necessarily lead to better results ([Inoue et al., 2021](https://aclanthology.org/2021.wanlp-1.10.pdf)). [Inoue et al., 2021](https://aclanthology.org/2021.wanlp-1.10.pdf) showed that Arabic variants (i.e., MSA vs. dialects) proximity of pretraining data to fine-tuning task-specific data is more important than pretraining data size. Although AraT5 and AraBART are Seq2Seq models, we hypothesize that a similar conclusion can be observed from our results. Particularly because AraT5 was pretrained on a mix of MSA and dialectical text, whereas AraBART was pretrained on MSA-only text. Given that the GEC datasets we use are in MSA, this explains why AraBART is performing better than AraT5 in most cases, even though it was prertrained on less data.\n\nWe hope that our response clarifies your concerns. We will make sure to mention this explanation in the extra page we are allowed to add in the camera-ready version if our paper is accepted.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is well-written and easy to understand; the readers who do not speak Arabic, such as me, can also understand the nature and background of Arabic GEC quickly by reading Sections 2 and 3. The newly proposed alignment algorithm and the strong baseline are essential contributions to the Arabic GEC community. ", "reasons_to_reject": "I only have one concern about the missing explanations of results. In Section 7, Line 509, the paper only describes the results. In Section 5, we can see that AraT5 has a larger pre-trained data size than AraBART but performs worse in most of the situations on the GEC dataset. It would be better to provide some explanations about this kind of result.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper builds a new benchmark for the Arabic grammatical error correction (GEC) area. To build the benchmark, it starts with the grammatical error detection (GED) task. In this section, the paper proposes a new alignment algorithm for error type annotation and achieves better results compared with previous methods. In the GEC section, it finds that the pre-trained model BART can be enhanced with GED and Morphological Disambiguation output, and this combination performs best among most of the settings. It also shows how GED granularity affects final GEC performance.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper tackles an underexplored problem in the NLP community, namely the grammatical error correction (GEC) for Arabic language. The paper proposes two Transformer-based models and demonstrated state-of-the-art performance on multiple shared task datasets. The reviewers all agreed on the soundness and excitement of this work.
Decision: Accept-Main