Targeted Syntactic Evaluation for Grammatical Error Correction
Authors: Aomi Koyama, Masato Mita, Su-Youn Yoon, Yasufumi Takama, Mamoru Komachi
Accepted at: acl25

Targeted Syntactic Evaluation for Grammatical Error Correction

Aomi Koyama1    Masato Mita1,2    Su-Youn Yoon3,2
Yasufumi Takama1    Mamoru Komachi2,1
1Tokyo Metropolitan University   2Hitotsubashi University   3EduLab, Inc.
koyama-aomi@ed.tmu.ac.jp, mita@edu.sds.hit-u.ac.jp, su-youn.yoon@edulab-inc.com
ytakama@tmu.ac.jp, mamoru.komachi@r.hit-u.ac.jp

Abstract

Language learners encounter a wide range of grammar items across the beginner, intermediate, and advanced levels. To develop grammatical error correction (GEC) models effectively, it is crucial to identify which grammar items are easier or more challenging for models to correct. However, conventional benchmarks based on learner-produced texts are insufficient for conducting detailed evaluations of GEC model performance across a wide range of grammar items due to biases in their distribution. To address this issue, we propose a new evaluation paradigm that assesses GEC models using minimal pairs of ungrammatical and grammatical sentences for each grammar item. As the first benchmark within this paradigm, we introduce the CEFR-based Targeted Syntactic Evaluation Dataset for Grammatical Error Correction (CTSEG), which complements existing English benchmarks by enabling fine-grained analyses previously unattainable with conventional datasets. Using CTSEG, we evaluate three mainstream types of English GEC models: sequence-to-sequence models, sequence tagging models, and prompt-based models. The results indicate that while current models perform well on beginner-level grammar items, their performance deteriorates substantially for intermediate and advanced items.

1 Introduction

Grammatical error correction (GEC) is a task that involves correcting errors in text to produce grammatically correct expressions. It primarily aims to support language learners by addressing various types of errors they make (Bryant et al., 2023). Currently, sequence-to-sequence models (Kiyono et al., 2020; Rothe et al., 2021) and sequence tagging models (Omelianchuk et al., 2020; Tarnavskyi et al., 2022) are widely used in GEC, both of which have achieved high performance. In recent years, models leveraging instruction-based prompts (Brown et al., 2020; OpenAI et al., 2024) have also gained significant attention in this field, demonstrating promising results (Wu et al., 2023; Fang et al., 2023; Coyne et al., 2023; Loem et al., 2023; Davis et al., 2024).

In GEC, learner-generated texts have traditionally been used for model evaluation. The CoNLL-2014 shared task test set (Ng et al., 2014), one of the most widely recognized benchmarks in GEC, is based on essays written by students at the National University of Singapore. Similarly, the JFLEG test set (Napoles et al., 2017), which focuses on measuring correction fluency (Sakaguchi et al., 2016), includes essays written by learners at various proficiency levels as part of English proficiency tests. These test sets enable GEC models to be evaluated in contexts that closely reflect real-world usage.

However, evaluating GEC models using learner-

(a) Conventional evaluation datasets based on learner-produced text: Since these datasets are not explicitly designed for specific grammar items, they may not cover all the items that developers intend to evaluate.

(b) A proposed evaluation dataset: Developers create dedicated evaluation datasets for specific grammar items, enabling targeted assessment of GEC models.

Figure 1: Comparison of evaluation methods: Conventional vs. proposed evaluation dataset for GEC models. Red wavy lines indicate errors for each grammar item.
<table>
  <tr>
    <th>Dataset</th>
    <th>Sents.</th>
    <th>Refs.</th>
    <th>Error Tags</th>
    <th>CEFR Level</th>
  </tr>
  <tr>
    <td>FCE (Yannakoudakis et al., 2011)</td>
    <td>2,695</td>
    <td>1</td>
    <td>71</td>
    <td>B1–B2</td>
  </tr>
  <tr>
    <td>KJ (Nagata et al., 2011)</td>
    <td>3,199</td>
    <td>1</td>
    <td>22</td>
    <td>A1–A2?</td>
  </tr>
  <tr>
    <td>CoNLL-2014 (Ng et al., 2014)</td>
    <td>1,312</td>
    <td>2</td>
    <td>28</td>
    <td>C1</td>
  </tr>
  <tr>
    <td>AESW (Daudaravičius et al., 2016)</td>
    <td>143,804</td>
    <td>1</td>
    <td>N/A</td>
    <td>C1–C2 (+Native)</td>
  </tr>
  <tr>
    <td>JFLEG (Napoles et al., 2017)</td>
    <td>747</td>
    <td>4</td>
    <td>N/A</td>
    <td>A1–C2?</td>
  </tr>
  <tr>
    <td>BEA-2019 (Bryant et al., 2019)</td>
    <td>4,384</td>
    <td>5</td>
    <td>25</td>
    <td>A1–C2 (+Native)</td>
  </tr>
  <tr>
    <td>GMEG (Napoles et al., 2019)</td>
    <td>2,960</td>
    <td>4</td>
    <td>N/A</td>
    <td>B1–B2 (+Native)</td>
  </tr>
  <tr>
    <td>CWEB (Flachs et al., 2020)</td>
    <td>6,845</td>
    <td>2</td>
    <td>25</td>
    <td>(Native)</td>
  </tr>
  <tr>
    <td>CTSEG (Ours)</td>
    <td>1,578</td>
    <td>1–3</td>
    <td>263</td>
    <td>A1–B2</td>
  </tr>
</table>

Table 1: A summary of English GEC evaluation datasets and other relevant datasets. A question mark (?) in the CEFR level column indicates unknown or approximated information.

produced texts presents several challenges. For instance, the range of grammar items that can be assessed depends on the content of learners’ essays. As a result, certain grammar items that developers wish to examine (e.g., grammar item C in Figure 1a) may be absent, leaving the model’s performance on those items unknown. From the perspective of learner support, this is undesirable, as it may lead to certain grammar items being introduced to learners with suboptimal correction accuracy. Moreover, when a single sentence contains multiple errors, the corrections may interact with one another (Mita and Yanaka, 2021), making it difficult to isolate and accurately assess the model’s performance on specific grammar items.

To address these issues, we propose integrating Targeted Syntactic Evaluation (TSE) into the GEC evaluation framework (Figure 1b). TSE assesses grammatical and syntactic knowledge by comparing the probabilities assigned to minimal pairs of acceptable and unacceptable sentences, enabling fine-grained performance evaluation across specific linguistic phenomena (Linzen et al., 2016; Marvin and Linzen, 2018; Warstadt et al., 2020; Someya et al., 2024). Although TSE has traditionally been used for evaluating language models, its application to downstream tasks such as GEC is not straightforward. However, unlike other generative tasks, such as machine translation and summarization, GEC inherently aligns with the minimal pair setting, where only specific parts of a sentence are modified rather than the entire sentence. This structural similarity enables the adaptation of the TSE concept for GEC evaluation.

Building on this framework, we introduce the CEFR-based Targeted Syntactic Evaluation Dataset for Grammatical Error Correction (CTSEG), the first dataset developed under this paradigm, designed to facilitate proficiency-aligned evaluations of GEC models.1 Grounded in the CEFR (Council of Europe, 2001), CTSEG ensures that evaluations align with learner proficiency levels, providing a more structured and pedagogically relevant assessment. By constructing minimal pairs for different grammar items, it enables a targeted analysis of model performance on specific grammatical structures. To assess the effectiveness of our dataset, we conducted extensive experiments on three major types of GEC approaches: sequence-to-sequence models, sequence tagging models, and prompt-based models. The results reveal strengths and weaknesses in how different approaches handle specific grammatical phenomena, highlighting limitations that conventional leaderboard-style evaluations fail to capture and underscoring the necessity of fine-grained assessment for advancing GEC research.

In addition, leveraging CTSEG, we conducted an experiment to explore the applicability of prompt-based models—an area of growing interest in recent years—for learner support. Specifically, we investigated whether such models could perform corrections using a target grammar item specified in the prompt, while varying the type of reference sentence used for evaluation. The results showed that the model was indeed capable of incorporating the specified grammar item into its corrections. This suggests a promising feature for learners who wish to focus on practicing specific grammar items.

2 Related Work

2.1 GEC Evaluation

Various evaluation datasets have been published for GEC research. Table 1 lists evaluation datasets for GEC and related resources. The CoNLL-2014

1The CTSEG dataset is publicly available at https://github.com/SDS-NLP/CTSEG.
shared task test set (Ng et al., 2014) is one of the most widely used benchmarks in GEC and is based on the NUCLE corpus (Dahlmeier et al., 2013). The JFLEG test set (Napoles et al., 2017) was developed to assess the fluency-oriented performance of GEC models and is based on the GUG corpus (Heilman et al., 2014). The BEA-2019 shared task test set (Bryant et al., 2019) is derived from the W&I and LOCNESS corpora (Yannakoudakis et al., 2018; Granger, 1998) and includes both learner-written texts categorized by CEFR levels and essays written by native speakers. All these datasets, as well as related resources, are based on essays written by learners or native speakers. Therefore, the types of errors present in these evaluation datasets are not systematically controlled. As a result, these datasets do not necessarily include the specific types of errors that developers need to test in GEC models. To address this issue, we propose the creation of a controlled evaluation dataset designed to assess GEC models on specific error types.

In GEC evaluation, error tags in evaluation datasets can be used to assess model performance across different error types. The CoNLL-2014 test set includes 28 types of error tags, primarily focusing on common learner errors such as tense and article usage. The BEA-2019 test set employs 25 types of error tags based on the ERRANT (Felice et al., 2016; Bryant et al., 2017)2 annotation scheme. However, since it relies on rule-based methods, its error tags are restricted to errors that can be classified using predefined rules. As a result, the error tags in the CoNLL-2014 and BEA-2019 test sets do not allow for a detailed performance analysis of rare errors or those that are difficult to categorize using rule-based methods. In contrast, our tags offer the advantage of providing detailed information about the target grammatical expressions associated with grammatical errors.3 For instance, verb tense errors, which are categorized as a single type in both the CoNLL-2014 and BEA-2019 test sets, are further classified into 15 subcategories in our dataset, including present, present perfect, present progressive, and present perfect progressive. This granularity allows us to pinpoint specific grammatical patterns where GEC systems underperform. Thus, CTSEG enables a more precise measurement of performance for error types that are challenging to evaluate using existing datasets.

2.2 GEC Approaches

Recent GEC research primarily employs three approaches: sequence-to-sequence methods, sequence tagging methods, and prompt-based methods. Sequence-to-sequence methods (Kiyono et al., 2020; Rothe et al., 2021) utilize encoder-decoder architectures, such as Transformer (Vaswani et al., 2017), to generate corrected sentences. The introduction of sequence-to-sequence methods has substantially improved GEC model performance compared to earlier approaches based on language models (Gamon et al., 2008) or classifiers (Dahlmeier and Ng, 2011). Sequence tagging methods (Omelianchuk et al., 2020; Tarnavskyi et al., 2022) predict edit operation tags for input sentences, which are then applied to generate corrected outputs. Compared to sequence-to-sequence methods, sequence tagging methods tend to achieve higher precision but often exhibit lower recall (Omelianchuk et al., 2020). Prompt-based methods (Wu et al., 2023; Davis et al., 2024) leverage large language models, such as GPT-4 (OpenAI et al., 2024), to generate corrected sentences. Compared to other methods, prompt-based methods typically produce more natural and fluent corrections (Davis et al., 2024). Additionally, modifying the prompts allows for some degree of control over the corrections (Loem et al., 2023). In this study, we select and analyze a representative model from each of these three approaches, evaluating their performance using our newly created dataset.

3 The CTSEG Dataset

This section describes the methodology used to construct CTSEG dataset designed for grammar-item-specific evaluation.

3.1 Preliminary Preparation

This subsection provides an overview of the annotators involved in creating minimal pairs and presents the grammar item list used as evaluation criteria for GEC models.

Annotators We recruited two experienced English teachers to create minimal pairs of ungrammatical and grammatical sentences. Both teachers have over 10 years of experience teaching English. They possess extensive expertise in coaching English composition and grammar. These experienced educators were selected to ensure that the

2 ERRANT is a rule-based tool that automatically identifies error types and assigns corresponding error tags.
3 A list of grammar items included in CTSEG can be found in Appendix A.
<table>
  <tr>
    <th>ID</th>
    <th>CL</th>
    <th>Grammar Item</th>
    <th>Sentence</th>
    <th>Procedure</th>
  </tr>
  <tr>
    <td>12</td>
    <td>A1</td>
    <td>Demonstrative adjective (these/those + noun)</td>
    <td>*<b>That</b> pants are too big.<br><b>Those</b> pants are too big.</td>
    <td>Step 2<br>Step 2</td>
  </tr>
  <tr>
    <td>39</td>
    <td>A2</td>
    <td>Comparative (superiority) (more + adjective/adverb)</td>
    <td>*This dictionary is <b>most</b> useful than the one you bought.<br>This dictionary is <b>more</b> useful than the one you bought.</td>
    <td>Step 2<br>Step 2</td>
  </tr>
  <tr>
    <td>173</td>
    <td>B1</td>
    <td>Relative pronoun (nominative) (which)</td>
    <td>*I am looking for a game <b>what</b> will amuse my children.<br>I am looking for a game <b>which</b> will amuse my children.<br>I am looking for a game <b>that</b> will amuse my children.</td>
    <td>Step 2<br>Step 2<br>Step 3</td>
  </tr>
  <tr>
    <td>175</td>
    <td>B2</td>
    <td>Relative pronoun (objective) (who)</td>
    <td>*The person <b>which</b> I met yesterday is a doctor.<br>The person <b>who</b> I met yesterday is a doctor.<br>The person <b>that</b> I met yesterday is a doctor.<br>The person <b>that</b> I met yesterday is a doctor.</td>
    <td>Step 2<br>Step 2<br>Step 3<br>Step 3</td>
  </tr>
</table>

Table 2: Examples of minimal pairs for each grammar item. The ID column represents a unique number assigned to each grammar item. The CL column specifies the CEFR level associated with each grammar item. An asterisk (*) denotes an ungrammatical sentence.

ungrammatical expressions in the sentences accurately reflect common student errors. Additionally, involving multiple annotators in the sentence creation process helps mitigate potential biases in error representation.

Grammar Items List In this study, we use the CEFR-J Grammar Profile (Ishii and Tono, 2018) as the grammar item list for evaluating GEC models.4 The CEFR-J Grammar Profile primarily targets grammar items taught to English learners in Japan and incorporates insights from other established frameworks, including the T-series (Van Ek and Trim, 1991, 2001a,b), the Core Inventory for General English5, and the English Grammar Profile6. It defines 263 grammar items, each mapped to a specific CEFR level (see Table 2 for examples). The reason we chose the CEFR-J Grammar Profile is that it defines grammar items not only based on observations of learner errors but also on the learning objectives specified for each CEFR level. This dual foundation makes it particularly well suited for evaluating GEC models at specific proficiency levels, as it allows us to assess whether a model can handle grammar items that learners are expected to master. Furthermore, because the grammar items closely reflect what annotators are accustomed to teaching, they support the creation of authentic and pedagogically meaningful minimal pairs.

4CEFR-J is a framework adapted from CEFR for English education in Japan. As part of the CEFR-J project, the CEFR-J Grammar Profile was developed to provide a grammar item list aligned with this framework. For this study, we used version 20200220 of the CEFR-J Grammar Profile.
5https://www.eaquals.org/resources/the-core-inventory-for-general-english
6https://www.englishprofile.org/english-grammar-profile

3.2 Construction Scheme
This subsection outlines the procedures and guidelines for dataset creation.

Procedures Each annotator described in Section 3.1 followed the steps below to create the dataset. Since GEC corrections can have multiple valid forms (Bryant and Ng, 2015; Choshen and Abend, 2018), the process involved generating both minimal pairs of ungrammatical and grammatical sentences as well as multiple grammatical variants for a single ungrammatical sentence.

Step 1. Identify and understand the target grammar item.

Step 2. Construct an ungrammatical sentence containing an error related to the target grammar item, along with its corresponding grammatical version that correctly applies the item.

Step 3. If alternative corrections exist, generate additional grammatical sentences that do not depend on the target grammar item.

Each annotator followed this procedure for every grammar item, generating three sentence pairs per item.7 During Step 1, if ambiguities arose regarding the target grammar item, annotators and authors discussed them for clarification. In contrast, for Steps 2 and 3, each annotator worked independently to create ungrammatical and grammatical sentences. To ensure quality, we hired an additional

7The number of sentence pairs for each grammar item was determined based on budgetary constraints.
checker8, in addition to the two annotators, to assess the produced text for compliance with the rules described below and to provide feedback. Sentence pairs deemed inappropriate by the checker were returned to the annotators for revision, and this process was repeated until the checker approved the sentence pair.9

Rules To ensure the reliability of the dataset, we established four guidelines for sentence creation.

Rule 1. Errors in ungrammatical sentences must be limited to the target grammar item.

Rule 2. In Step 3, grammatical sentences should be generated for all relevant correction patterns.

Rule 3. Sentences should be simple and reflect the writing style typically used by language learners.

Rule 4. Within each grammar item, errors should reflect common mistakes made by English learners.

Since GEC errors can interact, Rule 1 prevents unintended errors from affecting correction performance (Mita and Yanaka, 2021). Reference-based evaluation methods may penalize valid corrections if they do not appear in the reference sentence (Bryant and Ng, 2015; Choshen and Abend, 2018). Rule 2 ensures a more comprehensive assessment of GEC model performance. Additionally, since GEC models are influenced by the text domain (Napoles et al., 2019; Mita et al., 2019; Flachs et al., 2020), Rules 3 and 4 help ensure that the evaluation conditions closely resemble real-world applications.10 Table 2 presents examples of actual minimal pairs across different CEFR levels. Each ungrammatical sentence contains only one error related to the target grammar item. The grammatical sentence in Step 2 corrects the error using the target grammar item, while the sentences in Step 3 employ alternative correction strategies. Alternative corrections in Step 3 were generated only when applicable.

8The reviewer is a certified English teacher with approximately 20 years of experience in developing educational materials for middle and high school English language learners.
9Approximately 29% of the sentence pairs were returned to the annotators by the checker at least once for revision.
10In this study, we developed CTSEG following the conventions of American English typically taught to English learners.

![Edit coverage of each test/dev set relative to BEA-train, computed by varying the frequency threshold and retaining only edits that appear in BEA-train at or above that threshold.](page_768_120_484_384.png)

Figure 2: Edit coverage of each test/dev set relative to BEA-train, computed by varying the frequency threshold and retaining only edits that appear in BEA-train at or above that threshold.

Post-processing To prepare the dataset for GEC model evaluation, we applied the following post-processing steps. First, we tokenized all sentences using the spaCy toolkit (Honnibal and Montani, 2017). Second, we converted the dataset into M^2 format using ERRANT.11 Finally, we divided the resulting M^2 file into separate files for each grammar item. These M^2 files will be publicly available for research purposes.

3.3 Edit Coverage

In this study, experienced English teachers created erroneous sentences to mimic common learner errors, which were then carefully reviewed by an independent checker. This section investigates the extent to which such artificially constructed errors overlap with those found in actual learner corpora. Specifically, we analyze how many of the edits in CTSEG overlap with those in BEA-train—a learner corpus consisting of FCE, the Lang-8 Corpus (Mizumoto et al., 2011), NUCLE, and W&I+LOCNESS, containing 564,684 sentence pairs. The overlap between CTSEG and BEA-train, referred to as edit coverage, is calculated as follows:

\[
\text{Edit coverage} := \frac{|E_{CTSEG} \cap E_{BEA\text{-}train}|}{|E_{CTSEG}|}
\]

Here, \( E_{CTSEG} \) denotes the set of edits in CTSEG, and \( E_{BEA\text{-}train} \) denotes the set of edits in BEA-train.

Figure 2 presents the edit coverage for CTSEG. For comparison, we also measured the edit coverage of CoNLL-2014 and BEA-dev using the same

11We use ERRANT only to convert the data into M^2 format for evaluation with the M^2 scorer (Dahlmeier and Ng, 2012). In the experiments, we evaluate each individual M^2 file based on CTSEG error categories rather than ERRANT labels.
method. To focus on more frequently occurring errors, we applied a frequency threshold to the edits in BEA-train. A threshold of 1 includes all edits, while a threshold of 2 includes only those that appear at least twice. As shown in Figure 2, approximately 60% of the edits in CTSEG also appear at least once in BEA-train. This ratio is comparable to those of CoNLL-2014 and BEA-dev, indicating that CTSEG effectively mimics learner-like errors. Furthermore, even as the frequency threshold increases, the edit coverage of CTSEG remains similar to that of CoNLL-2014 and BEA-dev. This suggests that CTSEG captures frequent edits typical of actual learner corpora.

3.4 Comparing Datasets

Table 1 summarizes the dataset statistics, including its size, the number of reference sentences, the number of error tag types, and CEFR levels. Our dataset contains a comparable number of sentences to the CoNLL-2014 shared task test set, a widely used GEC benchmark. Moreover, each ungrammatical sentence is assigned up to three grammatical reference sentences when necessary. A notable distinction of our dataset is its large number of error tags, totaling 263, which reflects its foundation in the CEFR-J Grammar Profile. Additionally, our dataset covers grammar items from CEFR levels A1 to B2, aligning with CEFR-J. However, it does not yet include C1 or C2-level grammar items, highlighting the need for future expansion.

To compare correction tendencies, we analyze the distribution of edit distances (Levenshtein, 1966) across datasets. Figure 3 shows word-level edit distance distributions for CTSEG, the CoNLL-2014 test set, and the JFLEG test set.12 We use the first target sentence in each dataset as the reference. CoNLL-2014 follows a minimal edit approach, while JFLEG focuses on fluency edits. Figure 3 shows that CTSEG has a smaller edit distance than CoNLL-2014, as it targets one error per sentence, allowing for item-specific evaluation. Since real-world sentences often contain multiple errors, existing benchmarks better reflect practical GEC conditions. Thus, depending on the evaluation goal, CTSEG and existing benchmarks can be used separately or together.

![Distribution of word-level edit distances between source and target sentences across datasets.](page_1012_180_377_246.png)

Figure 3: Distribution of word-level edit distances between source and target sentences across datasets.

4 Experiments

4.1 GEC Models

As mentioned in Section 2.2, current GEC research is primarily guided by three main paradigms. For our experiments, we selected representative models from each paradigm, as detailed below.

Transformer This model follows an encoder-decoder architecture (Vaswani et al., 2017). We used the Transformer-based model proposed by Kiyono et al. (2020), pre-trained on 70 million pseudo data and fine-tuned on real data.13 Previous studies (Kiyono et al., 2020; Kaneko et al., 2020) have shown that this paradigm generally achieves higher recall than sequence tagging models like GECToR (Omelianchuk et al., 2020).

GECToR This model is based on a sequence tagging architecture. We adopted the model proposed by Omelianchuk et al. (2020), which employs approximately 5,000 tags, and RoBERTa (Liu et al., 2019) for tag prediction.14 Prior research (Omelianchuk et al., 2020; Tarnavskyi et al., 2022) has demonstrated that sequence tagging models generally achieve higher precision than encoder-decoder architectures.

GPT-4 This model is a large language model (LLM) that generates corrections based on prompts. We used GPT-4 (OpenAI et al., 2024)15, developed by OpenAI16. A key characteristic of GPT-4 is that its output varies based on prompt design (Coyne et al., 2023; Loem et al., 2023; Davis et al., 2024). For this study, we employed the English teacher

12 Appendix B shows example sentence pairs from CTSEG for each word-level edit distance.
13 https://github.com/butsugiri/gec-pseudodata
14 https://github.com/grammarly/gector
15 gpt-4-0613
16 https://openai.com
prompt from Davis et al. (2024), which achieved the highest performance in their study.17

4.2 Evaluation
We evaluate model performance using our dataset, considering two types of scores: overall scores and individual scores. Overall scores aggregate grammar item scores across the dataset or by CEFR level, providing insights into general correction trends in GEC models. In contrast, individual scores assess performance for each grammar item separately, enabling a fine-grained analysis of a model’s ability to correct specific grammatical errors. The choice between overall and individual scores depends on the evaluation objective.

The core contribution of our evaluation paradigm lies in individual scores, which allow for a detailed analysis of model performance across specific grammatical items. This level of granularity is essential for diagnosing strengths and weaknesses that remain hidden in conventional aggregate evaluations. Since our evaluation paradigm is designed to complement rather than replace existing leaderboard-style evaluations, our primary focus is on assessing how accurately models correct specific grammatical items. While conventional evaluations offer a broad assessment of overall model capability, our approach enables a more targeted analysis by examining correction accuracy at the individual grammar item level.

To compute these scores, we use the M^2 scorer (Dahlmeier and Ng, 2012). For overall evaluation, we report Precision, Recall, and F_{0.5}, whereas for individual evaluation, we focus on Recall, as it directly measures the extent to which the targeted errors are corrected. By integrating both overall and individual evaluations, our framework provides a comprehensive perspective on GEC model performance, balancing general correction trends with fine-grained linguistic insights.

4.3 Results
Overall Scores Table 3 summarizes model performance across the entire dataset and by CEFR level. GPT-4 outperforms both Transformer and GECToR across all metrics. Consistent with previous studies (Omelianchuk et al., 2020; Tarnavskyi et al., 2022), Transformer achieves relatively high recall, reflecting its strength in covering a broad range of errors. GECToR demonstrates higher precision, which aligns with its sequence tagging-based architecture. Regarding CEFR-level performance, all models exhibit notably lower performance at the B2 level compared to other levels. This suggests that complex grammar items at this level pose greater challenges for GEC models. While overall scores reveal general trends, the following analysis of individual scores focuses on specific grammar items where performance is particularly weak.

<table>
  <tr>
    <th>GEC Model</th>
    <th>Category</th>
    <th>P</th>
    <th>R</th>
    <th>F<sub>0.5</sub></th>
  </tr>
  <tr>
    <td rowspan="5">Transformer</td>
    <td>ALL</td>
    <td>76.81</td>
    <td>77.25</td>
    <td>76.53</td>
  </tr>
  <tr>
    <td>A1</td>
    <td>78.91</td>
    <td>79.52</td>
    <td>78.74</td>
  </tr>
  <tr>
    <td>A2</td>
    <td>79.41</td>
    <td>81.75</td>
    <td>79.58</td>
  </tr>
  <tr>
    <td>B1</td>
    <td>80.42</td>
    <td>79.25</td>
    <td>79.79</td>
  </tr>
  <tr>
    <td>B2</td>
    <td>69.63</td>
    <td>70.34</td>
    <td>69.35</td>
  </tr>
  <tr>
    <td rowspan="5">GECToR</td>
    <td>ALL</td>
    <td>79.83</td>
    <td>57.83</td>
    <td>71.66</td>
  </tr>
  <tr>
    <td>A1</td>
    <td>85.11</td>
    <td>60.42</td>
    <td>75.07</td>
  </tr>
  <tr>
    <td>A2</td>
    <td>84.80</td>
    <td>63.80</td>
    <td>77.79</td>
  </tr>
  <tr>
    <td>B1</td>
    <td>78.42</td>
    <td>57.44</td>
    <td>70.42</td>
  </tr>
  <tr>
    <td>B2</td>
    <td>73.75</td>
    <td>52.19</td>
    <td>66.15</td>
  </tr>
  <tr>
    <td rowspan="5">GPT-4</td>
    <td>ALL</td>
    <td>83.47</td>
    <td>85.29</td>
    <td>83.64</td>
  </tr>
  <tr>
    <td>A1</td>
    <td>85.68</td>
    <td>86.48</td>
    <td>85.68</td>
  </tr>
  <tr>
    <td>A2</td>
    <td>85.05</td>
    <td>86.81</td>
    <td>85.22</td>
  </tr>
  <tr>
    <td>B1</td>
    <td>86.36</td>
    <td>87.78</td>
    <td>86.49</td>
  </tr>
  <tr>
    <td>B2</td>
    <td>77.64</td>
    <td>80.74</td>
    <td>78.00</td>
  </tr>
</table>

Table 3: Overall scores for each GEC model. The values represent macro-average scores for grammar items in each category. Darker shades of red indicate lower scores.

Individual Scores Table 4 presents the five grammar items with the lowest average recall for each CEFR level. The results demonstrate that GEC model performance varies across different grammar items. For example, for the grammar item “Imperative sentences (general verb),” GPT-4 achieves a recall of 83.33, whereas GECToR’s recall is 0. Beyond this specific grammar item, GECToR also fails to correct errors in other cases, with 8 out of 263 grammar items receiving no corrections at all. This limitation could not be identified using conventional benchmarks, highlighting the advantage of our proposed CTSEG benchmark.

Table 4 also indicates that certain grammar items, such as questions and imperative sentences, are hard to correct across the representative models. These grammar items are rarely used in existing benchmarks, such as the CoNLL-2014 test set, due to the nature of their content. As a result, despite their low performance, these grammar items have been largely overlooked in existing benchmarks. To enhance the effectiveness of GEC models in sup-

17For specific prompts, see the first row of Table 12 in Appendix C.
<table>
  <tr>
    <th>ID</th>
    <th>CL</th>
    <th>Grammar Item</th>
    <th>Transformer</th>
    <th>GECToR</th>
    <th>GPT-4</th>
    <th>Avg.</th>
  </tr>
  <tr>
    <td>168</td>
    <td>A1</td>
    <td>Present participle modifying a noun (pre-position)</td>
    <td>50.00</td>
    <td>0.00</td>
    <td>66.67</td>
    <td>38.89</td>
  </tr>
  <tr>
    <td>241</td>
    <td>A1</td>
    <td>Interrogative sentence (Which . . .?)</td>
    <td>50.00</td>
    <td>33.33</td>
    <td>50.00</td>
    <td>44.44</td>
  </tr>
  <tr>
    <td>117</td>
    <td>A1</td>
    <td>Imperative sentences (general verb)</td>
    <td>66.67</td>
    <td>0.00</td>
    <td>83.33</td>
    <td>50.00</td>
  </tr>
  <tr>
    <td>263</td>
    <td>A1</td>
    <td>Functional interrogative (What about . . .?)</td>
    <td>57.14</td>
    <td>42.86</td>
    <td>57.14</td>
    <td>52.38</td>
  </tr>
  <tr>
    <td>25</td>
    <td>A1</td>
    <td>Indefinite pronoun (supportive word “ones”)</td>
    <td>33.33</td>
    <td>66.67</td>
    <td>66.67</td>
    <td>55.56</td>
  </tr>
  <tr>
    <td>244</td>
    <td>A2</td>
    <td>Interrogative sentence (Whose + noun . . .?)</td>
    <td>16.67</td>
    <td>16.67</td>
    <td>66.67</td>
    <td>33.33</td>
  </tr>
  <tr>
    <td>243</td>
    <td>A2</td>
    <td>Interrogative sentence (Whose . . .?)</td>
    <td>30.00</td>
    <td>10.00</td>
    <td>60.00</td>
    <td>33.33</td>
  </tr>
  <tr>
    <td>191</td>
    <td>A2</td>
    <td>Exclamatory sentence (How alone)</td>
    <td>50.00</td>
    <td>16.67</td>
    <td>50.00</td>
    <td>38.89</td>
  </tr>
  <tr>
    <td>118</td>
    <td>A2</td>
    <td>Emphasis in affirmative imperative sentences with do</td>
    <td>50.00</td>
    <td>50.00</td>
    <td>50.00</td>
    <td>50.00</td>
  </tr>
  <tr>
    <td>179</td>
    <td>A2</td>
    <td>Omission of relative pronoun (objective)</td>
    <td>50.00</td>
    <td>50.00</td>
    <td>66.67</td>
    <td>55.56</td>
  </tr>
  <tr>
    <td>30</td>
    <td>B1</td>
    <td>Pronoun (the other/others)</td>
    <td>16.67</td>
    <td>33.33</td>
    <td>50.00</td>
    <td>33.33</td>
  </tr>
  <tr>
    <td>116</td>
    <td>B1</td>
    <td>Imperative sentences (be verb)</td>
    <td>33.33</td>
    <td>0.00</td>
    <td>66.67</td>
    <td>33.33</td>
  </tr>
  <tr>
    <td>45</td>
    <td>B1</td>
    <td>so + adjective/adverb + (that) clause</td>
    <td>57.14</td>
    <td>0.00</td>
    <td>57.14</td>
    <td>38.10</td>
  </tr>
  <tr>
    <td>220</td>
    <td>B1</td>
    <td>as if/though + subjunctive past</td>
    <td>50.00</td>
    <td>16.67</td>
    <td>50.00</td>
    <td>38.89</td>
  </tr>
  <tr>
    <td>184</td>
    <td>B1</td>
    <td>Preposition + relative pronoun</td>
    <td>66.67</td>
    <td>16.67</td>
    <td>50.00</td>
    <td>44.44</td>
  </tr>
  <tr>
    <td>214</td>
    <td>B2</td>
    <td>Participial construction (past participle at the beginning)</td>
    <td>16.67</td>
    <td>0.00</td>
    <td>33.33</td>
    <td>16.67</td>
  </tr>
  <tr>
    <td>182</td>
    <td>B2</td>
    <td>Pseudo-relative pronoun (as)</td>
    <td>28.57</td>
    <td>0.00</td>
    <td>50.00</td>
    <td>26.19</td>
  </tr>
  <tr>
    <td>213</td>
    <td>B2</td>
    <td>Participial construction (present participle at the beginning)</td>
    <td>42.86</td>
    <td>14.29</td>
    <td>28.57</td>
    <td>28.57</td>
  </tr>
  <tr>
    <td>80</td>
    <td>B2</td>
    <td>Passive voice (future continuous)</td>
    <td>41.67</td>
    <td>16.67</td>
    <td>41.67</td>
    <td>33.33</td>
  </tr>
  <tr>
    <td>218</td>
    <td>B2</td>
    <td>wish + subjunctive past</td>
    <td>33.33</td>
    <td>33.33</td>
    <td>33.33</td>
    <td>33.33</td>
  </tr>
</table>

Table 4: Recall for each grammar item across GEC models. For each CEFR level, the five grammar items with the lowest average recall were selected.

porting language learners, future research should focus on improving the performance of these underrepresented grammar items.

5 Leveraging GPT-4 to Support Language Learners

In the previous section, we found that the performance of GEC models is low for certain grammar items. In this section, we examine whether incorporating grammar item information into prompts enhances GPT-4’s performance, assuming that it will be used to assist language learners. Additionally, we investigate whether specifying grammar items enables GPT-4 to refine its corrections to ensure the inclusion of the target grammar items.

5.1 Motivation

An example application of GEC models is supporting learners in verifying a specific grammar item they are striving to master. For instance, after learning relative pronouns in class, a learner writes a sentence using a relative pronoun and inputs it into a GEC model to check whether it is used correctly. In such cases, it is reasonable to assume that information about the target grammar item can be integrated into the prompts. Furthermore, from the perspective of learner support, the model should not only correct errors but also ensure that the target grammar item is preserved in the correction. However, since GEC models often allow for multiple valid corrections, the learner’s intended grammar item may not always be present in the output.

To address this issue, we investigate whether modifying the prompt enhances GPT-4’s performance and enables it to generate corrections that explicitly include the target grammar item. This experiment assumes a scenario in which the learner’s target grammar item is predetermined, as described in the use case above. Therefore, we hypothesize that explicitly incorporating information about the target grammar item into the prompt can effectively guide GPT-4’s output.

5.2 Settings

Prompts We evaluate two types of prompts: a baseline prompt and a grammar-item-guided prompt. The baseline prompt, adapted from Davis et al. (2024) and used in Section 4, allows GPT-4 to generate corrections without imposing any constraints on grammar item usage. In contrast, the grammar-item-guided prompt explicitly specifies the target grammar item that the learner aims to practice.18 By explicitly guiding GPT-4, this prompt encourages the model to prioritize corrections that incorporate the specified grammar item.

Evaluation In addition to standard metrics, we introduce a novel metric called targeted recall to

18For specific prompts, see the second row of Table 12 in Appendix C.
<table>
  <tr>
    <th>Sentence (target grammar item: “Participial construction (present participle at the beginning)”)</th>
    <th>Perplexity</th>
  </tr>
  <tr>
    <td>Source</td>
    <td><b>The boy seeing</b> me , he ran away .</td>
    <td>289.48</td>
  </tr>
  <tr>
    <td>Baseline prompt (same as a reference at Step 3)</td>
    <td><b>When the boy saw</b> me, he ran away .</td>
    <td>23.59</td>
  </tr>
  <tr>
    <td>Grammar-item-guided prompt (same as a reference at Step 2)</td>
    <td><b>Seeing</b> me , the boy ran away .</td>
    <td>42.50</td>
  </tr>
</table>

Table 5: Example outputs for each prompt with the target grammar item.

evaluate whether GEC models’ corrections incorporate the target grammar item. Specifically, we assess the model using only the grammatical sentences generated in Step 2 of the dataset construction procedure as reference sentences. Since Step 2 involves generating grammatical sentences that explicitly include the target grammar item, this evaluation allows us to quantify the extent to which the model adheres to the target grammar item constraint in its corrections.

5.3 Results

Table 6 presents the evaluation results for each prompt. Single Ref refers to the evaluation using only grammatical sentences that include the target grammar item, while All Ref refers to the evaluation using all grammatical sentences. As shown in Table 6, GPT-4’s performance improves when incorporating the target grammar item into the prompt. The baseline prompt achieves a recall of 78.33 when evaluated against grammar-item-specific references, whereas explicitly specifying the target grammar item in the prompt increases recall to 91.67.

<table>
  <tr>
    <th>Prompt</th>
    <th>Single Ref</th>
    <th>All Ref</th>
  </tr>
  <tr>
    <td>Baseline</td>
    <td>78.33</td>
    <td>85.29</td>
  </tr>
  <tr>
    <td>Grammar-item-guided</td>
    <td>91.67</td>
    <td>94.02</td>
  </tr>
  <tr>
    <td>Δ</td>
    <td>+13.34</td>
    <td>+8.73</td>
  </tr>
</table>

Table 6: Targeted recall of GPT-4 for each prompt.

models. CTSEG is a relatively small dataset, with a size comparable to that of CoNLL-2014 and JFLEG. Nevertheless, as shown in Table 4, CTSEG successfully reveals model-specific weaknesses that are not easily captured by existing benchmarks, even with its limited size. These findings highlight the distinct capabilities and limitations of different models, thereby offering valuable insights for future development and evaluation of GEC systems.

We hope that CTSEG will be used alongside existing benchmarks. CTSEG enables fine-grained analysis of model weaknesses, making it a valuable resource for comprehensive GEC evaluation. For future work, given that manual dataset creation is resource-intensive, we plan to explore automated generation methods—including rule-based approaches and data augmentation using large language models—to enhance scalability. In addition, we aim to investigate the extent to which humans can correct errors made by annotators in CTSEG. This investigation will help determine whether the errors that GEC models fail to correct can instead be corrected by humans, or whether they are inherently difficult for both models and humans to address. We believe that this analysis will provide valuable insights for advancing GEC research.

6 Conclusions

In this study, we proposed integrating TSE into the GEC evaluation framework. As the first evaluation dataset based on TSE, we developed CTSEG to facilitate proficiency-aligned evaluations of GEC

Limitations

Dataset Size Limitation One limitation of this study is the relatively small number of minimal pairs per grammar item. In this study, minimal pairs were manually created by two English teachers to ensure high-quality evaluation data that adhered to the rules outlined in Section 3.2. To construct larger and more reliable evaluation datasets in the future, it is essential to explore methods for automating the generation of minimal pairs.
Error Operation Ratio Limitation   One limitation of CTSEG is that it does not control the ratio of error operations (i.e., Missing, Replacement, Unnecessary). As a result, the distribution of error operations may differ from that of existing test sets, such as CoNLL-2014 and BEA-2019. In this study, we deliberately chose not to control the operation ratio in order to reduce the burden on annotators and to allow for the creation of more natural, learner-like errors. However, Davis et al. (2024) has shown that GEC model performance can vary depending on the type of error operation, suggesting that evaluation at the operation level is important for GEC research. To enable more fine-grained analysis, it would be desirable in future work to construct test sets in which the distribution of error operations is explicitly controlled for each grammar item.

Language Limitation   In this study, we constructed a dataset for English to facilitate the evaluation of GEC models at the grammar item level. However, GEC research has also been conducted in languages such as Chinese (Tang et al., 2023; Yang and Quan, 2024) and Russian (Rozovskaya and Roth, 2019; Palma Gomez and Rozovskaya, 2024), where learner-produced texts are utilized in a similar manner to English. Therefore, grammar-item-specific evaluation datasets can also be developed for languages other than English.

Ethical Considerations

We collaborated with an annotation company to create our evaluation dataset. Through this company, we provided appropriate compensation to two English teachers and a reviewer. Specifically, each English teacher received 300 yen per sentence, while the reviewer was compensated 50 yen per sentence. Including fees and commissions, the total cost of dataset production amounted to approximately 1.5 million yen, exceeding the average wage in Japan.

Acknowledgments

This work was partly supported by JSPS KAKENHI Grant Numbers 22H03651 and 25K03178.

References

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language Models are Few-Shot Learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877–1901.

Christopher Bryant, Mariano Felice, Øistein E. Andersen, and Ted Briscoe. 2019. The BEA-2019 Shared Task on Grammatical Error Correction. In Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 52–75.

Christopher Bryant, Mariano Felice, and Ted Briscoe. 2017. Automatic Annotation and Evaluation of Error Types for Grammatical Error Correction. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 793–805.

Christopher Bryant and Hwee Tou Ng. 2015. How Far are We from Fully Automatic High Quality Grammatical Error Correction? In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing, pages 697–707.

Christopher Bryant, Zheng Yuan, Muhammad Reza Qorib, Hannan Cao, Hwee Tou Ng, and Ted Briscoe. 2023. Grammatical Error Correction: A Survey of the State of the Art. Computational Linguistics, pages 643–701.

Leshem Choshen and Omri Abend. 2018. Inherent Biases in Reference-based Evaluation for Grammatical Error Correction. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, pages 632–642.

Council of Europe. 2001. Common European Framework of Reference for Languages: Learning, Teaching, Assessment. Cambridge University Press, Cambridge.

Steven Coyne, Keisuke Sakaguchi, Diana Galvan-Sosa, Michael Zock, and Kentaro Inui. 2023. Analyzing the Performance of GPT-3.5 and GPT-4 in Grammatical Error Correction. Preprint, arXiv:2303.14342.

Daniel Dahlmeier and Hwee Tou Ng. 2011. Grammatical Error Correction with Alternating Structure Optimization. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 915–923.

Daniel Dahlmeier and Hwee Tou Ng. 2012. Better Evaluation for Grammatical Error Correction. In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 568–572.
Daniel Dahlmeier, Hwee Tou Ng, and Siew Mei Wu. 2013. Building a Large Annotated Corpus of Learner English: The NUS Corpus of Learner English. In Proceedings of the Eighth Workshop on Innovative Use of NLP for Building Educational Applications, pages 22–31.

Vidas Daudaravicius, Rafael E. Banchs, Elena Volodina, and Courtney Napoles. 2016. A Report on the Automatic Evaluation of Scientific Writing Shared Task. In Proceedings of the 11th Workshop on Innovative Use of NLP for Building Educational Applications, pages 53–62.

Christopher Davis, Andrew Caines, O Andersen, Shiva Taslimipoor, Helen Yannakoudakis, Zheng Yuan, Christopher Bryant, Marek Rei, and Paula Buttery. 2024. Prompting open-source and commercial language models for grammatical error correction of English learner text. In Findings of the Association for Computational Linguistics ACL 2024, pages 11952–11967.

Tao Fang, Shu Yang, Kaixin Lan, Derek F. Wong, Jinpeng Hu, Lidia S. Chao, and Yue Zhang. 2023. Is ChatGPT a Highly Fluent Grammatical Error Correction System? A Comprehensive Evaluation. Preprint arXiv:2304.01746.

Mariano Felice, Christopher Bryant, and Ted Briscoe. 2016. Automatic Extraction of Learner Errors in ESL Sentences Using Linguistically Enhanced Alignments. In Proceedings of the 26th International Conference on Computational Linguistics: Technical Papers, pages 825–835.

Simon Flachs, Ophélie Lacroix, Helen Yannakoudakis, Marek Rei, and Anders Søgaard. 2020. Grammatical Error Correction in Low Error Density Domains: A New Benchmark and Analyses. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 8467–8478.

Michael Gamon, Jianfeng Gao, Chris Brockett, Alexandre Klementiev, William B. Dolan, Dmitriy Belenko, and Lucy Vanderwende. 2008. Using Contextual Speller Techniques and Language Modeling for ESL Error Correction. In Proceedings of the Third International Joint Conference on Natural Language Processing, pages 449–456.

Sylviane Granger. 1998. The computerized learner corpus: a versatile new source of data for SLA research. In Learner English on Computer, pages 3–18. Addison Wesley Longman.

Michael Heilman, Aoife Cahill, Nitin Madnani, Melissa Lopez, Matthew Mulholland, and Joel Tetreault. 2014. Predicting Grammaticality on an Ordinal Scale. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, pages 174–180.

Matthew Honnibal and Ines Montani. 2017. spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing. To appear.

Yasutake Ishii and Yukio Tono. 2018. Investigating Japanese EFL Learners’ Overuse/Underuse of English Grammar Categories and Their Relevance to CEFR Levels. In Proceedings of the 4th Asia Pacific Corpus Linguistics Conference, pages 160–165.

Masahiro Kaneko, Masato Mita, Shun Kiyono, Jun Suzuki, and Kentaro Inui. 2020. Encoder-Decoder Models Can Benefit from Pre-trained Masked Language Models in Grammatical Error Correction. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4248–4254.

Shun Kiyono, Jun Suzuki, Tomoya Mizumoto, and Kentaro Inui. 2020. Massive Exploration of Pseudo Data for Grammatical Error Correction. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 28:2134–2145.

Vladimir Iosifovich Levenshtein. 1966. Binary Codes Capable of Correcting Deletions, Insertions and Reversals. Soviet Physics Doklady, 10(8):707–710.

Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. 2016. Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies. Transactions of the Association for Computational Linguistics, 4:521–535.

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A Robustly Optimized BERT Pretraining Approach. Preprint, arXiv:1907.11692.

Mengsay Loem, Masahiro Kaneko, Sho Takase, and Naoki Okazaki. 2023. Exploring Effectiveness of GPT-3 in Grammatical Error Correction: A Study on Performance and Controllability in Prompt-Based Methods. In Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications, pages 205–219.

Rebecca Marvin and Tal Linzen. 2018. Targeted Syntactic Evaluation of Language Models. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1192–1202.

Masato Mita, Tomoya Mizumoto, Masahiro Kaneko, Ryo Nagata, and Kentaro Inui. 2019. Cross-Corpora Evaluation and Analysis of Grammatical Error Correction Models — Is Single-Corpus Evaluation Enough? In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1309–1314.

Masato Mita and Hitomi Yanaka. 2021. Do Grammatical Error Correction Models Realize Grammatical Generalization? In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 4554–4561.

Tomoya Mizumoto, Mamoru Komachi, Masaaki Nagata, and Yuji Matsumoto. 2011. Mining Revision Log
of Language Learning SNS for Automated Japanese Error Correction of Second Language Learners. In Proceedings of 5th International Joint Conference on Natural Language Processing, pages 147–155.

Ryo Nagata, Edward Whittaker, and Vera Sheinman. 2011. Creating a manually error-tagged and shallow-parsed learner corpus. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 1210–1219.

Courtney Napoles, Maria Nädejde, and Joel Tetreault. 2019. Enabling Robust Grammatical Error Correction in New Domains: Data Sets, Metrics, and Analyses. Transactions of the Association for Computational Linguistics, 7:551–566.

Courtney Napoles, Keisuke Sakaguchi, and Joel Tetreault. 2017. JFLEG: A fluency corpus and benchmark for grammatical error correction. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics, pages 229–234.

Hwee Tou Ng, Siew Mei Wu, Ted Briscoe, Christian Hadiwinoto, Raymond Hendy Susanto, and Christopher Bryant. 2014. The CoNLL-2014 Shared Task on Grammatical Error Correction. In Proceedings of the Eighteenth Conference on Computational Natural Language Learning: Shared Task, pages 1–14.

Kostiantyn Omelianchuk, Vitaliy Atrasevych, Artem Chernodub, and Oleksandr Skurzhanskyi. 2020. GECToR – Grammatical Error Correction: Tag, Not Rewrite. In Proceedings of the Fifteenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 163–170.

OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmid, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgium, Irwan Bello, Jake Berdine, Gabriel Bernardett-Shapiro, Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman, Justin Forte, Isabella Fulford, Leo Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Gontijo-Lopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Hee-woo Jun, Tomer Kaftan, Łukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Jan Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Łukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Manning, Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Reichiro Nakano, Rajeev Nayak, Arvind Neelakantan, Richard Ngo, Hyeonwoo Nob, Long Ouyang, Cullen O’Keefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emry Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael Pokorny, Michelle Pokrass, Vitchyr H. Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Proehl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schurn, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin Sokolowsky, Yang Song, Natalie Staudacher, Felipe Petroski Such, Natalie Summers, Ilya Sutskever, Jie Tang, Nikolas Tezak, Madeleine B. Thompson, Phil Tillett, Amin Tootoonchian, Elizabeth Tseng, Preston Tuggle, Nick Turley, Jerry Twork, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vijayaragya, Chelsea Voss, Carroll Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang, Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojciech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. 2024. GPT-4 Technical Report. Preprint, arXiv:2303.08774.

Frank Palma Gomez and Alla Rozovskaya. 2024. Multi-
Reference Benchmarks for Russian Grammatical Error Correction. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics, pages 1253–1270.

Sascha Rothe, Jonathan Mallinson, Eric Malmi, Sebastian Krause, and Aliaksei Severyn. 2021. A Simple Recipe for Multilingual Grammatical Error Correction. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 702–707.

Alla Rozovskaya and Dan Roth. 2019. Grammar Error Correction in Morphologically Rich Languages: The Case of Russian. Transactions of the Association for Computational Linguistics, 7:1–17.

Keisuke Sakaguchi, Courtney Napoles, Matt Post, and Joel Tetreault. 2016. Reassessing the Goals of Grammatical Error Correction: Fluency Instead of Grammaticality. Transactions of the Association for Computational Linguistics, 4:169–182.

Taiga Someya, Ryo Yoshida, and Yohei Oseki. 2024. Targeted Syntactic Evaluation on the Chomsky Hierarchy. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation, pages 15595–15605.

Chenming Tang, Xiuyu Wu, and Yunfang Wu. 2023. Are Pre-trained Language Models Useful for Model Ensemble in Chinese Grammatical Error Correction? In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics, pages 893–901.

Maksym Tarnavskyi, Artem Chernodub, and Kostiantyn Omelianchuk. 2022. Ensembling and Knowledge Distilling of Large Sequence Taggers for Grammatical Error Correction. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, pages 3842–3852.

J. A. Van Ek and J. L. M. Trim. 1991. Threshold 1990. Cambridge University Press.

J. A. Van Ek and J. L. M. Trim. 2001a. Vantage. Cambridge University Press.

J. A. Van Ek and J. L. M. Trim. 2001b. Waystage 1990. Cambridge University Press.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is All you Need. In Advances in Neural Information Processing Systems 30, pages 5998–6008.

Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohanane, Wei Peng, Sheng-Fu Wang, and Samuel R. Bowman. 2020. BLiMP: The Benchmark of Linguistic Minimal Pairs for English. Transactions of the Association for Computational Linguistics, 8:377–392.

Haoran Wu, Wenxuan Wang, Yuxuan Wan, Wenxiang Jiao, and Michael Lyu. 2023. ChatGPT or Grammarly? Evaluating ChatGPT on Grammatical Error Correction Benchmark. Preprint, arXiv:2303.13648.

Haihui Yang and Xiaojun Quan. 2024. Alirector: Alignment-Enhanced Chinese Grammatical Error Corrector. In Findings of the Association for Computational Linguistics: ACL 2024, pages 2531–2546.

Helen Yannakoudakis, Ted Briscoe, and Ben Medlock. 2011. A New Dataset and Method for Automatically Grading ESOL Texts. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 180–189.

Helen Yannakoudakis, Øistein E Andersen, Ardeshir Geranpayeh, Ted Briscoe, and Diane Nicholls. 2018. Developing an automated writing placement system for ESL learners. Applied Measurement in Education, 31(3):251–267.

Appendix

A CTSEG Error Categories and GEC Model Performance

The CTSEG error categories and the recall scores of each GEC model are presented in Tables 7–10. Compared to ERRANT, CTSEG provides a significantly more fine-grained classification of grammatical errors. For example, ERRANT’s broad categories—such as VERB:TENSE or DET (determiners)—group together a variety of learner errors. In contrast, CTSEG, based on the CEFR-J Grammar Profile, breaks these down into more specific constructions. ERRANT’s VERB:TENSE category, for instance, may map to several CTSEG items such as “present,” “past,” or “future.” Likewise, DET may correspond to more specific categories in CTSEG like “a/an/the,” “some/any,” and “no.”

However, it is difficult to fully map the CEFR-J-based error categories in CTSEG to ERRANT’s error types due to fundamental differences in their design goals. CTSEG is pedagogically motivated and structured around what learners are expected to acquire at each proficiency level, whereas ERRANT is error-oriented and curriculum-agnostic. As a result, many CTSEG categories lack clear counterparts in ERRANT, making one-to-one mapping between the two frameworks challenging.

B Example Sentence Pairs by Edit Distance

Table 11 presents example sentence pairs corresponding to each word-level edit distance. Because
CTSEG instructs annotators to create minimal pairs, the edit distance is often between 1 and 2, as shown in Figure 3. However, depending on the grammar item, it can range from 3 to 5, as also illustrated in Table 11.

C   Prompt for GPT-4

The prompts used for GPT-4 in the experiment are shown in Table 12.
<table>
  <tr>
    <th>ID</th>
    <th>CL</th>
    <th>Grammar Item</th>
    <th>Transformer</th>
    <th>GECToR</th>
    <th>GPT-4</th>
    <th>Avg.</th>
  </tr>
  <tr><td>1</td><td>A1</td><td>Personal pronoun (nominative case) (I) + be: I am</td><td>83.33</td><td>66.67</td><td>100.00</td><td>83.33</td></tr>
  <tr><td>2</td><td>A1</td><td>Personal pronoun (nominative case) (you) + be: You are</td><td>100.00</td><td>100.00</td><td>100.00</td><td>100.00</td></tr>
  <tr><td>3</td><td>A1</td><td>Personal pronoun (nominative case) (he/she) + be: He/She is</td><td>66.67</td><td>83.33</td><td>100.00</td><td>83.33</td></tr>
  <tr><td>4</td><td>A1</td><td>Personal pronoun (nominative case) (we) + be: We are</td><td>100.00</td><td>83.33</td><td>100.00</td><td>94.44</td></tr>
  <tr><td>5</td><td>A1</td><td>Personal pronoun (nominative case) (they) + be: They are</td><td>83.33</td><td>50.00</td><td>83.33</td><td>72.22</td></tr>
  <tr><td>6</td><td>A1</td><td>Personal pronoun (possessive case): my/our/your/her/their</td><td>71.43</td><td>71.43</td><td>100.00</td><td>80.95</td></tr>
  <tr><td>7</td><td>A1</td><td>Personal pronoun (objective case): me/us/him/her/them</td><td>100.00</td><td>66.67</td><td>83.33</td><td>83.33</td></tr>
  <tr><td>8</td><td>A1</td><td>Demonstrative pronoun (this/that) + be: This/That is</td><td>75.00</td><td>75.00</td><td>100.00</td><td>83.33</td></tr>
  <tr><td>9</td><td>A1</td><td>Demonstrative pronoun (these/those) + be: These/Those are</td><td>85.71</td><td>57.14</td><td>85.71</td><td>76.19</td></tr>
  <tr><td>10</td><td>A1</td><td>Demonstrative pronoun (it) + be: It is</td><td>85.70</td><td>28.57</td><td>82.50</td><td>59.25</td></tr>
  <tr><td>11</td><td>A1</td><td>Demonstrative adjective (this/that + noun)</td><td>62.50</td><td>28.57</td><td>100.00</td><td>43.69</td></tr>
  <tr><td>12</td><td>A1</td><td>Demonstrative adjective (these/those + noun)</td><td>100.00</td><td>62.50</td><td>100.00</td><td>87.50</td></tr>
  <tr><td>13</td><td>A1</td><td>Indefinite article</td><td>100.00</td><td>66.67</td><td>83.33</td><td>83.33</td></tr>
  <tr><td>14</td><td>A1</td><td>Definite article</td><td>83.33</td><td>33.33</td><td>66.67</td><td>61.11</td></tr>
  <tr><td>15</td><td>A1</td><td>Determiner (some/any)</td><td>83.33</td><td>50.00</td><td>100.00</td><td>77.78</td></tr>
  <tr><td>16</td><td>A1</td><td>Determiner (no)</td><td>90.00</td><td>37.50</td><td>100.00</td><td>75.83</td></tr>
  <tr><td>17</td><td>A1</td><td>Determiner (another)</td><td>85.71</td><td>50.00</td><td>100.00</td><td>78.57</td></tr>
  <tr><td>18</td><td>A2</td><td>Adjective + uncountable noun</td><td>100.00</td><td>42.86</td><td>100.00</td><td>80.95</td></tr>
  <tr><td>19</td><td>A2</td><td>Adjective + uncountable noun</td><td>83.33</td><td>33.33</td><td>83.33</td><td>58.67</td></tr>
  <tr><td>20</td><td>A2</td><td>few + plural noun</td><td>83.33</td><td>66.67</td><td>85.71</td><td>78.57</td></tr>
  <tr><td>21</td><td>A2</td><td>Prepositions</td><td>100.00</td><td>83.33</td><td>100.00</td><td>94.44</td></tr>
  <tr><td>22</td><td>A1</td><td>Possessive pronouns</td><td>100.00</td><td>66.67</td><td>100.00</td><td>88.89</td></tr>
  <tr><td>23</td><td>A2</td><td>Reflexive pronouns</td><td>62.50</td><td>37.50</td><td>87.50</td><td>62.50</td></tr>
  <tr><td>24</td><td>A1</td><td>Indefinite pronouns (-thing/-one/-body)</td><td>71.43</td><td>42.86</td><td>85.71</td><td>66.67</td></tr>
  <tr><td>25</td><td>A1</td><td>Indefinite pronoun (supportive word 'ones')</td><td>33.33</td><td>66.67</td><td>66.67</td><td>55.56</td></tr>
  <tr><td>26</td><td>A1</td><td>Indefinite pronoun (one)</td><td>87.50</td><td>28.57</td><td>100.00</td><td>70.83</td></tr>
  <tr><td>27</td><td>B1</td><td>Reciprocal pronoun (each other)</td><td>100.00</td><td>57.14</td><td>83.33</td><td>80.16</td></tr>
  <tr><td>28</td><td>B1</td><td>Reciprocal pronoun (one another)</td><td>75.00</td><td>42.86</td><td>77.78</td><td>65.21</td></tr>
  <tr><td>29</td><td>B1</td><td>Pronoun (others) (excluding "the others")</td><td>100.00</td><td>50.00</td><td>83.33</td><td>77.78</td></tr>
  <tr><td>30</td><td>B1</td><td>Pronoun (the other/others)</td><td>16.67</td><td>33.33</td><td>50.00</td><td>33.33</td></tr>
  <tr><td>31</td><td>A2</td><td>-thing + adjective</td><td>100.00</td><td>66.67</td><td>100.00</td><td>88.89</td></tr>
  <tr><td>32</td><td>A2</td><td>Adverb (frequency)</td><td>57.14</td><td>42.86</td><td>71.43</td><td>57.14</td></tr>
  <tr><td>33</td><td>B1</td><td>Adverb (emphasis)</td><td>100.00</td><td>37.50</td><td>100.00</td><td>79.17</td></tr>
  <tr><td>34</td><td>B1</td><td>Adverb (negation)</td><td>28.57</td><td>28.57</td><td>44.44</td><td>37.50</td></tr>
  <tr><td>35</td><td>B1</td><td>Adverb (quasi-negation)</td><td>100.00</td><td>44.44</td><td>100.00</td><td>81.48</td></tr>
  <tr><td>36</td><td>B1</td><td>Adverb (quasi-negation)</td><td>66.67</td><td>33.33</td><td>88.89</td><td>62.96</td></tr>
  <tr><td>37</td><td>A2</td><td>Comparison of equality (as + adjective/adverb + as)</td><td>83.33</td><td>50.00</td><td>100.00</td><td>77.78</td></tr>
  <tr><td>38</td><td>A1</td><td>Comparative (superiority) (-er) (including irregular forms like "better")</td><td>88.89</td><td>55.56</td><td>88.89</td><td>77.78</td></tr>
  <tr><td>39</td><td>A2</td><td>Comparative (superiority) (+ more + adjective/adverb)</td><td>100.00</td><td>83.33</td><td>100.00</td><td>94.44</td></tr>
  <tr><td>40</td><td>A1</td><td>Superlative (superiority) (-est) (including irregular forms like "best")</td><td>100.00</td><td>57.14</td><td>100.00</td><td>85.71</td></tr>
  <tr><td>41</td><td>A2</td><td>Superlative (superiority) (+ most + adjective/adverb)</td><td>100.00</td><td>66.67</td><td>83.33</td><td>83.33</td></tr>
  <tr><td>42</td><td>B1</td><td>Emphasis (intensity)</td><td>66.67</td><td>37.50</td><td>83.33</td><td>62.50</td></tr>
  <tr><td>43</td><td>B1</td><td>Adjective/adverb + each</td><td>100.00</td><td>28.57</td><td>100.00</td><td>76.19</td></tr>
  <tr><td>44</td><td>B1</td><td>too + adjective/adverb + to-infinitive</td><td>100.00</td><td>66.67</td><td>100.00</td><td>88.89</td></tr>
  <tr><td>45</td><td>B1</td><td>so + adjective/adverb + (that) clause</td><td>57.14</td><td>0.00</td><td>57.14</td><td>38.10</td></tr>
  <tr><td>46</td><td>B1</td><td>such (+a/an)+ adjective + noun</td><td>100.00</td><td>66.67</td><td>100.00</td><td>88.89</td></tr>
  <tr><td>47</td><td>B1</td><td>so + adjective + a/an + noun</td><td>100.00</td><td>33.33</td><td>100.00</td><td>77.78</td></tr>
  <tr><td>48</td><td>B1</td><td>too + adjective + a/an + noun</td><td>71.43</td><td>0.00</td><td>90.91</td><td>54.11</td></tr>
  <tr><td>49</td><td>B1</td><td>Comparative and comparative (same comparative)</td><td>45.45</td><td>18.18</td><td>100.00</td><td>54.55</td></tr>
  <tr><td>50</td><td>B1</td><td>too + adjective + a/an + the + comparative</td><td>88.89</td><td>44.44</td><td>88.89</td><td>74.07</td></tr>
  <tr><td>51</td><td>B1</td><td>Emphasis of comparative (e.g., even)</td><td>85.71</td><td>55.56</td><td>83.33</td><td>74.87</td></tr>
  <tr><td>52</td><td>B1</td><td>Emphasis of superlative (e.g., by far)</td><td>50.00</td><td>37.50</td><td>62.50</td><td>50.00</td></tr>
  <tr><td>53</td><td>B1</td><td>Emphasis with do/does</td><td>33.33</td><td>50.00</td><td>83.33</td><td>55.56</td></tr>
  <tr><td>54</td><td>B1</td><td>Emphasis with did</td><td>83.33</td><td>50.00</td><td>83.33</td><td>72.22</td></tr>
  <tr><td>55</td><td>B2</td><td>Phrasal verbs (verb + particle)</td><td>85.71</td><td>14.29</td><td>100.00</td><td>66.67</td></tr>
  <tr><td>56</td><td>B2</td><td>Phrasal verbs (verb + noun phrase/pronoun + particle)</td><td>85.73</td><td>50.00</td><td>83.33</td><td>72.22</td></tr>
  <tr><td>57</td><td>B2</td><td>Phrasal verbs (verb + particle + preposition + noun phrase/pronoun)</td><td>85.73</td><td>50.00</td><td>100.00</td><td>77.78</td></tr>
  <tr><td>58</td><td>A1</td><td>Tense/aspect (present) (be verb)</td><td>57.14</td><td>57.14</td><td>85.71</td><td>66.67</td></tr>
  <tr><td>59</td><td>A1</td><td>Tense/aspect (present) (general verb, except third-person singular)</td><td>100.00</td><td>100.00</td><td>83.33</td><td>94.44</td></tr>
  <tr><td>60</td><td>A1</td><td>Tense/aspect (present) (general verb, third-person singular)</td><td>100.00</td><td>57.14</td><td>100.00</td><td>85.71</td></tr>
  <tr><td>61</td><td>A1</td><td>Tense/aspect (present continuous)</td><td>71.43</td><td>71.43</td><td>71.43</td><td>71.43</td></tr>
  <tr><td>62</td><td>A2</td><td>Tense/aspect (present perfect)</td><td>66.67</td><td>50.00</td><td>100.00</td><td>72.22</td></tr>
  <tr><td>63</td><td>B2</td><td>Tense/aspect (present perfect continuous)</td><td>100.00</td><td>66.67</td><td>100.00</td><td>88.89</td></tr>
  <tr><td>64</td><td>A1</td><td>Tense/aspect (past (be verb)</td><td>83.33</td><td>83.33</td><td>83.33</td><td>83.33</td></tr>
  <tr><td>65</td><td>A1</td><td>Tense/aspect (past (general verb)</td><td>100.00</td><td>100.00</td><td>100.00</td><td>100.00</td></tr>
  <tr><td>66</td><td>A2</td><td>Tense/aspect (past continuous)</td><td>100.00</td><td>100.00</td><td>100.00</td><td>100.00</td></tr>
  <tr><td>67</td><td>A2</td><td>Tense/aspect (past perfect)</td><td>66.67</td><td>66.67</td><td>83.33</td><td>72.22</td></tr>
  <tr><td>68</td><td>B2</td><td>Tense/aspect (past perfect continuous)</td><td>50.00</td><td>16.67</td><td>66.67</td><td>44.44</td></tr>
  <tr><td>69</td><td>A1</td><td>Tense/aspect (future)</td><td>66.67</td><td>33.33</td><td>66.67</td><td>55.56</td></tr>
  <tr><td>70</td><td>B1</td><td>Tense/aspect (future continuous)</td><td>50.00</td><td>83.33</td><td>83.33</td><td>72.22</td></tr>
  <tr><td>71</td><td>B2</td><td>Tense/aspect (future perfect)</td><td>50.00</td><td>83.33</td><td>100.00</td><td>77.78</td></tr>
  <tr><td>72</td><td>B2</td><td>Tense/aspect (future perfect continuous)</td><td>33.33</td><td>33.33</td><td>83.33</td><td>55.56</td></tr>
  <tr><td>73</td><td>A1</td><td>Passive voice (present)</td><td>100.00</td><td>100.00</td><td>100.00</td><td>100.00</td></tr>
  <tr><td>74</td><td>B2</td><td>Passive voice (present continuous)</td><td>100.00</td><td>66.67</td><td>83.33</td><td>83.33</td></tr>
  <tr><td>75</td><td>B2</td><td>Passive voice (present perfect)</td><td>83.33</td><td>100.00</td><td>100.00</td><td>94.44</td></tr>
  <tr><td>76</td><td>A2</td><td>Passive voice (past)</td><td>83.33</td><td>50.00</td><td>90.00</td><td>74.44</td></tr>
  <tr><td>77</td><td>B2</td><td>Passive voice (past continuous)</td><td>66.67</td><td>50.00</td><td>50.00</td><td>55.56</td></tr>
  <tr><td>78</td><td>B2</td><td>Passive voice (past perfect)</td><td>33.33</td><td>33.33</td><td>100.00</td><td>55.56</td></tr>
  <tr><td>79</td><td>B1</td><td>Passive voice (future)</td><td>100.00</td><td>100.00</td><td>100.00</td><td>100.00</td></tr>
  <tr><td>80</td><td>B2</td><td>Passive voice (future continuous)</td><td>41.67</td><td>16.67</td><td>41.67</td><td>33.33</td></tr>
</table>

Table 7: Recall for each grammar item (ID1–ID80) across GEC models.
<table>
  <tr>
    <th>ID</th>
    <th>CL</th>
    <th>Grammar Item</th>
    <th>Transformer</th>
    <th>GECToR</th>
    <th>GPT-4</th>
    <th>Avg.</th>
  </tr>
  <tr><td>81</td><td>B2</td><td>Passive voice (future perfect)</td><td>66.67</td><td>66.67</td><td>83.33</td><td>72.22</td></tr>
  <tr><td>82</td><td>B1</td><td>Passive voice (modal verb)</td><td>100.00</td><td>85.71</td><td>100.00</td><td>95.24</td></tr>
  <tr><td>83</td><td>B2</td><td>Passive voice (modal verbs + continuous)</td><td>50.00</td><td>40.00</td><td>66.67</td><td>52.22</td></tr>
  <tr><td>84</td><td>B2</td><td>Passive voice (modal verbs + perfect)</td><td>100.00</td><td>100.00</td><td>100.00</td><td>100.00</td></tr>
  <tr><td>85</td><td>B2</td><td>Passive voice (give/pass/send/show/teach/tell with indirect object as subject)</td><td>54.55</td><td>54.55</td><td>72.73</td><td>60.61</td></tr>
  <tr><td>86</td><td>B2</td><td>Passive voice (give/pass/send/show/teach/tell with direct object as subject)</td><td>100.00</td><td>83.33</td><td>83.33</td><td>88.89</td></tr>
  <tr><td>87</td><td>B1</td><td>get + past participle</td><td>100.00</td><td>83.33</td><td>100.00</td><td>94.44</td></tr>
  <tr><td>88</td><td>A2</td><td>to-infinitive (to DO)</td><td>100.00</td><td>75.00</td><td>100.00</td><td>91.67</td></tr>
  <tr><td>89</td><td>B1</td><td>Negative form of to-infinitive (not to DO)</td><td>83.33</td><td>83.33</td><td>83.33</td><td>83.33</td></tr>
  <tr><td>90</td><td>B2</td><td>Perfect to-infinitive</td><td>83.33</td><td>66.67</td><td>50.00</td><td>66.67</td></tr>
  <tr><td>91</td><td>B2</td><td>Perfect passive to-infinitive</td><td>50.00</td><td>50.00</td><td>50.00</td><td>50.00</td></tr>
  <tr><td>92</td><td>B2</td><td>Perfect passive to-infinitive</td><td>100.00</td><td>100.00</td><td>83.33</td><td>94.44</td></tr>
  <tr><td>93</td><td>B1</td><td>to-infinitive with logical subject</td><td>71.43</td><td>37.50</td><td>60.00</td><td>56.31</td></tr>
  <tr><td>94</td><td>B1</td><td>-thing + to-infinitive</td><td>100.00</td><td>100.00</td><td>100.00</td><td>100.00</td></tr>
  <tr><td>95</td><td>B2</td><td>in order to DO</td><td>71.43</td><td>42.86</td><td>85.71</td><td>66.67</td></tr>
  <tr><td>96</td><td>B2</td><td>in order not to DO</td><td>71.43</td><td>57.14</td><td>100.00</td><td>76.19</td></tr>
  <tr><td>97</td><td>B2</td><td>so as to DO</td><td>83.33</td><td>85.71</td><td>100.00</td><td>89.68</td></tr>
  <tr><td>98</td><td>B2</td><td>so as not to DO</td><td>85.71</td><td>28.57</td><td>85.71</td><td>66.67</td></tr>
  <tr><td>99</td><td>B2</td><td>but to DO</td><td>85.71</td><td>33.33</td><td>57.14</td><td>58.73</td></tr>
  <tr><td>100</td><td>B2</td><td>be about to DO</td><td>100.00</td><td>66.67</td><td>100.00</td><td>88.89</td></tr>
  <tr><td>101</td><td>A2</td><td>Verb + to-infinitive</td><td>100.00</td><td>100.00</td><td>100.00</td><td>100.00</td></tr>
  <tr><td>102</td><td>B1</td><td>Verb + not + to-infinitive</td><td>100.00</td><td>14.29</td><td>100.00</td><td>71.43</td></tr>
  <tr><td>103</td><td>A1</td><td>Verb + object + to-infinitive</td><td>100.00</td><td>83.33</td><td>100.00</td><td>94.44</td></tr>
  <tr><td>104</td><td>B2</td><td>Verb + object + not + to-infinitive</td><td>70.00</td><td>0.00</td><td>66.67</td><td>45.56</td></tr>
  <tr><td>105</td><td>A2</td><td>Verb -ing form</td><td>83.33</td><td>66.67</td><td>83.33</td><td>77.78</td></tr>
  <tr><td>106</td><td>B2</td><td>Verb + verb -ing form</td><td>50.00</td><td>18.75</td><td>66.67</td><td>47.62</td></tr>
  <tr><td>107</td><td>B2</td><td>have + past participle</td><td>50.00</td><td>50.00</td><td>83.33</td><td>61.11</td></tr>
  <tr><td>108</td><td>B2</td><td>being + past participle</td><td>50.00</td><td>66.67</td><td>100.00</td><td>72.22</td></tr>
  <tr><td>109</td><td>B2</td><td>having been + past participle</td><td>33.33</td><td>66.67</td><td>66.67</td><td>55.56</td></tr>
  <tr><td>110</td><td>A1</td><td>Preposition + verb -ing form</td><td>83.33</td><td>83.33</td><td>83.33</td><td>83.33</td></tr>
  <tr><td>111</td><td>B2</td><td>Gerund with logical subject (possessive pronoun)</td><td>66.67</td><td>50.00</td><td>100.00</td><td>72.22</td></tr>
  <tr><td>112</td><td>A2</td><td>Verb + verb -ing form</td><td>83.33</td><td>83.33</td><td>83.33</td><td>83.33</td></tr>
  <tr><td>113</td><td>B2</td><td>Verb + not + verb -ing form</td><td>83.33</td><td>66.67</td><td>83.33</td><td>77.78</td></tr>
  <tr><td>114</td><td>A2</td><td>Verb + object + verb -ing form</td><td>66.67</td><td>50.00</td><td>83.33</td><td>66.67</td></tr>
  <tr><td>115</td><td>B2</td><td>Verb + object + not + verb -ing form</td><td>42.86</td><td>42.86</td><td>87.50</td><td>57.74</td></tr>
  <tr><td>116</td><td>B1</td><td>Imperative sentences (be verb)</td><td>33.33</td><td>0.00</td><td>66.67</td><td>33.33</td></tr>
  <tr><td>117</td><td>A1</td><td>Imperative sentences (general verb)</td><td>66.67</td><td>0.00</td><td>83.33</td><td>50.00</td></tr>
  <tr><td>118</td><td>A2</td><td>Emphasis in affirmative imperative sentences with do</td><td>50.00</td><td>50.00</td><td>50.00</td><td>50.00</td></tr>
  <tr><td>119</td><td>A1</td><td>Please + affirmative imperative sentence</td><td>83.33</td><td>85.71</td><td>55.56</td><td>74.87</td></tr>
  <tr><td>120</td><td>A1</td><td>Let's</td><td>75.00</td><td>62.50</td><td>75.00</td><td>70.83</td></tr>
  <tr><td>121</td><td>B1</td><td>Modal verbs (be able to)</td><td>100.00</td><td>66.67</td><td>100.00</td><td>88.89</td></tr>
  <tr><td>122</td><td>A2</td><td>Modal verbs (be going to)</td><td>100.00</td><td>83.33</td><td>100.00</td><td>88.89</td></tr>
  <tr><td>123</td><td>A2</td><td>Modal verbs (can)</td><td>100.00</td><td>85.71</td><td>100.00</td><td>95.24</td></tr>
  <tr><td>124</td><td>B1</td><td>Modal verbs (could)</td><td>71.43</td><td>100.00</td><td>100.00</td><td>90.48</td></tr>
  <tr><td>125</td><td>B2</td><td>Modal verbs (dare to)</td><td>42.86</td><td>28.57</td><td>71.43</td><td>47.62</td></tr>
  <tr><td>126</td><td>B1</td><td>Modal verbs (had better)</td><td>75.00</td><td>28.57</td><td>85.71</td><td>63.10</td></tr>
  <tr><td>127</td><td>A2</td><td>Modal verbs (have to)</td><td>100.00</td><td>100.00</td><td>100.00</td><td>100.00</td></tr>
  <tr><td>128</td><td>B1</td><td>Modal verbs (have got to)</td><td>57.14</td><td>71.43</td><td>85.71</td><td>71.43</td></tr>
  <tr><td>129</td><td>B1</td><td>Modal verbs (may)</td><td>33.33</td><td>50.00</td><td>83.33</td><td>55.56</td></tr>
  <tr><td>130</td><td>B1</td><td>Modal verbs (might as well)</td><td>50.00</td><td>33.33</td><td>66.67</td><td>50.00</td></tr>
  <tr><td>131</td><td>B1</td><td>Modal verbs (may well)</td><td>66.67</td><td>66.67</td><td>83.33</td><td>72.22</td></tr>
  <tr><td>132</td><td>B1</td><td>Modal verbs (might)</td><td>100.00</td><td>66.67</td><td>100.00</td><td>88.89</td></tr>
  <tr><td>133</td><td>B1</td><td>Modal verbs (might as well)</td><td>100.00</td><td>50.00</td><td>100.00</td><td>83.33</td></tr>
  <tr><td>134</td><td>B2</td><td>Modal verbs (might well)</td><td>77.78</td><td>66.67</td><td>66.67</td><td>70.37</td></tr>
  <tr><td>135</td><td>B1</td><td>Modal verbs (must)</td><td>100.00</td><td>83.33</td><td>100.00</td><td>94.44</td></tr>
  <tr><td>136</td><td>A2</td><td>Modal verbs (need to)</td><td>85.71</td><td>57.14</td><td>71.43</td><td>71.43</td></tr>
  <tr><td>137</td><td>B1</td><td>Modal verbs (ought to)</td><td>100.00</td><td>100.00</td><td>100.00</td><td>100.00</td></tr>
  <tr><td>138</td><td>B1</td><td>Modal verbs (shall)</td><td>83.33</td><td>83.33</td><td>100.00</td><td>88.89</td></tr>
  <tr><td>139</td><td>B1</td><td>Modal verbs (should)</td><td>100.00</td><td>100.00</td><td>100.00</td><td>100.00</td></tr>
  <tr><td>140</td><td>B1</td><td>Modal verbs (used to)</td><td>83.33</td><td>66.67</td><td>83.33</td><td>77.78</td></tr>
  <tr><td>141</td><td>A2</td><td>Modal verbs (will)</td><td>66.67</td><td>57.14</td><td>66.67</td><td>63.49</td></tr>
  <tr><td>142</td><td>B1</td><td>Modal verbs (would)</td><td>100.00</td><td>57.14</td><td>100.00</td><td>85.71</td></tr>
  <tr><td>143</td><td>B1</td><td>Modal verbs (would rather)</td><td>100.00</td><td>83.33</td><td>83.33</td><td>88.89</td></tr>
  <tr><td>144</td><td>B1</td><td>Modal verbs + continuous</td><td>50.00</td><td>50.00</td><td>50.00</td><td>50.00</td></tr>
  <tr><td>145</td><td>B1</td><td>Modal verbs + perfect</td><td>100.00</td><td>100.00</td><td>100.00</td><td>100.00</td></tr>
  <tr><td>146</td><td>A1</td><td>There of existence (There + be)</td><td>100.00</td><td>20.00</td><td>100.00</td><td>83.33</td></tr>
  <tr><td>147</td><td>B1</td><td>There of existence (perfect)</td><td>100.00</td><td>66.67</td><td>100.00</td><td>88.89</td></tr>
  <tr><td>148</td><td>B1</td><td>There of existence (modal verbs)</td><td>57.14</td><td>42.86</td><td>71.43</td><td>57.14</td></tr>
  <tr><td>149</td><td>A1</td><td>Here of existence (Here is/are)</td><td>57.14</td><td>57.14</td><td>57.14</td><td>57.14</td></tr>
  <tr><td>150</td><td>A1</td><td>Coordinating conjunctions</td><td>71.43</td><td>66.67</td><td>71.43</td><td>69.84</td></tr>
  <tr><td>151</td><td>B2</td><td>that-clause (appositive)</td><td>83.33</td><td>66.67</td><td>100.00</td><td>83.33</td></tr>
  <tr><td>152</td><td>A1</td><td>that-clause (object)</td><td>85.71</td><td>71.43</td><td>85.71</td><td>80.95</td></tr>
  <tr><td>153</td><td>B1</td><td>who-clause (object) (excluding "whether")</td><td>100.00</td><td>85.71</td><td>85.71</td><td>90.48</td></tr>
  <tr><td>154</td><td>B1</td><td>who-clause (subject)</td><td>66.67</td><td>33.33</td><td>77.78</td><td>59.26</td></tr>
  <tr><td>155</td><td>A1</td><td>Adverbial clause (when)</td><td>83.33</td><td>83.33</td><td>100.00</td><td>88.89</td></tr>
  <tr><td>156</td><td>A2</td><td>Adverbial clause (if)</td><td>83.33</td><td>50.00</td><td>83.33</td><td>72.22</td></tr>
  <tr><td>157</td><td>A2</td><td>Adverbial clause (as)</td><td>90.00</td><td>30.00</td><td>60.00</td><td>60.00</td></tr>
  <tr><td>158</td><td>B1</td><td>Adverbial clause (as soon as)</td><td>100.00</td><td>25.00</td><td>100.00</td><td>75.00</td></tr>
  <tr><td>159</td><td>B1</td><td>Adverbial clause (by the time)</td><td>100.00</td><td>100.00</td><td>100.00</td><td>100.00</td></tr>
  <tr><td>160</td><td>B1</td><td>Adverbial clause (so that)</td><td>62.50</td><td>66.67</td><td>100.00</td><td>76.39</td></tr>
</table>

Table 8: Recall for each grammar item (ID81–ID160) across GEC models.
<table>
  <tr>
    <th>ID</th>
    <th>CL</th>
    <th>Grammar Item</th>
    <th>Transformer</th>
    <th>GECToR</th>
    <th>GPT-4</th>
    <th>Avg.</th>
  </tr>
  <tr><td>161</td><td>B1</td><td>Subordinate clauses (main subordinating conjunctions other than as/if/that/when/whether)</td><td>66.67</td><td>50.00</td><td>83.33</td><td>66.67</td></tr>
  <tr><td>162</td><td>A1</td><td>Omission of that in subordinate clauses (e.g., hope/know/think)</td><td>66.67</td><td>66.67</td><td>83.33</td><td>72.22</td></tr>
  <tr><td>163</td><td>A2</td><td>It as a formal subject + to-infinitive</td><td>85.71</td><td>57.14</td><td>100.00</td><td>80.95</td></tr>
  <tr><td>164</td><td>B2</td><td>It as a formal object + to-infinitive</td><td>100.00</td><td>100.00</td><td>100.00</td><td>100.00</td></tr>
  <tr><td>165</td><td>B1</td><td>It as a formal subject + that-clause</td><td>71.43</td><td>33.33</td><td>85.71</td><td>63.49</td></tr>
  <tr><td>166</td><td>B1</td><td>It as a formal object + that-clause</td><td>60.00</td><td>77.78</td><td>85.71</td><td>74.50</td></tr>
  <tr><td>167</td><td>B1</td><td>wh-word + to-infinitive</td><td>66.67</td><td>66.67</td><td>83.33</td><td>72.22</td></tr>
  <tr><td>168</td><td>A1</td><td>Present participle modifying a noun (pre-position)</td><td>50.00</td><td>0.00</td><td>66.67</td><td>38.89</td></tr>
  <tr><td>169</td><td>A1</td><td>Present participle modifying a noun (post-position)</td><td>75.00</td><td>83.33</td><td>100.00</td><td>86.11</td></tr>
  <tr><td>170</td><td>A1</td><td>Past participle modifying a noun (pre-position)</td><td>57.14</td><td>33.33</td><td>85.71</td><td>58.33</td></tr>
  <tr><td>171</td><td>A1</td><td>Past participle modifying a noun (post-position)</td><td>85.71</td><td>57.14</td><td>100.00</td><td>90.48</td></tr>
  <tr><td>172</td><td>B1</td><td>Relative pronoun (non-restrictive) (who)</td><td>100.00</td><td>83.33</td><td>100.00</td><td>94.44</td></tr>
  <tr><td>173</td><td>B1</td><td>Relative pronoun (nominative) (which)</td><td>100.00</td><td>50.00</td><td>100.00</td><td>83.33</td></tr>
  <tr><td>174</td><td>B1</td><td>Relative pronoun (nominative) (that)</td><td>50.00</td><td>33.33</td><td>50.00</td><td>44.44</td></tr>
  <tr><td>175</td><td>B2</td><td>Relative pronoun (objective) (who)</td><td>100.00</td><td>83.33</td><td>100.00</td><td>94.44</td></tr>
  <tr><td>176</td><td>B2</td><td>Relative pronoun (objective) (whom)</td><td>100.00</td><td>50.00</td><td>100.00</td><td>83.33</td></tr>
  <tr><td>177</td><td>B2</td><td>Relative pronoun (objective) (which)</td><td>100.00</td><td>66.67</td><td>100.00</td><td>88.89</td></tr>
  <tr><td>178</td><td>B2</td><td>Relative pronoun (objective) (that)</td><td>83.33</td><td>50.00</td><td>83.33</td><td>72.22</td></tr>
  <tr><td>179</td><td>B2</td><td>Objective relative pronoun (objective)</td><td>50.00</td><td>50.00</td><td>66.67</td><td>55.56</td></tr>
  <tr><td>180</td><td>B1</td><td>Relative pronoun (possessive)</td><td>100.00</td><td>50.00</td><td>100.00</td><td>83.33</td></tr>
  <tr><td>181</td><td>B1</td><td>Relative pronoun (non-restrictive clause)</td><td>83.33</td><td>66.67</td><td>83.33</td><td>77.78</td></tr>
  <tr><td>182</td><td>B2</td><td>Pseudo-relative pronoun (as)</td><td>28.57</td><td>0.00</td><td>50.00</td><td>26.19</td></tr>
  <tr><td>183</td><td>A2</td><td>Compound relative pronoun (what)</td><td>83.33</td><td>66.67</td><td>83.33</td><td>77.78</td></tr>
  <tr><td>184</td><td>B1</td><td>Preposition + relative pronoun</td><td>66.67</td><td>16.67</td><td>50.00</td><td>44.44</td></tr>
  <tr><td>185</td><td>A2</td><td>Relative adverb (with antecedent)</td><td>83.33</td><td>50.00</td><td>83.33</td><td>72.22</td></tr>
  <tr><td>186</td><td>B1</td><td>Relative adverb (without antecedent)</td><td>83.33</td><td>66.67</td><td>83.33</td><td>77.78</td></tr>
  <tr><td>187</td><td>B1</td><td>Relative adverb (non-restrictive clause)</td><td>66.67</td><td>66.67</td><td>83.33</td><td>72.22</td></tr>
  <tr><td>188</td><td>B2</td><td>wh-ever</td><td>50.00</td><td>50.00</td><td>83.33</td><td>61.11</td></tr>
  <tr><td>189</td><td>B1</td><td>Preposition stranding in relative clauses</td><td>71.43</td><td>71.43</td><td>81.82</td><td>74.89</td></tr>
  <tr><td>190</td><td>A2</td><td>Exclamatory sentence (How + adjective/adverb)</td><td>100.00</td><td>42.86</td><td>85.71</td><td>76.19</td></tr>
  <tr><td>191</td><td>A2</td><td>Exclamatory sentence (How alone)</td><td>50.00</td><td>16.67</td><td>50.00</td><td>38.89</td></tr>
  <tr><td>192</td><td>A2</td><td>Exclamatory sentence (What)</td><td>71.43</td><td>71.43</td><td>71.43</td><td>71.43</td></tr>
  <tr><td>193</td><td>B1</td><td>Tag question (following affirmative sentence)</td><td>57.14</td><td>28.57</td><td>100.00</td><td>61.90</td></tr>
  <tr><td>194</td><td>S</td><td>S + V (become/feel/go/look/seem/sound) + C (adjective)</td><td>66.67</td><td>66.67</td><td>66.67</td><td>66.67</td></tr>
  <tr><td>195</td><td>B1</td><td>S + V (become/feel/go/look/seem/sound) + C (adjective)</td><td>100.00</td><td>66.67</td><td>100.00</td><td>88.89</td></tr>
  <tr><td>196</td><td>A2</td><td>S + V + O</td><td>100.00</td><td>85.71</td><td>100.00</td><td>95.24</td></tr>
  <tr><td>197</td><td>B1</td><td>S + V (give/pass/send/show/teach/tell) + indirect object + direct object</td><td>100.00</td><td>87.50</td><td>100.00</td><td>95.83</td></tr>
  <tr><td>198</td><td>B1</td><td>S + V (give/pass/send/show/teach/tell) + direct object + to + indirect object</td><td>100.00</td><td>71.43</td><td>100.00</td><td>90.48</td></tr>
  <tr><td>199</td><td>B1</td><td>S + V (make) + O + C (adjective)</td><td>87.50</td><td>62.50</td><td>87.50</td><td>79.17</td></tr>
  <tr><td>200</td><td>A2</td><td>Reported speech (explain/report/say)</td><td>83.33</td><td>100.00</td><td>100.00</td><td>94.44</td></tr>
  <tr><td>201</td><td>B1</td><td>Reported speech (tell)</td><td>83.33</td><td>83.33</td><td>100.00</td><td>88.89</td></tr>
  <tr><td>202</td><td>B1</td><td>Indirect questions (ask/remind/show/teach/tell)</td><td>85.71</td><td>71.43</td><td>85.71</td><td>80.95</td></tr>
  <tr><td>203</td><td>B1</td><td>Indirect questions (ask/remind/show/teach/tell)</td><td>100.00</td><td>100.00</td><td>100.00</td><td>100.00</td></tr>
  <tr><td>204</td><td>B2</td><td>Cleft sentence (emphasizing prepositional phrase/adverb)</td><td>50.00</td><td>16.67</td><td>62.50</td><td>43.06</td></tr>
  <tr><td>205</td><td>B2</td><td>Pseudo-cleft sentence with what</td><td>66.67</td><td>33.33</td><td>83.33</td><td>61.11</td></tr>
  <tr><td>206</td><td>A2</td><td>Causative construction (have/let/make)</td><td>100.00</td><td>83.33</td><td>100.00</td><td>94.44</td></tr>
  <tr><td>207</td><td>B2</td><td>have/get + object + past participle</td><td>66.67</td><td>33.33</td><td>66.67</td><td>52.22</td></tr>
  <tr><td>208</td><td>B1</td><td>get + object + present participle</td><td>66.67</td><td>100.00</td><td>100.00</td><td>88.89</td></tr>
  <tr><td>209</td><td>B1</td><td>ask/let + object + to-infinitive</td><td>83.33</td><td>83.33</td><td>83.33</td><td>77.78</td></tr>
  <tr><td>210</td><td>B1</td><td>Perception verb + bare infinitive</td><td>85.71</td><td>85.71</td><td>71.43</td><td>80.95</td></tr>
  <tr><td>211</td><td>B2</td><td>Perception verb + present particle</td><td>66.67</td><td>50.00</td><td>50.00</td><td>55.56</td></tr>
  <tr><td>212</td><td>B2</td><td>Perception verb + past particle</td><td>50.00</td><td>50.00</td><td>50.00</td><td>50.00</td></tr>
  <tr><td>213</td><td>B2</td><td>Participial construction (present participle at the beginning)</td><td>42.86</td><td>14.29</td><td>28.57</td><td>28.57</td></tr>
  <tr><td>214</td><td>B2</td><td>Participial construction (past participle at the beginning)</td><td>16.67</td><td>0.00</td><td>33.33</td><td>16.67</td></tr>
  <tr><td>215</td><td>B2</td><td>Subjunctive past in if-clauses (past tense verb in if-clause)</td><td>50.00</td><td>33.33</td><td>100.00</td><td>61.11</td></tr>
  <tr><td>216</td><td>B2</td><td>Subjunctive past perfect in if-clauses (past perfect verb in if-clause)</td><td>83.33</td><td>33.33</td><td>100.00</td><td>88.89</td></tr>
  <tr><td>217</td><td>B2</td><td>Subjunctive present in that-clauses (bare infinitive in that-clause)</td><td>77.78</td><td>40.00</td><td>100.00</td><td>74.07</td></tr>
  <tr><td>218</td><td>B2</td><td>wish + subjunctive past perfect</td><td>83.33</td><td>33.33</td><td>33.33</td><td>53.33</td></tr>
  <tr><td>219</td><td>B2</td><td>wish + subjunctive past perfect</td><td>83.33</td><td>66.67</td><td>100.00</td><td>83.33</td></tr>
  <tr><td>220</td><td>B1</td><td>as if/though + subjunctive past</td><td>50.00</td><td>16.67</td><td>50.00</td><td>38.89</td></tr>
  <tr><td>221</td><td>B2</td><td>as if/though + subjunctive past perfect</td><td>50.00</td><td>50.00</td><td>66.67</td><td>55.56</td></tr>
  <tr><td>222</td><td>B2</td><td>if only + subjunctive past</td><td>83.33</td><td>50.00</td><td>83.33</td><td>72.22</td></tr>
  <tr><td>223</td><td>B2</td><td>if only + subjunctive past perfect</td><td>100.00</td><td>66.67</td><td>66.67</td><td>77.78</td></tr>
  <tr><td>224</td><td>B2</td><td>should in if-clauses</td><td>71.43</td><td>57.14</td><td>71.43</td><td>66.67</td></tr>
  <tr><td>225</td><td>B2</td><td>Subjunctive inversion for past subjunctive</td><td>83.33</td><td>50.00</td><td>83.33</td><td>72.22</td></tr>
  <tr><td>226</td><td>B2</td><td>Subjunctive inversion for past perfect subjunctive</td><td>85.71</td><td>33.33</td><td>100.00</td><td>73.02</td></tr>
  <tr><td>227</td><td>B2</td><td>Inversion using should</td><td>83.33</td><td>66.67</td><td>100.00</td><td>83.33</td></tr>
  <tr><td>228</td><td>B2</td><td>if it were/was not for ...</td><td>66.67</td><td>50.00</td><td>83.33</td><td>66.67</td></tr>
  <tr><td>229</td><td>B2</td><td>were it not for ...</td><td>85.71</td><td>85.71</td><td>100.00</td><td>90.48</td></tr>
  <tr><td>230</td><td>B2</td><td>if it hadn’t been for ...</td><td>66.67</td><td>33.33</td><td>83.33</td><td>61.11</td></tr>
  <tr><td>231</td><td>B2</td><td>had it not been for ...</td><td>100.00</td><td>50.00</td><td>100.00</td><td>83.33</td></tr>
  <tr><td>232</td><td>B2</td><td>Inversion (See + be/verb/do/modal verb + personal pronoun)</td><td>66.67</td><td>16.67</td><td>83.33</td><td>55.56</td></tr>
  <tr><td>233</td><td>B2</td><td>Inversion (Neither/nor + be/have/be/do/modal verb + personal pronoun)</td><td>83.33</td><td>50.00</td><td>66.67</td><td>61.11</td></tr>
  <tr><td>234</td><td>B2</td><td>Inversion (Hardly/Little/Never/No sooner/Scarcely/Seldom ...)</td><td>88.89</td><td>42.86</td><td>100.00</td><td>77.25</td></tr>
  <tr><td>235</td><td>A1</td><td>Interrogative sentence (Why ...?)</td><td>62.50</td><td>75.00</td><td>77.78</td><td>71.76</td></tr>
  <tr><td>236</td><td>A1</td><td>Interrogative sentence (When ...?)</td><td>88.89</td><td>55.56</td><td>88.89</td><td>77.78</td></tr>
  <tr><td>237</td><td>A1</td><td>Interrogative sentence (Who ...?)</td><td>50.00</td><td>37.50</td><td>87.50</td><td>58.33</td></tr>
  <tr><td>238</td><td>A1</td><td>Interrogative sentence (Whom ...?)</td><td>69.23</td><td>33.33</td><td>85.71</td><td>62.76</td></tr>
  <tr><td>239</td><td>A1</td><td>Interrogative sentence (What ...?)</td><td>83.33</td><td>66.67</td><td>100.00</td><td>83.33</td></tr>
  <tr><td>240</td><td>A1</td><td>Interrogative sentence (What + noun ...?)</td><td>62.50</td><td>30.00</td><td>62.50</td><td>38.89</td></tr>
</table>

Table 9: Recall for each grammar item (ID161–ID240) across GEC models.
<table>
  <tr>
    <th>ID</th>
    <th>CL</th>
    <th>Grammar Item</th>
    <th>Transformer</th>
    <th>GECToR</th>
    <th>GPT-4</th>
    <th>Avg.</th>
  </tr>
  <tr>
    <td>241</td>
    <td>A1</td>
    <td>Interrogative sentence (Which ...?)</td>
    <td>50.00</td>
    <td>33.33</td>
    <td>50.00</td>
    <td>44.44</td>
  </tr>
  <tr>
    <td>242</td>
    <td>A1</td>
    <td>Interrogative sentence (Which + noun ...?)</td>
    <td>88.89</td>
    <td>33.33</td>
    <td>100.00</td>
    <td>74.07</td>
  </tr>
  <tr>
    <td>243</td>
    <td>A2</td>
    <td>Interrogative sentence (Whose ...?)</td>
    <td>30.00</td>
    <td>10.00</td>
    <td>60.00</td>
    <td>33.33</td>
  </tr>
  <tr>
    <td>244</td>
    <td>A2</td>
    <td>Interrogative sentence (Whose + noun ...?)</td>
    <td>16.67</td>
    <td>16.67</td>
    <td>66.67</td>
    <td>33.33</td>
  </tr>
  <tr>
    <td>245</td>
    <td>A1</td>
    <td>Interrogative sentence (Where ...?)</td>
    <td>71.43</td>
    <td>57.14</td>
    <td>85.71</td>
    <td>71.43</td>
  </tr>
  <tr>
    <td>246</td>
    <td>A2</td>
    <td>Interrogative sentence (How ...?)</td>
    <td>88.89</td>
    <td>66.67</td>
    <td>100.00</td>
    <td>85.19</td>
  </tr>
  <tr>
    <td>247</td>
    <td>B1</td>
    <td>Interrogative sentence (How + adjective/adverb ...?)</td>
    <td>42.86</td>
    <td>42.86</td>
    <td>100.00</td>
    <td>61.90</td>
  </tr>
  <tr>
    <td>248</td>
    <td>B1</td>
    <td>Interrogative sentence (Preposition + interrogative ...?)</td>
    <td>42.86</td>
    <td>14.29</td>
    <td>100.00</td>
    <td>48.68</td>
  </tr>
  <tr>
    <td>249</td>
    <td>A1</td>
    <td>Functional interrogative (Can you ...?)</td>
    <td>100.00</td>
    <td>66.67</td>
    <td>100.00</td>
    <td>88.89</td>
  </tr>
  <tr>
    <td>250</td>
    <td>A2</td>
    <td>Functional interrogative (Could you ...?)</td>
    <td>100.00</td>
    <td>83.33</td>
    <td>100.00</td>
    <td>94.44</td>
  </tr>
  <tr>
    <td>251</td>
    <td>A2</td>
    <td>Functional interrogative (Will you ...?)</td>
    <td>100.00</td>
    <td>85.71</td>
    <td>100.00</td>
    <td>95.24</td>
  </tr>
  <tr>
    <td>252</td>
    <td>A1</td>
    <td>Functional interrogative (Would you ...?)</td>
    <td>66.67</td>
    <td>66.67</td>
    <td>66.67</td>
    <td>66.67</td>
  </tr>
  <tr>
    <td>253</td>
    <td>A1</td>
    <td>Functional interrogative (Can I ...?)</td>
    <td>66.67</td>
    <td>83.33</td>
    <td>100.00</td>
    <td>83.33</td>
  </tr>
  <tr>
    <td>254</td>
    <td>A2</td>
    <td>Functional interrogative (May I ...?)</td>
    <td>100.00</td>
    <td>100.00</td>
    <td>100.00</td>
    <td>100.00</td>
  </tr>
  <tr>
    <td>255</td>
    <td>A2</td>
    <td>Functional interrogative (Should I ...?)</td>
    <td>83.33</td>
    <td>83.33</td>
    <td>100.00</td>
    <td>88.89</td>
  </tr>
  <tr>
    <td>256</td>
    <td>A2</td>
    <td>Functional interrogative (Should we ...?)</td>
    <td>85.71</td>
    <td>66.67</td>
    <td>100.00</td>
    <td>84.13</td>
  </tr>
  <tr>
    <td>257</td>
    <td>A2</td>
    <td>Functional interrogative (Should I ...?)</td>
    <td>100.00</td>
    <td>83.33</td>
    <td>100.00</td>
    <td>94.44</td>
  </tr>
  <tr>
    <td>258</td>
    <td>A2</td>
    <td>Functional interrogative (Why don’t you ...?)</td>
    <td>83.33</td>
    <td>83.33</td>
    <td>100.00</td>
    <td>88.89</td>
  </tr>
  <tr>
    <td>259</td>
    <td>A2</td>
    <td>Functional interrogative (Why don’t we ...?)</td>
    <td>66.67</td>
    <td>55.56</td>
    <td>83.33</td>
    <td>68.52</td>
  </tr>
  <tr>
    <td>260</td>
    <td>A2</td>
    <td>Functional interrogative (Why not ...?)</td>
    <td>57.14</td>
    <td>50.00</td>
    <td>71.43</td>
    <td>59.52</td>
  </tr>
  <tr>
    <td>261</td>
    <td>A1</td>
    <td>Functional interrogative (How about ...?)</td>
    <td>83.33</td>
    <td>66.67</td>
    <td>83.33</td>
    <td>77.78</td>
  </tr>
  <tr>
    <td>263</td>
    <td>A1</td>
    <td>Functional interrogative (What about ...?)</td>
    <td>57.14</td>
    <td>42.86</td>
    <td>57.14</td>
    <td>52.38</td>
  </tr>
</table>

Table 10: Recall for each grammar item (ID241–ID263) across GEC models.

<table>
  <tr>
    <th>ID</th>
    <th>Grammar Item</th>
    <th>ED</th>
    <th>Sentence</th>
  </tr>
  <tr>
    <td>101</td>
    <td>Verb + to-infinitive</td>
    <td>1</td>
    <td>*I want __ play soccer .<br>I want to play soccer .</td>
  </tr>
  <tr>
    <td>162</td>
    <td>Omission of that in subordinate clauses<br>(e.g., hope/know/think)</td>
    <td>2</td>
    <td>*We know him innocent .<br>We know he is innocent .</td>
  </tr>
  <tr>
    <td>11</td>
    <td>Demonstrative adjective<br>(this/that + noun)</td>
    <td>3</td>
    <td>*This my dictionary is good .<br>This dictionary of mine is good .</td>
  </tr>
  <tr>
    <td>198</td>
    <td>S + V + direct object + to + indirect object<br>(V: give/pass/send/show/teach/tell)</td>
    <td>4</td>
    <td>*I gave to my friend the book .<br>I gave the book to my friend .</td>
  </tr>
  <tr>
    <td>85</td>
    <td>Passive voice<br>(give/pass/send/show/teach/tell with indirect object as subject)</td>
    <td>5</td>
    <td>*A letter was sent my parents by the teacher .<br><span style="color:green">My parents were sent a letter</span> by the teacher .</td>
  </tr>
</table>

Table 11: Examples of sentence pairs in CTSEG with word-level edit distances ranging from 1 to 5. The ED column indicates the word-level edit distance. An asterisk (*) denotes an ungrammatical sentence.

<table>
  <tr>
    <th>Name</th>
    <th>Prompt</th>
  </tr>
  <tr>
    <td>Baseline</td>
    <td>You are an English language teacher. A student has sent you the following text. \n{text}\nProvide a grammatical correction for the text, making only necessary changes. Do not provide any additional comments or explanations. If the input text is already correct, return it unchanged.</td>
  </tr>
  <tr>
    <td>Grammar-item-guided</td>
    <td>You are an English language teacher. You taught {grammar_item} in today’s class. A student has sent you the following text. \n{text}\nProvide a grammatical correction for the text, making only necessary changes. Do not provide any additional comments or explanations. If the input text is already correct, return it unchanged.</td>
  </tr>
</table>

Table 12: Prompt for GPT-4: {text} represents the source sentence, and {grammar_item} specifies the target grammar item.

Reviewer paper summaries: This paper introduces a targeted approach to evaluating GEC systems by focusing on specific grammar items. The authors present the CTSEG dataset, which contains teacher-created minimal pairs of ungrammatical and grammatical sentences, organized by grammar items aligned with CEFR proficiency levels. Unlike conventional GEC evaluation datasets based on learner-produced text with naturally occurring errors, CTSEG enables fine-grained assessment of GEC model performance on specific grammatical structures. The authors evaluate three main GEC approaches using CTSEG, and provide detailed analyses of model performance per grammar item. Additionally, the paper explores how explicitly including grammar item information in prompts improves GPT-4’s ability to generate corrections that preserve the targeted structure, demonstrating a practical application for language learning contexts.
Summary of strengths: 1. The paper addresses a clear gap in current GEC evaluation by enabling targeted assessment of specific grammar items--something conventional benchmarks struggle to offer due to uneven error distributions in learner-produced texts. This approach allows developers to systematically identify model strengths and weaknesses across grammatical structures.

2. The CTSEG dataset is carefully designed, with high-quality control and alignment with CEFR levels, making it pedagogically relevant and useful for both research and educational applications.

3. The empirical evaluation is comprehensive, covering three major types of GEC approaches.
Summary of weaknesses: 1. The introduction claims that existing datasets, such as CoNLL-2014 and BEA-2019, are limited in their ability to analyze rare errors or those difficult to categorize with rule-based methods. However, Rule 4 of the dataset creation guidelines specifies that the focus is on common learner mistakes. This creates a slight inconsistency. It would be helpful if the authors could clarify whether their goal is to capture fine-grained distinctions within common errors, address rarer error types, or both.

2. In the post-processing section, the authors mention using ERRANT, which employs a more limited set of error labels. The paper would benefit from including a mapping between their fine-grained labels and the original ERRANT categories to establish clear connections with existing evaluation frameworks. For example, do all 15 subcategories of verb tense errors in CTSEG correspond to the general "Verb Tense" category in ERRANT (i.e., correctly being categorized as verb tense errors by ERRANT)? In addition, how many of the fine-grained CTSEG error tags are classified as OTHER or Unknown by ERRANT? Providing this mapping would clarify how CTSEG aligns with and extends existing tagging schemes.

3. The paper does not address how different error operations (missing, replacement, unnecessary) interact with error tags in the dataset creation process. These operations represent different correction challenges, and their distribution across grammar items would likely affect model performance.
Comments, suggestions and typos: While Sections 1–4 form a coherent narrative focused on the CTSEG dataset and its evaluation, Section 5, which explores grammar-item-guided prompting for GPT-4, feels somewhat disconnected. The introduction does not adequately prepare readers for this application-oriented section. Introducing the practical use case of grammar-item-guided prompting earlier in the introduction would improve the paper’s flow and provide a clearer sense of the full scope of the research.
Overall assessment scores: 3.5