Detection-Correction Structure via General Language Model for Grammatical Error Correction (2024)

Abstract: Grammatical error correction (GEC) is a task dedicated to rectifying texts with minimal edits, which can be decoupled into two components: detection and correction. However, previous works have predominantly focused on direct correction, with no prior efforts to integrate both into a single model. Moreover, the exploration of the detection-correction paradigm by large language models (LLMs) remains underdeveloped. This paper introduces an integrated detection-correction structure, named DeCoGLM, based on the General Language Model (GLM). The detection phase employs a fault-tolerant detection template, while the correction phase leverages autoregressive mask infilling for localized error correction. Through the strategic organization of input tokens and modification of attention masks, we facilitate multi-task learning within a single model. Our model demonstrates competitive performance against the state-of-the-art models on English and Chinese GEC datasets. Further experiments present the effectiveness of the detectioncorrection structure in LLMs, suggesting a promising direction for GEC.

heading: 1 Introduction
text: Grammatical error correction (GEC) is a task focused on automatically rectifying grammatical errors in human-written text (Wang et al., 2021). GEC models are applied in language learning (Katinskaia and Yangarber, 2021; Caines et al., 2023; Kaneko et al., 2022), enhancing automatic speech recognition (Liao et al., 2023), and aiding in text data labeling (Sun et al., 2023). The two primary approaches in GEC are Sequence-to-Sequence (Seq2Seq) and Sequenceto-Edit (Seq2Edit). Without detection, Seq2Seq treats GEC as the direct generation for correct text, providing high flexibility (Junczys-Dowmunt et al., 2018; Ge et al., 2018). On the other hand, Seq2Edit views GEC as a sequence labeling task for edit la-
bels, showcasing high precision by controlled edits (Awasthi et al., 2019; Stahlberg and Kumar, 2020; Omelianchuk et al., 2020). The advent of large language models (LLMs) has further expanded Seq2Seq model capabilities (Ouyang et al., 2022; Zeng et al., 2022). Despite their unprecedented performance in various tasks (Chang et al., 2024), LLMs underperform than low-parameter models in GEC due to the over-correction phenomenon (Qu and Wu, 2023; Coyne et al., 2023).
While the detection-correction structure can harness the strengths of both Seq2Seq and Seq2Edit, most existing works merely utilize detection as additional input for Seq2Seq models (Yuan et al., 2021a; Li et al., 2022, 2023a). Moreover, all previous detection-correction systems comprise separate models (Chen et al., 2020). In contrast, we introduce a novel GEC model, named DeCoGLM, based on the General Language Model (GLM) (Du et al., 2022). This model employs an integrated detection-correction structure to detect errors and generate localized corrections. As depicted in Figure 1, the error detection phase employs a template rule to construct masked text based on detection results. During the correction phase, the model leverages the autoregressive mask infilling capabil-
1748
ity of the GLM to generate correct text pieces for erroneous parts, thereby saving inference time. To incorporate both detection and correction within a single model, we devise a multi-task learning approach, organizing input text with attention mask adjustments. Results on English and Chinese GEC benchmarks demonstrate that our proposed model surpasses previous detection-correction models and is comparable to state-of-the-art (SOTA) models. To further explore the potential of applying the detection-correction structure to LLMs, the detection and correction phases are separated, termed DeGLM and CoGLM respectively. Our proposed single system, comprising a small detection model and an LLM corrector, outperforms other Seq2Seq LLMs. In summary, our primary contributions are:
• A novel GEC model, DeCoGLM, which incorporates a detection-correction structure based on the GLM.
• The design of a multi-task training method that integrates detection and correction within a single model.
• The exploration of using LLMs for GEC, which involves deploying large error correction models with the support of small detection models.

heading: 2 Related Work

heading: 2.1 Sequence-to-Sequence GEC
text: Seq2Seq models (Lewis et al., 2019; Raffel et al., 2020) have demonstrated high performance in GEC (Junczys-Dowmunt et al., 2018; Choe et al., 2019; Zhao et al., 2019; Katsumata and Komachi, 2020). Techniques such as data synthesis (Stahlberg and Kumar, 2021; Grundkiewicz et al., 2019), training schedule (Lichtarge et al., 2020; Bout et al., 2023), and decode reranking methods (Kaneko et al., 2019; Zhang et al., 2023; Zhou et al., 2023) have been incorporated into previous Seq2Seq GEC models. SOTA model architectures typically supplement Seq2Seq models with additional information (Li et al., 2023a; Zhang et al., 2022b; Fang et al., 2023a). However, a significant drawback of Seq2Seq GEC models is the inference cost, as these models generate tokens sequentially and waste time copying source tokens (Sun et al., 2021).
As the latest Seq2Seq models, LLMs have emerged as a new paradigm for natural language processing (NLP) tasks following the introduction
of GPT-3 and ChatGPT (Brown et al., 2020). Nevertheless, recent studies have shown that LLMs underperform current SOTA models on both English and Chinese GEC benchmarks (Coyne et al., 2023; Loem et al., 2023; Qu and Wu, 2023; Li et al., 2023b). Existing datasets and evaluation methods (Bryant et al., 2017) favor minimum edits as the rule for correction. However, GPT models often produce over-corrected sentences with unnecessary edits (Fang et al., 2023b; Coyne et al., 2023). In contrast to the Seq2Seq GEC methods that directly perform overall generation, our work only focuses on localized error correction, which not only saves inference time but also mitigates the over-correction phenomena in LLMs.

heading: 2.2 Sequence-to-Edit GEC
text: Seq2Edit methods generate edit operations for ungrammatical sentences (Stahlberg and Kumar, 2020). For instance, LaserTagger (Malmi et al., 2019) predicts token-level edit operations, which has been adopted in subsequent methods like PIE and GECToR (Awasthi et al., 2019; Omelianchuk et al., 2020). As a representative model, GECToR predicts four classes of edits and grammatical transformations, achieving high-precision results. Lai et al. (2022) further enhances it by addressing its deficiencies in multi-round correction. However, Seq2Edit methods necessitate intricate designs for edits, which are not language-agnostic. In contrast, our proposed model retains a limited set of language-agnostic edit operations and can flexibly conduct edits by autoregressive generation.

heading: 2.3 Detection-Correction GEC
text: The GEC task can be divided into two processes: detection and correction (Rei and Yannakoudakis, 2016; Bell et al., 2019). Prior research incorporates detection results as supplementary information for Seq2Seq correction models (Kaneko et al., 2020; Yuan et al., 2021b; Li et al., 2023a). The methods proposed by Mallinson et al. (2020) and Yakovlev et al. (2023) employ the Masked Language Model (MLM) (Devlin et al., 2018) to obtain corrections, which are constrained by mask number. Chen et al. (2020) introduces error span detection and correction to address the GEC problem, which allows for flexible corrections while maximizing time efficiency. Building on this, we further integrate the detection and correction tasks into a single GLM model, enabling mutual benefits between the two tasks, which is not achieved by previous works.

heading: 3 Methods
text: Our proposed model leverages the design of the GLM. Given a sentence with MASK tokens, GLM utilizes autoregressive blank infilling (Du et al., 2022) to generate a corresponding segment for each MASK position. These segments are termed as text pieces. This section describes how GLM is utilized to integrate detection and correction into a single model, as depicted in Figure 2. Additionally, the design of multi-task training is also outlined here.

heading: 3.1 Error Detection
text: Drawing from the four edit classes by Omelianchuk et al. (2020), we utilize token-level detection labels that do not include any specific word or grammar. Given that the mask-infilling process can generate empty text pieces, the REPLACE and DELETE operations are consolidated into the ERROR label. Consequently, the detection labels comprise KEEP (K), ERROR (E), and INSERT (I). Given the tokens of source text as:
xs = x 1 sx 2 s . . . x n s (1)
, the objective of error detection is to predict detection labels derived by the alignment between the source text and the target text (correct text):
d = d1d2 . . . dn, di ∈ L = {K,E, I} (2) Detection Model The proposed model begins by extracting the representations of the source text tokens by GLM as Equation 3. The final detection label predictions are generated through a detection head, implemented by a feed-forward network FN and softmax function, as shown in Equation 4:
hs = h 1 sh 2 s . . . h n s = GLM(xs) (3)
p ( d̂i = l|xs ) = Softmax(FN ( his ) ), l ∈ L (4)
Fault-tolerant Template The source text xs is transformed into masked text xm based on the detection labels using the following template rules. Each continuous interval containing only ERROR labels is replaced with a MASK token. For each position of INSERT, a MASK token is inserted. The form of masked text is shown in Equation 5:
xm = xs1m1xs2m2 . . .mkxsk+1 , (5)
where mi is the i-th MASK token introduced in xs, and xsi denotes the i-th correct subinterval of source text. If all the labels are KEEPs, the source text is directly output as the corrected result. Despite potential inaccuracies in detections, our model can tolerate a certain degree of false positives. In the instance where the correct token is identified as ERROR or INSERT, the corrector can mitigate such errors by either restoring the original text piece or generating an empty text piece. Aggressive Detection Utilizing the fault-tolerant template enables more aggressive detection, emphasizing the recall of ERROR and INSERT. Focal Loss (Lin et al., 2020) is used as the loss function to tackle the issue of imbalanced classification because the majority of tokens correspond to KEEP labels. The training objective for error detection is given by Equation 6:
ℓD = −αD (1− pθ (d|xs))γ log (pθ (d|xs)) (6)
where θ represents the model parameters and γ is a hyper-parameter set to 2. αD denotes the corresponding weight factors for detection labels. To strengthen aggressive error detection, αK for the
KEEP category is set to 1, while αEI = 2 is set for the ERROR and INSERT categories.

heading: 3.2 Localized Error Correction
text: In the training data, detection labels are derived from the alignment of sequences between the source text xs and the target text y. The corresponding masked text xm can be formulated in Equation 5 with xsi representing the i-th aligned segment. For each unaligned position replaced with mi, the correct text piece is denoted as ci. Consequently, the target text can be represented as:
y = xs1c1xs2c2 . . . ckxsk+1 (7)
Leveraging the GLM pretrained by autoregressive blank infilling task, we fine-tune the GLMs for localized error correction. The probability distribution prediction for the j-th token in the i-th text piece ci is given in Equation 8:
p ( ˆci,j = w|xs,xm, c<i, c<ji ) =
GLMH ( xs,xm, c<i, c <j i ) , w ∈ V
(8)
where GLMH denotes the GLM model with its original token prediction head, w is any token in the vocabulary, and c<ji refers to all tokens with index < j in text piece ci.

heading: 3.3 Multi-Task Organization
text: Multi-task Learning. The cross-entropy loss function, shown in Equation 9, is used as the training objective for error correction task:
ℓC = − ∑
i,j
log ( pθ ( ci,j |xs,xm,c<i,c<ji )) (9)
For multi-task learning, we utilize a weighted loss function to enable the model to concurrently acquire error detection and correction capabilities. The training objective for this DeCoGLM model is to minimize the loss function given by:
ℓ = ℓ̄C + wD ℓ̄D (10)
where ℓ̄C and ℓ̄D are the token-level averages of ℓC and ℓD respectively. The detection loss weight wD is set to 10 to balance the scales of the two losses. For the impact of the loss weights on the model’s performance, please refer to Section 5.2. Attention Mask To unify the two tasks into a single model, source text xs, masked text xm, and text pieces c are concurrently fed into the GLM model. The prediction of detection labels is conditioned
on xs, while the autoregressive text prediction relies on xs, xm, and all previously generated text pieces. Therefore, the attention from xm to xs is eliminated to prevent detection from using xm and c, with other part adhering to the original GLM attention mask. This is depicted in Figure 3. Two Stage Supervised Fine-tuning Given that error detection is not infallible, the input during the correction phase may contain inaccuracies, with a distribution deviation from the training samples constructed with right detection labels. This issue is also observed in other detection-correction works (Chen et al., 2020; Li et al., 2023a). To address this, we add a second supervised fine-tuning stage (SFT2), which employs a detection-enhanced approach: initially, all detection results on the training set are obtained using the model trained with data constructed by perfect detection (SFT1). Then, new training data is generated by augmenting the original labels with the fault detection results, leading to a secondary training of the SFT1 model. Examples of the two-stage training samples are provided in Table 7 in the appendix.

heading: 3.4 Separate Models
text: The detection and correction phases can be implemented using two separate GLMs, named DeGLM and CoGLM respectively in this paper. Their training objectives are defined by Equation 6 and 9,
respectively. This decomposition facilitates the customization of distinct models for the detection and correction phases. However, in scenarios with limited computational resources, the ParameterEfficient Fine-Tuning (PEFT) (Fu et al., 2023) is ineffective for DeCoGLM due to the significant disparities between the sequence labeling task of the error detection module and the mask-infilling pretraining task of GLM. To apply our approach to LLMs, we propose training a large version of CoGLM using the detection-enhanced method, similar to the second stage fine-tuning discussed in Section 3.3.

heading: 3.5 Detection Control
text: During inference, the model needs to predict detection labels for the source text first, then transform it into masked text. Subsequently, both of them are input together for generating text pieces. This decoupling allows us to regulate the correction process using the probabilities of the three detection labels, thereby harnessing the model’s potential to enhance benchmark performance. Three control modes are designed: KEEP Threshold (δ): Any prediction with KEEP probability exceeding δ is directly set to KEEP. ERROR Lower Bound (ϕe): Any ERROR probability prediction falling below ϕe is directly set to 0, thereby precluding the prediction of ERROR when pe < ϕe. INSERT Lower Bound (ϕi): Any INSERT probability prediction below ϕi is directly set to 0, precluding the prediction of INSERT when pi < ϕi.
The three inference hyper-parameters can be determined using a greedy grid search based on the metrics on the validation set. We discuss them in Section 5.5.

heading: 4 Experiments

heading: 4.1 Datasets and Evaluation
text: For the English GEC task, we evaluate the performance on the CoNLL-14 test set (Ng et al., 2014) using the M2 Scorer (Dahlmeier and Ng, 2012), and on the BEA-19 test set (Bryant et al., 2019) using the ERRANT scorer (Bryant et al., 2017). The model is pretrained on synthetic dataset C4-200M (Stahlberg and Kumar, 2021) and fine-tuned on the cleaned Lang8 dataset (CLang8) (Rothe et al., 2021). For the large version of CoGLM model, we utilize smaller datasets including FCE (Yannakoudakis et al., 2011), NUCLE (Dahlmeier et al.,
2013), and W&I+LOCNESS (Bryant et al., 2019) for fine-tuning, following Zhou et al. (2023). The BEA-19 dev set is used for model selection.
For the Chinese GEC task, we synthesize pretraining data from the People’s Daily corpus1 using rule-based insertion, replacement, and deletion. The models are fine-tuned on the Chinese Lang8 dataset (Zhao et al., 2018) and the HSK dataset, following Zhang et al. (2022a), and on the FCGEC training set, respectively. The models are evaluated on MuCGEC and FCGEC test sets using ChERRANT (Zhang et al., 2022a; Xu et al., 2022). Further details are provided in Appendix A.

heading: 4.2 Model Settings
text: Proposed Models The open-source GLMs are utilized as the backbones for both DeCoGLM and separate models. The detection head comprises a feed-forward network with a single hidden layer, the dimension of which matches that of the GLM hidden state. The English base model employs glm-roberta-large, while glm-large-chinese is used as the Chinese base model. The large CoGLM models for error correction, denoted as CoGLM (10B), uses glm-10b and glm-10b-chinese as backbones. Due to the restriction of computational resources, large models are fine-tuned on the relatively small fine-tuning dataset mentioned in Section 4.1 by LoRA (Hu et al., 2021), without datasets for pretraining. Refer to Appendix B.2 for detailed configurations. Comparison with Previous Works In the main experiment, we present the results of single systems trained on parallel data without any reranker. GECToR (Omelianchuk et al., 2020) represents the Seq2Edit models, while BART and T5 (Lewis et al., 2019; Raffel et al., 2020) are SOTA backbones of Seq2Seq GEC methods. SynGEC (Zhang et al., 2022b) incorporates syntactic information into the BART model. The performance of GECToR and BART model on the Chinese dataset is the reproduced result under our data configuration, and the results for BART on the English dataset are reported by Zhang et al. (2022b). We also present the results of four models involving the detection-correction process. SpanDC (Chen et al., 2020) comprises a span detector and a generator. Multi-Encoder (Yuan et al., 2021a) encodes error categories as auxiliary information. GEC-DePend (Yakovlev et al., 2023) integrates error detection
1https://github.com/shibing624/pycorrector
with correction by the MLM. TemplateGEC (Li et al., 2023a) uses the output of the GECToR model as supplementary information for Seq2Seq models. Comparison with LLMs For the LLMs treating GEC as a Seq2Seq task, we fine-tune ChatGLM2, ChatGLM3 (Du et al., 2022), and Llama2 (Touvron et al., 2023) with LoRA. As Llama2 is not optimized for Chinese, the results on the Chinese dataset are obtained using the Baichuan (Yang et al., 2023) models. GPT-4 We report the zero-shot performance of GPT-4 on four datasets with prompting. We attempt to incorporate detection results in the form of masked text into the prompt of GPT-4, aiming to enhance the performance on GEC tasks.

heading: 4.3 Main Results
text: Table 1 presents the main results. According to the last two rows of primary results, the integrated detection-correction model outperforms the separate models in most cases in terms of the F0.5 metric, despite having only half the parameter count. This suggests that the designed multi-task learning mutually reinforces detection and correction, which will be further discussed in Section 5.1. DeCoGLM achieves the highest or secondhighest F0.5 performance on three datasets, demonstrating comparable performance to SOTA GEC models. Considering the model parameter counts, our model outperforms all previous works with
the detection-correction process, indicating that the well-designed detection-correction structure can achieve the SOTA level in GEC, a field typically dominated by Seq2Seq models. Furthermore, the inference speed of the localized error correction is significantly faster than the globalized error correction of the Seq2Seq method, with the details provided in Appendix C. These results also underscore the potential of GLM in the GEC field.
Despite limitations of data quantity and finetuning methods, fine-tuning LLMs with over 10B parameters yields results approaching SOTA level, suggesting that LLMs can reduce the need for extensive supervised data for fine-tuning. The strategy of small detection models assisting large models in localized correction yields improved performance across all datasets, primarily due to higher precision. This suggests that the model reduces over-correction at the expense of a certain level of recall. On the English dataset, GPT-4 exhibits a similar trend when incorporated with detection results, indicating that detection results can stably improve the GEC capability of LLMs, thus presenting a promising future direction for GEC.

heading: 5 Analysis

heading: 5.1 Interaction of Detection and Correction
text: In Section 4.3, we mentioned that detection and correction tasks can mutually benefit each other.
To further verify this, we conduct experiments using the integrated model (DeCoGLM, 335M) and two separate models for detection and correction (DeGLM, 335M; CoGLM, 335M), as shown in the primary results in Table 1. Without employing two-stage fine-tuning involving data enhancement, the models are trained on the same dataset, and their performance on the development set for detection and correction metrics is presented in Table 2. The integrated model exhibits superior detection and correction capabilities over separate models. A fairer comparison should involve two separate models with a parameter count of 335M/2 = 167.5M, but currently, there is no GLM backbone of approximately this size. In this scenario with fewer parameters, the advantage of the integrated model is expected to be even greater.

heading: 5.2 Weights of Multi-Task Training
text: To establish two weights that significantly impact the training objective: the detection loss weight wD in Equation 10, and the ERROR and INSERT loss weight αEI in Equation 6, we conduct preliminary experiments, which include only the two stages of fine-tuning. The obtained results are presented in Table 3. Based on a preliminary observation on the loss scale, we initially set wD = 10 and explore experimental results under varying αEI .
The outcomes suggest that the Focal Loss along with moderately increasing αEI to achieve aggressive detection introduced in Section 3.3 is effective. After setting αEI = 2, we conducted additional experiments with different wD. The overall experimental results indicate that αEI = 2 and wD = 10 constitute a suitable setup.

heading: 5.3 Detection Label Set
text: In the design outlined in Section 3.1, ERROR includes both replacement and deletion, as the deletion can be considered as replacing with zerolength text. The results for this design are shown in the second row of Table 4. INSERT can also be further merged into the ERROR label. This can be achieved by considering the INSERT operation as replacing the token xi at the insertion position with xicj , where cj represents tokens to be inserted. The results corresponding to this approach are shown in the first row of Table 4. Additionally, we demonstrate the results of applying four detection labels (KEEP, ERROR, INSERT, DELETE) in the last row. Overall, our designed three-label scheme performs relatively better, as the insertion operation in the two-label mode requires disrupting the correct part of the source text, and encountering DELETE in the four-label mode will lead to direct deletion, which makes the model unable to recover from faults in the error correction phase.

heading: 5.4 Ablation Study
text: To explore the effectiveness of various components in the designed detection-correction model, we conduct an ablation study focusing on synthetic data,
backbone, two-stage fine-tuning, and detection control. The results are shown in Table 5.
Effectiveness of synthetic data In the proposed model, both the English and Chinese models undergo pretraining with a large-scale synthetic dataset of GEC. A comparison between the top and middle rows of Table 5 reveals that pretraining indeed provides a stable improvement in model performance, although the data used is not from real scenarios.
Effectiveness of GLM backbone The detectioncorrection structure can also be implemented in Seq2Seq models. We applied the proposed method to the BART model and conducted experiments. An additional detection head is integrated into the BART encoder, while the decoder generates text pieces for localized error correction. The experimental results, depicted in the bottom rows of Table 5, consistently demonstrate superior performance when employing GLM as the backbone compared to using BART. This can be attributed, in part, to the consistency between the original pretraining task of GLM and the training objective of the correction task, as defined in Equation 9. However, the pretraining pattern of BART differs. Additionally, the separation of BART’s encoder and decoder into two distinct modules may not effectively foster the mutual enhancement of detection and correction abilities in multi-task learning.
Effectiveness of Two Stage Fine-tuning As described in Section 3.3, two fine-tuning stages differ in the training data: SFT1 constructs training samples using only ground-truth detection labels, while SFT2 utilizes both ground-truth detection labels and the detection results from the model trained in the first stage. As evident from the comparison in Table 5, SFT1 significantly improves the
model’s performance than the model pretrained on the synthetic dataset. Comparing the results exclusively differing in SFT2 in Table 5, it is observed that SFT2 consistently enhances F0.5, primarily attributed to the improvement in precision while maintaining recall relatively constant. This validates the effectiveness of the two-stage supervised fine-tuning design. Detection Control From Table 5, it is evident that, under the scenario of employing the same trained model, setting three hyper-parameters for the detection phase also enhances the F0.5 performance. This approach primarily aims at improving precision. However, upon closer inspection, it is noticeable that this technique results in a more substantial reduction in recall compared to the secondstage fine-tuning. For all GLM models incorporating detection control, the recall on the CoNLL-14 test set is consistently below 50%, and the recall on the BEA-19 test set is consistently below 65%. Thus, the effectiveness of detection control stems more from the trade-off between precision and recall, as discussed in the next section.

heading: 5.5 Precision-Recall Trade off
text: Adjusting the threshold for KEEP prediction probability (δ) and the probability lower bounds for ERROR and INSERT predictions (ϕe, ϕi) defined in Section 3.5 allows for further adjustment of precision and recall, resulting in improved F0.5 scores. We performed a parameter search on the validation set to identify configurations maximizing F0.5, and the results are depicted in Figure 4.
Without setting ϕe and ϕi, δ = 0.38 achieved the highest F0.5 of 63.2 on BEA-19 dev set. Then, we fix δ = 0.38 and perform a grid search for ϕe, ϕi. All results are presented as points in the right plot of
Figure 4, and nearly all points are located within the region enclosed by the dashed line in the bottomleft. The dashed line represents the boundary of the model’s capability, and the intersection point with the F0.5 contour line represents the optimal performance attainable by the model. The point with the highest F0.5 = 63.5 is the one closest to the intersection point, with ϕe = 0.5 and ϕi = 0.6. Under this parameter configuration, the model achieved an F0.5 value of 74.43 on the BEA-19 test set, as shown in Table 1. The detection control offers such a straightforward implementation of the precision-recall trade-off.

heading: 6 Conclusion
text: We introduce a novel language-agnostic detectioncorrection structure via GLM for the GEC task. The structure employs a three-label error detection pattern and uses Focal Loss for aggressive detection. The correction phase leverages the maskinfilling capability of GLM to generate correct text pieces. A multi-task learning approach is designed to integrate both functionalities within the same model, optimized using a weighted loss function. Experimental results show proposed model DeCoGLM outperforms previous detectioncorrection structures and achieves F0.5 scores comparable to SOTA on English and Chinese GEC benchmarks. The effectiveness of the detectioncorrection structure is further validated by applying it to open-source LLMs and GPT-4, indicating that incorporating error detection information improves the performance of LLMs on GEC datasets by reducing over-correction. Ablation studies confirm the efficacy of our model design and the ability to trade off precision and recall can be realized by detection control. We aim for this work to further guide GEC research within the detectioncorrection paradigm. The code and related models are available at https://github.com/GMagoLeWay/GECFramework.
Limitations
Incremental methods proven effective on Seq2Seq models, such as incorporating syntactic information (Zhang et al., 2022b), refining training data (Mita et al., 2020), and employing additional models for reranking during the generation phase (Zhang et al., 2023; Zhou et al., 2023), are not implemented in this work. The main objective of this paper is to propose a novel GEC architecture, with these additional tricks serving as potential avenues for future extensions. Furthermore, due to resource restrictions, we are unable to apply our integrated detection-correction structure to LLMs. This is because the sequence labeling task differs from the generative tasks that LLMs are designed to perform, necessitating full-parameter fine-tuning to integrate the two tasks. Additionally, in our investigation of LLMs as correction models, models with parameters exceeding 13B are not utilized. The absence of full-parameter fine-tuning on LLMs and experiments with larger models due to resource constraints leaves room for further exploration of the application of the detection-correction paradigm on LLMs.
Ethics Statement
The datasets and models we used are publicly available and utilized only for research purposes. The datasets do not contain any information that names or uniquely identifies individual people or offensive content. LLMs are utilized in our experiments, consistent with their intended use in natural language processing tasks. The models we designed will be published and intended for academic research in the field of grammatical error correction, in accordance with the original access conditions of the models used.
The detection-correction structure we designed limits the model to making only localized modifications to the text, preventing it from generating text without constraints, thereby significantly reducing the potential risks associated with the model. However, It is worth noting that the modifications made by the designed model may alter certain facts in the text, leading to hallucination, especially when modifications occur in named entities.
ChatGPT is utilized as the AI Assistant to polish the paper writing.

heading: Acknowledgements
text: We thank all the reviewers for their valuable comments on improving our paper. This work was supported by National Natural Science Foundation of China (62036001). The corresponding author is Houfeng Wang.

heading: A Dataset
text: A.1 Dataset Statistics
In the experiments described in Section 4.1, the datasets used are outlined in Table 6. Due to constraints on our computational resources, the CoGLM (10B) models are fine-tuned on relatively smaller datasets, and the models are not pre-trained on synthetic datasets.
A.2 Dataset for Training
As shown in Table 6, for relatively small models, we first pretrain using the publicly available C4200M English GEC synthetic dataset and our synthesized Chinese GEC dataset to obtain two pretrained models. Subsequently, the English model is fine-tuned using the CLang8 dataset, while the Chinese model is fine-tuned using the Lang8 Chinese dataset and the FCGEC dataset to yield two individual models. For the results of the models used for comparison in the primary results of Table 1, the GECToR and BART results on the two Chinese datasets are reproduced according to our training procedure, while the rest are as reported in the original papers, where they utilized various training data configurations. Most models are finetuned using the NUCLE, W&I+LOCNESS, and FCE datasets. Besides, GECToR uses the PIE-9M as the pretraining dataset (Awasthi et al., 2019).
For LLMs, we did not perform pretraining; instead, we directly applied the LoRA method using the NUCLE, W&I+LOCNESS, and FCE datasets for the English model. For Chinese, we trained two models using Lang8 (CH) and FCGEC. The training data for CoGLM (10B) and other LLMs used for comparison are completely consistent.
A.3 Dataset Examples In Sections 3.1 and 3.2, we describe the construction of training data. By aligning the source text with the target text, we derive error detection labels and masked text, thereby constructing training samples as illustrated in Figure 2. In Section 3.3, we elaborate on a two-stage supervised fine-tuning approach, where the training data for the second stage is reconstructed based on the detection predictions made by the model trained in the first stage. During data construction, model-induced false positives for ERROR and INSERT are incorporated to generate new masked text and corresponding text pieces. It is crucial to note that this process is solely aimed at creating new masked text to enhance the model’s ability to address false positives during the correction phase, while the detection labels used in training remain unchanged. Examples of the constructed training data are provided in Table 7, where "
" denotes the "begin of sentence" token and "
" represents the "end of sentence" token. For the sake of brevity, these tokens are omitted in the content of this paper except in Figure 2.

heading: B Details of Experiments
text: B.1 Loss Weight We pre-determine the weights in multi-task learning by intuitively observing the scales of two losses. This preliminary experiment was conducted on the CLang8 dataset, and the loss curves are depicted in Figure 5. It is evident from the figure that the detection loss ℓD and correction loss ℓC differ by roughly an order of magnitude. Consequently, we initially set wD = 10, determine the weights for ERROR and INSERT categories in Focal Loss denoted by αEI , and subsequently test whether wD = 10 is an optimal choice, as discussed in Section 5.2.
B.2 Model Configurations
The training configurations for the integrated detection-correction model (DeCoGLM) and the parameters used during inference are presented in Table 8. To conserve computational resources during training, early stopping is employed, which requires the pre-definition of evaluation metrics on the validation set. Two primary metrics are utilized: (1) AD-Accuracy, defined as the sum of the recall for ERROR and INSERT and the accuracy of next token prediction by GLM, aiming to reinforce the aggressive detection principle mentioned in Section 3.1; (2) General-Accuracy, the geometric mean between the recall for the three detection labels and the accuracy of next token prediction
by GLM. The configurations for training the separate models, DeGLM and CoGLM, are similar to those in Table 8. The pre-trained models include glm-roberta-large, glm-large-chinese, glm-10b, and glm-10b-chinese, accessible through HuggingFace2. We implement all the designed models using PyTorch, including DeCoGLM, DeGLM, and CoGLM.
All models are trained by the Trainer from the transformers3 package in Python, on NVIDIA RTX 4090 GPUs. Due to resource constraints, all experiments are conducted with a fixed random seed (111), and single-run results are reported. We adopt the approach recommended by Rothe et al. (2021)
2https://huggingface.co 3https://huggingface.co/docs/transformers/index
to post-process the model’s predictions on English test datasets, aiming to ensure greater alignment of tokenization with the evaluation data.
B.3 GPT-4 Prompts The prompts utilized during the inference of GPT-4 are illustrated in Table 9. For the Chinese tasks, the prompts are the direct translation of the corresponding English prompts. The API version of GPT-4 used in this paper is Preview-0315.
C Inference Speed
We conduct a brief evaluation of the inference speed of our proposed detection-correction structure, and the average inference speeds on the CoNLL-14 and BEA-19 test sets are presented in Table 10. The models are trained exclusively on the CLang8 dataset, and during the inference phase, no hyperparameters are adjusted, utilizing only beam search. Our proposed model achieves slightly better performance while maintaining a faster inference speed (≈3x) than the Seq2Seq model. The experiments are conducted on an NVIDIA RTX 4090 GPU, with the same constrained batch size of 1 during inference.

heading: Future Work
text: The main objective of this paper is to propose a novel GEC architecture, with these additional tricks serving as potential avenues for future extensions. Furthermore, due to resource restrictions, we are unable to apply our integrated detection-correction structure to LLMs. This is because the sequence labeling task differs from the generative tasks that LLMs are designed to perform, necessitating full-parameter fine-tuning to integrate the two tasks. Additionally, in our investigation of LLMs as correction models, models with parameters exceeding 13B are not utilized. The absence of full-parameter fine-tuning on LLMs and experiments with larger models due to resource constraints leaves room for further exploration of the application of the detection-correction paradigm on LLMs