Chinese Spelling Corrector Is Just a Language Learner (2024)

Abstract: This paper emphasizes Chinese spelling correction by means of self-supervised learning, which means there are no annotated errors within the training data. Our intuition is that humans are naturally good correctors with exposure to error-free sentences, which contrasts with current unsupervised methods that strongly rely on the usage of confusion sets to produce parallel sentences. In this paper, we demonstrate that learning a spelling correction model is identical to learning a language model from error-free data alone, with decoding it in a greater search space. We propose Denoising Decoding Correction (D2C), which selectively imposes noise upon the source sentence to determine the underlying correct characters. Our method is largely inspired by the ability of language models to perform correction, including both BERT-based models and large language models (LLMs). We show that the self-supervised learning manner generally outperforms the confusion set in specific domains because it bypasses the need to introduce error characters to the training data which can impair the error patterns not included in the introduced error characters.

heading: 1 Introduction
text: Chinese spelling correction (CSC) is a fundamental natural language processing task for a series of AI applications (Martins and Silva, 2004; Gao et al., 2010; Yang et al., 2024; Afli et al., 2016; Gupta et al., 2021). Recent studies (Wu et al., 2023b; Liu et al., 2024) show that simply using the supervised signals within parallel sentences to fine-tune pretrained language models (PLMs) achieves notable results across a series of benchmarks.
3 Equal contribution. † Corresponding author; This research was supported by the Joint Research Project of Yangtze River Delta Science and Technology Innovation Community (No. 2022CSJGG1400).
Source code: https://github.com/Jianglai-0023/ self-supervised-csc
However, the high cost of annotation is blamed for the low accessibility of parallel sentences. Therefore, these models remain mediocre in handling massive domains in real applications, which makes the application of powerful self-supervised learning to CSC a pivotal issue that has received broad attention in the community. This paper emphasizes the value of self-supervised learning, where only error-free data is used to adapt models to specific target domains, which has still achieved marginal progress in recent years.
Previous unsupervised methods (Zhao and Wang, 2020; Liu et al., 2021; Li, 2022) focus on synthesizing pseudo parallel sentences, while the supervised signals do not derive from the real distribution but from the confusion set (an empirically constructed word set of common misspelled cases). By replacing certain characters in the original sentences with those in the confusion set, parallel sentences are obtained for fine-tuning the models. However, the gap between the confusion set and the real error patterns in the target domain can induce a high
6933
false positive rate (Wu et al., 2023b). This paper raises a bold idea: Can machine spelling correction learn from error-free data alone?
Intriguingly, humans naturally learn to rectify mistakes in a sentence with minimal exposure to parallel data. We illustrate in Figure 1 that humans only learn to use correct sentences (error-free data) in daily life. When encountering a sentence with an error character “模” (mold), they can correct it to “膜” (cornea) with ease based on their knowledge. In contrast, machine spelling correction models cannot do this if they are not exposed to annotated edit pairs like “模” → “膜” in the training process.
In this paper, we demonstrate that a machine spelling corrector can also be learned from solely error-free data as illustrated at the bottom of Figure 1. The key is to have the model learn semantics rather than character-to-character editing, where the source sentence will first be encoded into the semantic space, and then rephrased to the correct sentence, demonstrate this ability. We call this manner self-supervised spelling correction. However, the resultant models still exhibit a low recall.
To address this problem, we propose a novel decoding algorithm Denoising Decoding Correction (D2C), which selectively imposes noise upon the source sentence to solve the underlying correct characters. We apply D2C to two architectures: bidirectional models (represented by ReLM (Liu et al., 2024), the state-of-the-art model in Chinese spelling correction) and auto-regressive models (represented by a series of LLMs (OpenAI, 2023; Touvron et al., 2023; Yang et al., 2023; Wu et al., 2024)). D2C achieves a significant performance boost over raw language models trained with errorfree data.
To evaluate our method across different domains, we created a synthesized training set for LEMON (Wu et al., 2023b) using GPT-3.5 as a sentence generator, which contains only error-free sentences. This dataset permits the fine-tuning and evaluation of self-supervised models in various domains.
We summarize the contributions of this paper. • We demonstrate that spelling correction can be directly transferred from language modeling on error-free data. • We propose a novel decoding algorithm, creating an effective self-supervised learning procedure that allows spelling correction models to adapt to target domains with minimal expense. • We build synthetic error-free training data from
LEMON to benchmark unsupervised domain adaption in the community.

heading: 2 Related Works
text: Correcting spelling errors poses a challenging yet crucial task in natural language processing. Early endeavors primarily relied on unsupervised techniques, assessing sentence perplexity as a key metric (Yeh et al., 2013; Yu and Li, 2014; Xie et al., 2015). Recent methods model spelling correction as a sequence tagging problem that maps each character in a given sentence to its accurate counterpart (Wang et al., 2018, 2019). On top of pre-trained language models (PLMs), some BERT-based models with the sequence tagging training objective are proposed. Zhang et al. (2020) identify the potential error characters by a detection network and then leverage the soft masking strategy to enhance the eventual correction decision. Zhu et al. (2022a) use a multi-task network to minimize the misleading impact of the misspelled characters (Cheng et al., 2020). There is also a line of work that incorporates phonological and morphological knowledge through data augmentation and enhances the BERTbased encoder to assist mapping the error to the correct one (Guo et al., 2021; Li et al., 2021; Liu et al., 2021; Cheng et al., 2020; Huang et al., 2021; Zhang et al., 2021). Recent studies (Liu et al., 2024) focus on the rephrasing training objective, which achieves notable results.
While in the unsupervised spelling correction domain, previous works focus on generating pseudo annotated data or detecting error characters with confusion dataset (Zhao and Wang, 2020; Liu et al., 2021; Li, 2022). While these methods are based on heuristics, our method is based on self-supervised learning (Devlin et al., 2019; Gao et al., 2021; Wu et al., 2022, 2023a) which seeks to perturb the language representation of PLMs.

heading: 3 From Language Modeling to Spelling Correction
text: This section provides the motivation for our work. The basic goal is to learn spelling correction from error-free data, which we term self-supervised spelling correction. First, we discuss the transferability between language modeling and spelling correction. Second, we highlight that rephrasing is the primary training objective for self-supervised spelling correction.
We discuss the transferability from two perspectives: (1) The coherence of training objectives between rephrasing spelling correction and language modeling, and (2) The inclusion of knowledge about spelling correction into the pre-training process.

heading: 3.1 Language Modeling
text: First, we introduce the training objectives of language modeling.
Given an input sentence Y = {y1, y2, · · · , yn} of n characters, auto-regressive language modeling seeks to predict the character yi based on its left context, namely P (yi|y1, y2, · · · , yi−1).

heading: 3.2 Spelling Correction
text: Second, we introduce the training objectives of spelling correction. A spelling correction model can be learned by two dominant objectives, sequence tagging and rephrasing.
Spelling correction aims to rectify the underlying misspelled characters in the source sentence. Denote the source sentence as X = {x1, x2, · · · , xn} and the target sentence as Y = {y1, y2, · · · , yn} and suppose xi is one of the typos in X , the model learns to correct xi to yi based on the entire source sentence, namely P (yi|x1, x2, · · · , xn). Tagging The above modeling process can also be viewed as sequence tagging from X to Y . While this has been widely adopted in previous work, a recent study (Liu et al., 2024) shows that taggingbased spelling correction models will lean towards point-to-point editing, thus ignoring the specific context. The final training objective degenerates into P (yi|xi). Rephrasing In comparison, rephrasing (Liu et al., 2024) is shown to be a more effective training objective for spelling correction. It specifically seeks to rewrite the entire sentence, namely P (yi|x1, x2, · · · , xn, y1, y2, · · · , yi−1). To ensure that the rephrasing process is based on semantics instead of copying, a ratio of noise (e.g., masking with an unused token) is introduced to the source sentence, written as P (yi|x̃1, x̃2, · · · , x̃n, y1, y2, · · · , yi−1).

heading: 3.3 Self-supervised Spelling Correction
text: The unsupervised learning setting is naturally akin to language modeling, where the model is trained on error-free data. Comparing the above two training objectives with language modeling, we find that
rephrasing and language modeling are formally the same. In rephrasing, the input sentence is the concatenation of the source and target. This implies that the spelling correction model can better utilize the knowledge in a pre-trained language model and be transferred from it.

heading: 3.4 Knowledge in Vanilla PLM
text: We hypothesize that, after large-scale pre-training, the language model already contains the knowledge needed for spelling correction.
To verify this hypothesis, we mask the error characters in the source sentence and have the vanilla model (non-fine-tuned one) output the corrected sentence. We then check the predicted characters at the positions of the mask tokens of the output sentence and compare them with the right characters.
As shown in Table 1, we see that the vanilla model can already recall the correct characters in its top-k candidates without any fine-tuning on spelling correction. For example, in about 90% of the cases, the model’s top 10 predictions have covered the correct answer. This indicates that pretrained language models already possess the necessary knowledge for spelling correction through mask-infilling.

heading: 3.5 Tagging Model vs. Rephrasing Model
text: In this section, we evaluate the tagging model and the rephrasing model (Liu et al., 2024) through two small-scale experiments, uncovering their different emphases during the spelling correction task. The tagging model excels at remembering the characters’ mapping relations while the rephrasing model performs better in understanding the meaning of the sentences.
Error-free Data Table 2 shows that the tagging model trained on error-free data is ineffective. We
conjecture that the model only learns point-to-point copying since the source is always the same as its target, thus losing the ability to make modifications to the source sentence. In contrast, the rephrasing model can learn well even with error-free data. This confirms that pre-trained language models can learn spelling correction from error-free data alone.
Shuffling of Characters Specifically, we shuffle the characters in the source and target sentences pairwise to spoil their semantics. We use these highly noisy samples to fine-tune the rephrasing and tagging models. From Table 2 (Shuf.), we find that the tagging model outperforms the rephrasing model on samples that do not convey semantic information.
Conversely, it verifies that the tagging model focuses more on point-to-point editing at the expense of semantics. As mentioned before, it is the semantics that are key to learning spelling correction from error-free data. Therefore, in this paper, we choose to rephrase as the primary training objective for self-supervised spelling correction.

heading: 4 Synthetic LEMON Training Set
text: To evaluate self-supervised models’ performance across multiple domains, we release a GPT-3.5generated synthetic LEMON training set.
LEMON (Wu et al., 2023b) is a multi-domain benchmark that allows us to evaluate the multidomain generalization of CSC models. However, it only includes a test set without a training set. The synthetic data is generated in two steps: (1) Extract the words in each domain. (2) Randomly select words and request GPT-3.5 to generate error-free sentences mimicking the style of specific domains. See our prompts in Appendix A.
Statistical information about the size of the different training sets is provided in Table 3.

heading: 2389 2489 1707 2222 2381 3669 4273

heading: 5 Method
text: In this section, we first introduce two rephrasing architectures. Then, we propose an enhanced decoding method to unleash the potential of pre-trained language models. Additionally, we suggest using a confusion dataset to improve the recall score.

heading: 5.1 Two Rephrasing Architectures
text: Our method can be implemented using two architectures: non-auto-regressive rephrasing and autoregressive rephrasing.
Auto-regressive Model Auto-regressive models, such as GPT-like models (Brown et al., 2020), are the primary choice for generating rephrasing.
To improve the quality of rephrasing, it is an easy yet effective way to mask a ratio of characters in the source sentence with an unused token (Wu et al., 2023b). In this paper, we denote the masked source sentence as X̃ = {x̃1, x̃2, · · · , x̃n}.
ReLM Rephrasing Language Model (ReLM) (Liu et al., 2024) is the current state-of-theart spelling correction model based on BERT (Devlin et al., 2019). It rephrases the source sentence by filling the masked slots. Specifically, the model is fed with the concatenation of the source sentence and a sequence of mask tokens. Due to the bidirectional nature of BERT, the rephrasing process can be expressed as P (yi|x̃1, x̃2, · · · , x̃n,m1,m2, · · · ,mn), where mi refers to the mask token. Unlike auto-regressive models, ReLM predicts all characters simultaneously.

heading: 5.2 Denoising Decoding Correction
text: The model trained with rephrasing still suffers from low recall when tested on real sentences because there are no mask tokens present. The situation becomes even more challenging when multiple errors occur in a single sentence. The cascade effect of these errors makes it increasingly difficult to correct the erroneous characters.
To address these problems, we propose a novel decoding algorithm, where we actively introduce noise to the source sentence to encourage the model to recall more candidates. Since the mask operation in the inference stage is consistent with that in the training stage of rephrasing, the model’s correction capability can be boosted. We call this method Denoising Decoding Correction (D2C).
Specifically, we mask the leftmost character in the source sentence if its confidence level falls below β (0.995). Such a character is considered a potential error. Then we send this masked sentence to the model and figure out whether the original character appears in the prediction’s top-k candidates. If it does, we keep the original character; otherwise, we note the new character and its confidence if this confidence is bigger than a threshold ϵ. We then mask the second character from the left and repeat the procedure. We refer to the process of masking all characters in the sentence as an iteration. After each iteration, we select the character with the highest recorded confidence and update the original sentence with it. We continue iterations until no further updates are needed. As the number of errors decreases, the challenges associated with correcting multiple typos also diminish. Thus, this iterative decoding method is robust against multiple errors.
We notice that picking a character with the biggest confidence in each iteration results in a large decoding overhead. Given that there is always a small number of errors in a sentence, we rank the characters in the sentence by their confidence from the lowest to highest, mask the top α of them respectively, and send the sentence to the model to figure out whether the original character appears in its top-k candidates. If it does, we remain the original character (same as original D2C strategy), else we update it with a new character that has the highest confidence if this confidence is bigger than a threshold ϵ.
Pseudo Code The overall procedure of D2C is described in Algorithm 1.

heading: 5.3 Fine-tune with Confusion Set
text: Given the low recall rate, we can improve the model by replacing some tokens with a confusion set instead of mask tokens during the fine-tuning process. The confusion set is constructed based on Chinese pronunciations and fonts. Using this confusion set, we can create a parallel dataset for training.
Algorithm 1: D2C Input: Input sentence Y ; Threshold ϵ;
Top-k; Language Model LM ; Set S. Output: Predict Result Z
1 for t ∈ [0,length(Y)] do 2 Clear S; 3 for i ∈ [0,length(Y)] do 4 Mask yi; 5 Get top-k predictions
{y1i , y2i , · · · , yki } and confidences {p1i , p2i , · · · , pki } from LM ;
6 if yi /∈ {y1i , · · · , yki } and p1i > ϵ then 7 Store y1i and p 1 i to S; 8 else 9 Continue;
10 end

heading: 11 end
text: 12 if S is empty then 13 Break; 14 else 15 Choose y1i with biggest p 1 i from S; 16 Replace yi with y1i ; 17 end 18 end 19 Z = Y ;
For an error-free sentence, we randomly select one character and replace it with a character from the confusion set.
Specifically, our method for using the confusion set is as follows: we initially train our selfsupervised model using entirely error-free data. Then, we generate parallel error-annotated data by introducing a rate of error-free data into the confusion set. Finally, we continue to fine-tune the model using this parallel data.

heading: 6 Experiments
text: In this section, we report the empirical results of a series of spelling correction benchmarks.
We concentrate on two benchmarks: • ECSpell (Lv et al., 2023): a small-scale multidomain Chinese spelling correction dataset of Law (LAW), medical treatment (MED), and official document writing (ODW), which is particular due to its large number of errors in the test set that do not appear in the training set; • Syn-LEMON: it is generated from LEMON
(Wu et al., 2023b) which spans 7 different domains with a total of 19,130 synthetic train samples.
We consider the following methods: • BERT (Devlin et al., 2019): the fine-tuned tagging model based on BERT; • MDCSpell (Zhu et al., 2022b): the strongest tagging model with a multi-task network of error detection and correction; • Masked-FT (MFT) (Wu et al., 2023b): a simple yet effective fine-tuning technique on tagging models to uniformly mask the non-error characters in the source sentence; • ReLM (Liu et al., 2024): the newly released state-of-the-art models on spelling correction, which rephrases the sentence in a non-autoregressive manner; • Baichuan2-7b (Yang et al., 2023): one of the strongest Chinese LLMs following the autoregressive architecture; • User Dictionary (UD) (Lv et al., 2023): an enhanced decoding method that leverages an expertise dictionary (law, medical treatment, and official document writing) to bias the beam search.

heading: 6.1 Training Settings
text: For BERT-based models, we set the batch size to 128 and the learning rate to 5e-5, swept from grid search. For Baichuan2, we set the batch size to 32 and the learning rate to 3e-4, and use LoRA (Hu et al., 2022) to reduce the training budget. For supervised spelling correction, the masking ratio is
chosen from {0.2, 0.3}, while for self-supervised spelling correction, it is set to 0.5.
When fine-tuning with the confusion set, we set the batch size to 64 and the learning rate to 5e-5.

heading: 6.2 Results on ECSpell
text: Table 4 highlights the effectiveness of rephrasing models when using error-free data and demonstrates the robust performance of D2C.
We first find that ReLM outperforms MDCSpellMFT by 35.1, 47.7, and 46.0 absolute points of F1 respectively on LAW, MED, and ODW,
When empowered with D2C, it further significantly produces the increase of 18.9, 7.1, and 14.0 absolute points. The biggest increase is in the recall rate, which is consistent with the design of D2C. Furthermore, we find that D2C is competitive against using a user dictionary (UD), or even more powerful. It suggests that some of the domain knowledge in the user dictionary has already been stored in the pre-trained language models, and D2C plays a key role in unlocking their great power.
When utilizing the confusion set, the increase is weaker. The confusion set method increases the FPR score, reaching 41.1 on MED. A higher confusion rate is related to a higher FPR score.

heading: 6.3 Results on Syn-LEMON
text: Table 5 summarizes the results of self-supervised methods on Syn-LEMON. It indicates that, except for GAM (game) and NOV (novel), using the con-
fusion set outperformed D2C’s F1 score. These variances reveal that different domains possess distinct data properties, significantly influencing performance outcomes when employing the confusion set versus D2C.

heading: 7 Discussion

heading: 7.1 D2C vs. Using Confusion Set
text: We compare D2C and the data augmentation method using the confusion set, a widely used technique in previous work. In Table 4, we find that D2C outperforms using the confusion set on two of the chosen datasets. Table 5 indicates that D2C surpasses using confusion set on GAM and NOV.
First, the results indicate that both D2C and using the confusion set can increase the recall rate. The common phenomenon is caused by different reasons. The confusion set introduces character-tocharacter corrections during the training process that are similar to test examples. While D2C introduces mask tokens to the test examples, which is inherited from the fine-tuning process. However, using the confusion set has a disadvantage compared with D2C. The non-matching segments in the confusion set can cause gaps in the real error patterns in the testing time. Therefore, using the confusion set always has lower P scores and higher FPR scores. D2C is a more suitable choice when it comes to domains that contain professional knowledge.
Second, compared to D2C, using the confusion set is relatively straightforward and efficient. Employing the confusion set presents an alternative approach in various application scenarios, offering efficiency but potentially posing a risk to performance.

heading: 7.2 Seen and Unseen Errors
text: To take a closer look at the correction ability, we divide the test set into two subsets, exclusive (E) and inclusive (I) sets, which refer to the test errors
that occur or do not occur in the training set. From Table 6, it is clear that supervised models fit the internal error set well but the performances drop sharply on the external error set. While models trained with error-free data have a high degree of similarity between the performance on the external error set and the internal error set. Besides, D2C boosts the performance on the external and internal sets simultaneously.
Surprisingly, MDCSpell-MFT performs even better on self-supervised learning than supervised on the exclusive set. This suggests that the tagging objective degenerates the learned representation in the pre-trained language model, leading to a drop in generalizability.

heading: 7.3 Effect of Mask Rate
text: We also investigate the impact of the mask rate. From Figure 2, it is apparent that the F1 scores for ECSpell’s Law improve consistently as the mask rate increases from 0% to approximately 30%, after which they experience a slight decline. A closer examination reveals that increasing the mask rate significantly enhances recall (R) scores more than precision (P) scores, while P scores tend to remain unchanged or even decline. Since the error-free fine-tuning process introduces noise solely through
mask tokens, the models are more inclined to preserve the source sentences without modification, resulting in lower R scores. During the evaluation stage, error characters serve as noise for the model; therefore, a higher mask rate improves the models’ performance on R scores.

heading: 7.4 Effect of Hyperparameters
text: We assess the effect of hyperparameters in D2C. As a representative, we depict the curves on ReLM in Figure 2.
Threshold Figure 2 shows that a higher threshold (ϵ) leads to improved performance. For instance, D2C with a higher ϵ (0.9) achieves better results on LAW.
Top-k There is a common phenomenon in Figure 2 that a higher top-k character count uplifts the F1 score under different thresholds ϵ.

heading: 7.5 Efficiency
text: We compare the decoding efficiency of D2C and normal decoding in Table 7. Our observations indicate that, compared to directly decoding each sentence, D2C requires approximately twice the time on ReLM and three times the time on Baichuan.

heading: 8 Case Study
text: We further showcase some examples to illustrate how D2C improves the decoding process.
Multi-typo In this case, (What are the innovations in meniscal (半月板) calcification (钙化)
), error characters are (钙→改) and (半→伴) , which are very similar in pronunciation but meaningless as words in the sentence. We noticed in the experiment that ReLM without D2C failed to correct this sentence with two error characters while successful with a single error character if one of the two errors has been corrected before. Therefore, with D2C we introduce noise into the source sentence to correct “伴” and “改” step by step.
Can’t Recall Considering sentences in spelling correction sometimes have short lengths, models receive limited semantic information and tend to under-correct error characters just like the case in Table 9. This case (How to calculate child’s weight (体重) ) has the error pattern of (体→休), which are similar in terms of their visual appearance. In the presence of semantics limitations, D2C directs models to reword specified positions to incorporate more suitable characters and effectively mitigate the issue of under-correction.

heading: 9 Conclusion
text: This paper studies self-supervised spelling correction based on rephrasing-based models. We demonstrate that machine spelling correction does not necessitate parallel data and can be learned from error-free data alone. We propose a novel decoding algorithm named D2C to effectively enhance the recall ability of the self-supervised model. We also compare the D2C method with the confusion set method. Results on Chinese spelling correction showcase the significant improvement brought by our method. We hope this paper can bring new in-
sight and vigor to future research on self-supervised spelling correction.
Limitations
Our work focuses on Chinese. Other languages, such as Korean have not been studied in this work. Additionally, D2C leads to a decline in the decoding speed.

heading: A Prompts
text: A.1 Extract Words Sentences are extracted from the original LEMON dataset.

heading: 1. Please extract the words in the given
text: sentences 2. Your answer should be in Chinese and JSON format {sentences} Your answer format: "words": ["word1", "word2",...], ["word1", "word2",...] ... ]}
A.2 Generate Data We propose the GAM domain’s prompt as an example.

heading: 1. You are a professional game writer.
text: Try to use your professional knowledge and think step by step.

heading: 2. Please make your answers diverse in formats, words, and expressions.

heading: 3. Generate 5 smooth sentences Using the given word sets

heading: 4. Your answer should be abundant and include details, but not too long

heading: 5. Try to generate realistic and fluent sentences like a human writer

heading: 6. Your answer should be in Chinese in JSON format
text: 7. Your generated sentence should follow the style of my given example sentences This is my given word sets: "words": ["word1", "word2",...], ["word1", "word2",...] ... ]} This is my given example sentences: {sentences} Your answer: [sentence1,sentence2,...]

heading: Future Work
text: We hope this paper can bring new in-
sight and vigor to future research on self-supervised spelling correction. Limitations
Our work focuses on Chinese. Other languages, such as Korean have not been studied in this work. Additionally, D2C leads to a decline in the decoding speed