Two Issues with Chinese Spelling Correction and A Refinement Solution (2024)

Abstract: The Chinese Spelling Correction (CSC) task aims to detect and correct misspelled characters in Chinese text, and has received lots of attention in the past few years. Most recent studies adopt a Transformer-based model and leverage different features of characters such as pronunciation, glyph and contextual information to enhance the model’s ability to complete the task. Despite their state-of-theart performance, we observe two issues that should be addressed to further advance the CSC task. First, the widely-used benchmark datasets SIGHAN13, SIGHAN14 and SIGHAN15, contain many mistakes. Hence the performance of existing models is not accurate and should be re-evaluated. Second, existing models seem to have reached a performance bottleneck, where the improvements on the SIGHAN’s testing sets are increasingly smaller and unstable. To deal with the two issues, we make two contributions: (1) we manually fix the SIGHAN datasets and re-evaluate four representative CSC models using the fixed datasets; (2) we analyze the new results to identify the spelling errors that none of the four models successfully corrects, based on which we propose a simple yet effective refinement solution. Experimental results show that our solution improves the four models in all metrics by notable margins.

text: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 196–204 August 11-16, 2024 ©2024 Association for Computational Linguistics
The Chinese Spelling Correction (CSC) task aims to detect and correct misspelled characters in Chinese text, and has received lots of attention in the past few years. Most recent studies adopt a Transformer-based model and leverage different features of characters such as pronunciation, glyph and contextual information to enhance the model’s ability to complete the task. Despite their state-of-theart performance, we observe two issues that should be addressed to further advance the CSC task. First, the widely-used benchmark datasets SIGHAN13, SIGHAN14 and SIGHAN15, contain many mistakes. Hence the performance of existing models is not accurate and should be re-evaluated. Second, existing models seem to have reached a performance bottleneck, where the improvements on the SIGHAN’s testing sets are increasingly smaller and unstable. To deal with the two issues, we make two contributions: (1) we manually fix the SIGHAN datasets and re-evaluate four representative CSC models using the fixed datasets; (2) we analyze the new results to identify the spelling errors that none of the four models successfully corrects, based on which we propose a simple yet effective refinement solution. Experimental results show that our solution improves the four models in all metrics by notable margins.

heading: 1 Introduction
text: Chinese Spelling Correction (CSC) aims to detect and correct misspelled characters in Chinese text. The task is challenging yet important, being used in various NLP applications such as search engines (Martins and Silva, 2004), optical character recognition (Afli et al., 2016) and international Chinese education (Liu et al., 2011). To solve the task, recent studies have employed Transformer (Vaswani et al., 2017) or BERT (Kenton
∗ Xuesong Lu is the corresponding author.
and Toutanova, 2019) as the base model and incorporated rich semantic features of characters to promote performance (Cheng et al., 2020; Liu et al., 2021; Xu et al., 2021; Li et al., 2022a; Liu et al., 2022; Liang et al., 2023; Huang et al., 2023).
Despite the promising results, we observe two issues with the current research for CSC. First, the widely-used benchmark datasets, SIGHAN13 (Wu et al., 2013), SIGHAN14 (Yu et al., 2014) and SIGHAN15 (Tseng et al., 2015), contain many mistakes, most of which are the meaningless sentences and the spelling errors in the target sentences. The former are the common mistakes made by Chinese beginners, as the SIGHAN datasets are collected from the Chinese essay section of Test for foreigners. These mistakes make the meaning of the sentences unclear and may affect the correction of spelling errors. The latter are the spelling errors that were not identified by the Chinese teachers in the test. Specifically, it is known that SIGHAN13 contains many misuses of “的”, “地” and “得” in the target sentences. These mistakes definitely affect the accuracy of the evaluation results. Surprisingly, previous studies have never attempted to fix the mistakes to better evaluate their models. Second, recent models seem to have reached a performance bottleneck on the SIGHAN’s testing sets, as evidenced by the increasingly smaller and unstable improvements (i.e., a newly proposed model does not perform better in all metrics) in the evaluation metrics. For instance, SCOPE (Li et al., 2022a) performs worse than MLM-phonetics (Zhang et al., 2021) in detection recall and correction recall on SIGHAN14 and performs worse than REALISE (Xu et al., 2021) in detection precision and correction precision on SIGHAN15. Furthermore, SCOPE combined with DR-CSC (Huang et al., 2023) improves SCOPE by only around 1 point in all metrics and also performs worse than comparative models in several metrics on SIGHAN13 and SIGHAN14. While
196
these models are focused on different aspects of spelling errors, we speculate the reason is that there exist certain errors which none of them can stably correct.
To tackle the two issues, we make two contributions in this paper. First, we examine the SIGHAN datasets sentence by sentence and fix all possible mistakes. Then, we retrain four representative CSC models using the fixed datasets and re-evaluate their performance. Second, we analyze the evaluation results and identify the spelling errors that none of the models successfully corrects, based on which we propose a simple solution to refine the output of the models without training. Experimental results show that our simple solution improves the four models in all metrics by notable margins.

heading: 2 Fixing SIGHAN and Re-evaluating Four Models
text: Two authors of the paper independently examine the SIGHAN datasets and identify the sentences with mistakes. Then they review each identified sentence and discuss whether it should be fixed and how to fix it. To ensure the accuracy of fixing, we fix the datasets in two rounds and both rounds take the same steps. First, we examine the fluency of the sentences and identify those that are meaningless. In this case, both a source sentence and the corresponding target sentence need to be fixed, and the spelling errors remain unchanged. Second, we identify the spelling errors in the target sentences.
Third, we identify the traditional Chinese characters that are not converted into simplified ones by OpenCC1 in both source and target sentences. Table 1 shows the example sentences with mistakes and the corresponding fixes. More examples are presented in Table 6 of the appendix.
Table 2 shows the statistics of fixes for the three datasets as well as the original statistics. The numbers in the parentheses are the numbers of sentences with spelling errors. Note that the rows indicated by "Fixed" show the statistics for the fixed sentences only. We observe that all three datasets have a considerable number of lines2 fixed, with many spelling errors including the newly-identified errors indicated in the square brackets. Note that a new spelling error is identified when a spelling error in a target sentence is fixed. That is, the numbers in the square brackets are the numbers of spelling errors in the target sentences of the original SIGHAN datasets.
Then, we select four representative CSC models and re-evaluate them on the fixed datasets, namely, PLOME (Liu et al., 2021), REALISE (Xu et al., 2021), LEAD (Li et al., 2022b) and SCOPE (Li et al., 2022a). The four models generally have the strongest performance among existing models according to the literature, and the authors have released the source code3 that are easily run. For each
1https://github.com/BYVoid/, Apache License 2.0. 2A line consists of a source sentence and a target sentence. 3PLOME: https://github.com/liushulinle/PLO ME, REALISE: https://github.com/DaDaMrX/ReaLiSe, LEAD: https://github.com/geekjuruo/LEAD, SCOPE: https://github.com/jiahaozhenbang/SCOPE
model, we adopt the training settings in the original paper. We train each model four times with random seeds and report the average results on the testing sets. We use the widely-adopted sentence-level precision, recall and F1 (Wang et al., 2019) to evaluate the models, which are also used in their original papers. The evaluation is conducted on detection and correction sub-tasks. The results are reported in Table 3, 4 and 5, where the rows indicated by “Original” are the results on the original SIGHAN datasets, and the rows indicated by “Retrained” are the results of the models retrained using the fixed SIGHAN datasets. The “Original” results are all copied from the corresponding papers except for PLOME on SIGHAN13 and SIGHAN14. The authors have not reported the results which we have to reproduce.
Comparing the results of “Original” and “Re-
trained”, we observe that the results are largely changed. On SIGHAN13, all results decrease drastically. This is mainly because the “Original” results are calculated after excluding “的”, “地” and “得”, since the targets are almost not correct, whereas the “Retrained” results are calculated on all spelling errors. This indicates the models can still not correct “的”, “地” and “得” well, especially for SCOPE which has the largest performance drop. On SIGHAN14 and SIGHAN15, the results generally increase after the datasets are fixed. Based on the results, we suggest to use the fixed datasets for more accurate evaluation in the future.
An interesting observation is that the “Retrained” results generally show the models ranked by performance from high to low are SCOPE4, LEAD, REALISE and PLOME, which coincides with the “Original” results. This indicates that we have correctly retrained the models and the fixed SIGHAN can reflect their performance discrepancies.

heading: 3 A Refinement Solution using ChineseBERT
text: We extract the sentences from the testing sets that none of the four models successfully reproduces the target sentence, and analyze the reasons of failures. We observe three main types of failures. First, the models often fail to correct the particles “的”, “地” and “得” and the pronouns such as “他（们）”, “她（们）”, “它（们）”, “那” and
4SCOPE seems to be much more affected by “的”, “地” and “得”. After excluding them, SCOPE performs better on the fixed SIGHAN13 as shown in Table 7 of the appendix.
“哪”. Second, the models often fail to correct the spelling errors in special terms, including idioms, proverbs, proper nouns and other commonly-used expressions. Third, the models often make overcorrections.
Most of the above failures can be solved by inferring the correct character using the contextual information of the corresponding sentence. Based on the idea, we propose a simple refinement solution with ChineseBERT (Sun et al., 2021) on top of the output of the four models. Specifically, given a sentence output by any model, we mask the character pertaining to the above failure cases, and let ChineseBERT infer the new character without training. Then we measure the phonological distance between the masked character and the inferred character, where the distance is calculated as the edit distance between the pinyins (with tone) of the two characters. If the distance is below a threshold5, we keep the inferred character; otherwise, we keep the masked character. The intuition is that about 83% spelling errors have similar pronunciation with the correct character (Liu et al., 2010), so if the inferred character has a very different pinyin than the masked character, it is unlikely to be the correct character. If there are multiple characters to mask in a sentence, we mask them one at a time and infer using ChineseBERT, from beginning to end. Once there is no character to mask, we stop the process and use the last output of ChineseBERT as the refined sentence. Note that if a sentence output by the above four models contains no character to mask, the sentence is the final output and the refinement process does not run.
The problem at hand is how to identify the characters to be masked. We design three strategies for the three failure types, respectively. First, we directly mask the particles “的”, “地” and “得” and the pronouns “他”, “她”, “它”, “那” and “哪”. Second, for a special term with spelling errors, we notice that the jieba6 tokenizer produces different tokens with and without the Hidden Markov Model (HMM). The former tends to regard it as a new word and the latter tends to tokenize it into single characters. Hence, for a sentence output by the above models, we use the two methods to tokenize it and regard the parts with different tokenization results as the special terms to mask. Note that this approach may mask phrases other than special
5In the experiments, we set the threshold to 3. 6https://github.com/fxsjy/jieba
terms if there exist spelling errors. Third, to identify over-corrections, we calculate the edit distance between the pinyins (with tone) of the changed character and the original character in the source sentence. If the distance is above 3 as discussed in the last paragraph, we regard it as a potential over-correction and mask the character.
The results are presented in Table 3, 4 and 5, indicated by “Refined”. We observe that after refinement, the performances of all the four models are improved by notable margins in all metrics on the three datasets, compared to the “Retrained” results. The results show our simple solution is very effective, even without training.

heading: 4 Related Work
text: Recent studies mainly adopt Transformer or BERT/ChineseBERT as the base model to solve the CSC task, and incorporate rich semantic features of the Chinese language to enhance the ability of the base model. For instance, Cheng et al. (2020) and Nguyen et al. (2021) use the confusion sets7 to exclude unlikely candidates output by BERT. More studies such as Xu et al. (2021); Huang et al. (2021); Liu et al. (2021); Li et al. (2022a,b); Liang et al. (2023); Zhang et al. (2023); Wei et al. (2023) leverage phonological and/or visual features of characters to boost the performance. Studies like Zhang et al. (2020, 2021); Li et al. (2021); Zhu et al. (2022); Huang et al. (2023) adopt the detection-correction framework to increase the accuracy of identifying potential spelling errors. Other studies learn contextual information in sentences to detect and correct spelling errors (Guo et al., 2021; Wang et al., 2021; Liu et al., 2022; Li et al., 2022c).

heading: 5 Conclusion
text: In this work, we discuss two issues with the Chinese Spelling Correction task: the existence of mistakes in the SIGHAN datasets and the smaller and unstable improvements of new models. We manually fix the mistakes and re-evaluate four representative CSC models on the fixed datasets. We analyze the common types of failures of the models and propose a simple yet effective refinement solution. Experimental results show our solution can stably improve the base models in all metrics. While the current refinement solution is purely rule
7The confusion sets are a collection of sets, where each set is formed with phonologically or visually similar characters.
based, in the future we will develop data-driven methods to further improve the performance.

heading: Limitations
text: There are two main limitations in the current work. First, the four models evaluated in the experiments belong to the category that incorporate phonological and visual features of Chinese characters. We choose them because they are reported in their papers to have the strongest performance among existing models and the source code are well maintained and released by the authors for reproducing and training. However, we should evaluate diverse models in the future, such as those using the detection-correction framework and those incorporating the contextual information. Second, our strategy to identify the characters in special terms and over-corrections to be masked is rule based and is not very accurate. For special terms with spelling errors, the identification depends on whether the jieba tokenizer with and without HMM yield different tokenization results. For over-corrections, we empirically identify them based on the edit distance between the pinyins (with tone) of a changed character and the original character. The threshold of the distance is set empirically and the visual distance is not considered, which is also the case for deciding whether to preserve the character inferred by ChineseBERT or not at the final output. While the current refinement solution is simple yet effective, we will explore more complex methods to further improve the accuracy of identifying the characters to be masked, as well as the final performance for CSC.

heading: Ethical Statement
text: The datasets and the models used in the current study are all released and authorized by the original authors for research purpose. These datasets contain neither identifying information nor any other ethical issues. The output of the models do not contain any violence, pornography or other inappropriate information. Hence, there is no ethical issue in the current study.

heading: Acknowledgement
text: This work is supported by the grant from the National Natural Science Foundation of China (Grant No. 62277017).

heading: Appendix

heading: Future Work
text: However, we should evaluate diverse models in the future, such as those using the detection-correction framework and those incorporating the contextual information. Second, our strategy to identify the characters in special terms and over-corrections to be masked is rule based and is not very accurate. For special terms with spelling errors, the identification depends on whether the jieba tokenizer with and without HMM yield different tokenization results. For over-corrections, we empirically identify them based on the edit distance between the pinyins (with tone) of a changed character and the original character. The threshold of the distance is set empirically and the visual distance is not considered, which is also the case for deciding whether to preserve the character inferred by ChineseBERT or not at the final output. While the current refinement solution is simple yet effective, we will explore more complex methods to further improve the accuracy of identifying the characters to be masked, as well as the final performance for CSC