SlowGuess/ACL_2020_OCR huggingface dataset:
Table 4: Ablation Study of Soft-Masked BERT on News Title
Method	Detection	Correction
Acc.	Prec.	Rec.	F1.	Acc.	Prec.	Rec.	F1.
BERT-Finetune +Force(Upper Bound)	89.9	75.6	90.3	82.3	82.9	58.4	69.8	63.6
Soft-Masked BERT	80.8	65.5	64.0	64.8	77.6	55.8	54.5	55.2
Soft-Masked BERT-R	81.0	75.2	53.9	62.8	78.4	64.6	46.3	53.9
Rand-Masked BERT	70.9	46.6	48.5	47.5	68.1	38.8	40.3	39.5
BERT-Finetune	80.0	65.0	61.5	63.2	76.8	55.3	52.3	53.8
Hard-Masked BERT (0.95)	80.6	65.3	63.2	64.2	76.7	53.6	51.8	52.7
Hard-Masked BERT (0.9)	77.4	57.8	60.3	59.0	72.4	44.0	45.8	44.9
Hard-Masked BERT (0.7)	65.3	38.0	50.9	43.5	58.9	24.2	32.5	27.7
Table 5: Impact of Different Values of  $\lambda$
λ	Detection	Correction
Acc.	Pre.	Rec.	F1.	Acc.	Pre.	Rec.	F1.
0.8	72.3	50.3	49.6	50.0	68.2	37.9	37.4	37.6
0.5	72.3	50.0	49.3	49.7	68.0	37.5	37.0	37.3
0.2	71.5	48.6	50.4	49.5	66.9	35.7	37.1	36.4
in the model is removed. In Hard-Masked BERT, if the error probability given by the detection network exceeds a threshold (0.95, 0.9, 07), then the embedding of the current character is set to the embedding of the [MASK] token, otherwise the embedding remains unchanged. In Rand-Masked BERT, the error probability is randomized with a value between 0 and 1. We can see that all the major components of Soft-Masked BERT are necessary for achieving high performance. We also tried 'BERT-Finetune + Force', whose performance can be viewed as an upper bound. In the method, we let BERT-Finetune to only make prediction at the position where there is an error and select a character from the rest of the candidate list. The result indicates that there is still large room for Soft-Masked BERT to make improvement.
3.7 Discussions
We observed that Soft-Masked BERT is able to make more effective use of global context information than BERT-Finetune. With soft masking the likely errors are identified, and as a result the model can better leverage the power of BERT to make sensible reasoning for error correction by referring to not only local context but also global context. For example, there is a typo in the sentence '我会说一点儿，不过一个汉子也看不懂，所以我迷路
了'(I can speak a little Chinese, but I don't understand man. So I got lost.). The word '汉子'(man) is incorrect and should be written as '汉字'(Chinese character). BERT-Finetune can not rectify the mistake, but Soft-Masked BERT can, because the error correction can only be accurately conducted with global context information.
We also found that there are two major types of errors in almost all methods including Soft-Masked BERT, which affect the performances. For statistics of errors, we sampled 100 errors from test set. We found that  $67\%$  of errors require strong reasoning ability,  $11\%$  of errors are due to lack of world knowledge, and the remaining  $22\%$  of errors have no significant type.
The first type of errors is due to lack of inference ability. Accurate correction of such typos requires stronger inference ability. For example, for the sentence '他主动拉了姑娘的手, 心里很高心, 嘴上故作生气' (He intentionally took the girl's hand, and was very x, but was pretending to be angry.) where the incorrect word 'x' is not comprehensible, there might be two possible corrections, changing '高心' to '寒心'(chilled) and changing '高心' to '高兴'(happy), while the latter is more reasonable for humans. One can see that in order to make more reliable corrections, the models must have stronger inference ability.
The second type of errors is due to lack of world knowledge. For example, in the sentence '芜湖: 女子落入青戈江,众人齐救援' (Wuhu: the woman fell into the Qingge River, and people tried hard to rescue her.), '青戈江' (Qingge River) is a typo of '青弋江' (Qingyu River). Humans can discover the typo because the river in Wuhu city China is called Qingyu not Qingge. It is still very challenging for the existing models in general AI systems to detect
and correct such kind of errors.
Various studies have been conducted on spelling error correction so far, which plays an important role in many applications, including search (Gao et al., 2010), optical character recognition (OCR) (Afli et al., 2016), and essay scoring (Burstein and Chodorow, 1999).
Chinese spelling error correction (CSC) is a special case, but is more challenging due to its conflation with Chinese word segmentation, which received a considerable number of investigations (Yu et al., 2014; Yu and Li, 2014; Tseng et al., 2015; Wang et al., 2019). Early work in CSC followed the pipeline of error detection, candidate generation, and final candidate selection. Some researchers employed unsupervised methods using language models and rules (Yu and Li, 2014; Tseng et al., 2015) and the others viewed it as a sequential labeling problem and employed conditional random fields or hidden Markov models (Tseng et al., 2015; Zhang et al., 2015). Recently, deep learning was applied to spelling error correction (Guo et al., 2019; Wang et al., 2019), and for example, a Seq2Seq model with BERT as encoder was employed (Hong et al., 2019), which transforms the input sentence into a new sentence with spelling errors corrected.
BERT (Devlin et al., 2018) is a language representation model with Transformer encoder as its architecture. BERT is first pre-trained using a very large corpus in a self-supervised fashion (mask language modeling and next sentence prediction). Then, it is fine-tuned using a small amount of labeled data in a down-stream task. Since its inception BERT has demonstrated superior performances in almost all the language understanding tasks, such as those in the GLUE challenge (Wang et al., 2018a). BERT has shown strong ability to acquire and utilize knowledge for language understanding. Recently, other language representation models have also been proposed, such as XLNET (Yang et al., 2019), Roberta (Liu et al., 2019), and ALBERT (Lan et al., 2019). In this work, we extend BERT to Soft Masked BERT for spelling error correction and as far as we know no similar architecture was proposed before.
In this paper, we have proposed a novel neural network architecture for spelling error correction,
more specifically Chinese spelling error correction (CSC). Our model called Soft-Masked BERT is composed of a detection network and a correction network based on BERT. The detection network identifies likely incorrect characters in the given sentence and soft-masks the characters. The correction network takes the soft-masked characters as input and makes correction on the characters. The technique of soft-masking is general and potentially useful in other detection-correction tasks. Experimental results on two datasets show that Soft-Masked BERT significantly outperforms the state-of-art method of solely utilizing BERT. As future work, we plan to extend Soft-Masked BERT to other problems like grammatical error correction and explore other possibilities of implementing the detection network.
Haithem Afli, Zhengwei Qiu, Andy Way, and Páraic Sheridan. 2016. Using smt forOCR error correction of historical texts. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16), pages 962-966.
Jill Burstein and Martin Chodorow. 1999. Automated essay scoring for nonnative english speakers. In Proceedings of a Symposium on Computer Mediated Language Assessment and Evaluation in Natural Language Processing, pages 68-75. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
Jianfeng Gao, Xiaolong Li, Daniel Micol, Chris Quirk, and Xu Sun. 2010. A large scale ranker-based system for search query spelling correction. In COLING 2010, 23rd International Conference on Computational Linguistics, Proceedings of the Conference, 23-27 August 2010, Beijing, China, pages 358-366.
Jinxi Guo, Tara N Sainath, and Ron J Weiss. 2019. A spelling correction model for end-to-end speech recognition. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5651-5655. IEEE.
Yuzhong Hong, Xianguo Yu, Neng He, Nan Liu, and Junhui Liu. 2019. Faspell: A fast, adaptable, simple, powerful chinese spell checker based on daedecoder paradigm. In Proceedings of the 5th Workshop on Noisy User-generated Text (W-NUT 2019), pages 160-169.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019. Albert: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv:1909.11942.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
Bruno Martins and Mário J. Silva. 2004. Spelling correction for search engine queries. In Advances in Natural Language Processing, 4th International Conference, EstAL 2004, Alicante, Spain, October 20-22, 2004, Proceedings, pages 372-383.
Yuen-Hsien Tseng, Lung-Hao Lee, Li-Ping Chang, and Hsin-Hsi Chen. 2015. Introduction to sighan 2015 bake-off for chinese spelling check. In Proceedings of the Eighth SIGHAN Workshop on Chinese Language Processing, pages 32-37.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2018a. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461.
Dingmin Wang, Yan Song, Jing Li, Jialong Han, and Haisong Zhang. 2018b. A hybrid approach to automatic corpus generation for chinese spelling check. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2517-2527.
Dingmin Wang, Yi Tay, and Li Zhong. 2019. Confusionset-guided pointer networks for Chinese spelling check. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5780-5785.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. arXiv preprint arXiv:1906.08237.
Junjie Yu and Zhenghua Li. 2014. Chinese spelling error detection and correction based on language model, pronunciation, and shape. In Proceedings of The Third CIPS-SIGHAN Joint Conference on Chinese Language Processing, pages 220-223, Wuhan, China. Association for Computational Linguistics.
Liang-Chih Yu, Lung-Hao Lee, Yuen-Hsien Tseng, and Hsin-Hsi Chen. 2014. Overview of sighan 2014 bake-off for chinese spelling check. In Proceedings of The Third CIPS-SIGHAN Joint Conference on Chinese Language Processing, pages 126-132.
Shuiyuan Zhang, Jinhua Xiong, Jianpeng Hou, Qiao Zhang, and Xueqi Cheng. 2015. Hanspeller++: A unified framework for Chinese spelling correction. In Proceedings of the Eighth SIGHAN Workshop on Chinese Language Processing, pages 38-45.