SlowGuess/ACL_2020_OCR huggingface dataset:
Shaohua Zhang $^{1}$ , Haoran Huang $^{1}$ , Jicong Liu $^{2}$  and Hang Li $^{1}$
$^{1}$ ByteDance AI Lab
$^{2}$ School of Computer Science and Technology, Fudan University
{zhangshaohua.cs,huanghaoran,lihang.lh}@bytedance.com
jcliu15@fudan.edu.cn
Spelling error correction is an important yet challenging task because a satisfactory solution of it essentially needs human-level language understanding ability. Without loss of generality we consider Chinese spelling error correction (CSC) in this paper. A state-of-the-art method for the task selects a character from a list of candidates for correction (including non-correction) at each position of the sentence on the basis of BERT, the language representation model. The accuracy of the method can be sub-optimal, however, because BERT does not have sufficient capability to detect whether there is an error at each position, apparently due to the way of pre-training it using mask language modeling. In this work, we propose a novel neural architecture to address the aforementioned issue, which consists of a network for error detection and a network for error correction based on BERT, with the former being connected to the latter with what we call soft-masking technique. Our method of using 'Soft-Masked BERT' is general, and it may be employed in other language detection-correction problems. Experimental results on two datasets demonstrate that the performance of our proposed method is significantly better than the baselines including the one solely based on BERT.
Spelling error correction is an important task which aims to correct spelling errors in a text either at word-level or at character-level (Yu and Li, 2014; Yu et al., 2014; Zhang et al., 2015; Wang et al., 2018b; Hong et al., 2019; Wang et al., 2019). It is crucial for many natural language applications such as search (Martins and Silva, 2004; Gao et al., 2010), optical character recognition (OCR) (Afli et al., 2016; Wang et al., 2018b), and essay scoring (Burstein and Chodorow, 1999). In this pa
Table 1: Examples of Chinese spelling errors
Wrong: 埃及有金子塔。Egypt has golden towers.
Correct: 埃及有金字塔。Egypt has pyramids.
Wrong: 他的求胜欲很强，为了越狱在挖洞。  
He has a strong desire to win and is digging for prison breaks
Correct: 他的求生欲很强，为了越狱在挖洞。  
He has a strong desire to survive and is digging for prison breaks.
per, we consider Chinese spelling error correction (CSC) at character-level.
Spelling error correction is also a very challenging task, because to completely solve the problem the system needs to have human-level language understanding ability. There are at least two challenges here, as shown in Table 1. First, world knowledge is needed for spelling error correction. Character字 in the first sentence is mistakenly written as子, where金子塔 means golden tower and金字塔 means pyramid. Humans can correct the typo by referring to world knowledge. Second, sometimes inference is also required. In the second sentence, the 4-th character生 is mistakenly written as胜. In fact, 胜 and the surrounding characters form a new valid word 求胜欲 (desire to win), rather than the intended word 求生欲 (desire to survive).
Many methods have been proposed for CSC or more generally spelling error correction. Previous approaches can be mainly divided into two categories. One employs traditional machine learning and the other deep learning (Yu et al., 2014; Tseng et al., 2015; Wang et al., 2018b). Zhang et al. (2015), for example, proposed a unified framework for CSC consisting of a pipeline of error de
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 882-890
tection, candidate generation, and final candidate selection using traditional machine learning. Wang et al. (2019) proposed a Seq2Seq model with copy mechanism which transforms an input sentence into a new sentence with spelling errors corrected.
More recently, BERT (Devlin et al., 2018), the language representation model, is successfully applied to many language understanding tasks including CSC (cf., (Hong et al., 2019)). In the state-of-the-art method using BERT, a character-level BERT is first pre-trained using a large unlabelled dataset and then fine-tuned using a labeled dataset. The labeled data can be obtained via data augmentation in which examples of spelling errors are generated using a large confusion table. Finally the model is utilized to predict the most likely character from a list of candidates at each position of the given sentence. The method is powerful because BERT has certain ability to acquire knowledge for language understanding. Our experimental results show that the accuracy of the method can be further improved, however. One observation is that the error detection capability of the model is not sufficiently high, and once an error is detected, the model has a better chance to make a right correction. We hypothesize that this might be due to the way of pre-training BERT with mask language modeling in which only about  $15\%$  of the characters in the text are masked, and thus it only learns the distribution of masked tokens and tends to choose not to make any correction. This phenomenon is prevalent and represents a fundamental challenge for using BERT in certain tasks like spelling error correction.
To address the above issue, we propose a novel neural architecture in this work, referred to as Soft-Masked BERT. Soft-Masked BERT contains two networks, a detection network and a correction network based on BERT. The correction network is similar to that in the method of solely using BERT. The detection network is a Bi-GRU network that predicts the probability that the character is an error at each position. The probability is then utilized to conduct soft-masking of embedding of character at the position. Soft masking is an extension of conventional 'hard masking' in the sense that the former degenerates to the latter, when the probability of error equals one. The soft-masked embedding at each position is then inputted into the correction network. The correction network conducts error correction using BERT. This approach can force the model to learn the right context for error correc
tion under the help of the detection network, during end-to-end joint training.
We conducted experiments to compare Soft-Masked BERT and several baselines including the method of using BERT alone. As datasets we utilized the benchmark dataset of SIGHAN. We also created a large and high-quality dataset for evaluation named News Title. The dataset, which contains titles of news articles, is ten times larger than the previous datasets. Experimental results show that Soft-Masked BERT significantly outperforms the baselines on the two datasets in terms of accuracy measures.
The contributions of this work include (1) proposal of the novel neural architecture Soft-Masked BERT for the CSC problem, (2) empirical verification of the effectiveness of Soft-Masked BERT.
2.1 Problem and Motivation
Chinese spelling error correction (CSC) can be formalized as the following task. Given a sequence of  $n$  characters (or words)  $X = (x_{1}, x_{2}, \dots, x_{n})$ , the goal is to transform it into another sequence of characters  $Y = (y_{1}, y_{2}, \dots, y_{n})$  with the same length, where the incorrect characters in  $X$  are replaced with the correct characters to obtain  $Y$ . The task can be viewed as a sequential labeling problem in which the model is a mapping function  $f: X \to Y$ . The task is an easier one, however, in the sense that usually no or only a few characters need to be replaced and all or most of the characters should be copied.
The state-of-the-art method for CSC is to employ BERT to accomplish the task. Our preliminary experiments show that the performance of the approach can be improved, if the erroneous characters are designated (cf., section 3.6). In general the BERT based method tends to make no correction (or just copy the original characters). Our interpretation is that in pre-training of BERT only  $15\%$  of the characters are masked for prediction, resulting in learning of a model which does not possess enough capacity for error detection. This motives us to devise a new model.
2.2 Model
We propose a novel neural network model called Soft-Masked BERT for CSC, as illustrated in Figure 1. Soft-Masked BERT is composed of a detection network based on Bi-GRU and a correction net
883
Figure 1: Architecture of Soft-Masked BERT
work based on BERT. The detection network predicts the probabilities of errors and the correction network predicts the probabilities of error corrections, while the former passes its prediction results to the latter using soft masking.
More specifically, our method first creates an embedding for each character in the input sentence, referred to as input embedding. Next, it takes the sequence of embeddings as input and outputs the probabilities of errors for the sequence of characters (embeddings) using the detection network. After that it calculates the weighted sum of the input embeddings and [MASK] embeddings weighted by the error probabilities. The calculated embeddings mask the likely errors in the sequence in a soft way. Then, our method takes the sequence of soft-masked embeddings as input and outputs the probabilities of error corrections using the correction network, which is a BERT model whose final layer consists of a softmax function for all characters. There is also a residual connection between the input embeddings and the embeddings at the final layer. Next, we describe the details of the model.
2.3 Detection Network
The detection network is a sequential binary labeling model. The input is the sequence of embeddings  $E = (e_1, e_2, \dots, e_n)$ , where  $e_i$  denotes the embedding of character  $x_i$ , which is the sum of word embedding, position embedding, and segment
embedding of the character, as in BERT. The output is a sequence of labels  $G = (g_{1}, g_{2}, \dots, g_{n})$ , where  $g_{i}$  denotes the label of the  $i$  character, and 1 means the character is incorrect and 0 means it is correct. For each character there is a probability  $p_{i}$  representing the likelihood of being 1. The higher  $p_{i}$  is the more likely the character is incorrect.
In this work, we realize the detection network as a bidirectional GRU (Bi-GRU). For each character of the sequence, the probability of error  $p_i$  is defined as
$$
p _ {i} = P _ {d} (g _ {i} = 1 | X) = \sigma \left(W _ {d} h _ {i} ^ {d} + b _ {d}\right) \quad (1)
$$
where  $P_{d}(g_{i} = 1|X)$  denotes the conditional probability given by the detection network,  $\sigma$  denotes the sigmoid function,  $h_i^d$  denotes the hidden state of Bi-GRU,  $W_{d}$  and  $b_{d}$  are parameters. Furthermore, the hidden state is defined as
$$
\overrightarrow {h _ {i} ^ {d}} = \operatorname {G R U} \left(\overrightarrow {h} _ {i - 1} ^ {d}, e _ {i}\right) \tag {2}
$$
$$
\overleftarrow {h _ {i} ^ {d}} = \operatorname {G R U} \left(\overleftarrow {h} _ {i + 1} ^ {d}, e _ {i}\right) \tag {3}
$$
$$
h _ {i} ^ {d} = \overrightarrow {\left[ h _ {i} ^ {d} ; h _ {i} ^ {d} \right]} \tag {4}
$$
where  $\overrightarrow{[h_i^d; h_i^d]}$  denotes concatenation of GRU hidden states from the two directions and GRU is the GRU function.
Soft masking amounts to a weighted sum of input embeddings and mask embeddings with error probabilities as weights. The soft-masked embed
ding  $e_i^{\prime}$  for the  $i$ -th character is defined as
$$
e _ {i} ^ {\prime} = p _ {i} \cdot e _ {\text {m a s k}} + (1 - p _ {i}) \cdot e _ {i} \tag {5}
$$
where  $e_i$  is the input embedding and  $e_{mask}$  is the mask embedding. If the probability of error is high, then soft-masked embedding  $e_i'$  is close to the mask embedding  $e_{mask}$ ; otherwise it is close to the input embedding  $e_i$ .
2.4 Correction Network
The correction network is a sequential multi-class labeling model based on BERT. The input is the sequence of soft-masked embeddings  $E' = (e_1', e_2', \dots, e_n')$  and the output is a sequence of characters  $Y = (y_1, y_2, \dots, y_n)$ .
BERT consists of a stack of 12 identical blocks taking the entire sequence as input. Each block contains a multi-head self-attention operation followed by a feed-forward network, defined as:
MultiHead(Q,K,V)
$$
= \operatorname {C o n c a t} \left(\operatorname {h e a d} _ {1}; \dots , \operatorname {h e a d} _ {h}\right) W ^ {O} \tag {6}
$$
$\mathrm{head}_i = \mathrm{Attention}(QW_i^Q,KW_i^K,VW_i^V)$  (7)
$$
\operatorname {F F N} (X) = \max  \left(0, X W _ {1} + b _ {1}\right) W _ {2} + b _ {2} \tag {8}
$$
where  $Q, K,$  and  $V$  are the same matrices representing the input sequence or the output of the previous block, MultiHead, Attention, and FNN denote multi-head self-attention, self-attention, and feedforward network respectively,  $W^{O}, W_{i}^{Q}, W_{i}^{K}, W_{i}^{V}, W_{1}, W_{2}, b_{1}$ , and  $b_{2}$  are parameters. We denote the sequence of hidden states at the final layer of BERT as  $H^{c} = (h_{1}^{c}, h_{2}^{c}, \dots, h_{n}^{c})$
For each character of the sequence, the probability of error correction is defined as
$$
P _ {c} \left(y _ {i} = j \mid X\right) = \operatorname {s o f t m a x} \left(W h _ {i} ^ {\prime} + b\right) [ j ] \tag {9}
$$
where  $P_{c}(y_{i} = j|X)$  is the conditional probability that character  $x_{i}$  is corrected as character  $j$  in the candidate list, softmax is the softmax function,  $h_i^\prime$  denotes the hidden state,  $W$  and  $b$  are parameters. Here the hidden state  $h_i^\prime$  is obtained by linear combination with the residual connection,
$$
h _ {i} ^ {\prime} = h _ {i} ^ {c} + e _ {i} \tag {10}
$$
where  $h_i^c$  is the hidden state at the final layer and  $e_i$  is the input embedding of character  $x_i$ . The last layer of correction network exploits a softmax function. The character that has the largest probability is selected from the list of candidates as output for character  $x_i$ .
The learning of Soft-Masked BERT is conducted end-to-end, provided that BERT is pre-trained and training data is given which consists of pairs of original sequence and corrected sequence, denoted as  $= \{(X_1,Y_1),(X_2,Y_2),\ldots ,(X_N,Y_N)\}$ . One way to create the training data is to repeatedly generate a sequence  $X_{i}$  containing errors given a sequence  $Y_{i}$  without an error, using a confusion table, where  $i = 1,2,\dots ,N$
The learning process is driven by optimizing two objectives, corresponding to error detection and error correction respectively.
$$
\mathcal {L} _ {d} = - \sum_ {i = 1} ^ {n} \log P _ {d} \left(g _ {i} | X\right) \tag {11}
$$
$$
\mathcal {L} _ {c} = - \sum_ {i = 1} ^ {n} \log P _ {c} \left(y _ {i} | X\right) \tag {12}
$$
where  $\mathcal{L}_d$  is the objective for training of the detection network, and  $\mathcal{L}_c$  is the objective for training of the correction network (and also the final decision). The two functions are linearly combined as the overall objective in learning.
$$
\mathcal {L} = \lambda \cdot \mathcal {L} _ {c} + (1 - \lambda) \cdot \mathcal {L} _ {d} \tag {13}
$$
where  $\lambda \in [0,1]$  is coefficient.
We made use of the SIGHAN dataset, a benchmark for  $\mathrm{CSC}^1$ . SIGHAN is a small dataset containing 1,100 texts and 461 types of errors (characters). The texts are collected from the essay section of Test of Chinese as Foreign Language and the topics are in a narrow scope. We adopted the standard split of training, development, and test data of SIGHAN.
We also created a much larger dataset for testing and development, referred to as News Title. We sampled the titles of news articles at Toutiao, a Chinese news app with a large variety of content in politics, entertainment, sports, education, etc. To ensure that the dataset contains a sufficient number of incorrect sentences, we conducted the sampling from lower quality texts, and thus the error rate of
$^{1}$ Following the common practice (Wang et al., 2019), we converted the characters in the dataset from traditional Chinese to simplified Chinese.
the dataset is higher than usual. Three people conducted five rounds of labeling to carefully correct spelling errors in the titles. The dataset contains 15,730 texts. There are 5,423 texts containing errors, in 3,441 types. We divided the data into test set and development set, each containing 7,865 texts.
In addition, we followed the common practice in CSC to automatically generate a dataset for training. We first crawled about 5 million news titles at the Chinese news app. We also created a confusion table in which each character is associated with a number of homophonous characters as potential errors. Next, we randomly replaced  $15\%$  of the characters in the texts with other characters to artificially generate errors, where  $80\%$  of them are homophonous characters in the table and  $20\%$  of them are random characters. This is because in practice about  $80\%$  of spelling errors in Chinese are homophonous characters due to the use of Pinyin-based input methods by people.
For comparison, we adopted the following methods as baselines. We report the results of the methods from their original papers.
NTOU is a method of using an n-gram model and a rule-based classifier (Tseng et al., 2015). NCTU-NTUT is a method of utilizing word vectors and conditional random field (Tseng et al., 2015). HanSpeller++ is an unified framework employing a hidden Markov model to generate candidates and a filter to re-rank candidates (Zhang et al., 2015). Hybrid uses a BiLSTM-based model trained on a generated dataset (Wang et al., 2018b). Confusionset is a Seq2Seq model consisting of a pointer network and copy mechanism (Wang et al., 2019). FASPell adopts a Seq2Seq model for CSC employing BERT as a denoising autoencoder and a decoder (Hong et al., 2019). BERT-Pretrain is the method of using a pre-trained BERT. BERT-Finetune is the method of using a fine-tuned BERT.
3.3 Experiment Setting
As evaluation measures, we utilized sentence-level accuracy, precision, recall, and F1 score as in most of the previous work. We evaluated the accuracy of a method in both detection and correction. Obviously correction is more difficult than detection, because the former is dependent on the latter.
The pre-trained BERT model utilized in the experiments is the one provided at https://github.com/huggingface/transformers. In fine-tuning of BERT, we kept the default hyperparameters and only fine-tuned the parameters using Adam. In order to reduce the impact of training tricks, we did not use the dynamic learning rate strategy and maintained a learning rate  $2e^{-5}$  in fine-tuning. The size of hidden unit in Bi-GRU is 256 and all models use a batch size of 320.
In the experiments on SIGHAN, for all BERT-based models, we first fine-tuned the model with the 5 million training examples and then continued the fine-tuning with the training examples in SIGHAN. We removed the unchanged texts in the training data to improve the efficiency. In the experiments on News Title, the models were fine-tuned only with the 5 million training examples.
The development sets were utilized for hyperparameter tuning for both SIGHAN and News Title. The best value for hyper-parameter  $\lambda$  was chosen for each dataset.
Table 2 presents the experimental results of all methods on the two test datasets. From the table, one can observe that the proposed model SoftMasked BERT significantly outperforms the baseline methods on both datasets. Particularly, on News Title, Soft-Masked BERT performs much better than the baselines in terms of all measures. The best results for recall of correction level on the News Title dataset are greater than  $54\%$ , which means more than  $54\%$  errors will be found and correction level precision are better than  $55\%$ .
HanSpeller++ achieves the highest precision on SIGHAN, apparently because it can eliminate false detections with its large number of manually-crafted rules and features. Although the use of rules and features is effective, the method has high cost in development and may also have difficulties in generalization and adaptation. In some sense, it is not directly comparable with the other learning-based methods including Soft-Masked BERT. The results of all methods except Confusionset are at sentence level not at character level. (The results at character level can look better.) Nonetheless, Soft-Mask BERT still performs significantly better.
The three methods of using BERT, Soft-Masked BERT, BERT-Finetune, and FASPell, perform better than the other baselines, while the method of
Table 2: Performances of Different Methods on CSC
Test Set	Method	Detection	Correction
Acc.	Prec.	Rec.	F1.	Acc.	Prec.	Rec.	F1.
SIGHAN	NTOU (2015)	42.2	42.2	41.8	42.0	39.0	38.1	35.2	36.6
NCTU-NTUT (2015)	60.1	71.7	33.6	45.7	56.4	66.3	26.1	37.5
HanSpeller++ (2015)	70.1	80.3	53.3	64.0	69.2	79.7	51.5	62.5
Hybird (2018b)	-	56.6	69.4	62.3	-	-	-	57.1
FASPell (2019)	74.2	67.6	60.0	63.5	73.7	66.6	59.1	62.6
Confusionset (2019)	-	66.8	73.1	69.8	-	71.5	59.5	64.9
BERT-Pretrain	6.8	3.6	7.0	4.7	5.2	2.0	3.8	2.6
BERT-Finetune	80.0	73.0	70.8	71.9	76.6	65.9	64.0	64.9
Soft-Masked BERT	80.9	73.7	73.2	73.5	77.4	66.7	66.2	66.4
News Title	BERT-Pretrain	7.1	1.3	3.6	1.9	0.6	0.6	1.6	0.8
BERT-Finetune	80.0	65.0	61.5	63.2	76.8	55.3	52.3	53.8
Soft-Masked BERT	80.8	65.5	64.0	64.8	77.6	55.8	54.5	55.2
Table 3: Impact of Different Sizes of Training Data
Train Set	Method	Detection	Correction
Acc.	Prec.	Rec.	F1.	Acc.	Prec.	Rec.	F1.
500,000	BERT-Finetune	71.8	49.6	48.2	48.9	67.4	36.5	35.5	36.0
Soft-Masked BERT	72.3	50.3	49.6	50.0	68.2	37.9	37.4	37.6
1,000,000	BERT-Finetune	74.2	54.7	51.3	52.9	70.0	41.6	39.0	40.3
Soft-Masked BERT	75.3	56.3	54.2	55.2	71.1	43.6	41.9	42.7
2,000,000	BERT-Finetune	77.0	59.7	57.0	58.3	73.1	48.0	45.8	46.9
Soft-Masked BERT	77.6	60.0	58.5	59.2	73.7	48.4	47.3	47.8
5,000,000	BERT-Finetune	80.0	65.0	61.5	63.2	76.8	55.3	52.3	53.8
Soft-Masked BERT	80.8	65.5	64.0	64.8	77.6	55.8	54.5	55.2
BERT-Pretrain performs fairly poorly. The results indicate that BERT without fine-tuning (i.e., BERT-Pretrain) would not work and BERT with fine-tuning (i.e., BERT-Finetune, etc) can boost the performances remarkably. Here we see another successful application of BERT, which can acquire certain amounts of knowledge for language understanding. Furthermore, Soft-Masked BERT can beat BERT-Finetune by large margins on both datasets. The results suggest that error detection is important for the utilization of BERT in CSC and soft masking is really an effective means.
3.5 Effect of Hyper Parameter
We present the results of Soft-Masked BERT on (the test data of) News Title to illustrate the effect of parameter and data size.
Table 3 shows the results of Soft-Masked BERT as well as BERT-Finetune learned with different sizes of training data. One can find that the best result is obtained for Soft-Masked BERT when the
size is 5 million, indicating that the more training data is utilized the higher performance can be achieved. One can also observe that Soft-Masked BERT is consistently superior to BERT-Finetune.
A larger  $\lambda$  value means a higher weight on error correction. Error detection is an easier task than error correction, because essentially the former is a binary classification problem while the latter is a multi-class classification problem. Table 5 presents the results of Soft-Masked BERT in different values of hyper-parameter  $\lambda$ . The highest F1 score is obtained when  $\lambda$  is 0.8. That means that a good compromise between detection and correction is reached.
We carried out ablation study on Soft-Masked BERT on both datasets. Table 4 shows the results on News Title. (We omit the results on SIGHAN due to space limitation, which have similar trends.) In Soft-Masked BERT-R, the residual connection