SlowGuess/ACL_2020_OCR huggingface dataset:
Xingyi Cheng* Weidi Xu* Kunlong Chen
Shaohua Jiang Feng Wang Taifeng Wang Wei Chu Yuan Qi
{fanyin.cxy,weidi.xwd,kunlong.ckl,shaohua.jsh,zifan.wf, taifeng.wang,weichu.cw,yuan.qi}@alibaba-inc.com
Chinese Spelling Check (CSC) is a task to detect and correct spelling errors in Chinese natural language. Existing methods have made attempts to incorporate the similarity knowledge between Chinese characters. However, they take the similarity knowledge as either an external input resource or just heuristic rules. This paper proposes to incorporate phonological and visual similarity knowledge into language models for CSC via a specialized graph convolutional network (SpellGCN). The model builds a graph over the characters, and SpellGCN is learned to map this graph into a set of inter-dependent character classifiers. These classifiers are applied to the representations extracted by another network, such as BERT, enabling the whole network to be end-to-end trainable. Experiments 1 are conducted on three human-annotated datasets. Our method achieves superior performance against previous models by a large margin.
Spelling errors are common in our daily life, caused typically by human writing, automatic speech recognition, and optical character recognition systems. Among these errors, misspelling a character frequently occurs due to the similarity between characters. In Chinese, many characters are phonologically and visually similar, but semantically very different. According to Liu et al. (2010), about  $83\%$  of errors are related to phonological similarity and  $48\%$  are related to visual similarity. The Chinese Spelling Check (CSC) task aims to detect and correct such misuse of the Chinese language. Despite recent development, CSC remains a challenging task. Notably, the spelling checking on Chinese is very different from English, due to its language
Input
(phonics)	餐厅的换经费产适合约会
cān ting de huan jing fèi chán shi hé yuē huì
BERT
(phonics)	餐厅的月消费最适合约会
cān ting de yuè xiāo fèi zuì shi hé yuē huì
Gold Label
(phonics)	餐厅的环境非常适合约会
cān ting de huan jing fèi cháng shi hé yuē huì
Table 1: A CSC data sample from SIGHAN 2014 (Yu et al., 2014) with ID B1-3440-2, the incorrect/correct characters are in orange/blue. A BERT model modifies the text into a sentence that is semantically reasonable but dissimilar in pronunciation. By incorporating both phonological and visual similarities, our new method SpellGCN can generate a sentence that is both semantically sensible and phonically similar to the original sentence. The sentence output from SpellGCN means "this restaurant is very suitable for dating".
nature. Chinese is a language consisting of many pictographic characters without word delimiters. And the meaning of each character changes dramatically when the context changes. Therefore, a CSC system needs to recognize the semantics and aggregate the surrounding information for necessary modifications.
Previous methods followed the line of generative models. They used either language models (Liu et al., 2013, 2010; Yu and Li, 2014) or sequence-to-sequence models (Wang et al., 2019). To fuse the external knowledge of the similarity between characters, some of them leveraged a confusion set, which contains a set of similar character pairs. For instance, Yu and Li (2014) proposed to produce several candidates by retrieving the confusion set and then filter them via language models. Wang et al. (2019) used a pointer network to copy a similar character from the confusion set. These methods attempted to utilize the similarity information to confine the candidates, rather than modeling the relationship between characters explicitly.
In this paper, we propose a novel spelling check convolutional graph network (SpellGCN) that cap
<sup>1</sup>The dataset and all code for this paper is available at https://github.com/ACL2020SpellGCN/SpellGCN
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 871-881 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
tures the pronunciation/shape similarity and explore the prior dependencies between characters. Specifically, two similarity graphs are constructed for the pronunciation and shape relationship correspondingly. SpellGCN takes the graphs as input and generates for each character a vector representation after the interaction between similar characters. These representations are then constructed into a character classifier for the semantic representation extracted from another backbone module. We use BERT (Devlin et al., 2019) due to its powerful semantic capacity. Combining the graph representations with BERT, SpellGCN can leverage the similarity knowledge and generate the right corrections accordingly. Regarding the example as in Table 1, SpellGCN is able to modify the sentence correctly within the pronunciation constraint.
Experiments were conducted on three open benchmarks. The results demonstrate that Spell-GCN improves BERT evidently, outperforming all competitor models by a large margin.
- We propose a novel end-to-end trainable Spell-GCN to integrate the pronunciation and shape similarities into the semantic space. Its essential components such as the specialized graph convolution and attentive combination operations are carefully investigated.
- We investigate the performance of SpellGCN both quantitatively and qualitatively. Experimental results indicate that our method achieves the best results on three benchmark datasets.
The CSC task is a long-standing problem and has attracted much attention from the community. The research emerges in recent years (Jia et al., 2013; Xin et al., 2014; Yu and Li, 2014; Tseng et al., 2015; Fung et al., 2017; Wang et al., 2019; Hong et al., 2019), together with other topics, e.g., grammar error correction (GEC) (Rao et al., 2018; Ji et al., 2017; Chollampatt et al., 2016; Ge et al., 2018). CSC focuses on detecting and correcting character errors, while GEC also includes errors that need deletion and insertion. Previous work handles CSC using unsupervised language models (Liu et al., 2013; Yu and Li, 2014). The errors are detected/corrected by evaluating the perplexity of sentences/phrases. However, these models were
unable to condition the correction on the input sentence. To circumvent this problem, several discriminative sequence tagging methods were adopted for CSC (Wang et al., 2018). For more flexibility and better performance, several sequence-to-sequence models were also employed (Wang et al., 2019; Ji et al., 2017; Chollampatt et al., 2016; Ge et al., 2018), as well as BERT (Hong et al., 2019).
Recent attention was paid to utilizing the external knowledge of character similarity. The similarity knowledge can be gathered into a dictionary, i.e., confusion set, where similar pairs are stored. Yu and Li (2014) first used the dictionary to retrieve similar candidates for potential errors. Wang et al. (2019) incorporated a copy mechanism into a recurrent neural model. When given similar characters as input, their model uses the copy mechanism to directly copy the character to the target sentence. In a sense, these models face difficulty in modeling the relationship between similar characters as the similarity information is solely used for candidate selection. To capture the pronunciation/shape similarity and explore the prior dependencies between characters, we propose to use graph convolution network (GCN) (Kipf and Welling, 2017) to model character inter-dependence, which is combined with the pre-training of BERT (Devlin et al., 2019; Cheng et al., 2019) for the CSC task.
GCN has been applied to model the relationship on several tasks. Yan et al. (2019) equipped it into the relation extraction task where relations construct a hierarchical tree. Li et al. (2018); Cheng et al. (2018) use it to model spatial-temporal to predict traffic flow. GCN was also used to model the relationship between labels in a multi-label task (Chen et al., 2019). In this paper, it is the first time that GCN is applied successfully into the CSC task. The relationship in CSC is much different from those tasks where objects in the graph are semantically related. By contrast, the similar characters are semantically distinct in CSC. Therefore, we deeply investigate the effect of our SpellGCN and propose several essential techniques.
In this section, we elaborate on our method for CSC. Firstly, the problem formulation is presented. Then, we introduce the motivations for SpellGCN, followed by its detailed description. At last, we present its application in the CSC task.
The Chinese Spelling Check task aims to detect and correct the errors in the Chinese language. When given a text sequence  $\mathbf{X} = \{x_{1}, x_{2}, \ldots, x_{n}\}$  consisting of  $n$  characters, the model takes  $\mathbf{X}$  as input and output a target character sequence  $\mathbf{Y} = \{y_{1}, y_{2}, \ldots, y_{n}\}$ . We formulate the task as a conditional generation problem by modeling and maximizing the conditional probability  $p(\mathbf{Y}|\mathbf{X})$ .
3.2 Motivations
The framework of the proposed method is depicted in Figure 1. It consists of two components, i.e., a character representation extractor and a SpellGCN. The extractor derives a representation vector for each character. Above the extractor, SpellGCN is used to model the inter-dependence between characters. It outputs target vectors containing the information of similar characters after interactions.
As illustrated in Table 1, a vanilla language model is able to provide feasible corrections in semantic meaning but faces the difficulty in meeting the pronunciation constraint. Although the correction "月消费最" is semantically plausible, its phonics differs much from "换经费产" and "环境非常". This indicates that the similarity information between characters is necessary so that the model can learn to generate related answers. Previous methods have taken the similarity into consideration. However, they typically regarded similar characters as potential candidates, neglecting their inter-relationship in terms of pronunciation and shape. This work makes a preliminary attempt to handle this issue, trying to fuse both the symbolic space (phonological and visual similarity knowledge) and the semantic space (language semantic knowledge) into one model. To achieve this, we leverage the power of graph neural network (GNN) to infuse the similarity knowledge directly. The essential idea is to update the representations by aggregating the information between similar characters. Intuitively, a model is likely to have a sense of similar symbols when equipped with our method.
Among various GNN models, we use GCN in our implementation. Since there are up to 5K Chinese characters in the graph, the light-weight GCN is more suitable for our problem. The proposed SpellGCN is depicted as follows in detail.
3.3 Structure of SpellGCN
SpellGCN requires two similarity graphs  $\mathbf{A}^p$ ,  $\mathbf{A}^s$  for pronunciation and shape similarities correspondingly, which are derived from an open-sourced confusion set (Wu et al., 2013). For simplicity, the superscript will be omitted if unnecessary and  $\mathbf{A}$  denotes one of these two similarity graphs. Each similarity graph is a binary adjacent matrix of size  $\mathbb{R}^{N\times N}$ , constructed from  $N$  characters in the confusion set. The edge  $\mathbf{A}_{i,j}\in \{0,1\}$  between  $i$ -th character and  $j$ -th character denotes whether the  $(i,j)$  pair exists in the confusion set.
The goal of SpellGCN is to learn a map function to map the input node embedding  $\mathbf{H}^l\in \mathbb{R}^{N\times D}$  of  $l$  -th layer (where  $D$  is the dimensionality of character embedding) to a new representation  $\mathbf{H}^{l + 1}$  via convolutional operation defined by A. This map function has two main sub-components: a graph convolution operation and an attentive graph combination operation.
Graph Convolution Operation The graph convolution operation is to absorb the information from neighboring characters in the graph. In each layer, the light-weight convolution layer in GCN (Kipf and Welling, 2017) is adopted:
$$
f (\mathbf {A}, \mathbf {H} ^ {l}) = \hat {\mathbf {A}} \mathbf {H} ^ {l} \mathbf {W} _ {g} ^ {l}, \tag {1}
$$
where  $\mathbf{W}_g^l\in \mathbb{R}^{D\times D}$  is a trainable matrix and  $\hat{\mathbf{A}}\in \mathbb{R}^{N\times N}$  is the normalized version of the adjacent matrix  $\mathbf{A}$ . For the definition of  $\hat{\mathbf{A}}$ , we direct you to the original paper (Kipf and Welling, 2017). Note that we use the character embedding of BERT as the initial node features  $\mathbf{H}^0$ , and we omit the non-linearity function after convolution. Since we adopted BERT as our extractor, which has its own learned semantic space, we remove the activation function from the equation to keep the derived representation identical with original space, rather than a completely different space. During our experiments, using non-linearity activation such as ReLU is ineffective, resulting in a performance drop.
Attentive Graph Combination Operation The graph convolution operation handles the similarity of a single graph. To combine the pronunciation and shape similarity graphs, the attention mechanism (Bahdanau et al., 2015) is adopted. For each character, we represent the combination operation
Figure 1: The framework of the proposed SpellGCN. Left: The characters in the input sentence are processed by the extractor to obtain the semantic representation vectors. Right: The phonological or visual similarity knowledge of characters is learned by our SpellGCN. Two similarity graphs are used to model the pronunciation and shape similarities respectively, and they are combined via an attentive combination operation. Middle: The character embedding vectors derived from SpellGCN are used as the target character classifiers.
$$
\mathbf {C} _ {i} ^ {l} = \sum_ {k \in \{s, p \}} \alpha_ {i, k} ^ {l} f _ {k} \left(\mathbf {A} ^ {k}, \mathbf {H} ^ {l}\right) _ {i}, \tag {2}
$$
where  $\mathbf{C}^l\in \mathbb{R}^{N\times D}$  and  $f_{k}(\mathbf{A}^{k},\mathbf{H}^{l})_{i}$  is the  $i$  -th row of convolved representation of graph  $k,\alpha_{i,k}$  is a scalar for  $i$  -th character denoting the weight of graph  $k$  . The weight  $\alpha_{i,k}$  is computed by
$$
\alpha_ {i, k} = \frac {\exp \left(\mathbf {w} _ {a} f _ {k} \left(\mathbf {A} ^ {k} , \mathbf {H} ^ {l}\right) _ {i} / \beta\right)}{\sum_ {k ^ {\prime}} \exp \left(\mathbf {w} _ {a} f _ {k ^ {\prime}} \left(\mathbf {A} ^ {k ^ {\prime}} , \mathbf {H} ^ {l}\right) _ {i} / \beta\right)}, \tag {3}
$$
where  $\mathbf{w}_a\in \mathbb{R}^D$  is a learnable vector shared across the layers and  $\beta$  is a hyper-parameter which controls the smoothness of attention weights. We found  $\beta$  essential for the attention mechanism.
Accumulated Output After graph convolution and attentive combination operations, we obtain a representation  $\mathbf{C}^l$  for  $l$ -th layer. To maintain the original semantic of the extractor, all outputs of previous layers are accumulated as the output:
$$
\mathbf {H} ^ {l + 1} = \mathbf {C} ^ {l} + \sum_ {i = 0} ^ {l} \mathbf {H} ^ {i}. \tag {4}
$$
In this way, SpellGCN is able to focus on capturing the knowledge of character similarity, leaving the responsibility of semantic reasoning to the extractor. Hopefully, each layer can learn to aggregate the information for the specific hop. During the experiments, the model failed when excluding  $\mathbf{H}^0$ .
3.4 SpellGCN for Chinese Spelling Check
Here, we introduce how to apply SpellGCN to the CSC task. Motivated by recent applications of
GCN in relationship modeling (Chen et al., 2019; Yan et al., 2019), we use the final output of Spell-GCN to be classifiers of the target characters.
Similarity Graphs from Confusion Set The similarity graphs used in SpellGCN are constructed from the confusion set provided in (Wu et al., 2013). It is a pre-defined set consisting of similar characters for most of ( $\sim 95\%$ ) the Chinese characters and these characters are categorized into five categories, i.e., (1) similar shape, (2) same pronunciation and same tone, (3) same pronunciation and different tone, (4) similar pronunciation and same tone, (5) similar pronunciation and different tone. Since the pronunciation similarity is more fine-grained compared with the shape similarity category, we combine the pronunciation similarities into one graph. Consequently, we construct two graphs corresponding to pronunciation and shape similarities.
Character Representation by Extractor The representation of characters used for final classification is given by an extractor. We can use any model that is able to output representation vectors  $\mathbf{V} = \{\mathbf{v}_1, \mathbf{v}_2, \dots, \mathbf{v}_n\}$  (where  $\mathbf{v}_i \in \mathbb{R}^D$ ) for  $n$  characters  $\mathbf{X} = \{x_1, x_2, \dots, x_n\}$ . In our experiment, we adopt BERT as the backbone model. It takes  $\mathbf{X}$  as input and uses the output of the last layer as  $\mathbf{V}$ . We conduct the experiment using the base version, which has 12 layers, 12 self-attention heads with a hidden size of  $768^2$ .
2This means  $D = 768$  in our experiment.
SpellGCN as Character Classifier When given the representation vector  $\mathbf{v}_i$  of a character  $x_i$ , the model needs to predict a target character through a fully-connected layer whose weight  $\mathbf{W} \in \mathbb{R}^{M \times D}$  is configured by the output of SpellGCN ( $M$  is the size of the extractor vocabulary):
$$
p \left(\hat {y} _ {i} | \mathbf {X}\right) = \operatorname {s o f t m a x} \left(\mathbf {W v} _ {i}\right). \tag {5}
$$
Concretely, the output vectors of SpellGCN plays the role of the classifier in our task. We use the output of the last layer of SpellGCN  $\mathbf{H}^L$  (where  $L$  is the number of layers) to classify the characters in the confusion set. And since the confusion set only covers a subset of vocabulary, we use the word embedding of the extractor as the classifier for those excluded by the confusion set. In this way, denoting  $u_i \in \{1, \dots, N\}$  is the index of confusion set for the  $i$ -th character in the extractor vocabulary,  $\mathbf{W}$  is presented by:
$$
\mathbf {W} _ {i} = \left\{ \begin{array}{l l} \mathbf {H} _ {u _ {i}} ^ {L}, & \text {i f i - t h c h a r a c t e r \in c o n f u s i o n s e t} \\ \mathbf {E} _ {i}, & \text {o t h e r w i s e ,} \end{array} \right. \tag {6}
$$
where  $\mathbf{E} \in \mathbb{R}^{M \times D}$  is the embedding matrix of extractor. In brief, we use the embedding from Spell-GCN if the character is in the confusion set. Otherwise, the embedding vectors are used as in BERT. Instead of modeling a large compact graph containing all characters in the extractor vocabulary, we chose this implementation for computational efficiency, since there are around 5K characters in the confusion set and more than 20K characters in the extractor vocabulary.
Overall, the objective is to maximize the log likelihood of target characters:
$$
\mathcal {L} = \sum_ {\mathbf {X}, \mathbf {Y}} \sum_ {i} \log p \left(\hat {y} _ {i} = y _ {i} \mid \mathbf {X}\right). \tag {7}
$$
3.5 Prediction Inference
The CSC task consists of two sub-tasks in evaluation, i.e., detection and correction. Some previous work (Yu and Li, 2014; Liu et al., 2013) used two models for these sub-tasks separately. In this work, we simply use the character with the highest probability  $\arg \max_{\hat{y}_i} p(\hat{y}_i | \mathbf{X})$  as the prediction for the correction task. And the detection is achieved by checking whether the prediction matches the target character  $y_i$ .
In this section, we describe our experiment in detail. We first present the training data and test data,
Training Data	# Line	Avg. Length	# Errors
(Wang et al., 2018)	271,329	44.4	382,704
SIGHAN 2013	350	49.2	350
SIGHAN 2014	6,526	49.7	10,087
SIGHAN 2015	3,174	30.0	4,237
Total	281,379	44.4	397,378
Test Data	# Line	Avg. Length	# Errors
SIGHAN 2013	1000(1000)	74.1	1,227
SIGHAN 2014	1062(526)	50.1	782
SIGHAN 2015	1100(550)	30.5	715
Graph	# Character	# Edges
Pronunciation Similarity Graph	4753	112,687
Shape Similarity Graph	4738	115,561
Table 2: Statistics information of the used data resources. The number in the bracket in #Line column denotes the number of sentences with errors.
as well as the evaluation metrics. Then we introduce our main results for SpellGCN. After that, the ablation studies are made to analyze the effect of the proposed components, followed by a case study. Finally, quantitative results are provided.
Training Data The training data is composed of three training datasets (Wu et al., 2013; Yu et al., 2014; Tseng et al., 2015), which has 10K data samples in total. Following (Wang et al., 2019), we also include additional 271K samples as the training data, which are generated by an automatic method (Wang et al., 2018)  $^{3}$ .
Test Data To evaluate the performance of the proposed method, we used three test datasets from the SIGHAN 2013, SIGHAN 2014, SIGHAN 2015 benchmarks (Wu et al., 2013; Yu et al., 2014; Tseng et al., 2015) as in (Wang et al., 2019). We also follow the same data pre-processing procedure, i.e., the characters in these datasets are converted to simplified Chinese using OpenCC  $^{4}$ . The statistic of the data is listed in Table 2.
Baseline Models We compare our method with five typical baselines.
- LMC (Xie et al., 2015): This method utilizes the confusion set to replace the characters and then evaluates the modified sentence via a Ngram Language Model.
- SL (Wang et al., 2018): This method proposes a pipeline where a Sequence Labeling model
<sup>3</sup>https://github.com/wdimmy/Automatic-CorpusGeneration
4https://github.com/BYVoid/
Character-level	Sentence-level
Detection-level	Correction-level	Detection-level	Correction-level
SIGHAN 2013	D-P	D-R	D-F	C-P	C-R	C-F	D-P	D-R	D-F	C-P	C-R	C-F
LMC (Xie et al., 2015)	79.8	50.0	61.5	77.6	22.7	35.1	(-)	(-)	(-)	(-)	(-)	(-)
SL (Wang et al., 2018)	54.0	69.3	60.7	(-)	(-)	52.1	(-)	(-)	(-)	(-)	(-)	(-)
PN (Wang et al., 2019)	56.8	91.4	70.1	79.7	59.4	68.1	(-)	(-)	(-)	(-)	(-)	(-)
FASpell (Hong et al., 2019)	(-)	(-)	(-)	(-)	(-)	(-)	76.2	63.2	69.1	73.1	60.5	66.2
BERT	80.6	88.4	84.3	98.1	87.2	92.3	79.0	72.8	75.8	77.7	71.6	74.6
SpellGCN	82.6	88.9	85.7	98.4	88.4	93.1	80.1	74.4	77.2	78.3	72.7	75.4
SIGHAN 2014	D-P	D-R	D-F	C-P	C-R	C-F	D-P	D-R	D-F	C-P	C-R	C-F
LMC (Xie et al., 2015)	56.4	34.8	43.0	71.1	50.2	58.8	(-)	(-)	(-)	(-)	(-)	(-)
SL (Wang et al., 2018)	51.9	66.2	58.2	(-)	(-)	56.1	(-)	(-)	(-)	(-)	(-)	(-)
PN (Wang et al., 2019)	63.2	82.5	71.6	79.3	68.9	73.7	(-)	(-)	(-)	(-)	(-)	(-)
FASpell (Hong et al., 2019)	(-)	(-)	(-)	(-)	(-)	(-)	61.0	53.5	57.0	59.4	52.0	55.4
BERT	82.9	77.6	80.2	96.8	75.2	84.6	65.6	68.1	66.8	63.1	65.5	64.3
SpellGCN	83.6	78.6	81.0	97.2	76.4	85.5	65.1	69.5	67.2	63.1	67.2	65.3
SIGHAN 2015	D-P	D-R	D-F	C-P	C-R	C-F	D-P	D-R	D-F	C-P	C-R	C-F
LMC (Xie et al., 2015)	83.8	26.2	40.0	71.1	50.2	58.8	(-)	(-)	(-)	(-)	(-)	(-)
SL (Wang et al., 2018)	56.6	69.4	62.3	(-)	(-)	57.1	(-)	(-)	(-)	(-)	(-)	(-)
PN (Wang et al., 2019)	66.8	73.1	69.8	71.5	59.5	69.9	(-)	(-)	(-)	(-)	(-)	(-)
FASpell (Hong et al., 2019)	(-)	(-)	(-)	(-)	(-)	(-)	67.6	60.0	63.5	66.6	59.1	62.6
BERT	87.5	85.7	86.6	95.2	81.5	87.8	73.7	78.2	75.9	70.9	75.2	73.0
SpellGCN	88.9	87.7	88.3	95.7	83.9	89.4	74.8	80.7	77.7	72.1	77.7	75.9
Table 3: The performance of our method and baseline models (\%). D, C denote the detection, correction, respectively. P, R, F denote the precision, recall and F1 score, respectively. The results of BERT are from our own implementation. Best results are in bold. We performed additional fine-tuning on SIGHAN13 for 6 epochs as the data distribution in SIGHAN13 differs from other datasets, e.g. “的”, “得” and “地” are rarely distinguished.
is adopted for detection. The incorrect characters are marked as 1 (0 otherwise).