SlowGuess/ACL_2020_OCR huggingface dataset:
- PN (Wang et al., 2019): This method incorporates a Pointer Network to consider the extra candidates from the confusion set.
- FASpell (Hong et al., 2019): This model utilizes a specialized candidate selection method based on the similarity metric. This metric is measured using some empirical methods, e.g., edit distance, rather than a pre-defined confusion set.
- BERT (Devlin et al., 2019): The word embedding is used as the softmax layer on the top of BERT for the CSC task. We trained this model using the same setting, i.e., the comparable model w/o SpellGCN.
Evaluation Metrics The precision, recall and F1 scores are reported as the evaluation metrics, which are commonly used in the CSC tasks. These metrics are provided for the detection and correction sub-tasks. Besides the evaluation on the character level, we also report the sentence-level metrics on the detection and correction sub-tasks, which is more appealing for real-world applications. On the sentence level, we consider a sentence to be
correctly annotated only if all errors in the sentence are corrected as in (Hong et al., 2019)  $^{5}$ . On the character level, we calculate the metrics using the evaluation script from (Wang et al., 2019)  $^{6}$ . We also evaluated BERT and SpellGCN by the official evaluation metrics tools, which gives False Positive Rate (FTR), Accuracy and Precision/Recall/F1.
4.2 Hyper-parameters
Our code is based on the repository of BERT  $^{8}$ . We fine-tune the models using AdamW (Loshchilov and Hutter, 2018) optimizer for 6 epochs with a batch size of 32 and a learning rate of 5e-5. The number of the layer in SpellGCN is 2, and the attentive combination operation with factor 3 is used. All experiments were conducted for 4 runs and the averaged metric is reported. The code and trained models will be released publicly after review (currently, the code is attached in the supplementary files).
$^{5}$ https://github.com/iqiyi/FASPell
$^{6}$ https://github.com/wdimmy/Confusionset-guided-Pointer-Networks-for-Chinese-Spelling-Check
<sup>7</sup>http://nlp.ee.ncu.edu.tw/resource/csc.html
<sup>8</sup>https://github.com/google-research/bert
SIGHAN 2014	FPR	D-A	C-A	D-P	D-R	D-F	C-P	C-R	C-F
BERT	15.3	76.8	75.7	81.9	68.9	74.9	81.4	66.7	73.3
SpellGCN	14.1	77.7	76.9	83.1	69.5	75.7	82.8	67.8	74.5
SIGHAN 2015	FPR	D-A	C-A	D-P	D-R	D-F	C-P	C-R	C-F
BERT	13.6	83.0	81.5	85.9	78.9	82.3	85.5	75.8	80.5
SpellGCN	13.2	83.7	82.2	85.9	80.6	83.1	85.4	77.6	81.3
Table 4: The performance of BERT and SpellGCN evaluated by official tools on SIGHAN 2014 and SIGHAN 2015. FPR denotes the false positive rate and A denotes the accuracy. D-A and C-A denote detection accuracy and correction accuracy.
Table 3 shows the performance of the proposed method on the three CSC datasets, compared with five typical CSC systems. When using SpellGCN, the model achieves better results in all test sets against vanilla BERT, which verifies its effectiveness. The improvement is considerable with such a large amount of training data (cf. the comparison in Figure 2). This indicates the similarity knowledge is essential for CSC and it can hardly be learned by simply increasing the data amount. In terms of sentence-level F1score metric in the correction subtask, i.e., C-F score in the last column, the improvements against previous best results (FASpell) are  $9.2\%$ ,  $9.7\%$  and  $13.3\%$  respectively. Nevertheless, it should be noted that FASpell was trained on different training data while this paper follows the setting mentioned in the PN paper (Wang et al., 2019). Ideally, our method is compatible with FASpell and better results can be achieved when FASpell is employed.
FASpell used their own metrics, which are different from the sentence-level false positive and false negate to compute their metrics for fair comparison. We further add the official evaluation results of BERT and SpellGCN in Table 4. Actually, SpellGCN consistently improves the performance when evaluated by the PGNet/FASpell scripts and the official evaluation toolkit. We will add the FPR results in our revision. The FPR scores are  $14.1\%$  (SpellGCN) v.s.  $15.3\%$  (BERT) on SIGHAN 14, and  $13.2\%$  (SpellGCN) v.s.  $13.6\%$  (BERT) on SIGHAN 15. FPR on SIGHAN 13 is statistically meaningless since almost all the tested sentences have the spelling errors.
Figure 2: The test curves for sentence-level correction metrics with and without SpellGCN w.r.t. the number of training epochs on SIGHAN 2015.
In this subsection, we analyze the effect of several components, including the number of layers and the attention mechanism. The ablation experiments were performed using 10K training data.
Effect of the Number of Layers Generally, the performance of a GCN varies with the number of layers. We investigate how the number of SpellGCN layers influence the performance in CSC. In this comparison, the number of layers changes from 1 to 4, and the results are illustrated in Figure 3. For clarity, we report the character-level C-F on the three test datasets. The results indicate that SpellGCN is able to make use of multiple layers. With multiple layers, SpellGCN can aggregate the information in more hops and therefore, achieve better performance. However, the F1score drops when the number of layers is larger than 3. This is reasonable due to the over-smooth problem noted in (Yan et al., 2019). When the number of GCN layers increases, the representations of neighboring characters in the similarity graph will get more and more similar since they all are calculated via those of their neighbors in the similarity graph.
Figure 3: The character-level C-F results  $(\%)$  w.r.t. the depth of SpellGCN. The results were obtained with 10K training samples.
combination method	C-F
baseline (w/o SpellGCN)	67.0
sum pooling	66.3
mean pooling	67.5
attentive combination (β=1)	67.8
attentive combination (β=3)	68.2
attentive combination (β=5)	68.0
attentive combination (β=10)	67.7
Effect of Attention Mechanism We investigate how to better combine the graphs in the SpellGCN layer. Here, we compare the attention mechanism against sum-pooling and mean-pooling, with different hyper-parameter  $\beta$  mentioned in Section 3.3. The experiments are conducted based on the 2-layer SpellGCN on SIGHAN 2013 test set. The results presented in Table 5 show that the sum pooling fails in the CSC task. We suggest that the sum pooling is inconsistent with the normalization of GCN and fails to combine the information from different channels (i.e., graphs). The mean pooling is feasible but is surpassed by the attention mechanism. This indicates that the adaptive combination for each character node is beneficial. We incorporate a hyper-parameter  $\beta$  into the attention operation since the dot products may grow large in magnitude, pushing the softmax function into regions where it has extremely small gradients. With these results, we chose the attention mechanism with a  $\beta$  of 3 in SpellGCN.
Table 5: The ablation results for graph combination method  $(\%)$ . The averaged character-level C-F scores of 4 runs on the SIGHAN 2013 are reported. The models were trained with 10K training samples. Mean pooling denotes that the output representation  $C^l$  of each layer is the average of  $f_{k\in \{P,S\}}(\mathbf{A}_k,\mathbf{H}^l)$ , while sum pooling summarizes  $f_{k\in \{P,S\}}(\mathbf{A}_k,\mathbf{H}^l)$ .
Pronunciation: fǎng→fán, wàng→wàng
...走路真的麻坊，我也没有喝的东西，在家汪了...
...走路真的麻木，我也没有喝的东西，在家呆了...
...走路真的麻烦，我也没有喝的东西，在家忘了...
Pronunciation: yīn→yǐng
...因为妈妈或爸爸在看录音机...帮小孩子解决问题...
...因为妈妈或爸爸在看录音机...帮小孩子解决问题...
...因为妈妈或爸爸在看录影机...帮小孩子解决问题...
Shape: 向→尚
...不过在许多传统文化的国家，女人向未得到平等...
...不过在许多传统文化的国家，女人从未得到平等...
...不过在许多传统文化的国家，女人尚未得到平等...
Table 6: Several prediction results on the test set. The first line in the block is the input sentence. The second line is corrected by BERT without SpellGCN. And the last line is the result from SpellGCN. We highlight the incorrect/correct characters by orange/blue color.
We show several correction results to demonstrate the properties of SpellGCN. In addition to the sample illustrated in Table 1, several prediction results are given in Table 6. From these cases, we can tell that our SpellGCN is capable of revising the incorrect characters into correct ones with the pronunciation and shape constraint. For instance, in the first case, "麻坊(fāng)" is detected as errors and modified into "麻烦(fán)". Without pronunciation similarity constraint, "麻木(mù)" becomes the most probable answer. And surprisingly, in the second case, our SpellGCN successfully modifies the character reasonable in the context. The meaning of input sentence "看录音机" is "watch the audio recorder", and our method corrects it into "看录影机" which means "watch the video recorder". We suggest that SpellGCN injects a prior similarity between "音" and "影" in the representation space so that the model derives a higher posterior probability of "影". In the last case, we show a correction result under the shape constraint. In the confusion set, "向" is similar to "尚" and therefore, using SpellGCN is able to retrieve the correct result.
4.6 Character Embedding Visualization
Previous experiments have explored the performance of SpellGCN quantitatively in detail. To qualitatively study whether SpellGCN learns meaningful representations, we dive into the target embedding space  $\mathbf{W}$  derived from SpellGCN.
In Figure 4, the embedding of characters with phonics "chang" and "si" is presented using t-SNE (Maaten and Hinton, 2008). The embedding
(a) w/ SpellGCN
(b) w/o SpellGCN
Figure 4: The scatter of similar characters of "长" and "祀" in terms of pronunciation by t-SNE.
(a) w/ SpellGCN
Figure 5: The scatter of similar characters of "长" and "祀" in terms of shape by t-SNE.
learned by BERT captures the semantic similarity but fails to model the similarity in terms of pronunciation for the CSC task. This is reasonable as this similarity knowledge is absent in the modeling. In contrast, our SpellGCN successfully infuses this prior knowledge into the embedding and the resulting embedding exhibits cluster patterns. The embedding of characters with these two different pronunciations forms two clusters, corresponding to "cháng" and "si" respectively. Due to this property, the model tends to recognize similar characters and hence is able to retrieve the answers under pronunciation constraint. Figure 5 shows the same situation for the shape similarity, where two sets of characters with the shape similar to "长" and "祀" are scattered. This verifies the ability of SpellGCN in modeling shape similarity.
We proposed SpellGCN for CSC to incorporate both phonological and visual similarities into language models. The empirical comparison and the results of analytical experiments verify its effectiveness. Beyond CSC, SpellGCN can be generalized to other situations where specific prior knowledge is available, and to other languages by leveraging specific similarity graphs analogously. Our method can also be adapted to grammar error correction, which needs insertion and deletion, by utilizing more flexible extractors such as Levenshtein Trans
former (Gu et al., 2019). We leave this direction to future work.
Zhao-Min Chen, Xiu-Shen Wei, Peng Wang, and Yanwen Guo. 2019. Multi-label image recognition with graph convolutional networks. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pages 5177-5186.
Xingyi Cheng, Weidi Xu, Kunlong Chen, Wei Wang, Bin Bi, Ming Yan, Chen Wu, Luo Si, Wei Chu, and Taifeng Wang. 2019. Symmetric regularization based BERT for pair-wise semantic reasoning. CoRR, abs/1909.03405.
Xingyi Cheng, Ruiqing Zhang, Jie Zhou, and Wei Xu. 2018. Deeptransport: Learning spatial-temporal dependency for traffic condition forecasting. In 2018 International Joint Conference on Neural Networks, IJCNN 2018, Rio de Janeiro, Brazil, July 8-13, 2018, pages 1-8. IEEE.
Shamil Chollampatt, Kaveh Taghipour, and Hwee Tou Ng. 2016. Neural network translation models for grammatical error correction. In Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence, IJCAI 2016, New York, NY, USA, 9-15 July 2016, pages 2768-2774.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 4171-4186.
Gabriel Pui Cheong Fung, Maxime Debosschere, Dingmin Wang, Bo Li, Jia Zhu, and Kam-Fai Wong. 2017. NLPTEA 2017 shared task - chinese spelling check. In Proceedings of the 4th Workshop on Natural Language Processing Techniques for Educational Applications, NLP-TEA@IJCNLP 2017, Taipei, Taiwan, December 1, 2017, pages 29-34.
Tao Ge, Furu Wei, and Ming Zhou. 2018. Fluency boost learning and inference for neural grammatical error correction. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Long Papers, pages 1055-1065.
Jiatao Gu, Changhan Wang, and Jake Zhao. 2019. Levenshtein transformer. CoRR, abs/1905.11006.
Yuzhong Hong, Xianguo Yu, Neng He, Nan Liu, and Junhui Liu. 2019. Faspell: A fast, adaptable, simple, powerful chinese spell checker based on dae-decoder paradigm. In Proceedings of the 5th Workshop on Noisy User-generated Text (W-NUT 2019), pages 160-169.
Jianshu Ji, Qinlong Wang, Kristina Toutanova, Yongen Gong, Steven Truong, and Jianfeng Gao. 2017. A nested attention neural hybrid model for grammatical error correction. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30 - August 4, Volume 1: Long Papers, pages 753-762.
Zhongye Jia, Peilu Wang, and Hai Zhao. 2013. Graph model for chinese spell checking. In Proceedings of the Seventh SIGHAN Workshop on Chinese Language Processing, SIGHAN@IJCNLP 2013, Nagoya, Japan, October 14-18, 2013, pages 88-92.
Thomas N. Kipf and Max Welling. 2017. Semi-supervised classification with graph convolutional networks. In 5th International Conference on Learning Representations, ICLR 2017, Toulouse, France, April 24-26, 2017, Conference Track Proceedings.
Jing Li, Hao Peng, Lin Liu, Guixi Xiong, Bowen Du, Hongyuan Ma, Lihong Wang, and Md. Zakirul Alam Bhuiyan. 2018. Graph cnns for urban traffic passenger flows prediction. In 2018 IEEE SmartWorld, Ubiquitous Intelligence & Computing, Advanced & Trusted Computing, Scalable Computing & Communications, Cloud & Big Data Computing, Internet of People and Smart City Innovation, SmartWorld/SCALCOM/UIC/ATC/CBDCom/IOP/SCI 2018, Guangzhou, China, October 8-12, 2018, pages 29-36. IEEE.
Chao-Lin Liu, Min-Hua Lai, Yi-Hsuan Chuang, and Chia-Ying Lee. 2010. Visually and phonologically similar characters in incorrect simplified chinese words. In COLING 2010, 23rd International Conference on Computational Linguistics, Posters Volume, 23-27 August 2010, Beijing, China, pages 739-747.
Xiaodong Liu, Kevin Cheng, Yanyan Luo, Kevin Duh, and Yuji Matsumoto. 2013. A hybrid chinese spelling correction using language model and statistical machine translation with reranking. In Proceedings of the Seventh SIGHAN Workshop on Chinese Language Processing, SIGHAN@IJCNLP 2013, Nagoya, Japan, October 14-18, 2013, pages 54-58.
Ilya Loshchilov and Frank Hutter. 2018. Decoupled weight decay regularization.
Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-sne. Journal of machine learning research, 9(Nov):2579-2605.
Gaoqi Rao, Qi Gong, Baolin Zhang, and Endong Xun. 2018. Overview of NLPTEA-2018 share task chinesegrammatical error diagnosis. In Proceedings of the 5th Workshop on Natural Language Processing Techniques for Educational Applications, NLP-TEA@ACL 2018, Melbourne, Australia, July 19, 2018, pages 42-51.
Yuen-Hsien Tseng, Lung-Hao Lee, Li-Ping Chang, and Hsin-Hsi Chen. 2015. Introduction to SIGHAN 2015 bake-off for chinese spelling check. In Proceedings of the Eighth SIGHAN Workshop on Chinese Language Processing, SIGHAN@IJCNLP 2015, Beijing, China, July 30-31, 2015, pages 32-37.
Dingmin Wang, Yan Song, Jing Li, Jialong Han, and Haisong Zhang. 2018. A hybrid approach to automatic corpus generation for chinese spelling check. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 - November 4, 2018, pages 2517-2527.
Dingmin Wang, Yi Tay, and Li Zhong. 2019. Confusionset-guided pointer networks for chinese spelling check. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 5780-5785.
Shih-Hung Wu, Chao-Lin Liu, and Lung-Hao Lee. 2013. Chinese spelling check evaluation at SIGHAN bake-off 2013. In Proceedings of the Seventh SIGHAN Workshop on Chinese Language Processing, SIGHAN@IJCNLP 2013, Nagoya, Japan, October 14-18, 2013, pages 35-42.
Weijian Xie, Peijie Huang, Xinrui Zhang, Kaiduo Hong, Qiang Huang, Bingzhou Chen, and Lei Huang. 2015. Chinese spelling check system based on n-gram model. In Proceedings of the Eighth SIGHAN Workshop on Chinese Language Processing, SIGHAN@IJCNLP 2015, Beijing, China, July 30-31, 2015, pages 128-136.
Yang Xin, Hai Zhao, Yuzhu Wang, and Zhongye Jia. 2014. An improved graph model for Chinese spell checking. In Proceedings of The Third CIPS-SIGHAN Joint Conference on Chinese Language Processing, Wuhan, China, October 20-21, 2014, pages 157-166.
Haoran Yan, Xiaolong Jin, Xiangbin Meng, Jiafeng Guo, and Xueqi Cheng. 2019. Event detection with multi-order graph convolution and aggregated attention. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5770-5774.
Junjie Yu and Zhenghua Li. 2014. Chinese spelling error detection and correction based on language model, pronunciation, and shape. In Proceedings of
The Third CIPS-SIGHAN Joint Conference on Chinese Language Processing, Wuhan, China, October 20-21, 2014, pages 220-223.
Liang-Chih Yu, Lung-Hao Lee, Yuen-Hsien Tseng, and Hsin-Hsi Chen. 2014. Overview of SIGHAN 2014 bake-off for chinese spelling check. In Proceedings of The Third CIPS-SIGHAN Joint Conference on Chinese Language Processing, Wuhan, China, October 20-21, 2014, pages 126-132.