# MLSL-Spell: Chinese Spelling Check Based on Multi-Label Annotation
## 3 Approach
### Spelling Detection of MLSL-Spell  
MLSL-Spell's spelling detection module is shown in Figure 3. Suppose the sentence to be corrected is formulated as  
\[X=\{x_{1},x_{2},\dots,x_{n}\}\]  
and the label sequence output after spelling detection as \(Y=\{y_{1},y_{2},\dots,y_{n}\}\), where \(x_{i}\) represents a Chinese character, \(y_{i}\) denotes \"T\", \"P\", \"S\", \"B\" or \"O\", and \(i\) refers to the position of the Chinese character in the sentence. We fuse the context vector \(e_{i}\) and the Pinyin vector \(p_{i}\) (each pair of \(e_{i}\) and \(p_{i}\) corresponds to one character in the sentence) into the vector \(e_{i}^{f}\), and input it to the bidirectional GRU model. The calculation formula of the fusion vector is as follows:  
\[e_{i}^{f}=[e_{i};p_{i}] \tag{1}\]  
The hidden layer vector \(\overline{h_{i}}\) of a certain direction at the current time step depends on the fusion vector \(e_{i}^{f}\) at the current time step and the hidden layer vector \(\overline{h_{i\pm 1}}\) at the previous time step. We concatenate the hidden layer vectors in each direction as the output vector of the BiGRU model. The calculation formulas are shown below.  
\[\overrightarrow{h_{i-1}}=BiGRU\left(\overrightarrow{h_{i-1}},e_{i}^{f}\right) \tag{2}\]  
\[\overleftarrow{h_{i+1}}=BiGRU\left(\overleftarrow{h_{i+1}},e_{i}^{f}\right) \tag{3}\]  
\[h_{i}^{f}=\left[\overrightarrow{h_{i}};\overleftarrow{h_{i}}\right] \tag{4}\]  
\[y_{pred}=argmax\left(s(X,\vec{y})_{\vec{y}\in y_{x}}\right) \tag{5}\]  
In the bidirectional GRU model, the hidden layer vectors fully learn the contextual information under the current sentence meaning. Next, MLSL-Spell sends the trained hidden layer vectors to CRF model to label the text in sequence. We use a linear function to normalize the hidden layer vector at each time step and obtain the respective scores of the five labels. With these scores attained, we can calculate the emission matrix of CRE  
Figure 3: Architecture of MLSL-Spell spelling detection module.  
\(E\in R^{n\times k}\), where \(n\) is the length of the input sentence, and \(k\) represents the number of the labels. In this paper, we define 5 subtypes of the sequence label: \"T\", \"P\", \"S\", \"B\" and \"O\", representing the correct character, the misspelled character of similar pronunciation, misspelled character of similar shape, misspelled character of both similar pronunciation and similar shape, and the misspelled character caused by other reasons, respectively. Therefore, \(k\) equals 5. The calculation formula of CRF model for constructing the emission matrix is as follows:  
\[P_{i,y_{i}}(y_{i}=j,0\leq j\leq 4|x_{i})=Softmax(Wh_{i}+b)[j] \tag{6}\]  
CRF model calculates the scores of sequence labels through the emission matrix and the label transfer matrix. The calculation formulas are as follows:  
\[s(X,y)=\sum_{i=0}^{n}T_{y_{i},y_{i+1}}+\sum_{i=1}^{n}P_{i,y_{i}} \tag{7}\]  
\[P(y|X)=\frac{e^{s(X,y)}}{\sum_{\beta\in y_{all}}e^{s(X,\beta)}} \tag{8}\]  
We adopt the negative logarithmic maximum likelihood function as the loss function of the model, as shown below:  
\[loss=-\log p(y|X) \tag{9}\]  
\(y\) represents the real label corresponding to \(X\). In the end, we adopt the label sequence with the highest probability as the spelling detection result of MLSL-Spell, which is formulated as follows:  
\[y_{pred}=argmax\Big{(}s(X,\vec{y})_{\hat{y}\in y_{x}}\Big{)} \tag{10}\]  
In addition, we use the adamW optimizer to optimize it, which has the L2 regularization and higher computational efficiency than the adam optimizer.