# A Neural N-Gram-Based Classifier for Chinese Clinical Named Entity Recognition
## 3 The Proposed Approach
### Neural Entity Recognition Model  
In this section, we discuss the proposed approach to deal with Chinese CNER in the EMRs. The neural model adopted in this research mainly relies on Attention, BiLSTM and CRF layers to obtain a more semantic representation of Chinese characters. Attention Layer: The attention method has been widely used in many tasks, especially NLP applications to capture the context information and dependency between tokens for the given sentence [24; 25]. The mechanism computes attention weights between every two tokens and uses a summation operation to obtain the representation [26]. The calculation of attention on our n-gram character embeddings is described as follows. Given an input sequence \(E_{n}=(\text{e}_{1},\text{e}_{2}\ldots,\text{e}_{t})\) where \(\text{e}_{i}\in R^{1\times dn}\) is the n-gram character embeddings (\(n\) from 1 to 3) and \(d_{n}\) is the size of the embeddings, \(E_{n}\) is converted to query \(Q_{n}\), key \(K_{n}\) and value \(V_{n}\) through linear transformations by the following expression:  
\[Q_{n},K_{n},V_{n}=E_{n}W_{q},E_{n}W_{k},E_{n}W_{v} \tag{1}\]  
where \(W_{q}\), \(W_{k}\), \(W_{v}\) are learnable parameters, and \(Q_{n}\in R^{t\times dn}\), \(K_{n}\in R^{t\times dn}\), \(V_{n}\in R^{t\times dn}\). The attention score is, then, calculated as follows:  
\[A_{n}(Q_{n},\ K_{n},\ V_{n})=\text{softmax}(\frac{Q_{n}{K_{n}}^{T}}{\sqrt{d_{ n}}})\ V_{n} \tag{2}\]  
In this paper, we use a special kind of attention, self-attention, to learn the feature of one unit in a sentence by attending to all units within the same sentence. In the self-attention mechanism, the query \(Q_{n}\), key \(K_{n}\) and value \(V_{n}\) are all the same. BiLSTM Layer: After the above self-attention calculation, we concatenate \(A_{1}\), \(A_{2}\) and \(A_{3}\) to obtain the final embedding matrix \(A\in R^{t\times(d1+d2+d3)}\) as shown in the Concatenation Layer of Figure 1. We then pass \(A\) into a BiLSTM layer. The LSTM is proposed to capture long-term dependencies by introducing gated memory units to address the gradient problems and control the information flow [27]. At each timestamp \(t\) for the given input \(A=(a_{1},\,a_{2}\ldots,a_{t})\), the LSTM updates its hidden state \(h_{t}\) based on the current input \(a_{t}\) and the previous hidden state \(h_{t-1}\) by computing the following equations:  
\[i_{t}=\textit{sigmoid}(W_{i}[a_{t},h_{t-1}]) \tag{3}\]  
\[f_{t}=\textit{sigmoid}\left(W_{f}[a_{t},h_{t-1}]\right) \tag{4}\]  
\[o_{t}=\textit{sigmoid}(W_{o}[a_{t},\ h_{t-1}]) \tag{5}\]  
\[c_{t}=f_{t}c_{t-1}+i_{t}tanh\left(W_{g}[a_{t},h_{t-1}]\right) \tag{6}\]  
\[h_{t}=o_{t}tanh(c_{t}) \tag{7}\]  
where \(i_{t},f_{t}\), \(o_{t}\) and \(c_{t}\) are input gate, forget gate, output gate and cell vector. \(W_{i}\), \(W_{f}\), \(W_{o}\) and \(W_{g}\) are the corresponding weight matrices. Since the output of each time step in the LSTM only considers the previous state, BiLSTM is carried out to use both forward and backward information. The output is created by concatenating the two hidden vectors from both directions as \(h_{t}=[h_{t}^{\prime};h_{t}^{\prime}]\). CRF Layer: In the CNER task, there exist several constraints and dependencies in the BIO tagging scheme. For instance, the I tag must follow the B tag. It is, therefore, important to take the above factors into consideration and we adopt CRF to predict a label sequence by learning the correlations between the current label and its neighbors [28]. Given the input sequence obtained from the output of the BiLSTM Layer \(h=(h_{1},h_{2}\ldots,h_{t})\), we use \(y=(y_{1},y_{2}\ldots,y_{t})\) to represent a sequence of labels for \(h\). The CRF model definesthe conditional probability distribution over all label sequences \(y\) given \(h\) and uses the following equation [29]:  
\[p(y|h;W,b)\propto exp(\sum_{i=1}^{t}W_{y_{i-1},y_{i}}^{T}h_{i}+b_{y_{i-1},y_{i}}) \tag{8}\]  
where \(W\) denotes the weight and \(b\) the bias term corresponding to the neighboring pair \((y_{i-1},y_{i})\).