# A Neural N-Gram-Based Classifier for Chinese Clinical Named Entity Recognition
## 4 Experiments
### Experiment and Results  
To study the effectiveness of the proposed model, two experiments are conducted. The first one evaluates the model against other competitive algorithms. In the second experiment, we test our model on different lengths of n-gram character embeddings. The experiments are carried out on the Windows system with an Intel(R) Core(TM) i7-8750H CPU, 8 GB RAM, and NVIDIA GeForce GTX 1050 Ti GPU. The neural network model is composed of one character embedding lookup table, a self-attention layer, a BiLSTM layer and a final CRF layer. The hyper-parameter setting of the model is shown in Table 3.  
To verify the proposed approach, we conduct the first experiment and compare our results with the following methods:  
* LSTM-CRF: A LSTM neural network model with a CRF layer.
* BiLSTM-CRF: A bidirectional LSTM model with a CRF layer [32].
* RD-CNN-CRF: A residual dilated Convolutional Neural Network with CRF where dictionary features are utilized according to the drug information in Shanghai Shuguang Hospital and some medical literature [33].
* ID-CNN-CRF: A Convolutional Neural Network-based model with iterated dilated convolutions and a domain-specific lexicon for word embeddings matching [34; 35].
* BERT-BiLSTM-CRF: A pre-trained language model BERT to enhance the semantic representation, a BiLSTM network and a CRF layer [36].  
The comparison results are demonstrated in Table 4. Our model obtains the best F-score among all competitors. We push the F-score to 89.33% and outperform the second-best system (RD-CNN-CRF) by 0.82%. In general, LSTM based approaches have better results in the Recall metric while CNN based methods perform better in the Precision score. RD-CNN-CRF achieves the best Precision score (88.64%) and our approach is the second-best (88.53%). The best Recall score is reported by BERT-BiLSTM-CRF (90.48%) and our system is ranked second (90.13%).  
\begin{table}
\begin{tabular}{c c c c} \hline \hline
**Models** & **P** & **R** & **F** \\ \hline LSTM-CRF & 83.59 & 85.28 & 84.42 \\ \hline BiLSTM-CRF & 88.22 & 88.53 & 88.37 \\ \hline RD-CNN-CRF & **88.64** & 88.38 & 88.51 \\ \hline ID-CNN-CRF & 88.30 & 87.21 & 87.75 \\ \hline BERT-BiLSTM-CRF & 86.50 & **90.48** & 88.45 \\ \hline Our model & 88.53 & 90.13 & **89.33** \\ \hline \hline \end{tabular}
\end{table}
Table 4: Performance comparison results of each model.  
\begin{table}
\begin{tabular}{c c} \hline \hline
**Parameter** & **Value** \\ \hline n-gram & 1, 2, 3 \\ character embedding size & 100 \\ LSTM hidden units & 100 \\ batch size & 16 \\ dropout rate & 0.5 \\ learning rate & 0.001 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Hyper-parameter settings of the proposed approach.  
We perform the second experiment to further investigate the effect of different lengths of n-gram embeddings. Table 5 demonstrates the experimental results of 1-g, 2-g, 3-g and our approach which combines all of them. Our model achieves the highest F-score of 89.33% and Precision of 88.53%. In terms of Recall, the 2-g method is the best (90.47%) and our model achieves the second-highest score (90.13%).  
In addition to the overall performance evaluation discussed above, we also show the detailed results for all five entity categories in Table 6. Our model achieves the highest F-scores in 4 entity categories except for the \"Body\" category. The most challenging entity types are Disease and Treatment where the F-scores for all models are less than 80%.  
Though our approach is generally applicable, there are several limitations that have to be addressed. First, since our model uses a Bi-LSTM layer that is not able to fully utilize GPU for parallel processing, it is important to handle this issue to ensure both high performance and high computational efficiency. Second, we incorporate an embeddings layer to learn the distributed representation without applying any pre-trained embeddings. We expect that pre-trained embeddings learned from large Chinese medical corpora can help in the Chinese CNER task.