## 3 Results
### Results  
We trained the model with different ransom seeds 10 times, and the mean and standard deviations of metrics are reported. Table 3 shows the performance of our model, and the performances from other models which experimented on the same dataset. Our solution in single-model mode achieved an exact F1 score of \(87.53\pm 0.11\%\), exact precision of \(87.75\pm 0.18\%\), and exact recall of \(87.32\pm 0.26\%\). Before our work, the best performing single model was the \"ELMo (clinical) + BiLSTM-CRF (single) model\" [24], which also used clinical pretrained ELMo word embeddings. We obtained an improvement of \(0.69\%\) in mean F1, an improvement of \(0.31\%\) in mean precision, and an improvement of \(1.06\%\) in mean recall compared with the best performing model. Additionally, we build an ensemble model based on 10 single models trained on different random seeds [53]. The ensemble model classified tokens based on a voter mechanism that chose the most voted label by the 10 single models. The results of our ensemble model are also shown in Table 3. We can see that our ensemble model achieved an F1 score of \(88.78\), precision of \(89.11\), and recall of \(88.46\). Compared to the previous best solution \"ELMo (clinical) + BiLSTM-CRF (ensemble)\", our model improved by \(0.18\%\) in F1 and \(0.59\%\) in recall, but had a lower performance for precision (\(-0.23\%\)). The F1 and precision had a slight variation between our model and the previous best solution, but we can see a noticeable improvement in recall, just like the comparison between the single models. The improvement in recall agrees with the aim of the multi-task mechanism in our model, which was \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Solutions** & **F1** & **Precision** & **Recall** \\ \hline GloVe-biLSTM–CRF [7] & 83.88 & 84.36 & 83.41 \\ Clinical Named Entity Recognition system & 83.8 & 84.0 & 83.6 \\ (CliNER) 2.0 [52] & 85.71 & 86.27 & 85.15 \\ Att-biLSTM–CRF + Transfer [19] & 85.71 & 86.27 & 85.15 \\ ELMo (General) + BiLSTM–CRF (Single) [24] & 82.54 \(\pm\) 0.14 & 83.26 \(\pm\) 0.25 & 81.84 \(\pm\) 0.22 \\ \hline Word2 vec + multitask-Att-biLSTM–CRF & 78.70 & 79.98 & 77.47 \\ ELMo (General) + multitask-Att-biLSTM–CRF & 83.00 & 82.91 & 83.09 \\ \hline ELMo (Clinical) + BiLSTM–CRF (Single) [24] & 86.84 \(\pm\) 0.16 & 87.44 \(\pm\) 0.27 & 86.25 \(\pm\) 0.26 \\ Our model (Single) & **87.53 \(\pm\) 0.11** & **87.75\(\pm\) 0.18** & **87.32 \(\pm\) 0.26** \\ \hline ELMo (Clinical) + BiLSTM–CRF (Ensemble) [24] & 88.60 & 89.34 & 87.87 \\ Our model (Ensemble) & **88.78** & 89.11 & **88.46** \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Results of experiments on the 2010 i2b2/VA dataset. to enhance the model's sensibility to unknown tokens. To see how the multitask mechanism performed if different embedding methods were implemented, we changed the embedding part of our system to general pretrained Word2 vec and general pretrained ELMo, and the results are shown in Table 3. The result shows that the system with Word2 vec embeddings performs not as good as we expected. The reason may be that the hyperparameters and label format of our system were adjusted for the contextual word embedding. However, the system with general ELMo performed better compared to the result in Reference [24]. The result indicates that the multitask mechanism may have better performance with contextual word embeddings. The evaluation of the prediction for each type of medical entities is shown in Table 4. We can see that the performance of our system on predicting problem entities was better than the other two kinds of entities in all three indicators, and the prediction of medical test was a little worse than problem and treatment entities. The reason for this difference may be the imbalance of the training dataset, in which there were 7073 problem entities, 4844 test entities, and 4606 entities.