## 2 Materials and Methods
### Related Work on Clinical Named Entity Recognition
Feature engineered machine learning systems can learn rules and patterns through a training process, which dramatically improves efficiency, and studies showed that these systems achieve considerable performance [32; 33; 34; 35]. However, the system's performance highly relies on the features that humans selected, which decreases the robustness of the system. Along with the development of deep learning in NLP, systems based on deep learning methods were proposed. Compared to systems based on machine learning algorithms, one of the best advantages of deep learning systems is the avoidance of manual work. It benefits from unsupervised pretrained embeddings like Word2 vec [22] and GloVe [23]. Actually, the first neural network architecture proposed by [PERSON] et al. for NER tasks constructed features by orthographic information and lexicons which also contained manual work [25], whereas [PERSON] et al. improved his model by replacing those manually built features with word embedding, which converts a word into an N-dimension vector through an unsupervised training process [36]. Studies on CNER based on deep learning methods mainly follow two directions. One is to optimize the learning model, and the other is to construct or pre-train better embeddings, which can provide more information for the learning model. For the studies on learning models, [PERSON] et al. firstly proposed a model with convolution layers to capture local information in the sequence [36]. Models based on Recurrent Neural Network (RNN) were proposed due to its superior ability in sequence learning. [PERSON] et al. proposed a bi-directional LSTM model for sequence labeling and showed that assembling a CRF layer on top of an LSTM could improve performance [37]. [PERSON] et al. proposed the biLSTM-CRF model for NER [38]. The biLSTM with CRF-based model showed its success in many CNER studies. [PERSON] et al. extracted clinical concepts with a biLSTM-CRF architecture, and achieved 83.88% F1 score, 84.36% precision score, and 83.41% recall score on the 2010 I2B2/VA dataset (the version with 170 training notes), which was better than all prior work [7]. [PERSON] et al. extracted disease named entities with the same architecture, and achieved 80.22% F1, which was also better than prior work [39]. [PERSON] et al. compared CRFs, Structured Support Vector Machines (SSVMs), semi-Markov, Convolutional Neural Network (CNN), and biLSTM-CRF on the 2010 I2B2/VA dataset (the version with 349 training notes) and found that biLSTM-CRF achieved the best performance among all learning models with 85.94% F1 score [14]. [PERSON] et al. combined biLSTM-CRF with a global attention mechanism, and conducted experiments on the 2010 I2B2/VA dataset (the version with 170 training notes). They achieved 85.71% F1 score, 86.27% precision score, and 85.15% recall score, which performed the best compared to prior work [19]. At present, biLSTM-CRF is the most approved learning architecture for CNER tasks. For studies on embeddings, word embedding was widely used in NER tasks. [PERSON] et al. compared random embedding, Word2 vec, and GloVe in biLSTM-CRF, and found that the system with GloVe outperformed others [7]. [PERSON] et al. showed that the pre-training process of word embedding is crucial for NER systems, and, for domain-specific NER tasks, domain-specific embeddings could improve the system's performance [40]. [PERSON] et al. used pretrained Word2 vec embeddings on (Medical Literature Analysis and Retrieval System Online) MEDLINE and Wikipedia corpus and achieved considerable performance compared to other studies [41]. As a word can be seen as a sequence of characters, and characters in a word contain parts of a word's meaning and orthographic information, character-level embedding is quite useful for NER tasks. Normally, character-level embeddings are not pretrained; they are initialized randomly and trained by a sub-CNN or sub-RNN in the whole architecture. [PERSON] et al.