# An Improved Method for Named Entity Recognition and Its Application to CEMR+
## 3 Method
### Attention-Based ID-CNNs-CRF Model
#### 3.3.2 Attention Mechanism  
Our model can rapidly aggregate broad context by using iterated dilated CNNs. However, the extracted broad context usually ignores the importance of the current word for the current tag.  
Figure 4: Normal CNN (**left**) acts on a continuous position of the input matrix and continuously slides to do convolution and pooling. The dilated CNN (**right**) adds a dilation width to the filter. When it acts on the input matrix, it skips the input data in the middle of all the dilation width, and the size of the filter matrix itself remains the same, so the filter achieves a wider input.  
Figure 5: A dilated CNN block. The input is a sequence of texts, each of which is the word representation of Figure 2 (for example, 15 words are entered). The circles just represent the concept of dimensions. Layer 1 has a coefficient of expansion of 1 to obtain a receptive field of 3. Layer 2 expands by 1 on the basis of Layer 1; although the receptive field in Layer 2 is still 3, the receptive field in the input is 7. Take the orange unit of Layer 3 as an example: all related neurons are connected by thick lines, and the receptive field is 15. Each unit of each layer is connected to the three units of the previous layer; the number of parameters remains unchanged, but the receptive field increases geometrically.  
To alleviate this problem, we applied the attention mechanism to our model. The attention mechanism has recently become popular in image processing and natural language processing. [PERSON] et al. (2018) utilized an attention mechanism to enforce tagging consistency across multiple instances of the same token in a document. [PERSON] et al. (2018) introduced the attention mechanism to enhance their model performances. However, their attention mechanism focuses on which encoded elements contribute to the generation of the current unit. Different from them, we applied the attention mechanism to focus on the related tokens in the sequence.  
In the attention layer, we calculated the attention weight by Equation (5) between the current token and all tokens in the sequence as the projection matrix and normalized it with the softmax function.  
\[\alpha_{t,i}=\frac{exp(sim(b_{t}^{L},b_{i}^{L}))}{\sum_{k=1}^{n}exp(sim(b_{t} ^{L},b_{k}^{L}))} \tag{5}\]  
where \(b_{t}^{L}\) is the final output position \(t\) in Equation (4) and \(sim\) denotes the similarity between two vectors. The similarity between two vectors was measured by cosine similarity according to Equation (6).  
\[sim(b_{t}^{L},b_{i}^{L})=\frac{W_{a}(b_{t}^{L}b_{i}^{L})}{\left\|b_{t}^{L} \right\|\left\|b_{t}^{L}\right\|} \tag{6}\]  
where \(W_{a}\) is a weight matrix, which is learned in the training process. We took the weight vector computation of the unit at position \(b_{t}^{L}\) as an example to explain the detailed computation process. According to Equation (7), we can achieve the coefficient of each output in the attention layer, then we calculate the output \(a_{t}\) under this attention coefficient.  
\[a_{t}=\sum_{i=1}^{N}\alpha_{t,i}b_{i}^{L} \tag{7}\]  
Then, the output of the current position and the output of the attention layer are concatenated as the output of this module.  
\[o_{t}=W_{o}[a_{t}:b_{t}^{L}] \tag{8}\]  
where \(W_{o}\) is a weight matrix that maps the output to the category space. Finally, like the general NER method, the CRF layer is added for the final sequence labeling.