# A Novel Neural Network-Based Method for Medical Text Classification
## 3 Our Work
### Sentence Representation
#### 3.1.3 Bidirectional GRU and Attention Mechanism  
GRU is specified for sequential modeling and can extract the contextual information from the feature sequences obtained by the convolutional layer. The effect of the bidirectional GRU is to establish the word vector representation. Because each word contributes differently to the sentence, extracting important words in the sentence is a common way of solving the problem. Attention mechanisms can quickly extract important features of sparse data to enhance understanding of sentence. Therefore, a combination of bidirectional GRU and attention mechanism can obtain more important features in sentence.  
Bidirectional GRU obtains the annotations for words by summarizing the two directions (forward and backward) information of words, so the annotations contain contextual information for the sentence. Bidirectional GRU include the forward GRU (represented as \(\overrightarrow{GRU}\)) which reads the feature sequence \(Ls\) and the backward GRU (represented as \(\overleftarrow{GRU}\)) which reads \(Ls\). Formally, bidirectional GRU outputs in two directions are stated as follows:  
\[\overrightarrow{h_{if}}=\overrightarrow{GRU}(Ls_{n}),n\in[1,100] \tag{7}\]  
\[\overleftarrow{h_{ib}}=\overleftarrow{GRU}(Ls_{n}),n\in[100,1]. \tag{8}\]  
An annotation for a given feature sequence \(Ls\) of sentence i is obtained by the forward hidden state \(\overrightarrow{h_{if}}\) and the backward hidden state \(\overleftarrow{h_{ib}}\). These states summarize the information of the sentence centered around \(Lsn\) and implement the word encoding.  
Attention mechanism can focus on the features of the keywords to reduce the impact of non-keywords on the text. The workflow of attention mechanism in sentence representation is detailed below.  
The word annotation \(\overrightarrow{h_{if}}\) is first fed to get \(\overrightarrow{u_{if}}\) by one layer perceptron a hidden representation of \(\overrightarrow{h_{if}}\). The \(\overrightarrow{u_{if}}\) is formulated as follows:  
\[\overrightarrow{u_{if}}=\tanh(\overrightarrow{w_{if}}+b), \tag{9}\]  
where \(w\) and \(b\) are represented as the weight and bias in the neuron, \(\tanh(.)\) is hyperbolic tangent function. The model uses the similarity between \(\overrightarrow{u_{if}}\) and a word level context vector \(\overrightarrow{v_{if}}\) to measure the importance of each word. And then it uses the softmax function to get the normalized weight \(\overrightarrow{a_{if}}\) of each word. \(\overrightarrow{a_{if}}\) is formulated as follows:  
\[\overrightarrow{a_{if}}=\frac{exp(\overrightarrow{u_{if}}*\overrightarrow{v _{if}})}{\sum_{j=1}^{M}(exp(\overrightarrow{u_{if}}*\overrightarrow{v_{if}} ))}, \tag{10}\]  
where \(M\) is the number of words in the text and \(exp(.)\) is the exponential function. \(*\) is multiplication. The word level context vector \(\overrightarrow{v_{if}}\) in sentence \(si\) can be seen as a high-level representation of the informative words over the words and is randomly initialized and jointly learned during the training process.  
After that, a weighted sum of the forward read word annotations based on the weight \(\overrightarrow{a_{if}}\) is computed as the forward sentence representation \(F_{i}\). The \(F_{i}\) is the part of the output of the attention layer and it can be expressed as:  
\[F_{i}=\sum(\overrightarrow{a_{if}}*\overrightarrow{h_{if}}). \tag{11}\]  
Similar to \(\overrightarrow{a_{if}}\), \(\overleftarrow{a_{ib}}\) can be calculated using the backward hidden state \(\overleftarrow{h_{ib}}\). Like \(F_{i}\), the backward sentence representation \(H_{i}\) is also the part of the output of the attention layer and it can be expressed as:  
\[H_{i}=\sum(\overleftarrow{a_{ib}}*\overleftarrow{h_{ib}}). \tag{12}\]  
In this way, we obtain an annotation for a given feature sequence \(Ls\) by concatenating the forward sentence representation \(F_{i}\) and backward sentence representation \(H_{i}\). The sentence representations \(s_{i}=[F_{i},H_{i}]\) are obtained.