## 3 Materials and Methods
### Attentive Bi-LSTM Circulating  
Bi-LSTM, which can remember the information from both directions, was adopted. A dropout operation [31] could randomly ignore a part of layer's neurons during training. Thus, this paper introduces dropout to reduce complex coadaptation relationships between neurons. When the neurons are reduced, it can have an advantage of model fusion in addition to allowing the network to achieve better parameters. The dropout process is shown in Figure 3.  
The vectors that are obtained by the previous layer are trained forward and backward several times to obtain the hidden state vector \(h_{t}\). \(h_{t}\) is the output of the last timestamp to  
Figure 3: The dropout operation in the neural network. The dashed lines indicate randomly deleted neurons to prevent overfitting.  
represent the meaning of the inputs. The Bi-LSTM circulating process is shown in Figure 4. The calculation process can be expressed by the following equations.  
\[f_{t}=\sigma\Big{(}W_{f}\cdot[h_{t-1},x_{t}]+b_{f}\Big{)}, \tag{1}\]  
\[i_{t}=\sigma(W_{i}\cdot[h_{t-1},x_{t}]+b_{i}), \tag{2}\]  
\[g_{t}=tanh(W_{c}\cdot[h_{t-1},x_{t}]+b_{c}), \tag{3}\]  
\[c_{t}=i_{t}g_{t}+f_{t}c_{t-1}, \tag{4}\]  
\[o_{t}=\sigma(W_{o}\cdot[h_{t-1},x_{t}]+b_{0}) \tag{5}\]  
\[h_{t}=o_{t}tanh(c_{t}) \tag{6}\]  
As shown in the equations above, the output of the model at moment \(t\) is noted as \(h_{t}\). \(W\) and \(b\) are denoted as weight and bias, respectively. A _sigmoid_ function is used in various gates to forget and retain information. A _tanh_ function is used to process data for the state and output.  
We redesigned the Bi-LSTM encoder-decoder in a circulating layer by constructing an attention mechanism, which fuses with the Bi-LSTM. The attention dependent on the Bi-LSTM part is shown in Figure 5. Before the hidden state vector \(h_{t}\) passes to the next layer, it is weighted by attention. Finally, we obtain \(h_{t}\)\({}^{a}\).  
Figure 4: The process of Bi-LSTM circulation.  
Figure 5: Attention operation depend on Bi-LSTM.  
The attention mechanism includes an attention weight layer and a weight sum layer. First, the attention weight layer calculates the weights of each word in the hidden state vector \(h_{t}\) that is obtained from the Bi-LSTM encoder-decoder. The similarity \(s\) between \(h_{t}\) and \(h\) is calculated by the inner product and regularized by the _Softmax_ function to obtain the weights of each word. Then, the weight sum layer calculates the weighted sum of \(a\) and \(h_{t}\) and outputs the context vector \(c\), as is shown in Figure 6.  
The attention mechanism calculation process can be divided into three stages. Firstly, the similarity of the query and key is computed as follows:  
\[Similarity(Query,Key_{i})=\frac{(Query*Key_{i})}{(||Query||).||Key||} \tag{7}\]  
Then, normalization of the obtained weights is formulated as follows:  
\[a_{i}=Softmax[Sim_{i}]=\frac{e^{Sim_{i}}}{\sum_{j=1}^{L_{x}}e^{Sim_{j}}}. \tag{8}\]  
Finally, in the weighted summation of the sequence elements above, \(a_{i}\) is represented as follows:  
\[Attention(Query,Source)=\sum_{i=1}^{L_{x}}a_{i}*Value_{i} \tag{9}\]  
Max pooling: To reduce information redundancy and prevent overfitting, the max pooling method is utilized to represent the questions and answers. The maxing pooling formula is as follows:  
\[r_{j}=\underset{1<i<l}{\max}P_{i,j}, \tag{10}\]  
where \(r_{j}\) denotes the fixed size vector representation of the \(jth\) element and \(l\) denotes the input length.