# Global and Local Information Adjustment for Semantic Similarity Evaluation
## 3 Materials and Methods
### Bidirectional Long Short-Term Memory  
RNN is a neural network with good performance for sequential data processing, but learning ability is poor due to gradient vanishing, in which the gradient disappears during backpropagation, as the timestep, which refers to the order of input data, increases [7]. This problem is called long-term dependency. To solve the shortcomings of RNN, LSTM has been proposed. LSTM has a structure in which a cell state (\(c_{t}\)) that can have long-term memory is added to the hidden state (\(h_{t}\)) of RNN and solves long-term dependency [7]. To obtain \(h_{t}\) and \(c_{t}\), LSTM uses an input gate (\(i_{t}\)) a forget gate (\(f_{t}\)), an output gate (\(o_{t}\)), and a cell (\(g_{t}\)) that analyzes \(h_{t-1}\) of the previous timestep and (\(x_{t}\)) of the input information of the current timestep. \(f_{t}\) is a gate that determines how much information of \(c_{t-1}\) is preserved by calculating the Hadamard product of \(f_{t}\) and \(c_{t-1}\). \(i_{t}\) is a gate that determines how much information of \(x_{t}\) is also reflected by calculating the Hadamard product of \(i_{t}\) and \(g_{t}\). \(o_{t}\) is a gate that determines \(h_{t}\), the current hidden state, by calculating the Hadamard product of \(o_{t}\) and \(\tan\h(c_{t})\). The equations of LSTM are as follows:  
In Equation (1), the sigmoid function in \(i_{t}\), \(f_{t}\), and \(o_{t}\) results in adjusting how much of that information will be reflected. \(W_{xi}\), \(W_{xf}\), \(W_{wo}\), and \(W_{xg}\) are the learning weight matrices connected to \(x_{t}\), \(W_{hi}\), \(W_{hf}\), \(W_{ho}\), and \(W_{hg}\) are connected to \(h_{t-1}\), and \(b_{i}\), \(b_{f}\), \(b_{o}\), and \(b_{g}\) are biases of the corresponding layers. \(c_{t}\) is updated by adding the Hadamard product of \(f_{t}\) and \(c_{t-1}\) and the Hadamard product of \(i_{t}\) and \(g_{t}\). Finally, \(h_{t}\) is created by Hadamard product of \(\tan\h(c_{t})\) and \(o_{t}\).  
\[\begin{split} i_{t}&=\ \sigma\ (W_{xi}x_{t}+W_{hi}h_{t-1}+b_{i})\\ f_{t}&=\ \sigma\ (W_{xf}x_{t}+W_{hf}h_{t-1}+b_{f} )\\ o_{t}&=\ \sigma\ (W_{xo}x_{t}+W_{ho}h_{t-1}+b_{o})\\ c_{t}&=\ f_{t}\odot c_{t-1}+i_{t}\odot g_{t}\\ y_{t}&=h_{t}&=\ o_{t}\odot\tan\h(c_{t} )\end{split} \tag{1}\]  
Bi-LSTM has a structure with a forward LSTM that sequentially approaches from \(x_{1}\) to \(x_{L}\) and a backward LSTM that sequentially approaches from \(x_{L}\) to \(x_{1}\). Hidden states sequentially processed in Bi-LSTM are defined as follows:  
Equation (2) is the concatenation of the hidden state of the forward LSTM of timestep \(i\) and the hidden state of the backward LSTM, where \(L\) refers to the length of the sequence. In Equation (3), \(H\) is the concatenation of the hidden state of all timesteps. This \(H\) is then inputted to the attention mechanism.  
\[\h_{i}=\left[\text{LSTM}\Big{(}x_{i}^{\text{f}}\Big{)}\ ;\text{LSTM}\Big{(}x_{L}^{\text{b}} \Big{)}\right] \tag{2}\]  
\[H\ =\ [h_{1},\ h_{2},\ \cdots h_{L-1},\ h_{L}] \tag{3}\]  
The global features that contain the information of the whole sentence are defined as follows:  
Equation (4) refers to the concatenation of the last hidden state of Bi-LSTM. The global features are used with local features, to be mentioned in the section on the capsule network, to further improve the accuracy of the model in the future.  
\[h_{L}\ =\ \left[LSTM\Big{(}h_{L}^{f}\Big{)}\ ;\ LSTM\Big{(}h_{1}^{\text{b}} \Big{)}\right] \tag{4}\]