# A Hybrid Deep Learning Model for Protein-Protein Interactions Extraction from Biomedical Literature
## 3 The Model Description
### Intermediate Structure  
As shown in Figure 1, the two channel embeddings each separately pass through a bidirectional LSTM network consequent with a CNN network. bidirectional LSTM enables learning long-term dependencies for both the preceding and succeeding contexts. After that, CNN encodes the important information contained in the bidirectional LSTM networks.  
The bidirectional LSTM network is a combination of Bidirectional RNNs with LSTM, using an input word embedding sequence \((e_{1},e_{2},\ldots,e_{N})\), where \(N\) denotes the max length of input sequences. As illustrated in Figure 3, bidirectional RNNs compute the forward hidden sequence \(\overset{\rightarrow}{h}\), the backward hidden sequence \(\overset{\leftarrow}{h}\), and the output sequence \(y\) by iterating the forward layer from \(t=(1,\ldots,N)\), the backward layer from \(t=(N,\ldots,1)\) and then updating the output layer as follows:  
\[\overset{\rightarrow}{h}_{t}=S\Big{(}W_{\overset{\leftarrow}{e_{h}}}e_{t}+W_ {\overset{\rightarrow}{h}\overset{\rightarrow}{h}}h_{t-1}e_{t}+b_{\overset{ \rightarrow}{h}}\Big{)} \tag{1}\]  
\[\overset{\leftarrow}{h}_{t}=S\Big{(}W_{\overset{\leftarrow}{e_{h}}}e_{t}+W_ {\overset{\leftarrow}{h}\overset{\leftarrow}{h}}h_{t+1}e_{t}+b_{\overset{ \rightarrow}{h}}\Big{)} \tag{2}\]  
\[y_{t}=W_{\overset{\rightarrow}{h}y}\overset{\rightarrow}{h}_{t}+W_{ \overset{\leftarrow}{h}y}\overset{\leftarrow}{h}_{t}+b_{y} \tag{3}\]  
where \(W\) denotes weight matrices, \(b\) denotes bias vectors, and \(S\) is the hidden layer function on each element of a vector.  
In the bidirectional LSTM network, each unit of RNN is an LSTM (Figure 4):  
\[f_{t}=\sigma\Big{(}W_{\overset{\leftarrow}{e_{t}}}e_{t}+W_{\overset{ \leftarrow}{h}f}h_{t-1}+W_{\overset{\leftarrow}{c_{f}}}c_{t-1}+b_{f}\Big{)} \tag{4}\]  
\[i_{t}=\sigma(W_{\overset{\leftarrow}{e_{t}}}e_{t}+W_{\overset{\leftarrow}{ h}i}h_{t-1}+W_{\overset{\leftarrow}{c_{f}}}c_{t-1}+b_{i}) \tag{5}\]  
\[o_{t}=\sigma(W_{\overset{\leftarrow}{\alpha}}e_{t}+W_{\overset{\leftarrow}{ h}o}t_{t}+b_{\overset{\leftarrow}{o}}) \tag{6}\]  
\[c_{t}=f_{t}c_{t-1}+i_{t}\text{tanh}(W_{\overset{\leftarrow}{c_{f}}}e_{t}+W_{ \overset{\leftarrow}{h}c}h_{t-1}+b_{\overset{\leftarrow}{c}}) \tag{7}\]  
\[h_{t}=o_{t}\text{tanh}(c_{t}) \tag{8}\]  
where \(\sigma\) is the logistic sigmoid function, and \(f_{t},\ i_{t},\ o_{t},\ c_{t}\) are the forget gate, input gate, output gate, and cell state, respectively at the time step \(t\).  
Figure 3: Illustration of the bidirectional recurrent neural network.  
The bidirectional LSTM network outputs for both channels are each separately fed to CNN networks. Given an output sequence \(y_{i}\) of the bidirectional LSTM network, it is input into the convolutional layer. The convolutional layer contains a set of filters for different window sizes \(g\), and computes the output feature map \(C\) as follows:  
\[C=\left[m_{1},\ m_{2},\ \ldots,m_{N-g+1}\right] \tag{9}\]  
\[m_{i}=f\Big{(}W_{i}y_{i{i+g-1}}+b_{i}\Big{)} \tag{10}\]  
where \(f\) is an activation function, \(b\) is a bias term, and is element-wise multiplication.  
Then the concatenation operation is applied to join multiple outputs from the Max-Pooling layer into a single tensor for each channel. Finally, the CNN network outputs for both channels are concatenated to a unique vector and fed to a fully connected layer.