## 2 Model
### Deep Neural Network Module  
The deep neural network module is composed of two parts: Bi-GRU and CNN. As we all know, text is structured and organized. With the purpose of avoiding the destruction of the sequence structure of text, our model passes the output of feature-enhanced word-embedding layer to the Bi-GRU layer and then to the CNN layer, and obtain the final representation of the input sentence. In the following, we will detail the two parts in the order of the data flow. As shown in Figure 1, the Bi-GRU layer contains two sub-layers for the forward and backward sequences, respectively, which is beneficial to have access to the future context as well as the past context. Since the GRU unit is a variant of LSTM, we first briefly review the LSTM for the sequence modeling task. The main idea of the LSTM is to overcome the problem of gradient vanishing and expansion in the recurrent neural network by introducing an adaptive gating mechanism that controls the data flow to and from their memory cell units. Taking the sequence vector \(X=\left[x_{1},x_{2},\ \cdots,\ x_{n}\right]\) from the output of feature-enhanced word-embedding layer as an example, LSTM processes the data word-by-word corresponding to the time-step from the past to the future. At time step \(t\), the current hidden state \(h_{t}\) and the current memory-cell state \(c_{t}\) are calculated as follows:  
\[i_{t}=\textit{sigmoid}\ (W_{i}\cdot\left[h_{t-1},x_{t}\right]+b_{i}) \tag{8}\]  
\[f_{t}=\textit{sigmoid}\ \left(W_{f}\cdot\left[h_{t-1},x_{t}\right]+b_{f}\right) \tag{9}\]  
\[o_{t}=\textit{sigmoid}\ \left(W_{o}\cdot\left[h_{t-1},x_{t}\right]+b_{o}\right) \tag{10}\]  
\[\overline{c_{t}}=\ \textit{tanh}\ \left(W_{c}\cdot\left[h_{t-1},x_{t}\right]+b_{c}\right) \tag{11}\]\[c_{t}=f_{t}\odot c_{t-1}+\ i_{t}\odot\overline{c_{t}} \tag{12}\]  
\[h_{t}=o_{t}\odot\tan\h(c_{t}) \tag{13}\]  
where \(\cdot\) denotes matrix multiplication and \(\odot\) stands for element-wise multiplication. \(i_{t}\) refers to the input gate which controls the input of new information to the memory cell, \(f_{t}\) indicates the forget gate, which controls how long certain values are held in the memory cell, and \(o_{t}\) represents the output gate which controls how much the values stored in the memory cell affect the output activation of the block. \(W_{i}\), \(W_{f}\), and \(W_{o}\) are the weight matrixes for these three gates, respectively, while \(b_{i}\), \(b_{f}\), \(b_{o}\) are the bias vectors for these gates. As for GRU, it is regarded as an optimization of LSTM, and it can not only merge the input gate \(i_{t}\) and the forget gate \(f_{t}\) of LSTM into the reset gate \(r_{t}\), but also optimize the update mode of the hidden state \(h_{t}\). In addition, the output gate \(o_{t}\) corresponds to the update gate \(z_{t}\). Throughout this design, the GRU can maintain the advantage of the LSTM while having a simpler structure, fewer parameters, and better convergence than the LSTM [13]. As shown in Figure 3, at each time-step \(t\), the GRU transition functions are defined as follows:  
\[r_{t}=\textit{sigmoid}\ (W_{r}\cdot[h_{t-1},x_{t}]) \tag{14}\]  
\[z_{t}=\textit{sigmoid}\ (W_{z}\cdot[h_{t-1},x_{t}]) \tag{15}\]  
\[\widetilde{h_{t}}=\textit{tanh}\left(W_{\widetilde{h}}^{-}\cdot[r_{t}\odot h _{t-1},x_{t}]\right) \tag{16}\]  
\[h_{t}=(1-z_{t})\odot h_{t-1}+z_{t}\odot\widetilde{h_{t}} \tag{17}\]  
In the Bi-GRU layer, the forward GRU layer processes the sentence word-by-word in the order of the input sequence to obtain the hidden state \(\widetilde{h_{t}}\) at each time-step \(t\). The backward GRU layer does the same thing, except that its input sequence is reversed.