## 2 Materials and Methods
### The Att-biLSTM Model  
A recurrent neural network (RNN) is a type of neural network designed to handle sequential data. For sequential data such as stock price data within a period and every token in one sentence, the data in step \(t\) typically have some relationships with the previous step. In a language model, the RNN can \"remember\" the information before the current step, which makes it suitable for sequence prediction [46]. Particularly, for a sequential data series \(x_{task}=\{x_{0},x_{1},x_{2},\ldots,x_{t},\ldots,x_{n}\}\) where \(x_{t}\) is the \(t\) step of \(x_{task}\), the model calculates the hidden cell output \(\text{h}_{t}\) by \(x_{t}\) and \(\text{h}_{t-1}\) for each step at first; then, computing hidden state outputs of all the steps, the model computes its output \(o_{task}\), and each \(\text{o}_{t}\) is calculated by \(\text{h}_{t}\). The mathematical expression of the forward propagation process is as follows:  
\[i_{t}=\tan\text{h}(Ux_{t}), \tag{4}\]  
\[\text{h}_{t}=Wh_{t-1}+i_{t}+bias_{h}, \tag{5}\]  
\[\text{o}_{t}=Wh_{t}+bias_{o}, \tag{6}\]  
where \(U\) is the weight of the input layer, \(W\) is the weight in the hidden cell, and \(V\) is the weight of the output layer. \(U\), \(W\), and \(V\) are shared for all steps. Commonly, after computing the outputs, another layer is added based on the task. For a classification task, a softmax function is usually used to normalize the probability of each class. Theoretically, a naive RNN model can handle the previous information for each step. However, in practice, the problems of vanishing gradient and exploding gradient in backpropagation through time (BPTT) result in it failing to learn enough information from previous steps and handle long-term dependencies [47]. Facing this dilemma, the LSTM model was implemented. The LSTM model combats the vanishing gradient and exploding gradient problem by its gating and cell state mechanism [15]. The mechanism includes a forget gate \(f\), an input gate \(I\), and a cell state C. The forward propagation process in an LSTM cell is as follows:  
\[f_{t}=\sigma\Big{(}W_{fh}h_{t-1}+W_{fx}x_{t}+bias_{f}\Big{)}, \tag{7}\]  
\[I_{t}=\sigma(W_{ih}h_{t-1}+W_{ix}x_{t}+bias_{i}), \tag{8}\]  
\[\text{C}_{t}=f_{t}\text{C}_{t-1}+i_{t}\text{tanh}(W_{Clh}h_{t-1}+W_{cx}x_{t}+ bias_{C}), \tag{9}\]\[z_{t}=\sigma(W_{zh}h_{t-1}+W_{xx}x_{t}+bias_{o}), \tag{10}\]  
\[h_{t}=z_{t}\text{tanh}(C_{t}), \tag{11}\]  
where \(f_{t}\) is the forget weight, and \(\sigma\) is the sigmoid function which restricts \(f_{t}\) between [0, 1]. In the range, 0 means to completely forget previous information, and 1 means to completely remember the previous information. \(I_{t}\) is the input weight of cell \(t\), and it decides how much information should enter the cell. \(C_{t}\) is the value of the current cell \(t\). It consists of previous information adjusted by the forget gate and current information by the input gate. At last, the output of the LSTM cell \(h_{t}\) is the cell state value normalized by a tanh function and then adjusted by \(z_{t}\), in which \(z_{t}\) decides how much information should be added to the output. In our work, the model needs to decide a token's label not only by the previous tokens but also by the tokens behind it; thus, we use a two-layer biLSTM to gather information on each token from both directions (shown in Figure 1). As described above, a single bidirectional LSTM generates an output \(\overset{\rightarrow}{h}_{t}\) and, for a biLSTM, it uses two independent single LSTM layers to generate an output \(\overset{\rightarrow}{[h}_{t},\overset{\leftarrow}{h}_{t}]\). \([\overset{\rightarrow}{h}_{t},\overset{\leftarrow}{h}_{t}]\) is the final representation of token \(t\) in our model. Considering that, when human beings classify a token into some kind of entity, they may rely on some similar representations around it, we add a multi-head self-attention layer [48] after the biLSTM layer. In this layer, we compute the attention score by a query vector (Q), a key vector (K), and a value vector (V).