# Forward-Looking Element Recognition Based on the LSTM-CRF Model with the Integrity Algorithm
## 3 LSTM-CRF Model with the Integrity Algorithm
### LSTM-CRF Model  
LSTM networks are a specific form of recurrent neural network (RNN) proposed by [PERSON], [PERSON], and colleagues [22]. Unlike the general RNN model, conventional neurons are replaced with memory cells, each of which consists of input gates, forget gates, and output gates. This approach not only allows long-term dependencies to be identified but also avoids the problem of gradient disappearance or gradient expansion. In this article, the LSTM-CRF model is like that proposed by [PERSON] et al. [16]. The model can be formulated as follows:  
\[\dot{\mathbf{i}}_{t}=\sigma\left(\mathbf{W}_{xi}\,\mathbf{x}_{t}+\mathbf{W}_{hi}\,\mathbf{h}_{t-1}+ \mathbf{W}_{ci}\,\mathbf{c}_{t-1}+\mathbf{b}_{i}\right) \tag{1}\]  
\[\dot{\mathbf{f}}_{t}=\sigma\left(\mathbf{W}_{xf}\,\mathbf{x}_{t}+\mathbf{W}_{bf}\,\mathbf{h}_{t-1}+ \mathbf{W}_{ci}\,\mathbf{c}_{t-1}+\mathbf{b}_{j}\right) \tag{2}\]  
\[\widetilde{\mathbf{c}}\ =\ tanh\left(\mathbf{W}_{xc}\mathbf{x}_{t}\ +\ \mathbf{W}_{hc}\mathbf{h}_{t-1}\ +\ \mathbf{b}_{c}\right) \tag{3}\]  
\[\mathbf{c}_{t}\ =\mathbf{f}_{t}\odot\mathbf{c}_{t-1}\ +\ \dot{\mathbf{i}}_{t}\odot\widetilde{\mathbf{c}} \tag{4}\]  
\[\mathbf{o}_{t}=\sigma\left(\mathbf{W}_{x0}\,\mathbf{x}_{t}+\mathbf{W}_{ho}\,\mathbf{h}_{t-1}+\mathbf{W }_{co}\,\mathbf{c}_{t}+\mathbf{b}_{o}\right) \tag{5}\]  
Figure 1: The main architecture of our model. \[\mathbf{h}_{t}\ =\ \mathbf{o}_{t}\odot tanh(\mathbf{c}_{t}) \tag{6}\]  
where \(f\), \(i\), and \(o\) represent the forget, input, and output gates, respectively; \(c_{t-1}\) represents the state of the cell at time \(t-1\); \(c_{t}\) represents the state of the cell at time \(t\). \(h_{t}\) represents the output of the current state; \(h_{t-1}\) represents the output of the unit at the previous moment; \(\sigma\) is the logistic sigmoid function; W and b represent the weight and bias, respectively; and \(\odot\) is the element-wise product. The CRF model is an undirected graph model that was proposed by [PERSON] et al. (2013) in 2001. The model has obvious advantages in labeling and segmenting serialized data. It directly models the conditional distribution of data and can effectively avoid the problem of mark biasing experienced by other discriminant models. Moreover, the CRF model is different from other production models because it does not rely on the independence assumption, which allows it to fuse a variety of complex, non-local features and better capture the potential relationships among states. The goal of the CRF model is to calculate the conditional probability distribution of the optimal output sequence (predictive sequence) given the input sequence (observed sequence), i.e., \(P(Y\mid X)\). For example, assume that the random variable \(X=(x_{1},x_{2},\ldots,x_{n-1},x_{n})\) is the input sequence and the random variable \(Y=(y_{1},y_{2},\ldots,y_{n-1},y_{n})\) is the output sequence. Then, under the condition that the observation sequence is X, the conditional probability distribution of the prediction sequence Y is as shown in Formulas (7) and (8):  
\[P(X|Y) =\frac{1}{Z(X)}\exp(\sum_{k}\lambda_{k}\alpha_{k}(y_{i-1},y_{i},x, i)+\sum_{k}\mu_{k}\beta_{k}(y_{i},x,i)) \tag{7}\] \[Z(X) =\sum_{y}\exp(\sum_{k}\lambda_{k}\alpha_{k}(y_{i-1},y_{i},x,i)+ \sum_{k}\mu_{k}\beta_{k}(y_{i},x,i)) \tag{8}\]  
where \(\alpha_{k}\) and \(\beta_{k}\) are binary functions (0 or 1); \(\alpha_{k}\) is a transfer eigenfunction that reflects the correlation between adjacent marker variables at two positions, \(i-1\) and \(i\), and the effect of observation sequences on marker variables; \(\beta_{k}\) is a state eigenfunction that represents the effect of a node at observation sequence position i on the marker variable; \(\lambda_{k}\) and \(\mu_{k}\) are parameters corresponding to the characteristic functions \(\alpha_{k}\) and \(\beta_{k}\), respectively; and \(Z(X)\) is the normalization factor. The LSTM-CRF model is widely used in the NER task. The powerful data fitting ability of the LSTM model can be utilized, and sequence labeling can be directly implemented by the CRF.