# Rate of Penetration Prediction Method for Ultra-Deep Wells Based on LSTM-FNN
## 2 Artificial Intelligence Algorithm  
### Fully Connected Neural Network  
A fully connected neural network (FNN) is a computational model that can be used to approximate functional relationships between variables, identify features or perform other complex tasks. A typical fully connected neural network is shown in Figure 1, which consists of multiple layers, each of which contains many nodes. The neural network starts from the input layer, and the output of the previous layer acts as the input of the next layer until the final result is output. As neurons receive input data from other neurons, each with some weight, the total input value received by the neurons will be compared to the neuron threshold and then processed through the activation function to generate the neuron output. As shown in Equation (1), the input matrix of the neural network is recorded as \(X_{in}\), the output matrix is recorded as \(Y_{out}\), the output matrix of \(n\) layer is recorded as \(Y_{n}\), the weight matrix is \(W_{n}\) and the bias matrix is  
LSTM has two transmission states; cell state (\(C_{t}\)) and hidden state (\(h_{t}\)). Throughout the entire process, \(C_{t}\) changes quite slowly. Usually, the output \(C_{t}\) in each different node is \(C_{t-1}\) of the previous cell plus several signals, while \(h_{t}\) varies greatly. Additionally, in the LSTM algorithm, individual neurons are replaced by a unit structure. Each unit contains three gates: the forgetting gate, the input gate and the output gate. The whole process of LSTM can be briefly described by the Equation (4), below. In this equation, W is the weight in the whole process whereas \(b\) is the bias. \[ \begin{bmatrix}\widetilde{c}_{t}\\ o_{t}\\ i_{t}\\ f_{t}\end{bmatrix} = \begin{bmatrix}tanh\\ \sigma\\ \sigma\\ \sigma\end{bmatrix} \left(W \begin{bmatrix}x_{t}\\ h_{t-1}\end{bmatrix} +b\right)\\ c_{t}=f_{t}\cdot c_{t-1}+i_{t}\cdot\widetilde{c}_{t}\\ h_{t}=o_{t}\cdot\tanh(c_{t})\end{bmatrix} \tag{4}\]  
Forgetting gate: The forgetting gate is a key component in the LSTM unit. It can control which information of the input is retained and which is forgotten, and avoid the problems of gradient vanishing and gradient explosion. The structure of the forgetting gate is shown in Figure 3a. The forgetting gate determines what information the neural network discards from the previous cell state. The gate reads \(h_{t-1}\) and \(x_{t}\), then maps them to a signal \(f_{t}\) between 0 and 1 by a sigmoid function. Finally, this signal \(f_{t}\) is multiplied \(c^{t-1}\) of the previous cell state to determine what information is discarded. The signal of 1 indicates that the information is completely retained, while the signal of 0 indicates the information is completely discarded. The calculation expression is shown in Equation (5):  
\[f_{t}=\sigma\left(W_{f}\cdot[h_{t-1},x_{t}]+b_{f}\right) \tag{5}\]  
Input gate: The input gate is used to control how much of the input information is saved to the memory cell. The input gate consists of two parts. First, the input gate generates candidate state \(\widetilde{c}_{t}\) of the current unit through a tanh layer as seen in Equation (6):  
\[\widetilde{C}_{t}=\tanh(W_{C}\cdot[h_{t-1},x_{t}]+b_{C}) \tag{6}\]  
Second, the sigmoid function in the input gate produces a signal \(i_{t}\) between 0 and 1. This is used to control how much information the candidate state \(\widetilde{c}_{t}\) needs to save. The  
Figure 2: The LSTM structure [27]. structure of the signal acquisition in the input gate is shown in Figure 3b. The signal \(i_{t}\) will be generated by the following equation (Equation (7):  
\[i_{t}=\sigma(W_{i}\cdot[h_{t-1},x_{t}]+b_{i}) \tag{7}\]  
The structure of obtaining \(C_{t}\) in the input gate is shown in Figure 3c. Through the input gate, the old cell state will be updated to the new cell state.