# LSTM and Bat-Based RUSBoost Approach for Electricity Theft Detection
## 3 Proposed System Model
### Feature Extraction  
After the pre-processing stage, the input features are fed into the LSTM module. To get the refined features, the LSTM cell has been used [31]. Since a large amount of data is collected from SGCC, a traditional recurrent neural network (RNN) cannot be adopted. LSTM is a variant of RNN that solves the problems of gradient vanishing and gradient exploding. During the training of RNN, it uses past information and captures a temporal correlation between the previous state and the current input to predict the output. Due to its short memory, RNN fails to regain past information for large time series data.  
The LSTM has the ability to capture temporal correlations and classify large time series data. It has been used in many applications as it achieves significant results in speech recognition and image classification problems.  
The architecture of the LSTM is shown in Figure 4. It has a special type of memory cells in its architecture that use previous information and memorize the important features from large time series data. The property of the cell state is to keep this information.  
The LSTM has three gates: the input gate \(i_{t}\), the forget gate \(f_{t}\) and the output gate \(o_{t}\). The forget gate takes the previous hidden state information \(h_{t-1}\) and current input \(x_{t}\) through a pointwise multiplication operation and decides either to retain or remove the information from the cell state. This gate uses the sigmoid activation function and predicts an output of either 0 or 1. A value of 1 shows that the relevant information should be kept in the cell state, while 0 represents irrelevant information, which is discarded from the cell state. The forget gate, input gate and output gate are described in Equations (3)-(8), which are mentioned in [25]:  
\[f_{t}=\sigma(W_{f}[h_{t-1},x_{t}])+b_{f}. \tag{3}\]  
where \(W_{f}\) represents the weight and \(b_{f}\) is the bias of the forget gate \(f_{t}\). The \(\sigma\) is applied as the activation function on the forget gate.  
The input gate decides what information is going to be stored in the cell state \(C_{t}\). It takes the input \(x_{t}\) and previous hidden state \(h_{t-1}\) and applies \(f_{t}\) and _tanh_ activation functions through a pointwise multiplication operation as follows:  
\[i_{t}=\sigma(W_{i}[h_{t-1},x_{t}])+b_{i}, \tag{4}\]  
Figure 4: Architecture of LSTM.  
\[C_{t}=tanh(W_{c}[h_{t-1},x_{t}])+b_{o}. \tag{5}\]  
where \(W_{i}\) and \(W_{c}\) represent the weights of the input gate \(i_{t}\) and output gate \(o_{t}\), respectively. The \(b_{i}\) and \(b_{c}\) are the biases of the network, and \(C_{t}\) is the previous hidden cell state information. To update the information of the current cell state \(C_{t}^{{}^{\prime}}\), Equations (4) and (5) are summed through pointwise addition operation, given by Equation (6):  
\[C_{t}^{{}^{\prime}}=(f_{t}\times(C_{t}))+(i_{t}\times(C_{t})). \tag{6}\]  
Finally, in Equation (7), the output gate is determined. The output gate takes the current input \(x_{t}\) and previous hidden state \(h_{t-1}\) with the implication of the activation function \(\sigma\). The \(b_{o}\) is added as a bias to the output network.  
\[o_{t}=\sigma(W_{o}[h_{t-1},x_{t}])+b_{o}. \tag{7}\]  
The updated output gate \(o_{t}\) and the information from cell state \(C_{t}\) are used to perform the pointwise multiplication operation to get the next hidden state \(h_{t}\), given by Equation (8):  
\[h_{t}=\sigma(O_{t}\times tanh(C_{t}^{{}^{\prime}})). \tag{8}\]  
The optimal values are used to obtain better performance for the LSTM. These parameters play an important role in the performance of feature extraction. For better training, we set 50 neurons in each layer except the dense layer, which is a fully connected layer. The dropout is set to 20% in order to avoid the overfitting problem. The hyper-parameter values are shown in Table 4.