# Smart Grid Theft Detection Based on Hybrid Multi-Time Scale Neural Network
## 4 Experimental Analyses
### Ablation Experiments of the Model  
To further validate the effectiveness of the model, the study performed the following ablation experiments: (1) use only Bi-LSTM networks to extract data features on a daily time scale, and then feed them into the CatBoost classifier, denoted as Test 1; (2) use only ResNet to extract data features on a weekly time scale, and then feed them into the CatBoost classifier, denoted as Test 2; (3) use only AlexNet to extract data features on a monthly time scale, and then feed them into the CatBoost classifier, denoted as Test 3. The above models were trained separately, and the test data were fed into the models to draw PR and ROC curves, as shown in Figure 18.  
\begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Models** & **Precision** & **Recall** & **FPR** & **PR-AUC** & **ROC-AUC** \\ \hline SVM & 61.07\% & 69.42\% & 17.75\% & 0.3743 & 0.6144 \\ RF & 71.58\% & 71.77\% & 16.67\% & 0.4703 & 0.7053 \\ DT & 74.06\% & 73.81\% & 15.97\% & 0.5824 & 0.7444 \\ Wide \& Deep CNN & 80.79\% & 81.13\% & 13.15\% & 0.5724 & 0.8123 \\ LSTM \& MLP & 87.41\% & 87.36\% & 15.63\% & 0.6719 & 0.8789 \\ SOSTLink \& BiGRU & 93.42\% & 93.14\% & 10.34\% & 0.7795 & 0.9332 \\ Bi-RNN & 85.38\% & 85.47\% & 21.24\% & 0.7799 & 0.8488 \\ CNN \& LSTM & 88.02\% & 87.11\% & 17.56\% & 0.8146 & 0.8733 \\ LSTM \& GRU & 74.98\% & 73.24\% & 32.18\% & 0.6656 & 0.7328 \\ Proposed models & 96.64\% & 96.87\% & 3.77\% & 0.9469 & 0.9684 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Comparison of the performances of the proposed models.  
In Figure 18, it was clear that using only the Bi-LSTM network to extract data features on a daily time scale yielded a PR-AUC value of 0.7765 and ROC-AUC value of 0.8434. Using only ResNet to extract data features on a weekly time scale yielded a PR-AUC value of 0.7974 and ROC-AUC value of 0.8631. Using only AlexNet to extract data features on a monthly time scale yielded a PR-AUC value of 0.8988 and ROC-AUC value of 0.9363. Although each model achieved a certain feature extraction effect, it was notable that that fusing three-time scales of customer electricity consumption features could more comprehensively reflect the patterns of customer electricity consumption and achieve a better classification effect. Table 4 displays the detailed results.  
Figure 18: PR and ROC curves of the model ablation tests. (**a**) PR curves; (**b**) ROC curves.  
From Table 4, it was clear after testing that the model fusing three-time scales was more comprehensive than a single time scale model in responding to the features of normal and electricity theft customers. In terms of the accuracy rate, the model in this paper improved the accuracy by 12.38% compared to Test 1, 10.11% compared to Test 2, and 2.39% compared to Test 3. In terms of the false detection rate, the model in this paper reduced the false detection rate by 18.16% compared to Test 1, 15.99% compared to Test 2, and 5.44% compared to Test 3. By comparison, it was striking that using AlexNet to extract data features on a monthly time scale was the best, using ResNet to extract data features on a weekly time scale was the second best, and using Bi-LSTM to extract data features on a daily time scale was the worst.