# Smart Grid Theft Detection Based on Hybrid Multi-Time Scale Neural Network
## 4 Experimental Analyses
### Performance Comparison before and after Dataset Enhancement  
To test the effectiveness of the method proposed in this paper to balance the dataset using Time GAN-generated data, we selected commonly used methods such as ROS, RUS, SMOTE, and ADASYN, as well as the pre-processed raw data for comparison. Additionally, to predict the presence of electricity theft by electricity consumers, a hybrid multi-time scale neural network model was trained with the same parameters. At the end of the experiment, we plotted PR and ROC curves based on the experimental results, as shown in Figure 16, to evaluate the performance of the model. This helped us to determine whether the Time GAN method significantly improved the balance of the dataset and thus improved the prediction accuracy of the model. The experiment followed the following steps.  
Figure 16: _Cont._  
1. Collect and pre-process the data of electricity consumers, including the original data and the data processed by ROS, RUS, SMOTE, ADASYN, and Time GAN.
2. Design hybrid multi-time scale neural network models and train each dataset using the same parameters.
3. Evaluate the performance of each model separately and plot PR and ROC curves to determine whether the Time GAN approach could improve the balance of the dataset.  
With this experimental design, we could more accurately evaluate the effectiveness of the Time GAN method and determine whether it was a viable data-balancing method.  
In Figure 16, it was observed that the PR-AUC value of the raw data was only 0.5338 and the ROC-AUC value was only 0.5795, indicating inadequate performance in distinguishing normal and electricity theft customers. The RUS and ROS techniques improved the impact of data imbalance to some extent, but the test results were still relatively poor. The SMOTE technique achieved a PR-AUC value of 0.7915 and ROC-AUC value of 0.8527, but it could not overcome the problem of easy distribution marginalization, resulting in poorer results. ADASYN, with a PR-AUC value of 0.7832 and ROC-AUC value of 0.9358, achieved further improvements but was susceptible to outlier effects, which affected the balance of the sample. In addition, Time GAN achieved a great improvement, with a PR-AUC value of 0.9469 and ROC-AUC value of 0.9684, because it not only used the raw data to gradually supervise the loss, but also let the model capture the distribution in the time series data. Table 2 displays the detailed results.  
\begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Models** & **Precision** & **Recall** & **FPR** & **PR-AUC** & **ROC-AUC** \\ \hline Raw data & 57.11\% & 43.12\% & 42.36\% & 0.5338 & 0.5795 \\ RUS & 71.47\% & 71.39\% & 35.29\% & 0.6439 & 0.7058 \\ ROS & 82.07\% & 81.72\% & 24.46\% & 0.7490 & 0.8156 \\ SMOTE & 86.21\% & 85.34\% & 18.26\% & 0.7915 & 0.8527 \\ ADASYN & 93.47\% & 92.08\% & 9.37\% & 0.7832 & 0.9358 \\ Time GAN & 96.64\% & 96.87\% & 3.77\% & 0.9469 & 0.9684 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Comparison of data balancing performance.  
Figure 16: PR and ROC curves generated by various balanced data methods. (**a**) PR curves; (**b**) ROC curves.  
From Table 2, it was observed that the method of generating electricity theft customer data to balance the dataset by Time GAN improved our test results by 39.53% over the original data in precision, 53.75% in recall, 38.59% in FPR reduction, 0.4131 in PR-AUC, and 0.3889 in ROC-AUC, which greatly improved the detection accuracy and reduced the false detection rate. These results qualitatively illustrated the effectiveness of the Time GAN-generated data utilized in this study and confirmed the proposed model's ability to accurately detect electricity theft customers.