## 3 Experimentation and Results

Table 8: Load recognition results obtained by the feed-forward ANN where it has been well trained and validated on the class-imbalanced training data. Figure 10: Training trajectory of the feed-forward, multilayer (1-23-12) ANN. As seen in Table 8, the 989 NIALM data instances make up a class-imbalanced dataset. Accuracy alone is not sufficient for evaluating a recognizer trained from class-imbalanced data, where in each class there may exist a significant disparity between positives (status: On) and negatives (status: Off). As a result, \(F_{1}\) score, the harmonic mean of precision and recall, is conducted and used to address the class-imbalanced problem (in order to fully evaluate the performance of the feed-forward ANN in terms of load recognition). Various metrics in addition to \(F_{1}\) score have been developed. Table 9 shows TPRs vs. FPRs, for ROC curves, obtained by the well trained and tested feed-forward, multilayer ANN applied on the class-imbalanced test data for NIALM, where one ROC curve can be shown per class (the maximum AUC is 1, which corresponds to a perfect recognizer as all positives above all negatives--100% sensitivity of no false negatives and 100% specificity of no false positives--are ranked). The authors of [24] used two different types of ANNs, deep NNs by autoencoder and long short-term memory (LSTM), as load recognizers to perform load disaggregation for the UK-DALE dataset. Comparison among the autoencoder, the LSTM and the presented feed-forward, multilayer ANN for load recognition is shown in Table 10. As shown in Tables 8-10, the feed-forward, multilayer ANN that gives similar performance, in load recognition, against autoencoder outperforms the LSTM. The load recognizer of the presented NIALM is a shallow neural network, not a deep neural work. As reported in this section, the presented feed-forward, multilayer ANN is able to discriminate the targeted electrical appliances from the house environment well. \begin{table}
\begin{tabular}{c c c c c c c c c c} \hline \hline \multirow{3}{*}{\begin{tabular}{c} **Electrical** \\ **Appliance** \\ \end{tabular} } & \multicolumn{2}{c}{\
\begin{tabular}{c} **The Presented ANN-Based** \\ **NIALM** \\ \end{tabular} } & \multicolumn{4}{c}{
\begin{tabular}{c} **A Deep ANN-Based Load** \\ **Quasgeregation by** **LSTM**[24] \\ \end{tabular} } \\ \cline{2-10}  & **Precision** & **Recall** & \(\mathbf{F_{1}}\) **Score** & **Precision** & **Recall** & \(\mathbf{F_{1}}\) **Score** & **Precision** & **Recall** & \(\mathbf{F_{1}}\) **Score** \\ \hline
**fridge** & 0.88 & 0.90 & 0.89 \({}^{1}\) & 0.85 & 0.88 & 0.87 & 0.72 & 0.77 & 0.74 \\ \hline
**dishwasher** & 0.89 & 0.86 & 0.88 \({}^{1}\) & 0.29 & 0.99 & 0.44 & 0.04 & 0.87 & 0.08 \\ \hline
**kettle** & 0.33 & 0.60 & 0.43 & 1.00 & 0.87 & 0.93 & 0.96 & 0.91 & 0.93 \\ \hline
**Avg.** & 0.70 & 0.79 & 0.73 & 0.71 & 0.91 & 0.75 & 0.57 & 0.85 & 0.58 \\ \hline \hline \end{tabular} \({}^{1}\) Across its all states.  
\end{table} 
Table 10: Comparison among the autoencoder, the LSTM and the presented feed-forward, multilayer ANN for load recognition. \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Class** & **FPR** & **TPR** & **AUC** \\ \hline
1 & 0.17 & 0.82 & 0.83 \\ \hline
2 & 0.02 & 0.98 & 0.98 \\ \hline
3 & 0.16 & 0.62 & 0.73 \\ \hline
4 & 0.01 & 0.91 & 0.95 \\ \hline
5 & 0.08 & 0.84 & 0.88 \\ \hline
6 & 0.00 & 0.90 & 0.95 \\ \hline
7 & 0.04 & 0.82 & 0.89 \\ \hline
8 & 0.02 & 0.60 & 0.79 \\ \hline
9 & 0.10 & 0.93 & 0.92 \\ \hline
10 & 0.02 & 0.71 & 0.84 \\ \hline
11 & 0.05 & 0.82 & 0.88 \\ \hline
12 & 0.02 & nan & nan \\ \hline \hline \end{tabular}
\end{table} 
Table 9: Obtained TPRs vs. FPRs, for ROC curves, from the well trained and tested feed-forward, multilayer ANN applied on the class-imbalanced test data; for the 12 classes the AUCs obtained are also shown.