# IoT System for Detecting the Condition of Rotating Machines Based on Acoustic Signals
## 4 Experimental Results
### Preliminary Results without Hyperparameter Tuning
The total number of testing samples is 295 (64 from class 1, 82 from class 2, 74 from class 3, and 75 from class 4), and the resulting confusion matrix is shown in Table 3. Since the diagonal entries correspond to the correct classification, the accuracy of the algorithm (the number of correctly classified samples divided by a total number of samples) is \(acc=0.96\). It is important to verify that the classification procedure is able to correctly generalize the results and that the high accuracy achieved in this example is not accidental. For that reason, the training and testing of the neural network was repeated in order to validate the behavior of the algorithm. All the data from the previous example (the total of 933 samples) was again randomly divided into training (70%) and testing (30%) sets and this partitioning was conducted four times, simulating a form of cross-validation verification. The results are presented using boxplot visualization in Figure 14, where each train-test partitioning is shown in different color. For each class, classification results are presented with median value (denoted with circles) and 25 th and 75 th percentile (denoted with horizontal lines). It is evident that high classification accuracy is achieved with all data partitions which shows that overfitting has not occurred and the algorithm performs properly. The nature of state detection problem and the requirements of the experts from maintenance teams indicates that the classification errors are not equally significant. Classes 1, 2, and 3 correspond to the proper functioning of the machine, indicating how soon the machine maintenance will be required. This helps about planning regular maintenance, but does not signal the need for immediate response and replacement of worn parts. Class 4, on the other hand, alarms that the maintenance needs to be performed. For that reason, it is of crucial importance to determine reliably if machine operates in class 4 regime, while misclassification between classes 1, 2, and 3 does not substantially affect maintenance performing. This means that it is worth to examine the algorithm behavior with respect to class 4, apart from accuracy which is a metric that describes overall functioning of the algorithm. If class 4 is denoted as positive, and classes 1, 2, and 3 as negative, then the new confusion matrix is shown in Table 4, where TP means true positive estimation, TN--true negative, FP--false positive and FN--false negative estimation. Like all unbalanced classification problems, accuracy is not enough to assess the performance of the algorithm. Metrics such as precision (out of all predicted positives, how many of them are actually positive), recall \begin{table}
\begin{tabular}{c c c c c c} \hline \hline  & & \multicolumn{4}{c}{**PREDICTED**} \\ \cline{3-6}  & & **1** & **2** & **3** & **4** \\ \hline \multirow{4}{*}{**ACTUAL**} & **1** & 64 & 0 & 0 & 0 \\ \cline{2-6}  & **2** & 0 & 82 & 0 & 0 \\ \cline{1-1} \cline{2-6}  & **3** & 0 & 3 & 64 & 7 \\ \cline{1-1} \cline{2-6}  & **4** & 0 & 0 & 2 & 73 \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Confusion matrix for fixed hyperparameter classification. (out of all actual positives, how many are predicted as a positive), and F1 score (harmonic mean of precision and recall) are far more informative:  
\[\text{Precision}=\frac{TP}{TP+FP}, \tag{5}\]  
\[\text{Recall}=\frac{TP}{TP+FN}, \tag{6}\]  
\[\text{F1 score}\ =\frac{2}{\frac{1}{\text{Recall}}+\frac{1}{\text{Precision}}}. \tag{7}\]  
In the experiment given in Table 4 these values are: precision = 0.91, recall = 0.97, and F1 score = 0.94.