# IoT System for Detecting the Condition of Rotating Machines Based on Acoustic Signals
## 4 Experimental Results
### Comparative Analysis with Hyperparameter Tuning
\begin{table}
\begin{tabular}{c c c c c c c c} \hline \hline  & & **Mill A, P1** & **Mill A, P2** & **Mill A, P3** & **Mill B** & **Mill C** \\ \hline  & ACCURACY & 0.96 & 0.96 & 0.86 & 0.96 & 0.93 \\ \cline{2-7} DEFAULT & PRECISION & 0.91 & 0.99 & 0.93 & 0.97 & 1 \\ \cline{2-7} HYPERPARAMETERS & RECALL & 0.97 & 0.97 & 0.72 & 1 & 0.97 \\ \cline{2-7}  & **F1 SCORE** & **0.94** & **0.98** & **0.81** & **0.98** & **0.98** \\ \hline  & ACCURACY & 0.99 & 0.99 & 0.97 & 0.97 & 0.96 \\ \cline{2-7} Tuned hyperparameters & PRECISION & 1 & 0.99 & 1 & 1 & 1 \\ \cline{2-7}  & RECALL & 0.97 & 1 & 0.93 & 0.98 & 1 \\ \cline{2-7}  & **F1 SCORE** & **0.98** & **0.99** & **0.96** & **0.99** & **1** \\ \hline \hline \end{tabular}
\end{table} 
Table 7: Comparative results of algorithm performance. \begin{table}
\begin{tabular}{c c c c c c c c c} \hline \hline  & \multicolumn{4}{c}{**Training Samples per Class**} & \multicolumn{4}{c}{**Testing Samples per Class**} \\ \cline{2-9}  & **1** & **2** & **3** & **4** & **1** & **2** & **3** & **4** \\ \hline
**MILL A, POSITION 1** & 134 & 190 & 188 & 126 & 64 & 82 & 74 & 75 \\ \hline
**MILL A, POSITION 2** & 116 & 180 & 162 & 124 & 67 & 85 & 83 & 76 \\ \hline
**MILL A, POSITION 3** & 142 & 203 & 201 & 151 & 65 & 89 & 73 & 76 \\ \hline
**MILL B** & 162 & 214 & 204 & 145 & 64 & 84 & 83 & 71 \\ \hline
**MILL C** & 147 & 192 & 168 & 150 & 72 & 81 & 92 & 85 \\ \hline \hline \end{tabular}
\end{table} 
Table 5: Number of testing and training samples for different machines. The results are interesting for several reasons. First, the algorithm with tuned hyperparameters always has a better performance on all metrics than the algorithm with default parameters, which is expected. The tuning itself lasts between several minutes and an hour, depending on the size of the training dataset. Since it should be done only once for each machine, it is recommended to always perform it. Next, even with default hyperparameters the algorithm shows solid performance. F1 score is taken as a reference parameter because it considers both recall and precision, and in all but one case default parameters show satisfactory results. Another phenomenon that requires further discussion is the significant difference in algorithm performance on the same machine (Mill A) when recordings are taken from different positions. This is especially noticeable with default hyperparameters. An algorithm trained on signals from Position 2 gives almost perfect classification results, while signals from Position 3 have significantly worse performance. This is due to the nature of acoustic signals and their susceptibility to ambient noise. Position 2, near the back of the mill, is furthest from the motor (which is the loudest component of this machine), while Position 3, at the front, is nearest to the motor. This difference in the amount of surrounding noise directly affects classification and indicates that the user should carefully choose the recording position so that the amount of noise is minimized. On the other hand, even when it is impossible to avoid the noise, these results show that algorithms with tuned hyperparameters will be able to obtain satisfactory performance. In the example in Table 7, F1 score has increased from 0.81 for default hyperparameters to 0.96 when hyperparameters are tuned.