# Application of Predictive Maintenance Concepts Using Artificial Intelligence Tools
## 3 Application Example
### Test Set Behavior  
The validation set has been used, so far, to fine-tune the models and respective hyper-parameters, in order to seek better performance. It is now important to check how the models behave in the test set. Although, in a real case, it is advisable to evaluate only the model that is intended to be implemented [8], this section presents the results obtained for the evaluation of the two best models (in the validation set), in this case, Random Forest and Artificial Neural Networks, with min-max scaling normalization.  
Tables 14 and 15 show the values obtained for Precision, Recall and F1 Score, for the Random Forest and Artificial Neural Network model, respectively, in the validation and test sets.  
As expected, there is a generalized decrease in performance of the evaluation metrics in the test set. Still, the results remain satisfactory. As previously mentioned, in predictive maintenance, as a general rule, the most important is the number of real failures that the model is capable of predicting, that is, the value of the model's Recall parameter [28]. This parameter becomes even more important as the consequences of false negatives, that is, true failures that the model was unable to predict, exceed the consequences of false positives, that is, a false prediction of a failure [33; 34].  
\begin{table}
\begin{tabular}{c c c c c c} \hline  & **None** & **comp1** & **comp2** & **comp3** & **comp4** \\ \hline Precision & 0.9997 & 0.9451 & 0.9917 & 1.0000 & 0.9520 \\ \hline Conj. Develop.Recall & 0.9998 & 0.9337 & 0.9972 & 0.9167 & 0.9954 \\ \hline F1 Score & 0.9997 & 0.9394 & 0.9945 & 0.9565 & 0.9732 \\ \hline Precision & 0.9990 & 0.9030 & 0.9941 & 0.9858 & 0.9725 \\ \hline Conj. Test Recall & 0.9995 & 0.8425 & 0.9853 & 0.9392 & 0.9833 \\ \hline F1 Score & 0.9993 & 0.8717 & 0.9897 & 0.9619 & 0.9779 \\ \hline \end{tabular}
\end{table}
Table 15: Performance for the Artificial Neural Network model in the validation and test sets, with 100 hidden layers (hidden layers = 100) and min-max scaling normalization.  
\begin{table}
\begin{tabular}{c c c c c c} \hline  & **None** & **comp1** & **comp2** & **comp3** & **comp4** \\ \hline Precision & 0.9997 & 0.9244 & 0.9916 & 1.0000 & 0.9812 \\ \hline Conj. Develop. Recall & 0.9999 & 0.9578 & 0.9861 & 0.9514 & 0.9543 \\ \hline F1 Score & 0.9998 & 0.9408 & 0.9889 & 0.9751 & 0.9676 \\ \hline Precision & 0.9988 & 0.9718 & 0.9711 & 0.9855 & 0.9830 \\ \hline Conj. Test Recall & 0.9998 & 0.8150 & 0.9882 & 0.9189 & 0.9611 \\ \hline F1 Score & 0.9993 & 0.8865 & 0.9796 & 0.9510 & 0.9719 \\ \hline \end{tabular}
\end{table}
Table 14: Performance for the Random Forest model in the validation and test sets, with ‘n_estimators = 70.  
For both models, there is a drop in the value of Recall (and, consequently, of F1 Score) to values below 90% for component 1 in the test set. In the present application, the four components were considered to be of equal importance.  
In a real application, where it may be possible to know more information about each one of them (such as cost, importance in the process, location in the equipment, ease of replacement), the analysis may involve trying to optimize certain metrics that are considered to be of greater relevance.