## 4 Results and Discussion
### Evaluation Results  
Within the CRISP-DM methodology, the evaluation phase holds significant importance as it allows for a comprehensive assessment of the quality and effectiveness of the developed machine learning models. As already mentioned, to address the challenge of predicting pallet collapse, this study employed Decision Tree and Random Forest due to their proven effectiveness in handling classification tasks, particularly in scenarios with class imbalances. Decision Tree was chosen for its simplicity and interpretability, making it easier to understand the decision-making process. Random Forest, an ensemble method, was selected for its ability to reduce overfitting and improve predictive accuracy by averaging the results of multiple trees. During this phase, the primary objective was to assess the models' generalization capabilities, specifically their accuracy in predicting unseen data. To achieve this, 20% of the dataset was reserved during the initial split as a dedicated test set. This reserved set was used to evaluate the models' performance on data that was not seen during training. By employing this separate test dataset, the evaluation phase ensured an unbiased assessment of each model's predictive performance and its ability to generalize beyond the training data. The evaluation of the developed models, which addressed a binary classification problem, involved constructing a confusion matrix for each model. The confusion matrix provides a detailed view of a model's accuracy, allowing for the calculation of various performance metrics. Given the objective of identifying pallets at risk of collapsing during transportation, minimizing false negatives is critical. At the same time, it is important to \begin{table}
\begin{tabular}{c c} \hline \hline
**Hyperparameter** & **Range of Values** \\ \hline _splitrule_ & Information Gain or Gini index \\ _nttree_ & 100 to 120, with increments of 1 (one) \\ _mtry_ & 1 to 4, with increments of 1 (one) \\ _nodesize_ & 2 to 20, with increments of 1 (one) \\ \hline \hline \end{tabular}
\end{table} 
Table 10: Random Forest model’s tuned hyperparameters and correspondent values. avoid excessive false positives to prevent unnecessary alerts. The \(F1\)-_score_ was chosen as the primary metric for optimization during the hyperparameter tuning process. The \(F1\)-_score_ metric, which represents the harmonic mean of _Precision_ and _Recall_, offers a balanced evaluation of the model's performance:  
\[F1\text{-}score=\frac{2\times\text{Precision}\times\text{Recall}}{\text{ Precision}+\text{Recall}}, \tag{5}\]  
where _Precision_ and _Recall_ are calculated as follows:  
\[Precision=\frac{TP}{TP+FP}, \tag{6}\]  
\[Recall=\frac{TP}{TP+FN}, \tag{7}\]  
and \(TP\) stands for true positives, \(TN\) for true negatives, \(FP\) for false positives, and \(FN\) for false negatives. To facilitate a comparison among the developed models, Table 11 was built, presenting a comprehensive overview of each model's performance. This table allows for an analysis of each model's strengths and weaknesses based on the _Precision_, _Recall_, and \(F1\)-_score_ values. To aid in interpreting Table 11, it is important to note the naming convention used: for example, \"DecisionTrees/525\" refers to the Decision Tree model with a class imbalance, where 75% indicates the prevalence of the majority class (\"No\") and 25% represents the minority class (\"Yes\"). This convention helps identify each model's specific configuration and clarifies the class distribution used during training. The confusion matrix for the model with the best performance, RandomForests6040, based on the \(F1\)-_score_ results, is presented in Table 12. This matrix shows the results for the 1206 examples in the test set, facilitating the calculation of the metrics previously discussed. Tables 13 and 14 present the optimal values of the hyperparameters that resulted in the maximum \(F1\)-_score_ values during the tuning process for the Decision Tree and Random Forest models, respectively.