# AI-Driven Predictive Maintenance for Workforce and Service Optimization in the Automotive Sector
## 4 Methodology
### Model Training
#### 4.2.7 Performance Metrics  
Accuracy is a measure for evaluating classification models. Accuracy is the ratio of correct predictions made by the model to all predictions. It is a facile method most used to measure model success [40]. Accuracy is not an efficient success metric in models trained using unbalanced data. The accuracy is calculated by the third formula below. \[Accuracy=\frac{(TN+TP)}{TN+TP+FN+FP} \tag{18}\]  
Error rate is calculated by dividing the number of misclassified samples by the sum of the samples [41]. The error rate is calculated by the formula below. \[Error\ Rate=\frac{(FN+FP)}{TN+TP+FN+FP} \tag{19}\]  
Recall is a measure of how many positive identifications are correct [42]. The recall is calculated by the formula below. \[Recall=\frac{(TP)}{TP+FN} \tag{20}\]  
Precision shows how many of the values we estimated as positive are actually positive [42]. The precision is calculated by the formula below. \[Precision=\frac{(TP)}{TP+FP} \tag{21}\]  
F1-score value shows the harmonic average of precision and recall values [40]. The F1-score is calculated by the formula below. \[F1\ Score=2*\frac{precision*recall}{precision+recall} \tag{22}\]  
The receiver operating characteristic (ROC) curve is a graphical representation used to assess the performance of a binary classifier across various threshold settings. It illustrates the trade-off between the true positive rate (TPR) and the false positive rate (FPR), offering insight into the model's discriminative ability irrespective of the decision threshold. The relevant metrics are defined as follows:  
\[\text{True Positive Rate }(\text{TPR})=\frac{(\text{TP})}{\text{TP}+\text{FN}} \tag{23}\]  
\[\text{False Positive Rate }(\text{FPR})=\frac{(\text{FP})}{\text{FP}+\text{TN}} \tag{24}\]  
The Area Under the ROC Curve (AUC) provides a single scalar value summarizing the model's overall performance. An AUC of 0.5 indicates no discriminative power (i.e., random guessing), while an AUC of 1.0 signifies perfect classification ability. The confusion matrix is a fundamental evaluation tool in classification tasks, providing a comprehensive breakdown of predicted versus actual class labels. It is particularly useful for identifying not only the overall performance of a model but also the nature of specific classification errors [36]. The matrix comprises four key components:  
True Positives (TP): correctly predicted positive cases;  
True Negatives (TN): correctly predicted negative cases;  
False Positives (FP): negative cases incorrectly predicted as positive;  
False Negatives (FN): positive cases incorrectly predicted as negative. This structure enables the calculation of a variety of performance metrics and allows for class-specific analysis. For instance, a high number of false negatives in predictive maintenance may indicate critical missed service opportunities, while excessive false positives could reflect inefficient use of service resources. As such, the confusion matrix is an essential diagnostic tool, particularly in imbalanced classification settings where global accuracy may be misleading. In addition to conventional evaluation metrics, domain-specific indicators offer a more practical and context-aware assessment of model utility, particularly in the automotive after-sales domain. These metrics are tailored to reflect the operational and business consequences of model predictions, which may not be adequately captured by generic measures such as accuracy or F1-score. For instance, in predictive maintenance applications, a false negative--i.e., failing to predict a customer's upcoming service visit--may result in a missed opportunity for revenue, underutilization of service capacity, or customer dissatisfaction. Conversely, a false positive may lead to over-preparation, misallocated technician effort, or unnecessary inventory usage.