## 4 The Experiments
### Model Evaluation Procedure  
The model evaluation procedure comprised two steps. In the first step, the predictive model underwent parameter optimization, and in the second step, the quality of the best obtained predictive model was assessed using test datasets, which included different levels of damage to the valve plate as described in Section 3. The full data processing schema is presented in Figure 6. It starts with the initial standardization of the training data, where the standardization parameters (mean and variance) were determined (fitted) using the training set and then applied to the test sets. Then, the grid search procedure starts with a 10-fold stratified cross-validation test. The range of optimized parameters for each model is provided in Table 6. Subsequently, the best model obtained from parameter optimization was used to verify the accuracy of the test. Here, the test data described two different levels of valve plate damage. In this case, the standardization coefficients derived from the entire dataset were applied to the test sets. After this initial processing, predictions were made using the model trained in the Failure 1 dataset and applied to both test datasets. Figure 6: Model evaluation scheme. The next step in the model evaluation process was assessing attribute importance using the permutation method. Applying this method to both test sets and comparing the obtained results allows for the verification of the model in terms of the consistency of the results. Obtaining similar attribute importance for two independent test datasets confirms the validity of the model's quality. The analysis of attribute importance using the permutation method has certain limitations (the problem also applies to the MDI method). This limitation arises in the case of strongly correlated attributes. In such situations, replacing an attribute with a new one containing permuted values causes the other strongly correlated attribute to take over its role, making the decrease in prediction accuracy invisible. To circumvent the above-described problem, this work includes an analysis of the correlations between attributes. For attributes identified as strongly correlated, i.e., those with a correlation above 0.9, a PCA analysis was conducted, leaving only the first principal component for all correlated attributes. The analysis of attribute importance was then performed on the dataset with a reduced number of attributes, starting by retraining the model and then evaluating its performance on the test data. An important factor worth highlighting was the model evaluation method during cross-validation. The traditional approach to using cross-validation involves randomly splitting the data into individual folds. This procedure was abandoned in this study due to the nature of the data. The recorded data should be treated to some extent as a signal, which means that with randomized data and the step-wise nature of the pump load operation, we would obtain almost identical training and test data. This would consequently lead to an overestimation of the model's quality. Therefore, a linear partitioning approach was adopted in this work. In this schema a dataset is divided into k subsets (we used \(k=10\)), but not randomly, rather divided into k following subsets where each subset contains \(\frac{n}{k}\) samples ordered by time. In each of the \(k\) iterations, one of these subsets was used for testing and the remaining were used for training. In the calculations performed, simple accuracy was used as the measure of prediction performance, calculated as the number of correctly predicted cases relative to the total number of evaluated cases. The use of simple accuracy is justified because the datasets evaluated were balanced, with the class distribution being 47%, 49%, and 48%, respectively, for the Failure 1, Failure 2, and Failure 3 datasets. As a complement to the accuracy, a confusion matrix was used for evaluation, which allows for the assessment of false positive and false negative errors.