## 6 Experiments and Discussion
### Evaluation of the CBR Method for PIAC  
Figure 4 shows the average reduction in PAR-10 obtained during evaluation when using the CBR method instead of the default configuration, for different case base sizes the (number of unique problem instances in the case base) and for different tuning time budgets (case base types). We can see how the CBR can lead to reductions of up to 17.2% in PAR-10 when using a case base with 800 unique instances and a tuning time budget per instance of 600 s (Case Base 3). Surprisingly, a case base with as few as 100 instances can lead to reductions of up to 16% for the same tuning budget. In fact, increasing the case base size seems to have a smaller effect on performance than increasing the tuning time budget (Table 3). However, smaller case base sizes seem to result in higher performance variance and thus less robust predictions (Figure 5). Performance gains are more limited from the point of view of CPU time reductions. In the best case, the selection of a configuration via the CBR method reduces CPU time by as much as 3.3% on average (Figure 6). Once again, increasing the case base size seems to have a smaller effect on performance than increasing the tuning time budget (Table 3). The drastic difference in performance observed between the PAR-10 and CPU time can be better understood when we look at the average reduction in the number of timed-out replications (Figure 7). The CBR method results in a reduced number of timed-out replications per instance (Figure 7). The reduction can be as much as 19.78% and falls in line with the magnitude of the average reduction in PAR-10 (Figure 4). Just as the case for the reduction in PAR-10, increasing the case base size seems to have a smaller effect on performance than increasing the tuning time budget (Table 3). In our experiments we found that the difference in performance between the CBR method and Gurobi's default configuration is significant for all case base types and sizes (Table 2). Once again, good performance is obtained with as little as 100 instances (for Case Base 3), however, the large variance observed during the evaluation may be an indication of a lack of robustness. An increase in model robustness seems to be linked to an increase in case base size. As we can observe, the coefficient of variation (CV) of the performance estimate (the average reduction in PAR-10) seems to decrease monotonically as the case base increases (Figure 5). The simplicity of the CBR approach, however, allows us to easily augment the model by simply drawing new instances and performing tuning on each of them, without the need for re-training of the predictive model. This, in combination with standard cross-validation approaches (e.g., K-fold cross-validation), could help practitioners in the better planning of their sampling activities via a better understanding of the accuracy-robustness trade-off. Figure 5: Coefficient of variation of the average reduction in PAR-10 obtained when using the configurations obtained with the CBR method instead of [PERSON]’s default configuration. Results are presented per case base type and per case base size. Figure 6: Average reduction in CPU time observed when using the configurations obtained with the CBR method instead of [PERSON]’s default configuration. Results are presented per case base type and per case base size. The dotted lines connect the mean values. Figure 7: Average reduction in the number of timed-out replications when using the configurations obtained with the CBR method instead of [PERSON]’s default configuration. Results are presented per case base type and per case base size. The dotted lines connect the mean values. The application of the CBR method for PIAC in the process planning use case for RAS seems to indicate that significant performance improvements can be achieved with such a simple approach in the face of highly homogenous instance spaces.