## 6 Experiments and Discussion
### Discussion
However, in a homogeneous instance space, such as in our use case, we observed that configurations lacked the ability to be strictly dominant over a given region and presented significant performance variations at a granular level. In a further analysis of the training data for a given pair of configurations (C1, C2) in a given portfolio, we observed that the distribution of the number of timed-out replications was somewhat similar between the configurations (Figure 14). The color regions have different thicknesses, and the areas of transition between color regions show different levels of overlap, but the distribution of the regions follows a similar fashion. Nevertheless, the most concerning aspect is perhaps that both configurations have a large number of instances in common (Figure 15), with the areas where three replications time out being those of most concern. We can see in Figure 14 that, in that same area, configuration C1 has a large number of iterations with two, one or even zero timed-out replications. That entire set of instances where both C1 and C2 time out for the three replications has no impact during the training of the RF for the pair of configurations (C1, C2) (since the associated weight is zero). Nonetheless, that same RF may cast a vote that would prefer one configuration over the other, which may result in wrongly selecting a configuration with very poor performance and thus a very large error (in terms of PAR-10). In general, we observed significant performance overlap within the space of each configuration (cf. Figure 14) and between configurations in the portfolio (cf. Figure 15). In fact, a configuration from the portfolio may be dominant over a certain region but present the worst performance among those configurations in the portfolio for sparsed instances within their dominant region. In such cases, selecting the dominant configuration instead of the best configuration in the portfolio can lead to differences (percentual errors) of up to 2900% in PAR-10. In short, the RF methodology proposed in Hydra seems to have trouble dealing with the strong performance overlap observed in the homogenous instance space. Figure 14: Number of timed-out instances for a given pair, C1 (**a**) and C2 (**b**), of configurations in a given portfolio. Each axis represents the total number of product units of each product type. Furthermore, we can verify the impact of the inference mechanism of Hydra on performance by replacing the RF methodology with the nearest neighbor approach used in the CBR method. Since the CBR method makes use of all training data for inference instead of a summary of the data (as the RFs do in Hydra), small local groupings can be more easily identified. To investigate this further, we build a case base that only contains the portfolio data used to train the RFs in Hydra, and use the same similarity-based approach to inference used in the CBR method; we will refer to this method as the CBR portfolio method. Note that the aforementioned CBR portfolio method is in essence the same as that in in [26] but in the context of algorithm configuration instead of algorithm selection. Figure 16 shows the performance in terms of the average PAR-10 reduction observed when using the configurations predicted by the CBR portfolio method for PIAC instead of [PERSON]'s default configuration. We observed a significant improvement in performance when replacing the RF prediction method from Hydra with the nearest neighbor approach for prediction used in the CBR-portfolio method. The effects attributed to the robustness and length of the portfolios observed in Hydra remain present for the CBR portfolio method, namely the large performance variance (Figures 9 and 17) and the poor performance observed in Case Base 3. However, even for Case Base 3, the nearest neighbor approach to inference used in CBR approaches results in significantly better performance than the approach used in Hydra. Figure 15: Instances with the same number of time outs for a given pair of configurations, C1 and C2, in a given portfolio.