## 6 Experiments and Discussion
### Discussion  
During our experiments, we observed that the CBR method outperformed Hydra in the prediction of better configurations to solve each unique problem instance. These results fall in line with the findings of [18]. However, and perhaps surprisingly, Hydra's performance during our experiments was not only inferior to that observed with the CBR method, but it also failed to improve upon the performance obtained when using the default configuration. Furthermore, Hydra's performance did not improve consistently with an increased tuning time budget (as was the case for the CBR method). As such, to explain the poor performance of Hydra, one may need to take a look at _(i)_ the characteristics of the portfolios that are built for each tuning time budget and _(ii)_ the size of the instance set, and the way Hydra predicts a configuration. A close look into the resulting portfolios shows, for Case Base 3 (the case base with the worst performance), that the portfolio building process results in smaller portfolios (Figure 12) than those in the case of Case Base 1 and 2. Furthermore, for Case Base 3, the portfolio size decreases as the case base size increases. This contrasts Case Base 1 and 2, where we can see an increase in portfolio size for larger case bases. As such, the smaller portfolios built in Case Base 3 may not be able to capture the full complexity of the problem space, which may, at least in part, explain the poor performance of Case Base 3. Considering the way in which the portfolios are built (see Section 3.2), it is possible that the most prevalent configurations used to build the portfolios are all present and have similar performance (with respect to one another) in the same area(s) of the input space. Consequently, the portfolio building process converges before the complexity of the input space can be captured. Case Base 1 and 2 both result in portfolios that are of similar lengths across the board (Figure 12), which may help explain their similar performance (Figure 8). However, as we can see in Figure 13, the portfolio building process yields portfolios with more variance in terms of their composition for Case Base 2. In fact, for almost all case base sizes, each replication of the portfolio building process yields a unique portfolio. For Case Base 1 and 2, on the other hand, the composition of the portfolios becomes more uniform as the case base size increases. The larger performance variance observed in  
Figure 11: Average reduction in the number of timed-out replications when using the configurations obtained with Hydra instead of Gurobi’s default configuration. Results are presented per case base type and per case base size. The doted lines connect the mean values. Case Base 2 (Figure 9) may then be attributed to the varying composition of the resulting portfolios across the 10 replications. Figure 12: Average length of the portfolios used by Hydra. The doted lines connect the mean values. Figure 13: Frequency of unique portfolios used by Hydra among all 10 replications. Note that a unique portfolio can be fully contained within another unique larger portfolio. The other aspect that may explain the poor performance of Hydra in the present use case is the way Hydra makes use of the portfolio data to predict a new configuration. Recall that inference in Hydra considers a voting system in which a set of pairwise (i.e., for each pair of configurations in the portfolio) weighted cost-sensitive RFs predicts/votes in favor of one configuration out of each pair of configurations. As such, the configuration with the most votes is selected. Furthermore, for each trained RF, a weight proportional to the performance delta observed between each configuration pair is assigned to the training data. This methodology has proven effective in dealing with (highly) heterogeneous instance spaces where the configurations found in the portfolios are strictly dominant in significant regions of the instance space.