# A General Framework Based on Machine Learning for
## 4 Experiments and Results
### Challenging the Oracle  
In a more challenging analysis, we compared the Machine-Learning-based algorithm selectors against the Oracle (a method that always chooses the best among the three heuristics for each instance). The average number of consistency checks per instance required by the Oracle is 2,996,917.37, which is still lower than any of the studied methods. In fact, both MRL and MNN require around 9% additional consistency checks compared to the Oracle to solve the whole test set. This means that, although the Machine-Learning-based algorithm selectors are capable solvers, there is still room for improvement. Since the Oracle is a method that would achieve a 100% accuracy on the test set, it seems reasonable to analyze the accuracy of the Machine-Learning-based algorithm selectors as a way to estimate how close such methods are to the best possible decision. Table 2 shows the accuracy of all the methods under analysis on the test set. Although heuristics always apply the same criteria and no selection process is involved, the accuracy reflects the fraction of instances where each heuristic is the best choice. Additionally, it is important to remark that we do not present the accuracy of GA and SA since their application model does not allow us to calculate it properly. GA and SA are allowed to dynamically change the heuristic as the search takes place. Then, such methods do not rely on the accuracy but on their capacity to modify the search so it reduces its cost.  
As expected, the good performance of the Machine-Learning-based algorithm selectors, in terms of consistency checks, is supported by a large accuracy of their corresponding classifiers. It is interesting to observe that, a small difference in the accuracy can translate into a huge change in the number of consistency checks required to solve the set. For example, The difference in the accuracies of MNN and MDJ is only 1.875%. However, this represents around 25% additional consistency checks when solving the test set. This finding suggests that, a few poor decisions can tremendously impact the overall performance of the algorithm selectors.  
Additional to the accuracy of the methods, in Table 2, we included the \(p\)-value of a paired two-sided \(t\)-test that compares each method against the Oracle, based on the number of consistency checks per instance. By using 5% significance, the statistical evidence does not allow us to state that any of the generated Machine-Learning-based algorithm selectors is different in performance to the Oracle. However, the statistical evidence suggests that DOM, KAPPA, WDEG, GA, and SA are not equal to the Oracle in performance. Thus, we can conclude that all the Machine-Learning-based algorithm selectors produced as part of this work are statistically equivalent to the Oracle for the instances in the test set.  
\begin{table}
\begin{tabular}{l c c} \hline \hline
**Method** & **Accuracy (\%)** & \(p\)**-Value** \\ \hline DOM & 27.500 & 4.809\(\times 10^{-7}\) \\ KAPPA & 57.500 & 9.370\(\times 10^{-4}\) \\ WDEG & 15.000 & 7.283\(\times 10^{-13}\) \\ \hline GA & N/A & 3.176\(\times 10^{-7}\) \\ SA & N/A & 2.679\(\times 10^{-9}\) \\ \hline MLR & 76.875 & 0.771068 \\ MNN & 76.875 & 0.771068 \\ MDF & 71.875 & 0.269258 \\ MDJ & 75.000 & 0.345435 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Accuracy of the different methods.  
Additionally, we found that there is an improvement concerning individual use of the heuristics and two metaheuristic-based algorithm selectors.