# Addressing the Algorithm Selection Problem through an Attention-Based Meta-Learner Approach
## 4 Experiments and Results
Table 2 shows the results corresponding to the values for the accuracy, precision, recall, and F1-score, all computed on the 14.2% of the dataset previously described that was reserved exclusively for reporting the final results of the selectors' evaluation. The results displayed in Table 2 show that the AM is consistently selecting, more often than the other five selectors, the specific algorithm that generates, on average, the lowest cost solution given a particular VRPTW instance. Moreover, compared to the other algorithm selectors, the AM shows a remarkable improvement in reducing the proportion of false positives when performing a selection. Furthermore, a remarkable result is the significant difference in how often the AM selects the best algorithm for a given instance concerning the MLP. Overall, the AM selector shows an improvement of roughly 14% compared to the MLP, which confirms the idea that exploring more novel neural network architectures improves the learning of the mappings from problem instances to the algorithm space. This improvement is possible due to the \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Method** & **Hyperparameter** & **Ranges of Values** & **Optimal Value** \\ \hline \multirow{4}{*}{AM} & Epochs & \((1,1000)\) & \(1000\) \\  & Learning rate & \((0.0,0.01)\) & \(0.00043197\) \\  & Dropout & \((0.1,0.9)\) & \(0.12383\) \\  & Embedding size & \(\mathrm{d}_{h}\) (\(16,256\)) & \(196\) \\  & Attention heads & \((2,12)\) & \(8\) \\ \hline \multirow{4}{*}{MLP} & Epochs & \((1,1000)\) & \(108\) \\  & Learning rate & \((0.0,0.1)\) & \(0.0048487\) \\  & L2 regularizartion & \((0.0001,1000)\) & \(0.30201379\) \\  & Hidden layer size & \((1,1000)\) & \(222\) \\ \hline \multirow{2}{*}{KNN} & Neighbors & \((1,15)\) & \(1\) \\  & Leaf size & \((1,100)\) & \(42\) \\ \hline \multirow{2}{*}{SR} & L2 regularization & \((0.0001,1000)\) & \(876.51276\) \\  & Maximum iterations & \((1,1000)\) & \(895\) \\ \hline \multirow{2}{*}{SVM} & Kernel & Linear, Radial Basis Function, \\  & Kernel & Sigmoid, Polynomial & Radial Basis Function \\  & L2 regularization & \((0.0001,1000)\) & \(957.93164\) \\ \hline DT & Maximum depth & \((1,100)\) & \(95\) \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Hyperparameter ranges of values and optimal values obtained from Bayesian optimization. \begin{table}
\begin{tabular}{c c c c c c c} \hline \hline
**Metric** & **AM** & **SVM** & **KNN** & **DT** & **MLP** & **SR** \\ \hline Accuracy & **87.38** & 86.72 & 84.32 & 81.98 & 72.79 & 70.72 \\ Precision & **89.03** & 87.56 & 85.20 & 83.23 & 73.93 & 71.80 \\ Recall & **87.33** & 86.67 & 83.87 & 82.00 & 72.67 & 70.77 \\ F1 & **86.90** & 86.63 & 83.76 & 81.46 & 72.43 & 70.63 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Selector results. Final values for accuracy, precision, recall and F1-score, with the best value for each metric marked in bold. enhanced pattern-extraction process and further leveraging of the meta-features used to characterize the problem instances. Overall, based on the results shown in Table 2, the AM selector performs better for all the analyzed metrics in selecting the most suitable algorithm to solve a particular problem instance. The latter supports the idea that leveraging the multi-head attention mechanism is a good approach for generating an improved representation of a problem instance based on its underlying structure and, thus, better learning the problem-algorithm mapping. This improvement in performance for selecting the best algorithm for a given VRPTW instance was further validated via an F-test [66] employing the implementation provided by [PERSON] [67] used for similar studies, focusing on the accuracy as the main point of comparison.