# Addressing the Algorithm Selection Problem through an Attention-Based Meta-Learner Approach
## 3 Solution Model
### Multi-Head Attention for Problem-Algorithm Mapping  
This work relies on the multi-head attention mechanism in the transformer [9], explicitly using it within an attention-based neural network as a meta-learner. The rationale behind using multi-head attention lies, in part, in the type of features that can be computed to represent the structure of a VRPTW instance and how this representation can be transformed to encode it better. We do this by following an approach similar to that of [PERSON] et al. [40] and perform an intermediate feature-to-feature mapping step, with the main difference being that we use multi-head attention in an attention block to do this. This step aims to generate a new and improved representation for a given VRPTW instance. We express this as a three-step process where an input vector \(x_{i}\), containing \(d_{x}\) features that characterize a given VRPTW instance, is incrementally transformed into a new representation \(h_{i}^{(\ell)}\), where \(\ell\in 1,\ldots,n\), and \(n\) corresponds to the number of attention blocks used. The number of attention blocks determines the number of times the intermediate feature-to-feature mapping step is performed so that mapping from a particular problem instance to a point in the algorithm space can be learned. As shown by [PERSON] et al. [41], multiple features can be computed to represent different aspects of a particular problem instance. For example, we can mention generic ones, such as those related to the graph topology, as well as specific ones related to the location of the depot, clients and the demand, among others. With the proposed three-step process, we aim to provide an approach to better extract the patterns encoded by these features. Additionally, we strive to avoid saturating a single attention mechanism due to using multiple types of features (geometrical or topological, for example) by employing multiple heads and thus splitting the cognitive load. Moreover, the decision to use multiple attention heads instead of a single one was mainly taken due to the added capability enabling joint attention to information from different representation subspaces at different positions from the input vector \(x_{i}\in\mathbb{R}^{d_{x}}\)[10]. The motivation for this was the variety of features used to characterize the VRPTW instances. The goal was to leverage the approach to facilitate specialization behavior among the attention heads, wherein certain heads develop specialized functions within the model [11]. The three-step process is described as follows:* Initially, for the first step in the input layer, a new representation of \(d_{h}\) dimensions of the input vector \(x_{i}\) is learned via a linear transformation \(h_{i}^{(0)}=Wx_{i}+b\), where the weight matrix \(W\in\mathbb{R}^{d_{x}\times d_{h}}\) and the bias \(b\in\mathbb{R}^{d_{h}}\) correspond to learnable parameters of the network [38]. In this work, \(d_{x}\) and \(d_{h}\) are set to 26 and 196, respectively. The value of \(d_{x}\) was defined empirically and corresponds to the 26 features described in Section 3.3 used to characterize each VRPTW instance. In contrast, \(d_{h}\) was defined via Bayesian optimization as the optimal value for this hyperparameter, as described in Section 4. The idea of using a linear transformation to generate a new initial representation of the input vector \(x_{i}\) was adopted since it has been previously used in attention-based networks with promising results [9; 38; 39]. By improving the representation of the input data, this approach can enhance the effectiveness of the updates made by the attention block in the next step. * In the second step, which corresponds to the intermediate feature-to-feature mapping process [40], this new and improved representation \(h_{i}^{(0)}\)--which characterizes the given VRPTW instance--is then fed to the attention block.