# Addressing the Algorithm Selection Problem through an Attention-Based Meta-Learner Approach
## 2 Background and Related Work
All the meta-learners considered for comparison share the three main components considered for this work: machine learning, meta-heuristics, and combinatorial optimization problems (which in this work happens to be the VRPTW). Moreover, we opted for such meta-learners because the literature consistently presents them as baseline methods for this problem. As attention mechanisms have become more relevant, they have been used in tasks such as text classification [36] and image-based analysis [37]. However, although these mechanisms have been implemented differently, they all share the same principles, based on which [PERSON] et al. [10] identified four main categories. The first category concerns the softness of attention, which includes soft and hard attention [10]. Soft attention is deterministic and differentiable and computes a weighted average of all keys to build the context vector, while hard attention samples keys to do this [10]. This category also includes global attention, similar to soft attention, and local attention, a blend of hard and soft attention that considers only a subset of the input at a time. The second category is related to the form of the input feature. It is divided into item-wise, which processes explicit items or a sequence of items, or location-wise, which processes a feature map or an image at each decoder step [10]. The third category concerns the input representation, which includes distinctive, multi-input, self-, and hierarchical attention [10]. The fourth and final category refers to how the output is represented, such as a probability distribution, sequence of tokens, or fixed-length vector [10]. Moreover, [PERSON] et al. [10] also identified seven types of score functions used within the attention mechanisms to determine how to compute the relevance or similarity between the keys and queries to generate the attention weights. The proposed attention-based meta-learner has a similar architecture to the encoder block used in transformers [9], which employs a scaled multiplicative score function within the attention mechanism [10]. Transformers represent a network architecture for processing sequential data without necessarily processing it in order. Instead, transformers leverage self-attention to look at the whole sequence and give more importance to the most relevant input data segments. This network architecture was initially developed for natural language processing (NLP) [9]. Nevertheless, it has also been proven helpful in other domains, including combinatorial optimization and, specifically, for solving VRPs using modified transformer architectures to learn heuristics and construct solutions incrementally [38; 39]. Nonetheless, as opposed to [PERSON] et al. [38] and [PERSON] et al. [39], this work will not use the proposed attention-based meta-learner to solve specific VRPTW instances directly. Instead, it will select a specific algorithm that will, on average, generate the best solution for a given problem instance. Therefore, to the best of the authors' knowledge, no previous work has proposed using an attention-based neural network as a meta-learner for selecting, from a given set of meta-heuristics, the one that best solves a particular VRPTW instance, as proposed in this work.