# Reinforcement Learning for Efficient Drone-Assisted Vehicle Routing
## 5 Computational Studies
### Training and Evaluation Configurations  
**Data Generation. ** To demonstrate the computational efficiency of the proposed approach, results are reported on the well-known benchmark dataset first proposed in [28]. In particular, ref. [28] randomly generated customer locations using a uniform distribution within the [1, 100] \(\times\) [1, 100] area, while the depot location is generated with a uniform distribution within the [0, 1] \(\times\) [0, 1] range, ensuring it always resides in a corner. Other studies have widely used the same benchmark dataset, including [9, 30]. Throughout the experiments, it is assumed that the drone's speed is double that of the truck, thus \(\alpha\) = 2, consistent with existing literature. **Gap. ** To evaluate the quality of solutions produced by different approaches to solve the VRPD, the GAP is measured and defined as:  
\[\text{Gap}_{i}=\frac{Z_{i}-Z_{i}^{*}}{Z_{i}^{*}}\times 100\% \tag{11}\]  
where \(Z_{i}\) is the cost of a solution method for each instance \(i\) and \(Z_{i}^{*}\) is the cost of the best-performing solution method among all methods compared for instance \(i\). In Tables 1-3, the average GAP for each setting of the problem as the mean of GAPs for 10 instances of the problem is reported. **Baselines. ** The efficiency of the proposed neural architecture is analyzed using [9] as a benchmark. In [9], the authors proposed the MILP formulation of the problem, solving the small instances exactly with and without lazy constraints, denoted as Solver and Solver Lazy, respectively in Tables 1-3. Beyond the exact method, the authors suggested a more practical strategy involving VNS and TS methods. This approach entails incorporating drones into delivery routes and a divide-and-conquer strategy. VNS is a method of exploring various possibilities by altering the neighbourhood structure, while TS is another approach that learns from previous solutions to avoid retracing steps. The results of VNS and TS-based heuristics are denoted as Heuristic in Tables 1-3. The computational time and costs for each instance of [9] have been kindly shared by the authors, where the best solution obtained by their heuristic in five runs and its corresponding computational time are reported. **Decoding Strategies. ** Three approaches were used to extract solutions from the trained model. The first method, referred to as \"greedy\", involves consistently choosing nodes with the highest visitation probability at each time step. In the second method, \"sampling\", multiple solutions are independently generated from the trained model with a sampling size of 4800. Subsequently, the solution with minimal cost among the samples is identified, and its results are reported. In the third method, \"Ensemble\", multiple models saved throughout the training process are utilized. In particular, the models' weights in the interval [400k, \(\ldots\), 1000k] are selected with an increment of 100k. Solutions are sampled using a sample size of 4800 for each model, and the best results are reported. **Computational Resources and Hyperparameters. ** Computers with the following specifications were utilized: Intel Core i9 12900K and PCI-E 24576 Mb Palit RTX 3090 GamingPro OC, GeForce RTX 3090 video card were used in two of the systems, while a Core i9 12900K with RTX 4090 24 GB was employed on another computer for training. An Intel Core i9 12900K and PCI-E 24576 Mb Palit RTX 3090 GamingPro OC computer were used to conduct testing and report the computational time. The same hyperparameters as those in [13] were used to train the neural networks, including 256 as the hidden dimension for embedding, 0.0001 as the learning rate, and 1 million iterations for training.