# Reinforcement Learning for Efficient Drone-Assisted Vehicle Routing
## Appendix A  
We consider a limited set \(K\) of vehicles. Furthermore, each vehicle \(k\in K\) holds an equal number of \(D\in\mathbb{Z}\) drones assigned to it. Each drone may travel a maximum range of \(E\) distance units per operation, where a drone-operation is characterized by a triple \((i,w,j)\) as follows: the drone is launched from a vehicle at a node \(i\in V_{L}\), delivers a parcel to \(w\in V_{N}\), and is retrieved by the vehicle from which it was launched at node \(j\in V_{R}\) (\(i\
eq w\
eq j\) and \(i\
eq j\)). We use the following decision variables for modeling the VRPD:  
* \(x^{k}_{ij}\): \(\forall i,j\in V,k\in K-\) Binary variables that state whether edge \((i,j)\) belongs to the route of vehicle \(k\). * \(y^{k}_{inj}\): \(\forall i,w,j\in V,k\in K-\) Binary variables that specify whether drone delivery \((i,w,j)\) is performed by a drone associated with vehicle \(k\). * \(p^{k}_{ij}\): \(\forall i,j\in V,k\in K-\) Binary variables that indicate whether node \(i\) is served before (but not necessarily consecutive to) node \(j\) in the route of vehicle \(k\). * \(u^{k}_{i}\): \(\forall i\in V,k\in K-\) Integer variables with a lower bound of \(0\) that state the position of node \(i\) in the route of vehicle \(k\), if customer \(i\) is served by vehicle \(k\). - Binary variables that specify whether the drone delivery \((a,w,e)\) is performed by a drone associated with vehicle \(k\) and node \(a\) precedes node \(i\) while node \(e\) follows node \(j\) in the route of vehicle \(k\). * Continuous variables with a lower bound of 0 that indicate the earliest time at which customer \(i\) is served by either the vehicle \(k\) or a drone associated with vehicle \(k\). If \(i\) is used as a retrieval node, \(a^{k}_{i}\) is the earliest time at which both the (latest) drone and the vehicle have arrived at node \(i\). The MILP formulation of the VRPD is given in (A1)-(A19), where (A1) is the objective function and the constraints are given by (A2)-(A19):  
\[\min\tau\] (A1)  
\[\text{s.t. }\tau\geq a^{k}_{n+1}\quad\forall k\in K\] (A2)  
\[\sum_{j\in\mathbb{V}_{N}}x^{k}_{0j}-\sum_{i\in\mathbb{V}_{N}}x^{k}_{i,n+1}=0 \quad\forall k\in K\] (A3)  
\[\sum_{i\in\mathbb{V}_{L}}x^{k}_{i\hbar}-\sum_{j\in\mathbb{V}_{R}}x^{k}_{\hbar j }=0\quad\forall k\in K,h\in V_{N}\] (A4)  
\[\sum_{j\in\mathbb{V}_{N}}x^{k}_{0j}\leq 1\quad\forall k\in K\] (A5)  
\[\sum_{j\in\mathbb{V}_{N}}x^{k}_{ij}+\sum_{l\in\mathbb{V}_{L}}\sum_{m\in \mathbb{V}_{R}}y^{k}_{ljm}=1\quad\forall j\in V_{N}\] (A6)  
\[2y^{k}_{i\
j}\leq\sum_{h\in V_{L}}x^{k}_{i\hbar}+\sum_{l\in\mathbb{V}_{N}}x^{k }_{lj}\quad\forall k\in K,i\in V_{N},w\in V_{N},j\in V_{R}\] (A7)  
\[y^{k}_{0\
j}\leq\sum_{h\in V_{L}}x^{k}_{\hbar j}\quad\forall k\in K,w\in V_{N},j\in V_{R}\] (A8)  
\[u^{k}_{i}-u^{k}_{j}\leq(n+1)(1-x^{k}_{ij})\quad\forall k\in K,i\in V_{L},j\in V _{R}\] (A9)  
\[u^{k}_{j}-u^{k}_{i}\geq 1-(n+1)(1-y^{k}_{i\
j})\quad\forall k\in K,i\in V_{L},w \in V_{N},j\in V_{R}\] (A10)  
\[\text{s.t. }\tau\geq a^{k}_{n+1}\quad\forall k\in K\]0 (A11)  
\[\text{s.t. }\tau\geq a^{k}_{n+1}\quad\forall k\in K\]1 (A12)  
\[\text{s.t. }\tau\geq a^{k}_{n+1}\quad\forall k\in K\]2 (A13)  
\[\text{s.t. }\tau\geq a^{k}_{n+1}\quad\forall k\in K\]3 (A14)  
\[\text{s.t. }\tau\geq a^{k}_{n+1}\quad\forall k\in K\]4 (A15)  
\[\text{s.t. }\tau\geq a^{k}_{n+1}\quad\forall k\in K\]5 (A16)  
\[\text{s.t. }\tau\geq a^{k}_{n+1}\quad\forall k\in K\]6 (A17)\[\text{s.t. }\tau\geq a^{k}_{n+1}\quad\forall k\in K\]7  
\[\text{s.t. }\tau\geq a^{k}_{n+1}\quad\forall k\in K\]8 \[\text{s.t. }\tau\geq a^{k}_{n+1}\quad\forall k\in K\]9 \[\sum_{j\in\mathbb{V}_{N}}x^{k}_{0j}-\sum_{i\in\mathbb{V}_{N}}x^{k}_{i,n+1}=0 \quad\forall k\in K\]0  
The constraints (110)-(112) ensure the flow preservation, synchronization between vehicles and drones, and maximum range of drone operations, as well as satisfying other operational restrictions. The objective function minimizes the arrival time \(a_{n+1}^{k}\), which represents the time when the last vehicle or its associated drones return to the depot.