## 2 Background
### Ant Colony Algorithm  
The ant colony algorithm (ACO) [44] is a widely studied intelligent evolutionary algorithm. The ACO algorithm consists of two main steps: calculation of the state transfer probabilities and updating of the routing pheromone concentrations:  
(1) The node transfer probabilities:  
When the ant selects the next transfer node, it selects nodes by a roulette wheel after calculating the transfer probability of each node. To prevent ants from repeatedly selecting nodes, the selected node is added to the tabu list. (2) Updating of routing pheromone concentrations:  
The ACO algorithm mimics the behavior of ants that leave pheromones on each route on which they walk and introduces a pheromone volatilization mechanism to prevent the algorithm from falling into local optima. After all the ants have explored all the nodes, the pheromone concentration of each route is updated according to the pheromone update strategy. The ACO algorithm is used in various fields, such as assignment problems, scheduling problems, and resource allocation. [PERSON] [45] et al. proposed the product ant colony optimization (PACO) algorithm which combines the ACO algorithm with pro duction products to improve the productivity of textile manufacturers and balance the total operating time of all machines in each process. [PERSON] [46] et al. combined the binary ant colony algorithm (BACA) with other search optimization algorithms, such as hill climbing (HC) and simulated annealing (SA) to ensure the optimal Wireless Sensor Networks (WSNs) Coverage. The ACO algorithm has good performance and effectiveness in solving VRPs. [PERSON] [47] et al. employed an ACO to solve the feeder vehicle routing problem (FVRP). [PERSON] [48] et al. applied an improved ACO which used an innovative approach in updating the pheromone to solve the multidepot GVRP (MDGVRP). Current research on the improvement of the ACO for VRPs has focused on the state transfer strategy, pheromone update strategy, introduction of heuristic rules, individual learning strategy, and integration with other algorithms. [PERSON] [49] et al. proposed a polymorphic ACO with the adaptive state transfer strategy and the adaptive information updating strategy to improve the global search capability of the algorithm. [PERSON] [50] proposed a new improved ACO to solve the VRP with split pickup and delivery of multicategory goods. The algorithm adopts a TS operator that contains five neighborhood operators to improve the local search ability and introduces SA mechanisms to update the global pheromones. [PERSON] [51] et al. proposed an improved ACO which adopted a pseudo-random state transition rule and avoided the blindness search at early planning through the unequal allocation initial pheromone. [PERSON] [52] proposed the multiobjective ant colony system algorithm for epidemic situations (MOACS4E) to solve a new multiobjective VRP model for epidemic situations that considers the traditional travel cost and the prevention cost of the VRP in epidemic situations. MOACS4E adopted two ant colonies to optimize the problem and proposed a new pheromone fusion-based solution generation method. In addition to this, algorithms such as the simulated annealing algorithm [53] and the artificial fish swarm algorithm [54] were used to combine with the ACO. Deep reinforcement learning (DRL) is currently one of the important research directions in the field of machine learning. Common reinforcement learning algorithms are Deep Q-Networks (DQNs), Deep Deterministic Policy Gradient (DDPG), and Proximal Policy Optimization (PPO). DRL can realize end-to-end learning by directly interacting with the environment. With excellent feature extraction ability to realize the migration between different tasks, it can effectively overcome various perturbations in the system, as well as have good ability to solve high-dimensional and large-scale problems.