## 4 Proposed Algorithm
### Local Search Strategy
#### 4.5.1 Environment Setup  
The evolutionary algorithm guides the reinforcement learning for searching. The following is the process of establishing the DDQN environment and directing its learning direction:  
(1) Neural Network Construction  
The input to the problem in this paper is a three-dimensional vector which has low dimensionality, so this paper constructs a function-fitting neural network which is structured as a fully connected network with three hidden layers. The input layer has three nodes. Each hidden layer has 40 nodes.  
The activation functions are chosen to be linear transfer function (purelin), saturated linear transfer function (satlin), and positive linear transfer function (poslin). Multilayer networks use different activation functions for each layer to allow the neural network to approximate arbitrary functions.  
The network chooses a multilayer neural network training function: the variable learning rate backpropagation algorithm (traingdx). It uses momentum optimization to accelerate convergence, and improves stability and accuracy through an adaptive learning rate to make the gradient descent smoother.  
The network structure is shown in Figure 6.  
After the initial construction of the neural network is completed, the initial parameter training of the neural network is required. In order to improve the effectiveness of parameter training and the rate of neural net approximation, the Pareto frontier is chosen as the input for initial training in this paper. The Pareto frontier, named \(Hpareto\), is selected in the solution set G obtained by nondominated ordering; meanwhile the set with the largest nondominated rank, named \(Lpareto\), is selected. The mean square error between the estimated Q value and the corresponding reward of \(Hpareto\) is then used as the loss function to train the initial evaluation network, as shown in Equation (44).  
\[\mathrm{L}(\theta_{t})=\mathrm{E}_{(s,a,r,s^{\prime})\sim\mathrm{U}(Hpareto)} [(r(s,a)-Q(s,a;\ \theta_{t}))^{2}] \tag{44}\]  
\(r(s,a)\) represents the reward corresponding to Hpareto in the \((s,a)\) environment. \(Q(s,a;\ \theta_{t})\) represents the estimated Q value obtained by the evaluation network in the \((s,a)\) environment.  
(2) Reward Return Design  
This paper studies the multiobjective VRP problem where the reward value is related to the pheromone between the routes, the heuristic function, and the advance time. The rewards are shown in Equation (45):  
\[r_{t}=\tau_{ij}^{a}(t)\times\eta_{ij}^{\beta}(t)\times(1/w_{j}(t_{j}))^{\delta} \tag{45}\]  
This equation simultaneously takes full account of multiobjectives, so that when an agent completes an action choice that causes a change in the state, its corresponding reward also changes. This design allows for the effective mapping of the multiple objectives of the study to the rewards of deep reinforcement learning.  
For the VRP problem, the optimal solution of the subpaths can easily cause the algorithm to fall into a local optimum, so this paper proposes an overall learning rate  
Figure 6: Construction of the network.  
called \(Alpha_{target}\). After the action selection is finished to obtain a complete path for all the workstation requirements, the IGD [57] metrics of this path with the two path sets of \(Hpareto\) and \(Lpareto\) are calculated to obtain \(IGD\_Lpareto\) and \(IGD\_Hpareto\).Then \(Alpha_{target}\) is calculated according to Equation (46), and the \(r_{t}\) is updated according to Equation (47). \(Alpha_{target}\) quantifies the advantages and disadvantages of the obtained path as a ratio of distances from \(Hpareto\) and \(Lpareto\), in order to correct the rewards of the various subpaths of this path, so that paths with smaller IGDs have a higher advantage, whereas paths with larger IGDs have a larger disadvantage.  
\[Alpha_{target}=IGD\_Lpareto/IGD\_Hpareto \tag{46}\]  
\[r_{t}=Alpha_{target}*r_{t} \tag{47}\]  
The structure of the DDQN algorithm is shown in Figure 7.