## 4 Proposed Algorithm
### Local Search Strategy  
The DDQN [56] is an off-policy DRL algorithm which is improved from the DQN. The DDQN decouples the two steps of selection of the target Q value action and computation of the target Q value to address the problem that the DQN potentially leads to maximization bias, which results in overestimation.  
The decoupling process is as follows:  
The DDQN constructs two networks: the evaluation network and the target network. The structure of the evaluation network and the target network are identical, but the weighting parameters are different.  
First, the DDQN selects the action corresponding to the maximum Q value by the evaluation network, as shown in Equation (40):  
\[a_{\max}=argmax_{a^{\prime}}Q\big{(}s_{t+1},a^{\prime};\;\theta^{\prime}_{t} \big{)} \tag{40}\]  
Then, the target network calculates the target Q value as shown in Equation (41):  
\[y_{t}=r_{t}+\gamma Q(s_{t+1},a_{max};\;\theta_{t}) \tag{41}\]Combining the states, the target Q value of is shown in Equation (42):  
\[y_{t}=\left\{\begin{array}{ll}r_{t}&\text{if $s^{\prime}$ is terminal}\\ r_{t}+\gamma Q(s_{t+1},argmax_{a^{\prime}}Q(s_{t+1},a^{\prime};\ \theta^{\prime}_{t});\ \theta_{t})&\text{otherwise}\end{array}\right. \tag{42}\]  
As shown in the Equation (43), the DDQN adopts the mean square error (MSE) of the estimated Q value and the target Q value as the loss function, and updates the evaluation network by gradient descent backpropagation.  
\[L(\theta_{t})=E[(y_{t}-Q(s_{t},a;\ \theta_{t}))^{2}] \tag{43}\]  
The DDQN learns past experience offline by building a replay buffer. After \(N^{-}\) rounds of the replay buffer sampling, the weight parameters of the evaluation network are copied to the target network so as to realize the model self-learning.  
To solve the MOPs, the DDQN adopts the Pareto solution set to train the network and guide the learning direction (details are in Sections 4.5.1 and 4.5.2). The pseudo-code is shown in the table below (Algorithm 2):  
```
Input: \(D:empty\ replay\ buffer;\ \theta:\ network\ parameters;\ \theta^{-}:\ copy\ of\ \theta;\) \(N_{t}:replay\ buffer\ maximum\ size;\ N_{b}:training\ batch size;\ N^{-}:target\ network\ replacement\ freq;\) \(G:the\ set\ of\ Pareto\ solutions\) that have been obtained; \(G^{H}\):the Pareto frontier in \(G\) For\(i\) from \(1\) to \(\text{size}\ (G^{H})\) for\(j\) from \(1\) to \(\text{size}\ (G^{H}_{i})\)  Take \(\left(s_{j},a_{j}\right)\) and \(r_{j}\). Do a gradient descent step with loss \(\left\|r_{j}\left(s_{j},a_{j}\right)-Q\left(s_{j},a_{j};\ \theta\right)\right\|^{2}\) to train \(\theta\).  end for For episode\(e\in\{1,2,\dots M\}\)do  Initialize frame sequence x\(\leftarrow\)(0) for\(t\in\{0,1,\dots\}\)do  Set state s \(\leftarrow\) x, sample action \(a\)\(\sim\pi_{B}\)  Sample next frame \(x^{\prime}\) from environment \(\varepsilon\) given \((s,a)\) and receive \(r\), and append \(x^{\prime}\) to x  Compare the \(\varepsilon\) to \(G\) to obtain the overall learning rate \(Alpha_{target}\), set \(r^{\prime}\gets r*Alpha_{target}\) if\(|x|\)>\(N_{f}\)then delete oldest frame \(x_{t_{\text{train}}}\) from x end  Set \(s^{\prime}\leftarrow\) x, and add \((s,a,r^{\prime},s^{\prime})\) to D,  Replacing the oldest tuple if\(|D|\)>\(N_{t}\)  Sample a minibatch of \(N_{b}\) tuples \((s,a,r^{\prime},s^{\prime})\)\(\sim\) Unif\((D)\)  Construct target values, one for each of the \(N_{b}\) tuples \(a^{max}(s^{\prime};\ \theta)=argmax_{a^{\prime}}Q(s^{\prime},a^{\prime};\ \theta)\); \(y_{j}=\left\{\begin{array}{ll}&if\ s^{\prime}\ is terminal\\ r+\gamma Q(s^{\prime},a^{max}(s^{\prime};\ \theta);\ \theta^{-}),&otherwise.\end{array}\right.\) ;  Do a gradient descent step with loss \(\left\|y_{i}-Q(s,a;\ \theta)\right\|^{2}\)  Set \(\theta^{-}\leftarrow\theta\) every \(N^{-}\) steps  end for  end for  end for
```  
**Algorithm 2**Double DQN in MOPs