## 4 Optimization Method of MPDQN Based on DRL
### Solution to Optimization Problems  
For action tuple \((\Gamma,p_{\Gamma})\), a policy network is used to match them. \[p_{\Gamma}^{t}=x^{Q}\big{(}s^{t},\Gamma;\theta_{x}  
where the network weights are denoted as \(\theta_{Q}\). The process where the agent obtains the action with the highest action value is illustrated in Figure 2.  
The loss functions for the Q-Network and the network of the actor are defined as follows:  
\[L_{Q}\big{(}\theta_{Q}\big{)}=\underset{\big{(}s^{t},(\Gamma,p_{\Gamma}),r^{t},s ^{t+1}\big{)}\sim M}{\mathbb{E}}\left[\tfrac{1}{2}\big{(}y^{t}-Q\big{(}s^{t},( \Gamma,p_{\Gamma});\theta_{Q}\big{)}\big{)}^{2}\right]\, \tag{23}\]  
\[L_{x}(\theta_{x})=\underset{s^{t}\sim M}{\mathbb{E}}\left[-\sum\limits_{\Gamma =\{20,50,10\}}Q\big{(}s^{t},\Gamma,x^{Q}(s,\Gamma;\theta_{x});\theta_{Q}\big{)} \right]\, \tag{24}\]  
where \(y_{t}\) is defined as follows:  
\[y^{t}=r^{t}+\gamma\max_{\Gamma}Q\Big{(}s^{t+1},\Gamma^{t+1},x^{Q}\Big{(}s^{t+ 1};\theta_{x}\Big{)};\theta_{Q}\Big{)}. \tag{25}\]  
Finally, the network weights are updated using the learning rates \(lr_{x}\) and \(lr_{Q}\) for each network, aiming to approach the optimization objective. \[\theta_{Q}^{t+1}=\theta_{Q}^{t}-lr_{Q}\
abla_{\theta_{Q}}L_{Q}\big{(}\theta_{ Q}\big{)}, \tag{26}\]  
\[\theta_{x}^{t+1}=\theta_{x}^{t}-lr_{x}\
abla_{\theta_{x}}L_{x}(\theta_{x}), \tag{27}\]  
Next, we will describe the algorithm process in detail. First, the parameters of both networks are randomly initialized, and an experience replay buffer of size \(M\) is established. Then, the algorithm iterates over \(EP\) episodes. At the beginning of each episode, the system parameters are reset. The RSU selects initial action tuples based on the initial state and the network and observes the next state. Subsequently, the algorithm iterates from timeslot 1 to timeslot \(T\). For each timeslot \(t\), the RSU allocates actions to vehicles needing resource reallocation based on the current state. When selecting actions, the RSU either explores randomly with a certain probability or chooses the action with the maximum Q-value, introducing exploration noise to avoid local optima. Finally, the tuple \(\big{[}s^{t},\,(\Gamma^{t},p_{\Gamma}),r^{t},s^{t+1}\big{]}\) is stored in the experience replay buffer. When the number of tuples in the buffer exceeds the sample size \(B\), they will be employed to update the network parameters. The pseudocode is shown in Algorithm 1. Figure 2: The process of agent selecting actions in MPDQN. ```
Input:\(\gamma\), \(lr_{x}\), \(lr_{Q}\) Output: optimized \(\theta_{Q}\), \(\theta_{x}\)  Initialize the learning rates \(lr_{x}\) and \(lr_{Q}\), experience replay buffer \(M\), sample size \(B\), network weights \(\theta_{x}\) and \(\theta_{Q}\). forepisode from \(1\) to \(EP\)do  Reset the model parameters;  Observe the initial state \(s^{0}\) and output the initial action \(\left(\Gamma_{i}^{0},p_{\Gamma}\right)\);  Obtain the state \(s_{i}^{1}=\left[N_{i}^{1},d_{i}^{1},Rn_{i}^{1},RC_{i}^{0}\right]\). for slot \(t\) from \(1\) to \(T\)do  Obtain action tuples based on the policy network or through random exploration: \(a^{t}=\left\{ \begin{array}{cc}random\left(\Gamma^{t},p_{\Gamma}\right)+ \Delta&P_{ran}\\ \arg\max Q\left(s^{t},\Gamma^{t},x;\theta_{Q}\right)+\Delta&1-P_{ran}\end{array} \right.\)  Execute action \(\left(\Gamma_{i}^{t},p_{\Gamma}\right)\), observe state \(\mathbf{s}^{t+1}\) and reward \(r^{t}\);  Store transition tuple \(\left[s^{t},\left(\Gamma^{t},p_{\Gamma}\right),r^{t},s^{t+1}\right]\) to \(M\); ifnumber of tuples in \(M\) is larger than \(B\)then  Randomly sample \(B\) transitions tuples from \(M\);  Iterate networks according to Equation (23), (24), (26) and (27). end if  end for
```  
**Algorithm 1**Optimization algorithm for AoI and energy consumption based on MPDQN  
During the testing phase, there is no need to update the parameters. The actions are assigned to the required vehicles based on the optimized strategy in the training phase to carry out the test. The corresponding pseudocode is shown in Algorithm 2.