## 4 Proposed PPO-Based Method
### MHPPO-Based Solution Strategy  
Figure 2 illustrates the overall architecture of the proposed MHPPO method. The method consists of two main networks: the actor network and the critic network. The actor network generates both continuous and discrete action strategies. The continuous actor network, parameterized by \(\theta_{c}\), outputs the mean and variance which are used to construct a Gaussian distribution over the continuous action space. The discrete actor network, parameterized by \(\theta_{d}\), outputs a probability distribution over the discrete action space. The critic network, parameterized by \(\phi\), evaluates the value of the state. The agent interacts with the environment in the following way: the actor network outputs the probability distribution for discrete actions and the mean and variance for continuous actions to construct a Gaussian distribution based on the input state \(s_{t}\) and generates specific actions \(a_{t}\) by sampling. After interacting with the environment, a reward \(r_{t}\) will received by the agent, followed by a transition to the next state \(s_{t+1}\). So, the data \(\left\{s_{t},a_{t},r_{t},s_{t+1}\right\}\) are obtained and stored in the replay buffer. In the process of action selection, due to the distance limitation of inter-vehicle communication, the action space of unloading objects may contain some targets that cannot be  
Figure 2: The framework of the MHPPO method. unloaded. To address the problem, an action masking mechanism is introduced. The specific realization idea is to set the sampling probability of invalid actions to zero in the action selection stage. Specifically, if the distance between the task vehicle and the \(n\)-th service vehicle exceeds the predefined communication range, the sampling probability of selecting this service vehicle in the action space for offloading decisions is set to 0. With the action masking mechanism applied, the discrete action probability distribution is transformed into a mask categorical distribution, where invalid actions are masked out by assigning them zero probability. In the update phase, a random mini-batch of \(M\)samples is extracted from the experience replay buffer and the actor and critic networks perform parameter updates by minimizing their respective objective functions. After several iterations, the optimized policy is applied to the environment interaction and new data are continuously collected to further improve performance. In order to continuously optimize the actor networks, their parameters \(\theta_{c}\) and \(\theta_{d}\) need to be updated by continuous training. In this paper, we adopt the PPO-Clip approach, which relies on specialized tailoring of the objective function to reduce the differences between the old and new policies. The strategies are updated by multi-step stochastic gradient descent to maximize the objective function. The continuous actor network's objective function is  
\[L_{\mathrm{c}}^{\mathrm{clip}}(\theta_{c})=\mathbb{E}_{t}\bigg{[}\min\big{(}r _{t}^{\mathrm{c}}(\theta_{c})\hat{A}_{t},\mathrm{clip}(r_{t}^{\mathrm{c}}( \theta_{c}),1-\epsilon,1+\epsilon)\hat{A}_{t}\big{)}\bigg{]}, \tag{19}\]  
where \(\hat{A}_{t}\) is the advantage function and \(r_{t}^{\mathrm{c}}(\theta_{c})\) is the ratio coefficient between the new continuous strategy and the old continuous strategy, which is represented as  
\[r_{t}^{\mathrm{c}}(\theta_{c})=\frac{\pi_{\theta_{c}}(a_{t}^{\mathrm{c}}|s_{t })}{\pi_{\theta_{c}(\mathrm{old})}(a_{t}^{\mathrm{c}}|s_{t})}. \tag{20}\]  
Similarly, the objective function of the discrete actor network is  
\[L_{\mathrm{d}}^{\mathrm{clip}}(\theta_{d})=\mathbb{E}_{t}\bigg{[}\min\big{(}r _{t}^{\mathrm{d}}(\theta_{\mathrm{d}})\hat{A}_{t},\mathrm{clip}(r_{t}^{\mathrm{ d}}(\theta_{d}),1-\epsilon,1+\epsilon)\hat{A}_{t}\big{)}\bigg{]}, \tag{21}\]  
where \(r_{t}^{\mathrm{d}}(\theta_{\mathrm{d}})\) is the ratio coefficient between the new discrete strategy and the old discrete strategy.