## 4 Multi-Objective Deep Reinforcement Learning Based on the PPO Algorithm
### PPO Algorithm Application  
The application of the PPO algorithm is directed at optimizing the policy function using a neural network. This approach is particularly adapted to navigating the complexities of the non-convex optimization challenges characteristic of intricate MEC scenarios. PPO distinguishes itself by balancing the exploration of innovative strategies with the exploitation of established, efficacious actions facilitated by its distinct objective function and policy update methodology. This allows for effective handling of the non-convex optimization problem prevalent in complex MEC scenarios. The PPO algorithm, with its objective function and policy update mechanism, ensures a balance between the exploration of new strategies and the exploitation of known effective actions [33]. To evaluate the versatility of various reinforcement learning (RL) algorithms and to fine-tune hyperparameters for broader applicability, we decoupled the environmental module from the neural network. This separation enables the environmental module to focus exclusively on generating environmental feedback and computing rewards, a critical step in enhancing the scheme's generality, as illustrated in Figure 3. The environmental module's interfaces, namely reset and step, are integral to the RL process. The reset interface initializes or refreshes the environment to a baseline state, providing the initial observation post-training initiation or episode completion. In contrast, the step interface progresses the environment in response to the agent's actions, delivering subsequent observations, instant rewards, termination flags, and additional context-specific data. This delineation of responsibilities ensures a coherent and systematic interaction between the PPO algorithms and the environment module, promoting an efficient learning cycle predicated on action-derived feedback. Within the neural network module, the PPO algorithm updates the neural network parameters based on state returns by sampling historical offloading data. This process involves two distinct neural networks: the actor network, which is responsible for updating the current policy by determining action probabilities, and the critic network, which evaluates the policy's expected returns. The functions of the actor and critic neural networks are to update and evaluate the current strategy, respectively. Upon receiving the current environment state as input, the actor network employs the neural network's parameters, \(\theta\), to ascertain the likelihood of undertaking a particular action within state s. Concurrently, the critic network computes the value of the current policy, effectively estimating the expected returns. This dual-structured approach enables the PPO algorithm to harmoniously balance the need for exploration, which involves experimenting with novel actions with the necessity of exploitation or optimizing known effective actions. Through this iterative interaction, the PPO algorithm learns and hones strategies, progressively navigating towards an optimal policy. The objective function of the policy, denoted as \(J\) (\(\theta\)) (Equation (30)), integrates with \(\hat{A}_{t}\), the estimator of the advantage function with time step \(t\), and \(\pi_{\theta}\) is the strategy function. \[J(\theta)=E_{t}\bigg{[}\frac{\pi_{\theta}(a_{t}|s_{t})}{\pi_{\theta_{old}}(a_ {t}|s_{t})}\hat{A}_{t}\bigg{]}, \tag{30}\]  
\[J(\theta)=\hat{E}_{t}\big{[}\min\big{(}r_{t}(\theta)\hat{A}_{t},clip(r_{t}( \theta),1-e,1+e)\hat{A}_{t}\big{)}\big{]}. \tag{31}\]  
The PPO algorithm utilizes the principle of importance sampling to update the strategy with the experience generated by the old strategy. It solves the redundant process in reinforcement learning that requires recollecting experience after each policy update, thus effectively utilizing past experience and speeding up the learning process. The principle of importance sampling requires \(r_{t}(\theta)\) (expressed as Equation (32)) to be as close to 1 as  
Figure 3: The interaction process between two modules.