## 4 Multi-Objective Deep Reinforcement Learning Based on the PPO Algorithm
### Markov Decision Process  
Our approach constructs the computational offloading and resource allocation problem as an MDP. The MDP is defined by a state space encompassing key metrics like transfer latency, computational delay, energy consumption, an action space for offloading strategies, and a reward function designed to minimize latency and energy consumption. This formulation is crucial for generating optimized offloading policies based on the current state of the network. In the context of computational offloading and resource allocation within the edge environments, we define the MDP as a quintuple, encompassing the state space (\(S\)), action space (\(A\)), state transition probability matrix (\(P\)), reward function (\(R\)), and the discount factor (\(\gamma\)). The crux of the MDP's application in this scenario lies in its ability to generate a policy that dictates offloading decisions based on the current state.  
* State: The state of the environment consists of several metrics for the local device, the MEC server, and the collaborator devices. Transfer latency used by the computational tasks, computational latency, transfer energy consumption, computational energy consumption, CPU utilization of the current device, and the number of computational tasks.
* Action: The action space consists of four offloading strategies: local computing, D2D computing, edge computing, and migration computing.
* Probability: The state probability transfer matrix can be expressed as \[P_{ss^{\prime}}^{d}=P\big{[}S_{t+1}=s^{\prime}\big{|}S_{t}=s,A_{t}=a\big{]}.\] (25)
* Reward: The rewards are provided at each episode. Considering the goal of minimizing UE latency and energy consumption, the defined reward should be negatively correlated with this goal. It can be expressed as follows: \[R(s,a)=\sum_{n=i}^{N}\gamma\Big{(}C_{i}^{local}-C_{i}^{d2d}-C_{i}^{dage}-C_{i} ^{mig}\Big{)},\forall i\in U_{i}.\] (26)
* \(\gamma\): is the discount factor, \(\gamma~{}\in~{}[0,1]\).  
For the computational model of the system, the state can be composed of a number of metrics from the local device, the MEC server, collaborator devices, and the offloaded device. After executing each decision, the agent will receive an immediate reward to measure the goodness of the current decision [32]. The reward function is shown in Equation (26), and the value of the state is dependent on the current reward versus the possible future reward. The return of state is defined as follows:  
\[G_{t}=R_{t+1}+\gamma R_{t+2}+\gamma^{3}R_{t+3}+\ldots+\gamma^{n}R_{t+n}=\sum_ {n=0}^{\infty}\gamma^{n}R_{t+n+1}. \tag{27}\]  
\(\gamma\) is used to estimate the current value of future rewards, then the long-term payoff can be expressed as the expected value of \(G_{t}\), defined as  
\[V(s_{t},\pi)=E_{\pi}[G_{t}]=E_{\pi}[R_{t+1}+\gamma V(s_{t+1},\pi)]. \tag{28}\]  
\(V(s_{t},\pi)\) denotes the value function of state \(s\) under a given strategy \(\pi\), \(S_{t}\) denotes the next state, and \(\pi\) denotes the strategy.  
The problem is thus formulated as solving the optimal policy \(\pi^{*}\) of the system model to maximize the long-term value of the model \(V(s,~{}\pi)\), expressed as follows:  
\[\pi^{*}=\underbrace{argmax}_{\pi}V((s_{t+1},\pi)). \tag{29}\]