## 3 PPO Resource Scheduling  
Based on the above content, the heterogeneous edge computing network resource scheduling problem is established. In this section, it is constructed as a deep reinforcement learning problem and solved by PPO resource scheduling algorithm. First, the problem is modeled as an MDP, and the edge server (i.e., the control node) is regarded as an agent to learn the resource scheduling strategy. The state space, action space, state transition probability function, and reward function of MDP are designed as follows. Figure 2: Software-defined heterogeneous edge computing network. State space \(\mathcal{S}\). For the considered heterogeneous edge computing network, the edge computing tasks and edge computing node resource scheduling in the network determine the energy consumption of the entire network. Therefore, we take the subtasks in the network and edge computing node scheduling as the state \(s\), then the state space can be expressed as a set of all subtasks and node scheduling strategies. * Action space \(\mathcal{A}\). The scheduling of subtasks causes the above state to change. Therefore, we use the scheduling strategy of the next subtask as the action of the agent \(a\). This design is more intuitive and convenient for the interaction between the agent and the environment. The defined action space can be represented as a collection of all possible scheduling policies for the next subtask. * State transition probability function \(\mathcal{P}\). According to the design of agent state and action, the state transition probability function of the design environment in this section is given as \[P(s^{\prime}|s,a)=\left\{ \begin{array}{cc}1,s^{\prime}=s(a)&\\ 0,\text{others}&\end{array} ,s\in\mathcal{S},a\in\mathcal{A}.\right.\] (7) The above expression indicates that the update of the state is uniquely determined by the current state and the next action. * Reward function \(\mathcal{R}\). We transform the optimization objective (6) into a cumulative reward maximization problem by designing a reward function. Specifically, the reward of each step is defined as a weighted function of the computing, communication energy consumption and load balancing degree brought by the subtask allocation, which is expressed as \[r=-\lambda E^{\mathit{tolda}}E^{\mathit{balance}},\] (8) where \(\lambda\) is a positive weight to limit the reward into a reasonable range. The design of the above reward function enables the agent to continuously move toward a low-energy, load-balanced state to maximize the accumulation of rewards. The role of the Actor network is to fit the strategy of the agent \(\pi_{\theta}(a|s)\), and the strategy is expressed in the form of a Gaussian distribution. The Gaussian distribution can be fully described by its mean and standard deviation. When the agent needs to take action, the Gaussian distribution is recovered first, and then an action is selected by random sampling. The PPO initializes a network with the same structure and parameters as the Actor network, named old-Actor to keep the old policy unchanged for each update round. The old-Actor network does not participate in training. It only copies parameters from the Actor network before each update to maintain the old strategy of the current round \(\pi_{\theta_{g}}\). The parameters of the Actor network are updated by  
\[ \begin{array}{c}\theta_{g+1}=\underset{\theta}{\arg\max}\;\frac{1}{T}\sum \limits_{l=0}^{T}\min\biggl{(}\frac{\pi_{\theta}(a_{l}|s_{l})}{\pi_{\theta_{g} }(a_{l}|s_{l})}A_{t}^{\pi_{\theta_{g}}},\\ \text{clip}\biggl{(}\frac{\pi_{\theta}(a_{l}|s_{l})}{\pi_{\theta_{g}}(a_{l}|s_ {l})},1-\varepsilon,1+\varepsilon\biggr{)}A_{t}^{\pi_{\theta_{g}}}\biggr{)}, \end{array}  \tag{9}\]  
where \(g\) indicates the number of rounds of updates, \(A_{t}^{\pi_{\theta_{g}}}=R_{t}+\gamma V(s_{t+1})-V(s_{t})\) is the advantage function, \(R_{t}\) and \(V(s_{t})\) are the reward value of the time step \(t\) and the state value of \(s_{t}\). The Critic network completes the estimation of the state value \(V(s)\).