## 3 Method
### Scheduling Policy Optimization Algorithm Based on HEGNN-PPO
#### 3.4.2 Algorithm Training  
The PPO algorithm is used for training, employing an actor-critic architecture [27]. The actor is the policy network \(\pi_{\omega}\), responsible for generating the probability distribution of actions based on the current state \(s_{t}\); the critic is another network \(v_{\phi}\), which predicts the value \(v(s_{t})\) of state \(s_{t}\). The critic network takes the state embedding \(h_{global}\) computed by the HEGNN as input and outputs the estimated value of that state.  
Policy network update: Using the collected experience data, the PPO algorithm updates the policy network parameters \(\omega\) with the objective of maximizing the expected cumulative reward, optimized through the following loss function:  
\[L^{\text{clip}}(\omega)=\hat{\mathbb{E}}_{t}\Big{[}min\bigg{(}\frac{\pi_{ \omega}(a_{t}|s_{t})}{\pi_{\omega_{\text{old}}}(a_{t}|s_{t})}\hat{A}_{t}, \text{clip}\bigg{(}\frac{\pi_{\omega}(a_{t}|s_{t})}{\pi_{\omega_{\text{old}}} (a_{t}|s_{t})},1-\epsilon,1+\epsilon\bigg{)}\hat{A}_{t}\bigg{)}\Big{]} \tag{23}\]  
where \(\hat{A}_{t}\) is the advantage estimate, representing the advantage value of choosing action \(a_{t}\) in state \(s_{t}\), and \(\epsilon\) is the clipping parameter used to control the extent of policy updates, preventing instability due to overly large updates.  
Value network update: The goal of the critic network \(v_{\phi}\) is to minimize the value prediction error, optimized through the following loss function:  
\[L^{V}(\phi)=\hat{\mathbb{E}}_{t}\Big{[}\Big{(}v_{\phi}(s_{t})-\hat{R}_{t})^{2 }\Big{]} \tag{24}\]  
where \(\hat{R}_{t}\) is the discounted reward estimate, and \(\hat{\mathbb{E}}_{t}\) denotes the expectation at time step \(t\).  
The training flowchart of the algorithm is illustrated in Figure 5.  
Algorithm 1 outlines the complete training procedure of the proposed HEGNN-PPO scheduling strategy.  
```
Initialize HEGNN network parameters \(\theta\);  Initialize policy network parameters \(\omega\);  Initialize critic network parameters \(\phi\);  Sample an initial \(S\) of FJSP instances with batch size \(B\); for\(i=1,2,\ldots,I\)do: if\(i\bmod 20==1\)then  sample (\(B\)) new FJSP instances;  Initialize the experience buffer \(D\); for\(b=1,2,\ldots,B\)do:  Initialize the state \(s_{t}\) based on instance \(b\); while\(s_{t}\) is not a terminal state do:  Extract features \(h_{t}=\) HEGNN (\(s_{t}\));  Sample action \(a_{t}\) according to the policy \(\pi_{\omega}(\cdot|h_{t})\);  Select action \(a_{t}\), obtain reward \(r_{t}\) and next state \(s_{t+1}\);  Store the experience in buffer \(D\);  Update the state \(s_{t}\gets s_{t+1}\); endwhile endfor for\(k=1\) to \(K\)do:  Randomly shuffle \(D\) and divide it into batches  Compute the estimated advantages \(\hat{A}_{t}\) and the loss \(L\) for each batch;  Update the network parameters \(\theta\), \(\omega\) and \(\phi\); endfor if\(i\bmod 10=0\)then  evaluate the current policy performance using the validation set; endif endfor Output: the trained network parameters;
```  
**Algorithm 1** Training Process  
Figure 5: Training flowchart of the algorithm.  
Based on the algorithm design, the proposed scheduling strategy optimization algorithm effectively leverages the rich feature information extracted by the HEGNN. By integrating the optimization capabilities of the PPO algorithm, it learns high-quality scheduling strategies. This enables efficient and flexible scheduling decisions in complex and dynamic scheduling environments.