## 5 Deep Reinforcement Learning for Workflow Scheduling
### Training Algorithm  
The Actor-Critic method exhibits lower sample utilization and instability during training, and training the Critic network to predict baselines is challenging. To address these issues, the PPO method is adopted, combined with the Self-Critic mechanism [39], using Rollout Baseline [32] as the Critic network.  
For the Actor network, the training objective function is defined as the expectation of the reward.  
\[J(\theta|V)=E_{O^{V}\sim\pi_{\theta}(|V)}reward(O^{V}|V) \tag{17}\]  
According to the definition of the PPO algorithm, the network parameters \(\theta_{old}\) are saved for Actor-old, and the network parameters \(\theta\) are saved for Actor-new. A mini-batch of data are fed into the corresponding networks with these two sets of parameters, resulting in two sets of action probability distributions. The important weights are then calculated based on these distributions. The important weights represent the similarity between the new and old policies.  
\[r(\theta)=\frac{\pi_{\theta}(O^{V}|V)}{\pi_{\theta_{old}}(O^{V}|V)} \tag{18}\]  
After obtaining the importance weights, the update range of the Actor network is restricted using the clipping technique to prevent gradients from becoming too large. The clip function, \(clip(r(\theta),1-\epsilon,1+\epsilon)\), is applied to confine the weights within the range of \((1-\epsilon,1+\epsilon)\). Consequently, the new training objective function is as follows:  
\[\begin{split} J_{PPO}(\theta|V)&=E_{O^{V}\sim\pi_{ \theta}(|V)}[max(r(\theta)D,\\ clip(r(\theta),1-\epsilon,1+\epsilon)D)]\\ \end{split} \tag{19}\]  
\[D=reward(O^{V}|V)-b(V) \tag{20}\]  
For the baseline \(b(V)\), three methods are mainly employed: Exponential Baseline (exponential moving average of the rewards), Critic Baseline, and Rollout Baseline. The first two baselines are used for comparison. For the Rollout Baseline, during the first epoch, the Exponential Baseline is used for initial training. Two parameters, mini-batch-actor and mini-batch-critic, are defined to control the training process. After each mini-batch-actor of data, the Actor network \(\pi_{\theta}\) is updated. After each mini-batch-critic of data, the Rollout network \(\pi_{\theta}\) is updated using the current network parameters \(\theta\) of the Actor network. Three variations of the Actor-Critic method based on different baselines, as well as the PPO method based on the Rollout Baseline, have been implemented for comparison. The algorithm's flow is illustrated in Algorithm 4, where the presence of both \"epoch\" and \"episode\" variables, controlling the number of training iterations, is for the purpose of comparison with non-PPO methods.  
```
1:Actor-network parameter \(\theta\); Rollout-network parameter \(\phi\); number of epoch \(E\); number of episodes \(I\); PPO-Epoch \(PE\); batch-size \(B\); mini-batch-actor \(BA\); mini-batch-critic BC;
2:for epoch \(e\Leftarrow 1:E\)do
3: Generate \(D\) problem instances randomly;
4:for episodes \(i\Leftarrow 1:I\)do
5:for batch \(b\Leftarrow 1:B\)do
6: Get sub-tasks sequence \(x\) from \(D\);
7: The agent run policy \(\pi_{\theta_{old}}\) to get the scheduling sequence \(O^{V}\) base on \(V\);
8: Store the (\(O^{V}\), \(V\), \(reward\)) in buffer;
9:if\(b\) mod \(BA==0\)then
10:for\(pe\Leftarrow 1:PE\)do
11: Sample \(BA\) batches of data from the buffer;
12: Update \(\theta\) by a gradient method to optimize the objective function \(J_{PPO}(\theta|V)\)
13:endfor
14:endif
15:if\(b\) mod \(BC\) == \(0\)then
16:if OneSidedPairedTTest(\(\pi_{\theta},\pi_{\phi}<\lambda\)) then
17:\(\phi\Leftarrow\theta\);
18:endif
19:endif
20:endfor
21: Update \(\pi_{\theta_{old}}\) to \(\pi_{\theta}\)
22:endfor
23:endfor
```  
**Algorithm 4** Training Algorithm