# MASA: Multi-Application Scheduling Algorithm for Heterogeneous Resource Platform
## 4 Multi-Application Scheduling Algorithm (MASA) Based on DRL
### Network Training  
This paper addresses the problem of training neural networks based on the policy gradient algorithm, Actor Critic (AC) [26; 27]. AC evaluates the merits of an action by using a dominance function, and if the action value function \(\mathsf{Q}\)\((\mathsf{S},\mathsf{A})\) is better than the state value function \(\mathsf{V}\)\((\mathsf{S})\), the corresponding action is improved. At each time step \(\mathsf{t}\), the actor network observes the environment state \(\mathsf{s_{t}}\) and chooses to execute action \(\{\mathsf{a_{n,t}}\}_{n=1}^{n^{\prime}}\sim\pi_{0,n}(\mathsf{s_{t}})\). n\({}^{\prime}\) denotes the number of tasks with the same priority.  
The Actor network's loss function is  
\[\mathsf{Loss_{actor}}=\sum_{t=0}^{\mathsf{T}}\log\pi_{0}(\mathsf{a_{n,t}}| \mathsf{s_{t}})\big{(}\mathsf{Q}_{\pi_{0}}(\mathsf{s_{t}},\mathsf{a_{n,t}})- \mathsf{V}_{\pi_{0}}(\mathsf{s_{t}})\big{)} \tag{16}\]The Critic network assesses the state value function \(\mathrm{V}_{\pi_{0}}\left(\mathrm{s}_{t}\right)\) and computes its loss function as shown in Equation (17).  
\[\mathrm{Loss}_{\mathrm{critic}}=\frac{1}{2}\big{(}\mathrm{Q}_{\pi_{0}}\big{(} \mathrm{s}_{t},\mathrm{a}_{n,t}\big{)}-\mathrm{V}_{\pi_{0}}\big{(}\mathrm{s}_{ t}\big{)}\big{)}^{2} \tag{17}\]  
\[\mathrm{V}_{\Theta}\mathrm{J}(\theta)\triangleq\mathrm{V}_{\Theta}\big{[} \mathrm{Loss}_{\mathrm{actor}}\ +\mathrm{Loss}_{\mathrm{critic}}\ +\eta\cdot\mathrm{E}_{5}\big{[}\mathrm{H}(\pi_{0} \left(\cdot|\mathrm{s}_{t}\right))\big{]} \tag{18}\]  
The training policy learns the optimal strategy using the regular entropy to make the output probability distribution of actions more uniform and to prevent convergence to a single output action. This paper considers introducing entropy regularity \(\eta\cdot\mathrm{E}_{5}\big{[}\mathrm{H}(\pi_{0}\left(\cdot|\mathrm{s}_{t} \right))\big{]}\) into the loss function, represented with entropy \(\mathrm{H}\left(\cdot\right)\) and weight \(\eta\). In summary, the gradient strategy is as follows.  
To achieve stable and fast training convergence, this paper also employs the following methods to optimize the training process: (1) Reward Dynamic Alignment (RDA) is used to address the problem of incorrect reward propagation. This problem arises due to inconsistent interaction time steps between SCHEDULER and the environment. (2) Gradually increasing the length of episodes can improve the initial training phase's inference ability for neural networks and help address the challenge of making multiple job DAG scheduling decisions. (3) Asynchronous multi-agent joint training is another method used to speed up the neural network's training process.  
Algorithm 2 shows the pseudocode of the network training process in MASA. Throughout the training process, asynchronous multi-agent joint training is performed. In order to avoid wasting training time, the initial events are terminated early, and Line 4 samples the event length \(\mathrm{T}\) from an exponential distribution with a small initial expectation \(\mathrm{T}_{\mathrm{mean}}\). (See Training Optimization Method 2 for more information). In order to avoid errors caused by randomness during job arrivals and inconsistent task execution times, Line 11 calculates the correct time-step reward based on the clock reward (see Training Optimization Method 1). Additionally, Line 22 implements the increment of the average length of episodes \(\mathrm{T}_{\mathrm{mean}}\). Lastly, the strategy parameter \(\theta\) of MASA is updated in Line 23.