# MASA: Multi-Application Scheduling Algorithm for Heterogeneous Resource Platform
## 4 Multi-Application Scheduling Algorithm (MASA) Based on DRL
### Framework of MASA
#### 4.1.3 Description of the Empirical Data  
The NN-Scheduler obtains state information \(\mathsf{s_{t}}\) and executes action \(\{\mathsf{a_{n,t}}\}_{n=1}^{n^{\prime}}\) at each interaction time step \(\mathsf{t}\). This paper differs from standard MDP processes in that the moment when the environment state changes is uncertain. The policy network generates multiple task decisions in time step \(\mathsf{t}\). When all the actions in time slot \(\mathsf{t}\) (time step \(\mathsf{[t,t+1)}\)) are completed, rewards are generated and the environment state is updated. The environment state changes if a new job DAG is reached during action execution.  
1. State space  
The NN-Scheduler receives an input state represented by \(\mathsf{S}=[\mathsf{s_{1}},\mathsf{s_{2}},\ldots,\mathsf{s_{0}}]\). \(\mathsf{s_{n}}\) represents the state of the nth job, and \(\mathsf{o}\) is the number of concurrent jobs in the job queue at the current time, where \(\mathsf{o}\) is a variable and its value cannot exceed the maximum job queue capacity \(\mathsf{C}\). Concurrent jobs include previously unfinished jobs and newly arrived jobs, and the NN-Scheduler can learn about the potential interference between concurrent jobs through training. The state \(\mathsf{s_{n}}\) of the nth job can be represented by the following equation:  
\[\mathsf{s_{n}}=\Big{(}\big{(}\overline{\mathsf{p_{v}}},\overline{\mathsf{ EET_{v}}},\overline{\mathsf{status_{v}}}\big{)}_{v=1}^{\|},|\mathsf{OT}|,| \mathsf{RT}|,|\mathsf{WT}\Big{)} \tag{14}\]  
The task \(\mathsf{n_{v}}\)'s assignable PE number is \(\mathsf{p_{v}}\); \(\mathsf{EET_{v}}\) is the estimated execution time corresponding to the task; \(\mathsf{status_{v}}\) is the current state of the task, classified by the labels ready, completed, or outstanding; \(\|\mathsf{j}\|\) is the number of task nodes in job \(\mathsf{j}\). \(|\mathsf{OT}|\) refers to the number of outstanding tasks that are waiting for the parent node to finish its execution. Similarly, \(|\mathsf{RT}|\) refers to the number of ready tasks in the set of ready tasks \(\mathcal{A}_{t}\), and JWT is the time that job \(\mathsf{j_{n}}\) has been running from the time it arrived on the system to the time it is currently running.  
2. Action space  
At each time step, the SCHEDULER observes the state \(\mathsf{s_{t}}\) of the environment and performs the number of n/ actions, \(\{\mathsf{a_{n,t}}\sim\pi_{0}(\mathsf{a}|\mathsf{s_{t}})\}_{n=1}^{n^{\prime}}\). The action space is shown in the red box in Figure 3, and the NN-Scheduler outputs the set of selected high-priority tasks \(\mathsf{n_{v}^{i}}\).  
3. Reward  
The correct reward is computed according to the RDA method (Section 4.2.1), taking into account the inconsistency between the time step of the scheduler's interaction with the environment and the time step of the environment's interaction with the scheduler. \(\mathsf{r_{t}}\) is computed by constructing the reward under each clock signal as \(\mathsf{r_{clk}^{\prime}}\), and then calculating \(\mathsf{r_{t}}\) from it. \(\mathsf{r_{clk}^{\prime}}\) is given by Equation (15), where \(\big{|}\mathsf{n_{comp}}\big{|}\) is the number of newly completed tasks under each clock.  
\[\mathsf{r_{clk}^{\prime}}=\big{|}\mathsf{n_{comp}}\big{|} \tag{15}\]