# MASA: Multi-Application Scheduling Algorithm for Heterogeneous Resource Platform
## 4 Multi-Application Scheduling Algorithm (MASA) Based on DRL
### Network Training
#### 4.2.1 Reward Dynamic Alignment Method  
The standard MDP process is suitable for round-based decision problems, where the change in the state of the environment occurs after the agent has executed an action [28]. In heterogeneous resource systems, external factors such as the random arrival of jobs and internal factors such as the different completion times of multiple actions at the same timestep can cause the state of the environment to change while the SCHEDULER is executing an action. This paper proposes the RDA method as a solution to incorrect reward propagation during finite MDP. 1. incorrect rewards  
**Scenario 1**. : _SCHEDULER can achieve real-time task scheduling by observing the real-time environment state and taking immediate actions while making decisions in the current time step. For instance, as depicted on the Figure 3(a), when the action \(a_{n,t}\) ends at time step, denoted by \(t^{\prime}\), the immediate reward for the action is also granted at \(t^{\prime}\). While the action \(a_{i,t}\) is being executed, the random arrival of a job can modify the priority relation between tasks, and SCHEDULER observes \(S_{t+1}\) when \(t^{\prime}\in(t+1,t+2]\). This circumstance leads to the incidence of the incorrect reward for the action at time \(t+1\)._  
**Scenario 2**. : _Scheduling decisions of tasks with the same priority are formed at the same timestep. However, due to the differences in the performance of heterogeneous PE as well as the task requirements, each decision process completes at a different time, resulting in an inconsistency between the immediate rewards and the observed rewards, a situation that also leads to the occurrence of incorrect rewards. For example, as shown on the Figure 3(b), although task \(T_{1}\) has been accomplished earlier than expected, the scheduling policies for the subsequent tasks \(T_{3}\), \(T_{4}\) and \(T_{5}\) are applied after the completion of task \(T_{2}\). As a result, there is a delay in the reward for task \(T_{1}\)._  
2. Implementation of RDA  
The RL process of an MDP can be summarized as a \(\left\{\mathsf{s}_{\mathsf{t}},\left\{\mathsf{a}_{\mathsf{n}}\right\}_{ \mathsf{n}=\mathsf{1}}^{n},\left\{\mathsf{r}_{\mathsf{n}}\right\}_{ \mathsf{n}=\mathsf{1}}^{n}\right\}_{\mathsf{t}=\mathsf{1}}^{\mathsf{T}}\) sequence. The reward \(\mathsf{r}_{\mathsf{t}}\) corresponding to the previous decision affects the SCHEDULER's action decision \(\mathsf{a}_{\mathsf{t}+1}\) at time step \(\mathsf{t}+1\), so how to obtain the correct \(\mathsf{r}_{\mathsf{t}}\). The essential reason for generating incorrect rewards is the inconsistency of the time step at which the SCHEDULER interacts with the environment. Therefore, this paper proposes to compute the reward \(r_{\mathsf{clk}}^{\prime}\) on each clock signal so that the computation of \(r_{\mathsf{clk}}^{\prime}\) is independent of the  
Figure 4: (**a**) Incorrect reward lead by external factors; (**b**) Incorrect reward lead by internal factors. interaction time step. The time-step reward \(\mathrm{r_{t}}\) consists of the clock reward \(\mathrm{r^{\prime}_{clk}}\), i.e., the value of \(\mathrm{r_{t}}\) consists of a superposition of \(\{\mathrm{r^{\prime}_{clk}}\}\). The RDA method is specifically described below. * Step 1: Decouple the empirical data associated with the time step to obtain the sequence \(\{\mathrm{s_{t}},\mathrm{a_{t}}\}_{\mathrm{t}=1}^{\mathrm{T}}\) as well as \(\{\mathrm{r_{t}}\}_{\mathrm{t}=1}^{\mathrm{T}}\). T is the last interaction time step in the episodes. To obtain the sequences \(\{\mathrm{r^{\prime}_{clk}}\}_{\mathrm{clk}=1}^{\mathrm{CLK}}\) and \(\left\{\sum_{\mathrm{clk}=1}^{\mathrm{clk}_{\mathrm{know}}}\mathrm{r^{ \prime}_{clk}}\right\}_{\mathrm{clk}_{\mathrm{know}=1}^{\mathrm{CLK}}}^{ \mathrm{CLK}}\), compute the \(\mathrm{r^{\prime}_{clk}}\) on each clock signal and perform the \(\sum_{\mathrm{clk}=1}^{\mathrm{clk}_{\mathrm{know}}}\mathrm{r^{\prime}_{clk}}\) computation independently of the interaction timestep. Here, CLK refers to the clock length of the simulation.