# SLA-DQTS: SLA Constrained Adaptive Online Task Scheduling Based on DDQN in Cloud Computing
## 3 Proposed Online Task Scheduling Model  
### Deep Learning Technique  
RL is a branch of ML, which learns continuously through the agent's interaction with the environment based on the rewards or punishments obtained, to better adapt to the environment ([PERSON] et al., 2019). RL can learn without a priori knowledge of the environment and obtain the decision-making ability of the corresponding environment through repeated interactive learning. Figure 1 shows a classic RL model. In each step, the agent makes a decision based on the strategy \(\pi\) through the state \(S_{t}\) obtained from the environment and takes action \(a=\pi(s)\). The environment transfers accordingly from state \(S_{t}\) to state \(S_{t+1}\). The agent obtains \(R_{t}\) returned by this process. RL assumes the process has a Markov property, i.e., the probability and reward of state transition only depend on \(S_{t}\) and \(A_{t}\), and have nothing to do with the state action before \(t-1\). Through interaction with the environment, the agent can record these quantities for learning, so as to maximize the expected cumulative reward \(G_{t}\). Therefore, RL can obtain decision-making strategies through observation and learning without prior knowledge of the environment. The strategy used by the agent to make decisions in RL is defined by the probability distribution \(\pi(a|s)\) of action \(a\) under state \(s\). In the classic reinforcement learning algorithm Q-learning, it is composed of state and action. Q-table is used for storage, and the s-a mapping with the highest Q-value in state s is selected from the table when making a decision. In Q-learning, the optimal strategy is that \(\arg\max_{a\in A}(Q(a|S_{t}))\) is the maximum cumulative return of the selection action an in state \(S_{t}\). If \(Q\) can predict the cumulative  
Figure 1: Reinforcement learning paradigm. return of each action in the current state, the optimal decision can be made. The following is the updated formula of \(Q\):  
\[Q(S_{t},A_{t})=Q(S_{t},A_{t})+\alpha[R_{t}+\lambda max_{a}Q(S_{t+1},a)-Q(S_{t},A _{t})] \tag{1}\]  
The \(R_{t}\) is the return of the action \(A_{t}\) in the state \(S_{t}\). The \(\lambda\) is the discount factor, which is used to calculate the cumulative return. Take the currently observed return \(R_{t}\) plus \(\lambda\) multiplied by the maximum cumulative return of the next state \(S_{t+1}\) as the current maximum cumulative return. Make the difference between it and the expected return of the current state \(S_{t}\) to obtain the loss, and multiply the loss by the learning factor \(\alpha\) to update the current prediction value. After repeated iterations, the more accurate the prediction of \(Q\), the smaller the loss value, and the more stable the value predicted by \(Q\) until convergence. At the same time, to make the algorithm converge faster and more stable, it needs suitable training samples, a suitable exploration strategy. The most common strategy is the \(\epsilon\)-greedy policy, which is a simple strategy. During exploration, it is determined whether to randomly select an action or choose the action with the highest return under Q prediction according to the probability \(\epsilon\). In the early stage of training, a higher \(\epsilon\) value is required to enhance the exploration to accelerate the convergence. In the later stage, the \(\epsilon\) value can be reduced to increase the convergence accuracy. Actual problems have a large number of state actions. It is inappropriate to store the probability distribution in the form of a table, and if a state has never appeared, Q-learning cannot handle it. That is, it lacks generalization ability, so a function with a certain number of parameters is generally used for fitting. As the amount of data and number of features increase, the advantages of deep neural networks have become more obvious, and DRL, which combines deep neural networks and RL, has achieved good results in many fields.