## 3 System Design
### Design of DGRL Training Method, Controller Node Features and Reward Function  
#### 3.4.1 The Training Method of DGRL  
The model proposed in this paper is based on the deterministic policy gradient and this framework is based on Q-value. The Critic neural network of DGRL is used for Q-Value fitting. The quality of the policy can be expressed by the expected value according to the following equation:  
\[Q(S_{t},a_{t})=E\left[\sum_{K}\gamma^{k}R(S_{t+k},a_{t+k})\right] \tag{8}\]  
where \(R_{t}=R(S_{t},a_{t})\) is the feedback value of the environment, and \(\gamma\) is the time series decrement factor.  
Taking \(t=2\) as an example, the iterative formula of \(Q(S_{2},a_{2})\) can be obtained by approximating Equation (8) as follows:  
\[\begin{split} Q(S_{2},a_{2})&=E\Big{[}R_{2}+\gamma R _{3}+\cdots+\gamma^{n-2}R_{n}\Big{]}\\ &=E[R_{2}]+E\Big{[}\gamma R_{3}+\cdots+\gamma^{n-2}R_{n}\Big{]}\\ &\approx R_{2}+E\Big{[}\gamma R_{3}+\cdots+\gamma^{n-2}R_{n} \Big{]}\\ &\approx R_{2}+\gamma Q(S_{3},a_{3})\end{split} \tag{9}\]Inductive Equation (9) can get the iterative formula as shown in (10):  
\[\begin{split} Q(S_{t},a_{t})&=E[R_{t}+\gamma Q(S_{t+1}, a_{t+1})]\\ &\approx R_{t}+\gamma Q(S_{t+1},a_{t+1})\end{split} \tag{10}\]  
Define \(y_{t}=R_{t}+\gamma Q^{*}(S_{t+1},a_{t+1}^{*})\), where \(a_{t+1}^{*}\) is the output of the _Target Policy_ and \(Q^{*}(S_{t+1},a_{t+1}^{*})\) is the output of the _Target Q_. Then the loss function of the _Online_\(Q\) can be defined as follows:  
\[\begin{split} L(\theta^{Q})&=\frac{1}{N}\sum_{t}( Q(S_{t},a_{t})-y_{t})^{2}\\ &=\frac{1}{N}\sum_{t}(Q(S_{t},a_{t})-(R_{t}+\gamma Q^{*}(S_{t+1}, a_{t+1})))^{2}\end{split} \tag{11}\]  
where \(\theta^{Q}\) is all the weights that need to be trained in the _Online_\(Q\) model, \(Q(S_{t},a_{t})\) is the output of the _Online_\(Q\) model, and \(n\) is the batch size. At this point, replay can be done by sampling the four tuples (Last State, Action, Reward, New State) in the experience pool of DGRL model. The error back-propagation algorithm can be used to train the _Online_\(Q\) network and update all its internal weights \(\theta^{Q}\).  
All weights of the _Online Policy_ are represented by \(\theta^{\mu}\). Its update is slightly different from the Critic network because _Online Policy_ does not have explicit label data. However, through the deterministic gradient policy, it can be known that a good policy will get a larger Q value. Given the gradient of the loss function for Critic network and the output policy \(\mu(S_{t})\) of the _Online Policy_, the weight gradient update of the Q value can be obtained as follows:  
\[\
abla_{\theta^{\mu}}L=\frac{1}{n}\sum_{t}\
abla_{a}Q(S_{t},a_{t}\mid\theta^{ Q})\
abla_{\theta^{\mu}}\mu(S_{t}\mid\theta^{\mu}) \tag{12}\]  
Where \(a_{t}=\mu(S_{t}\mid\theta^{\mu})+N_{t}\), \(N_{t}\) is an Ornstein-Uhlenbeck stochastic process with 0 mean characteristics [34]. It enables the agent to explore beyond the learned policy, and avoids the network from falling into a local optimal policy.  
The initial weights of the _Target_\(Q\) model and the _Target Policy_ model \(\theta^{Q^{\ast}}\), \(\theta^{\mu^{\ast}}\) are completely copied from the respective corresponding models: \(\theta^{Q^{\ast}}\leftarrow\theta^{Q};\theta^{\mu^{\ast}}\leftarrow\theta^{\mu}\). After the weights of the Critic and Actor models have been updated through training, the Target model uses a soft update to update the weights, as shown in Equation (13). A soft update is used to assist the training process to reach the convergence state.  
\[\begin{split}\theta^{Q^{\ast}}&=\tau\theta^{Q}+(1- \tau)\theta^{Q^{\ast}}\\ \theta^{\mu^{\ast}}&=\tau\theta^{\mu}+(1-\tau)\theta ^{\mu^{\ast}}\end{split} \tag{13}\]