# Toward Low Time Fragmentation of Equipment: A Double-DQN Based TT&C Task Planning Approach
## 5 Low Time Fragmentation-Oriented TT&C Task-Planning Algorithm
### Evaluation Function Based on DDQN Network Training
When constructing an independent set, the reward obtained for each additional vertex is not only from the successful planning of the task represented by this vertex, but also from subsequent task rewards in non-conflicting task planning. For this delayed reward scenario, the Q function updated at each step is too short-sighted to reflect the long-term revenue situation well. Therefore, this paper uses the \(n\)-step Q-learning method to update the parameters. Unlike single-step learning, \(n\)-step Q-learning needs to wait \(n\) steps for parameter update. Only in this way can the Q-function provide a more precise evaluation of future rewards. The parameter update method is the same as the single-step learning (Equation (7)). Therefore, there is a difference in the calculation of the target value:  
\[h=\left\{ \begin{array}{ll}\sum\limits_{i=1}^{n-1}r(IS_{\tau+i},v_{\tau+i})+ \gamma\text{Q}(IS_{\tau+i},\text{argmax}_{v^{\prime}}\text{Q}(IS_{\tau+n},v^{ \prime},\Theta_{e});\Theta_{t})&IS_{\tau+n-1}\
eq\widehat{IS}\\ \sum\limits_{i=1}^{n-1}r(IS_{\tau+i},v_{\tau+i})&IS_{\tau+n-1}=\widehat{IS}\\ \end{array} \right. \tag{9}\]  
Moreover, there is a strong correlation between the continuous actions of the independent set construction process. In order to reduce the impact of the correlation between continuous samples on neural network training, this paper uses experience replay technology to speed up the convergence speed of neural network and improve the robustness of training results. Specifically, samples are continuously added to the data set \(\Pi\) at each step in every round. For example, in step \(\tau+n\), the tuple \((IS_{\tau+n},v_{\tau+n},R_{\tau+n,\tau},IS_{\tau})\) is added to \(\Pi\), where \(R_{\tau+n,\tau}=\sum\limits_{i=1}^{n-1}r(IS_{\tau+i},v_{\tau+i})\). In summary, we find that there is a reward delay problem during independent set construction and that there is a strong correlation between consecutive samples. Based on this, we first address the reward-delay situation during independent set construction using n-step Q-learning. Then, we use the experience replay technique to reduce the correlation between consecutive samples. At last, this paper designs an independent set algorithm based on double deep Q-learning to complete the training of parameter \(\Theta\), as shown in Algorithm 2. \(CG(V_{c},E_{c})\) is a task-planning conflict graph constructed from a set of tasks drawn from the sample space. ```
1: Initialize experience replay storage space \(\Pi\). 2: for round \(e=1\) to \(L\) do
3: Extract a task set and construct the corresponding task-planning conflict graph \(CG(V_{c},E_{c})\)
4: Initialize \(\tau=1\), \(IS_{1}=\text{Q}\), and random parameters \(\Theta_{t},\Theta_{e}\). 5: while\(V_{\overline{IS_{\tau}}}\
eq\varnothing\):
6:\(v_{\tau}=\left\{ \begin{array}{ll}\text{random node }v\in V_{ \overline{IS_{\tau}}}&\text{w.p.}\varepsilon\\ \text{argmax}_{v\in V_{\overline{IS_{\tau}}}}\text{Q}(IS_{\tau},v;\Theta_{e} )&\text{w.p.}1-\varepsilon\\ \end{array} \right..\)
7: Add vertex \(v_{t}\) into the local solution: \(IS_{\tau+1}=IS_{\tau}\cup\{v_{\tau}\}\)
8: Delete vertex \(v_{\tau}\) and its neighbor vertices from \(CG_{\overline{IS}}\). 9: if\(\tau\geq n\) then
10: Add tuple \((IS_{\tau+n},v_{\tau+n},R_{\tau+n,\tau}IS_{\tau})\) to \(\Pi\). 11: Sample random batch from B-\(\Pi\). 12: Use the samples in B to train and update the value of \(\Theta_{e}\) based on the gradient descent method. 13: Replace target parameters \(\Theta_{t}\leftarrow\Theta_{e}\) every c steps
14: end if
15:\(\tau=\tau+1\). 16: end while
17: end for
18: return \(\Theta_{t}\)
```  
**Algorithm 2**Evaluation function training based on DDQN learning