# Toward Low Time Fragmentation of Equipment: A Double-DQN Based TT&C Task Planning Approach
## 5 Low Time Fragmentation-Oriented TT&C Task-Planning Algorithm
### Evaluation Function Based on DDQN Network Training  
As a typical representative of the value-based algorithm in the reinforcement learning algorithm, the DQN algorithm combines deep learning and reinforcement learning to predict the Q value. Q value can be understood as the value of state action, i.e., the expected revenues of the agent through acting in a certain state. Although the DQN algorithm has achieved good results in deep reinforcement learning, it sometimes overestimates the behavior value too optimistically. This will cause the algorithm to unanimously regard the suboptimal behavior value as the optimal behavior value, eventually failing to converge to the best value function. However, the calculation method of the target value of DDQN and DQN is different. In DQN, the largest Q value in the target network is directly selected as the target value. Additionally, DDQN calculates the Q value according to the parameters of the current Q network. Therefore, the Q value in DDQN is less than or equal to the Q value in DQN. This reduces the overestimation to a certain extent, making the Q value closer to the real value. Therefore, in this section, solving the independent set in the task-planning conflict graph is modeled as a DDQN learning process. Additionally, the Q evaluation function has been trained. This paper defines the state, action, and reward under the reinforcement learning framework. The specific content is as follows:  
1. State: The state in reinforcement learning is the local solution _IS_. Because the impact of local solution on the subsequent independent set construction process is equal to the task-planning conflict graph in the iteration process, the q-dimension embedded vector \(\sum_{u\in IS}\mu_{u}\) of the task-planning conflict subgraph is used to represent the current state. 2. Action: The action \(v\) is a vertex that does not belong to the current local solution _IS_. Additionally, the action \(v\) is expressed as a q-dimensional embedding vector \(\mu_{v}\). 3. Reward: This paper aims to minimize time fragments and maximize task revenue. Therefore, the designed reward is the revenue of vertex \(v\) minus the total time fragment's increment after vertex \(v\) is added to the current local solution _IS_, namely: \[r(IS,v)=\lambda\omega_{v}-\Delta T(IS,v)\] (6) where \(\Delta T(IS,v)\) is the total time fragment generated by adding the feasible scheme represented by vertex \(v\) to the set of feasible schemes corresponding to the current solution _IS_, and \(\lambda\)\(\in\)(0,1) is a constant. The evaluation function Q(\(IS,v;\Theta\)) gives the value of action \(v\) in a given state _IS_. DDQN is used to optimize parameter \(\Theta\) of the evaluation Q(\(IS,v;\Theta\)) function in the reinforcement learning process. In this paper, two DDQN neural networks are constructed to train model parameters. i.e., target network and evaluation network. The evaluation network is used to select an action, and the target network is used to evaluate the value of the action. The two networks update parameters alternately with each other. Specifically, target network has parameter \(\Theta_{t}\) and evaluation network has parameter \(\Theta_{e}\) are established. Additionally, the parameter \(\Theta_{e}\) is updated by minimizing the square loss through the gradient descent method:  
\[L(\Theta)=E\Big{[}(h-\text{Q}(IS,v;\Theta_{e}))^{2}\Big{]} \tag{7}\]  
During the training process, the neural network copies parameter \(\Theta_{t}\) to parameter \(\Theta_{e}\) every several step. In DDQN, the target value is defined as  
\[h=\left\{ \begin{array}{ll}r(IS_{\tau},v_{\tau})+\gamma\text{Q}(IS_{\tau}, \underset{v^{\prime}}{\text{argmax}}\text{Q}(IS_{\tau+1},v^{\prime},\Theta_{ e});\Theta_{t})&IS\
eq\widehat{IS}\\ r(IS_{\tau},v_{\tau})&IS=\widehat{IS}\end{array} \right. \tag{8}\]  
where \(\gamma\) is the Q-learning decay coefficient.