## 3 Heuristic Scheduling Algorithm for Deep Reinforcement Learning
### Drl-Ha
#### 3.1.1 HCNN Model  
The HCNN is based on the sequence-to-sequence (Seq2 Seq) structure, which includes an encoder, decoder, and global function. 1. Encoder: the encoder's objective is to convert the static feature parameters of the to-be-scheduled tasks into a high-dimensional parameter eigenvector by one-dimensional convolution. In this way, the network is able to exhaustively exploit the parameter information of the task to make a reasonable decision. Because each imaging satellite task is independent, the output of the neural network is irrelevant to the sequence of the task data. Therefore, a simple embedding layer is necessary to solve this problem, which additionally reduces the time and space complexity of the encoder model. In this study, the features of a task are composed of the static feature \(s\) and the dynamic feature \(d\). The static features are attribute features, including task observation time window \(tw_{i}\), profit \(p_{i}\), duration \(l_{i}\), attitude angle parameters \(\theta_{i}^{roll,s}\), \(\theta_{i}^{roll,s}\), \(\theta_{i}^{pitch,s}\), \(\theta_{i}^{pitch,s}\), and so on. The dynamic features are the to-be-scheduled status: the value is 0 or 1; 0 means available, while 1 means unavailable. 2. Global variables: since the eigenvector of an individual task is independent in the encoder, these variables are not able to indicate the interaction features between tasks. Therefore, this study applies an improved multi-head attention for extracting related features. Figure 4 displays the structure of the neural network. By inputting the eigenvector set of the task set, we get the variable G, which contains the historical scheduling information between tasks. Figure 3: DRL-HA operation process diagram. * Decoder: unlike the encoder, each decoding step points to the input task. In this research, because selecting the current task is based on previous historical decision information, the HCNN model accomplishes the decoding process through this historical decision information. To obtain the historical decision variables, a neural network module with a memory function is required to realize the acquisition of historical decision information. Fortunately, the RNN network is a great choice for solving this problem. In this algorithm, the gated recurrent unit (GRU) network structure [26] is selected for achieving the extraction of the historical decision information. * Attention mechanism: the fundamental goal of the attention mechanism is to select the information that is the most compatible with the current target and has the highest degree of attention from numerous information. Based on the encoder-decoder structure, this mechanism utilizes the neural network model for completing feature interaction between the outputs from the encoder and decoder. In this way, the related distribution of the next task is obtained. At the softmax layer, this related distribution is able to be converted into the probability distribution diagram of task selection, which is greatly advantageous for selecting the next task. The overall structure of the HCNN model is shown in Figure 5. The blue cells are the structural parts of this model; the dynamic process, where dynamic parameters are updated during the process, is indicated by the orange arrows and cells; there is one black arrow from the probability function to the embedding layer, representing the task which should be concealed; the grey arrows and cells demonstrate that all parameters are constant. At the encoder layer, the static parameter eigenvector \(H_{s}\) and the dynamic parameter eigenvector \(H_{d}\) are obtained by embedding. At the decoder layer, the historical decision information \(h_{t}\) is obtained by GRU. Figure 4: Structure of multi-head attention. In the HCNN model, this study designs two attention layers for selecting tasks. In the first layer, the total feature parameter \(M\) is obtained from the connection layer in formula (12).