# Deep Reinforcement Learning for the Agile Earth Observation Satellite Scheduling Problem
## 3 Methodology
### Feature Engineering  
Appropriate feature extraction is the foundation of network decisions. We describe the AEOSSP as the time-attitude adjacency graph in which node attributes and edge weights are equally important. Therefore, the features of the AEOSSP comprise ten node features and five edge features, as shown in Figure 5. The following parts describe the meaning of each feature. All features must be normalized to improve network generalization ability and avoid weakening or failure of the network decision-making effect caused by the difference in data distribution.  
Node features can be divided into task, VTW, and status features. The profit \(pri_{i}\) and the observation duration \(ct_{i}\) are task features proposed by the users. The former indicates the importance of \(task_{i}\), and the latter indicates the shortest time required for completing the task observation. The VTW features represent the VTW of the task, including the overhead time \(t_{i}^{\text{side}}\), the overhead roll angle \(\theta_{i}^{\text{side}}\), the start time \(wb_{i}\) and end time \(we_{i}\) of the VTW, the earliest start time \(t_{i}^{m}\), and its corresponding roll angle \(\theta_{i}^{m}\) of \(task_{i}\). The status features \(l_{i}^{\text{wait}}\) and \(l_{i}^{\text{last}}\) are updated after each decision. \(l_{i}^{\text{wait}}\) indicates whether \(task_{i}\) is the candidate task in the current state. When \(task_{i}\) is among the candidate tasks, \(l_{i}^{\text{wait}}=1\). Otherwise, \(l_{i}^{\text{wait}}=0\). \(l_{i}^{\text{last}}\) indicates whether \(task_{i}\) is the last task in the current solution sequence. If \(task_{i}\) is the last task, \(l_{i}^{\text{last}}=1\). Otherwise, \(l_{i}^{\text{last}}=0\). After normalization, we obtain the node features \(\widetilde{v}_{i}=\left(\widehat{pr}i_{i},\widehat{i}_{i}^{\text{side}},\widehat{ \theta}_{i}^{\text{side}},\widehat{c}i_{i},\widehat{w}\widetilde{v}_{i}, \widehat{w}\widetilde{e}_{i},\widehat{t}_{i}^{n},\widehat{\theta}_{i}^{n},I_{i} ^{\text{wait}},I_{i}^{\text{last}}\right)\in\mathbb{R}^{10}\) and the node feature vector \(\mathbf{v}\).  
The matrix of edge features \(\mathbf{E}\) indicates the edge features between two nodes, where \(\widetilde{e}_{ij}=\left(\widehat{d}_{ij},I_{ij}^{\text{n}},I_{ij}^{\text{n}}, I_{ij}^{\text{n}},I_{ij}^{\text{n}}\right)\in\mathbb{R}^{5}\). \(d_{ij}\) indicates the distance between two nodes. In time-attitude coordinates, the distance between two nodes indicates the satellite attitude transition angle between two tasks, which can be calculated as Equation (13). In practice, the satellite attitude transition angle is primarily determined by roll and pitch angles. The pitch angle of the satellite is time-dependent and related to the length of the VTW. Therefore,  
Figure 4: Solution construction process for the AEOSSP.  
Figure 5: Problem feature extraction.  
we use the overhead time \(t_{i}^{\text{side}}\) to represent the pitch angle, thus linking time to the attitude transition angle, as shown in Equation (15).  
\[d_{ij}=\left|t_{i}^{\text{side}}-t_{j}^{\text{side}}\right|*2\varphi_{\text{max}}/ tw_{\text{max}}+\left|\theta_{i}^{\text{side}}-\theta_{j}^{\text{side}}\right| \tag{15}\]  
where \(tw_{\text{max}}\) is the length of the longest VTW and \(\varphi_{\text{max}}\) is the maximum pitch angle of the satellite. In addition, \(l_{ij}^{\text{n1}}\), \(l_{ij}^{\text{n5}}\), \(l_{ij}^{\text{n0}}\) and \(l_{ij}^{\text{n20}}\) are features used to represent the relationship of two nodes. For \(node_{i}\), we sort all \(d_{ij}\) in ascending order. If \(d_{ij}\) is ranked within the first \(K\)th, then \(l_{i}^{\text{nK}}=1\). Otherwise, \(l_{i}^{\text{nK}}=0\).