# Deep Reinforcement Learning for the Agile Earth Observation Satellite Scheduling Problem
## 3 Methodology
### GNDD Structure  
The graph attention network (GAT) is a graph neural network structure proposed by [PERSON] [30]. The network introduces an attention mechanism into the graph neural network structure and can weigh the relationships between graph nodes. By extracting problem features, the GAT can calculate the probability of the following actions based on the features of the current state. In the proposed method, we design the GAT-based decision neural network (GDNN) for problem sequence decision-making. As shown in Figure 6, the GDNN consists of nine layers. The first four layers are embedding layers, and each embedding layer is a single-layer GAT network using the attention mechanism to weigh the node and edge features. The following five layers are all fully connected layers, which are only responsible for updating the attributes of features. The fifth layer is the middle layer, which converts the network dimensions. The sixth to eighth layers are hidden layers whose dimensions remain the same. The last layer is the output layer and outputs a one-dimensional action probability. The feature update of the entire network is independent of the graph structure. In the proposed method, the node feature is \(\widetilde{v}_{i}\in\mathbb{R}^{10}\) and the edge feature is \(\widetilde{e}_{ij}\in\mathbb{R}^{5}\). To avoid complicating the network, the intermediate network structure has the same dimension, which is unified as \(F_{3}\). The transfer process of the extracted features in the network layer is as follows, where \(l\) is the network layer identifier, \(l\in[1,9]\wedge l\in\mathbb{N}^{+}\). (1) Embedded layer and the transfer network (\(l\in[1,4]\)). The node feature vector \(v\) is transferred in the embedding layer network through Equations (16)-(18), where the LeakyReLU function is proposed in [31]. The condition shown in Equation (19) is satisfied, and the ReLU function [32] is used to activate between layers. The edge feature matrix \(E\) is transferred in the embedding layer network though the Equation (20), and the conditions shown in Equations (21) and (22) are satisfied. \[\widetilde{z}_{ij}^{(l)}=\mathbf{W}_{\text{v}}^{(l)}\left[\widetilde{v}_{i}^{(l)} \left\|\widetilde{e}_{ij}^{(l)}\left\|\widetilde{v}_{j}^{(l)}\right.\right\| \widetilde{v}_{j}^{(l)}\right] \tag{16}\]  
Figure 6: GDNN structure for the AEOSSP. \[\alpha_{ij}^{(l)}=\frac{\exp\biggl{(}\text{LeakyReLU}\biggl{(}\vec{a}^{(l)}{}^{T} \widetilde{z}_{ij}^{(l)}\biggr{)}\biggr{)}}{\sum_{k\in\mathbf{N}_{i}}\exp\biggl{(} \text{LeakyReLU}\biggl{(}\widetilde{z}_{ik}^{(l)}\biggr{)}\biggr{)}},\,\vec{a}^{( l)}\in\mathbf{R}^{\mathbf{F}_{3}} \tag{17}\]  
\[\overline{v}_{i}^{(l+1)}=\text{ReLU}\Biggl{(}\sum_{j\in\mathbf{N}_{i}} \boldsymbol{\bar{v}}_{ij}^{(l)}\widetilde{z}_{ij}^{(l)}\biggr{)} \tag{18}\]  
\[\mathbf{W}_{\text{v}}^{(l)}\in\left\{ \begin{array}{c}\mathbf{R}^{\mathbf{F}_ {3}\times(2\mathbf{F}_{1}+\mathbf{F}_{2})},l=1\\ \mathbf{R}^{\mathbf{F}_{3}\times(2\mathbf{F}_{3})},l\in[2,4]\end{array} \right. \tag{19}\]  
\[\widetilde{e}_{ij}^{(l+1)}=\text{ReLU}\biggl{(}\mathbf{W}_{\text{E}}^{(l)} \overline{v}_{i}^{(l)}+\mathbf{W}_{\text{E}}^{(l)}\overline{v}_{j}^{(l)}+ \mathbf{W}_{\text{EE}}^{(l)}\widehat{e}_{ij}^{(l)}\biggr{)} \tag{20}\]  
\[\mathbf{W}_{\text{E}}^{(l)}\in\left\{ \begin{array}{c}\mathbf{R}^{\mathbf{F}_ {3}\times\mathbf{F}_{1}},l=1\\ \mathbf{R}^{\mathbf{F}_{3}\times\mathbf{F}_{3}},l\in[2,4]\end{array} \right. \tag{21}\]  
\[\mathbf{W}_{\text{EE}}^{(l)}\in\left\{ \begin{array}{c}\mathbf{R}^{\mathbf{F} _{3}\times\mathbf{F}_{2}},l=1\\ \mathbf{R}^{\mathbf{F}_{3}\times\mathbf{F}_{3}},l\in[2,4]\end{array} \right. \tag{22}\]  
(2) Middle layer and hidden layer network (\(l\in[5,8]\)). The middle layer and hidden layer are all fully connected layers. The input and output dimensions are both \(F_{3}\), and the feature transfer adopts the method shown in Equation (23).