# DRL-Based Dynamic Destroy Approaches for Agile-Satellite Mission Planning
## 3 Modeling Process
### Markov Decision Process Modeling
#### 3.3.1 State Space  
In order to facilitate the extraction of node features by the graph neural network, the task information (\(t^{d},t^{s},t^{e},Roll,Pitch\)) and environmental information (\(P\), \(T\)) are embedded into a seven-dimensional vector, as shown in the input and embedding of Figure 3:  
\[s_{i}=\left[t_{i}^{d},t_{i}^{s},t_{i}^{e},Roll_{i},Pitch_{i},P_{i},T_{i}\right] \tag{14}\]  
where, in Equation (14), the first three dimensions, \(t^{d}\), \(t^{s}\), and \(t^{e}\), are defined in Table 1, representing the task-execution time, task-start execution time, and task-end execution time. \(Roll_{i}\) and \(Pitch_{i}\) represent the roll and pitch attitude angles observed by the satellite to the task node \(i\), respectively [35]. \(s_{i}^{t}\) denotes the embedded vector in the \(t\)-th iteration, while \(P_{i}\) and \(T_{i}\) represent the cumulative profit and execution time, respectively, of the scheduled list corresponding to the \(t\)-th iteration. \(t^{d}\), \(Roll\), and \(Pitch\) are normalized by the corresponding maximum value in the task node; \(t^{s}\) and \(t^{e}\) are normalized by the total planning time; and \(P\) and \(T\) are normalized by the total cumulative value of the scheduling process.  
Therefore, the state space is defined as a vector containing task information, scheduled task lists, and environmental information:  
\[S_{t}=[s_{1}^{t},s_{2}^{t},\dots,s_{n}^{t}]^{T} \tag{15}\]  
\begin{table}
\begin{tabular}{c c} \hline \hline
**Name** & **Description** \\ \hline \(r_{i}\) & The \(i\)-th task \\ \(x_{i}\) & A binary variable representing whether task \(r_{i}\) is selected \\ \([w_{i}^{s},w_{i}^{s}]\) & The selected visible time window for task \(r_{i}\) \\ \([t_{i}^{s},t_{i}^{e}]\) & The selected execution time window for task \(r_{i}\), which is a subset of \(\mathbf{OW_{i}}\) \\ \(t_{i}^{e}\) & Execution time for task \(r_{i}\) \\ \(w_{i}\) & The profit for task \(r_{i}\) \\ \(t_{ij}^{e}\) & Satellite attitude maneuver time between tasks \(r_{i}\) and \(r_{j}\) \\ \(\theta_{i}\) & Satellite observation angle for task \(r_{i}\) \\ \(\mathbf{OW_{i}}\) & The set of task-execution time windows for task \(r_{i}\) \\ \(v_{i}\) & The \(i\)-th embedded graph node \\ \(W\) & The weight matrix of transformation \\ \(Roll_{i}\) & Roll attitude angle observed by the satellite to the task node \(i\) \\ \(Pitch_{i}\) & Pitch attitude angle observed by the satellite to the task node \(i\) \\ \(S_{t}\) & The state in \(t\)-th iteration \\ \(A_{t}\) & The action in \(t\)-th iteration \\ \(\mathbf{\Omega}\) & The set of removing action nodes \\ \(\mathbf{\Lambda}\) & The set of corresponding removing coefficients \\ \(G_{t}\) & The reward function \\ \(R\) & The discounted cumulative reward \\ \hline \hline \end{tabular}
\end{table}
Table 1: Summary of variables and notation.  
Figure 4: Agent-environment interaction in AEOS.  
\(S_{t}\) represents the current state in the \(t\)-th iteration. As shown in Equation (15) and Figure 3, each element of \(S_{t}\) represents a scheduled embedding task vector \(s_{i}^{t}\).