## 3 Solving Method
### Markov Decision Process Model  
The Markov decision process presents the state space of the environment, the action space of the agent, the state transition function, and the reward function. the state describes the features of the environment, based on which the proposed encoder-decoder network selects a task action. According to the state transition function, the next state is reached based on the current state, task action, and resource action decision made by the heuristic algorithm. Then, a reward from the environment is received. * State space The state, \(S_{t}\), is described by the following equations: \[S_{t}=\left\{s_{1}^{t},s_{2}^{t},\cdots,s_{M}^{t}\right\},\] (8) \[s_{i}^{t}=\left\{x_{i},d_{i}^{t}\right\},\] (9) \[x_{i}=\left\{d_{i},P_{i}\right\},\] (10) \[d_{i}^{t}=\left\{sch_{i}^{t},ttw_{i}^{t},ect_{i}^{t},st_{i}^{t},rc_{i}^{t}\right\}\] (11) \[rc_{i}^{t}=\sum_{j\in J}\left(C_{j}-\sum_{i\in I}x_{ij}^{t}s_{i}\right)\] (12) where \(s_{i}^{t}\) is the state element of task \(i\), including static elements, \(x_{i}\), and dynamic elements, \(d_{i}^{t}\). The value of static elements is unchanged in the entire decision process, and the value of dynamic elements may vary in each new state. If task \(i\) has been scheduled before, \(sch_{i}^{t}=1\), and \(sch_{i}^{t}=0\), otherwise. Following a previous study [48], we use two dynamic elements, \(ttw_{i}^{t}\) and \(etet_{i}^{t}\), to include the VTW information for each task. This state representation for the MDP can be applied to cases where tasks may have various numbers of VTWs. \(ttw_{i}^{t}\) is the total length of the feasible time windows of all satellites for task \(i\) at time step \(t\). The feasible time windows for task \(i\) at time step \(t\), denoted as \(ftw_{i}^{t}\), are the intersection of the required observation time by users and the VTWs for task \(i\). Figure 2 describes the feasible time windows for task \(i\). \(eet_{i}^{t}\) is the earliest end time of task \(i\) in all satellites at time step \(t\). \(ftw_{i}^{t}\) and \(eet_{i}^{t}\) are updated at each time step, since a newly scheduled task may occupy the VTWs of task \(i\). The value of each element is normalized by the maximum value. \(st_{i}^{t}\) is the value of storage occupied by task \(i\). \(rc_{i}^{t}\) is the value of the remaining capacity in all satellites for task \(i\). \(rc_{i}^{t}\) is calculated by Equation (12). \(x_{ij}^{t}=1\) if task \(i\) is observed by satellite \(j\) at time step \(t\), and \(x_{ij}^{t}=0\) otherwise. \(st_{i}^{t}\) was originally a static element; here, \(st_{i}^{t}\) and \(rc_{i}^{t}\) are normalized together by the maximum value of the two types of elements. Thus, the value of \(st_{i}^{t}\) may change over time steps. * Action space The action space is decomposed into the task action and resource action subspaces: \[A=\{A_{t},A_{r}\}\] (13) \[A_{t}=\{i|i=1,2,\cdot\cdot\cdot,M\}\] (14) \[A_{r}=\{j|j=1,2,\cdot\cdot\cdot,N\}\] (15) \(A_{t}\) and \(A_{r}\) are the collections of task number and satellite number, respectively. The action is selecting a task and a satellite, and the selected task is executed at the earliest of its feasible time windows by the selected satellite. * State transition If task \(a\) and satellite \(b\) are determined at time step \(t\), the state, \(S_{t}\), is then updated to \(S_{t+1}\). In \(S_{t+1}\), the static elements of all tasks remain unchanged. \(sch_{a}^{t+1}\) is set to 0. The \(ttw_{i}^{t+1}\) and \(er_{i}^{t+1}\) values of the other tasks are updated according to the time interval occupied by task \(a\) in satellite \(b\). The \(rc_{i}^{t}\) for other tasks is updated according to the storage occupied by task \(a\) in satellite \(b\). The \(s_{i}^{t}\) for all tasks is normalized by the updated maximum value. * Reward function In previous studies [42; 47; 48], the reward function is usually described as the increment in total profit.