## 3 Solving Method
### Markov Decision Process Model  
The Markov decision process is the fundamental framework of reinforcement learning, presenting the state of the environment, the action of the agent, the state transition process, and the reward. The state is a description of the environment based on which the agent (decision network in this study) selects an action, resulting in the next state, and receives a reward from the environment.  
* State space  
The state is the input of the decision network and the basis for decision-making in the network. According to the problem description in Section 2, the state should consist of the task and satellite attributes.  
In related studies, the state is typically composed of the original attributes, including the duration, profit, latitude, longitude of the task, and the start and end of the VTW. The network must extract features from the original attributes and learn to fit the relationship between the original attributes and rewards under complex constraints, which increases the  
Figure 3: Framework of the solving method.  
difficulty of training decision networks. Moreover, in a DRL-based method, the sequence length of network inputs for each task usually needs to be the same, but different tasks may have different numbers of VTWs, resulting in various sequence lengths of inputs if only original attributes are used.  
In this study, we input the direct attributes calculated from the original attributes into the state of MDP to improve the learning efficiency of decision-making networks. The VTW information is included in two direct attributes, the total length of the feasible time intervals and the earliest end time, so that the sequence length of inputs for all tasks is the same. The state \(S_{t}\) is given by the following equations:  
\[S_{t}=\left\{s_{1}^{t},s_{2}^{t},\cdots,s_{M}^{t}\right\}, \tag{6}\]  
\[s_{i}^{t}=\left\{o_{i},d_{i}^{t}\right\}, \tag{7}\]  
\[o_{i}=\left\{d_{i},p_{i}\right\}, \tag{8}\]  
\[d_{i}^{t}=\left\{sch_{i}^{t},ttw_{i}^{t},etet_{i}^{t}\right\}, \tag{9}\]  
where \(s_{i}^{t}\) represents the state element of task \(i\). \(o_{i}\) is the collection of the duration and profit of task \(i\). \(d_{i}^{t}\) is the collection of direct attributes. \(sch_{i}^{t}=1\) if task \(i\) has been scheduled and \(sch_{i}^{t}=0\) otherwise. \(ttw_{i}^{t}\) is the total length of the feasible time intervals at time step \(t\). \(et_{i}^{t}\) is the earliest end time of task \(i\) at time step \(t\).  
* Action space The action space \(A\) is the collection of the task number: \[A=\left\{i|i=1,2,\cdots,M\right\}\] (10)  
The action is selecting a task to schedule, and the selected task begins to be executed at the earliest of its feasible time intervals.  
* State transition Task \(a=A_{t}\) is determined based on the probabilities of each task output by the decision network at time step \(t\). The state is then updated. The original attributes \(o_{a}\) of task \(a\) remain unchanged. We set \(sch_{a}^{t+1}=0\) and update \(ttw_{i}^{t+1}\) and \(etet_{i}^{t+1}\) of other tasks according to the time period occupied by observation task \(a\) and the transition time between \(a\) and other tasks (refer to Figure 2).
* Reward Here, the reward is the increment in total profit after determining an action, as defined in the following equation:  
\[R=\left\{r_{1},r_{2},\cdots r_{t},\cdots\right\}, \tag{11}\]  
\[r_{t}=f(X_{t})-f(X_{t-1}), \tag{12}\]  
where \(X_{t}=\left\{x_{1t},x_{1t},\cdots,x_{Mt}\right\}\), \(x_{it}=\left\{0,1\right\}\), and \(x_{it}=1\) if task \(i\) is a scheduled task at time step \(t\). \(f(X_{t})\) represents the total profit at time step \(t\).