# Revising the Observation Satellite Scheduling Problem Based on Deep Reinforcement Learning
## 3 Solving Method
### DRL-Based Method for Optimization
#### 3.2.3 Task Scheduling Method  
As an off-policy DRL algorithm, the DDPG allows us to train the EOS task scheduling network without knowing the prior information. Moreover, it is noted that in the training process, valid observation tasks are sorted according to the start time of the VTW.  
(1) Network architecture  
As mentioned above, the model consists of two separate networks, which are, respectively, the actor and the critic. Figure 6 shows the network architecture.  
Figure 5: Illustration of DDPG algorithm.  
The input of the network is a sequence of state information (denoted as s), including the start and end time of the VTW, start and end time of the OW, priority and the observation duration. The L2-normalization is applied to the state input layer and the output of the architecture has two parts: an estimation Q-value of total expected profit (denoted as \(Q(s,a\mid\theta_{q})\)) as critic and the policy (denoted as \(\mu(s\mid\theta_{\mu})\)) for the task as actor. The output value of the actor network is mapped to \([-1,1]\) by applying a non-linear activation function _Tanh_, which has the same value range as the action space in Equation (19).  
\[Tanh(x)=\frac{e^{x}-e^{-x}}{e^{x}+e^{-x}} \tag{22}\]  
(2) Reward function design  
In the task scheduling problem, the EOS must guarantee that high-priority observation tasks in the execution sequence can be carried out. Simultaneously, to meet actual requirements, conflicting tasks are not permitted. Therefore, the reward function, denoted by \(r_{t}\), is modeled to supervise the task scheduler to achieve an optimal result. In Section 2.2, we propose the optimization objective function \(f\) in Equation (13). Thus, to acquire the largest global profit, the reward function is formulated as follows:  
\[r_{t}=r(s_{t},a_{t})=\lambda(f_{t}-f_{t-1}) \tag{23}\]  
where \(\lambda\) is an amplification factor to improve a good action's reward and accelerate the process of training the policy network.  
(3) Training method  
In the training process, the scheduling network makes a decision when to start observing targets, and then receives an instant reward based on the reward function. For considering the future reward given by the current policy, an accumulated reward with a discount factor \(\gamma\) is used to estimate the Q-value in the critic network. Meanwhile, the actor could learn to make an optimal scheduling policy based on the actor network. During each training step, the critic improves its prediction ability by gradient descent of error between actual profit and estimated profit. The actor updates its parameters based on the prediction from the critic. To improve exploration efficiency, noise is introduced into the decision as follows:  
\[a_{t}=\mu(s_{t}\mid\theta_{\mu})+\alpha\mathcal{N}(0,\sigma^{2}) \tag{24}\]  
The noise obeys the normal distribution and \(\alpha\) is the attenuation coefficient, \(\sigma\) is the standard deviation. At the end of each training episode, a soft synchronization method is adopted to update network parameters, which is formulated as follows:  
\[\left\{\begin{array}{l}\theta^{\prime}_{\mu}=\tau\theta_{\mu}+(1-\tau)\theta ^{\prime}_{\mu}\\ \theta^{\prime}_{q}=\tau\theta_{q}+(1-\tau)\theta^{\prime}_{q}\end{array}\right. \tag{25}\]  
where \(\tau\) is the coefficient of synchronization, which is used to make the target networks slowly track the learned networks, significantly improving the stability of learning [45]. The pseudocode for the scheduling algorithm is shown in Algorithm 2:  
Figure 6: Architecture of DDPG actor and critic network.  
(4) Resolving conflicts  
After the OW is determined by the DDPG, conflicts may still remain. Constraint checking and conflict resolving are performed, and the breadth first search (BFS) is adopted to check the task sequence and update the list \(\{x_{1},x_{2},\ldots,x_{N}\}\) to \(\{0,1,1,0,\cdots\}\). After resolving conflicts, the profit obtained in the current step is calculated according to Equation (13).