# Revising the Observation Satellite Scheduling Problem Based on Deep Reinforcement Learning
## 4 Experimental Simulation
### Results and Discussion
The profit score of scheduling is fluctuates upward with increasing episodes, which shows that the network could learn from experience and achieve a better and more stable scheduling policy with higher profit. Additionally, the trained DRL scheduling network gives a solution with observation profit of 3.25 for 50 original tasks, but the profit of allocating 100, 150 and 200 original tasks decreases it to 2.31, 1.83 and 1.56, respectively. Therefore, the desired profit is greatly influenced by the number of original tasks, representing the complexity of the EOSSP. The main possible reason for this is that the observation tasks are under the limitation of the VTW, energy, storage and other constraint conditions, and task scheduling becomes more and more difficult while the number of tasks increases. A compromising solution is to ensure tasks with higher priority are executed. Figure 11 demonstrates the observation time period selection of the DDPG methods in the task scheduling phase for 50 original tasks. The horizontal axis represents the time and the vertical axis represents different clustered observation tasks. The left figure shows the initial state, where all tasks are arranged at the start of the VTW. The scheduling result is shown in the right figure, where valid tasks (marked as green) are executed, and the invalid tasks (marked as red) are not performed because of constraint conflicts. Moreover, a series of comparisons with the genetic algorithm (GA), the simulated annealing algorithm (SA) and the GA-SA hybrid algorithm is performed to examine the superiority of the DDPG method. Note that the GA-SA hybrid algorithm has been validated in our previous work [24]. In addition, the DDPG without considering the task clustering is simulated to withstand the effect of the preprocess, and it is defined as NTC-DDPG. Correspondingly, the DDPG with task clustering is represented by TC-DDPG. Comparison results are given in Figure 12. It is indicated that the TC-DDPG method always gives a good optimization result compared with other methods. Interestingly, it was found that for 50 tasks, NTC-DDPG has a relatively low profit, but with the increase in the task number, NTC-DDPG exceeds the non-DRL methods, even though NTC-DDPG does not take the task clustering into account. This indicates that the DDPG could contribute to  
Figure 10: Profit curves of training performance. a high EOS efficiency. It is necessary to point out that the non-DRL methods also include the task clustering preprocess. Additionally, it is obvious that traditional optimization algorithms achieve a worse profit with the increases in the number of original tasks, and the SA algorithm is even out of work in the 150 and 200 task-scale situations. Hence, the DRL method has practical advantages when addressing a large-scale EOSSP. The results shown in Table 5 illustrate the feasibility of the proposed RL method and this method is rather competitive in the EOSSP, with good profit performance and adaptability to practical applications. In addition, the task clustering algorithm greatly improves the DDPG algorithm, obtaining a higher observation profit and reducing the running time, with a preprocess time of less than 0.5 s.  
Figure 11: Observation time period selection results. Figure 12: Optimization performance contrast. \begin{table}
\begin{tabular}{c c c c c c c c c} \hline \hline  & \multicolumn{4}{c}{**Profit**} & \multicolumn{4}{c}{**Running Time (s)**} \\ \hline
**Task Numbers** & **50** & **100** & **150** & **200** & **50** & **100** & **150** & **200** \\ \hline TC-DDPG & 3.25 & 2.31 & 1.83 & 1.56 & 163.7 & 228.3 & 295.3 & 362.4 \\ NTC-DDPG & 3.11 & 2.17 & 1.77 & 1.36 & 181.1 & 255.2 & 352.1 & 454.7 \\ GA-SA & 3.14 & 1.95 & 1.54 & 1.13 & 155.3 & 163.7 & 362.9 & 530.2 \\ GA & 2.96 & 1.56 & 1.29 & 0.79 & 3.7 & 6.4 & 10.3 & 15.6 \\ SA & 2.79 & 1.59 & 1.07 & 0.63 & 47.1 & 83.4 & 379.2 & 702.3 \\ \hline \hline \end{tabular}
\end{table} 
Table 5: Algorithm performance comparison.