# Revising the Observation Satellite Scheduling Problem Based on Deep Reinforcement Learning
## 4 Experimental Simulation
### Results and Discussion  
According to the analysis in Section 3, task clustering is an effective approach promoting scheduling efficiency and saving EOS resources. For the task clustering preprocess phase, the proposed minimum clique partitioning algorithm consists of two steps. Firstly, it establishes the clustering graph by considering constraint conditions, and then it partitions original tasks into minimum clique aiming to update the task execution sequence. Figure 8 shows the clique partition result of 50 original tasks, where \(\{t_{1},t_{2},t_{3},t_{4}\}\) are inaccessible. The interconnected vertexes such as \(\{t_{29},t_{34},t_{37}\}\) can be seen as a merged task. \begin{table}
\begin{tabular}{c c c} \hline \hline
**Parameters** & **Value** \\ \hline Semi-major axis of orbit \(a\) & 7000 km \\ Orbital eccentricity \(e\) & 0 \\ Orbital inclination \(i\) & 60\({}^{\circ}\) \\ Longitude of ascending node \(\Omega\) & 285\({}^{\circ}\) \\ Argument of perihelion \(\omega\) & 0\({}^{\circ}\) \\ Mean anomaly \(M_{0}\) & 0\({}^{\circ}\) \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Orbital element settings. \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Parameters** & **Value** & **Parameters** & **Value** \\ \hline \(M\) & 600 & \(c_{i}\) & 1 \\ \(FOV\) & 10\({}^{\circ}\) & \(e_{i}\) & 1 \\ \(maxT\) & 150 s & \(\varepsilon_{uv}\) & 0.5 \\ \(max\theta\) & \(\pm\)40\({}^{\circ}\) & \(E\) & 1200 \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Constraint conditions in the task scheduling. Figure 7: Observation target point map. Furthermore, other groups of comparisons with 100, 150 and 200 original tasks were simulated, as shown in Figure 9. The results indicate that 4, 14, 19 and 25 corresponding invalid tasks are eliminated and 36, 70, 102 and 133 clustering tasks are generated. Moreover, it is found that the task clustering running time is less than 0.5 s. Thus, it could be stated that the improved graph-based minimum clique partition algorithm achieves the desired objective and reduces the task scale obviously and quickly. On the basis of the results from the clustering, the performance of the RL-based algorithm is examined, where the Pytorch deep learning framework is utilized to implement the scheduling networks. Table 4 gives the hyperparameters adopted in the training process. The DRL algorithm runs on an Nvidia GTX1660 GPU and Intel i5-8400 CPU device. In each training episode, there are 100 exploratory steps, and the maximum training episode is 400, which means the total amount of experience is 40,000. The experience memory pool keeps updated data with a size of 3000. In each training step, experiences are randomly sampled from the memory pool with a batch size of 32. Parameters of the actor and the critic networks are initialized randomly before training, and the models are trained with the ADAM optimizer [46]. \begin{table}
\begin{tabular}{l c} \hline \hline
**Parameters** & **Value** \\ \hline Learning rate for the critic & 0.002 \\ Learning rate for the actor & 0.001 \\ Discount factor \(\gamma\) & 0.95 \\ Memory capacity & 3000 \\ Batch size & 32 \\ Noise attenuation coefficient \(\alpha\) & 0.9995 \\ Standard deviation \(\sigma\) & 0.1 \\ Soft synchronization coefficient \(\tau\) & 0.001 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Training hyperparameter settings. Figure 8: Result of the minimum task clique partition. Figure 9: Performance demonstration of clustering algorithm. The task scheduling profit \(p\) proposed in Section 2.2 is selected as the evaluation indexes of the training performance. As shown in Figure 10 the episode maximum profit (gray curves) and average profit (blue curves) are selected to demonstrate the algorithm's performance in each training episode. The trendline (red curves) indicates that the task scheduling network could achieve a higher profit in every simulation, which demonstrates that the proposed method is working.