## 4 Computational Experiments
### Comparison with the State-of-the-Art Algorithms  
To verify the effect of DRLLA, we compare it with seven state-of-the-art algorithms for satellite scheduling problems or other combination optimization problems, including Transformer (TRFM) [42], PrtNet [31], RLPT [32], a reinforcement learning-based genetic algorithm (RLGA) [43], a hybrid discrete artificial bee colony algorithm (HDABC) [21], an improved simulated annealing algorithm (ISA) [37], and an improved ant colony optimization algorithm (IACO) [44]. TRFM, PrtNet, and RLPT are neural network models with encoder-decoder structures, and RLGA, HDABC, ISA, and IACO are heuristic algorithms with improved search mechanisms. The heuristic algorithms maintain their original settings. The comparison experiments contain two parts: first, these algorithms are tested on the dataset _Testing_R1_ to validate the effectiveness of DRLLA under different problem scales; then, they are tested on the other three datasets _Testing_R2_, _Testing_R3_, and _Testing_G_, whose task distribution is different from _Testing_R1_, to demonstrate the generalization ability of DRLLA in various instances. Figures 8 and 9 and Table 7 show the detailed testing results of the first comparison experiment on the testing dataset _Testing_R1_. The boxplot graphs in Figure 8 illustrate the profit ratio distribution of solutions obtained by different algorithms. In these boxplot graphs, the top lines denote maximum values, the bottom lines denote minimum values, the middle boxes denote interquartile ranges, the red lines denote medians, the blue dashed lines denote mean values, and the red circles denote outliers. As shown in these boxplot graphs, the profit ratio distributions of DRLLA are highly concentrated, and its median lines are higher than the comparison algorithms, demonstrating that DRLLA can obtain solutions with the highest profit ratio when handling various AOS scheduling instances. The profit ratio distributions of TRFM are also concentrated, and its median lines are higher than other comparison algorithms but lower than DRLLA. The profit ratio distributions of the remaining algorithms are relatively scattered. In general, the stability and generalization of DRLLA and TRFM are superior to the remaining comparison algorithms, and DRLLA can obtain solutions with higher profit ratios than the comparison algorithms. Figure 7: Training process of DRLLA. (**a**) The reward curve and the learning rate curve in the whole training process. (**b**) The reward curve and the learning rate curve in the second training epoch. Table 7 presents the detailed testing results of the first comparison experiment on the dataset _Testing_R1_. The indicators include the mean profit ratio (MPR), mean profit (MP), mean completion number (MCN), and mean computational time (MCT), and the best values are bolded. Figure 9 depicts the MP curves and MCN curves of different algorithms. The testing results are presented as follows:  
1. As for the DRL-based algorithms, DRLLA can obtain better solutions with the higher profit and completion number than these comparison algorithms in a relatively small amount of computational time under different task scales; TRFM, only second to DRLLA, can also gain acceptable solutions with a little more computational time; although RLPT has the shortest computational time in these testing cases, it is hard to achieve satisfactory solutions. 2. Among the heuristic algorithms, ISA and HDABC can obtain relatively acceptable solutions, but the computational time of HDABC far exceeds other algorithms; RLGA and IACO perform worse than the other algorithms in terms of solution quality. Comparing with the DRL-based algorithms, these heuristic algorithms need more computational time due to the huge computation burden of population iteration. 3. As the task number increases, the MPR values of all algorithms continuously decrease, while the MP and MCN values of most algorithms keep growing. The main reason is