## 4 Computational Experiments
### Training Process  
The parameter settings of the DRLLA are listed in Table 6, containing the network parameters and the training parameters. The total number of training epochs is 10. In every epoch, the network is trained on the training dataset \(Training\_R1\) in the order of 200-task, 150-task, 100-task, and 50-task samples, resulting in its reward curve in the training process appearing in a stepped shape. One training epoch contains 80 training steps because of the training parameter settings.  
The training process is shown in Figure 7. As depicted in Figure 7a, the reward curve fluctuates sharply in the first three training epochs but trends toward stability and convergence in the last three training epochs. The learning rate curve overall shows a downward trend but rises multiple times.  
\begin{table}
\begin{tabular}{l} \hline \hline
**Actor network** \\ \hline Static embedding layer: Linear(input\_size = 2, out\_size = 128), \\ LSTMcell(input\_size = 6, hidden\_size = 128), \\ Linear(input\_size = 256, out\_size = 128); \\ Static encoder: MultiHeadAttention(embedded\_dim = 128, num\_heads = 2, dropout = 0.1), \\ Conv1d(input\_size = 128, output\_size = 128, kernel\_size = 1); \\ Dynamic embedding layer: Linear(input\_size = 6, out\_size = 128); \\ Dynamic encoder: LSTMcell(input\_size = 128, hidden\_size = 128), \\ Linear(input\_size = 256, out\_size = 128); \\ Decoder: LSTMcell(input\_size = 6, hidden\_size = 128), \\ Local attention (which only considers the unselected tasks with VTWs in the current orbit or the next one at every decoding step). \\ \hline
**Critic network** \\ \hline Embedding layer: the same as the static embedding layer; \\ Encoder: the same as the static encoder; \\ Decoder: Conv1d(input\_size = 128, output\_size = 64, kernel\_size = 1), \\ Conv1d(input\_size = 64, output\_size = 16, kernel\_size = 1), \\ Conv1d(input\_size = 16, output\_size = 128, kernel\_size = 1). \\ \hline
**Training parameters** \\ \hline Actor network: adaptive learning rate (whose initial value is \(lr^{1}=0.1\) and decay rate is \(\gamma=0.999\)); \\ Critic network: fixed learning rate (\(lr_{c}=0.001\)); \\ Number of training epochs: \(Epoch=10\). \\ \hline \hline \end{tabular}
\end{table}
Table 6: Parameter settings of DRLLA.  
\begin{table}
\begin{tabular}{l c c c c c} \hline \hline
**Dataset** & **Task Distribution** & **Latitude Range** & **Longitude Range** & **Number of Tasks** & **Number of Instances for Each Type** \\ \hline \multirow{2}{*}{
\begin{tabular}{c} _Training\_R1 \\ _Testing\_R1 \\ _Testing\_R1 \\ _Testing\_R2 \\ _Testing\_R3 \\ _Testing\_G_ \\ \end{tabular} } & Regional distribution & [3\({}^{\circ}\) N, 53\({}^{\circ}\) N] & [73\({}^{\circ}\) E, 133\({}^{\circ}\) E] & 50, 100, 150, 200 & 2560 \\  & Regional distribution & [3\({}^{\circ}\) N, 53\({}^{\circ}\) N] & [73\({}^{\circ}\) E, 133\({}^{\circ}\) E] & 50, 75, 100, 125, 150, 128 \\  & Regional distribution & [3\({}^{\circ}\) N, 53\({}^{\circ}\) N] & [3\({}^{\circ}\) E, 63\({}^{\circ}\) E] & 50, 100, 150, 200, 250 & 128 \\  & Regional distribution & [3\({}^{\circ}\) N, 53\({}^{\circ}\) N] & [63\({}^{\circ}\) W, 123\({}^{\circ}\) W] & 50, 100, 150, 200, 250 & 128 \\  & Global distribution & [70\({}^{\circ}\) S, 70\({}^{\circ}\) N] & [180\({}^{\circ}\) W, 180\({}^{\circ}\) E] & 50, 100, 150, 200, 250 & 128 \\ \hline \hline \end{tabular}
\end{table}
Table 5: Details of the training and testing datasets.  
Figure 7b shows the details of the training process in the second training epoch (from the 80 th training step to the 160 th training step), and the learning rate rises obviously five times in this epoch. It can be seen from these two figures that the adaptive learning rate strategy can balance the early exploration and the later convergence of the actor network and the actor network is trained well through the actor-critic training algorithm with the adaptive learning rate.