# BeiDou Short-Message Satellite Resource Allocation Algorithm Based on Deep Reinforcement Learning
## 4 Simulation and Performance Analysis
### Analysis of Simulation Results  
#### 4.2.1 Algorithm Performance Comparison  
We compared the DRL-SRA with the DQN, TI-SHA, and GA. Using different numbers of short-message terminals, the optimization effects of the total path transmission loss (\(L_{total}\)), load balancing index (\(LI\)), and service satisfaction index (\(SI\)) were analyzed. The comparison algorithm is described as follows:  
1. DQN: The Q network and target Q network with the same network structure are included, the Q network has three hidden layers, and the number of neurons in each layer is 64. The DQN strategy is shown in Equation (38). 2. TS-HIA: The Hungarian algorithm is satisfied by adding a virtual satellite and terminal. 3. GA: The population size is 50, the termination evolution number is 400, the crossover probability is 0.9, and the mutation probability is 0.01. The above four algorithms have the same weights for the three objectives, i.e., \(\alpha_{1}=\alpha_{2}=\alpha_{3}=\frac{1}{3}\). In addition, after the training has been completed, the network parameters in DRL-SRA are not updated in subsequent experiments. First of all, we evaluated the convergence of the proposed algorithm. Figure 11 shows the convergence effect of the DRL-SRA when the number of short-message terminals was 1500, the number of training steps was 500, and the number of iterations was 500. The abscissa shows the number of iterations, and the ordinate shows the objective function for Equation (18), which is compared with the DQN. In the process of comparison, the EMA was also used to smooth each original data curve for these objective function values. As shown in Figure 11, the objective function value curves of the two algorithms grew at different rates as the training process proceeded. In the first 50 iterations of DRL-SRA, the value of the objective function quickly reached 0.492. After 350 iterative periods, the value of the objective function increased to 0.957, obtaining a state of convergence. The value of the objective function of the DQN reached 0.823 in the first 400 iterations and did not continue to grow. At the end of the training period, the DRL-SRA algorithm had a higher objective function value than the DQN algorithm; that is, a better convergence performance. \begin{table}
\begin{tabular}{c c} \hline \hline
**Simulation Parameter** & **Value** \\ \hline Uplink operating frequency/MHz & 1620 \\ Downlink operating frequency/MHz & 1207.14 \\ Orbital altitude/km & 21,528 \\ Total number of short-message terminals & 1100–2400 \\ \(C_{\text{max}}\)/bit & 560 \\ \(\tau\)/s & 60 \\ Time taken up by each snapshot/s & 60 \\ \(n_{t}\) & 1000 \\ \(f_{j}\)/Mbit/s & 50 \\ \(v_{\text{max}}\) & 10 \\ Episodes & 500 \\ Steps T & 500 \\ Batch size & 8 \\ Discount factor & 0.9 \\ Soft update factor & 0.01 \\ Learning rate & 0.01 \\ Activation function & Relu \\ \(\alpha\) & 64 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: The parameters of the feature extraction network. Furthermore, the running times of the four algorithms were compared. As shown in Figure 12, the average time taken by the four algorithms to complete a strategy selection process was calculated when there were 1100, 1800, and 2400 short-message terminals. When there were 1100, 1800, and 2400 short-message terminals, the average completion time for the DRL-SRA algorithm was 3.87 s, 3.93 s, and 3.92 s, respectively. The average completion time for the DQN is slightly lower than that of the DRL-SRA because there are fewer network parameters in the DQN compared with the DRL-SRA. With an increase in the number of terminals in the TS-IHA and GA, the time required to run the algorithm obviously increased. However, the time consumed by the DRL-SRA and DQN remained unchanged, which shows that the average completion time of decision actions input into the DRL-SRA and DQN is determined by the parameters of the network (such as the number of network layers). In contrast, the network parameters are fixed after the DRL-SRA and DQN complete the training period.