# BeiDou Short-Message Satellite Resource Allocation Algorithm Based on Deep Reinforcement Learning
## 3 Algorithm Design
### DRL-SAR Algorithm
Using Equation (18), the gain of the optimization objective can be expressed as: \[\Delta U=U(t+1)-U(t).\] (31) For \(r(s_{t},a_{t})\), there is: \[r(s_{t},a_{t})=\left\{ \begin{array}{ll}r_{h},\,\Delta U>0\\ r_{l},\,\Delta U\leq 0\end{array} \right..\] (32)  
When \(\Delta U>0\), the system revenue is increasing, \(r(s_{t},a_{t})=r_{h}\). When \(\Delta U\leq 0\), the system revenue is unchanged or decreasing, \(r(s_{t},a_{t})=r_{l}\), and \(0\leq r_{l}<r_{h}\leq 1\). For the short-message satellite resource allocation scenario, the proposed DRL-SAR framework is shown in Figure 6. The basic process is that the short-message satellite and the short-message terminal continuously interact to determine the current state of the environment and transmit environmental information to the ground station. Based on the state of the current system, the ground station sends the action instructions to the short-message satellite to be executed. After executing the instructions, the system environment moves from the current state to the next state and receives rewards through the environmental feedback. At the same time, the ground station stores the quadruple \(\langle s_{t},s_{t+1},a_{t},r(s_{t},a_{t})\rangle\) as a sample in the memory pool, which is composed of the current environment state, the next state, executes actions and feeds back rewards. In the DRL-SAR training process, the training speed can be accelerated through experience replay. The above algorithm framework is divided into two steps:  
STEP 1: DRL-SAR input data reconstruction  
The deep learning process carried out in the DRL-SAR involves the use of a feature extraction network to extract state features. The essence of the feature extraction network is the use of convolutional neural networks (CNNs). CNNs usually require input data to conform to the form of a graph tensor. For the state \(s_{t}=\left\{\mathbf{L}_{t},\mathbf{A}_{n\times n_{t}}^{r},\mathbf{L}_{t} \right\}\), \(\mathbf{L}_{t}\), \(\mathbf{A}_{n\times n_{t}}^{r}\), and \(\mathbf{L}_{t}\) are split into one-dimensional vectors, and \(s_{t}\) is transformed into \(n+2\) graph tensors of \(\alpha\times\alpha\) through operations such as zero padding and matrix transformation. The resource allocation of \(n\) short-message satellites and information about the path transmission loss of the terminal in the subregion and the service satisfaction of the short-message terminal are recorded. The graphic tensor outputs a one-dimensional vector with dimensions of \(14\times v_{\max}\) through the feature extraction network. The one-dimensional vector records the  
Figure 6: DRL-SAR algorithm framework. characteristics of each short-message satellite for its covered subregion and inputs this information into the action decision mechanism. STEP 2: DRL-SAR training and update  
In state \(s_{t}\) of the SMSCS, the ground station sends the instruction to execute action \(a_{t}\). At this time, it receives a reward \(r(s_{t},a_{t})\) and is transferred to state \(s_{t+1}\). Assuming that the initial state of the SMSCS is \(s_{0}\), strategy \(\pi\) is transferred from the initial state \(s_{0}\) to the state \(s_{t+1}\), as follows:  
\[\pi=\{\pi(s_{0}|a_{0}),\cdots,\pi(s_{t}|a_{t})\}. \tag{33}\]  
In DRL-SAR, for \(\pi(s_{t}|a_{t})\), the action value function \(Q^{\pi}(s_{t},a_{t})\) is used to evaluate the benefits of action \(a_{t}\) in the current state \(s_{t}\) of the SMSCS. According to the Bellman equation, the action function is:  
\[Q_{\pi}(s_{t},a_{t})=r(s_{t},a_{t})+\gamma\sum_{s_{t+1}\in\mathbf{S}}P(s_{t+1}| s_{t},a_{t})V_{\pi}(s_{t+1}). \tag{34}\]  
The state function is:  
\[V_{\pi}(s_{t})=\sum_{a_{t}\in A}P(a_{t}|s_{t})Q_{\pi}(s_{t},a_{t}) \tag{35}\]  
where \(\gamma\) is the attenuation factor. \(P(s_{t+1}|s_{t},a_{t})\) is the probability that the SMSCS transfers to \(s_{t+1}\) under state \(s_{t}\) and action \(a_{t}\). \(P(a_{t}|s_{t})\) is the probability of performing action \(a_{t}\) under state \(s_{t}\).