# BeiDou Short-Message Satellite Resource Allocation Algorithm Based on Deep Reinforcement Learning
## 3 Algorithm Design
### DRL-SAR Algorithm  
When the short-message satellite receives a task request from short-message terminals in all subregions, it forwards the information, such as the short-message satellite status and the short-message terminal task request, to the ground station. The ground station determines the optimal strategy and returns the information to the short-message satellite system by using the satellite resource allocation algorithm to respond to the task request. Because the task request from the short-message terminal occurs randomly in the time dimension, the state transition probability of the system is difficult to calculate, and it is challenging to solve the problem by using the traditional value iteration method. The critical problem is determining the optimal strategy for allocating the short-message satellite response to short-message terminal tasks. As one of the basic methods of DRL, DQN is widely used in many optimization fields and can be used to effectively deal with tasks with a large state space and action space. However, because the output of DQN is discrete and the resource capacity of the short-message satellite and the energy of the short-message terminal are continuous variables, in order to meet the requirements for DQN input, the above continuous variables need to be quantized into discrete variables. This causes the action space to grow exponentially, which makes it challenging to guarantee the performance of DQN. In order to solve the resource allocation problem of the short-message satellite system in continuous space, we propose a satellite short-message resource allocation algorithm based on deep reinforcement learning (DRL-SAR). The DRL-SAR algorithm takes Equation (18) as the optimization goal, models the short-message satellite as the agent, considers the response to the short-message terminal request as the action of the agent, and models the satellite-to-ground link as an interactive environment. The three elements of the DRL-SAR --status, action, and reward--can be described as follows. 1. Status Suppose the state space of the DRL-SAR is \(\mathbf{S}=\{s_{1},s_{2},\cdots,s_{t}\}\), where \(s_{t}\) is defined as the system state under snapshot \(t\). For \(s_{t}\) there is: \[s_{t}=\left\{\mathbf{L}_{t},\mathbf{A}_{t\times n_{t}}^{r}\mathbf{L}_{t}\right\}\] (29) where \(\mathbf{L}_{t}=\{L_{t,1},L_{t,2},\cdots,L_{t,v}\}\). \(L_{t,v}\) is the total path transmission loss of the terminal in subregion \(v\) under \(s_{t}\). \(\mathbf{A}_{n\times s}^{r}\) is the resource matrix under \(s_{t}\), which involves the use information of satellite resources, such as resource occupancy and resource allocation. In \(\mathbf{L}_{t}=\{L_{t,1},L_{t,2},\cdots,L_{t,v}\}\), \(L_{t,v}\) is the satisfaction degree of the terminal in area \(v\) under \(s_{t}\). Since satellites and terminals are mobile, their levels of mobility are mapped as the change in distance between the subregion and the covered satellite and then further mapped to show the path transmission loss of the entire region. The mobility mentioned above only affects the state change of the DRL-SAR, but does not affect the overall framework design of the algorithm. 2. Action Suppose the action space is \(\mathbf{A}\), when all possible resource allocation decisions \(\mathbf{b}_{n}(t)\) of the satellite under \(s_{t}\) are included: \[a_{t}=\left\{\mathbf{b}_{1}(t),\mathbf{b}_{2}(t),\cdots,\mathbf{b}_{n}(t)\right\}\] (30) where \(b_{n}(t)\) is the resource allocation decision of \(SAT_{n}\) under snapshot \(t\). \(\mathbf{b}_{n}(t)=\left[p_{1,n},p_{2,n},\cdots,p_{v,n}\right]\), where \(p_{i,n}\) is the proportion of resources allocated to subregion \(a_{i}^{r}\) by the \(SAT_{n}\) using its own resources, and there is \(\sum\limits_{i=1}^{v}p_{i,n}=1\). The allocation ratio has a continuous quantity, so it is necessary for the DRL-SAR to effectively deal with the continuous action space to solve the action dimension problem. 3. Reward Suppose that, under status \(s_{t}\), the reward obtained by the system is \(r(s_{t},a_{t})\).