# BeiDou Short-Message Satellite Resource Allocation Algorithm Based on Deep Reinforcement Learning
## 3 Algorithm Design
### DRL-SAR Algorithm
Equations (34) and (35) can be used to obtain the optimal action function \(Q^{*}_{\pi}(s_{t},a_{t})\) and the optimal state function \(V^{*}(s_{t})\):  
\[Q^{*}(s_{t},a_{t})=r(s_{t},a_{t})+\gamma\sum_{s_{t+1}\in\mathbf{S}}P(s_{t+1}| s_{t},a_{t})\max_{a_{t+1}\in A}Q_{\pi}(s_{t+1},a_{t+1}) \tag{36}\]  
\[V*(s_{t})=\max_{a_{t}\in A}Q^{*}(s_{t},a_{t}). \tag{37}\]  
The optimal strategy is \(\pi^{*}(s_{t}|a_{t})\), the corresponding optimal action is \(a^{*}_{t}\), and its expression is:  
\[a^{*}_{t}=\underset{a_{t}\in A}{\operatorname{argmax}}Q_{\pi}(s_{t},a_{t}). \tag{38}\]  
Each snapshot in the SMSCS corresponds to the state action function \(Q_{\pi}(s_{t},a_{t})\), the state function \(V_{\pi}(s_{t})\), and the optimal action \(a^{*}_{i}\). The optimal strategy \(\pi^{*}\) for transferring from the initial state \(s_{0}\) to the state \(s_{t+1}\) is:  
\[\pi^{*}=\{\pi^{*}(s_{0}|a_{0}),\cdots,\pi^{*}(s_{t}|a_{t})\}. \tag{39}\]  
However, it is usually challenging to determine the state transition probability of the SMSCS, the state of the resource allocation problem is continuous, and the scale of the state set is large. We include the DDPG in the action decision mechanism of the DRL-SAR. Through the introduction of actor-critic, the continuous spatial state is parameterized. First, by introducing \(\widetilde{V}_{\pi}\) and \(\widetilde{Q}_{\pi}\), the state function and the action function are approximated. They are:  
\[\widetilde{V}_{\pi}(s_{t},\theta) \approx V_{\pi}(s_{t}) \tag{40}\] Figure 6:0  
Similarly, to approximate the strategy function, we have:  
Figure 6:1  
where \(\theta\) and \(\omega\) are the weight parameters in the network. DDPG includes four networks, namely, the target critic network, the critic network, the target actor network, and the actor network. The basic idea is that the strategy gradient is approximated by the strategy function and the value function. In this process, the strategy function can evaluate and optimize the strategy based on the value function. The optimized strategy function can also make the value function reflect the value of the state more accurately, and the functions can influence each other to obtain the optimal solution [44]. The actor network has a policy function and is responsible for agent selection and environment interactions. The critic network has a value function and is used to evaluate the behavior of the actor. In the DRL-SAR, the main functions of the four networks are as follows. 1. The target critic network is responsible for calculating targets \(\widetilde{V}_{\pi}(s_{t+1},\theta^{\prime})\) and \(\widetilde{Q}_{\pi}(s_{t+1},a_{t+1},\theta^{\prime})\) based on the state sampled in the experience replay pool. Parameter \(\theta^{\prime}\) in the target critic network is regularly copied from \(\theta\) in the critic network, i.e., Figure 6:2 (43) where \(\mu\) is the updated coefficient, and \(0<\mu\ll 1\). 2. The critic network is responsible for iteratively updating parameter \(\theta\) in the value function and calculating the current values of \(\widetilde{V}_{\pi}(s_{t},\theta)\) and \(\widetilde{Q}_{\pi}(s_{t},a_{t},\theta)\). The loss function of the critic network can be defined as: Figure 6:3 (44) where \(N\) is the number of samples drawn from the experience playback pool and \(N>0\). 3. The target actor target network is responsible for selecting the optimal action \(a_{t+1}\) based on the state \(s_{t+1}\) sampled in the experience replay pool. The parameter \(\omega^{\prime}\) in the target actor target network is periodically copied from \(\omega\) in the actor network, i.e., Figure 6:4 (45)
4. The actor network is responsible for iteratively updating parameter \(\omega\) in the strategy function. According to the state \(s_{t}\) of the SMSCS in the current snapshot \(t\), it selects the current action \(a_{t}\), obtains the reward \(r(s_{t},a_{t})\), and determines the initial state \(s_{t+1}\) of snapshot \(t+1\).