# DQN-Based Shaped Reward Function Mold for UAV Emergency Communication
## 3 Improvement and Experiment  
### Improvement of Algorithm  
The simple reinforcement learning algorithm has high computational cost, which often requires millions of training data or more, and the training cost is relatively large. At the same time, it still faces the problems of poor training effect and lengthy training time. Above all, the main reason is the sparse reward problem. As for the agent, the reward can only be obtained in the final state of the game, while the intermediate state cannot be learned [12]. In the discrete space, the current stage focuses on the algorithm level, while the reward function is designed from the perspective of general theory. The reward function of DQN also mainly depends on the environment, which has a large room for improvement. The Q function is represented by a convolutional layer [13]. The training efficiency, convergence speed and stability of RL can be improved by reasonable reward function design. RL requires a lot of training resources for training, so if the experiment is repeated on the reward function, it will bring additional reward, and reward shaping refers to the training process of acceleration reinforcement learning by adding a potential energy function according to the corresponding mathematical form on the basis of the original reward function. However, in many practical cases, the agent will receive little or no reward, which will lead to learning failure and ineffective exploration [14]. The shaping function refers to the sparse reward environment and the reward environment with action evaluation [15]. There is a reward-shaping module and a control module that outputs a series of actions to maximize the reward signal [16]. Under the condition of constant external rewards, the designs of internal rewards are equivalent to adding prior human knowledge. But at the same time, strategy invariance is very important for reward shaping. The following is the frame of reward shaping, as shown in Figure 4.  
Figure 3: Main work chart. The following formula can prove that potential-based reward function shaping does not change the original strategy:  
\[\mathrm{R}^{\prime}\big{(}\mathrm{s},\mathrm{a},\mathrm{s}^{\prime}\big{)}= \mathrm{R}\big{(}\mathrm{s},\mathrm{a},\mathrm{s}^{\prime}\big{)}+\mathrm{F} \big{(}\mathrm{s}^{\prime}\big{)}. \tag{5}\]  
\[\mathrm{F}\big{(}\mathrm{s}^{\prime}\big{)}=\varphi\big{(}\mathrm{s}^{\prime} \big{)}-\gamma\varphi(\mathrm{s}). \tag{6}\]  
The shaping function based on potential energy is that the optimal strategy of reinforcement learning remains unchanged after the dynamic potential energy is introduced. And based on the DQN algorithm, this paper introduces a reward-shaping mechanism. At the same time, we classify the actions of the agent and set the progress toward the goal as a positive reward, and the rest is classified as a negative or no reward. When the agent takes the next action according to the action value function, it will preferentially select the action that develops toward the goal progress. The reward function based on dynamic potential energy can be expressed as  
\[\mathrm{F}\big{(}\mathrm{s},\mathrm{t},\mathrm{s}^{\prime},\mathrm{t}^{\prime }\big{)}=\mathrm{y}\varphi\big{(}\mathrm{s}^{\prime},\mathrm{t}^{\prime}\big{)} -\varphi(\mathrm{s},\mathrm{t}). \tag{7}\]  
And the formula proves that the Q value of the shaping function is independent of the action selection, and the optimal strategy remains unchanged:  
\[ \begin{array}{l}Q^{*}_{i,\varphi}(s,a)=\sum_{s^{\prime}}Pr(s^{\prime}|s,a) U_{i,\varphi}(s^{\prime})=\sum_{s^{\prime}}Pr(s^{\prime}|s,a)(U_{i}(s^{\prime})- \varphi(s,t))=\\ \sum_{s^{\prime}}Pr(s^{\prime}|s,a)U_{i}(s^{\prime})-\sum_{s^{\prime}}Pr(s^{ \prime}|s,a)\varphi(s,t)=Q^{*}_{i}(s,a)-\varphi(s,t).\end{array}  \tag{8}\]  
where F represents the additional reward function, which is the state value after the action is taken at the next moment, and the formula indicates that both F and will change dynamically with time.