# DQN-Based Shaped Reward Function Mold for UAV Emergency Communication
## 3 Improvement and Experiment
### Improvement of Algorithm
This paper adds the reward shaping on DQN, and the following is the pseudocode, as Algorithm 1 shows. ```
1. Initialize replay memory to capacity
2. Initialize action-value function Q with random weights
3. Initialize target action-value function with weights
4. For episode = 1, X do
5. Update the position of UAV
6. For step = 1, M do
7. Select an action with softmax policy
8. Set the reward based on the step
9. Implement and move on to the next state
10. Store transition
11. Gradient descent policy to loss function to improve
12. Every K step reset
13. End for

Figure 4: Reward shaping. ### Experiment

Our experiments were performed on a computer with an Intel Core i7-8700K CPU (Intel, Santa Clara, CA, USA) and an NVIDIA GeForce RTX 3060 (NVIDIA, Santa Clara, CA, USA). The experiments were run on the Windows system, while the machine learning-related components made use of PyTorch 1.8.1. In this paper, the algorithm was designed and simulated based on the environment of Atari from Open AI. We used Open AI's Atari platform to build a new experimental environment. Only one UAV in the experiment was able to move left and right at different speeds and was able to explore different areas. In the experiment, baseline3 of PyTorch was used as the experimental baseline, and the DQN algorithm was compared with the DQN algorithm, adding reward shaping. The hyperparameter settings are shown in Table 1:

In this experiment, 2,000,000 rounds of training are set for the UAV agent. The comparison results between the improved algorithm with the reward shaping and the baseline algorithm are shown in Figures 5 and 6. As shown in Figures 5 and 6, after 2 million training sessions, the final DQN with a reward function can obtain a higher reward than before. The improved algorithm obtained \begin{table}
\begin{tabular}{c c} \hline \hline
**Hyperparameter** & **Number** \\ \hline Seed & 25 \\ Entry 2 & data \\ Learning\_rate & 0.0001 \\ Grad\_clipping\_value & 5 \\ Replay\_buffer\_size & 1,000,000 \\ Batch\_size & 32 \\ Gamma & 0.98 \\ Discount factor & 0.99 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Number of hyperparameters. Figure 5: Training reward. Figure 6: Training reward over episodes. a reward of 7 on the 500,000 th training, while the original DQN algorithm still obtained a reward of about 5 after training on 2 million. This means that the improved algorithm is able to train the UAV agent faster, make the agent adapt to the new environment, and receive the reward quicker. In summary, compared with the baseline algorithm and the traditional DQN algorithm, the final DQN with a reward function has better performance and convergence, and it can enable the UAV agent to obtain better scores in shorter training rounds. ## 4 Conclusions and Future Work

Aiming at the problem of using UAVs as communication nodes in emergency situations, this paper constructs a virtual environment through abstract modeling. At the same time, based on DQN and combined with the dynamic reward function-shaping method, the reward function is set to improve. At the same time, the hyperparameters are modified repeatedly according to the experimental results to improve the performance of the algorithm. Experiments show that this algorithm can optimize the performance of the original algorithm, shorten the training time, improve the training efficiency, and provide a reference for the autonomous location selection of UAVs under emergency conditions. A high-performance intelligent decision-making approach for wargames that still has some limitations was presented in this paper. This paper does not evaluate the merits and demerits of the reward function, and it only verifies the positive effect of the reward function on improving the training efficiency without verifying whether it will affect the final training effect of the agent. So, the next step is to optimize the shaping method of the dynamic reward function and make a reasonable design for specific problems.