# Autonomous Obstacle Avoidance with Improved Deep Reinforcement Learning Based on Dynamic Huber Loss
## 2 Related Works
### RL
#### 2.1.3 Reward Function  
The learning goal of the drone is to obtain a strategy for long-term reward maximization, which is related to its instant rewards as follows:  
\[R=\sum_{t=0}^{\infty}\gamma^{t}R(s_{t})\ 0\leq\gamma<1 \tag{2}\]  
where \(R(s_{t})\) is the instant reward after the drone takes an action, and \(R\) is the long-term reward after the agent performs a set of actions. However, directly adding instant rewards would generate no tendencies in an infinite time series and result in an endless loop of states. To deal with this issue, a discount coefficient, whose value is between 0 and 1, is introduced to make the drone pay more attention to the instant reward when interacting with its environment.  
This study aims to guide the UAV to reach the target location safely and efficiently within a limited area, around which clear mission objectives and constraints are set. In terms of goal decomposition, rapidity is achieved by a fixed penalty of \(-1\) per step to minimize the task completion time; safety is ensured by a collision penalty of \(-50\) and an out-of-bounds penalty of \(-20\); and accuracy is achieved by giving a terminal reward of 100 when the distance between the UAV and the target is less than a threshold of 1.0. As for the environment, the UAV is restricted to operate within \(|x|\leq 5\) and \(|y|\leq 5\), and there are dynamic obstacles, and successful arrival, collision, or boundary crossing will terminate the round. Based on the above objectives, the reward function adopts a segmented form and gives corresponding rewards based on the Euclidean distance between the UAV and the target whether it collides with the obstacle or not and whether it crosses the boundary or not. From the parameter interpretability and theoretical analysis, the time penalty \(-1\) can motivate the UAV to reduce redundant actions, and either too large or too small a penalty value will harm the strategy; the safety penalty \(-50\) is significantly higher than the out-of-bounds penalty \(-20\), and the weighting ratio of 2.5 highlights the principle of \"safety first\"; and the termination reward of 100 needs to be sufficiently large so that it can cover the potential cumulative punishment to avoid the UAV giving up exploration due to the fear of punishment. This segmented design of the reward function has sparse rewards to facilitate initial strategy exploration and a high penalty mechanism to force drones to avoid dangerous behaviors. The reward function is defined in the UAV autonomous obstacle avoidance system as follows:  
\[R(s_{t})=\left\{\begin{array}{ll}-50,&\text{collision}\\ -20,&|x|>5,|y|>5\\ 100,&Dist<1\\ -1,&other\end{array}\right. \tag{3}\]  
where \(Dist\) is the distance of the drone to its preset destination.  
\begin{table}
\begin{tabular}{c c} \hline \hline
**Action Number** & **Action Description** \\ \hline
0 & Drone moves up 1 unit length \\
1 & Drone moves down 1 unit length \\
2 & Drone moves 1 unit length to the left \\
3 & Drone moves 1 unit length to the right \\ \hline \hline \end{tabular}
\end{table}
Table 1: Drone action space description.