# Autonomous Obstacle Avoidance with Improved Deep Reinforcement Learning Based on Dynamic Huber Loss
## 4 Experimental Setup and Simulation Analysis
### Analysis of Training Results
In the Smooth L1 loss experiments, the Soft updating method significantly increases the reward value at the late stage of the training, which shows that the Soft updating method enhances the later obstacle avoidance ability of the UAV and enables it to adapt to the environmental changes. In the Huber loss experiments, the reward value of the Soft updating method is always high with minimal fluctuation, which ensures that the UAV makes correct decisions at high frequency and steadily improves its obstacle avoidance skills. For the Dynamic  
Figure 4: UAV obstacle avoidance training reward maps for different loss functions: (**a**) static environment with Soft update; (**b**) dynamic environment with Hard update; (**c**) dynamic environment with Soft update; (**d**) dynamic environment with Soft update and wind interference. Huber loss experiments, the fluctuation of the reward value is relatively smooth, the convergence tendency is obvious and fast in the later stage, and the upper limit of the loss value of Soft updating is low, which can avoid the situation of poor performance. In conclusion, Soft updating can update the parameters more smoothly when dealing with outliers or large errors to keep the reward value relatively stable, which is conducive to the stable training and optimization of the model. The change of loss during UAV obstacle avoidance training with different loss functions is shown in Figure 6. Figure 6a shows the Soft-update-based UAV obstacle avoidance training loss variations in a static environment. In the early stage of training (0-10,000 steps), the loss values of MSE, Smooth L1 loss, and Huber loss fluctuate greatly, while Dynamic Huber loss fluctuates less, and the early optimization is smoother. Meanwhile, MSE, Smooth L1 loss, and Huber loss several times during training show higher loss value conditions. In contrast, Dynamic Huber loss has the lowest loss value at the beginning of training and the fastest decline, and its loss value is already significantly lower than the other loss functions at about 10,000 steps. Figure 6b shows the loss variations of Hard-update-based UAV obstacle avoidance training in a dynamic environment. At the beginning of the training (0-10,000 steps), the loss values of MSE, Smooth L1 loss, and Huber loss fluctuate a lot, while Dynamic Huber loss fluctuates less and the initial optimization is smoother. After 50,000 steps, the fluctuation ranges of loss values of Dynamic Huber loss decrease and stabilize, while the fluctuations of the other functions are larger. Figure 6c shows the  
Figure 5: Reward maps for UAV obstacle avoidance training with different update methods ((**a–d**) are the reward maps based on MSE, Smooth L1 loss, Huber loss, and Dynamic Huber loss, respectively). Soft-update-based UAV obstacle avoidance training loss variations in a dynamic environment. In the early stage of training (0-5000 steps), the loss values of MSE, Smooth L1 loss, and Huber loss fluctuate greatly, and Dynamic Huber loss fluctuates less, with smoother optimization in the early stage. Then, 20,000 steps later, the range of fluctuation of loss values of Dynamic Huber loss gradually narrows down and tends to be stable, while the other functions fluctuate more. Figure 6d shows the loss variations in dynamic Soft-update-based UAV obstacle avoidance training with wind interference. In the initial training stage (0-8000 steps), the loss values of MSE, Smooth L1 loss, and Huber loss fluctuate sharply due to wind interference. However, the Dynamic Huber loss fluctuates less, with more stable optimization. After 25,000 steps, the loss value of Dynamic Huber loss stabilizes at a low level, while the others still fluctuate significantly. In summary, Dynamic Huber loss has the lowest loss value and the fastest decline in the early stage of training, indicating that it can converge to a lower loss value faster in the early stage of training.