# Autonomous Obstacle Avoidance with Improved Deep Reinforcement Learning Based on Dynamic Huber Loss
## 1 Introduction  
In recent years, the rapid development of machine learning theories and methods, along with advancements in implementation hardware, has significantly enhanced the expansion of UAV application areas. Today, UAVs are increasingly used in autonomous operations across various fields, including power grid inspection, logistics, agricultural and forestry plant protection, and security patrols. During actual operations, UAVs often encounter dynamic, unstructured, and complex environments, which pose challenges to their performance. Among these challenges, the effectiveness of path planning and the efficiency of autonomous obstacle avoidance directly influence the UAV's ability to successfully complete its mission. Traditional obstacle avoidance methods, such as artificial potential field techniques [1, 2], particle swarm optimization algorithms [3, 4, 5], and feedback control methods [6], have matured significantly. However, these methods require the construction of models based on specific application scenarios and often necessitate the generation of offline maps for path planning. This reliance on pre-defined parameters severely limitstheir applicability in unknown or dynamic environments. In contrast, machine learning approaches, including neural networks [7; 8; 9], reinforcement learning (RL) [10; 11; 12], and deep reinforcement learning (DRL), have been increasingly applied to the domain of autonomous UAV obstacle avoidance [13]. Nonetheless, methods based on neural networks tend to lack versatility due to their high data requirements and scene specificity. Reinforcement learning reframes the UAV obstacle avoidance challenge as an optimal decision-making problem, aiming to develop strategies that maximize long-term rewards through interaction with the environment [10]. However, RL's effectiveness is often constrained by its difficulty in managing high-dimensional state and action spaces. Deep reinforcement learning addresses some of these limitations by combining the strengths of deep learning and reinforcement learning [14]. Collectively, these efforts highlight the ongoing research dedicated to overcoming the challenges associated with UAV obstacle avoidance. The core of RL lies in the fact that the agent explores and optimizes strategies with the help of a loss function through constant interaction with the environment, thus obtaining greater cumulative rewards and improving decision-making ability and task performance. Commonly used loss functions include mean absolute error (MAE), mean square error (MSE), and Smooth L1 loss. Nevertheless, the functions mentioned above encounter issues such as model oscillation, sensitivity to outliers, shape fixation, parameter sensitivity, and function instability when addressing complex challenges like UAV obstacle avoidance, which impede the training outcomes and degrade model performance. To enhance both the speed of convergence and the accuracy of DRL in autonomous UAV obstacle avoidance, this paper proposes an improved DRL method based on Dynamic Huber loss. Unlike traditional methods, this method substitutes Soft update for Hard update when updating the target network in DRL. Soft updating is a parameter updating strategy, which does not directly copy the parameters of the online network to the target network as Hard updating does but fuses the parameters of the online network with those of the current target network according to a certain ratio so that the parameters of the target network can be updated smoothly. This approach helps to enhance training stability and avoid model oscillation due to large parameter updates, which improves the effectiveness of UAV obstacle avoidance decision-making. At the same time, the parameters of Huber loss can be dynamically adjusted, and different parameter values are set in the early, middle, and late stages of training so that the gradient can be flexibly regulated according to the error and the training stage to optimize the obstacle avoidance strategy to improve the performance of obstacle avoidance.