# Autonomous Obstacle Avoidance with Improved Deep Reinforcement Learning Based on Dynamic Huber Loss
## 3 A DRL Framework Based on Dynamic Huber Loss and Soft Updates
### Loss Function
#### 3.2.4 Dynamic Huber Loss Function  
This study designs the Dynamic Huber loss function to optimize the Huber loss function further and improve the UAV obstacle avoidance performance. It is applied in deep reinforcement learning to solve the UAV obstacle avoidance problem. Compared with the traditional loss function, the advantages of the Dynamic Huber loss function are mainly reflected in the flexibility of gradient adjustment, convergence speed, and convergence accuracy. To deeply understand the characteristics of the Dynamic Huber loss function, a corresponding mathematical model is established for analysis in the UAV obstacle avoidance scenario with wind interference. Compared with the Huber loss function, the Dynamic Huber loss function dynamically adjusts the threshold value \(\delta\) during different stages of training. In the early stage, the \(\delta\) (denoted by \(1\leq\) episode \(\leq\) 300) is set to 1, allowing the gradient to be larger for larger errors. This adjustment helps guide the parameter updates more effectively while tolerating outliers. In the mid-stage, the \(\delta\) (denoted by 301 \(\leq\) episode \(\leq\) 1000) is linearly reduced to 0.5, allowing the gradient to adjust dynamically according to the error and the training phase, ensuring consistent guidance for parameter updates. In the later stage, the \(\delta\) (denoted by 1001 \(\leq\) episode \(\leq\) 2000) is fixed at 0.3. This smaller, fixed threshold makes the gradient more sensitive in the small error region, which helps accelerate convergence. In the gradient descent algorithm [21], convergence speed is closely related to both the gradient and the step size of the loss function. The parameter update rule is as follows:  
\[\theta_{t+1}=\theta_{t}-\alpha\
abla L(\theta_{t}) \tag{15}\]  
where \(\theta_{t}\) is the current parameter, \(\alpha\) is the learning rate, and \(\
abla L(\theta_{t})\) is the gradient of the loss function to the current parameters. Convergence speed is typically measured by the change in the distance from the optimal solution after each iteration. If, at the t-th iteration, the distance between the current parameter and the optimal solution is \(d_{t}\), rapid convergence occurs if \(d_{t+1}<d_{t}\) and \(d_{t}\) approaches zero quickly in each iteration. During the pre-training phase, the larger threshold of the Dynamic Huber loss function results in a larger gradient when facing large errors. According to the parameter update rule, this larger gradient leads to a larger update step, which helps reduce errors quickly. As training progresses into the middle and late stages, the threshold gradually decreases, causing the gradient to adjust accordingly. This results in a smaller update step, ensuring stability as the algorithm approaches the optimal solution. When the error \(\varepsilon\) is large during training, the gradient of the Huber loss function, constrained by a fixed threshold, leads to a relatively small update step. In contrast, the Dynamic Huber loss function can reduce the error more efficiently since its larger threshold yields a larger gradient and, consequently, a larger update step. Let \(\
abla L_{H}(\varepsilon)\) be the gradient of the Huber loss function at this moment and \(\
abla L_{DH}(\varepsilon)\) be the gradient of the Dynamic Huber loss function. With the same learning rate \(\alpha\), the parameter update using the Dynamic Huber loss function will be larger, making it converge to the optimal solution more quickly and improving the overall convergence speed. When approaching the optimal solution, the second-order derivative property of the loss function has an important influence on the convergence accuracy [22]. In the late stage of training, the Dynamic Huber loss function is fixed at a threshold of 0.3, and, at this time, in the small error region \((|\varepsilon|\leq 0.3)\), the function is in the form of a quadratic function \(L(\varepsilon)=\frac{1}{2}\varepsilon^{2}\), and the second-order derivative is \(\varepsilon\).