# Autonomous Obstacle Avoidance with Improved Deep Reinforcement Learning Based on Dynamic Huber Loss
## 3 A DRL Framework Based on Dynamic Huber Loss and Soft Updates
### Loss Function
#### 3.2.4 Dynamic Huber Loss Function
As the Huber loss function has a fixed threshold, if the threshold is not set properly when approaching the optimal solution, the advantage of accurate convergence of the quadratic function cannot be fully utilized. Assuming that the error is within the range of \([-\varepsilon,\varepsilon]\) (\(\varepsilon\) is a very small positive number) when approaching the optimal solution, for the Dynamic Huber loss function, the quadratic form of the loss function can make the parameter update more accurately towards the optimal solution according to the second-order derivative property. In this region, the Huber loss function may not be able to update the parameters precisely due to the gradient changes. Through the analysis of the influence of second-order derivative on parameter updating, it can be seen that the Dynamic Huber loss function can adjust the parameter more accurately to make the model closer to the optimal solution and thus has higher convergence accuracy. To deeply understand the characteristics of the Dynamic Huber loss function, a corresponding mathematical model is established for the UAV obstacle avoidance scenario with wind interference. The model's behavior under wind interference and its suppression effect on such interference are analyzed in detail. Furthermore, mathematical analyses and theorems are rigorously applied to validate these findings. In the case of wind interference, the motion of the UAV is affected by wind. Let the actual position of the UAV at the moment of \(t\) be \((x_{t},y_{t})\), the velocity be \(\left(v_{x,t},v_{y,t}\right)\), and the velocity components of the wind in the directions of \(x\) and \(y\) be \(\left(w_{x},w_{y}\right)\), respectively. Then, the position and velocity of the UAV at the next moment are updated by the following:  
\[\left\{ \begin{aligned} & x_{t+1}=x_{t}+v_{x,t}\Delta t+w_{x}\Delta t \\ & y_{t+1}=y_{t}+v_{y,t}\Delta t+w_{y}\Delta t\\ & v_{x,t+1}=v_{x,t}+a_{x,t}\Delta t\\ & v_{y,t+1}=v_{y,t}+a_{y,t}\Delta t\\ \end{aligned} \right. \tag{16}\]  
where \(a_{x,t}\) and \(a_{y,t}\) are the acceleration of the UAV in the \(x\) and \(y\) directions, and \(\Delta t\) is the time interval. The Dynamic Huber loss function adjusts its threshold at different training stages. The threshold for each stage is defined as follows:  
\[L_{DH}(x,t)=\left\{ \begin{array}{cl}\frac{1}{2}\varepsilon^{2},&if| \varepsilon|\leq\delta_{t}\\ \delta_{t}|\varepsilon|-\frac{1}{2}\delta_{t}^{2},&if|\varepsilon|>\delta_{t} \\ \end{array} \right. \tag{17}\]  
In the scenario with wind interference, \(\varepsilon\) can represent the deviation of the actual UAV trajectory from the optimal trajectory. As the training phase progresses, \(\delta_{t}\) will change; \(\delta_{1}=1\) at the beginning of training, linearly decreases to \(\delta_{2}=0.5\) in the middle, and is fixed at \(\delta_{3}=0.3\) in the end. In the gradient descent algorithm, if the loss function \(L(\theta)\) satisfies the Lipschitz continuity, i.e., there exists a constant \(L\) such that \(\|\
abla L(\theta_{1})-\
abla L(\theta_{2})\|\leq L|\theta_{1}-\theta_{2}|\) and the learning rate \(\alpha\) satisfies \(0<\alpha<\frac{2}{L}\), the iterative sequence \(\{\theta_{t}\}\) converges to a locally optimal solution. In the presence of wind interference, the gradient of the Dynamic Huber loss function \((\
abla L_{DH}(\varepsilon,t))\) varies with the threshold \(\delta_{t}\) and the error \(\varepsilon\). Early in training, when the trajectory deviation due to wind interference is large, the gradient is larger, leading to a larger update step. As training progresses, the threshold decreases, and the gradient becomes smaller, reducing the update step to ensure stability. By dynamically adjusting the threshold at different stages, the Dynamic Huber loss function can satisfy the Lipschitz continuity condition across different error regions.