# Autonomous Obstacle Avoidance with Improved Deep Reinforcement Learning Based on Dynamic Huber Loss
## 3 A DRL Framework Based on Dynamic Huber Loss and Soft Updates
### Loss Function
#### 3.2.3 Huber Loss  
The Huber loss is a loss function with parameters for regression problems. The expression for the Huber loss is as follows:  
\[L_{\text{Huber}}(\theta)=\left\{\begin{array}{ll}\frac{1}{2}{\left(\varepsilon \right)^{2}}&|\varepsilon|\leq\lambda\\ \lambda(|\varepsilon|-\frac{1}{2}\lambda)&|\varepsilon|>\lambda\end{array}\right. \tag{13}\]  
For the neural network parameter \(\theta\), the gradient is as follows:  
\[\
abla L_{\text{Huber}}(\theta)=\frac{\partial L_{\text{Huber}}}{\partial \varepsilon}=\left\{\begin{array}{ll}\varepsilon&|\varepsilon|\leq\lambda \\ \lambda sign(\varepsilon)&|\varepsilon|>\lambda\end{array}\right. \tag{14}\]  
The Huber loss, like the Smooth L1 loss, combines the advantages of the MAE and MSE and can be trained with its parameters \(\lambda\) to achieve better outlier processing results. When \(|\varepsilon|\leq\lambda\), the gradient is \(\
abla L_{\text{Huber}}(\theta)=\varepsilon\), the Lipschitz constant \(L_{2}=1\), and when \(|\varepsilon|>\lambda\), the gradient is \(\
abla L_{\text{Huber}}(\theta)=\lambda sign(\varepsilon)\), whose derivative is 0, and due to \(\lambda<1\), the overall gradient changes more gently. For \(\theta_{1}\) and \(\theta_{2}\), it is easy to verify that the Lipschitz continuity is satisfied and the constants are 1 or \(\lambda\) when \(\theta_{1}\) and \(\theta_{2}\) are both in \((-\infty,-\lambda]\cup[\lambda,\infty)\) or in \([-\lambda,\lambda]\). When \(\theta_{1}\in[-\lambda,\lambda]\) and \(\theta_{2}\
otin[-\lambda,\lambda]\), then \(\|\
abla L_{\text{Huber}}(\theta_{2})-\
abla L_{\text{Huber}}(\theta_{1})\|= \|\lambda\cdot sign(\theta_{2})-\theta_{1}\|\leq|\lambda-\theta_{1}\|\leq\lambda\), and \(\|\theta_{2}-\theta_{1}\|\geq\theta_{2}-\lambda\), so \(\frac{\|\
abla L_{\text{Huber}}(\theta_{1})-\
abla L_{\text{Huber}}(\theta_{1 })\|}{\|\theta_{2}-\theta_{1}\|}\leq 1\); therefore, the overall Lipschitz constant is relatively small.  
For Huber loss, when \(|\varepsilon|>\lambda\), \(m_{t}=\beta_{1}m_{t-1}+(1-\beta_{1})\lambda sign(\varepsilon)\); since \(\lambda<1\), the range of fluctuations of \(m_{t}\) is limited between \(\beta_{1}m_{t-1}\pm(1-\beta_{1})\lambda\), and the fluctuation is smaller compared to Smooth L1 loss, making the parameter update more stable. Also, according to the recursive equation \(v_{t}=(1-\beta_{2})\lambda^{2}\sum\limits_{i=0}^{t-1}\beta_{2}^{i}+\beta_{2}^{ i}v_{0}\), when \(t\) is large, \(v_{t}\approx\frac{(1-\beta_{2})\lambda^{2}}{1-\beta_{2}}=\lambda^{2}\); since \(\lambda<1\), \(v_{t}\) reflects the gradient changes more finely, and it will not stabilize at an inappropriate value quickly due to the sudden change of the gradient. Because \(m_{t}\) of fluctuation is small and \(v_{t}=\lambda^{2}\), the parameter update step \(\Delta\theta_{Huber}\) is relatively stable. Additionally, because \(\lambda<1\), the denominator is relatively small, within a reasonable range of variation of \(m_{t}\), and the parameters can be adjusted more finely and move more stably towards the optimal solution. Compared with Smooth L1 loss, Huber loss's gradient is smoother, the Lipschitz constant is relatively small, and the upper bound of the convergence rate is smaller, i.e., the convergence rate is faster. At the same time, its stable gradient property makes it less likely to fall into the local optimal solution during the iteration process, and it can converge to the global optimal solution more stably.