# Research on Dynamic Path Planning of Multi-AGVs Based on Reinforcement Learning
## 2 K-L-Based Local Path Planning Algorithm
### Reward Function  
For the reward function of traditional Q-learning algorithms, if the AGV reaches the end point, it yields a reward, whereas if it collides with obstacles, it yields a penalty. For other situations, the reward function of the AGV yields a \"reward\" value of 0. The traditional reward function has the advantages of simple calculation and ease of implementation. However, for the path optimization problems of the AGVs, if the number of states is large and the number of dimensions is larger, when the feedbacks of many states are 0, the learning efficiency of the Q-learning algorithm is reduced and the convergence time increases. Therefore, we utilize potential energy to improve the traditional reward function.  
We simulate the working environment of the AGV as a potential energy field. The end point generates gravitational potential energy, whereas the obstacle generates repulsive potential energy. As a result, when the AGV approaches the end point, it obtains an extra reward value. Conversely, when the AGV moves away from the end point or collides with an obstacle, it loses the reward.  
The improved reward function is as follows:  
\[r_{t}=\left\{\begin{array}{c}r_{pos},\,s_{t}=s_{com}\\ r_{de},d_{obs}=0\\ r_{t-1}+F(t),Ep(t)<Ep(t-1)\\ r_{c}\,Ep(t)>Ep(t-1)\\ r_{c}\ \left(D_{obs}\ (t)-D_{obs}\ (t-1)\right),\ \ Ep(t)-Ep(t-1)\\ r_{c}\ \,other\end{array}\right. \tag{6}\]  
where \(r_{pos}\) is the reward obtained when the AGV reaches the target point, \(r_{de}\) is the penalty value obtained when \(d_{obs}\) (the distance between the AGV and the obstacle) is equal to 0, and \(r_{c}\) is a penalty value. \(Ep(t)\) is the potential energy of the current time (\(t\)), and \(F(t)\) is the current additional reward value. \(D_{obs}\ (t)\) is the distance between the AGV and obstacles. The calculation formulae of \(F(t)\) and \(Ep(t)\) are as follows:  
\[Ep(t)=d_{t}-d_{t-1} \tag{7}\]  
\[F(t)=Ep(t)-\mu Ep(t-1) \tag{8}\]  
where \(d_{t}\) is the straight-line distance between the AGV and the end point, and \(\mu\) is the punishment factor.  
If the AGV keeps approaching the target point, i.e., \(Ep(t)<Ep(t-1)\), a positive payoff (\(r_{t-1}+F(t)\)) is obtained; if the AGV keeps moving away from the target point, a negative payoff (\(r_{c}\) ) is obtained; and if the AGV reaches the destination, a large \(r_{pos}\) value is obtained. If the distance between the AGV and the target point is kept constant due to the presence of dynamic obstacles, the dynamic obstacle is used as a repulsive field; then, the payoff value is the value of the change in the distance between the AGV and the dynamic obstacle, as well as the change in the distance to the target point. If the action reduces the distance to the target point, then a positive payoff is obtained; otherwise, a negative payoff is obtained. This design allows the AGV to quickly avoid dynamic obstacles, in addition to providing the AGV with timely feedback on each action taken, ensuring continuity of the payoff function and accelerating the convergence of the algorithm.