## 3 Multiple Destination Global Path Planning Strategy Based on Improved Q-Learning Algorithm
### Path Planning Based on Improved Q-Learning Algorithm
Through the iteration of Equation (13), the final action value function is updated as Equation (16). \[q(s,a)=r+\gamma max_{a}q(s^{\prime},a^{\prime}), \tag{16}\]  
At the same time, the Q-learning algorithm may take the behavior with the highest expected reward every time. Other possible actions may not be explored during the training  
Figure 9: The flow of global path planning. process. It may even enter the \"local optimum\", failing to complete the path planning. Therefore, in order to avoid it, the setting function _path_ is used to represent all the previous exploration points. The safety distance coefficient \(D\) is set to avoid entering the \"local optimum\". If this exploration is a point in the path, the reward \(r\) obtained by the system returns to 1. If the safety distance \(D<0.5\) m, the reward \(r\) obtained by the system returns to 0. Step 4. Repeat step 3 until the iteration ends. The Q value table is obtained after the last iteration. Step 5. The agent updates the action value function by interacting with the environment. The updated Q value table is used to select the optimal strategy. According to the best path selected by the Q value table, it is planned many times in the order of multi-purpose arrival. It can be seen from Equation (16) that the Q-learning algorithm has nothing to do with the initial value. It can guarantee convergence without the environment model. This feature can be applied to robot path planning. However, according to the algorithm flow: Q-learning updates the Q value table by calculating the action value function, and then selects the optimal strategy according to the Q value table. **4. Multiple Destination Path Planning Simulation**  
_4.1. Multi-Destination Path Planning Algorithm Simulation Steps_  
Based on multi-destination path planning and design the algorithm structure is shown in Figure 10.  
Figure 10: Algorithm flow of multi-destination path planning. As shown in Figure 10, the algorithm of this article can be divided into several steps. The several concrete steps are shown in the following. Step 1. In order to confirm \(n\) destination coordinates, this article inputs the grid map model and initializes it. Step 2. Rebuild the environmental grid map. According to the coordinates of \(n\) destination, \((n^{2}+n)/2\) planning matrices are generated. The planning matrices are the matrices between each two destination points. According to the planning matrices, this article calculates the environmental obstacle value. Step 3. According to the obstacle value obtained in the previous step, this article uses the simulated annealing method to sort. The order of arrival sequence of multiple destinations is obtained. Step 4. According to the order of multi-destination arrival sequence obtained in Step 3, this article starts executing the planning instruction. The path planning is carried out for each two target points based on the improved Q-learning algorithm in turn. Step 5. This step needs to judge whether the current destination is the starting point. If the conditions are met, the exercise ends. Otherwise, the improved Q-learning algorithm prepares to move to the next destination point and goes to step 4.