## 3 Multiple Destination Global Path Planning Strategy Based on Improved Q-Learning Algorithm
### Path Planning Based on Improved Q-Learning Algorithm  
Global path planning means that the robot plans a global path from the starting point through all destinations while moving. The Q-learning algorithm has little connection with the initial value. It can guarantee the convergence without the environment model. This feature can be applied to the path planning of the mobile robot in this article. The Q-learning algorithm is based on the reinforcement learning framework. Firstly, a Q value table is set. The rows of the table represent different states. The columns of it represent different actions that can be taken. The initial state and reward value are defined, and the agent is executed. After a certain action, the environment will transform to a new state. For this new state, the environment will give a corresponding reward value. Then, the agent calculates and updates the Q value table according to the new state and reward feedback from the environment. The Q value table selects strategies and executes new actions. Finally, an optimal action set is found through continuous interaction with the environment. Aiming at the characteristics of multi-destination path planning and the possibility of wheeled mobile robots, the overall destination arrival sequence planning is obtained. The improved Q-learning algorithm is used to complete the global path planning. The structure is shown in Figure 9.  
As shown in Figure 9, global path planning can be divided into several steps. The several concrete steps are shown as following. Step 1. Initialize the Q value table. The Q value table has \(n\) rows and \(n\) columns (\(n\) is the number of destinations). All values in the table are initialized to zero. The Q-value table is updated by Equation (12). Therefore, the basic form of the algorithm is Equation (12). \[q(s,a)=q(s,a)+\alpha[r+\gamma max_{a}q(s^{\prime},a^{\prime})-q(s,a)], \tag{12}\]  
where \(q(s,a)\) is the of action value function. \(s\) is the current state. \(a\) is the current action. \(s^{\prime}\) is the next state. \(a^{\prime}\) is the action taken in the next state. \(r\) is the reward obtained by the system. \(\alpha\) is the learning rate. \(\gamma\) is the attenuation factor. \(max_{a}q(s^{\prime},a^{\prime})\) represents the next state \(s^{\prime}\) to select action \(a^{\prime}\) to maximize the value function. Step 2. Select the next action \(a\) based on the current Q value table. The Q value of the initial state is zero, and the agent can randomly choose the next action at the beginning. As the number of iterations increases, the Q value table is continuously updated, and this step will choose the action with the greatest Q value return. By transforming the Equation (12), this article can get Equation (13). \[q(s,a)=(1-\alpha)q(s,a)+\alpha[r+\gamma max_{a}q(s^{\prime},a^{\prime})], \tag{13}\]  
According to Equation (13), Equation (14) is carried out by iteration. \[ \begin{array}{c}q(s,a)=(1-\alpha)q(s,a)+\alpha[r+\gamma max_{a}q(s^{\prime}, a^{\prime})]=\\ \quad\quad\quad\quad\quad\quad\quad\quad(1-\alpha)[(1-\alpha)q(s,a)+\alpha[r+ \gamma max_{a}q(s^{\prime},a^{\prime})]]+\alpha[r+\gamma max_{a}q(s^{\prime}, a^{\prime})]=\\ \quad\quad\quad\quad\quad\quad(1-\alpha)^{2}q(s,a)+[1-(1-\alpha)^{2}][r+ \gamma max_{a}q(s^{\prime},a^{\prime})]\end{array}  \tag{14}\]  
The Equation (13) continues to be substituted into Equation (14) for iteration. By analogy, \(n\) iterations are required. Step 3. This step needs to calculate the action reward. According to the current state and reward, this article uses [PERSON]'s equation to update the previous state \(q\) (\(s\), \(a\)), after action \(a\). \[ \begin{array}{c}q(s,a)=(1-\alpha)q(s,a)+\alpha[r+\gamma max_{a}q(s^{\prime},a^{\prime})]=\\ \quad\quad\quad\quad\quad\quad\quad\quad(1-\alpha)^{n}q(s,a)+[1-(1-\alpha)^{n }][r+\gamma max_{a}q(s^{\prime},a^{\prime})]\end{array}  \tag{15}\]  
Due to the learning rate \(0<\alpha<1\), so \(0<(1-\alpha)<1\).When \(n\rightarrow\infty\), \(\left(1-\alpha\right)^{n}\to 0\).