# Path Planning Method of Mobile Robot Using Improved Deep Reinforcement Learning

Authors: Wei Wang; Zhenkui Wu; Huafu Luo; Bin Zhang
Journal: Journal of Electrical and Computer Engineering
 (2022)

---

## Abstract

A mobile robot path planning method based on improved deep reinforcement learning is proposed. First, in order to conform to the actual kinematics model of the robot, the continuous environmental state space and discrete action state space are designed. In addition, an improved deep Q-network (DQN) method is proposed, which takes the directly collected information as the training samples and combines the environmental state characteristics of the robot and the target point to be reached as the input of the network. DQN method takes the Q value at the current position as the output of the network model and usesε-greedy strategy for action selection. Finally, the reward function combined with the artificial potential field method is designed to optimize the state-action space. The reward function solves the problem of sparse reward in the environmental state space and makes the action selection of the robot more accurate. Experiments show that compared with the classical DQN method, the average loss function value is reduced by 36.87% and the average reward value is increased by 12.96%, which can effectively improve the working efficiency of mobile robot.


---

## Body

## 1. Introduction

With the continuous integration of informatization and industrialization, the intelligent industry represented by robot technology is booming, which is also the key development field of scientific and technological innovation in various countries [1]. In many application scenarios of robots, the working environment of robots is complex, diverse, and unpredictable. Completing the task of path planning in a complex and unknown environment requires the robot to have a certain degree of intelligence, that is, the ability of autonomous learning and the ability to explore the environment [2–4]. On the other hand, because the robot does not have enough environmental information in the unknown environment, in order for the robot to successfully and efficiently realize path planning in an unknown environment, the robot needs to have a certain degree of adaptability to its working scene and the ability to deal with emergencies [5–7].Artificial intelligence is one of the important methods of robot navigation [8–11]. In recent years, with the continuous efforts of experts and scholars at home and abroad in the field of artificial intelligence and the continuous progress of artificial intelligence technology, some technologies have been successfully applied to human daily life, including speech recognition and machine translation. As one of the research hotspots in artificial intelligence technology, deep reinforcement learning (DRL) has made a breakthrough [12]. AlphaGo defeated the world’s top Go experts. Through deep reinforcement learning algorithm, the DeepMind team can well control YALi arcade games after training, even surpassing the level of ordinary game players, showing the powerful advantages of this method and greatly increasing the confidence of researchers and experts in the research of artificial intelligence technology [13]. Deep reinforcement learning is an autonomous learning method, which does not need to mark the training data, learns appropriate behavior from the environmental state, and allows the robot to modify its strategy according to the received reward or punishment [14]. By introducing the deep reinforcement learning method into the robot path planning, the robot is equivalent to having a “brain” for autonomous learning and walking. In the unknown and complex information scenes, the robot can adjust the walking path and plan the path independently. In the actual robot path planning task, the deep reinforcement learning method still has some problems to be solved and optimized, such as algorithm training efficiency, reward function design, and convergence stability [15, 16]. Taking the deep reinforcement learning method as the path planning algorithm in the robot path planning task has strong scientific research significance and practical application value for improving the intelligent degree of the robot.In order to solve the problems of poor exploration ability and sparse reward of environmental state space in mobile robot path planning in unknown environment, a mobile robot path planning method based on improved deep reinforcement learning is proposed. The innovations of the proposed method lie as follows:(1)
The DQN method is improved. The sensor senses the surrounding environmental information, combines its own location information and the target point to form a state space as the input of the network, takes the Q value at the current location as the output of the network model, and uses -greedy strategy for action selection to improve the operation efficiency.(2)
Combining the artificial potential field method with the reward function, the state-action space is optimized, the reward value of the method is improved, and the reward sparsity of the environmental state space is improved.


## 2. Related Works

Traditional robot path planning methods cannot meet the requirements of modern robot path planning tasks, such as genetic algorithm, artificial potential field method, and neural network [17–19]. These traditional methods require the robot to fully understand the surrounding environment information. When the working environment of the robot is complex and the state is changeable, these traditional methods cannot effectively complete the task of path planning. At the same time, these path planning methods cannot improve the self-study ability, exploration ability, and environmental adaptability of the robot [15]. In order to overcome the shortcomings of these methods, researchers at home and abroad have explored various solutions. Reference [20] used the depth Q-network algorithm to conduct the robot navigation simulation experiment in the maze scene. The experiment directly took color image with a size of 160 × 120 as input. The output was in three states: straight ahead, right turn, and left turn. The experimental results showed that the robot not only successfully realized obstacle avoidance but also had the ability of self-learning. Reference [21] combined double DQN with competitive architecture DQN, adopted multiview strategy, and took the data collected by four cameras from four different directions as input. The experimental results verified the effectiveness of this method. In order to obtain more training data, reference [22] proposed a deep reinforcement learning method of nonstrategy training based on deep Q function by using the multithreading method. This method adopted the concurrent method, which could quickly collect the samples required for training to a certain extent without too much manual intervention and effectively ensure the training time. Reference [23] proposed a distributed asynchronous strategy method for multirobot cooperative learning because multiple robots could share their experience with each other and learn a strategy together. Each robot used local neural network strategy to optimize the behavior of each robot in different scenes and then saved the training samples of all robots to the same server, so as to supervise the training of global neural network strategy. This method effectively reduced the learning time of robot. Reference [24] applied DQN to model-free obstacle avoidance path planning, but there was a problem of overestimation of state-action values, resulting in sparse rewards for mobile robots, and the planned path was not optimal. Reference [25] proposed an end-to-end path planning method of safety constrained robot based on deep reinforcement learning. Reference [26] proposed a path planning method of mobile robot based on TPR-DDPG. This method preprocessed the state through various normalization methods and designed a complete reward function to make the mobile robot quickly reach the target point through the optimal path in the complex environment. Reference [27] proposed a deep reinforcement learning method based on double deep Q-network (DDQN), which made the robot have the ability of autonomous navigation. However, the above methods often have problems such as poor exploration ability and sparse reward in environmental state space, so it is difficult to obtain the optimal planning path.To overcome the above problems, a mobile robot path planning method based on improved deep reinforcement learning is proposed.


## 3. Path Planning Method Based on Deep Reinforcement Learning

### 3.1. Robot Kinematics Model

Pioneer3-AT is a four-wheel differential driving robot composed of front steering wheel and rear driving wheel. The driving wheel moves forward at the speedv and uses the speed difference between the left and right wheels for steering operation. The model diagram of Pioneer3-AT in plane coordinate system is shown in Figure 1. If the midpoint of the robot’s rear axle is selected as the reference point, the robot’s attitude at time t can be expressed by a three-dimensional vector Pxt,yt,θ. xt,yt is the coordinate of the robot in the system coordinate. δ is the steering angle, which is used to describe the angle of the robot’s steering wheel. θ is the direction angle, which is used to describe the included angle between the spatial fixed coordinate system and the robot’s fixed coordinate system, and d is the distance between the front wheel and the rear wheel.Figure 1
Robot motion coordinates.Under the spatial fixed coordinate system, the kinematic model of pioneer3-AT is(1)xtytθ=vcosθvsinθvtanθd.where xtytθT represents state vector at time t.The discretization equation for obtaining the position and attitude of the robot is(2)xt+1=xt+Tvcosθtyt+1=yt+Tvsinθtθt+1=θt+Tvtanδtd,where T is sampling time.


### 3.2. Environmental State Space Design

State space is the feedback of the whole environment and the basis for agents to choose action space. The robot is equipped with Lidar with a detection range of 10 meters and a scanning range of 360 degrees. Considering the accuracy and calculation, only 180 degrees in front of the robot is considered, and the Lidar data in 9 directions are taken, as shown in Figure2.Figure 2
Environmental state space diagram.We usesio to indicate the distance of obstacles detected by the Lidar in all directions and use the Lidar’s measurement span to normalize sio. The final azimuth state information is S1o,S2o,…,S9o.(3)Sio=sioS,where S is the maximum range of Lidar.In order to enable the robot to move towards the target point, the angleβ between the current orientation of the robot and the target point is taken as an input state, as shown in Figure 3. When the target point is on the left side of the robot, the value range of β is 0,180∘. When the target point is on the right side of the robot, the value range of β is −180∘,0.Figure 3
Target angle.


### 3.3. Action State Space Design

The motions of Pioneer3-AT robot include rotational motion and translational motion. The robot controls the moving direction according to the rotational motion. The action space of the robot is continuous and will produce a huge action space. Because the huge action space will make the deep reinforcement learning method difficult to converge, the translation motion of the robot is set as0.8m/s, and the rotation motion is divided into five discrete actions according to(4)A=Ai,i=1,2,3,4,5=0∘,±30∘,±45∘.When the robot turns right according to its own motion direction, the actionAi is a negative angle; When it turns left, Ai is a positive angle. Each rotation angle corresponds to a Q value. The Q value of each rotation angle is obtained through neural network, and then the corresponding rotation angle is selected according to the Q value.


### 3.4. Improved Action Selection Strategy

The model of the proposed method is shown in Figure4. The robot takes the directly collected depth information as the training sample, combines its own environmental state characteristics and the target point to be reached as the input of the network, takes the Q value at the current position as the output of the network model, and uses the ε-greedy strategy for action selection to reach the next state. When the next step is reached, the corresponding reward value r is calculated to obtain a complete data tuple s,a,r,s′, so the series of data is stored in the experience playback pool P, and then a small batch of samples is extracted from the experience playback pool P and put them into the neural network for training. In the process of exploring the optimal path, it is very important for the robot to select the reward value r from the experience playback pool P. The reward value r determines the quality of the robot path planning. The robot sends the reward value r to the optimization objective function to update the network parameters and iterates until the training is completed.Figure 4
Improved DQN method model.In the process of network training, if the robot uses sensors to identify obstacles, the robot can effectively avoid obstacles by improving the deep reinforcement learning method. The design of the improved deep reinforcement learning algorithm is shown in Algorithm 1.Algorithm 1: Improved deep reinforcement learning algorithm for mobile robot.
Inputs: informationε=st,at−1, target point G, the parameters of Q network is θ, and the parameters of target Q network is θ′.Output: action direction of robot.Initialization: initializest, experience playback pool P with capacity of N, action value function Q, the weights of target Q network is θ, θ′  =  θ.for episode = 1, M dofort = 1, T doGet from experience poolP, input st=sp,sp−1,…,s0The robot chooses a random actionat with a certain probability ε; otherwise, it chooses the optimal action at=maxaQ′st,a;θPerform actionat in the environment, get reward rt=2v2cos2vw−0.1 and st+1, put st,at,rt,st+1 into experience pool P.Randomly take a small batch of samplessj,aj,rj,sj+1 from the experience pool PLoss function isLθ=Ey′−Qs,a,θ2Perform policy gradient updateEnd forEnd for


### 3.5. Reward Function Design

The reward function is used to evaluate the quality of the decision-making of the deep learning framework and reward each step of the decision-making. It plays a guiding role in the whole learning process. The neural network makes specific decisions according to the state, the environment is updated according to the decision, and the reward value is calculated. The neural network updates the network parameters according to the feedback reward value, so that the network can make better decisions in the next calculation. The quality of reward function directly affects the effectiveness and convergence of the whole reinforcement learning framework. The reward function is designed as follows:(5)R=−100dt<do,−2others,−1dt=dt−1,1dt<dt−1,2dt<dg,20dt<dg,(6)R=rate∗normP0−P1+1−rate∗R,where dt represents the distance between the robot and the target point at the current time; dt−1 represents the distance between the robot and the target point at the last time; do represents the safe distance of the obstacle. If distance is less than do, it indicates the obstacle is encountered; dg represents the threshold value from the target point. If distance is less than dg, it is considered to have reached the target point; dn represents the distance close to the target point. A positive reward will be given when reaching this range, which can promote the robot to reach the target point faster. When the position of the robot is an obstacle, it will get a negative reward; When the robot reaches the target point, it will get a larger positive reward. When the robot approaches the target point, it will get 1 as a reward; When the robot stays in place, it will get −1 as a reward. In other cases, it will get −2 as a reward. This reward is smaller than 1 because it can prevent the robot from moving back and forth to obtain a positive reward and promote the robot to find the shortest path.In order to solve the problem of blind selection of action by traditional methods, the artificial potential field method is introduced as an auxiliary of early training, so that the system can train the model faster. After selecting the action space, first calculate the next positionP0 of the robot according to the artificial potential field method and then update the environment to get the actual position P1 of the robot. On the basis of the original reward function, add the distance between P0 and P1. In equation (5), normP0−P1 is calculation of the distance between 2 positions. rate is a weight value, which represents the weight of the artificial potential field method. In the early stage, rate has a large proportion and guides the robot to the end faster. With the increase of training times, the weight gradually decreases.


## 3.1. Robot Kinematics Model

Pioneer3-AT is a four-wheel differential driving robot composed of front steering wheel and rear driving wheel. The driving wheel moves forward at the speedv and uses the speed difference between the left and right wheels for steering operation. The model diagram of Pioneer3-AT in plane coordinate system is shown in Figure 1. If the midpoint of the robot’s rear axle is selected as the reference point, the robot’s attitude at time t can be expressed by a three-dimensional vector Pxt,yt,θ. xt,yt is the coordinate of the robot in the system coordinate. δ is the steering angle, which is used to describe the angle of the robot’s steering wheel. θ is the direction angle, which is used to describe the included angle between the spatial fixed coordinate system and the robot’s fixed coordinate system, and d is the distance between the front wheel and the rear wheel.Figure 1
Robot motion coordinates.Under the spatial fixed coordinate system, the kinematic model of pioneer3-AT is(1)xtytθ=vcosθvsinθvtanθd.where xtytθT represents state vector at time t.The discretization equation for obtaining the position and attitude of the robot is(2)xt+1=xt+Tvcosθtyt+1=yt+Tvsinθtθt+1=θt+Tvtanδtd,where T is sampling time.


## 3.2. Environmental State Space Design

State space is the feedback of the whole environment and the basis for agents to choose action space. The robot is equipped with Lidar with a detection range of 10 meters and a scanning range of 360 degrees. Considering the accuracy and calculation, only 180 degrees in front of the robot is considered, and the Lidar data in 9 directions are taken, as shown in Figure2.Figure 2
Environmental state space diagram.We usesio to indicate the distance of obstacles detected by the Lidar in all directions and use the Lidar’s measurement span to normalize sio. The final azimuth state information is S1o,S2o,…,S9o.(3)Sio=sioS,where S is the maximum range of Lidar.In order to enable the robot to move towards the target point, the angleβ between the current orientation of the robot and the target point is taken as an input state, as shown in Figure 3. When the target point is on the left side of the robot, the value range of β is 0,180∘. When the target point is on the right side of the robot, the value range of β is −180∘,0.Figure 3
Target angle.


## 3.3. Action State Space Design

The motions of Pioneer3-AT robot include rotational motion and translational motion. The robot controls the moving direction according to the rotational motion. The action space of the robot is continuous and will produce a huge action space. Because the huge action space will make the deep reinforcement learning method difficult to converge, the translation motion of the robot is set as0.8m/s, and the rotation motion is divided into five discrete actions according to(4)A=Ai,i=1,2,3,4,5=0∘,±30∘,±45∘.When the robot turns right according to its own motion direction, the actionAi is a negative angle; When it turns left, Ai is a positive angle. Each rotation angle corresponds to a Q value. The Q value of each rotation angle is obtained through neural network, and then the corresponding rotation angle is selected according to the Q value.


## 3.4. Improved Action Selection Strategy

The model of the proposed method is shown in Figure4. The robot takes the directly collected depth information as the training sample, combines its own environmental state characteristics and the target point to be reached as the input of the network, takes the Q value at the current position as the output of the network model, and uses the ε-greedy strategy for action selection to reach the next state. When the next step is reached, the corresponding reward value r is calculated to obtain a complete data tuple s,a,r,s′, so the series of data is stored in the experience playback pool P, and then a small batch of samples is extracted from the experience playback pool P and put them into the neural network for training. In the process of exploring the optimal path, it is very important for the robot to select the reward value r from the experience playback pool P. The reward value r determines the quality of the robot path planning. The robot sends the reward value r to the optimization objective function to update the network parameters and iterates until the training is completed.Figure 4
Improved DQN method model.In the process of network training, if the robot uses sensors to identify obstacles, the robot can effectively avoid obstacles by improving the deep reinforcement learning method. The design of the improved deep reinforcement learning algorithm is shown in Algorithm 1.Algorithm 1: Improved deep reinforcement learning algorithm for mobile robot.
Inputs: informationε=st,at−1, target point G, the parameters of Q network is θ, and the parameters of target Q network is θ′.Output: action direction of robot.Initialization: initializest, experience playback pool P with capacity of N, action value function Q, the weights of target Q network is θ, θ′  =  θ.for episode = 1, M dofort = 1, T doGet from experience poolP, input st=sp,sp−1,…,s0The robot chooses a random actionat with a certain probability ε; otherwise, it chooses the optimal action at=maxaQ′st,a;θPerform actionat in the environment, get reward rt=2v2cos2vw−0.1 and st+1, put st,at,rt,st+1 into experience pool P.Randomly take a small batch of samplessj,aj,rj,sj+1 from the experience pool PLoss function isLθ=Ey′−Qs,a,θ2Perform policy gradient updateEnd forEnd for


## 3.5. Reward Function Design

The reward function is used to evaluate the quality of the decision-making of the deep learning framework and reward each step of the decision-making. It plays a guiding role in the whole learning process. The neural network makes specific decisions according to the state, the environment is updated according to the decision, and the reward value is calculated. The neural network updates the network parameters according to the feedback reward value, so that the network can make better decisions in the next calculation. The quality of reward function directly affects the effectiveness and convergence of the whole reinforcement learning framework. The reward function is designed as follows:(5)R=−100dt<do,−2others,−1dt=dt−1,1dt<dt−1,2dt<dg,20dt<dg,(6)R=rate∗normP0−P1+1−rate∗R,where dt represents the distance between the robot and the target point at the current time; dt−1 represents the distance between the robot and the target point at the last time; do represents the safe distance of the obstacle. If distance is less than do, it indicates the obstacle is encountered; dg represents the threshold value from the target point. If distance is less than dg, it is considered to have reached the target point; dn represents the distance close to the target point. A positive reward will be given when reaching this range, which can promote the robot to reach the target point faster. When the position of the robot is an obstacle, it will get a negative reward; When the robot reaches the target point, it will get a larger positive reward. When the robot approaches the target point, it will get 1 as a reward; When the robot stays in place, it will get −1 as a reward. In other cases, it will get −2 as a reward. This reward is smaller than 1 because it can prevent the robot from moving back and forth to obtain a positive reward and promote the robot to find the shortest path.In order to solve the problem of blind selection of action by traditional methods, the artificial potential field method is introduced as an auxiliary of early training, so that the system can train the model faster. After selecting the action space, first calculate the next positionP0 of the robot according to the artificial potential field method and then update the environment to get the actual position P1 of the robot. On the basis of the original reward function, add the distance between P0 and P1. In equation (5), normP0−P1 is calculation of the distance between 2 positions. rate is a weight value, which represents the weight of the artificial potential field method. In the early stage, rate has a large proportion and guides the robot to the end faster. With the increase of training times, the weight gradually decreases.


## 4. Experiment and Analysis

### 4.1. Simulation Environment Construction and Parameter Setting

Considering that the environment of the actual robot is diverse, the shapes of obstacles in the experimental environment are diverse, and the placement positions are as random as possible. The configuration environment of the computer used in the experiment is NVIDIA GTX 2080Ti GPU server, and the operating system is Ubuntu 16 04. Pioneer3-AT robot and its working environment are simulated with ROS and Gazebo. Python programming language and TensorFlow framework are used. Some relevant parameter values in the experiment are shown in Table1.Table 1
Experimental parameters of robot path planning.
ParameterValueLearning rate0.005Reward decline0.8Network update frequency400Experience pool size10000Number of batches16Number of samples in each layer15β0.3Starting value,ε0.1Termination value,ε0.0001


### 4.2. Comparison between Loss Functions and between Average Value of Rewards

The parameters are updated iteratively through the loss functionLθ and Adam optimizer. Under the framework of Keras based on TensorFlow, using GPU acceleration, all training samples pass through 1000 epochs to complete the training of the proposed improved deep reinforcement learning model. Figure 5 shows the comparison of the loss function values in the training process of the network model. It can be seen that the loss values of the two methods tend to be stable with the continuous increase of the number of training iterations. The value of loss function tends to be stable when the training times of DQN algorithm reach about 270. When the training times of the proposed method reach about 200, the value of the loss function can quickly become stable. The average loss function value of DQN algorithm is 0.178, and the average loss function value of the proposed method is 0.112, which is 36.87% lower than that of DQN algorithm. Therefore, the improved method has certain advantages and speeds up the convergence speed of the network.Figure 5
Comparison diagram of loss function values.Figure6 shows the comparison of the average reward values between DQN algorithm and the proposed method. At the beginning of training, the reward value is between −9 and 0, which is the process that the robot just begins to explore, learn, and avoid obstacles, and it fails to make correct judgment on obstacles, so getting a negative reward value. When the reward value is (0∼80) stage, the training times reach 100, and the robot is in the exploratory learning stage, which means that the robot begins to recognize and can avoid some obstacles, but it is still learning interactively with the environment and further adjust the action selection strategy to obtain the corresponding positive reward value. When the reward value is (80∼160) stage and the training times are 100∼200, the reward value obtained by the robot in DQN algorithm and improved DQN algorithm is unstable. When the training times reach about 300, the reward value obtained by DQN algorithm tends to be balanced. When the training times of the improved method reach about 200, the average reward value tends to be stable. The average reward value of the improved DQN is 12.96% higher than that of DQN. Therefore, the improved method can shorten the network training time, improve the average reward value, and improve the reward sparsity.Figure 6
Comparison of average reward values.


### 4.3. Comparison of Several Methods of Path Planning

In order to prove the advantages of the proposed improved DQN mobile robot path planning method, path planning methods in reference [25, 26] are compared with the proposed method under the same experimental conditions. The comparison indicators include the following two: the planned path length and the number of turning points in planned path. The obtained path planning diagram is shown in Figure 7, and the comparison data is shown in Table 2.Figure 7
Comparison of path planning of different methods. (a) Ref [26]. (b) Ref [27]. (c) Proposed methods.
(a)(b)(c)Table 2
Comparison of path length and number of turning points of different methods.
MethodRef. [25]Ref. [26]Proposed methodPath length28.62728.62728.627Number of turning points574Figures7(a)–7(c) are the paths planned by reference [25, 26] and the proposed improved DQN method according to the action corresponding to the maximum Q value output by the network after 500 iterations. From the path planning map and Table 2, each method has planned a collision-free path. In terms of the advantages and disadvantages of the path, the proposed DQN method plans an optimal path with a path length of 28.627 and a number of turning points of 4, while the methods in reference [25, 26] do not plan an optimal path. Although the path length is the same as the proposed method, the number of turning points of the two methods are 5 and 7 respectively, which increases the time-consumption of robots. The proposed method aims at the global path planning and combines the artificial potential field method with the reward function to optimize the state-action space, improve the reward value of the method, and guide the robot to the target faster. The comparison method has the problem of overestimation of state-action value, resulting in sparse rewards for mobile robots, and the planned path is not optimal. In the actual environment, the reduction of turning points in path planning will also reduce the time spent by the robot from the starting point to the target point.


## 4.1. Simulation Environment Construction and Parameter Setting

Considering that the environment of the actual robot is diverse, the shapes of obstacles in the experimental environment are diverse, and the placement positions are as random as possible. The configuration environment of the computer used in the experiment is NVIDIA GTX 2080Ti GPU server, and the operating system is Ubuntu 16 04. Pioneer3-AT robot and its working environment are simulated with ROS and Gazebo. Python programming language and TensorFlow framework are used. Some relevant parameter values in the experiment are shown in Table1.Table 1
Experimental parameters of robot path planning.
ParameterValueLearning rate0.005Reward decline0.8Network update frequency400Experience pool size10000Number of batches16Number of samples in each layer15β0.3Starting value,ε0.1Termination value,ε0.0001


## 4.2. Comparison between Loss Functions and between Average Value of Rewards

The parameters are updated iteratively through the loss functionLθ and Adam optimizer. Under the framework of Keras based on TensorFlow, using GPU acceleration, all training samples pass through 1000 epochs to complete the training of the proposed improved deep reinforcement learning model. Figure 5 shows the comparison of the loss function values in the training process of the network model. It can be seen that the loss values of the two methods tend to be stable with the continuous increase of the number of training iterations. The value of loss function tends to be stable when the training times of DQN algorithm reach about 270. When the training times of the proposed method reach about 200, the value of the loss function can quickly become stable. The average loss function value of DQN algorithm is 0.178, and the average loss function value of the proposed method is 0.112, which is 36.87% lower than that of DQN algorithm. Therefore, the improved method has certain advantages and speeds up the convergence speed of the network.Figure 5
Comparison diagram of loss function values.Figure6 shows the comparison of the average reward values between DQN algorithm and the proposed method. At the beginning of training, the reward value is between −9 and 0, which is the process that the robot just begins to explore, learn, and avoid obstacles, and it fails to make correct judgment on obstacles, so getting a negative reward value. When the reward value is (0∼80) stage, the training times reach 100, and the robot is in the exploratory learning stage, which means that the robot begins to recognize and can avoid some obstacles, but it is still learning interactively with the environment and further adjust the action selection strategy to obtain the corresponding positive reward value. When the reward value is (80∼160) stage and the training times are 100∼200, the reward value obtained by the robot in DQN algorithm and improved DQN algorithm is unstable. When the training times reach about 300, the reward value obtained by DQN algorithm tends to be balanced. When the training times of the improved method reach about 200, the average reward value tends to be stable. The average reward value of the improved DQN is 12.96% higher than that of DQN. Therefore, the improved method can shorten the network training time, improve the average reward value, and improve the reward sparsity.Figure 6
Comparison of average reward values.


## 4.3. Comparison of Several Methods of Path Planning

In order to prove the advantages of the proposed improved DQN mobile robot path planning method, path planning methods in reference [25, 26] are compared with the proposed method under the same experimental conditions. The comparison indicators include the following two: the planned path length and the number of turning points in planned path. The obtained path planning diagram is shown in Figure 7, and the comparison data is shown in Table 2.Figure 7
Comparison of path planning of different methods. (a) Ref [26]. (b) Ref [27]. (c) Proposed methods.
(a)(b)(c)Table 2
Comparison of path length and number of turning points of different methods.
MethodRef. [25]Ref. [26]Proposed methodPath length28.62728.62728.627Number of turning points574Figures7(a)–7(c) are the paths planned by reference [25, 26] and the proposed improved DQN method according to the action corresponding to the maximum Q value output by the network after 500 iterations. From the path planning map and Table 2, each method has planned a collision-free path. In terms of the advantages and disadvantages of the path, the proposed DQN method plans an optimal path with a path length of 28.627 and a number of turning points of 4, while the methods in reference [25, 26] do not plan an optimal path. Although the path length is the same as the proposed method, the number of turning points of the two methods are 5 and 7 respectively, which increases the time-consumption of robots. The proposed method aims at the global path planning and combines the artificial potential field method with the reward function to optimize the state-action space, improve the reward value of the method, and guide the robot to the target faster. The comparison method has the problem of overestimation of state-action value, resulting in sparse rewards for mobile robots, and the planned path is not optimal. In the actual environment, the reduction of turning points in path planning will also reduce the time spent by the robot from the starting point to the target point.


## 5. Conclusion

In the face of mobile robot path planning in unknown environment, the current research often has the problems of poor exploration ability and sparse reward in environmental state space, so it is difficult to obtain the optimal planning path. In order to overcome the above problems, a mobile robot path planning method based on improved deep reinforcement learning is proposed. The continuous environmental state space and discrete action state space are designed, an improved DQN method is proposed to speed up the path search speed, and an improved reward and punishment function is designed to improve the reward value of the method, which alleviates the problem of sparse reward to a certain extent. Experiments show that compared with the comparative references, the proposed method can effectively improve the working efficiency of mobile robot. Using images as data input will get richer environmental information than Lidar. In the future, robot autonomous path planning through camera will be studied. In addition, multirobot cooperative path planning task is also the focus of the next research of this topic.



---

*Source: 5433988-2022-04-23.xml*