# A Vehicle Path Planning Algorithm Based on Mixed Policy Gradient Actor-Critic Model with Random Escape Term and Filter Optimization

Authors: Wei Nai; Zan Yang; Daxuan Lin; Dan Li; Yidan Xing
Journal: Journal of Mathematics
 (2022)

---

## Abstract

The transportation system of those countries has a huge traffic flow is bearing great pressure on transportation planning and management. Vehicle path planning is one of the effective ways to alleviate such pressure. Deep reinforcement learning (DRL), as a state-of-the-art solution method in vehicle path planning, can better balance the ability and complexity of the algorithm to reflect the real situation. However, DRL has its own disadvantages of higher search cost and earlier convergence to the local optimum, as vehicle path planning issues are usually in a complex environment, and their action set can be diverse. In this paper, a mixed policy gradient actor-critic (AC) model with random escape term and filter operation is proposed, in which the policy weight is both data driven and model driven. The empirical data-driven method is used to improve the poor asymptotic performance, and the model-driven method is used to ensure the convergence speed of the whole model. At the same time, in order to avoid the model converging local optimum, a random escape term has been added in the policy weight update process to overcome the problem that it is difficult to optimize the non-convex loss function, and the random escape term can help to explore the policy in more directions. In addition, filter optimization has been innovatively introduced in this paper, and the step size of each iteration of the model is selected through the filter optimization algorithm to achieve the better iterative effect. Numerical experiment results have shown that the model proposed in this paper can not only improve the accuracy of the solution without losing the accuracy but also speed up the convergence speed and improve the utilization of data.


---

## Body

## 1. Introduction

### 1.1. Backgrounds

In recent years, the rapid growth of China’s economy has greatly stimulated the growth of its domestic automobile consumption and has caused a rapid increase of the number of private vehicles. In the first half of 2020, the sales volume of China’s automobile market remained at the level of above 10 million, even if affected by the COVID-19 epidemic [1]. Traffic congestion caused by vehicle growth is one of the main reasons of traffic congestion, and it will also lead to problems of increased energy consumption and soaring carbon oxide emissions. In this context, the study of intelligent transportation system (ITS) attracts extensive attention. Vehicle path planning is a very important aspect of ITS, as it can not only help drivers to achieve the purpose of shortest distance, minimum cost and minimum travel duration under certain constraints, but help government or transportation management department to alleviate traffic congestion, and to improve the operation efficiency of the whole road-traffic system as well.At present, many solutions have been put forward according to the research on vehicle path planning in the field of ITS, and practical algorithms can be roughly divided into three categories: traditional path planning algorithms represented by graph search and sampling methods, intelligent optimization algorithms based on swarm intelligence algorithms such as ant colony algorithm, and algorithms based on reinforcement learning (RL) and its derivatives. Traditional path planning algorithms need a lot of data support so as to get the final solution, and their solution ability for the unknown environment is poor, so it cannot cope with the changes of the environment. Among the three categories, RL algorithms stand out in practical application because they can better reflect the sequence characteristics. However, RL algorithms also have their own shortcomings, such as low data utilization and long iteration time. In order to better meet the needs of efficient operation of the transportation system, it is of great significance and has become an urgent task to build a better vehicle path planning algorithm.


### 1.2. Objective of Paper

#### 1.2.1. Research Motivation

Among the existing algorithms for vehicle path planning using RL theory with hyperparameters introduced, the mainstream is data-driven RL algorithm, which relies on millions of iterations and a large number of empirical data to learn strategies. Data-driven RL algorithm has the advantage of the good asymptotic performance, but the convergence speed is slow. On the other hand, it is noteworthy that mode-driven RL algorithm uses a differentiable transition model to improve the convergence speed, but the numerical operation is unstable and the operation efficiency is low. The existing mixed policy gradient method can integrate both data-driven and model-driven algorithms, but the numerical operation form lacks the intermediate metamorphosis of the connection between the two, which makes the amount of learning information insufficient. In addition, the mixed policy gradient method also has non-convex optimization objective function, which is difficult to carry out global optimization. And the selection cost of the search step size (hyperparameter) in the algorithm is too high to be performed manually.


#### 1.2.2. Main Contribution of This Paper

Driven by research motivation, in this paper, a mixed policy gradient actor-critic (AC) model with random escape term and filter operation is proposed to solve the vehicle path planning problem, the main contributions of this paper are as follows:(1)
From the existing literature, this paper combines the mixed policy gradient method with filter optimization technology to solve the vehicle path planning problem for the first time.(2)
In view of the lack of intermediate metamorphosis between data-driven and model-driven convergence in the existing mixed policy gradient methods, this paper proposes a new mixed policy gradient form and proposes a novel AC model on the basis of such mixed policy gradient.(3)
In view of the non-convex objective function and the difficulty of manually selecting the search step size, this paper combines the filter optimization technology with the noise gradient descent method (gradient descent method with escape term) to solve the above problems. On one hand, filter optimization technology can adaptively select the optimal step size, and on the other hand, the escape term can make the optimization search jump out of the trap of local minimum, so as to find the global optimal solution properly.


### 1.3. Paper Organization

In the overall model establishment, the vehicle path planning problem is abstracted firstly to realize the construction of AC model, and its solution method of using a mixed policy gradient is then discussed, and finally, the real data simulation is carried out in the experiment. The remainder of this paper is organized as follows: a comprehensive literature review from three aspects will be done in Section2, including the traditional path planning method, RL method, and filter optimization algorithm. Then in Section 3, starting from the RL of Monte Carlo process, the policy gradient of function approximation will be given, and then the AC algorithm model will be built on the background of vehicle path planning. In Section 4, the integration of the idea of a mixed gradient algorithm into the basic AC algorithm framework to form a mixed AC model will be introduced in detail. In Section 5, the model will be further improved by combining the filter optimization algorithm and random escape term, and then the model will be applied to vehicle path planning. In Section 6, the related numerical experiment will be discussed, and the comparison of the performance of the proposed model with other representative ones is discussed. Finally, the conclusion will be drawn in Section 7.


## 1.1. Backgrounds

In recent years, the rapid growth of China’s economy has greatly stimulated the growth of its domestic automobile consumption and has caused a rapid increase of the number of private vehicles. In the first half of 2020, the sales volume of China’s automobile market remained at the level of above 10 million, even if affected by the COVID-19 epidemic [1]. Traffic congestion caused by vehicle growth is one of the main reasons of traffic congestion, and it will also lead to problems of increased energy consumption and soaring carbon oxide emissions. In this context, the study of intelligent transportation system (ITS) attracts extensive attention. Vehicle path planning is a very important aspect of ITS, as it can not only help drivers to achieve the purpose of shortest distance, minimum cost and minimum travel duration under certain constraints, but help government or transportation management department to alleviate traffic congestion, and to improve the operation efficiency of the whole road-traffic system as well.At present, many solutions have been put forward according to the research on vehicle path planning in the field of ITS, and practical algorithms can be roughly divided into three categories: traditional path planning algorithms represented by graph search and sampling methods, intelligent optimization algorithms based on swarm intelligence algorithms such as ant colony algorithm, and algorithms based on reinforcement learning (RL) and its derivatives. Traditional path planning algorithms need a lot of data support so as to get the final solution, and their solution ability for the unknown environment is poor, so it cannot cope with the changes of the environment. Among the three categories, RL algorithms stand out in practical application because they can better reflect the sequence characteristics. However, RL algorithms also have their own shortcomings, such as low data utilization and long iteration time. In order to better meet the needs of efficient operation of the transportation system, it is of great significance and has become an urgent task to build a better vehicle path planning algorithm.


## 1.2. Objective of Paper

### 1.2.1. Research Motivation

Among the existing algorithms for vehicle path planning using RL theory with hyperparameters introduced, the mainstream is data-driven RL algorithm, which relies on millions of iterations and a large number of empirical data to learn strategies. Data-driven RL algorithm has the advantage of the good asymptotic performance, but the convergence speed is slow. On the other hand, it is noteworthy that mode-driven RL algorithm uses a differentiable transition model to improve the convergence speed, but the numerical operation is unstable and the operation efficiency is low. The existing mixed policy gradient method can integrate both data-driven and model-driven algorithms, but the numerical operation form lacks the intermediate metamorphosis of the connection between the two, which makes the amount of learning information insufficient. In addition, the mixed policy gradient method also has non-convex optimization objective function, which is difficult to carry out global optimization. And the selection cost of the search step size (hyperparameter) in the algorithm is too high to be performed manually.


### 1.2.2. Main Contribution of This Paper

Driven by research motivation, in this paper, a mixed policy gradient actor-critic (AC) model with random escape term and filter operation is proposed to solve the vehicle path planning problem, the main contributions of this paper are as follows:(1)
From the existing literature, this paper combines the mixed policy gradient method with filter optimization technology to solve the vehicle path planning problem for the first time.(2)
In view of the lack of intermediate metamorphosis between data-driven and model-driven convergence in the existing mixed policy gradient methods, this paper proposes a new mixed policy gradient form and proposes a novel AC model on the basis of such mixed policy gradient.(3)
In view of the non-convex objective function and the difficulty of manually selecting the search step size, this paper combines the filter optimization technology with the noise gradient descent method (gradient descent method with escape term) to solve the above problems. On one hand, filter optimization technology can adaptively select the optimal step size, and on the other hand, the escape term can make the optimization search jump out of the trap of local minimum, so as to find the global optimal solution properly.


## 1.2.1. Research Motivation

Among the existing algorithms for vehicle path planning using RL theory with hyperparameters introduced, the mainstream is data-driven RL algorithm, which relies on millions of iterations and a large number of empirical data to learn strategies. Data-driven RL algorithm has the advantage of the good asymptotic performance, but the convergence speed is slow. On the other hand, it is noteworthy that mode-driven RL algorithm uses a differentiable transition model to improve the convergence speed, but the numerical operation is unstable and the operation efficiency is low. The existing mixed policy gradient method can integrate both data-driven and model-driven algorithms, but the numerical operation form lacks the intermediate metamorphosis of the connection between the two, which makes the amount of learning information insufficient. In addition, the mixed policy gradient method also has non-convex optimization objective function, which is difficult to carry out global optimization. And the selection cost of the search step size (hyperparameter) in the algorithm is too high to be performed manually.


## 1.2.2. Main Contribution of This Paper

Driven by research motivation, in this paper, a mixed policy gradient actor-critic (AC) model with random escape term and filter operation is proposed to solve the vehicle path planning problem, the main contributions of this paper are as follows:(1)
From the existing literature, this paper combines the mixed policy gradient method with filter optimization technology to solve the vehicle path planning problem for the first time.(2)
In view of the lack of intermediate metamorphosis between data-driven and model-driven convergence in the existing mixed policy gradient methods, this paper proposes a new mixed policy gradient form and proposes a novel AC model on the basis of such mixed policy gradient.(3)
In view of the non-convex objective function and the difficulty of manually selecting the search step size, this paper combines the filter optimization technology with the noise gradient descent method (gradient descent method with escape term) to solve the above problems. On one hand, filter optimization technology can adaptively select the optimal step size, and on the other hand, the escape term can make the optimization search jump out of the trap of local minimum, so as to find the global optimal solution properly.


## 1.3. Paper Organization

In the overall model establishment, the vehicle path planning problem is abstracted firstly to realize the construction of AC model, and its solution method of using a mixed policy gradient is then discussed, and finally, the real data simulation is carried out in the experiment. The remainder of this paper is organized as follows: a comprehensive literature review from three aspects will be done in Section2, including the traditional path planning method, RL method, and filter optimization algorithm. Then in Section 3, starting from the RL of Monte Carlo process, the policy gradient of function approximation will be given, and then the AC algorithm model will be built on the background of vehicle path planning. In Section 4, the integration of the idea of a mixed gradient algorithm into the basic AC algorithm framework to form a mixed AC model will be introduced in detail. In Section 5, the model will be further improved by combining the filter optimization algorithm and random escape term, and then the model will be applied to vehicle path planning. In Section 6, the related numerical experiment will be discussed, and the comparison of the performance of the proposed model with other representative ones is discussed. Finally, the conclusion will be drawn in Section 7.


## 2. Literature Review

Vehicle path planning has a long research history and has already made remarkable progress. The traditional path planning model will be reviewed in this section as a start, and then the vehicle path planning model under the RL framework will be reviewed later on, and finally, the review of the filter optimization algorithm will be done. During the review process, the advantages and disadvantages of the vehicle path planning model will also be summarized, so as to lay the foundation for follow-up discussions.


### 2.1. Traditional Path Planning

At present, for the methods of path planning, many scholars have given relevant research in many fields. Guo et al. have proposed an optimized Dijkstra algorithm applied to automated guided vehicle (AGV), which has the advantage of shorter planned path, smaller turning angle, and strong search ability [2]. Qin et al. have proposed a path planning method for vehicle based on a graph search algorithm and geometric curve, such method can be divided into three stages: global path planning, which forms the global path; path coordination strategy, which gives a transition path under coordination; and parking path planning, which finally guides the vehicle from the original place to the destination [3]. Fendji et al. have proposed a path planning method for the unmanned aerial vehicle (UAV) in order to meet the minimum charging requirements and energy supply station number reduction under energy constraints [4]. Shin et al. have pointed out that the environmental uncertainty encountered by AGV degrades the overall performance of vehicle autonomous navigation, and have proposed a limited optimization, dynamic path planning model based on model predictive control (MPC), such model regards the dynamic model of AGV as equality constraint, and the limited state range and control input as inequality constraints. The stability of the proposed model is improved by passive constraints and particle swarm optimization algorithm, and the superiority of the proposed model has also been verified via the path planned [5].As in many cases of path planning, the workload required to find the best path may be very large. In order to improve the robustness and the performance in handling complex computation, Blackmore et al. have proposed a disjunctive program to optimize the path planning algorithm, which utilizes the existing constraint methods to form an optimal path, and such method operates directly in the workspace and reduces a lot of costs [6]. Xiong et al. have pointed out that the distribution path is the key problem of cold chain distribution so as to ensure goods to be delivered in time and to reduce the cost as much as possible, but the search time of traditional path planning is too long, thus, they have proposed a cold chain distribution routing optimization scheme based on improved ant colony algorithm which can successfully obtain the optimal distribution path of the enterprise within a predefined time [7]. Zhai et al. have proposed an effective path planning method through the porous structure, generalized Tyson polygon method is also used to find the subregion, and then genetic algorithm (GA) is used to solve the traveling salesman problem, traverse the path of each subregion, merge the subregions and improve the calculating efficiency [8]. Che et al. have proposed an improved ant colony optimization (ACO) algorithm based on particle swarm optimization algorithm with an autonomous underwater vehicle (AUV) as the object, by considering the constraints including energy and vision, combining with the selected node information of seabed environment, it can be effectively applied to the path planning of AUV in complex seabed environment, and can achieve fast calculation speed and high accuracy [9]. Li et al. have also proposed a fast method to obtain the optimal path based on heuristic set sampling and artificial potential field, their combination of artificial potential field and heuristic sampling reduces unnecessary sampling times, and speeds up the efficiency of the algorithm [10]. Morsali et al. have defined vehicle path planning as a convex optimization problem, and have tried to use support vector machine (SVM) to represent the surrounding environment, by classifying the search space as with-obstacle and obstacle-free, its complexity has been reduced, its heuristic has been effectively derived, and the search efficiency has been significantly improved by pruning technology [11]. Ding et al. have proposed an automatic vehicle path planning system based on partially observable Markov decision process (POMDP), which improves the efficiency of traditional Markov decision process (MDP) by guiding branches in action space and observation space, and have proved that it can be applied to high interaction environment [12]. Sarvesh et al. have proposed a method to locally reshape the global path planning based on the observation of obstacles in the environment by sensors, by combining the path sensing steering torque field and non-myopically optimization of steering action, such a method can effectively reduce the vehicle moving space and realize online path efficient planning [13].During recent years, research studies on path planning have paid more attention to the improvement of safety and comfort for the autonomous vehicle. Wang et al. have considered the combination of vehicle path planning and human preferences and have proposed a driver-vehicle-road model-based method considering environmental constraints to provide personalized path planning for different drivers under complex maneuvering conditions, and linear time-varying (LTV) MPC has been used for path planning [14]. Sun et al. have proposed an effective solution for vehicle path planning based on the Frenet framework, by considering the unexpected behavior of the reference line-based method in generating possible paths, effective correction of planned paths can be realized when the curvature of the reference line is large and the vehicle deviates far from the reference line [15]. Hu et al. have established a three-dimensional Voronoi diagram including obstacles, which can be applied to the path planning of unmanned differential steering vehicles (UDSVs), and have expressed the vehicle movement cost in nonlinear optimal control (NOC) by establishing an integrated potential field function in it, which can effectively realize the fast path planning in the off-road environment [16].


### 2.2. Reinforcement Learning in Path Planning

The core of RL is to solve the control problem and obtain the highest overall reward under a certain good policy. There are two kinds of RL algorithms, one is based on value function, and the other is based on policy gradient. Value function represents the reward obtained after a certain state or action is taken. In a deterministic environment, when the state value function is gotten, that is, the reward value is obtained under various states or actions, the next action that should be implemented in a certain state to obtain the maximum reward can then be determined. However, in a random environment, it is not convenient to obtain the state value function at first, and then carry out the optimization policy, as it cannot directly show what action should be taken next. The policy gradient-based algorithm is more direct, it does not need to get the value function to find the optimal policy, but directly parameterizes the policy function, and then uses the performance evaluation function to optimize the parameters in the policy gradient.Due to the extensive application of RL in the fields of the transportation systems, the research on RL has become more and more in-depth in recent years. Hartmann et al. have proposed a distributed predictive cruise control algorithm based on RL to reduce driving time and fuel consumption [17]. Haydari et al. have pointed out that data-driven methodology will bring profound changes to its and play a key role in ITS in the future, have discussed in detail the application of deep reinforcement learning (DRL) in problems such as traffic signal control (TSC) and autonomous driving, and have pointed out that TSC can be abstracted into different problem formulas and involve different RL parameters [18]. Kiran et al. have focused on the application of DRL in different target tasks of autonomous driving, and have discussed solutions to verify, test, and improve the stability of existing DRL algorithms on the basis of discussing the challenges faced by autonomous driving agents in computing in actual deployment [19]. Aradi has pointed out that there are differences in the solutions given by DRL models for different task levels and has pointed out that the design keys of the DRL algorithm include environment modeling, modeling abstraction, description of state and perception models, appropriate rewards, and the implementation of the underlying neural network (NN) [20].In Internet of vehicles (IoV), each vehicle usually uses a routing protocol to choose its own path. In the traditional routing protocol, the network topology changes rapidly, the routing scheme is vulnerable to influence and causes data loss, Omar et al. have used RL combining with the traditional location-based routing protocol - greedy perimeter stateless routing (GPSR) to achieve the goal under changeable circumstances and has achieved good results [21]. Likewise, Eskandari et al. have introduced a strategy that uses the distributed mode of simple protocol to replace the original centralized structure in rapid changing vehicle networks and have used fuzzy logic management and RL with advantages in solving dynamic environment control problems, so as to improve the efficiency of the network and manage the movement of vehicles [22]. In addition to the path planning control of traditional vehicles, with the popularity of electric vehicles (EV), the issues of EV path planning as well as energy management are also essential. Zhou et al. have proposed a multi-step learning control algorithm based on RL to realize the online monitoring of EV path planning with energy management considered [23].The application of RL also plays a great role in the path control of UAV and self-driving vehicles. Zeng et al., Yang et al., and Li et al. have pointed out that the movement or task execution process of UAV is a continuous control problem in a changing environment, and RL and deep deterministic policy gradient in DRL can be used to better realize the control process of UAV [24–26]. Compared to the path control of UAV, driverless vehicle in recent years is more complicated and has a more complex environment. Ichikawa et al. have proposed a linear discrete-time model combining with RL to deal with the network delay between people driving vehicle and driverless vehicles [27]. Ye et al. have used DRL to put forward path planning for the vehicle network and calculate and control the path information of each step to achieve the overall path optimization [28].


### 2.3. Filter Optimization Algorithm

Filter optimization, which has been first proposed by Fletcher et al., has the main idea of accepting the heuristic point if it can meet certain filtering rules [29]. As it can ensure the global convergence of the algorithm for solving nonlinear constrained programming, the filter optimization algorithm has been widely studied in dealing with nonlinear programming problems [30–32]. Nie has used filter optimization to solve the nonlinear complementarity problem (NCP) and has also proposed a new filter algorithm that can establish local Q-superlinear and Q-quadratic convergence [33]. Long et al. have proposed a new Filter-Levenberg–Marquardt method on the basis of filtering optimization to solve the NCP which have been reformulated as the minimum of L2 norm solution, and the global convergence of the method proposed has been proved [34]. Zhang et al. have studied the nonlinear second-order cone programming problem and have combined sequence quadratic program (SQP) and filtering optimization together, under standard assumptions, it is proved that the iterative sequence generated by their proposed algorithm has global convergence [35]. Hou et al. have also introduced the filter optimization algorithm in solving NCP so as to make the calculation scale be reduced to a certain extent, to be specific, each iteration only needs to solve one set of linear equations; they have also proved the global convergence of the method under certain reasonable conditions [36]. Li et al. have proposed an affine scaling interior trust region method based on non-monotonic line search filtering technology to solve nonlinear optimization problems constrained by linear inequalities, by combining with the new backtracking correlation conditions, the global convergence of the solution can be ensured [37]. Li et al. have also proposed an affine scaling trust region method based online search filter technology to solve nonlinear optimization problems with bounded variables and have proved that under reasonable assumptions, this method has good global convergence and fast local convergence speed [38]. Wang et al. have reconstructed an NCP into a constrained optimization problem solved by a filter optimization algorithm, their defined regular conditions and regular points have proved that the level set of the constrained optimization objective function is bounded for a strongly monotonic function or a uniform P-function [39].


## 2.1. Traditional Path Planning

At present, for the methods of path planning, many scholars have given relevant research in many fields. Guo et al. have proposed an optimized Dijkstra algorithm applied to automated guided vehicle (AGV), which has the advantage of shorter planned path, smaller turning angle, and strong search ability [2]. Qin et al. have proposed a path planning method for vehicle based on a graph search algorithm and geometric curve, such method can be divided into three stages: global path planning, which forms the global path; path coordination strategy, which gives a transition path under coordination; and parking path planning, which finally guides the vehicle from the original place to the destination [3]. Fendji et al. have proposed a path planning method for the unmanned aerial vehicle (UAV) in order to meet the minimum charging requirements and energy supply station number reduction under energy constraints [4]. Shin et al. have pointed out that the environmental uncertainty encountered by AGV degrades the overall performance of vehicle autonomous navigation, and have proposed a limited optimization, dynamic path planning model based on model predictive control (MPC), such model regards the dynamic model of AGV as equality constraint, and the limited state range and control input as inequality constraints. The stability of the proposed model is improved by passive constraints and particle swarm optimization algorithm, and the superiority of the proposed model has also been verified via the path planned [5].As in many cases of path planning, the workload required to find the best path may be very large. In order to improve the robustness and the performance in handling complex computation, Blackmore et al. have proposed a disjunctive program to optimize the path planning algorithm, which utilizes the existing constraint methods to form an optimal path, and such method operates directly in the workspace and reduces a lot of costs [6]. Xiong et al. have pointed out that the distribution path is the key problem of cold chain distribution so as to ensure goods to be delivered in time and to reduce the cost as much as possible, but the search time of traditional path planning is too long, thus, they have proposed a cold chain distribution routing optimization scheme based on improved ant colony algorithm which can successfully obtain the optimal distribution path of the enterprise within a predefined time [7]. Zhai et al. have proposed an effective path planning method through the porous structure, generalized Tyson polygon method is also used to find the subregion, and then genetic algorithm (GA) is used to solve the traveling salesman problem, traverse the path of each subregion, merge the subregions and improve the calculating efficiency [8]. Che et al. have proposed an improved ant colony optimization (ACO) algorithm based on particle swarm optimization algorithm with an autonomous underwater vehicle (AUV) as the object, by considering the constraints including energy and vision, combining with the selected node information of seabed environment, it can be effectively applied to the path planning of AUV in complex seabed environment, and can achieve fast calculation speed and high accuracy [9]. Li et al. have also proposed a fast method to obtain the optimal path based on heuristic set sampling and artificial potential field, their combination of artificial potential field and heuristic sampling reduces unnecessary sampling times, and speeds up the efficiency of the algorithm [10]. Morsali et al. have defined vehicle path planning as a convex optimization problem, and have tried to use support vector machine (SVM) to represent the surrounding environment, by classifying the search space as with-obstacle and obstacle-free, its complexity has been reduced, its heuristic has been effectively derived, and the search efficiency has been significantly improved by pruning technology [11]. Ding et al. have proposed an automatic vehicle path planning system based on partially observable Markov decision process (POMDP), which improves the efficiency of traditional Markov decision process (MDP) by guiding branches in action space and observation space, and have proved that it can be applied to high interaction environment [12]. Sarvesh et al. have proposed a method to locally reshape the global path planning based on the observation of obstacles in the environment by sensors, by combining the path sensing steering torque field and non-myopically optimization of steering action, such a method can effectively reduce the vehicle moving space and realize online path efficient planning [13].During recent years, research studies on path planning have paid more attention to the improvement of safety and comfort for the autonomous vehicle. Wang et al. have considered the combination of vehicle path planning and human preferences and have proposed a driver-vehicle-road model-based method considering environmental constraints to provide personalized path planning for different drivers under complex maneuvering conditions, and linear time-varying (LTV) MPC has been used for path planning [14]. Sun et al. have proposed an effective solution for vehicle path planning based on the Frenet framework, by considering the unexpected behavior of the reference line-based method in generating possible paths, effective correction of planned paths can be realized when the curvature of the reference line is large and the vehicle deviates far from the reference line [15]. Hu et al. have established a three-dimensional Voronoi diagram including obstacles, which can be applied to the path planning of unmanned differential steering vehicles (UDSVs), and have expressed the vehicle movement cost in nonlinear optimal control (NOC) by establishing an integrated potential field function in it, which can effectively realize the fast path planning in the off-road environment [16].


## 2.2. Reinforcement Learning in Path Planning

The core of RL is to solve the control problem and obtain the highest overall reward under a certain good policy. There are two kinds of RL algorithms, one is based on value function, and the other is based on policy gradient. Value function represents the reward obtained after a certain state or action is taken. In a deterministic environment, when the state value function is gotten, that is, the reward value is obtained under various states or actions, the next action that should be implemented in a certain state to obtain the maximum reward can then be determined. However, in a random environment, it is not convenient to obtain the state value function at first, and then carry out the optimization policy, as it cannot directly show what action should be taken next. The policy gradient-based algorithm is more direct, it does not need to get the value function to find the optimal policy, but directly parameterizes the policy function, and then uses the performance evaluation function to optimize the parameters in the policy gradient.Due to the extensive application of RL in the fields of the transportation systems, the research on RL has become more and more in-depth in recent years. Hartmann et al. have proposed a distributed predictive cruise control algorithm based on RL to reduce driving time and fuel consumption [17]. Haydari et al. have pointed out that data-driven methodology will bring profound changes to its and play a key role in ITS in the future, have discussed in detail the application of deep reinforcement learning (DRL) in problems such as traffic signal control (TSC) and autonomous driving, and have pointed out that TSC can be abstracted into different problem formulas and involve different RL parameters [18]. Kiran et al. have focused on the application of DRL in different target tasks of autonomous driving, and have discussed solutions to verify, test, and improve the stability of existing DRL algorithms on the basis of discussing the challenges faced by autonomous driving agents in computing in actual deployment [19]. Aradi has pointed out that there are differences in the solutions given by DRL models for different task levels and has pointed out that the design keys of the DRL algorithm include environment modeling, modeling abstraction, description of state and perception models, appropriate rewards, and the implementation of the underlying neural network (NN) [20].In Internet of vehicles (IoV), each vehicle usually uses a routing protocol to choose its own path. In the traditional routing protocol, the network topology changes rapidly, the routing scheme is vulnerable to influence and causes data loss, Omar et al. have used RL combining with the traditional location-based routing protocol - greedy perimeter stateless routing (GPSR) to achieve the goal under changeable circumstances and has achieved good results [21]. Likewise, Eskandari et al. have introduced a strategy that uses the distributed mode of simple protocol to replace the original centralized structure in rapid changing vehicle networks and have used fuzzy logic management and RL with advantages in solving dynamic environment control problems, so as to improve the efficiency of the network and manage the movement of vehicles [22]. In addition to the path planning control of traditional vehicles, with the popularity of electric vehicles (EV), the issues of EV path planning as well as energy management are also essential. Zhou et al. have proposed a multi-step learning control algorithm based on RL to realize the online monitoring of EV path planning with energy management considered [23].The application of RL also plays a great role in the path control of UAV and self-driving vehicles. Zeng et al., Yang et al., and Li et al. have pointed out that the movement or task execution process of UAV is a continuous control problem in a changing environment, and RL and deep deterministic policy gradient in DRL can be used to better realize the control process of UAV [24–26]. Compared to the path control of UAV, driverless vehicle in recent years is more complicated and has a more complex environment. Ichikawa et al. have proposed a linear discrete-time model combining with RL to deal with the network delay between people driving vehicle and driverless vehicles [27]. Ye et al. have used DRL to put forward path planning for the vehicle network and calculate and control the path information of each step to achieve the overall path optimization [28].


## 2.3. Filter Optimization Algorithm

Filter optimization, which has been first proposed by Fletcher et al., has the main idea of accepting the heuristic point if it can meet certain filtering rules [29]. As it can ensure the global convergence of the algorithm for solving nonlinear constrained programming, the filter optimization algorithm has been widely studied in dealing with nonlinear programming problems [30–32]. Nie has used filter optimization to solve the nonlinear complementarity problem (NCP) and has also proposed a new filter algorithm that can establish local Q-superlinear and Q-quadratic convergence [33]. Long et al. have proposed a new Filter-Levenberg–Marquardt method on the basis of filtering optimization to solve the NCP which have been reformulated as the minimum of L2 norm solution, and the global convergence of the method proposed has been proved [34]. Zhang et al. have studied the nonlinear second-order cone programming problem and have combined sequence quadratic program (SQP) and filtering optimization together, under standard assumptions, it is proved that the iterative sequence generated by their proposed algorithm has global convergence [35]. Hou et al. have also introduced the filter optimization algorithm in solving NCP so as to make the calculation scale be reduced to a certain extent, to be specific, each iteration only needs to solve one set of linear equations; they have also proved the global convergence of the method under certain reasonable conditions [36]. Li et al. have proposed an affine scaling interior trust region method based on non-monotonic line search filtering technology to solve nonlinear optimization problems constrained by linear inequalities, by combining with the new backtracking correlation conditions, the global convergence of the solution can be ensured [37]. Li et al. have also proposed an affine scaling trust region method based online search filter technology to solve nonlinear optimization problems with bounded variables and have proved that under reasonable assumptions, this method has good global convergence and fast local convergence speed [38]. Wang et al. have reconstructed an NCP into a constrained optimization problem solved by a filter optimization algorithm, their defined regular conditions and regular points have proved that the level set of the constrained optimization objective function is bounded for a strongly monotonic function or a uniform P-function [39].


## 3. Vehicle Path Planning Based on Actor-Critic Model

In this section, the policy gradient method in RL is briefly deduced through the Markov decision process, and the AC algorithm is reviewed, moreover, the AC algorithm is combined with the background of vehicle path planning to build a vehicle path planning model based on AC model.


### 3.1. Derivation of Policy Gradient

The goal of RL is to find an optimal behavior policy for agents, so as to obtain the maximum return. The main feature of the policy gradient method is to directly model and optimize the policy. The method of using policy gradient to solve RL was first proposed in 1992 [40]. Then, Sutton et al. proposed a theoretical formula to estimate the gradient [41]. In this paper, MDP has been selected to model the RL problem, and tuple (S, A, p, R, γ) has been used to define the process, where(1)
S is the state space, st ∈ S represents the state of the agent at time t. When the system is at t = 0, the system will start from the initial state s0.(2)
A is the action space. at ∈ A represents the action of the agent at time t.(3)
p is the dynamics or transition distribution. After the action at is executed in the current state st, the probability distribution of transferring to the next state st+1 is recorded as p(st+1|st, at). This operation samples from the policy distribution, given by πθat|st,θ, and the parameter is θ. πθat|st,θ is expressed the corresponding policy of state st parameter vector θ.(4)
R is the reward function, expressed as S × A ⟶ R. After completing a state transition, a reward rt will be received, where rt ∼ R(st, at). The discount sum of the reward is expressed as Rτ=∑t=1Tγtrt, where γ ∈ (0, 1] is the discount coefficient, τ represents a sampling period with states and actions, τ = (s1, a1, ..., st, at), T represents the number of state action pairs on the period. Therefore, the probability distribution on sampling is as follows:(1)pτ=ρs1∏t=1Tpst+1|st,atπθat|st,θ,whereρ(s1) represents the distribution containing the initial state. In the policy gradient method, as long as ∇πθat|st,θ exists and is always limited, the policy πθat|st can be parameterized in any way.In solving the objective function here in this paper, the maximization of the objective functionθ will be focused. Solving RL based on approximation mainly includes 3 ideas: value based, policy based, and model based. In the process of function approximation, the relationship between policy parameters and policy gradient is as follows:(2)θ⟵θ+α∇θJθ,where α is a positive definite step size. In the learning process of RL, the objective function is an important index to measure its operation. For the calculation form of obtaining a reward for an action, taking the maximum discount sum as the objective function can well reflect the integrity of the algorithm. The J(θ) expression and the expression of exhausting all possibilities according to the law of large numbers can be given here as(3)Jθ=Es∼ρs∑tRst,at≈1N∑i∑tRsi,t,ai,t.The discount and reward of the sampled action can be defined by the average reward or start state. If∑aQws,a;w is used to represent the discount and reward obtained by the network according to the parameterized policy in state s, In the MDP process, whether ∑aQws,a;w is expressed by the average reward or start state formula, the policy gradient method can obtain the following formula:(4)∇Jθ∝∑sμs∑aQws,a;w∇θπθat|st,θ.In this formula, the gradient is proportional to the right-hand side, whereμ(s) is the stable distribution of the state. For the overall framework of RL, its core goal is to obtain the maximum objective function and obtain the corresponding policy of the objective function. However, such policy estimation has great variance and is not convenient for the iteration of the model.


### 3.2. Core Algorithm of Actor-Critic Model

In order to reduce the variance estimation when using the policy gradient, and to realize the online or continuous problem conveniently, in literature [42], an AC model with guiding critics has been proposed. In RL, if the policy function and value function have been learned at the same time, the method is generally called the AC method. To be specific, actor and critical will learn policy function and value function, respectively. There is no obvious difference between AC algorithm and the policy gradient algorithm in generating samples and policy improvement. The main change of the policy gradient algorithm is to obtain better gradient estimation by fitting a value function.In AC model, actors use the stochastic gradient descent (SGD) method to update the policyπθat|st,θ, and the direction of iteration of parameter θ is just the expectation of the maximized reward. At the same time, the critic uses the action value function estimated in the form of time difference error to replace the unknown action-value function, and the time difference error output drives the update process of actors and critics. In terms of algorithm principle, actors are responsible for calculating the evaluation function in the policy gradient theorem, and critics are responsible for updating the action value function about the policy.In the AC model, the original policy gradient (4) can be rewritten as(5)∇Jθ≈∑t=1TRst,at∑t=1T∇logπθst,at|θwhere ∑t=1TRst,at is the sum of all rewards and ∑t=1T∇logπθst,at|θ is the logarithmic expression of ∇πθst,at|θ. As in the process of solving the gradient, ∑t=1TRst,at is often unstable. Therefore, the baseline method b is often introduced in the calculation process, that is, the policy gradient is replaced by(6)∇Jθ≈∑t=1TRst,at−b∑t=1T∇logπθst,at|θ.In AC model,∑t=1TRst,at−b is usually called advantage function and expressed by Aπst,at. The advantage function refers to the change of the expected future return of action at compared with the average situation of baseline b in state st. The value function Vπst of the neural network is used to estimate the value of b, and its formula is as follows:(7)Vπst=Eat∼πθat|stQπst,at,where Qπst,at is called the reward function under the policy, and Vπst can also be called the expectation of the function under the given policy. In this paper, the Monte Carlo method is used to approximate Vπst after one trajectory sampling to obtain the following equation:(8)Vπst≈∑t′=tTRst′,at′.As it is complex to estimate the two networks in the algorithm, in order to reduce the complexity, when calculating the reward function, the expectation is estimated by the reward of the next state in the trajectory sample, thenQπst,at can be approximately expressed as(9)Qπst,at≈Rst,at+Vπst+1.And the advantage function can be approximated as(10)Aπst,at≈Rst,at+Vπst+1−Vπst.As the dynamic temporal difference (TD) errorδt of updating the network between actor and critic, it can be calculated by TD as(11)δt=rt+1−r¯t+1+ζVπst+1−Vπst,where r¯t+1 represents the average reward at the next moment, and its specific framework will be given together with the vehicle path planning problem, ζ is an error coefficient.


### 3.3. Actor-Critic Model for Vehicle Path Planning

According to the essence of vehicle path planning, the very planning problem can be abstracted into a set of nodes (n1, ..., nm) and a problem constituted by θ parameterized policy π. The feasible solution can be obtained through policy training. Each node corresponds to a solution, that is, the i-th action ai corresponds to the i-th node n. The sequence composed of ai is called the sequence of solutions (L = a1, ..., ai). Under the guidance of random strategy πθat|s, the neural network will generate a node solution every time it completes the regression. Here, the vehicle path planning problem can be understood as that the vehicle performs the i-th action ai, namely, selects a node. Through a series of solutions to guide the vehicle to different nodes, so as to achieve the function of path planning. According to this idea, Figure 1 is obtained.Figure 1
Basic rocket ship design. The rocket ship is propelled with three thrusters and features a single viewing window. The nose cone is detachable upon impact.First, a basic model of vehicle path planning has been proposed here. The policy networkπθa|s;θ is equivalent to the actor, that is, the vehicle’s choice of path. It makes action a based on state s. Value network Qws,a,w is equivalent to critic. It scores the performance of actors, that is, path selection and evaluates the quality of action a under the condition of state s. In order to alleviate the bootstrap problem, the target network is introduced, and the experience playback technology is used for training. The target network can be recorded as Qws,a,w−. Its structure is the same as that of the value network, but its parameters are different. Suppose that the policy parameter of the current network is θnow, and the value network parameter is wnow:(i)
Observe the current statest, make a decision according to the policy network: at∼πθ⋅|st;θnow, and let the agent perform the action at.(ii)
Observe rewardrt and new state st+1 from the environment.(iii)
Make decisionsa˜t∼πθ⋅|st+1;θnow according to the policy network, but do not let the agent perform action at+1.(iv)
Let the value network rate (st, at): Q^w,t=Qwst,at,wnow.(v)
Let the target network ratest+1,a˜t+1,: Q^w,t+1=Qwst+1,a˜t+1,wnow−.(vi)
Calculate TD target and TD error:y^t=rt+γ⋅Q^w,t+1, and δt=Q^w,t−y^t.(vii)
Update value network:wnew⟵wnow−α⋅δt∇wQwst,at;wnow, where α is a hyperparameter.(viii)
Update policy network:θnew⟵θnow−β⋅Q^w,t∇θlnπθ⋅|st;θnow, where β is also a hyperparameter.(ix)
Setτ0∈0,1 as a target smoothing coefficient and update the parameters of the target network by weighted average: wnew−⟵τ0⋅wnow+1−τ0⋅wnow−.


## 3.1. Derivation of Policy Gradient

The goal of RL is to find an optimal behavior policy for agents, so as to obtain the maximum return. The main feature of the policy gradient method is to directly model and optimize the policy. The method of using policy gradient to solve RL was first proposed in 1992 [40]. Then, Sutton et al. proposed a theoretical formula to estimate the gradient [41]. In this paper, MDP has been selected to model the RL problem, and tuple (S, A, p, R, γ) has been used to define the process, where(1)
S is the state space, st ∈ S represents the state of the agent at time t. When the system is at t = 0, the system will start from the initial state s0.(2)
A is the action space. at ∈ A represents the action of the agent at time t.(3)
p is the dynamics or transition distribution. After the action at is executed in the current state st, the probability distribution of transferring to the next state st+1 is recorded as p(st+1|st, at). This operation samples from the policy distribution, given by πθat|st,θ, and the parameter is θ. πθat|st,θ is expressed the corresponding policy of state st parameter vector θ.(4)
R is the reward function, expressed as S × A ⟶ R. After completing a state transition, a reward rt will be received, where rt ∼ R(st, at). The discount sum of the reward is expressed as Rτ=∑t=1Tγtrt, where γ ∈ (0, 1] is the discount coefficient, τ represents a sampling period with states and actions, τ = (s1, a1, ..., st, at), T represents the number of state action pairs on the period. Therefore, the probability distribution on sampling is as follows:(1)pτ=ρs1∏t=1Tpst+1|st,atπθat|st,θ,whereρ(s1) represents the distribution containing the initial state. In the policy gradient method, as long as ∇πθat|st,θ exists and is always limited, the policy πθat|st can be parameterized in any way.In solving the objective function here in this paper, the maximization of the objective functionθ will be focused. Solving RL based on approximation mainly includes 3 ideas: value based, policy based, and model based. In the process of function approximation, the relationship between policy parameters and policy gradient is as follows:(2)θ⟵θ+α∇θJθ,where α is a positive definite step size. In the learning process of RL, the objective function is an important index to measure its operation. For the calculation form of obtaining a reward for an action, taking the maximum discount sum as the objective function can well reflect the integrity of the algorithm. The J(θ) expression and the expression of exhausting all possibilities according to the law of large numbers can be given here as(3)Jθ=Es∼ρs∑tRst,at≈1N∑i∑tRsi,t,ai,t.The discount and reward of the sampled action can be defined by the average reward or start state. If∑aQws,a;w is used to represent the discount and reward obtained by the network according to the parameterized policy in state s, In the MDP process, whether ∑aQws,a;w is expressed by the average reward or start state formula, the policy gradient method can obtain the following formula:(4)∇Jθ∝∑sμs∑aQws,a;w∇θπθat|st,θ.In this formula, the gradient is proportional to the right-hand side, whereμ(s) is the stable distribution of the state. For the overall framework of RL, its core goal is to obtain the maximum objective function and obtain the corresponding policy of the objective function. However, such policy estimation has great variance and is not convenient for the iteration of the model.


## 3.2. Core Algorithm of Actor-Critic Model

In order to reduce the variance estimation when using the policy gradient, and to realize the online or continuous problem conveniently, in literature [42], an AC model with guiding critics has been proposed. In RL, if the policy function and value function have been learned at the same time, the method is generally called the AC method. To be specific, actor and critical will learn policy function and value function, respectively. There is no obvious difference between AC algorithm and the policy gradient algorithm in generating samples and policy improvement. The main change of the policy gradient algorithm is to obtain better gradient estimation by fitting a value function.In AC model, actors use the stochastic gradient descent (SGD) method to update the policyπθat|st,θ, and the direction of iteration of parameter θ is just the expectation of the maximized reward. At the same time, the critic uses the action value function estimated in the form of time difference error to replace the unknown action-value function, and the time difference error output drives the update process of actors and critics. In terms of algorithm principle, actors are responsible for calculating the evaluation function in the policy gradient theorem, and critics are responsible for updating the action value function about the policy.In the AC model, the original policy gradient (4) can be rewritten as(5)∇Jθ≈∑t=1TRst,at∑t=1T∇logπθst,at|θwhere ∑t=1TRst,at is the sum of all rewards and ∑t=1T∇logπθst,at|θ is the logarithmic expression of ∇πθst,at|θ. As in the process of solving the gradient, ∑t=1TRst,at is often unstable. Therefore, the baseline method b is often introduced in the calculation process, that is, the policy gradient is replaced by(6)∇Jθ≈∑t=1TRst,at−b∑t=1T∇logπθst,at|θ.In AC model,∑t=1TRst,at−b is usually called advantage function and expressed by Aπst,at. The advantage function refers to the change of the expected future return of action at compared with the average situation of baseline b in state st. The value function Vπst of the neural network is used to estimate the value of b, and its formula is as follows:(7)Vπst=Eat∼πθat|stQπst,at,where Qπst,at is called the reward function under the policy, and Vπst can also be called the expectation of the function under the given policy. In this paper, the Monte Carlo method is used to approximate Vπst after one trajectory sampling to obtain the following equation:(8)Vπst≈∑t′=tTRst′,at′.As it is complex to estimate the two networks in the algorithm, in order to reduce the complexity, when calculating the reward function, the expectation is estimated by the reward of the next state in the trajectory sample, thenQπst,at can be approximately expressed as(9)Qπst,at≈Rst,at+Vπst+1.And the advantage function can be approximated as(10)Aπst,at≈Rst,at+Vπst+1−Vπst.As the dynamic temporal difference (TD) errorδt of updating the network between actor and critic, it can be calculated by TD as(11)δt=rt+1−r¯t+1+ζVπst+1−Vπst,where r¯t+1 represents the average reward at the next moment, and its specific framework will be given together with the vehicle path planning problem, ζ is an error coefficient.


## 3.3. Actor-Critic Model for Vehicle Path Planning

According to the essence of vehicle path planning, the very planning problem can be abstracted into a set of nodes (n1, ..., nm) and a problem constituted by θ parameterized policy π. The feasible solution can be obtained through policy training. Each node corresponds to a solution, that is, the i-th action ai corresponds to the i-th node n. The sequence composed of ai is called the sequence of solutions (L = a1, ..., ai). Under the guidance of random strategy πθat|s, the neural network will generate a node solution every time it completes the regression. Here, the vehicle path planning problem can be understood as that the vehicle performs the i-th action ai, namely, selects a node. Through a series of solutions to guide the vehicle to different nodes, so as to achieve the function of path planning. According to this idea, Figure 1 is obtained.Figure 1
Basic rocket ship design. The rocket ship is propelled with three thrusters and features a single viewing window. The nose cone is detachable upon impact.First, a basic model of vehicle path planning has been proposed here. The policy networkπθa|s;θ is equivalent to the actor, that is, the vehicle’s choice of path. It makes action a based on state s. Value network Qws,a,w is equivalent to critic. It scores the performance of actors, that is, path selection and evaluates the quality of action a under the condition of state s. In order to alleviate the bootstrap problem, the target network is introduced, and the experience playback technology is used for training. The target network can be recorded as Qws,a,w−. Its structure is the same as that of the value network, but its parameters are different. Suppose that the policy parameter of the current network is θnow, and the value network parameter is wnow:(i)
Observe the current statest, make a decision according to the policy network: at∼πθ⋅|st;θnow, and let the agent perform the action at.(ii)
Observe rewardrt and new state st+1 from the environment.(iii)
Make decisionsa˜t∼πθ⋅|st+1;θnow according to the policy network, but do not let the agent perform action at+1.(iv)
Let the value network rate (st, at): Q^w,t=Qwst,at,wnow.(v)
Let the target network ratest+1,a˜t+1,: Q^w,t+1=Qwst+1,a˜t+1,wnow−.(vi)
Calculate TD target and TD error:y^t=rt+γ⋅Q^w,t+1, and δt=Q^w,t−y^t.(vii)
Update value network:wnew⟵wnow−α⋅δt∇wQwst,at;wnow, where α is a hyperparameter.(viii)
Update policy network:θnew⟵θnow−β⋅Q^w,t∇θlnπθ⋅|st;θnow, where β is also a hyperparameter.(ix)
Setτ0∈0,1 as a target smoothing coefficient and update the parameters of the target network by weighted average: wnew−⟵τ0⋅wnow+1−τ0⋅wnow−.


## 4. Mixed Policy Gradient Actor-Critic Model

In this section, the data-driven and model-driven policy gradient method based on the policy gradient formula is discussed; moreover, inspired by literature [43], the calculation formula of the mixed policy gradient has been modified, the intermediate state gradient information has been added, and make the algorithm can be made to get more learnable information. On this basis, a new AC algorithm has been proposed. Of course, this algorithm is still troubled by the problem of non-convex objective function and step size selection of hyperparameters, so in Section 5, an original solution will be given.


### 4.1. Data-Driven and Model-Driven Policy Gradient

Before integrating the mixed gradient algorithm into the AC algorithm, it is necessary to clarify what is data-driven and model-driven policy gradient algorithm. These two driven methods are also used to solve the gradient problem in the maximum problem of the objective function. During the discussion of the driven policy gradients, the objective function will be simply represented by the value functionqπs,a in the form of a future discount sum:(12)qπs,a=Esl,l>t∑l=t∞γl−tRsl,al|st=s,at=a,where l indicates the number of action state pairs. Then the gradient of the objective function can be expressed as(13)∇θJθ=Est∇atqπst,at∇θπθ.As for model-driven and data-driven policy gradient, the main difference lies in what form to approximate the functionqπ. Network Qw is used to approximate qπ, and define Q^w,t=Qws^t,a^t, where s^t=st,a^t=at. For model-driven gradient, as RL is modeled by Markov process, this property determines that the value function can be written as the n-th recursive form of the Behrman equation. Therefore, Q^w,t+n≜Qws^t+nπθs^t+n is used to represent the Behrman approximation function of each step, and the model-driven policy obtained by n-step Behrman can be expressed as(14)∇θJnMθ=Es^t∇θEs^t+1,...,s^t+n∑l=tn−1+tγl−tR^l+γnQ^w,t+n,where s^t+i is the state transition function under the parameterized policy s^t+i∼f⋅|s^t+i−1,πθs^t+i−1,i≥1, where f is the model specified to be used in the model-driven policy gradient, R^t+i≜Rs^t+i,πθs^t+i is expressed as the sum of returns under the parameterized policy, where i≥0.Data-driven gradient, as the name suggests, is a type of policy through a large number of iterations and empirical data. For data-driven, the value function can be approximated by theQw generated by a large amount of data [44]. The policy gradient driven by numerical value can be written as(15)∇θJDθ=Est∇atQw∇θπθ.


### 4.2. Data-Driven and Model-Driven Mixed Policy Gradient

No matter data-driven or model-driven, there will be some defects in the calculation. Data-driven RL highlights its excellent asymptotic performance at the cost of slow convergence speed, while model-driven RL uses a differential transformation model with the disadvantages of large numerical fluctuation and low error sensitivity to improve the convergence speed. If the two are mixed directly, the accuracy of the policy gradient will change with the error of the two after mixing, and the process will become uncontrollable. Therefore, in the process of constructing the mixed policy gradient, it is need to combine the advantages of the two ideas, so the unity between data-driven and model-driven policy gradient should be found firstly, that is, a unified gradient concept should be proposed to facilitate the subsequent model construction. Moreover, during this process, balancing the error of the two ideas is the key to realize the high-precision mixing of policy gradient. In addition to ensuring the accuracy of the model, the convergence speed of the model should also be considered. A faster convergence speed can better realize the transplantation of the algorithm to vehicle path planning practices and produce more practical benefits. Based on these considerations, in this sectionn-step Behrman is used to recursively approximate the value function to obtain a unified policy gradient form as follows:(16)∇θJnUθ=Es^t∇a^tEs^t+1,...,s^t+n∑l=tn−1+tγl−tR^l+γnQ^w,t+n∇θπθ.For this kind of mixed gradient unification, in the discount rater ⟶ 1, the transition state probability in the MDP process is a steady-state distribution, thus, the model-driven policy gradient acts to reduce the dependence on Q value. From the theorem [39]:(17)∇θJnMθ=∑k=0n−1γk∇θJn−kU,It can be proved that the model-driven policy gradient is the weighted average of the unified policy gradient, and each weight will increase exponentially with the recursive upper limit n - k in the unified policy gradient. However, in the overall unified policy gradient adjustment, the disadvantage in asymptotic performance in model-driven policy gradient will also lead to the irrationality of the model. Therefore, in this paper, a better way is supposed to be found so as to update the weight and establish a unified mixed policy gradient.Combining the above two (16) and (17), and the characteristics of data-driven and model-driven policy gradient, it can be known that in the early stage of model training, the real value is usually quite different from the Q value of data-driven numerical estimation. Therefore, a larger n in model-driven method can reduce the problems caused by the estimation error in data-driven method. With the continuous advancement of iteration, the n-step recursion of large values will destroy its asymptotic performance. Therefore, the value of n can be reduced for subsequent iterations.


### 4.3. Mixed Policy Gradient Actor-Critic Model

In this paper, the loss∇θJ0Uθ, ∇θJL/2Uθ, and ∇θJLUθ obtained from the reverse of the unified policy gradient are used to form the mixed policy gradient, where [·] is the integral function. According to the content of the unified policy gradient, the former is purely data-driven and the latter is model-driven by Behrman approximation. Since the approaching to infinity in n steps cannot be realized, L is expressed as a larger number in ∇θJLUθ. Then the form of mixed policy gradient can be expressed as:(18)∇θJMixθ=w0∇θJ0Uθ+wL/2∇θJL/2Uθ+wL∇θJLUθ,where w0, wL/2, and wL correspond to the weights to be determined for data-driven, intermediate metamorphosis, and model-driven policy gradients, respectively. Moreover, the following assumptions are made for the mixed policy gradient:(i)
At the beginning of training, the error of the mixed model mainly comes from the data-driven part; at the later stage of training, when theQ value estimation error is small, the error mainly comes from the model-driven part.(ii)
The unified policy gradient error changes exponentially, decreases in the early stage and increases in the later stage.(iii)
In order to simplify the formula, mixed policy gradient expressed in (16) can be expressed as(19)JMixθ=Es^t∼fsaww0X0s^t;θ+wL/2XL/2s^t;θ+wLXLs^t;θ,whereXis^t;θ=Es^t+1,…,s^t+i∑l=ti−1+tγl−tR^l+γiQ^w,t+i.Due to different error sources and different growth trends in each period, a simple rule-based method is used to determine the weight:(20)wi=Softmax1Rule−Errori,i=0,L/2,L,where Rule-Errori is a rule-based error within the prediction range i, it can be expressed as(21)Rule−Errori=λi,λ≤1,2−λL−i,λ>1,where λ is designed as a hyperparameter in model calculation, it increases continuously during the learning process from 1−η to 1 + η in iteration T, and then maintains at 1 + η, η ∈ (0,1).In order to learn the action value function, in this paper, the mean square error between the approximate value function and the target value is minimized and is represented byJQw, and its gradient is(22)∇wJQw=−Es,aRtargets,a−Qws,a∇wQws,a,where Rtargets,a is the n-step TD operation of running the current policy in the environment from each state in the batch. Its solution is as follows:(23)Rtargetst,at=∑tt+n−1γl−tRl+γnQw′st+n,πθ′st+n.After clarifying the gradient of the mixed policy and the way to determine its weight, the mixed policy gradient AC framework can be designed as follows:(i)
Initialization. Parameterw and θ represent critic network Qw and actor network πθ, respectively, target parameter is w′⟵w, θ′⟵θ, batch size is n, which means that the initial sampling area B has n samples, and the target smoothing coefficient is τ1, learning rates βw, β0 are hyperparameters.(ii)
For each iterationk,(1)
A batch of conversions is collected usingπθ and noise ε and stored in sampling area B.(2)
Use equation (22) to solve ∇wJQw and update w⟵w−βw∇wJQw.(iii)
If iterationk satisfies the time interval m,(1)
Obtainw0, wL/2 and wL according to equation (20).(2)
Use equation (19) to obtain JMixθ.(3)
Update the policyθ⟵θ+βθ∇θJMixθ.(4)
Update the objective networkw′⟵τ1w+1−τ1w′, θ’ ← τ1θ + (1−τ1)θ’.(iv)
End iterationk.


## 4.1. Data-Driven and Model-Driven Policy Gradient

Before integrating the mixed gradient algorithm into the AC algorithm, it is necessary to clarify what is data-driven and model-driven policy gradient algorithm. These two driven methods are also used to solve the gradient problem in the maximum problem of the objective function. During the discussion of the driven policy gradients, the objective function will be simply represented by the value functionqπs,a in the form of a future discount sum:(12)qπs,a=Esl,l>t∑l=t∞γl−tRsl,al|st=s,at=a,where l indicates the number of action state pairs. Then the gradient of the objective function can be expressed as(13)∇θJθ=Est∇atqπst,at∇θπθ.As for model-driven and data-driven policy gradient, the main difference lies in what form to approximate the functionqπ. Network Qw is used to approximate qπ, and define Q^w,t=Qws^t,a^t, where s^t=st,a^t=at. For model-driven gradient, as RL is modeled by Markov process, this property determines that the value function can be written as the n-th recursive form of the Behrman equation. Therefore, Q^w,t+n≜Qws^t+nπθs^t+n is used to represent the Behrman approximation function of each step, and the model-driven policy obtained by n-step Behrman can be expressed as(14)∇θJnMθ=Es^t∇θEs^t+1,...,s^t+n∑l=tn−1+tγl−tR^l+γnQ^w,t+n,where s^t+i is the state transition function under the parameterized policy s^t+i∼f⋅|s^t+i−1,πθs^t+i−1,i≥1, where f is the model specified to be used in the model-driven policy gradient, R^t+i≜Rs^t+i,πθs^t+i is expressed as the sum of returns under the parameterized policy, where i≥0.Data-driven gradient, as the name suggests, is a type of policy through a large number of iterations and empirical data. For data-driven, the value function can be approximated by theQw generated by a large amount of data [44]. The policy gradient driven by numerical value can be written as(15)∇θJDθ=Est∇atQw∇θπθ.


## 4.2. Data-Driven and Model-Driven Mixed Policy Gradient

No matter data-driven or model-driven, there will be some defects in the calculation. Data-driven RL highlights its excellent asymptotic performance at the cost of slow convergence speed, while model-driven RL uses a differential transformation model with the disadvantages of large numerical fluctuation and low error sensitivity to improve the convergence speed. If the two are mixed directly, the accuracy of the policy gradient will change with the error of the two after mixing, and the process will become uncontrollable. Therefore, in the process of constructing the mixed policy gradient, it is need to combine the advantages of the two ideas, so the unity between data-driven and model-driven policy gradient should be found firstly, that is, a unified gradient concept should be proposed to facilitate the subsequent model construction. Moreover, during this process, balancing the error of the two ideas is the key to realize the high-precision mixing of policy gradient. In addition to ensuring the accuracy of the model, the convergence speed of the model should also be considered. A faster convergence speed can better realize the transplantation of the algorithm to vehicle path planning practices and produce more practical benefits. Based on these considerations, in this sectionn-step Behrman is used to recursively approximate the value function to obtain a unified policy gradient form as follows:(16)∇θJnUθ=Es^t∇a^tEs^t+1,...,s^t+n∑l=tn−1+tγl−tR^l+γnQ^w,t+n∇θπθ.For this kind of mixed gradient unification, in the discount rater ⟶ 1, the transition state probability in the MDP process is a steady-state distribution, thus, the model-driven policy gradient acts to reduce the dependence on Q value. From the theorem [39]:(17)∇θJnMθ=∑k=0n−1γk∇θJn−kU,It can be proved that the model-driven policy gradient is the weighted average of the unified policy gradient, and each weight will increase exponentially with the recursive upper limit n - k in the unified policy gradient. However, in the overall unified policy gradient adjustment, the disadvantage in asymptotic performance in model-driven policy gradient will also lead to the irrationality of the model. Therefore, in this paper, a better way is supposed to be found so as to update the weight and establish a unified mixed policy gradient.Combining the above two (16) and (17), and the characteristics of data-driven and model-driven policy gradient, it can be known that in the early stage of model training, the real value is usually quite different from the Q value of data-driven numerical estimation. Therefore, a larger n in model-driven method can reduce the problems caused by the estimation error in data-driven method. With the continuous advancement of iteration, the n-step recursion of large values will destroy its asymptotic performance. Therefore, the value of n can be reduced for subsequent iterations.


## 4.3. Mixed Policy Gradient Actor-Critic Model

In this paper, the loss∇θJ0Uθ, ∇θJL/2Uθ, and ∇θJLUθ obtained from the reverse of the unified policy gradient are used to form the mixed policy gradient, where [·] is the integral function. According to the content of the unified policy gradient, the former is purely data-driven and the latter is model-driven by Behrman approximation. Since the approaching to infinity in n steps cannot be realized, L is expressed as a larger number in ∇θJLUθ. Then the form of mixed policy gradient can be expressed as:(18)∇θJMixθ=w0∇θJ0Uθ+wL/2∇θJL/2Uθ+wL∇θJLUθ,where w0, wL/2, and wL correspond to the weights to be determined for data-driven, intermediate metamorphosis, and model-driven policy gradients, respectively. Moreover, the following assumptions are made for the mixed policy gradient:(i)
At the beginning of training, the error of the mixed model mainly comes from the data-driven part; at the later stage of training, when theQ value estimation error is small, the error mainly comes from the model-driven part.(ii)
The unified policy gradient error changes exponentially, decreases in the early stage and increases in the later stage.(iii)
In order to simplify the formula, mixed policy gradient expressed in (16) can be expressed as(19)JMixθ=Es^t∼fsaww0X0s^t;θ+wL/2XL/2s^t;θ+wLXLs^t;θ,whereXis^t;θ=Es^t+1,…,s^t+i∑l=ti−1+tγl−tR^l+γiQ^w,t+i.Due to different error sources and different growth trends in each period, a simple rule-based method is used to determine the weight:(20)wi=Softmax1Rule−Errori,i=0,L/2,L,where Rule-Errori is a rule-based error within the prediction range i, it can be expressed as(21)Rule−Errori=λi,λ≤1,2−λL−i,λ>1,where λ is designed as a hyperparameter in model calculation, it increases continuously during the learning process from 1−η to 1 + η in iteration T, and then maintains at 1 + η, η ∈ (0,1).In order to learn the action value function, in this paper, the mean square error between the approximate value function and the target value is minimized and is represented byJQw, and its gradient is(22)∇wJQw=−Es,aRtargets,a−Qws,a∇wQws,a,where Rtargets,a is the n-step TD operation of running the current policy in the environment from each state in the batch. Its solution is as follows:(23)Rtargetst,at=∑tt+n−1γl−tRl+γnQw′st+n,πθ′st+n.After clarifying the gradient of the mixed policy and the way to determine its weight, the mixed policy gradient AC framework can be designed as follows:(i)
Initialization. Parameterw and θ represent critic network Qw and actor network πθ, respectively, target parameter is w′⟵w, θ′⟵θ, batch size is n, which means that the initial sampling area B has n samples, and the target smoothing coefficient is τ1, learning rates βw, β0 are hyperparameters.(ii)
For each iterationk,(1)
A batch of conversions is collected usingπθ and noise ε and stored in sampling area B.(2)
Use equation (22) to solve ∇wJQw and update w⟵w−βw∇wJQw.(iii)
If iterationk satisfies the time interval m,(1)
Obtainw0, wL/2 and wL according to equation (20).(2)
Use equation (19) to obtain JMixθ.(3)
Update the policyθ⟵θ+βθ∇θJMixθ.(4)
Update the objective networkw′⟵τ1w+1−τ1w′, θ’ ← τ1θ + (1−τ1)θ’.(iv)
End iterationk.


## 5. Model Improvement by Filter Optimization and Random Escape Term

In the process of obtaining the policy gradient, the step size is a hyperparameter, and its selection is always a difficult problem. As two main methods for solving nonlinear optimization problems, the line search method and trust region method can also be applied to gradient solution process. Most models just choose the trust region method to determine the appropriate hyperparameters, but the process is cumbersome and increases the calculation time of the model to a certain extent, which will lead the AC model to fall back into the problem of long calculation time. At the same time, although the AC model based on a mixed policy gradient has good convergence speed and asymptotic performance, it is rule-based in determining the weight which relies on practical experience, and will lead to the limitation of the solution space of the model to a certain extent, and it is easier to fall into local optimum in the rapid convergence. In this section, the method of filter optimization and additional escape term is introduced into the mixed policy gradient AC model, which makes it easier to determine the hyperparameters, and greatly improves the global convergence of the model.


### 5.1. Filter Optimization

Filter optimization is an algorithm in the field of nonlinear programming. As the filter algorithm always has good numerical results, it has also been extended to other fields, including the line search method mentioned above. The line search method is used to solve the step size in the process of policy gradient descent. The appropriate step size is selected by adding filter point conditions and a filtering mechanism, and the updated step size can be returned to the iteration process, thus, the convergence of the algorithm and the overall iteration rate of the model can be improved.No matter what algorithm is used, its purpose is to find the solution with a gradient equals to zero of the corresponding objective function and unconstrained optimization problems and the constrained optimization problems can have the following equivalent transformation:(24)minθ∈RnΦθ,equivalent to(25)minθ∈RnΦθs.t.∇θΦθ=0,where Φ(θ) is the optimization objective function. Through this equivalent transformation, the solution background of filter optimization can be better expanded, and filter optimization can be better applied to the step size of the mixed policy gradient.After the equivalence transformation is completed, according to the main idea of the filter optimization, the right-hand side in (24) needs to be regarded as a double objective optimization problem: minimizing the objective function Φ(θ) and the gradient norm hθ=∇θΦθ. In order to verify the step size in the iterative process of the algorithm, αk.ν is the v-th experimental step size in the k-th iteration, which can make the values of the objective function and gradient norm have a decreasing trend. In order to ensure the overall convergence of the algorithm and that the new step size can obtain a better convergence point at the iteration point than the last iteration, two filter point conditions are added:(26)Φθkαk.ν≤Φθk−γJαk.νΦ2θk,(27)Φθkαk.ν≤1−γhhθk,where γh∈0,1, γJ>0 is a constant. The selected new iteration point must conform to one of the above two equations.The filtering mechanism employed in this paper determines whetherθkαk.ν can become a new iteration point. The set used to store the objective function and gradient norm pair becomes the filter set, which is represented by Fk, Fk⊆0,∞×ℝ. When the test point satisfies (25) or (27), and the objective function and gradient norm correspond to the point are not in the filter set, the test point is accepted. The initial filter set can be defined as(28)F0=h,Φ∈ℝ2:h≥hmax,where hmax≥hθ0, hmax, and hθ0 are given values in the algorithm. When the gradient norm of the test point is greater than hmax, the test point cannot be accepted. In order to prevent the gradient norm and objective function obtained by the new iteration point from being the same as the previous point, resulting in the dead cycle of the algorithm, whenever a new iteration point is selected, the filter set will be expanded in the form of the original filter set plus new points.(29)Fk+1=F∪h,Φ∈ℝ2:h≥1−γhhθkandΦ≥Φθk−γJαk.νΦ2θk.Then, the filter line search method can be acquired as(i)
Initialization. Obtain the function Φ(θ) to be optimized, initialize the filter set according to (28), specify the error accuracy ε, fix the constants γh, γJ, and parameters 0<κ1≤κ2<1, randomly set the initial independent variable value θ0, and the number of iterations k = 0.(ii)
Test Convergence. Calculate the gradient norm, and stop the iteration ifhθk<ε.(iii)
Determine the negative gradient of Φ(θk) at point θk as the search direction.(iv)
Do the mixed policy gradient descent:(1)
Initialize policy gradient, letαk,0=1 and v = 0.(2)
Calculate the test point:θkαk.ν=θk+αk.νdk, dk is the negative gradient direction of the objective function.(3)
Check whether the test point belong to the points included in the filter set. Ifθkαk.ν∈Fk, jump to (6); if not, go to (4).(4)
If equation (25) is satisfied, jump to the next step “Accept the test point”; If not, go to (5).(5)
If equation (26) is satisfied, jump to the next step “Accept the test point”; If not, go to (6).(6)
Select a new step sizeαk,ν+1∈κ1αk.ν,κ2αk.ν, let v = v + 1, and go back to (2).(v)
Do the mixed policy gradient descent: Accept the test point. Letαk=αk.ν, θk=θαk=θk+αkdk.(vi)
Expand the filter set according to (28).(vii)
Letk = k + 1.


### 5.2. Mixed Policy Gradient Actor-Critic Model with Random Escape Term and Filter Optimization

Escape term has its positive effect in gradient descent calculation process. In terms of the constructor, the idea of adding escape term is just the same as that of adding a regularization term. Both of them are supposed to help avoid the early convergence under the influence of abnormal data to a certain extent.For the gradient descent non-convex optimization problem, when the point descents according to a certain step, it will reach a relatively lowest point, that is, the local optimum of the objective function. When it falls into the local optimum, it will be unable to continue the iteration because the descent direction tends to zero. Escape term provides a certain driving force for the gradient descent process to get rid of the situation that the next iteration cannot be carried out when falling into local optimum. For the model in this paper, the escape term can be added in (22). In order to simplify the calculation of the model, the escape term is set as a random variable ξ, which obeys the standard Gaussian distribution.The gradient with escape term added can be written as(30)∇wJ^Qw=−Es,aRtargets,a−Qws,a∇wQws,a+ξ.After the escape term has been determined, the overall model process can be described as follows:(i)
Initialization. Parameterw and θ represent critic network Qw and actor network πθ, respectively, target parameter is w′⟵w, θ′⟵θ, batch size is N, which means that the initial sampling area B has N samples, and the target smoothing coefficient is τ2, learning rates is βw and βθ, initialize the filter set according to (28), specify the error accuracy ε, fix the constants γh and γJ.(ii)
For each iterationk,(1)
A batch of conversions is collected usingπθ and noise ε and stored in sampling area B.(2)
Updateβw with equation (30) and filter, and then further update the value network: w⟵w−βw∇wJ^Qw.(iii)
If iterationk satisfies the time interval m,(1)
Use equation (20) to obtain w0, wL/2 and wL.(2)
Use equation (19) to obtain JMixθ.(3)
Update the step sizeβθ with filter, and then further update the policy network: θ⟵θ+βθ∇θJMixθ+ξ.(4)
Update the objective networkw′⟵τ2w+1−τ2w′, θ’ ← τ2θ + (1−τ2)θ’.(iv)
End iterationk.


## 5.1. Filter Optimization

Filter optimization is an algorithm in the field of nonlinear programming. As the filter algorithm always has good numerical results, it has also been extended to other fields, including the line search method mentioned above. The line search method is used to solve the step size in the process of policy gradient descent. The appropriate step size is selected by adding filter point conditions and a filtering mechanism, and the updated step size can be returned to the iteration process, thus, the convergence of the algorithm and the overall iteration rate of the model can be improved.No matter what algorithm is used, its purpose is to find the solution with a gradient equals to zero of the corresponding objective function and unconstrained optimization problems and the constrained optimization problems can have the following equivalent transformation:(24)minθ∈RnΦθ,equivalent to(25)minθ∈RnΦθs.t.∇θΦθ=0,where Φ(θ) is the optimization objective function. Through this equivalent transformation, the solution background of filter optimization can be better expanded, and filter optimization can be better applied to the step size of the mixed policy gradient.After the equivalence transformation is completed, according to the main idea of the filter optimization, the right-hand side in (24) needs to be regarded as a double objective optimization problem: minimizing the objective function Φ(θ) and the gradient norm hθ=∇θΦθ. In order to verify the step size in the iterative process of the algorithm, αk.ν is the v-th experimental step size in the k-th iteration, which can make the values of the objective function and gradient norm have a decreasing trend. In order to ensure the overall convergence of the algorithm and that the new step size can obtain a better convergence point at the iteration point than the last iteration, two filter point conditions are added:(26)Φθkαk.ν≤Φθk−γJαk.νΦ2θk,(27)Φθkαk.ν≤1−γhhθk,where γh∈0,1, γJ>0 is a constant. The selected new iteration point must conform to one of the above two equations.The filtering mechanism employed in this paper determines whetherθkαk.ν can become a new iteration point. The set used to store the objective function and gradient norm pair becomes the filter set, which is represented by Fk, Fk⊆0,∞×ℝ. When the test point satisfies (25) or (27), and the objective function and gradient norm correspond to the point are not in the filter set, the test point is accepted. The initial filter set can be defined as(28)F0=h,Φ∈ℝ2:h≥hmax,where hmax≥hθ0, hmax, and hθ0 are given values in the algorithm. When the gradient norm of the test point is greater than hmax, the test point cannot be accepted. In order to prevent the gradient norm and objective function obtained by the new iteration point from being the same as the previous point, resulting in the dead cycle of the algorithm, whenever a new iteration point is selected, the filter set will be expanded in the form of the original filter set plus new points.(29)Fk+1=F∪h,Φ∈ℝ2:h≥1−γhhθkandΦ≥Φθk−γJαk.νΦ2θk.Then, the filter line search method can be acquired as(i)
Initialization. Obtain the function Φ(θ) to be optimized, initialize the filter set according to (28), specify the error accuracy ε, fix the constants γh, γJ, and parameters 0<κ1≤κ2<1, randomly set the initial independent variable value θ0, and the number of iterations k = 0.(ii)
Test Convergence. Calculate the gradient norm, and stop the iteration ifhθk<ε.(iii)
Determine the negative gradient of Φ(θk) at point θk as the search direction.(iv)
Do the mixed policy gradient descent:(1)
Initialize policy gradient, letαk,0=1 and v = 0.(2)
Calculate the test point:θkαk.ν=θk+αk.νdk, dk is the negative gradient direction of the objective function.(3)
Check whether the test point belong to the points included in the filter set. Ifθkαk.ν∈Fk, jump to (6); if not, go to (4).(4)
If equation (25) is satisfied, jump to the next step “Accept the test point”; If not, go to (5).(5)
If equation (26) is satisfied, jump to the next step “Accept the test point”; If not, go to (6).(6)
Select a new step sizeαk,ν+1∈κ1αk.ν,κ2αk.ν, let v = v + 1, and go back to (2).(v)
Do the mixed policy gradient descent: Accept the test point. Letαk=αk.ν, θk=θαk=θk+αkdk.(vi)
Expand the filter set according to (28).(vii)
Letk = k + 1.


## 5.2. Mixed Policy Gradient Actor-Critic Model with Random Escape Term and Filter Optimization

Escape term has its positive effect in gradient descent calculation process. In terms of the constructor, the idea of adding escape term is just the same as that of adding a regularization term. Both of them are supposed to help avoid the early convergence under the influence of abnormal data to a certain extent.For the gradient descent non-convex optimization problem, when the point descents according to a certain step, it will reach a relatively lowest point, that is, the local optimum of the objective function. When it falls into the local optimum, it will be unable to continue the iteration because the descent direction tends to zero. Escape term provides a certain driving force for the gradient descent process to get rid of the situation that the next iteration cannot be carried out when falling into local optimum. For the model in this paper, the escape term can be added in (22). In order to simplify the calculation of the model, the escape term is set as a random variable ξ, which obeys the standard Gaussian distribution.The gradient with escape term added can be written as(30)∇wJ^Qw=−Es,aRtargets,a−Qws,a∇wQws,a+ξ.After the escape term has been determined, the overall model process can be described as follows:(i)
Initialization. Parameterw and θ represent critic network Qw and actor network πθ, respectively, target parameter is w′⟵w, θ′⟵θ, batch size is N, which means that the initial sampling area B has N samples, and the target smoothing coefficient is τ2, learning rates is βw and βθ, initialize the filter set according to (28), specify the error accuracy ε, fix the constants γh and γJ.(ii)
For each iterationk,(1)
A batch of conversions is collected usingπθ and noise ε and stored in sampling area B.(2)
Updateβw with equation (30) and filter, and then further update the value network: w⟵w−βw∇wJ^Qw.(iii)
If iterationk satisfies the time interval m,(1)
Use equation (20) to obtain w0, wL/2 and wL.(2)
Use equation (19) to obtain JMixθ.(3)
Update the step sizeβθ with filter, and then further update the policy network: θ⟵θ+βθ∇θJMixθ+ξ.(4)
Update the objective networkw′⟵τ2w+1−τ2w′, θ’ ← τ2θ + (1−τ2)θ’.(iv)
End iterationk.


## 6. Numerical Experiment

In this paper, data sets described in the literature [45] of traveling salesman problem (TSP) and capacitated vehicle routing problem (CVRP) have been used. For multiple routing with fixed fleet problems (MRPFF), data set in literature [43] has also been implemented. For the convenience of expression, the mixed policy gradient AC model with random escape term and filter optimization proposed in this paper is recorded as MPGACEF.For CVRP, MPGACEF has been compared with a set of representative methods proposed in the literature [45–48]. By using their public code to implement and report the results, 1000 random CVRP20, CVRP50, and CVRP100 instances have been given, and the average tour length, gap percentage, and average solution time of each method are shown in Table 1. In Table 1, the average tour lengths have been reported in kilometers (km) the groups of average solution time have been reported in minutes (min) or seconds (s), the groups of data marked with “∗” are directly referenced from the literature [46]. It is easy to see that the model proposed in this paper is superior to all existing methods.Table 1
The average tour length, gap percentage and average solution time of CVRP.
MethodCVRP20CVRP50CVRP100Tour lengthGap (%)Average solution timeTour lengthGap (%)Average solution timeTour lengthGap (%)Average solution timeStochastic RL framework [45]7.1015.156.42(s)11.9315.0219(s)17.8914.1842(s)Random CW∗ [46]6.8110.91—∗12.2517.90—18.9620.99—Random sweep∗ [46]7.0114.16—12.9624.73—20.3329.73—Or-tools∗ [46]6.434.73—11.4310.00—17.169.50—Attention-based graph layer policy [46]6.678.636.32(s)11.005.8520(s)16.998.4148(s)L2I [47]6.13—12(min)10.30—17(min)15.55—24(min)Reformed parallel GA [48]6.120.4—10.461.18—15.902.69—MPGACEF6.110.655.41(s)10.422.3614(s)16.123.1436(s)For MRPFF, MPGACEF proposed in this paper has also been compared with representative methods proposed in the literature [45, 46], and the average tour length, gap percentage, and average solution time of each method are shown in Table 2. In Table 2, results of applying 20, 50, and 100 customer nodes in the MRPFF experiment have been reported, it can also be seen that all the indicators of the model proposed in this paper are better than all existing methods.Table 2
The average tour length, gap percentage, and average solution time of MRPFF.
MethodMRPFF = 20MRPFF = 50MRPFF = 100Tour lengthGap (%)Average solution timeTour lengthGap (%)Average solution timeTour lengthGap (%)Average solution timeStochastic RL framework [45]6.8027.216(s)10.8718.9418.98(s)15.9721.3833 (s)Attention-based graph layer policy [46]5.9812.186(s)10.3012.9318(s)14.6711.4832 (s)MPGACEF5.424.834(s)9.077.3315(s)14.117.3830 (s)For TSP, MPGACEF proposed in this paper has also been compared with a set of representative methods proposed in the literature [45, 46, 49–52]. Again, the average tour length, gap percentage, and average solution time of each method are shown in Table 3. It can also be seen that the model proposed in this paper is superior to all existing methods.Table 3
The average tour length, gap percentage, and average solution time of TSP.
MethodTSP20TSP50TSP100Tour lengthGap (%)Average solution timeTour lengthGap (%)Average solution timeTour lengthGap (%)Average solution timeStochastic RL framework [45]4.004.444(s)7.0221.9812(s)9.4621.7525 (s)Attention-based graph layer policy [46]3.850.521(s)5.811.742(s)8.154.987 (s)Concorde [49]3.830.004(m)5.700.0010(m)7.770.0055 (m)GCN [50]3.870.776(s)5.862.9755(s)8.418.326 (m)PN [51]3.891.3013(s)6.6116.1572(s)10.8840.2073 (m)Neural combinatorial optimization [52]3.884.738(s)65.0423(s)9.6524.7121 (m)MPGACEF3.820.001(s)5.731.062(s)7.771.166 (s)Apart from MPGACEF proposed in this paper, the attention-based graph layer policy proposed by Kool et al. method performs best [46]. For the convenience of expression, the attention-based graph layer policy is recorded as AGLP. In the comparison, the performance evaluation index ρs(τ) of classical algorithm has been introduced [37], and the number of iterations is recorded as NIT. As shown in Figures 2–9, the performance of both MPGACEF and AGLP under the conditions of CVRP50 and TSP50, CVRP100 and TSP100, respectively. The higher the algorithm performance curve in those figures, the better performance of the corresponding model is, and it can be clearly seen that MPGACEF outperforms AGLP.Figure 2
Performance of CPU Time for MPGACEF and AGLP under the condition of CVRP50.Figure 3
Performance of NIT Time for MPGACEF and AGLP under the condition of CVRP50.Figure 4
Performance of CPU Time for MPGACEF and AGLP under the condition of TSP50.Figure 5
Performance of NIT Time for MPGACEF and AGLP under the condition of TSP50.Figure 6
Performance of CPU Time for MPGACEF and AGLP under the condition of CVRP100.Figure 7
Performance of NIT Time for MPGACEF and AGLP under the condition of CVRP50.Figure 8
Performance of CPU Time for MPGACEF and AGLP under the condition of TSP100.Figure 9
Performance of NIT Time for MPGACEF and AGLP under the condition of TSP100.


## 7. Conclusions

In this paper, a mixed policy gradient AC model with random escape term and filter operation, and via numerical experiment, its calculation efficiency and effectiveness have been proved to be superior to all existing methods. The main contributions of this paper can be concluded as follows:(1)
In the existing achievements of vehicle path planning, there is no precedent to solve the problem by employing the mixed policy gradient method, and there is no mixed policy gradient model considering both data-driven and model-driven architectures, and it has been creatively introduced in this paper in the vehicle path planning field.(2)
The vehicle path planning problem carries out sequential optimization through RL, which is generally transformed into a non-convex optimization problem. There is no precedent of parameter optimization using the line search method with escape term in the existing results. In this paper, the escape term technique is introduced into this field for the first time to solve the non-convex optimization problem.(3)
In the optimization process of the vehicle path planning problem realized by RL, a large number of hyperparameters (iteration step) need to be selected, and the selection of hyperparameters directly affects the performance of the model. The optimization process often depends on the accumulation of professional knowledge and long-term experience, and the time cost of manually selecting hyperparameters is too high. In the existing achievements of vehicle path planning, there is no precedent of using filter optimization to optimize the hyperparameters. In this paper, a filter optimization algorithm is introduced into this field for the first time.



---

*Source: 3679145-2022-08-29.xml*