# Adaptive Navigating Control Based on the Parallel Action-Network ADHDP Method for Unmanned Surface Vessel

Authors: Zhijian Huang; Xinze Liu; Jiayi Wen; Guichen Zhang; Yihua Liu
Journal: Advances in Materials Science and Engineering
 (2019)

---

## Abstract

The feedback PID method was mainly used for the navigating control of an unmanned surface vessel (USV). However, when the intelligent control era is coming now, the USV can be navigated more effectively. According to the USV character in its navigating control, this paper presents a parallel action-network ADHDP method. This method connects an adaptive controller parallel to the action network of the ADHDP. The adaptive controller adopts a RBF neural network approximation based on the Lyapunov stability analysis to ensure the system stability. The simulation results show that the parallel action-network ADHDP method has an adaptive control character and can navigate the USV more accurately and rapidly. In addition, this method can also eliminate the overshoot of the ADHDP controller when navigating the USV in various situations. Therefore, the adaptive stability design can greatly improve the navigating control and effectively overcome the ADHDP algorithm limitation. Thus, this adaptive control can be one of the intelligent ADHDP control methods. Furthermore, this method will be a foundation for the development of an intelligent USV controller.


---

## Body

## 1. Introduction

The development of science and technology is speeding up the informatization to intellectualization. Many systems are facing the new problems of intelligent control. The intelligent control can also be widely used in the unmanned surface vessel (USV) for its learning and adaptive ability. At present, the control research for the USV mainly focuses on the path-following, path planning, formation control, and experiment etc.Among the above researches, the path-following has been researched at most. In 2017, Shin et al. proposed a path-following control for the USV based on an identified dynamic model [1]. In 2018, Qin et al. solved the path-tracking control of the USV with input saturation and full-state constraints [2]; Qu et al. presented an exponential path-tracking control of the USV with external disturbance, system uncertainties [3], etc. These methods generally require the precise model of the USV or external environment, which are difficult to establish.The path planning is also one of the major research studies. In 2017, Kim et al. fulfilled the path optimization of the USV under environmental loads using the genetic algorithm [4]. In 2018, Lyu and Yin used a path-guided potential-field method to achieve the path planning in restricted waters [5]. In 2019, Wang et al. adopted an improved grey wolf optimizer to optimize the USV trajectory [6]. This is a new intelligent method, but it still has some shortcomings.In addition, in 2017, Klinger et al. evaluated the intelligent controller performance through a USV experiment [7]. In 2018, Conte et al. developed a ROS of multi-agent structure for the USV, and tested its path planning [8]. Dai et al. combined many methods to achieve the UAV formation control [9] and so on. For more detailed development and trend analysis, please refer to these review papers [10–12].However, the nonlinear adaptive optimal control of the USV has not been fully investigated. This is an important task in the intelligent USV control, and further research is needed. For the adaptive optimal control of the USV, it usually needs solving the Hamilton–Jacobi–Behrman (HJB) equation. This equation is difficult to solve in many cases except for a linear quadratic system. The classical dynamic programming method still suffers to the curse of dimensionality. Fortunately, in recent years, the approximate dynamic programming (ADP) method has arisen with neural network approximation [13]. Liu et al. gives the most recent developments of the ADP theory and its advances in industrial control [14]. Yang et al. presented a guaranteed cost neural tracking control for a class of uncertain nonlinear systems using the ADP [15]. These researches make the ADP suitable for the optimal control of minimizing a performance index. Thus, the ADP can also be used to learn and adjust the navigating according to the USV conditions.However, in practice, simply relying on the ADP cannot achieve enough accurate adjustment, especially under an external influence, while the adaptive control has the ability to adapt to different environment. If we combined the advantages of these two kinds of control, the USV navigating can be adjusted more quickly and accurately. Therefore, in this paper, these two methods are jointed for the navigating USV. Then, based on the analysis of the USV, this paper presents a parallel action-network ADP method. This method connects an adaptive controller parallel to the action network of the ADP. This parallel adaptive controller adopts a RBF neural network approximation based on the Lyapunov stability analysis. The simulation results show that the parallel action-network ADP has an adaptive control character and can navigate the USV more accurately and rapidly. In addition, this method can also eliminate the overshoot of the ADP controller when navigating the USV in various situations. Thus, this adaptive control can be one of the intelligent ADP control methods. This method will also be a foundation to the development of an intelligent USV controller.


## 2. The ADHDP Control Method

The action-dependent heuristic dynamic programming (ADHDP) is one of the main approaches in the ADP family. The critic and action networks of the ADHDP are usually established based on the BP neural network [16, 17]. As shown in Figure 1, the main function of the critic network uses the system state to approximate the cost function Jt. The correctness of the cost function will guide the action network to the optimal ut approximation. If the output of the action network is received as part of the input to the critic network, it forms a new critic network. The new critic network conceals a controlled object model and approximates the new cost function Qt. The output Qt is an estimation of the cost function in the next time. Thus, Qt=Jt+1, and the ADHDP can perform control without the mathematical plant model.Figure 1
A new critic network established in the ADHDP.


### 2.1. The Problem Formulation

The action-critic scheme of the ADHDP and its symbols are both shown in Figure2. Assuming that a nonlinear system is set up as(1)xt+1=Ftxt,ut,t,where xt∈Rn is the system state of time t, ut∈Rm is the system action of time t, and Ft is the state transition equation of the nonlinear system.Figure 2
The action-critic principle and structure of the ADHDP.In order to let the system run in an optimal state for the dynamic programming algorithm, the cost function must be defined as [18](2)Jxi,i=∑k=i∞αk−iγxk,uk,k,where α is the discount factor (0<α≤1), and γ is the utility function of each time step. The objective is to choose the control action, uk, k=i,i+1,…, so that the cost function Jt defined in equation (2) is minimized [19].


### 2.2. The Critic Network

As shown in Figure2, the input vector to the critic network is(3)s=x1t,x2t,…,xnt,u1t,u2t,…,umt.The inputqit to the hidden layer is calculated as(4)qit=∑j=1n+mwci,j1sj.s in the hidden layer of Figure 2 means the nonlinear Sigmodial function. Thus, the output of the hidden layer is(5)pit=1−exp−qit1+exp−qit.The / in the output layer means adopting a linear function, so the output layer is(6)Qt=∑i=1Nh1wci2pit.The ADHDP approximates the optimal solution by minimizing the new cost function; that is,Qt=Jxt+1,t+1. This is achieved by training the new critic network to minimize the following error measured over time:(7)Ect=∑t12ec2t=∑t12αQt−Qt−1−γt2,where Qt=Qxt,ut,t,wc, wc is the critic network weights. The output Qt of time t is an estimation of the dynamic programming at time t+1, which is the cost function in the next time.Then, the gradient descent algorithm is used to update the critic network weights and minimizeEct at each time step by(8)wct+1=wct+Δwct,Δwct=lct−∂Ect∂wct=lct−∂Ect∂ect⋅∂ect∂Qt⋅∂Qt∂wct=−lct⋅ect⋅α⋅∂Qt∂wct,where lct>0 is the learning rate of the critic network at time t, which decreases to a very small value with time t [20].


### 2.3. The Action Network

As shown in Figure2, the input to the action network is(9)x=x1t,x2t,…,xnt.The inputhit to the hidden layer of the action network is calculated as(10)hit=∑j=1nwai,j1xj.The hidden layer uses the nonlinear Sigmodial function; thus, the output of the hidden layer is(11)git=1−exp−hit1+exp−hit.The inputvkt to the output layer is calculated as(12)vkt=∑k=1Nh2wak,i2git.The output layer also uses the nonlinear Sigmodial function; thus, the output of the action network is(13)ukt=1−exp−vkt1+exp−vkt.When the critic network has been trained, the objective of training the action network is to minimize the following index:(14)Eat=12ea2t=12Q2t.Then, the gradient descent algorithm is also used to update the weight of the action network by minimizingEat with(15)wat+1=wat+Δwat,Δwat=lat−∂Eat∂wat=lat−∂Eat∂eat⋅∂eat∂Qt⋅∂Qt∂wat=−lat⋅eat⋅∂Qt∂wat,where lat>0 is the learning rate of the action network at time t, which also decreases to a very small value with time t [20].After getting the optimalQt function, the optimal control u∗t can be obtained by the following equation:(16)u∗t=argminutQ∗xt,ut,that is, to obtain the optimal control output of the action network, it is necessary to make the cost function Qt=0. This will also make the critic network output equal to zero as close as possible.As shown in Figure3, the dashed line means that the error should be eventually reduced to zero by calculating the error of each operation and the increment of the critic and action network weights.Figure 3
The schematic diagram of the ADHDP control method.


## 2.1. The Problem Formulation

The action-critic scheme of the ADHDP and its symbols are both shown in Figure2. Assuming that a nonlinear system is set up as(1)xt+1=Ftxt,ut,t,where xt∈Rn is the system state of time t, ut∈Rm is the system action of time t, and Ft is the state transition equation of the nonlinear system.Figure 2
The action-critic principle and structure of the ADHDP.In order to let the system run in an optimal state for the dynamic programming algorithm, the cost function must be defined as [18](2)Jxi,i=∑k=i∞αk−iγxk,uk,k,where α is the discount factor (0<α≤1), and γ is the utility function of each time step. The objective is to choose the control action, uk, k=i,i+1,…, so that the cost function Jt defined in equation (2) is minimized [19].


## 2.2. The Critic Network

As shown in Figure2, the input vector to the critic network is(3)s=x1t,x2t,…,xnt,u1t,u2t,…,umt.The inputqit to the hidden layer is calculated as(4)qit=∑j=1n+mwci,j1sj.s in the hidden layer of Figure 2 means the nonlinear Sigmodial function. Thus, the output of the hidden layer is(5)pit=1−exp−qit1+exp−qit.The / in the output layer means adopting a linear function, so the output layer is(6)Qt=∑i=1Nh1wci2pit.The ADHDP approximates the optimal solution by minimizing the new cost function; that is,Qt=Jxt+1,t+1. This is achieved by training the new critic network to minimize the following error measured over time:(7)Ect=∑t12ec2t=∑t12αQt−Qt−1−γt2,where Qt=Qxt,ut,t,wc, wc is the critic network weights. The output Qt of time t is an estimation of the dynamic programming at time t+1, which is the cost function in the next time.Then, the gradient descent algorithm is used to update the critic network weights and minimizeEct at each time step by(8)wct+1=wct+Δwct,Δwct=lct−∂Ect∂wct=lct−∂Ect∂ect⋅∂ect∂Qt⋅∂Qt∂wct=−lct⋅ect⋅α⋅∂Qt∂wct,where lct>0 is the learning rate of the critic network at time t, which decreases to a very small value with time t [20].


## 2.3. The Action Network

As shown in Figure2, the input to the action network is(9)x=x1t,x2t,…,xnt.The inputhit to the hidden layer of the action network is calculated as(10)hit=∑j=1nwai,j1xj.The hidden layer uses the nonlinear Sigmodial function; thus, the output of the hidden layer is(11)git=1−exp−hit1+exp−hit.The inputvkt to the output layer is calculated as(12)vkt=∑k=1Nh2wak,i2git.The output layer also uses the nonlinear Sigmodial function; thus, the output of the action network is(13)ukt=1−exp−vkt1+exp−vkt.When the critic network has been trained, the objective of training the action network is to minimize the following index:(14)Eat=12ea2t=12Q2t.Then, the gradient descent algorithm is also used to update the weight of the action network by minimizingEat with(15)wat+1=wat+Δwat,Δwat=lat−∂Eat∂wat=lat−∂Eat∂eat⋅∂eat∂Qt⋅∂Qt∂wat=−lat⋅eat⋅∂Qt∂wat,where lat>0 is the learning rate of the action network at time t, which also decreases to a very small value with time t [20].After getting the optimalQt function, the optimal control u∗t can be obtained by the following equation:(16)u∗t=argminutQ∗xt,ut,that is, to obtain the optimal control output of the action network, it is necessary to make the cost function Qt=0. This will also make the critic network output equal to zero as close as possible.As shown in Figure3, the dashed line means that the error should be eventually reduced to zero by calculating the error of each operation and the increment of the critic and action network weights.Figure 3
The schematic diagram of the ADHDP control method.


## 3. The Parallel Action-Network ADHDP: The Adaptive ADHDP Method

The addition of an adaptive control to the navigating of the USV is to ensure its system stability. This is because the gradient descent algorithm cannot guarantee the system stability [21] when updating the ADHDP parameters. In addition, it will lead to a local optimum in some cases. Therefore, the stability design based on the Lyapunov theory will greatly increase the stability of the navigating control and effectively overcome the limitation of the ADHDP algorithm.Based on the error minimization in equation (18), the error and its differential are used as the inputs to both the action network and adaptive controller, as shown in Figure 4. After getting the error and its differential inputs, the adaptive control algorithm is calculated separately.Figure 4
The scheme of the parallel action-network ADHDP method for the navigating control of the USV.When different control signalsut are obtained, the litter between the adaptive control and action network outputs is selected as input to the controlled object. This will also help the stability of the navigating USV system. In this way, the advantage of the AHDHP and adaptive control can be integrated to achieve the navigating stability of the USV.The adaptive control process is as follows: Supposing that the nonlinear system is(17)x¨=fx,x˙+gx,x˙u,(18)e=rin−y,where rin is the tracking navigation command, the error e is the difference between the command and the system output y, x is the system state, u is the control input, and fx,x˙ and gx,x˙ are the nonlinear state functions of the USV system.Supposing thatE=ee˙T, and the optimal control u∗ is designed as(19)u∗=1gx,x˙−fx,x˙+rin⋅⋅+KTE,where K is a coefficient. By substituting equations (18) and (19) in equation (17), and simplifying its form, the system error satisfies(20)e¨+k2e˙+k1e=0.To designK=k1k2T, the root of its polynomial function s2+k2s+k1=0 must be in the left half plane of the phase-plane. When t tends to infinity, there should be et⟶0 and e˙t⟶0.For the adaptive control, the RBF neural network is used for approximation. The input to the network isx=ee˙T, and the output of the network is(21)f^x=W^Thx,where W^ is the estimation of the network weight, hj=gx−cij2/bj2 is a Gauss basis function of No. j weight, and hx is a vector of the Gauss basis function, please see [22]. Suppose that the matrix P is symmetric positive-definite and satisfies the following Lyapunov equation:(22)ΛTP+PΛ=−I,where Λ=01−k1−k2 and I is a unit matrix. After obtaining the matrix P, the expected RBF network weights are(23)W^˙=−φETPBhx,where B=01T.After the weights of the RBF neural network are obtained, the output of this network can be obtained by equation (21). Finally, the optimal control signal can be obtained by equation (19).


## 4. The USV Model for Navigating Control

The ADHDP is based on the state system and does not require a controlled object model. Thus, the ADHDP is also known as a data-driven control method [23], which enables an online learning and control [24]. Thus, in this study, the USV model is only used as a simulation object, not for the controller design. However, the choice of a suitable controlled object model is also very important for numerical experiment.The motion model and state space of the USV used in this paper are summarized in [25]. This model was proposed by Abkowitz and other scholars with mathematical integrity and rigor. The standard state space of the USV is expressed as follows(24)x˙=Ax+Bδ,where A=a11a120a21a220010, B=b11b210, and δ is the rudder angle of the USV.In it,(25)a11=Izz′−Nr˙′Yv′−m′xC′−Yr˙′Nv′VS1,a12=Izz′−Nr˙′Yr˙′−m′−m′xC′−Yr˙′Nr˙′−m′xC′LVS1,a21=−m′xC′−Nv˙′Yv′+m′−Yv˙′Nv′V/LS1,a22=−m′xC′−Nv˙′Yr′−m′+m′−Yv˙′Nr′−m′xC′VS1,b11=Izz′−Nr˙′Yδ′−m′xC′−Yr˙′Nδ′V2S1,b21=−m′xC′−Nv˙′Yδ′+m′−Yv˙′Nδ′V2/LS1,S1=Izz′−Nr˙′m′−Yv˙′−m′xC′−Nv˙′m′xC′−Yr˙′L,m is the mass of the USV, m′ is its dimensionless mass during calculation, which is calculated with m′=1/2mρL3, ρ is the sea water density, L is the length between the perpendiculars; Izz′ is the inertial distance, which is calculated with Izz′=Izz1/2ρL2, and Izz=mL2/16; V is the USV speed; S1 is the reference area. For other related intermediate variables, please see [25], and they can be calculated with(26)Yv˙′=−1+0.16CbBT−5.1BL2πTL2,Yr˙′=−0.67BL−0.0033BT2πTL2,Nv˙′=−1.1BL−0.041BTπTL2,Nr˙′=−112+0.017CbBT−0.33BLπTL2,Yv′=−1+0.40CbBTπTL2,Yr′=−−12+2.2BL−0.080BTπTL2,Nv′=−12+2.4TLπTL2,Nr′=−14+0.039BT−0.56BLπTL2,Yδ′=3.0AδL2,Nδ′=−12Yδ′,where B is the width of the USV, Cb is the block coefficient, Aδ is the area of the rudder leaf, and T is the draught of the USV.Considering the marine environment effect to the movement of the USV, the interference of a white noiseω=ω1ω2ω3 is added. Thus,(27)x˙=Ax+Bδ+ω.


## 5. The Numerical Experiment

### 5.1. The Parameters of the Control System

The utility function of the adaptive ADHDP controller can be designed as(28)γt=4⋅et+4⋅e˙t2,whereet is the error in equation (18) at time t, and e˙t is its differential with respect to time t.Figure 5
A 90° navigating control response of the USV under a PID and the adaptive ADHDP controller.Figure 6
The PID and adaptive ADHDP controller response to a continue sharp navigating process of the USV.Figure 7
The ADHDP and adaptive ADHDP controller response to a continuous and sharp navigating process of the USV.The MATLAB/Simulink is used for the simulation. The S-function is used to establish the related model and algorithm because there is no ADHDP toolbox in MATLAB. The parameters of the simulating USV model are shown in Table1.Table 1
The related USV parameters for simulation.
L
B
V
T
C
b
m
A
δ
180 (m)
27.8 (m)
9.0 (m/s)
10.3 (m)
0.64
30,000 (m3)
38 (m2)Table 2
The comparison of a 90° navigating control response of the USV under a PID and the adaptive ADHDP controller.
Comparing items
PID
Adaptive ADHDP
Maximum response (°)
111.1079
90
Overshoot (%)
23.29
0
Delay time (s)
2.1400
7.5535
Rise time (s)
5.4600
23.9995
Peak time (s)
10.8650
23.9995
Adjustment time (s)
22.5850
16.3405


### 5.2. Simulation and Results

(1)
The navigating target is set as east (90°) and the initial navigation is north (0°). The simulation time is 50 seconds. The navigating responses of a PID and the adaptive ADHDP control are shown in Figure5. The detailed comparison information of result curves in Figure 5 is shown in Table 2.
From the data in Figure5 and Table 2, it can be seen that the combination of the ADHDP and adaptive control can shorten the adjusting time and eliminate overshoot. The data in figures and tables also prove that the combination of the ADHDP and adaptive control can adjust the navigating USV quickly and accurately, and it only needs one navigating change.
(2)
When navigating the USV in a narrow river, the adjusting frequency and time are significantly increased. For the simulation in this condition, the navigating target is set as east (90°) for the first 50 seconds, and the initial heading is north (0°). Then, the navigating target is set as north (0°) in the 50th seconds, and the navigating should turn around 90°. This stage runs for another 50 seconds. Then, the navigating target is set as east (90°) again in the 100th second, and the navigating should turn back 90°. This stage also runs for 50 seconds. The simulation results are shown in Figure6.
According to the simulation results, it can be seen that the PID controller has a long time to adjust the navigating USV, and its overshoot is very big. However, the navigating USV control with the adaptive ADHDP is significantly enhanced in overshoot and rapidity compared with the PID controller.
(3)
Figure7 shows the control effect and response under the above continuous and sharp navigating with the individual ADHDP, and compares to that of the adaptive ADHDP. It can be seen from Figure 7 that the ADHDP controller has a larger overshoot in a continuous and sharp navigating change. This is because the ADHDP is based on the data-driven and previous learning effects, which is difficult to make a right response in a short period of time. After combining the adaptive control, the adaptive ADHDP can avoid an overshoot in mutation state and can ensure a safe driving for the USV.


## 5.1. The Parameters of the Control System

The utility function of the adaptive ADHDP controller can be designed as(28)γt=4⋅et+4⋅e˙t2,whereet is the error in equation (18) at time t, and e˙t is its differential with respect to time t.Figure 5
A 90° navigating control response of the USV under a PID and the adaptive ADHDP controller.Figure 6
The PID and adaptive ADHDP controller response to a continue sharp navigating process of the USV.Figure 7
The ADHDP and adaptive ADHDP controller response to a continuous and sharp navigating process of the USV.The MATLAB/Simulink is used for the simulation. The S-function is used to establish the related model and algorithm because there is no ADHDP toolbox in MATLAB. The parameters of the simulating USV model are shown in Table1.Table 1
The related USV parameters for simulation.
L
B
V
T
C
b
m
A
δ
180 (m)
27.8 (m)
9.0 (m/s)
10.3 (m)
0.64
30,000 (m3)
38 (m2)Table 2
The comparison of a 90° navigating control response of the USV under a PID and the adaptive ADHDP controller.
Comparing items
PID
Adaptive ADHDP
Maximum response (°)
111.1079
90
Overshoot (%)
23.29
0
Delay time (s)
2.1400
7.5535
Rise time (s)
5.4600
23.9995
Peak time (s)
10.8650
23.9995
Adjustment time (s)
22.5850
16.3405


## 5.2. Simulation and Results

(1)
The navigating target is set as east (90°) and the initial navigation is north (0°). The simulation time is 50 seconds. The navigating responses of a PID and the adaptive ADHDP control are shown in Figure5. The detailed comparison information of result curves in Figure 5 is shown in Table 2.
From the data in Figure5 and Table 2, it can be seen that the combination of the ADHDP and adaptive control can shorten the adjusting time and eliminate overshoot. The data in figures and tables also prove that the combination of the ADHDP and adaptive control can adjust the navigating USV quickly and accurately, and it only needs one navigating change.
(2)
When navigating the USV in a narrow river, the adjusting frequency and time are significantly increased. For the simulation in this condition, the navigating target is set as east (90°) for the first 50 seconds, and the initial heading is north (0°). Then, the navigating target is set as north (0°) in the 50th seconds, and the navigating should turn around 90°. This stage runs for another 50 seconds. Then, the navigating target is set as east (90°) again in the 100th second, and the navigating should turn back 90°. This stage also runs for 50 seconds. The simulation results are shown in Figure6.
According to the simulation results, it can be seen that the PID controller has a long time to adjust the navigating USV, and its overshoot is very big. However, the navigating USV control with the adaptive ADHDP is significantly enhanced in overshoot and rapidity compared with the PID controller.
(3)
Figure7 shows the control effect and response under the above continuous and sharp navigating with the individual ADHDP, and compares to that of the adaptive ADHDP. It can be seen from Figure 7 that the ADHDP controller has a larger overshoot in a continuous and sharp navigating change. This is because the ADHDP is based on the data-driven and previous learning effects, which is difficult to make a right response in a short period of time. After combining the adaptive control, the adaptive ADHDP can avoid an overshoot in mutation state and can ensure a safe driving for the USV.


## 6. Discussion and Conclusion

The ADHDP control is a data-driven based method, which can be used without a plant model. In the simulation, it can be seen that the adjusting time of the ADHDP controller is much smaller than that of the PID controller, but the overshoot is larger than that of the PID. Therefore, an adaptive control is added to eliminate the overshoot of the ADHDP controller under the condition of a continuous and sharp navigating change.In addition, the simulation proves the feasibility of the adaptive ADHDP control method. This control method also does not need the model of the controlled object, which can simplify the controller design although it needs combining the adaptive method. This method can also increase the speed and stability of the controller. Thus, these methods can lay a foundation for the development of the intelligent USV and have a practical effect for a further enhancing of the intelligent USV control.



---

*Source: 7697143-2019-11-25.xml*