Adaptive dynamic programming-based optimal heading control for state constrained unmanned sailboat

Authors: shitong  zhang
yifei  xu
yingjie  deng
sheng  xu
Venue: IEEE_org_ICIST_2024_Conference
Published: 2024-07-27 10:47:18

Abstract: This paper proposes a new state-constrained adaptive optimal control strategy for unmanned sailboat heading angle tracking considering the motion constraints. To improve the tracking accuracy, a combination of backstepping and adaptive dynamic programming (ADP) is employed. Thus, the issue of virtual control rate derivatives in conventional backstepping control is resolved with satisfactory precision. Firstly, the motion constraints are considered by using the Barrier Lyapunov function (BLF), and the neural networks (NNs) is employed to approximate the model uncertainties and disturbances. Secondly, an adaptive backstepping feedforward controller is proposed, transforming the sailboat's affine nonlinear system tracking problem into a regulation problem. Thirdly, according to the ADP theory, critic NNs are constructed to approximate the analytical solution of the Hamilton-Jacobi-Bellman (HJB) equation, and the optimal feedback control is obtained by online learning. Finally, simulation results demonstrate the effectiveness and optimality of the proposed controller.

Main paper: 
 i. introduction: 
 In the past few decades, surface and underwater intelligent vehicles have played important roles in sea patrols, resource exploration, and rescue. However, due to the consumption of fuel, electricity, and other energy sources, unmanned ships and submarines require a large amount of power support to complete long-distance tasks, which also leads to huge cost problems in [1] - [6] . Since unmanned sailboats can use sails as power, have low costs, and can transmit data in real-time through their sensors, research on unmanned sailboats has gradually entered the interest of scholars. However, due to the This work is partially supported by the Natural Science Foundation of China (No.52101375), the Hebei Province Natural Science Fund (No.E2022203088, E2024203179), the Innovation Capacity Enhancement Program of Hebei Province (No.24461901D), the Joint Funds of the National Natural Science Foundation of China (No.U20A20332), and the Key Research and Development Project of Hebei Province (No.21351802D). complex marine environment, the disturbances of wind and waves can easily generate obstructive forces on the sails and keel, making the sails control difficult. Therefore, the authors of [7] proposed a path tracking method for unmanned sailboats that combines logic virtual ship (LVS) guidance law and dynamic event-triggered control. In [8] , a path tracking control scheme is proposed for unmanned sailboats that combined backstepping and dynamic surface control technology. In order to reduce the sideslip angle error during sailing, the authors of [9] proposed double finite-time observers-based line-of-sight guidance (DFLOS) and adaptive finite-time control (DFLOS-AFC) strategies. However, they did not consider the issue of state constraints. When subjected to significant interference, to return the sailboat to the reference heading, a larger rudder angle is required, which will damage the actuator as it cannot accept significant deflection in a short period. Therefore, while ensuring the accuracy of sailboat heading control, it is also necessary to ensure that the turning speed of the boat heading satisfies the prescribed constraints.In order to solve the problem of unmanned sailboat heading angle tracking accuracy, many scholars have conducted research on it. In [10] , a control strategy of adaptive echo state networks and backstepping is proposed to control the steering angle of the rudder. The authors of [11] designed a nonlinear heading controller of velocity vector direction to track the reference heading angle. In [12] , the L 1 adaptive control theory is proposed to complete heading control and ensure the stability of the required heading angle. However, most papers focused on the heading control of sailboats using backstepping or sign functions, which results in low tracking accuracy and cannot achieve optimal control effects.The key problem of the nonlinear optimal control is that the analytical solution of the Hamilton-Jacobi-Bellman (HJB) equation is difficult to solve. In order to obtain the analytical solution, the authors of [13] proposed the ADP theory, which approximates the solution of the equation online. However, the problem of slow approximation speed and multiple iterations has led to an explosion in computational complexity. Recently, with the rapid development of NNs, their approximation performance and speed for unknown nonlinear functions have become increasingly excellent. Therefore, the research combining ADP and NNs has solved the above problems. In [14] , a new event-triggered optimal trajectory tracking control method based on goal representation heuristic dynamic programming (GrHDP) for underactuated ships is proposed. In [15] , they proposed a model free dual heuristic dynamic programming (DHP) method for unmanned aerial vehicle attitude control. The authors of [16] applied ADP theory to the path planning problem of mobile robots. However, to the best of our knowledge, there is limited research on applying ADP theory to the heading control of unmanned sailboats.Taking inspiration from the above analysis, we introduce the LBF function to solve the state constraint problem. In the control design of heading tracking, while introducing the backstepping method, the ADP theory is also introduced to ensure the optimal tracking accuracy. We constructed an evaluation NN to approximate the analytical solution of the HJB equation and obtained the optimal feedback control through online learning. The main contributions are concluded below:• Compared with previous papers of [7] - [9] , we proposed a LBF-based method to overcome the issues of the motion constraints caused by the turning rate limitation of unmanned sailboats. • By using the ADP optimal feedback control strategy, our designed control method achieves optimal tracking accuracy in heading control compared to [9] - [12] . • According to the construction of the critic network, a training speed acceleration method is developed using online learning methods. The remaining of this paper is organized as follows. Section II elaborates on the heading angle control model of unmanned sailboats and lemma. Section III designs the backstepping feedforward controller and the optimal feedback controller. Section IV provides simulation verification of the proposed strategy superiority. Section V gives some conclusions.The sailboat is divided into four parts: the sail, rudder, keel, and hull. Combined with the theory of gas fluid dynamics, force analysis is conducted on each part, ignoring the undulating and pitching motion of the sailboat to establish a 3-DOF mathematical model of sailboat motion. The sailboat model considering external interference and control input rudder angle in this paper isEQUATIONwhere ψ ∈ R M and r ∈ R M are system state variables, respectively; u ∈ R M is the forward velocity; v ∈ R M is the lateral velocity; δ s ∈ R M is the sail angle; τ wr is the external disturbance; f r (•) is the nonlinear function of the unknown model of a sailboat; g r is the unknown control gain; τ r is the control input of the system. Assumption 1 [17] : Due to the fact that unmanned sailboats navigate in limited space, there exists a normal number that satisfies the heading angle and heading angular speed to be less than or equal to this normal number. That is, ψ ≤ k ψ and r ≤ k r .Lemma 1 [10] : Due to the outstanding ability of NNs in function approximation, they are often used to approximate nonlinear functions. Therefore, NNs can be used to approximate unknown functions as follows:EQUATIONwhere W = (w 1 , w 2 , ..., w n ) T denotes the desired weight of the NNs and ε ≤ ε denotes the approximating error.In this section, the backstepping method based on an adaptive NNs framework is adopted to transform system (1) into an affine nonlinear system. According to the system (1), define the error system function asEQUATIONwhere ψ d and r d are the reference heading angle and yaw speed, respectively. Differentiating ψ e along with (1) , it has:EQUATIONwhere r d = r α d +To construct the desired feedforward yaw speed virtual control input, consider the BLF asEQUATION)where k ψ is a positive value of the state constraint. Calculate the derivate of V 1 , we haveEQUATIONTherefore, the feedforward virtual yaw speed input r α d can be designed asEQUATIONwhere k 1 > 0 is a tuning parameters. Substituting ( 8) into (7), we haveEQUATIONTaking the derivative of the second equation of (3) yieldsEQUATIONwhere e d = [ψ d , r d ] T . The unknown model uncertainty function f r (•) and ṙd can be transferred by the following functionEQUATIONwhere z 2d = [e T d , ψ e , r e ] T . According to the Lemma 1, the above (11) can be approximated by NNs as follows:EQUATIONwhere W T 2 = W 2 -Ŵ2 is the NNs approximate error and Ŵ2 is the estimation of optimal weight vector W 2 . Through (11) and ( 12), f r (•) -f r (e d ) can be approximated as follows:EQUATIONwherep(e) = Ŵ T 2 σ 2 (z 2 ) -Ŵ T 2 σ 2 (z 2d ), e = [ψ e , r e ] T and F 2 (z 2 ) = f r (•) -ṙd is a function of r d and ṙd . The input z 2 is chosen as (e T d , ψ, r, δ s , u, v) T .According to ( 12) and ( 13), ( 10) can be written asEQUATIONTo construct the feedforward virtual control input τ α r , consider the BLF asEQUATIONEQUATIONwhere k r is a positive value of the motion constraint. Calculate the derivate of V 2 , we haveEQUATIONAccording to the Young's inequality, we can get thatEQUATIONSubstituting ( 17) into ( 16), we haveEQUATIONTherefore, the feedforward control input τ α r can be designed asEQUATIONwhere k 2 > 0 is a tuning parameters. The NNs weight vector adaptation law Ŵ2 can be designed asEQUATIONwhere β 2 > 0 is also a tuning parameters. Substituting ( 19) and ( 20) into ( 18), we haveV2 ≤ -k 1 ψ 2 e -k 2 r 2 e + 1 2 ε2 2 + β 2 W T 2 Ŵ2 + p(e)r e k 2 r -r 2 e + r e g r k 2 r -r 2 e τ * r + ψ e r * d k 2 ψ -ψ 2 e (21)According to the Young's inequality, we can get thatEQUATIONTherefore, the above (22) can be written as In previous research, the feedforward controller τ α r was calculated based on the derivative of the virtual controller ṙα d . However, in practical applications, it is not easy to get analytical solutions for ṙα d . Our proposed control method denotes r d = r α d + r * d , which is determined by both r α d and r * d and they are obtained through NNs weights and the system's state. Therefore, we approximate the derivative of the virtual controller using NNs in (11) . With this approach, the feedforward controller r α d and τ α r can be obtained directly from the current system without the need for the derivative of the virtual controller used in previous studies. Consequently, compared to previous work, the method we propose is more feasible to implement in practical applications.V2 ≤ -k E 2 + 1 2 ε2 2 + 1 2 β 2 W T 2 W 2 - 1 2 W T 2 W2 + p(e)Rewriting (23) as follow:EQUATION)whereẼ = [k 2 ψ -ψ 2 e , k 2 r -r 2 e ] T .The feedforward controller is expressed asU α = [r α d , τ α r ].The feedback optimal controller U * = [r * d , τ * r ] will be designed in the subsection. Therefore, they constitute the controller of the entire system.According to (24), the design of an individual feedforward controller for U α cannot guarantee the stability of the entire closed-loop system. Therefore, to ensure the stability of the last term in (24), a feedback optimal controller is designed based on ADP theory. With this design, not only the tracking ability of the system can be optimized, but also the system's stability can be ensured.The last term in (24) can be written as:Ė = 0 p(e) + 1 0 0 g r U * (25)Further, it can be obtained thatEQUATION)where E = [ψ e , r e ] T are the heading angle error and yaw speed error,P (E) = [0, p(e)] T , G = diag[1, g r ] T .According to the ADP theory, the performance index function can be define asEQUATIONwhere Q ∈ R 2×2 and R ∈ R 2×2 are positive definite matrices. The Hamiltonian function of the performance index function can be defined asEQUATIONwhere ∇J(E) = ∂J(E) ∂E denotes the derivative of J(E) with regard to E. In order to solve the HJB equation, the feedback optimal control U * can be designed asEQUATIONFrom (28), the optimal performance index function J * (E) can be obtained by minEQUATIONSubstituting the above (30) into (28), the HJB equation can be rewritten as follow:EQUATIONIt is obvious that the above equation is a nonlinear partial differential equation, so it is difficult to obtain its analytical solution. Therefore, to address this issue, the ADP theory is adopted. By constructing a single-layer NN to approximate the following optimal performance index function asEQUATIONwhere W c denotes the optimal weight vector of critic NNs, σ(•) is the activation function, ε c (E) is the critic NNs approximation error.The gradient of the optimal performance index function J * (E) with regard to E can be defined asEQUATIONFrom ( 33) and ( 32), (29) can be rewritten asEQUATIONTherefore, the HJB equation can be further designed asEQUATIONwhere ε HJB is the error. By using NNs to estimate the desired weights of performance index function as follow:EQUATIONwhere Ŵc and Ĵ(E) are estimations of W c and J(E), respectively. Let weight estimation error as Wc = W c -Ŵc , the estimate of optimal control U * can be designed asEQUATIONThen, the HJB equation can be approximated asEQUATIONThe objective error function of critic NNs is defined asEQUATION)From the [18] , we can design a appropriate critic NNs updating law, which can guarantee that Ŵc converges to W c and also minimize the objective error function (39).EQUATION)where k c > 0 is the tuning parameter, Γ = ∇σ(E)(P (E) + G Ū ), ζ = Γ 1+Γ T Γ , K 1 and K 2 are the tuning parameter. ∆ is designed asEQUATIONwhere V (E) is a Lyapunov function. From this, we can obtainV (E) = ∇V (E) T Ė = ∇V (E) T P (E) + GU * = -∇V (E) T S∇V (E) ≤ 0 (42)where S is a positive definite matrix. Specifically, V (E) is a function of the state variable E and can be chosen appropriately, for example, V (E) = E T E. Remark 1: The weight Ẇc update process consists of the following four components: The first component employs gradient descent for design. The second component ensures the boundedness of the weights. The third and fourth components guarantee the stability of the weights. Through this design, the proposed control strategy achieves a higher tracking accuracy while ensuring the rapid and stable update of the neural network weights.The model parameters of the unmanned sailboat are selected from [10] . In order to facilitate simulation analysis without losing generality, the reference heading is set as ψ d = sin(t). To verify the superiority of the proposed strategy, we will compare the "LSBG" strategy of [10] . Fig. 1 shows the realtime curve of heading angle tracking, and the results show that the designed optimal control method can track the reference signal with smaller errors and within state constraints. The heading angular velocity tracking and its state constraints are shown in Fig. 2 . Fig. 3 and Fig. 4 illustrate the error curves of heading angle and heading angular velocity, indicating that the proposed strategy achieves better tracking performance than the "LSBG" strategy. Fig. 5 displays the control inputs for control input τ r under "LSBG" strategy, backstepping feedforward control, optimal feedback control, and system control under the proposed strategy, respectively. Fig. 6 shows the update curve of the evaluation network weights, it is obvious that the speed of online learning has reached stability in a very short time. From the above analysis, the proposed strategy can not only ensure better tracking accuracy, but also avoid system state violations of constraints.ψ = 1.2, k r = 1.5, k 1 = 3, k 2 = 6, k c = 3.8, β 2 = 4, K 1 = 0.0001I, K 2 = 0.00001I, Q = I, R = 0.25I.In this paper, the optimal control method based on ADP is proposed for the tracking control of unmanned sailboats with heading turning constraints. The proposed LBF-based method solves the problem of state constraints. The feedforward backstepping controller and the feedback optimal controller were designed using the backstepping method and ADP theory, respectively. The learning ability of critic NNs has been accelerated through online learning strategies. Finally, the simulation verified the optimality of the proposed strategy. In the future, we will apply this method to the path-tracking task of unmanned sailboats in pratice. 
 
 Appendix:  



Reviews: review: title: title: This paper proposes a new state-constrained adaptive optimal control strategy for unmanned sailboat heading angle tracking considering the motion constraints. The tracking accuracy is satisfactory.
main_review: main_review: 1.Ensuring that the ADP algorithms converge to an optimal or near-optimal solution is challenging, especially in complex and high-dimensional environments, how to ensure convergence in this article
2.HJB equations are typically nonlinear partial differential equations. The nonlinearity complicates both analytical and numerical solutions, making it challenging to find precise solutions. How do the authors consider this problem?
3.The formula numbers mentioned in the paper should not have "the" in front, it is recommended to make modifications.
4.Please add a physical description on the vertical coordinate in Figures 6.
title: title: The topic under consideration is interesting. This paper can be accepted after some modifications.
main_review: main_review: In this paper, a new state-constrained adaptive optimal control strategy is proposed for unmanned sailboat heading angle tracking considering the motion constraints. The topic under consideration is interesting. Detailed comments and suggestions are listed as follows.
1.	The English writing of the paper needs to be further polished, and some typos should be fixed, such as, “Substituting the above (8) into the (7), and it has…” and “Calculate the derivate of V1, and it has…”.
2.	The detailed process of controller design cannot be easily followed. Please give an important theorem about the feasibility of the proposed controller on Chapter III.
3.	The format of References should be standardized.
title: title: Accept, some mistakes should be revised
main_review: main_review: 1. The essential difference between the contributions and the existing works should be highlighted.
2. The performance of the designed strategy is insufficient. Detailed demonstrations should be supplied to the simulation results.