Event-Triggered Optimal Tracking Control for Uncertain Nonlinear System Based on Reinforcement Learning

Authors: yuanhao  wang
weiwei  bai
Venue: IEEE_org_ICIST_2024_Conference
Published: 2024-08-14 08:29:22

Abstract: In this paper, an event-triggered optimal tracking
control problem is studied for uncertain nonlinear system based
on reinforcement learning (RL). Firstly, a class of nonlinear
dynamic system with general uncertainty is considered and
the augmented system comprising tracking error and reference signal is constructed. Secondly, an improved adaptive
dynamic programming (ADP) technique, involving actor-critic
algorithm and fuzzy logic system, is developed to solve the
Hamilton–Jacobi–Bellman (HJB) equation respect to nominal
augmented system. Thirdly, in order to reduce the mechanical
wear of actuator and energy consumption, event-triggered mechanism is performed in controller update. Finally, stability analysis
proofs the uniformly ultimately bounded (UUB) of all signals in
the closed-loop system via Lyapunov theory. Simulation results
verify feasibility of proposed scheme.

Main paper: 
 i. introduction: 
 Reinforcement Learning (RL) as an effective technique has competent in facilitating adaptive optimization strategy [1] , [2] . Generally, optimization is implemented via seeking minimized or maximized cost function to solve the Hamilton-Jacobi-Bellman (HJB) equation [3] . However, there exists a challenge about acquiring analytic solution of HJB equation directly for nonlinear dynamic systems [4] . Therefore many researchers proposed numerical solution of HJB equation [5] . Adaptive dynamic programming (ADP) as an advanced numerical solving method, has been widely applied to achieve the optimal tracking control of nonlinear systems.In contrast to traditional dynamic programming, ADP can be utilized to design optimal controller forward in time, which effectively avoids "curse of dimensionality" [6] , [7] . In addition, an improved ADP framework consists of actorcritic algorithm and fuzzy logic systems is constructed. So far, there have been many scholars devoting to developing ADP techniques [8] - [10] . In [11] , ADP method was implemented to solve a new neuro-optimal control problem of nonlinear dynamic systems by employing one critic and two actor networks. In [12] , a neural-network-based ADP method was developed to solve the optimal tracking control problem of a class of nonlinear systems with unmatched uncertainties. In [13] , linear singularly perturbed system was studied via employing ADP framework to achieve optimal control. These literatures concentrated on application and development of ADP and RL, but they did not consider the condition with mechanical wear of actuator and energy consumption. As a result, it is of necessity to perform event-triggered mechanism in control design for reducing mechanical wear and saving energy in actual engineering practice [14] .The key of event-triggered control algorithm is triggering threshold [14] . When signal exceeds triggering threshold, control policy will be updated [15] , [16] . In this paper, an event-triggered optimal tracking control scheme for uncertain nonlinear systems based on RL is developed. There are two main contributions:(1) An improved ADP and RL algorithm involving actorcritic and fuzzy logic systems is developed, which develops the optimal control strategy and effectively balances the tracking control performance and control costs.(2) Event-triggered mechanism is performed in controller design, the unnecessary control input is avoided, achieving the reduction of mechanical wear and the energy saving in engineering practice.The organization of this paper is shown as follows. System dynamic description and fuzzy logic systems are stated in Section II. Optimal controller and event-triggered controller are designed in Sections III and IV, respectively. Stability analysis, simulation and conclusion are shown in Sections V, VI and VII, respectively.Consider a class of continuous-time nonlinear dynamic systems which can be described byEQUATIONwhere x(t) ∈ R n is the state variable, u(t) ∈ R m is the control input, f (x(t)) ∈ R n and g(x(t)) ∈ R n×m are the unknown smooth function and unknown smooth function matrix respectively, D(x(t)) is the unknown disturbance with ∥D(x(t))∥ ≤ λ D and λ D is a positive parameter.To achieve tracking control, a reference signal is given byEQUATIONwhere r(t) ∈ R n is a bounded desired trajectory and δ(r(t)) is a Lipschitz continuous function. Let the tracking error beEQUATIONCombining equations ( 1), ( 2) and (3), one can yield the following dynamic of tracking errorė (t) = f (x (t)) + g (x (t)) u (t) + D(x(t)) -δ(r(t)) (4)Note that x (t) = e (t) + r (t), equation ( 4) can be rewritten asEQUATIONFor the sake of facilitating description, define ξ (t) = [e T (t), r T (t)] T ∈ R 2n , and then dynamic systems (2) and ( 5) can be augmented as a concise formEQUATIONwhere F (ξ (t)) and G (ξ (t)) are new matrices and ∆D(ξ(t)) can be still regarded as a new uncertain term. In par-ticular, F (ξ (t)) = f (e (t) + r (t)) -δ(t) δ(t) , G (ξ (t)) =g (e (t) + r (t)) 0 n×m and ∆D(ξ(t)) = D(e (t) + r (t)) 0 n×1 .Undoubtedly, the new uncertain term ∆D(ξ(t)) is still upper bounded since ∥∆D(ξ(t))∥ = ∥D(e (t) + r (t))∥ = ∥D(x (t))∥ ≤ λ D (7) To accomplish tracking control of dynamic system (1) to reference signal (2), the feedback controller u(ξ) will be constructed. One can yield that the closed-loop system is asymptotically stable under the controller u(ξ) for the uncertain and bounded term ∆D(ξ(t)). Therefore, the optimal control policy can be applied by considering appropriate cost function of the subsequent nominal system the same as that in [5] .Define a nonlinear continuous function P (x) over a compact set U, for any constant ε > 0, there exists fuzzy logic systems ω T φ (x) such that [17] EQUATIONwhere x = [x 1 , . . . , x j ] T is the input vector of fuzzy logic systems, ω = [ω 1 , ω 2 , . . . , ω L ] T ∈ R L is the degree of membership and L > 1 is the number of fuzzy rules, ε is the fuzzy minimum approximation error.φ (x) = [φ 1 (x) , φ 2 (x) , . . . , φ L (x)]T is fuzzy basic function vector and φ i (x) is selected as follows:EQUATIONwhere F l i (i = 1, . . . , j; l = 1, . . . , N ) is the fuzzy set and µ F l i (x i ) is the membership function.In this section, ADP comprising actor-critic algorithm and fuzzy logic systems will be employed to design the value function L * (ξ) and control policy u * (ξ), and design degree of membership update laws.In actor-critic framework, value function and control policy are approximated by critic and actor fuzzy systems, respectively. Optimal cost function (13) and feedback controller (15) represent value function and control policy for optimal tracking control problem, respectively.Consider the nominal part of the augmented system (6) , that isEQUATIONFor the nominal system (10) , this cost function is consideredEQUATIONwhereQ (ξ) = ξ T Qξ, R = R T . Q and R are positive defined matrix.Subsequently, one can define the Hamiltonian of the optimal problemH (ξ, u(ξ)) =Q (ξ) + u(ξ)T Ru(ξ)EQUATIONwhere ∇L (ξ) represents the partial derivative of L (ξ).Generally, as long as finding the optimal cost function can we derive the optimal controller. The infinitesimal version of cost function is regarded as the optimal cost function, one hasEQUATIONThe optimal cost function is the solution of the HJB equation which satisfiesEQUATIONConsequently, the optimal feedback controller is yieldedEQUATIONOne need to solve the HJB equation ( 14) and obtain the optimal controller (15) for nominal system (10) . However, the solution of HJB equation ( 14) is difficult to be obtained directly. Therefore, fuzzy logic systems and adaptive actorcritic will be utilized to find its estimated solution.Fuzzy logic systems are employed to reconstruct the value function L * (ξ)EQUATION)where ω is the degree of membership of fuzzy logic systems, φ (ξ) is the fuzzy basis function and ε (ξ) is the unknown fuzzy approximate error. Considering ( 15) and ( 16) yields the optimal controller described by fuzzy logic systems asEQUATIONIn order to clearly analyze, define a non-negative matrixEQUATIONOne can derive the HJB equation reconstructed by fuzzy logic systems, combining with ( 16), ( 17) and ( 18), one hasH (ξ, u * (ξ), L * (ξ)) =Q (ξ) + ω T ∇φ (ξ) F (ξ) - 1 4 ω T A (ξ) ω + ε HJB = 0 (19)and the residual error ε HJB is expressed asEQUATIONThe estimation of value function L * (ξ) and control policy u * (ξ) can be constructed by critic and actor fuzzy, respectively.EQUATION)where ωa is the actor estimated degree of membership and ωc is the critic estimated degree of membership. Noticing (21) and ( 22), one can derive the following estimated HamiltonianEQUATIONTo obtain the degree of membership update laws of fuzzy logic systems, defining the objective function asE c = 1 2 e c T e c, where e c = Ĥ ξ, û * (ξ), L * (ξ) -H(ξ, u * (ξ), L * (ξ)) is the Bellman error. In order to conquer the difficulties of searching controller and adaptive laws, the following assumption is made and the additional term can be constructed to improve the learning process. Assumption 1: [5] Define L s (ξ) is a continuous differentiable Lyapunov function candidate satisfyingLs (ξ) = ∇ T L s (ξ) (F (ξ) + u * (ξ)) < 0 (24)and then, there exists a positive matrix K ∈ R 2n×2n ensuring that∇ T L s (ξ) (F (ξ) + u * (ξ)) = -∇ T L s (ξ) K∇L s (ξ) ≤ -λ min (K)∇∥L s (ξ)∥ 2 (25)Based on the gradient decent, degree of membership update laws of fuzzy logic systems are designed, by considering these two Hamilton functions H (ξ, u * (ξ), L * (ξ)) and Ĥ ξ, û * (ξ), L * (ξ) , one hasEQUATIONwhere α a and α c are the basis learning parameters of actor and critic systems, respectively, and α s is the adjustable parameter for the additional term.The event triggering mechanism is defined asEQUATIONt d+1 = inf {t ∈ R| |Γ(t)| ≥ ∆ |u * e (ξ(t))| + M } , t 1 = 0 (29)where the event-triggered error Γ(t) = u * (ξ(t d )) -u * e (ξ(t)), the controller update time is t d , d ∈ Z + . Define the proper parameters 0 < ∆ < 1 and M > 0.When event is not triggered, the control policy will be chosen as u * (ξ(t d )). Otherwise, control policy will be updated and marked as u * e (ξ(t d+1 )). Assume two continuous and time-varying parameters ρ 1 (t) and ρ 2 (t), which results inu * (ξ(t)) = (1 + ρ 1 (t)∆)u * e (ξ(t)) + ρ 2 (t)M where |ρ 1 (t)| ≤ 1 and |ρ 2 (t)| ≤ 1.And then, the event-triggered controller can be rewritten asEQUATIONUsing ( 17) and (30) can yield that u * e (ξ(t)) = -EQUATIONwhere ρ = 1 + ρ 1 (t)∆, ε e (ξ (t)) = ∇ε (ξ (t)) + 2ρ 2 (t) RM .Similarly, based on critic fuzzy logic systems, the estimated event-triggered controller can be obtained, one has û * e (ξ(t)) = -EQUATIONConsidering the HJB equation ( 14), value function (21) and event-triggered controller (32), one can yield the following Hamilton function asEQUATIONSubsequently, degree of membership update laws with respect to event-triggered mechanism can be constructed, one hasEQUATIONTheorem 1: Considering the dynamic system (1), the optimal feedback controller (22), event-triggered controller (32) and the degree of membership update laws (26), ( 27), (34) and ( 35) are developed. Based on Lyapunov theory, all signals are uniformly bounded (UUB) in the closed-loop system.For the sake of investigating the stability of error dynamics and close-loop states, the following assumption is given by Assumption 2: On a compact set Ω, G (ξ), ∇φ (ξ), ∇ε(ξ), ξ * and ε HJB are bounded. ∥G (ξ)∥ ≤ λ g , ∥∇φ(η)∥ ≤ λ φ , ∥∇ε (η)∥ ≤ λ ε , ∥ξ * ∥ ≤ λ ξ and ∥ε HJB ∥ ≤ λ HJB , where λ g , λ φ , λ ε , λ ξ and λ HJB are positive constants.In this section, Lyapunov theory will be employed to demonstrate Theorem 1.Case1 : Event are not triggered. Consider the feedback controller ( 22) and the related degree of membership update laws (26) and ( 27).According to HJB equation ( 19), it can be transformed asQ (ξ) = -ω T ∇φ (ξ) F (η) + 1 4 ω T A (ξ) ω -ε HJB (36)Considering the degree of membership update laws ( 26) and ( 27), combining with ˙ ω a = -ωa and ˙ ω c = -ωc , one hasEQUATIONEQUATIONThen the following Lyapunov function can be chosen asEQUATIONits derivative isEQUATIONSubstituting ( 22) into (10) and observing the dynamic system ξ * = F (ξ)+G(ξ)u * (ξ) with optimal controller u * (ξ), one can acquireEQUATIONConsidering above formulations, one can further derive thatEQUATIONNext, equation ( 43) can be expended to conduct mathematical operations based on Assumption 2 and yields thatEQUATIONwhere λ 1 , λ 2 and λ 3 are positive constants.Considering Assumption 1 and equation (44), one can further derive thatEQUATIONwhereλ ∂ = λ 3 + λg 4 λε 4 (∥R -1 ∥) 4 16λmin(K) . As a result, if ∥ ω a ∥ ≥ 4 λ ∂ λ1 or ∥ ω c ∥ ≥ λ ∂ λ2 or ∥∇ L s (ξ)∥ ≥ λ ∂ λmin(K)αs( 1 αa + 1 αc ) + λg 2 λε 2 (∥R -1 ∥) 2 4λmin(K)hold, Ṡ (t) ≤ 0 will be satisfied. Finally, one can conclude that all signals are UUB.Case2 : Events are triggered. Consider the event-triggered controller (32) and the degree of membership update law (34) and (35).Choosing the following Lyapunov functionEQUATIONsame proof as that in Case1 , we can demonstrate all signals are UUB.Motivated by [14] , the derivative of event-triggered function can be written as d dt |Γ(t)| = d dt (Γ(t) × Γ(t))In this section, YUKUN of Dalian Maritime University is utilized to verify the validity and flexibility of the optimal control strategy considering event-triggered mechanism. The parameters of YUKUN are as follows: length between perpendiculars is 105 m, beam is 18 m, rudder area is 11.46 m 2 , loaded speed is 16.7 kn, full amidships draft is 5.2 m, full loaded displacement is 5735.5 m 3 , block coefficient is 0.5595. Maritime environment can be set that: wind direction ψ wind = 30 • , wind scale S = 6, current direction ψ current = 30 • , current velocity v current = 5kn.Therefore, a continuous-time ship dynamic system can be considered     ẋ1 = x 2 ẋ2 = -1 T α s x 2 + β s x 2 3 + K T (u + δ w ) y = x 1 (49)where x 1 and x 2 ∈ R are state variables, u ∈ R is the control input variable; reference signal x 1d = sin (πt/25); the rudder gain K = 0.314 and time constant T = 62.387; designed parameters α s = 100 and β s = 50. Design parameters α a = 0.001, α c = 1, α s = 100000, R = 0.067, ∆ = 0.39, M = 0.001. The initial state can be set that x 0 = [-0.3, 2.1, 0.1, 0.03] T , the initial degree of membership can be set thatω a0 = [-3.4, -4, -3.5, -1.8, -2, 0, -1.4, -0.8, -1.8, -2] T , ω c0 = [1, 1.3, 1.5, 1.3, 0, 0, 1.5, 3, 3.3, 3] T .Simulation results are illustrated in Fig. 1 2 3 4 . The tracking trajectory and error are shown in Fig. 1 , where the ship course can rapidly track the reference course in 10 seconds and tracking error can converge to a bounded compact set of zero based on the designed event-triggered adaptive optimal controller. Fig. 2 describes the general control input and the event-triggered control input. Its result illustrates eventtriggered controller is superior to common controllers under the same conditions. The numerical values of event-triggered controller are smaller than that of the general controller, which effectively verifies the competent in reducing mechanical wear and saving energy of the event-triggered mechanism. Fig. 3 describes the corresponding triggered time that highlights the advantages of cost saving for event-triggered controller. In the end, Fig. 4 gives the value function and policy function degree of memberships convergence exhibitions which demonstrate degree of membership signals can rapidly coverage to a bounded range.In this article, an event-triggered optimal tracking control scheme has been proposed for uncertain nonlinear systems based on RL. An improved ADP technique combining actorcritic algorithm and fuzzy logic systems have been implemented in solving HJB equation of nominal system. To reduce mechanical wear of actuator and save energy, event-triggered mechanism has been performed to update controller. All signals are UUB by Lyapunov demonstration and simulations verify the feasibility of proposed scheme. In the future, we will study the tracking control problem based on deep reinforcement learning and the multi-agent systems also is an interesting direction. 
 
 Appendix:  



Reviews: review: title: title: Accept
main_review: main_review: Comment: This paper study an event-triggered optimal tracking control problem for uncertain nonlinear system based on reinforcement learning. Simulation results verify feasibility of proposed scheme. The theory is correct and can be accepted after responding the following comments.
(1) In the introduction, it is not enough to state the current work. It should be expanded and reconstructed.
(2) There are many typos and grammar errors. The authors should have a native English speaker or software packages to perform the editing check.
(3) In the simulation section, the description of the simulation results is not detailed enough.
title: title: Event-Triggered Optimal Tracking Control for Uncertain Nonlinear System Based on Reinforcement Learning
main_review: main_review: In this paper, an event-triggered optimal tracking control problem is studied for uncertain nonlinear system based on reinforcement learning. There are some problems that should be replied. Comments for this submission are given as follows:
1.The paper makes a numerous of assumptions without providing a comprehensive justification or explanation. The mathematical integrity of the paper would be enhanced by the inclusion of more detailed explanations or justifications for these assumptions.
2. Although the paper references numerous pertinent works, the literature review could be enhanced, including adaptive dynamic programming and reinforcement learning in tracking control systems.
3.How to choose the main parameter of the proposed strategy to achieve the control object ? More details should be given.
4. The main contributions are not clear enough or not expressed clearly. Please provide a more detailed explanation.
title: title: This article is very interesting and a good one
main_review: main_review: This paper studied an event-triggered optimal tracking control problem for uncertain nonlinear system based on RL. The obtained result is valuable and can be accepted if the following problems can be clarified. 
(1) In the introduction, the shortages of those relevant studies are suggested to be further summarized. 
(2) In the end of Section 1, the organization of this study is suggested to be summarized. 
(3) There exist several spelling and grammar errors. Please check carefully and further polish 
(4) In the Simulation, more analysis can be added to better explain the main results of this paper. 
(5) The future work is missing in the Conclusion. (6) The references should be updated and their format standardized for enhanced consistency and accuracy.

Comments: title: This article is very interesting and a good one
comment: This paper studied an event-triggered optimal tracking control problem for uncertain nonlinear system based on RL. The obtained result is valuable and can be accepted if the following problems can be clarified.
(1)	In the introduction, the shortages of those relevant studies are suggested to be further summarized.
(2)	In the end of Section 1, the organization of this study is suggested to be summarized. 
(3)	There exist several spelling and grammar errors. Please check carefully and further polish
(4)	In the Simulation, more analysis can be added to better explain the main results of this paper.
(5)	The future work is missing in the Conclusion.
(6)	The references should be updated and their format standardized for enhanced consistency and accuracy.