# Online Adaptive Optimal Control of Vehicle Active Suspension Systems Using Single-Network Approximate Dynamic Programming

Authors: Zhi-Jun Fu; Bin Li; Xiao-Bin Ning; Wei-Dong Xie
Journal: Mathematical Problems in Engineering
 (2017)

---

## Abstract

In view of the performance requirements (e.g., ride comfort, road holding, and suspension space limitation) for vehicle suspension systems, this paper proposes an adaptive optimal control method for quarter-car active suspension system by using the approximate dynamic programming approach (ADP). Online optimal control law is obtained by using a single adaptive critic NN to approximate the solution of the Hamilton-Jacobi-Bellman (HJB) equation. Stability of the closed-loop system is proved by Lyapunov theory. Compared with the classic linear quadratic regulator (LQR) approach, the proposed ADP-based adaptive optimal control method demonstrates improved performance in the presence of parametric uncertainties (e.g., sprung mass) and unknown road displacement. Numerical simulation results of a sedan suspension system are presented to verify the effectiveness of the proposed control strategy.


---

## Body

## 1. Introduction

Improvement in suspension systems plays an important role in achieving the goals of pursuing more comfortable and safer vehicles. Naturally, the suspension systems are then expected to have more intelligence to accommodate themselves in various road conditions. Hence, many control methods as summarized in [1] have been used for suspension controller design. However, model uncertainties due to the parameter uncertainties and unknown road inputs in real suspension systems bring a great challenge for the controller design.It is well known that optimal controllers are normally designed offline by solving Hamilton-Jacobi-Bellman (HJB) equation. For linear system, linear quadratic regulator (LQR) controller is designed offline by solving the Riccati equation (special case of HJB). However, the main drawback of the conventional LQR method lies in the fact that the system model has to be known precisely in advance to find the optimal control law. In addition, the feedback control gains are obtained offline. Once the feedback gains of the controllers are obtained, they cannot be changed with the different driving environment. Thus, a more efficient control strategy is needed to adaptively cope with an active suspension control problem subject to time varying parameters under different driving situations in real time.Recent studies summarized in [2] show that approximate dynamic programming (ADP) based controller design merging some key principles from adaptive control and optimal control can overcome the need for the exact model and achieve the optimality at the same time. The learning mechanism of ADP supported by the actor-critic structure has two steps [3]. First, policy evaluation executed by the critic is used to assess the control action, and then policy improvement executed by the actor is used to modify the control policy. Most of the available adaptive optimal control methods are usually based on the dual NN architecture [4–7], where the critic NN and action NN are employed to approximate the optimal cost function and optimal control policy, respectively. The complicated structure and computational burden make the practical implementation difficult.In this paper, an adaptive optimal control method with simplified structure for the quarter-car active suspension system is proposed. The optimal control action is directly calculated from the critic NN instead of the action-critic dual networks. Meanwhile, robust learning rule driven by the parameter error is presented for the critic NN which is used to identify the optimal cost function. Finally, based on the critic NN, the optimal control action is calculated by online solving the HJB equation. Uniformly ultimately bounded (UUB) stability of the overall closed-loop system is guaranteed via Lyapunov theory. Compared with the conventional LQR control approach, simulation results from a quarter-car active suspension system verify the improved performance in terms of ride comfort, road holding, and suspension space limitation for the proposed ADP-based controller.The remainder of this paper is organized as follows. In Section2, the preliminary problem formulation is given. The proposed ADP-based control algorithm is introduced in Section 3. Section 4 presents the simulation results from a vehicle suspension system and, finally, the conclusions for the whole paper are drawn in Section 5.


## 2. Problem Formulation

The two-degree-of-freedom quarter-car suspension system is shown in Figure1, which has been widely used in the literatures [8, 9]. It represents the motion of the vehicle body at any one of the four wheels. The suspension system is composed of a spring ks, a damper bs, and an active force actuator F. The active force can be set to zero for the passive suspension. The sprung mass ms denotes the quarter equivalent mass of the vehicle body. The unsprung mass mu represents the equivalent mass of the tire assembly system. The springskt andbt represent the vertical stiffness and damping coefficient of the tire, respectively. Vertical displacements of the sprung mass, unsprung mass, and road are denoted byzs,zu, andzr, respectively.Figure 1
Quarter-car active vehicle suspension.The motion equations of the sprung and unsprung mass shown in Figure1 are given as follows [10]:(1)msz¨s+bsz˙s-z˙u+kszs-zu=F,muz¨u+btz˙u-z˙r+ktzu-zr-bsz˙s-z˙u-kszs-zu=-F.Define the state variables as(2)x1=zs-zu,x2=z˙s,x3=zu-zr,x4=z˙u,where zs-zu is the suspension deflection, z˙s is the velocity of sprung mass, zu-zr is the tire deflection, and z˙u is the velocity of unsprung mass.Then, (1) can be further rewritten in the following state space form:(3)x˙=Ax+Bu+Brz˙r,where(4)A=010-1-ksms-bsms0bsms0001ksmubsmu-ktmu-bs+btmu,B=01ms0-1mu,Br=00-1btmu.The objective of the controller design for the active suspension systems should consider the following three tasks [11].Firstly, one of the main tasks is to keep good ride quality. This task means to reduce the vibratory forces transmitted from the axle to the vehicle body via a well-designed suspension controller, that is, to reduce the sprung mass acceleration as far as possible when confronting parameter uncertaintiesms and unknown road displacementzr.Secondly, in order to assure the good road holding performance, the firm uninterrupted contact of wheels to road should be ensured, and the variations in normal tire load related to vertical tire deflectionzu-zr should be small.Lastly, the suspension space requirements should not exceed the maximum suspension deflectionzmax; that is, zs-zu≤zmax.The suspension control goal is to find a controller which can ensure the stability of the closed system and minimize the following performance index:(5)V=12∫0txTQx+uTRudτ,where x=(x1,x2)T and Q and R are chosen by the designer which determine the optimality in the optimal control law.Remark 1.
Most of the existing LQR controller designs are based on (3) with the assumption that all of the parameters are known in advance. Besides, the feedback control law of the LQR approach is usually obtained by solving the Riccati equation offline. Once it is obtained, the control law cannot be updated online when subjected to the parameter uncertaintiesms and unknown road displacementzr. Thus, it may affect the control performance. Therefore, high performance control approach that can be tolerant of various operating conditions should be developed for the active suspension system design.


## 3. Controller Design Based on Approximate Dynamic Programming

In this section, an adaptive optimal control method for active suspension systems is realized by using the ADP approach with only one critic network structure instead of the classic action-critic dual networks as presented in Figure2. The outputs parameters should be the state of the suspension. The implementation of the state feedback control algorithm for the suspension system is based on the requirement that all states of the system can be measurable or part of the states can be estimated by using some estimated methods like Kalman filter. In this paper, we assume that all of the suspension states are available.Figure 2
Schematic of the proposed controller system.The objective of the optimal regulator problem is to design an optimal controller to stabilize system (3) and minimize the infinite horizon performance cost function(6)Vx=∫t∞rxτ,uτdτ,where the utility function with symmetric positive definite matrices Q and R is defined as r(x,u)=xTQx+uTRu.According to the optimal regulator problem design in [4], an admissible control policy should be designed to ensure that infinite horizon cost function (6) related to (3) is minimized. So, the Hamiltonian of (3) is designed as(7)Hx,u,V=VxTAx+Bu+Brz˙r+xTQx+uTRu,where Vx≜∂V(x)/∂x denotes the partial derivative of the cost function V(x) with respect to x.The optimal cost functionV∗(x) is defined as(8)V∗x=minu∈ψΩ∫t∞rxτ,uτdτand it satisfies the HJB equation(9)0=minu∈ψΩ⁡Hx,u,Vx∗=Vx∗TAx+Bu+Brz˙r+xTQx+uTRu,where Vx∗≜∂V∗(x)/∂x. Based on the assumption that the minimum on the right-hand side of (8) exists and is unique, by solving ∂H(x,u,Vx∗)/∂u=0, the feedback form for admissible optimal control u∗ can be derived as(10)u∗=-12R-1BT∂Vx∗∂x.Substituting (10) into (9), we have(11)0=Vx∗TAx+Bu+Brz˙r+xTQx+-12R-1BTVx∗TR-12R-1BTVx∗.Assumption 2 (see [4]).
The solution to (9) is smooth, which allows us to bring in informal style of the Weierstrass higher-order approximation theorem. Then, there exists a complete independent basis ψ(x)∈RI such that the solution V∗(x) to (9) is uniformly approximated.From (10), one can learn that optimal control value u∗ is based on the optimal cost function V∗(x). However, it is difficult to solve the HJB equation (11) to obtain V∗(x) due to the uncertain parameters in matrixA and the unknown road displacement input. The usual method is to get the approximate solution via a critic NN as in [12, 13]. Hence, from Assumption 2, it is justified to assume that there exist weights W1 such that the value function V∗(x) is approximated as (12)V∗x=WcTψx+ξ,where Wc∈RI is the nominal weight vector, ξ is the approximation error, ψ(x)∈RI is the active function, and I is the number of neurons.Then, substituting (12) into (10), one obtains(13)u∗=-12R-1BT∇ψTxWc+∇ξ.In practical implementation, the critic NN is represented as(14)u=-12R-1BT∇ψTxW^c,where W^c is the estimation of nominal Wc.Remark 3.
There are many ways to get the estimation value ofWc such as least-squares [4] or gradient method [14, 15]. It has been proved in [16–18] that adaptive estimation method considering the parameter information can greatly improve the convergence speed in contrast to the conventional estimation method driven by the observer error. Inspired from these facts, a novel robust estimation method of Wc is presented in the following analysis.
Substituting (12) into (9), one obtains(15)Y=-WcTX-ξHJB,where X=∇ψ(x)(Ax+Bu), Y=xTQx+uTRu, and ξHJB=WcT∇ψ(x)Brz˙+∇ξ(Ax+Bu+Brz˙r) denotes the Bellman error caused by the approximation of the cost function.
The filtered variablesXf,Yf, and ξHJBf are defined as(16)ηY˙f+Yf=Y,Yf0=0,ηX˙f+Xf=X,Xf0=0,ηξ˙HJBf+ξHJBf=ξHJB,ξHJBf0=0,where η is a filter parameter. It should be noted that the fictitious filtered variable ξHJBf is just used for analysis.
Then, the auxiliary regression matrixE∈Rl×l and vector F∈Rl are defined as(17)E˙t=-ηEt+XfXfT,E0=0,F˙t=-ηFt+XfY-YfηF0=0,where η is a positive constant as defined in (16).
The solution to (17) is derived as (18)Et=∫0te-ηt-rXrXTrdr,Ft=∫0te-ηt-rXrYr-Yfrηdr.
Another auxiliary vectorM is denoted as (19)M=EtWc+Ft.
Finally, the adaptive law ofWc is provided by (20)W^˙c=-μM,where μ is the learning gain.Theorem 4.
For system (3) with adaptive optimal control signal u given in (14), and adaptive law (20), the optimal control u converges to a small bound around its ideal optimal solution u∗ in (13).Proof.
Define the Lyapunov function as(21)L=Lo+Lc,where Lo and Lc are defined as (22)Lo=12W~cTμ-1W~c,(23)Lc=ΓxTx+κV,where μ,Γ,κ>0 are positive constants.
Then, from (18) and (19), one obtains(24)M=EtW^c+Ft=-EtW~c+ςf,where ςf=-∫0te-η(t-r)XfξHJBfdr is bounded as ςf≤ς-f.
From [16], we know that the persistently excited (PE) X can guarantee that the matrix E(t) defined in (18) is positive definite; that is, λmin(E)>σ>0. Then, according to the fact W~˙c=-W^˙c, the time derivative of (22) is calculated as(25)L˙o=W~cTμ-1W~˙c=-EtW~cTW~c+W~cTςf≤-W~cσW~c-ς-f.
Finally,W~c converges to the compact set Ω:W~c≤ς-f/σ.
From the basic inequalityab≤a2δ/2+b2/2δ with δ>0, we can rewrite (25) as(26)L˙o≤-σ-12δW~c2+δς-f22.
The time derivative of (23) can be deduced from (3) and (8):(27)L˙c=2ΓxTAx+Bu+Brz˙r+κ-xTQx-uTRu≤-κλminQ-2ΓλminA+δx2-κλminR-ΓB2δu2+ΓBrz˙rδ2.
From (26) and (27), the time derivative ofL is L˙=L˙o+L˙c and satisfies the following inequality:(28)L˙≤-h1x2-h2u2-h3W~c2+ϑ,where h1=κλmin(Q)-2Γ[λmin(A)+δ], h2=-[κλmin(R)-ΓB2/δ], h3=(σ-1/2δ), and ϑ=Γ(Brz˙r/δ)2+δς-f2/2.
h
1
,
h
2
,
h
3 are all positive by choosing the appropriate parameters satisfying the following conditions: (29)κ>max-2ΓλminAλminQ,ΓB2/δλminR,σ>12δ.
ThenL˙<0 if (30)x>ϑh1oru>ϑh2orW~c>ϑh3,which means the system state x, control input u, and critic NN weight error W~c are all uniformly ultimately bounded (UUB).
Moreover, we have(31)u-u∗=12R-1BT∇ϕTW~c+12R-1BT∇ξ.
The steady state of the upper bound of (31) is(32)limt→∞u-u∗≤12R-1BT∇ϕTW~c+∇ξ≤ζ,where ζ depends on the critic NN approximation error.
From (32), we know that the optimal control u converges to a small bound around its ideal optimal solution u∗.Remark 5.
In this paper, a simplified ADP structure with a single critic NN approximator instead of the commonly used complex dual NN approximators (critic NN and actor NN) is proposed. Moreover, the weight updating law of the critic NN is based on the parameter estimation error rather than minimizing the residual Bellman error in the HJB equation by using the least-squares or the gradient methods, so that the weight error of the critic NN is guaranteed to converge to a residual set around zero with faster speed.


## 4. Simulation Results and Discussions

In this section, numerical simulations are carried out for a sedan active suspension model platform presented in Section2 to evaluate the effectiveness of the proposed ADP-based adaptive control method designed in Section 3. The main parameters set is listed in Table 1. Comparative results of passive suspension and active suspension with two different control methods (i.e., LQR method and the proposed ADP-based adaptive optimal control method) are presented in the following two different cases.Table 1
A sedan’s vehicle parameters used in simulation.
Parameter
Value/unit
Quarter-car sprung mass (ms)
250 kg
Quarter-car unsprung mass (mu)
35 kg
Suspension stiffness (ks)
15,000 N/m
Vertical stiffness of the tire (kt)
150,000 N/m
Suspension damper (bs)
450 N∗s/m
Damping coefficient of the tire (bt)
1000 N∗s/mCase 1 (with bump road displacement).
The suspension is excited by a bump road input as [11], which is given by(33)z˙rt=a21-cos⁡2πVslt,0≤t≤lVs0,t>lVs,where a and l are the height and the length of the bump, respectively, and Vs is the vehicle forward velocity. Here, we select a=0.1m, and l=5m and Vs=60km/h.For the LQR controller design, the weighing matricesQ and R have to be predetermined. The relationship between Q and R is that, for a fixed Q, a decrease in R matrix’s values will decrease the transition time and the overshoot but this action will increase the rise time and the steady state error. When R is kept fixed but Q is decreased, the transition time and overshoot will be increased but the rise time and steady state error will be decreased [19]. Based on this rule, Q and R are selected for the performance cost function (6) as R=2∗10-5I and Q=diag⁡[10,65,1.8,20]. Then, we can get the state feedback gain matrix from Matlab K=104∗[0.0166 0.5520 −5.5777 −0.2564]. The proposed ADP-based adaptive optimal control (14) with the updating laws (20) for the critic NN (12) is simulated with the following parameters: η=500, μ=1500 and the critic NN vector activation function is selected as ψx=[x12x1x2x1x3x1x4x22x2x3x2x4x32x3x4x42]T.Simulation results with two different sprung masses are presented in Figures3 and 4. Compared with the passive suspension system, the active suspension system with the LQR and the proposed ADP-based adaptive optimal control methods have the lower peak and less vibration. Meanwhile, one can see from Figures 3(a) and 4(a) that the proposed ADP-based adaptive optimal control method demonstrates smaller amplitude and faster transient convergence in terms of the suspension deflection and sprung mass acceleration than the LQR control method. Figures 3(b) and 4(b) further provide the simulation results of the suspension deflection and sprung mass acceleration with the varying sprung mass (from 250 kg to 350 kg). One may find that the proposed ADP-based adaptive optimal control method still shows improved performance compared with the LQR control method even under different sprung mass. Besides, Figure 3 also shows that the suspension working space falls into the acceptable ranges to satisfy the suspension space limitation requirement.Figure 3
The suspension deflection response.
(a)
(b)Figure 4
The sprung mass acceleration response.
(a)
(b)In order to ensure the good road holding performance, the ratio between tire dynamic load and stable load should be less than 1; that is,Ft+Fb/(ms+mu)g<1, where Ft=kt(zu-zr) and Ft=bt(z˙u-z˙r) are the elasticity force and damping force of the tires [20]. Here, the tire damping bt is assumed to be negligible. From Figure 5, one can see that the ratio between tire dynamic load and stable load is always less than 1 for both of the proposed ADP-based adaptive optimal control method and LQR control methods, which implies that the better road holding ability is guaranteed and thus ensures the stability of vehicle. Actuator output forces are presented in Figure 6, from which one can find that the proposed ADP-based adaptive optimal control method needs smaller actuator force than the LQR control approach.Figure 5
The ratio between tire dynamic load and stable load of active suspension systems.
(a)
(b)Figure 6
Control inputs.Furthermore, the performance index in terms of Root Mean Square (RMS) for the states in Figures3 and 4 is listed in Table 2. The fact that all of the RMS values of the proposed ADP-based adaptive optimal control method are smaller than the LQR control method further shows the improved performance of the proposed method.Table 2
The RMS values for the states (×10-8).
ADP
LQR
m
s = 250 kg
Suspension deflection
0.0048
1.7220
Sprung mass acceleration
0.0303
3.3670
m
s = 350 kg
Suspension deflection
0.0048
2.0800
Sprung mass acceleration
0.0239
3.9550Case 2 (with sinusoid road displacement).
In this case, a sinusoid road displacement of0.005sin⁡(5π) is used and all other simulation parameters are the same as those in Case 1. Comparative simulation results in Figures 7 and 8 further validate the improved performance of the proposed ADP-based adaptive optimal control method compared with the LQR control method. It should be pointed out that poor self-adaptive property of the LQR method leads to the small oscillation of the suspension deflection response and the sprung mass acceleration response as shown in Figure 3 and Figure 4. Meanwhile, the smaller oscillation for the proposed ADP-based adaptive optimal control method may come from the Bellman error caused by the approximation of the value function as shown in (15).
The reasons that lead to the improved performance of the proposed ADP-based adaptive optimal control method during the aforementioned simulation results are analyzed as follows.
From the LQR design process, one can see that it is based on the precise dynamic model (3) with the assumption that all the system parameters are time invariant. The optimal state feedback gain matrix is obtained by solving the Riccati equation offline and cannot be updated online according to the time varying parameters and unknown road input, respectively. So, it may influence the control performance and even lose stability. The proposed ADP-based adaptive optimal control approach provides a novel solution to the online optimal control of the uncertain system. the feedback control law of the proposed ADP-based adaptive optimal control approach can be updated online with the time varying parameters like sprung mass and unknown road displacement input. It therefore could be concluded that self-adaptive property of the proposed ADP-based optimal control method provides a more effective solution for the controller design of the active suspension systems and can greatly enhance the passenger’s comfort level and thus achieve the goals of pursuing more comfortable and safer vehicles.Figure 7
The suspension deflection response.Figure 8
The sprung mass acceleration response.


## 5. Conclusion

In this paper, an ADP-based adaptive optimal controller for active suspension systems considering the performance requirements has been proposed. Compared with the commonly used LQR method, self-adaptive property of the proposed ADP-based adaptive optimal control method has improved performance in terms of the basic tasks of suspension control like ride comfort, road holding, and suspension space limitation. This performance improvement thus will increase the passenger comfort level and at the same time enhance vehicle handling and stabilities when driving on the road. In future work, more comfortable and safer full vehicle suspension system considering the state constraints and actuator saturation limits will be designed.



---

*Source: 4575926-2017-04-05.xml*