# U-Model-Based Adaptive Sliding Mode Control Using a Deep Deterministic Policy Gradient

Authors: Changyi Lei; Quanmin Zhu
Journal: Mathematical Problems in Engineering
 (2022)

---

## Abstract

This paper presents a U-model-based adaptive sliding mode control (SMC) using a deep deterministic policy gradient (DDPG) for uncertain nonlinear systems. The configuration of the proposed methodology consisted of a U-model framework and an SMC with a variable boundary layer. The U-model framework forms the outer feedback loop that adjusts the overall performance of the nonlinear system, while SMC serves as a robust dynamic inverter that cancels the nonlinearity of the original plant. Besides, to alleviate the chattering problem while maintaining the intrinsic advantages of SMC, a DDPG network is designed to adaptively tune the boundary and switching gain. From the control perspective, this controller combines the interpretability of the U-model and the robustness of the SMC. From the deep reinforcement learning (DRL) point of view, the DDPG calculates nearly optimal parameters for SMC based on current states and maximizes its favourable features while minimizing the unfavourable ones. The simulation results of the single-pendulum system are compared with those of a U-model-based SMC optimized by the particle swarm optimization (PSO) algorithm. The comparison, as well as model visualization, demonstrates the superiority of the proposed methodology.


---

## Body

## 1. Introduction

A U-model is a generic and systematic control method that was proposed by Zhu et al. [1]. Different from other model-based and model-free control methods, it is a model-independent method in that it uses the dynamic model of the plant to design the controller, while the final performance is independent of the target plant. In doing this, the U-model provides a general routine to separate the system design and control design processes [2]. The gist of the U-model lies in designing a robust dynamic inverter that transforms the original plant into an identity matrix [2, 3]. This brings about two advantages. First, by cancelling the dynamics and nonlinearity, the overall system performance can be prescribed by a unit-negative feedback loop. Besides, the phase delay between the control and the output is eliminated, increasing the response speed of the system [4]. Due to its critical importance, the U-model has been combined with other control methods and has yielded a satisfying outcome. For examples, a U-model-based adaptive neural network [5], a U-model-based predictive control [6], a U-model-based fuzzy PID control [7], etc. However, the conventional U-controller has some drawbacks [8]. Firstly, it does not take into account disturbances and uncertainties. Second, difficulties in calculating dynamic inversion in continuous-time make it hard to be applied to continuous-time systems. Finally, the complexity of the U-control inverter depends on the target plant itself. If the plant is itself complex, then the U-model inverter is also hard to calculate. Therefore, finding a robust and simple dynamic inverter that can be concisely applied to continuous-time systems is a critical criterion for the success of the U-model.A sliding mode control (SMC) is a robust nonlinear controller. Its implementation is usually based on the Lyapunov stability theorem and is distinguished from other controllers by its discontinuity [9]. By constructing a sliding mode variable, it forces the state variables of the system to slide to equilibrium within a given trajectory. The outstanding characteristics of the SMC are robustness, quick response, and easy implementation [10]. Indeed, the SMC makes a good supplement to a U-model control, and a combination of those two methods has attained certain accomplishments [11, 12]. However, its discontinuity in rationale is afflicted by the chattering problem, which has been widely studied [13]. The chattering issue not only impairs the system’s performance but also causes damage to physical instruments. Some solutions have been proposed to alleviate the chattering problem of SMC, including the fuzzy system [14], the boundary layer method [15], the PID-based method [16], high-order SMC [17], a neural network and PSO [18], etc. Nonetheless, these approaches either compromise stability and precision or require tedious craftsmanship and expert experience.Reinforcement learning (RL) is a model-free methodology that optimizes its action on large-scale, complex problems through exploration and exploitation without explicit models [19]. Recently, with the development of deep learning, RL has been combined with deep neural networks to solve many control problems [20–22]. Actor-critic learning is one popular framework of RL. Compared with classical Q-learning [23] and deep Q-learning [24], the actor-critic (AC) framework works with continuous states and action space [25]. This is realized by using an actor network to output continuous action and a critic network to estimate the Q-value. This characteristic guarantees AC’s potential in combinatorial optimization problems [26–28]. Deep deterministic policy gradient (DDPG) is based on AC and was proposed in 2016 [29]. The appearance of DDPG enables the direct estimation of continuous action output in the RL realm. As its name suggests, the DDPG outputs a deterministic policy to the agent, with random noise added for exploration. The DDPG has been successfully implemented in many control scenarios [30–32].Based on the above discussion, the U-model framework has the potential to bridge the gap between linear and nonlinear systems, provided a robust dynamic inverter can be designed. An SMC is a special nonlinear control scheme that is highly regarded for its robustness. Although a combination of the U-model and SMC has been implemented and has yielded certain success, the chattering problem of SMC still an urgent need to be solved. Therefore, figuring out how to ease the chattering of SMC while maintaining its deserved performance is still challenging. In this paper, we propose a U-model-based adaptive SMC tuned by DDPG to tackle this problem. The parameter tuning problem of SMC is modeled as a combinatorial optimization problem, which is to be solved by DDPG. During the training phase, the DDPG undertakes exploration and exploitation to learn optimal action based on the current state automatically. Through penalizing the tracking error and DDPG output, the neuro network tries to minimize the error with minimal cost. Thus, the proposed adaptive SMC can attenuate the chattering issue without loss of stability or precision. Besides, this method does not require an estimation of the upper bound of the overall disturbance.The contribution of this paper can be summarized as follows:(1)
An adaptive SMC with variable thickness of the boundary layer, implemented as the dynamic inverter, based on DDPG, is proposed.(2)
An SMC optimized by the PSO algorithm is provided as the baseline for comparison.(3)
A nonlinear single-pendulum environment revised from Gym [33] is provided for simulation.(4)
The simulation tests are conducted to illustrate the advantages and rationality of the proposed method.(5)
Explainable artificial intelligence (XAI) methods are implemented to explain the trained DDPG model.The rest of this paper is organised as follows. Section 2 gives some preliminaries about the U-model framework and DDPG. Section 3 articulates the details of controller calculation step by step, including a conventional SMC, variable thickness of the boundary layer, a DDPG network, and an invariant controller. Section 4 presents the simulation results and analysis of two different target trajectories. The analysis focuses on settling time, accuracy, and chattering suppression. In addition, output visualization and the SHAP method are implemented to better understand the trained network. Section 5 gives a brief conclusion as well as future work suggestions.


## 2. Preliminaries

### 2.1. U-Model

#### 2.1.1. U-Model Control Framework

Considering a general U-model control framework as shown in Figure1. R is the reference signal; Y is the output vector; E=R−Y is the error vector. The middle part of the pathway is composed of the invariant controller GI, the dynamic inverter Gp−1, and the target plant Gp. The prioritized task of the U-model control is to design a robust dynamic inverter Gp−1 that cancels the dynamics of Gp. In other words, if there exists Gp−1 such that Gp−1Gp=I, and I is a unit matrix, then the overall system performance is only determined by the invariant controller. Assume the desired transfer function Gideal, then the invariant controller can be derived as GI=Gideal/1−Gideal. Therefore, the implementation of the U-model-based control framework enables the assignment of system performance using linear system theory regardless of the nonlinearity of the target plant [34].Figure 1
A general U-model control framework.


#### 2.1.2. General U-Model Expression

A SISO CT polynomial dynamic system can be expressed as follows[35]:(1)yMt=∑j=0JλjYM−1,UN−1,Θ,tuNtj,M>N,where yt∈ℝ is the output, ut∈ℝ is the input, and t is the time. yMt and uNt are the Mth and Nth orders of derivative of y and u, respectively. λjt∈ℝ is a variable that concludes all YM−1=yM−1t,…,yt∈ℝM, UN−1=uN−1t,…,ut∈ℝN and Θ. Θ contains all scalar coefficients. Throughout the study, it is assumed that the polynomial systems are strictly proper M>N, which guarantees the causality of the systems. Accordingly, for linear polynomial systems, M>N indicates when (1) is converted to its Laplace transform, the denominator Laplace polynomial has higher order than the numerator in the resultant transfer function.Extend (1) to MIMO expression is given as follows:(2)YMt=∑j=0JΛjtUNtj,where YM=y1m1t,y2m2t,…,yamatT is a vector containing all outputs.UN=u1n1tj1,u2n2tj2,…,ubnbtjbT is an input vector with the power j of all inputs. m∗ is the order of derivative of y∗ that is directly related to u, when u∗ has the derivative order n∗. Λjt∈ℝa×b is now a matrix, instead of λj being a scalar. For simplicity and without loss of generality, we will omit the dependent variables.Consider a generalised MIMO continuous-time state-space model expression given as follows:(3)X˙t=FXt,Ut,Yt=HXt,where Y∈ℝa is the output, U∈ℝb is the input, and X∈ℝn. F is the dynamics of the system that updates the state variables, and H calculates the system output. Extend it to MIMO state-space expression [7].(4)x˙1=∑i=0nλ1if1ix2,x˙2=∑i=0nλ2if2ix3,…x˙n=∑i=0nλnifniu1,u2,…,uby1=h1x1,x2,…,xn,y2=h2x1,x2,…,xn,…ya=hax1,x2,…,xn,,where λmi and fmi,0≤i≤n,0≤m≤n are time-varying parameters, and hl,1≤l≤a is a smooth mapping from state vector to a specific output. Take the following system as an example:(5)x˙1=x1+x1x2,x˙2=−x1+u1+x1u2,y1=2x1,y2=3x2..Convert it to a U-model expression based on the following absorbing rule:(6)x˙1=λ10+λ11f11x2,x˙2=λ20+λ21f21u1,u2,y1=h1x1,x2,y2=h2x1,x2,where λ10=x1,λ11=x1,λ20=−x1,λ21=1,f11x2=x2,f21u=u1+x1u2, h1x1,x2=2x1,h2x1,x2=3x2.


#### 2.1.3. U-Model Dynamic Inversion

Using the following equation, the U-model dynamic inversion is calculated as follows[36]:(7)Gp−1⇔U∈YdMt−∑j=0JΛjtUNtj=Oa,where Ydt is the desired output vector and Oa is a a×1 null vector. The prerequisites for the solution to exist are external stability and the nonminimum phase of the system. Reconsidering (4), apply derivative to the pth0≤p≤a output yp with respect to xi:(8)y˙p=∑i=0nhp′xix˙i.Replacex˙i with a polynomial equation and we have(9)y˙p=∑i=0n∑j=0nhp′xiλijfijxj.Repeating the above derivative and replacement procedures formp times. Rearrange the equation and combine similar terms as follows:(10)ypmp=∑i=1n−1PipX,tx˙i+PnpX,tx˙n,where X is the vector of state variables, and P is a function of X. Consequently, we need to solve an equation set of (10) to retrieve the final control.


### 2.2. DDPG

Deep deterministic policy gradient (DDPG) is an off-policy algorithm proposed in 2014 by Silver et al. [37]. DDPG is based on the actor-critic framework, which learns an action network and a Q network simultaneously. For every step, given current states, the actor network outputs the policy added with random noise. By executing the policy, the model receives a reward from the environment, and the policy network is updated using the temporal difference (TD) algorithm accordingly. The Q-value output by the critic network in turn guides the update of the actor network using the policy gradient algorithm [37]. To increase the stability of learning, a target actor network and a target critic network are added. They are updated through a soft update, namely a weighted average of the actor or critic networks and themselves. Besides, a replay buffer storing transition states and actions is implemented to increase the efficiency of sample utilization. Because of this double network configuration, DDPG can deal with situations when the action space and state space are all continuous [29]. The pseudo code of DDPG is shown in Algorithm 1.Algorithm 1: Deep deterministic policy gradient.
(1)
Initialize policy networkπϕs, critic network Qθs,a and empty replay buffer D(2)
Set target policy networkπϕ′′s and target critic network Qθ′′s,a, with ϕ′←ϕ, θ′←θ(3)
repeat(4)
Observe states and execute action a = clip πϕs+ε,alow,ahigh, where ε∼N(5)
Observe next states’, reward r, and done signal d to indicate whether s’ is terminal(6)
Store (s, a, r, s’, d) in the replay buffer D(7)
Ifs’ is terminal, reset environment state(8)
if it is time to update then(9)
for the number of updates do(10)
Randomly sample a batch of transitions,B = (s, a, r, s’, d) from D(11)
Compute targetsyr,s′,d=r+γ1−dQθ′s′,πϕs′(12)
UpdateQ-function by one step of gradient descent using ▽ϕ1/B∑s,a,r,s′,d∈BQθs,a−yr,s′,d2(13)
Update policy by one step of gradient ascent using▽ϕ1/B∑s∈BQθs,πϕs(14)
Update target networks withθ′←ρθ′+1−ρθϕ′←ρϕ′+1−ρϕ(15)
end for(16)
end if(17)
until convergence


## 2.1. U-Model

### 2.1.1. U-Model Control Framework

Considering a general U-model control framework as shown in Figure1. R is the reference signal; Y is the output vector; E=R−Y is the error vector. The middle part of the pathway is composed of the invariant controller GI, the dynamic inverter Gp−1, and the target plant Gp. The prioritized task of the U-model control is to design a robust dynamic inverter Gp−1 that cancels the dynamics of Gp. In other words, if there exists Gp−1 such that Gp−1Gp=I, and I is a unit matrix, then the overall system performance is only determined by the invariant controller. Assume the desired transfer function Gideal, then the invariant controller can be derived as GI=Gideal/1−Gideal. Therefore, the implementation of the U-model-based control framework enables the assignment of system performance using linear system theory regardless of the nonlinearity of the target plant [34].Figure 1
A general U-model control framework.


### 2.1.2. General U-Model Expression

A SISO CT polynomial dynamic system can be expressed as follows[35]:(1)yMt=∑j=0JλjYM−1,UN−1,Θ,tuNtj,M>N,where yt∈ℝ is the output, ut∈ℝ is the input, and t is the time. yMt and uNt are the Mth and Nth orders of derivative of y and u, respectively. λjt∈ℝ is a variable that concludes all YM−1=yM−1t,…,yt∈ℝM, UN−1=uN−1t,…,ut∈ℝN and Θ. Θ contains all scalar coefficients. Throughout the study, it is assumed that the polynomial systems are strictly proper M>N, which guarantees the causality of the systems. Accordingly, for linear polynomial systems, M>N indicates when (1) is converted to its Laplace transform, the denominator Laplace polynomial has higher order than the numerator in the resultant transfer function.Extend (1) to MIMO expression is given as follows:(2)YMt=∑j=0JΛjtUNtj,where YM=y1m1t,y2m2t,…,yamatT is a vector containing all outputs.UN=u1n1tj1,u2n2tj2,…,ubnbtjbT is an input vector with the power j of all inputs. m∗ is the order of derivative of y∗ that is directly related to u, when u∗ has the derivative order n∗. Λjt∈ℝa×b is now a matrix, instead of λj being a scalar. For simplicity and without loss of generality, we will omit the dependent variables.Consider a generalised MIMO continuous-time state-space model expression given as follows:(3)X˙t=FXt,Ut,Yt=HXt,where Y∈ℝa is the output, U∈ℝb is the input, and X∈ℝn. F is the dynamics of the system that updates the state variables, and H calculates the system output. Extend it to MIMO state-space expression [7].(4)x˙1=∑i=0nλ1if1ix2,x˙2=∑i=0nλ2if2ix3,…x˙n=∑i=0nλnifniu1,u2,…,uby1=h1x1,x2,…,xn,y2=h2x1,x2,…,xn,…ya=hax1,x2,…,xn,,where λmi and fmi,0≤i≤n,0≤m≤n are time-varying parameters, and hl,1≤l≤a is a smooth mapping from state vector to a specific output. Take the following system as an example:(5)x˙1=x1+x1x2,x˙2=−x1+u1+x1u2,y1=2x1,y2=3x2..Convert it to a U-model expression based on the following absorbing rule:(6)x˙1=λ10+λ11f11x2,x˙2=λ20+λ21f21u1,u2,y1=h1x1,x2,y2=h2x1,x2,where λ10=x1,λ11=x1,λ20=−x1,λ21=1,f11x2=x2,f21u=u1+x1u2, h1x1,x2=2x1,h2x1,x2=3x2.


### 2.1.3. U-Model Dynamic Inversion

Using the following equation, the U-model dynamic inversion is calculated as follows[36]:(7)Gp−1⇔U∈YdMt−∑j=0JΛjtUNtj=Oa,where Ydt is the desired output vector and Oa is a a×1 null vector. The prerequisites for the solution to exist are external stability and the nonminimum phase of the system. Reconsidering (4), apply derivative to the pth0≤p≤a output yp with respect to xi:(8)y˙p=∑i=0nhp′xix˙i.Replacex˙i with a polynomial equation and we have(9)y˙p=∑i=0n∑j=0nhp′xiλijfijxj.Repeating the above derivative and replacement procedures formp times. Rearrange the equation and combine similar terms as follows:(10)ypmp=∑i=1n−1PipX,tx˙i+PnpX,tx˙n,where X is the vector of state variables, and P is a function of X. Consequently, we need to solve an equation set of (10) to retrieve the final control.


## 2.1.1. U-Model Control Framework

Considering a general U-model control framework as shown in Figure1. R is the reference signal; Y is the output vector; E=R−Y is the error vector. The middle part of the pathway is composed of the invariant controller GI, the dynamic inverter Gp−1, and the target plant Gp. The prioritized task of the U-model control is to design a robust dynamic inverter Gp−1 that cancels the dynamics of Gp. In other words, if there exists Gp−1 such that Gp−1Gp=I, and I is a unit matrix, then the overall system performance is only determined by the invariant controller. Assume the desired transfer function Gideal, then the invariant controller can be derived as GI=Gideal/1−Gideal. Therefore, the implementation of the U-model-based control framework enables the assignment of system performance using linear system theory regardless of the nonlinearity of the target plant [34].Figure 1
A general U-model control framework.


## 2.1.2. General U-Model Expression

A SISO CT polynomial dynamic system can be expressed as follows[35]:(1)yMt=∑j=0JλjYM−1,UN−1,Θ,tuNtj,M>N,where yt∈ℝ is the output, ut∈ℝ is the input, and t is the time. yMt and uNt are the Mth and Nth orders of derivative of y and u, respectively. λjt∈ℝ is a variable that concludes all YM−1=yM−1t,…,yt∈ℝM, UN−1=uN−1t,…,ut∈ℝN and Θ. Θ contains all scalar coefficients. Throughout the study, it is assumed that the polynomial systems are strictly proper M>N, which guarantees the causality of the systems. Accordingly, for linear polynomial systems, M>N indicates when (1) is converted to its Laplace transform, the denominator Laplace polynomial has higher order than the numerator in the resultant transfer function.Extend (1) to MIMO expression is given as follows:(2)YMt=∑j=0JΛjtUNtj,where YM=y1m1t,y2m2t,…,yamatT is a vector containing all outputs.UN=u1n1tj1,u2n2tj2,…,ubnbtjbT is an input vector with the power j of all inputs. m∗ is the order of derivative of y∗ that is directly related to u, when u∗ has the derivative order n∗. Λjt∈ℝa×b is now a matrix, instead of λj being a scalar. For simplicity and without loss of generality, we will omit the dependent variables.Consider a generalised MIMO continuous-time state-space model expression given as follows:(3)X˙t=FXt,Ut,Yt=HXt,where Y∈ℝa is the output, U∈ℝb is the input, and X∈ℝn. F is the dynamics of the system that updates the state variables, and H calculates the system output. Extend it to MIMO state-space expression [7].(4)x˙1=∑i=0nλ1if1ix2,x˙2=∑i=0nλ2if2ix3,…x˙n=∑i=0nλnifniu1,u2,…,uby1=h1x1,x2,…,xn,y2=h2x1,x2,…,xn,…ya=hax1,x2,…,xn,,where λmi and fmi,0≤i≤n,0≤m≤n are time-varying parameters, and hl,1≤l≤a is a smooth mapping from state vector to a specific output. Take the following system as an example:(5)x˙1=x1+x1x2,x˙2=−x1+u1+x1u2,y1=2x1,y2=3x2..Convert it to a U-model expression based on the following absorbing rule:(6)x˙1=λ10+λ11f11x2,x˙2=λ20+λ21f21u1,u2,y1=h1x1,x2,y2=h2x1,x2,where λ10=x1,λ11=x1,λ20=−x1,λ21=1,f11x2=x2,f21u=u1+x1u2, h1x1,x2=2x1,h2x1,x2=3x2.


## 2.1.3. U-Model Dynamic Inversion

Using the following equation, the U-model dynamic inversion is calculated as follows[36]:(7)Gp−1⇔U∈YdMt−∑j=0JΛjtUNtj=Oa,where Ydt is the desired output vector and Oa is a a×1 null vector. The prerequisites for the solution to exist are external stability and the nonminimum phase of the system. Reconsidering (4), apply derivative to the pth0≤p≤a output yp with respect to xi:(8)y˙p=∑i=0nhp′xix˙i.Replacex˙i with a polynomial equation and we have(9)y˙p=∑i=0n∑j=0nhp′xiλijfijxj.Repeating the above derivative and replacement procedures formp times. Rearrange the equation and combine similar terms as follows:(10)ypmp=∑i=1n−1PipX,tx˙i+PnpX,tx˙n,where X is the vector of state variables, and P is a function of X. Consequently, we need to solve an equation set of (10) to retrieve the final control.


## 2.2. DDPG

Deep deterministic policy gradient (DDPG) is an off-policy algorithm proposed in 2014 by Silver et al. [37]. DDPG is based on the actor-critic framework, which learns an action network and a Q network simultaneously. For every step, given current states, the actor network outputs the policy added with random noise. By executing the policy, the model receives a reward from the environment, and the policy network is updated using the temporal difference (TD) algorithm accordingly. The Q-value output by the critic network in turn guides the update of the actor network using the policy gradient algorithm [37]. To increase the stability of learning, a target actor network and a target critic network are added. They are updated through a soft update, namely a weighted average of the actor or critic networks and themselves. Besides, a replay buffer storing transition states and actions is implemented to increase the efficiency of sample utilization. Because of this double network configuration, DDPG can deal with situations when the action space and state space are all continuous [29]. The pseudo code of DDPG is shown in Algorithm 1.Algorithm 1: Deep deterministic policy gradient.
(1)
Initialize policy networkπϕs, critic network Qθs,a and empty replay buffer D(2)
Set target policy networkπϕ′′s and target critic network Qθ′′s,a, with ϕ′←ϕ, θ′←θ(3)
repeat(4)
Observe states and execute action a = clip πϕs+ε,alow,ahigh, where ε∼N(5)
Observe next states’, reward r, and done signal d to indicate whether s’ is terminal(6)
Store (s, a, r, s’, d) in the replay buffer D(7)
Ifs’ is terminal, reset environment state(8)
if it is time to update then(9)
for the number of updates do(10)
Randomly sample a batch of transitions,B = (s, a, r, s’, d) from D(11)
Compute targetsyr,s′,d=r+γ1−dQθ′s′,πϕs′(12)
UpdateQ-function by one step of gradient descent using ▽ϕ1/B∑s,a,r,s′,d∈BQθs,a−yr,s′,d2(13)
Update policy by one step of gradient ascent using▽ϕ1/B∑s∈BQθs,πϕs(14)
Update target networks withθ′←ρθ′+1−ρθϕ′←ρϕ′+1−ρϕ(15)
end for(16)
end if(17)
until convergence


## 3. Controller Design

This section describes the details of the conceptual framework of the controller, which will be implemented in the following simulation. The combined controller is composed of a U-model controller, a sliding mode controller with a variable boundary layer, and a DDPG network.


### 3.1. Framework Overview

Figure2 illustrates the workflow of the proposed methodology. R∈ℝN represents the reference signal, where N is the degree of freedom (DoF) of the system. Y∈ℝN is the output vector, and E=R−Y represents the error vector. GI is the invariant controller, and Gp is the original target dynamic plant. Between them is the dynamic inverter performed by a sliding mode controller, and it outputs the control vector U∈ℝN. The parameters of the SMC are calculated by the DDPG module for every time step. Based on the gist of the U-model, if the dynamic inverter GSM−1 successfully performs dynamic inversion regardless of uncertainty, then the nonlinearity of the plant is cancelled, which means GSM−1Gp=IN. In this case, the system is equivalent to a unit negative feedback loop, of which the performance is assigned by the transfer function of GI. In the following part of this section, the details of SMC, DDPG, and GI will be introduced.Figure 2
Controller framework overview.


### 3.2. Sliding Mode Control with Variable Boundary Layer

In this section, we consider a general second-order dynamic system expression, based on which the SMC with variable boundary layer is designed using backstepping and the Lyapunov stability theorem.


#### 3.2.1. Conventional Sliding Mode Control with Backstepping

For simplicity and without loss of generality, consider the following single-input-single-output (SISO) dynamic model:(11)x˙1t=x2t,x˙2t=fx1,x2,t+hx1,x2,tut+dt,where x1,x2 are state variables, ut is the control input vector, and dt is the overall disturbance vector. fx1,x2,t and hx1,x2,t are time-varying functions dependent on the state variables. For simplicity, the expression afterwards will ignore all dependent variables.Assumption 1.
The overall disturbancedt is bounded and satisfies dt<D, in which D is a positively finite scalar.Remark 1.
In practice, the disturbance is often closely related to the state variables of the system. For example, viscous friction is a function of velocity. In practice, the state variables usually have bounds, so the total disturbance is usually bounded. Similar assumptions have been made in many control theory scenarios [38, 39].
The following is the construct virtual control variables:(12)z1=x1−zd,z2=x2+c1z1−z˙d,where zd,z˙d forms the reference signal. c1 is a constant positive value, and z1,z2 are virtual control variables. Construct the first partial Lyapunov function is given as follows:(13)V1=12z12.
Take the derivative of (13) and integrate with (12), we have(14)V˙1=z1z˙1=−c1z12+z1z2.
According to the Lyapunov stability theorem, sinceV1≥0, if z2=0, then V˙1<=0, and that z1 can converge to the equilibrium asymptotically. Regarding subsystem of x1,z2, design a second partial Lyapunov function as follows:(15)V2=V1+12z22.
Take the derivative of (15) and integrate with (11)–(14), we have(16)V˙2=−c1z12+z1z2+z2hu+f+c1z˙1−z¨d+d.
Selectz2 as the sliding mode variable and design control input as(17)u=1h−ηsgnz2−f−c1z˙1+z¨d−c2z2−z1,where sgn is the sign function. η>D is a positive scalar, and is called the switching gain.


#### 3.2.2. Stability Analysis

Theorem 1.
For a general second-order dynamic system as described in (11) and implementing a controller in (17), the system has asymptotic stability in the Lyapunov sense.Proof.
Integrating (17) into (16), we have(18)V˙2=−c1z12−c2z22−ηz2+z2d.
According to Assumption 1,z2d<ηz2, so V˙2<0 and V2≥0. Based on the Lyapunov asymptotic stability theorem [40], z1 and z2 will converge to equilibrium asymptotically. Therefore, it means that state variables x1 and x2 will follow the desired trajectory.


#### 3.2.3. Adding Variable Boundary Layer

The switching gainη plays a critical role in determining the performance of the SMC. If η is large, the system will converge quickly, but the chattering problem is also exacerbated because of the discontinuity of the sign function. To alleviate the chattering issue, the sign function is replaced with a saturation function as follows [10]:(19)sats,δ=1,s>δ,1δ,s<δ,−1s<−δ,where s is the sliding mode variable and δ is the thickness of the boundary layer. Thus, (17) becomes(20)u=1h−ηsatz2,δ−f−c1z˙1+z¨d−c2z2−z1.The introduction of the boundary layer constructs a space where the controller outputs a continuous torque so that the system trajectory can be smoother. However, the alleviation of chattering comes at the cost of lowered control accuracy. Figuring out how to select the optimal pair ofη,δ largely relies on the human experience.


### 3.3. DDPG Network Module

The purpose of introducing the DDPG module is to adaptively select the optimal pair ofη,δ for the sliding mode controller so that, when the error is large, the system will converge quickly; whereas when the system is near equilibrium, the chattering can be alleviated while ensuring the same level of accuracy. The DDPG belongs to the actor-critic framework of RL and, therefore, can deal with situations when state and action are all continuous values. In this paper, instead of directly outputting η,δT, it outputs 1/η=tanq,δT, where q∈0,π/2 is the inclination of the saturation function, as shown in Figure 19.Figure 3
Diagram of saturation function.The actor and the critic network are constructed as fully connected neural networks. The details of the network structure are shown in Figure4 and 5. The state is a 4×1 vector x,x˙,e_x,e_x˙T, where x,x˙ are state variables and e_x,e_x˙ are corresponding errors. The output of the actor network is a 2×1 vector u1,u2T=1/η,δT. The action ranges of those two outputs are set to 0,20. The actor network takes the state vector as the input, and then it goes through three fully connected hidden layers with 64 units of nodes. The activation functions of the first two layers are the Rectified Linear Unit (ReLu) function, and the last layer uses the Sigmoid function to map every dimension of the output to 0,1. The critic network takes the concatenation of state and action as the input. It also uses several fully connected layers to process the information, and then outputs a scalar, which is also called a state-action value Qstate,action. This is an estimate of how well the action is given the current state.Figure 4
Structure of an actor network.Figure 5
Structure of a critic network.


### 3.4. Invariant Controller

An invariant controller is utilized in the U-model to assign system performance using linear system techniques. It can also be viewd as the implementation of a smooth transition process. Ideally, we hope the system can converge to equilibrium without oscillation or overshoot. Luckily, a critically-damped second-order differential equation meets our requirements and is designed as the invariant controller. The ideal closed-loop system transfer function can be written as follows:(21)Gs=ωn2s2+2ξωns+ωn2,where ξ is the damping ratio and ωn is the natural frequency. They will be designed according to the required stability error and settling time. Also, the invariant controller can be calculated as follow:(22)GIs=Gs1−Gs=ωn2s2+2ξωns.


## 3.1. Framework Overview

Figure2 illustrates the workflow of the proposed methodology. R∈ℝN represents the reference signal, where N is the degree of freedom (DoF) of the system. Y∈ℝN is the output vector, and E=R−Y represents the error vector. GI is the invariant controller, and Gp is the original target dynamic plant. Between them is the dynamic inverter performed by a sliding mode controller, and it outputs the control vector U∈ℝN. The parameters of the SMC are calculated by the DDPG module for every time step. Based on the gist of the U-model, if the dynamic inverter GSM−1 successfully performs dynamic inversion regardless of uncertainty, then the nonlinearity of the plant is cancelled, which means GSM−1Gp=IN. In this case, the system is equivalent to a unit negative feedback loop, of which the performance is assigned by the transfer function of GI. In the following part of this section, the details of SMC, DDPG, and GI will be introduced.Figure 2
Controller framework overview.


## 3.2. Sliding Mode Control with Variable Boundary Layer

In this section, we consider a general second-order dynamic system expression, based on which the SMC with variable boundary layer is designed using backstepping and the Lyapunov stability theorem.


### 3.2.1. Conventional Sliding Mode Control with Backstepping

For simplicity and without loss of generality, consider the following single-input-single-output (SISO) dynamic model:(11)x˙1t=x2t,x˙2t=fx1,x2,t+hx1,x2,tut+dt,where x1,x2 are state variables, ut is the control input vector, and dt is the overall disturbance vector. fx1,x2,t and hx1,x2,t are time-varying functions dependent on the state variables. For simplicity, the expression afterwards will ignore all dependent variables.Assumption 1.
The overall disturbancedt is bounded and satisfies dt<D, in which D is a positively finite scalar.Remark 1.
In practice, the disturbance is often closely related to the state variables of the system. For example, viscous friction is a function of velocity. In practice, the state variables usually have bounds, so the total disturbance is usually bounded. Similar assumptions have been made in many control theory scenarios [38, 39].
The following is the construct virtual control variables:(12)z1=x1−zd,z2=x2+c1z1−z˙d,where zd,z˙d forms the reference signal. c1 is a constant positive value, and z1,z2 are virtual control variables. Construct the first partial Lyapunov function is given as follows:(13)V1=12z12.
Take the derivative of (13) and integrate with (12), we have(14)V˙1=z1z˙1=−c1z12+z1z2.
According to the Lyapunov stability theorem, sinceV1≥0, if z2=0, then V˙1<=0, and that z1 can converge to the equilibrium asymptotically. Regarding subsystem of x1,z2, design a second partial Lyapunov function as follows:(15)V2=V1+12z22.
Take the derivative of (15) and integrate with (11)–(14), we have(16)V˙2=−c1z12+z1z2+z2hu+f+c1z˙1−z¨d+d.
Selectz2 as the sliding mode variable and design control input as(17)u=1h−ηsgnz2−f−c1z˙1+z¨d−c2z2−z1,where sgn is the sign function. η>D is a positive scalar, and is called the switching gain.


### 3.2.2. Stability Analysis

Theorem 1.
For a general second-order dynamic system as described in (11) and implementing a controller in (17), the system has asymptotic stability in the Lyapunov sense.Proof.
Integrating (17) into (16), we have(18)V˙2=−c1z12−c2z22−ηz2+z2d.
According to Assumption 1,z2d<ηz2, so V˙2<0 and V2≥0. Based on the Lyapunov asymptotic stability theorem [40], z1 and z2 will converge to equilibrium asymptotically. Therefore, it means that state variables x1 and x2 will follow the desired trajectory.


### 3.2.3. Adding Variable Boundary Layer

The switching gainη plays a critical role in determining the performance of the SMC. If η is large, the system will converge quickly, but the chattering problem is also exacerbated because of the discontinuity of the sign function. To alleviate the chattering issue, the sign function is replaced with a saturation function as follows [10]:(19)sats,δ=1,s>δ,1δ,s<δ,−1s<−δ,where s is the sliding mode variable and δ is the thickness of the boundary layer. Thus, (17) becomes(20)u=1h−ηsatz2,δ−f−c1z˙1+z¨d−c2z2−z1.The introduction of the boundary layer constructs a space where the controller outputs a continuous torque so that the system trajectory can be smoother. However, the alleviation of chattering comes at the cost of lowered control accuracy. Figuring out how to select the optimal pair ofη,δ largely relies on the human experience.


## 3.2.1. Conventional Sliding Mode Control with Backstepping

For simplicity and without loss of generality, consider the following single-input-single-output (SISO) dynamic model:(11)x˙1t=x2t,x˙2t=fx1,x2,t+hx1,x2,tut+dt,where x1,x2 are state variables, ut is the control input vector, and dt is the overall disturbance vector. fx1,x2,t and hx1,x2,t are time-varying functions dependent on the state variables. For simplicity, the expression afterwards will ignore all dependent variables.Assumption 1.
The overall disturbancedt is bounded and satisfies dt<D, in which D is a positively finite scalar.Remark 1.
In practice, the disturbance is often closely related to the state variables of the system. For example, viscous friction is a function of velocity. In practice, the state variables usually have bounds, so the total disturbance is usually bounded. Similar assumptions have been made in many control theory scenarios [38, 39].
The following is the construct virtual control variables:(12)z1=x1−zd,z2=x2+c1z1−z˙d,where zd,z˙d forms the reference signal. c1 is a constant positive value, and z1,z2 are virtual control variables. Construct the first partial Lyapunov function is given as follows:(13)V1=12z12.
Take the derivative of (13) and integrate with (12), we have(14)V˙1=z1z˙1=−c1z12+z1z2.
According to the Lyapunov stability theorem, sinceV1≥0, if z2=0, then V˙1<=0, and that z1 can converge to the equilibrium asymptotically. Regarding subsystem of x1,z2, design a second partial Lyapunov function as follows:(15)V2=V1+12z22.
Take the derivative of (15) and integrate with (11)–(14), we have(16)V˙2=−c1z12+z1z2+z2hu+f+c1z˙1−z¨d+d.
Selectz2 as the sliding mode variable and design control input as(17)u=1h−ηsgnz2−f−c1z˙1+z¨d−c2z2−z1,where sgn is the sign function. η>D is a positive scalar, and is called the switching gain.


## 3.2.2. Stability Analysis

Theorem 1.
For a general second-order dynamic system as described in (11) and implementing a controller in (17), the system has asymptotic stability in the Lyapunov sense.Proof.
Integrating (17) into (16), we have(18)V˙2=−c1z12−c2z22−ηz2+z2d.
According to Assumption 1,z2d<ηz2, so V˙2<0 and V2≥0. Based on the Lyapunov asymptotic stability theorem [40], z1 and z2 will converge to equilibrium asymptotically. Therefore, it means that state variables x1 and x2 will follow the desired trajectory.


## 3.2.3. Adding Variable Boundary Layer

The switching gainη plays a critical role in determining the performance of the SMC. If η is large, the system will converge quickly, but the chattering problem is also exacerbated because of the discontinuity of the sign function. To alleviate the chattering issue, the sign function is replaced with a saturation function as follows [10]:(19)sats,δ=1,s>δ,1δ,s<δ,−1s<−δ,where s is the sliding mode variable and δ is the thickness of the boundary layer. Thus, (17) becomes(20)u=1h−ηsatz2,δ−f−c1z˙1+z¨d−c2z2−z1.The introduction of the boundary layer constructs a space where the controller outputs a continuous torque so that the system trajectory can be smoother. However, the alleviation of chattering comes at the cost of lowered control accuracy. Figuring out how to select the optimal pair ofη,δ largely relies on the human experience.


## 3.3. DDPG Network Module

The purpose of introducing the DDPG module is to adaptively select the optimal pair ofη,δ for the sliding mode controller so that, when the error is large, the system will converge quickly; whereas when the system is near equilibrium, the chattering can be alleviated while ensuring the same level of accuracy. The DDPG belongs to the actor-critic framework of RL and, therefore, can deal with situations when state and action are all continuous values. In this paper, instead of directly outputting η,δT, it outputs 1/η=tanq,δT, where q∈0,π/2 is the inclination of the saturation function, as shown in Figure 19.Figure 3
Diagram of saturation function.The actor and the critic network are constructed as fully connected neural networks. The details of the network structure are shown in Figure4 and 5. The state is a 4×1 vector x,x˙,e_x,e_x˙T, where x,x˙ are state variables and e_x,e_x˙ are corresponding errors. The output of the actor network is a 2×1 vector u1,u2T=1/η,δT. The action ranges of those two outputs are set to 0,20. The actor network takes the state vector as the input, and then it goes through three fully connected hidden layers with 64 units of nodes. The activation functions of the first two layers are the Rectified Linear Unit (ReLu) function, and the last layer uses the Sigmoid function to map every dimension of the output to 0,1. The critic network takes the concatenation of state and action as the input. It also uses several fully connected layers to process the information, and then outputs a scalar, which is also called a state-action value Qstate,action. This is an estimate of how well the action is given the current state.Figure 4
Structure of an actor network.Figure 5
Structure of a critic network.


## 3.4. Invariant Controller

An invariant controller is utilized in the U-model to assign system performance using linear system techniques. It can also be viewd as the implementation of a smooth transition process. Ideally, we hope the system can converge to equilibrium without oscillation or overshoot. Luckily, a critically-damped second-order differential equation meets our requirements and is designed as the invariant controller. The ideal closed-loop system transfer function can be written as follows:(21)Gs=ωn2s2+2ξωns+ωn2,where ξ is the damping ratio and ωn is the natural frequency. They will be designed according to the required stability error and settling time. Also, the invariant controller can be calculated as follow:(22)GIs=Gs1−Gs=ωn2s2+2ξωns.


## 4. Simulation

### 4.1. Dynamic Model Establishment

The simulation is implemented on a single pendulum. The structure graph is shown in Figure6.Figure 6
Structural graph of a single-pendulum.θ is the angle of the pendulum, l is the length, m is the weight, and g is the gravity coefficient. For simplicity and without loss of generality, we assume the mass of the pendulum is concentrated at the end of the link, and therefore, the dynamic equation can be derived as follows:(23)θ..=1ml2τ−bθ.−mglsinθ+d,where θ.. is the angular acceleration, θ. is the angular velocity, τ is the control torque, d is the disturbance, and b is the damping coefficient. Comparing (23) with (11), we have(24)f=−1ml2bθ.+mglsinθ,h=1ml2.Integrating (24) with (20), we have the sliding mode controller equation for a single-pendulum dynamics as follows:(25)τ=ml2−ηsatz2,δ−c1z˙1+z¨d−c2z2−z1−bθ.+mglsinθ.


### 4.2. DDPG Training Procedure

The training of DDPG follows the procedure of Algorithm 1, and the training-related parameters are shown in Table1. The network is trained for 5000 episodes, with 200-time steps for each episode. Due to the limited computational resources, the sampling time during training is set to 0.05s. The initial position of the pendulum is randomly set to be between −π,πT rad, and the initial velocity is between −10,10T rad/s. The target position, velocity, and acceleration of training are all set to 0. We later verify that, although the target of training is simplistic, the model can learn an effective policy that generalizes to time-varying trajectories (e.g., a sine wave). The state transition reward is designed as follows:(26)r=−r1+r2+r3,r1=2×e_θ+0.1×e_θ.2,r2=0.06×arctanu1,r3=0.06×u2..Table 1
Parameter of controllers.
ControllerParametersSMC controllerc1=2,c2=2DDPGlra=1e−6,lrc=2e−6,ρ=0.01,γ=0.9U-modelξ=1,ωn=5PSOa1=0.5,a2=0.5,N=40,ω=0.8The reward is composed of three parts.r1 takes a similar format as the sliding mode variable. An absolute operation is implemented on the angular error and velocity error, which implies a stricter punishment than the sliding mode variable because it requires both of them to converge to 0 simultaneously. Furthermore, there are slight adjustments to the weighting of the two terms. Since the priority goal is to track the desired position, we hope the reward can guide the model to pay more attention to angular error than velocity error. r2 calculates the inclination of the saturation function. r2 and r3 penalizes the interference of DDPG with the SMC controller. The final constant coefficients are determined through experiments. Intuitively, the model is instructed not to output unnecessarily large parameters to the SMC controller, and so that prevents the chattering issues. Figure 7 illustrates the episode reward during training. We can see that with training, the network gradually finds a nearly optimal policy and optimizes the episode reward.Figure 7
Training reward of DDPG.


### 4.3. PSO Tuning Procedure

PSO is a meta-heuristic global optimization method. It treats the model as a black box and tries to find optimal solutions through inputs and outputs. The update rule of PSO borrows ideas from swarm intelligence. Through interacting with each other, all particles strike a balance between exploration and exploitation. PSO has been proven to be simple in rationale but effective in practice. In this paper, we use PSO to find the optimal parameters for conventional SMC given certain reference signals, which will then be used as the baseline for comparison. The objective of PSO is to find a set ofX that produces the minimum value of a so-called cost function fX. In this paper, the cost function for PSO to optimize is set as follows:(27)fcost=∑z2dt=∑x2−z˙d+c1z1dt,which is the sum of the absolute sliding mode variable along one episode. The SMC parameter output by PSO is 0≤η≤20 in (17). For a general PSO framework with N particles of the swarm, the position of all particles at kth timestep can be represented as X=x1k,x2k,…,xnk. The position update function can be written as follows:(28)vik+1=ω×vik+a1×rand0,1×pbestik−xik+a2×rand0,1×gbestk−xik,xik+1=xik+vik+1.Among them,v is the velocity vector, which integrates the local best pbestik (the best position that the ith particle ever traversed) and global best gbestk (the best position that all particles ever traversed). a1,a2>0 are cognitive and social learning coefficients, which are related with local optimum and global optimum, respectively. ω≥0 is called inertia weight that adjusts the importance of the previous velocity. The pseudo code of PSO is shown in Algorithm 2.Algorithm 2: Particle swarm optimization for a minimization problem.
(1)
Initialization.(2)
for each one of the N particles do(3)
Initialize the positionxi and velocity vi(4)
Set particles’ best position as currentpbesti=xi(5)
Calculate the fitness of each particle and set the optimum as thegbest(6)
end for(7)
Update.(8)
while stopping Criterion not met do(9)
Update particles’ velocity usingvik+1=ωvik+c1R1pbestik−xik+c2R2gbestk−xik(10)
Update particles’ position usingxik+1=xik+vik+1(11)
Evaluate the fitness of all particlesfxik+1(12)
Iffxik+1<pbest, update individual best pbestik+1=fxik+1(13)
Iffxik+1<gbest, update global best gbestk+1=fxik+1(14)
end while(15)
Output best results


### 4.4. Parameters Initialization

The parameters of single-pendulum dynamics are shown in Table2. The parameters related to DDPG and sliding mode controller are shown in Table 1.Table 2
Parameter of the single-pendulum.
ParametersMass (kg)Length (m)DampingGravity (Nm2)Valuem=1l=1b=0.1g=9.8During testing, two target trajectories are assigned. One is a constant value targetzd,z˙d,z¨dT=0,0,0T, and the other is a time-varying sine wave target zd,z˙d,z¨dT=2sint,2cost,−2sintT. The sampling time is △T=0.005s, and one episode takes 4000 steps. The disturbance d is a random value between −0.1,0.1rad/s2. For comparison, the initial state of the system is fixed to θ,θ.,θ..T=2,1,0T.


### 4.5. Results and Evaluation

In this section, the testing is implemented using a constant value target and a sine wave target, respectively. The results, including the torque profile, angle profile, model outputs, a sliding mode variable, and velocity error, are presented, with analysis and discussion. The baseline is the conventional sliding mode control optimized by the PSO algorithm, and the optimization function is the sum of absolute sliding mode variables at all time steps, namelyfunc_opt=∑s. For label inside figures, we use “USMC” to represent a U-model-based SMC, and “USMC-RL” to represent a U-model-based adaptive SMC using DDPG. Through testing, the forward calculation time of our deep learning model is only about 600 μs, which satisfies real-time requirements.


#### 4.5.1. Constant Value Target

Figures8–19 show the simulation result on the tracking control of the constant value target zd,z˙d,z¨dT=0,0,0T. The optimal η=7.60 for conventional SMC is given by PSO. The sampling time is 0.005 s and the simulation lasts for 2400 steps.Figure 8
Sliding mode variable profile.Figure 9
Angular profile.Figure 10
Velocity error profile.Figure 11
Torque profile.Figure 12
Actor network output.Figure 13
Sliding mode variable profile.Figure 14
Angular output profile.Figure 15
Velocity error profile.Figure 16
Torque profile.Figure 17
Actor network output.Figure 18
The 1st output of an actor network.Figure 19
The 2nd output of an actor network.For the sliding mode variable in Figure8, we can see that USMC-RL renders a shorter settling time than USMC, and that the chattering issue is greatly alleviated in the vicinity of equilibrium. While the chattering amplitude of the USMC is about 0.07, the output of the USMC-RL remains smooth. Figure 9 shows the angle output and shows that USMC-RL converges quicker than USMC. Figure 10 is the profile of velocity error. We can see that, at the beginning, the velocity error of the USMC-RL is greater than that of the USMC. This is because the faster response of adaptive SMC produces a faster speed to converge to equilibrium. This can also be seen in the torque profile in Figure 11 where the USMC-RL performs severe chattering upon arriving at equilibrium. Besides, it is obvious that the USMC renders a much higher chattering amplitude (around 20 Nm) near equilibrium. The output of the model in Figure 19 illustrates that when the error is considerable at the beginning, the outputs of the model are near the upper bound 20; however, when the system is near equilibrium, the outputs are small (about only 2.5). This means that the model successfully learns to speed up the convergence while alleviating chattering according to the current situation. Also, we can explain the chattering upon arriving at the equilibrium of USMC-RL. This is because of the continuity of the model—it cannot jump from a very high output (20) to a very small output (2.5). Therefore, when it should produce a small output, it is still on its way down.


#### 4.5.2. Sine Wave Target

Figures13–18 show the simulation result of the tracking control of a sine wave target zd,z˙d,z¨dT=2sint,2cost,−2sintT. The optimal η=6.05052635 for conventional SMC is given by PSO. The sampling time is 0.005 s and the simulation lasts for 2400 steps.The results analysis of the sine wave target is similar to that of the constant value target. For the sliding mode variables in Figure13, we can see that the USMC-RL renders a shorter settling time than the USMC and that the chattering issue is greatly alleviated in the vicinity of equilibrium. While the chattering amplitude of the USMC is about 0.03, the output of the USMC-RL remains smooth. Figure 14 shows the angle output. We can see that the outputs of both controllers follow the reference signal with some slight lagging. This lagging is caused by the invariant controller of the U-model. Similar lagging is also witnessed in Figure 15. The same overshoot is also shown in Figure 15, which is induced by the chattering of USMC-RL, as shown in Figures 16 and 18, illustrates a periodic pattern of the model outputs. The outputs are elevated when the absolute reference acceleration reaches the peaks.


### 4.6. DDPG Output Visualization and Interpretation

Visualization is a simple but direct technique to interpret the deep learning model [41]. To justify the effectiveness of the learned policy, part of the output of the actor network is visualized. Figures 18 and 19 illustrate the 1st and 2nd output of DDPG when the state of the pendulum is θ,θ.T=0,0T. The x axis is the angular error, ranging from −2 rad to 2 rad, and the y axis is the velocity error, ranging from −2 rad/s to 2 rad/s. It is clear that both outputs present a similar shape of a valley. When the errors are small, the outputs are small; when the errors grow larger, the outputs are becoming larger. This fits the intuition because larger outputs mean larger control gains, and subsequently, quicker responses and higher accuracy. We think this indicates the correctness of the learned policy. However, we can also see that the outputs following the line y=x are smaller than y=−x, which is counterintuitive. If the angular error and velocity error have the same sign, then the error will keep increasing, and it is better to output larger values. We believe this is because, when y=−x, the velocity will force the absolute angular error to decrease, which renders a fake stimulation to the model, spurring the model to output larger values when y=−x. We believe this is caused by inadequate exploration.


### 4.7. DDPG Interpretation Using SHAP

While [42] visualization renders us a holistic picture of the model output, the information it can provide is only qualitative. To understand the model behaviour in a quantitative manner, we implemented the SHAP method for a model explanation, in terms of both global explanation and local explanation. First, an introduction of SHAP is given, and then, the global and local explanations are presented.SHAP (SHapley Additive exPlanations) is a model-agnostic, post-hoc explainable artificial intelligence (XAI) method. It explains the final output as the addition of attributions from all inputs. In this way, it can explain the influence of each input on the output, in a positive or negative direction. It borrows ideas from SHapley values [43] to calculate the marginal contribution of each input. To calculate SHapley values, we have to retrieve “background data” through sampling. Similar to the monte-carlo method, the more data points we sample, the more accurate the estimation is. In this paper, the sampling ranges of four input features are set as follows:(29)−2≤θ≤2,−2≤θ.≤2,−2≤e_θ≤2,−4≤e_θ.≤4.We took a uniform sample of 5,000 points from this domain, combined them with the respective model outputs, and formed our background data. Figure20 illustrates the distribution of our background data using a box plot. The mean values of four input features are −0.01758551,−0.01059006,0.02874098,0.05264564, and the corresponding standard deviations are 1.14712378,1.15875792,1.15254858,2.32362302.Figure 20
Distribution of background data.(Global explanation). The global explanation focuses on gauging how much influence each input feature has on each of the model outputs. This is carried out by calculating the expectation of the absolute SHapley value of each input feature with respect to the outputs. By using the background data, we have Figure21, which shows the estimated SHapley value of each input feature with respect to the two model outputs u1,u2. The feature names are listed and depicted in a descending order from top to bottom, according to the SHapley value. In other words, velocity error influences the outputs most, compared with other features. This fits with the intuition because the velocity range is larger than the angle range, and the velocity error accounts for the dynamic process. The second most influential feature is the angular error, and the last two features are angle and velocity. We can see that the first two features are related to the error information, which determines the robustness term of the SMC controller. The last two features are more related to the nominal control term. This result coincides with the rationale of the proposed method because the model outputs account for the robustness term of SMC, and it should take more consideration of the errors.Figure 21
Global explanation of the model.(Local explanation). The local explanation interprets quantitatively, figuring out why the model outputs certain values compared with the base value (the base value refers to the mean outputs among all the background data). Here, we take two extreme examples to show how our DDPG model can ease the chattering without compromising the settling time and accuracy. Figure22 and Figure 23 show the force plots of two DDPG outputs with the input features being θ,θ.,e_θ,e_θ.=0,0,0,0. We say this input is “under low error” because the errors are 0 and the system is in self-equilibrium. We can see from the figures that all four features are dragging the outputs from base values (13.87 and 14.13) to very low values (2.28 and 2.07). This behaviour fits with common sense since the system is now stable, so DDPG only needs small outputs to reject disturbance. In this way, the chattering issue is eased.Figure 22
Force plot ofu1 under low error.Figure 23
Force plot ofu2 under low error.Another extreme is when the errors are large. Consider the initial condition of our simulations, which isθ,θ.,e_θ,e_θ.=2,0,2,0. Figures 24 and 25 show the force plots under this situation. We can see that the velocity and velocity error drag the outputs to lower values, while the angle and angular error are dragging it higher. The reasons are twofold. Firstly, the low value of velocity and velocity error suggests that no large values are required. Secondly, both the high value of angle and the high value of angular error require larger outputs to stabilize. In this case, the system can converge quickly.Figure 24
Force plot ofu1 under high error.Figure 25
Force plot ofu2 under high error.In summary, the analysis results of the SHAP method show that our DDPG model can make rational decisions;when the errors are large, output large values to converge quickly; when the errors are small, output small values to reduce chattering.


## 4.1. Dynamic Model Establishment

The simulation is implemented on a single pendulum. The structure graph is shown in Figure6.Figure 6
Structural graph of a single-pendulum.θ is the angle of the pendulum, l is the length, m is the weight, and g is the gravity coefficient. For simplicity and without loss of generality, we assume the mass of the pendulum is concentrated at the end of the link, and therefore, the dynamic equation can be derived as follows:(23)θ..=1ml2τ−bθ.−mglsinθ+d,where θ.. is the angular acceleration, θ. is the angular velocity, τ is the control torque, d is the disturbance, and b is the damping coefficient. Comparing (23) with (11), we have(24)f=−1ml2bθ.+mglsinθ,h=1ml2.Integrating (24) with (20), we have the sliding mode controller equation for a single-pendulum dynamics as follows:(25)τ=ml2−ηsatz2,δ−c1z˙1+z¨d−c2z2−z1−bθ.+mglsinθ.


## 4.2. DDPG Training Procedure

The training of DDPG follows the procedure of Algorithm 1, and the training-related parameters are shown in Table1. The network is trained for 5000 episodes, with 200-time steps for each episode. Due to the limited computational resources, the sampling time during training is set to 0.05s. The initial position of the pendulum is randomly set to be between −π,πT rad, and the initial velocity is between −10,10T rad/s. The target position, velocity, and acceleration of training are all set to 0. We later verify that, although the target of training is simplistic, the model can learn an effective policy that generalizes to time-varying trajectories (e.g., a sine wave). The state transition reward is designed as follows:(26)r=−r1+r2+r3,r1=2×e_θ+0.1×e_θ.2,r2=0.06×arctanu1,r3=0.06×u2..Table 1
Parameter of controllers.
ControllerParametersSMC controllerc1=2,c2=2DDPGlra=1e−6,lrc=2e−6,ρ=0.01,γ=0.9U-modelξ=1,ωn=5PSOa1=0.5,a2=0.5,N=40,ω=0.8The reward is composed of three parts.r1 takes a similar format as the sliding mode variable. An absolute operation is implemented on the angular error and velocity error, which implies a stricter punishment than the sliding mode variable because it requires both of them to converge to 0 simultaneously. Furthermore, there are slight adjustments to the weighting of the two terms. Since the priority goal is to track the desired position, we hope the reward can guide the model to pay more attention to angular error than velocity error. r2 calculates the inclination of the saturation function. r2 and r3 penalizes the interference of DDPG with the SMC controller. The final constant coefficients are determined through experiments. Intuitively, the model is instructed not to output unnecessarily large parameters to the SMC controller, and so that prevents the chattering issues. Figure 7 illustrates the episode reward during training. We can see that with training, the network gradually finds a nearly optimal policy and optimizes the episode reward.Figure 7
Training reward of DDPG.


## 4.3. PSO Tuning Procedure

PSO is a meta-heuristic global optimization method. It treats the model as a black box and tries to find optimal solutions through inputs and outputs. The update rule of PSO borrows ideas from swarm intelligence. Through interacting with each other, all particles strike a balance between exploration and exploitation. PSO has been proven to be simple in rationale but effective in practice. In this paper, we use PSO to find the optimal parameters for conventional SMC given certain reference signals, which will then be used as the baseline for comparison. The objective of PSO is to find a set ofX that produces the minimum value of a so-called cost function fX. In this paper, the cost function for PSO to optimize is set as follows:(27)fcost=∑z2dt=∑x2−z˙d+c1z1dt,which is the sum of the absolute sliding mode variable along one episode. The SMC parameter output by PSO is 0≤η≤20 in (17). For a general PSO framework with N particles of the swarm, the position of all particles at kth timestep can be represented as X=x1k,x2k,…,xnk. The position update function can be written as follows:(28)vik+1=ω×vik+a1×rand0,1×pbestik−xik+a2×rand0,1×gbestk−xik,xik+1=xik+vik+1.Among them,v is the velocity vector, which integrates the local best pbestik (the best position that the ith particle ever traversed) and global best gbestk (the best position that all particles ever traversed). a1,a2>0 are cognitive and social learning coefficients, which are related with local optimum and global optimum, respectively. ω≥0 is called inertia weight that adjusts the importance of the previous velocity. The pseudo code of PSO is shown in Algorithm 2.Algorithm 2: Particle swarm optimization for a minimization problem.
(1)
Initialization.(2)
for each one of the N particles do(3)
Initialize the positionxi and velocity vi(4)
Set particles’ best position as currentpbesti=xi(5)
Calculate the fitness of each particle and set the optimum as thegbest(6)
end for(7)
Update.(8)
while stopping Criterion not met do(9)
Update particles’ velocity usingvik+1=ωvik+c1R1pbestik−xik+c2R2gbestk−xik(10)
Update particles’ position usingxik+1=xik+vik+1(11)
Evaluate the fitness of all particlesfxik+1(12)
Iffxik+1<pbest, update individual best pbestik+1=fxik+1(13)
Iffxik+1<gbest, update global best gbestk+1=fxik+1(14)
end while(15)
Output best results


## 4.4. Parameters Initialization

The parameters of single-pendulum dynamics are shown in Table2. The parameters related to DDPG and sliding mode controller are shown in Table 1.Table 2
Parameter of the single-pendulum.
ParametersMass (kg)Length (m)DampingGravity (Nm2)Valuem=1l=1b=0.1g=9.8During testing, two target trajectories are assigned. One is a constant value targetzd,z˙d,z¨dT=0,0,0T, and the other is a time-varying sine wave target zd,z˙d,z¨dT=2sint,2cost,−2sintT. The sampling time is △T=0.005s, and one episode takes 4000 steps. The disturbance d is a random value between −0.1,0.1rad/s2. For comparison, the initial state of the system is fixed to θ,θ.,θ..T=2,1,0T.


## 4.5. Results and Evaluation

In this section, the testing is implemented using a constant value target and a sine wave target, respectively. The results, including the torque profile, angle profile, model outputs, a sliding mode variable, and velocity error, are presented, with analysis and discussion. The baseline is the conventional sliding mode control optimized by the PSO algorithm, and the optimization function is the sum of absolute sliding mode variables at all time steps, namelyfunc_opt=∑s. For label inside figures, we use “USMC” to represent a U-model-based SMC, and “USMC-RL” to represent a U-model-based adaptive SMC using DDPG. Through testing, the forward calculation time of our deep learning model is only about 600 μs, which satisfies real-time requirements.


### 4.5.1. Constant Value Target

Figures8–19 show the simulation result on the tracking control of the constant value target zd,z˙d,z¨dT=0,0,0T. The optimal η=7.60 for conventional SMC is given by PSO. The sampling time is 0.005 s and the simulation lasts for 2400 steps.Figure 8
Sliding mode variable profile.Figure 9
Angular profile.Figure 10
Velocity error profile.Figure 11
Torque profile.Figure 12
Actor network output.Figure 13
Sliding mode variable profile.Figure 14
Angular output profile.Figure 15
Velocity error profile.Figure 16
Torque profile.Figure 17
Actor network output.Figure 18
The 1st output of an actor network.Figure 19
The 2nd output of an actor network.For the sliding mode variable in Figure8, we can see that USMC-RL renders a shorter settling time than USMC, and that the chattering issue is greatly alleviated in the vicinity of equilibrium. While the chattering amplitude of the USMC is about 0.07, the output of the USMC-RL remains smooth. Figure 9 shows the angle output and shows that USMC-RL converges quicker than USMC. Figure 10 is the profile of velocity error. We can see that, at the beginning, the velocity error of the USMC-RL is greater than that of the USMC. This is because the faster response of adaptive SMC produces a faster speed to converge to equilibrium. This can also be seen in the torque profile in Figure 11 where the USMC-RL performs severe chattering upon arriving at equilibrium. Besides, it is obvious that the USMC renders a much higher chattering amplitude (around 20 Nm) near equilibrium. The output of the model in Figure 19 illustrates that when the error is considerable at the beginning, the outputs of the model are near the upper bound 20; however, when the system is near equilibrium, the outputs are small (about only 2.5). This means that the model successfully learns to speed up the convergence while alleviating chattering according to the current situation. Also, we can explain the chattering upon arriving at the equilibrium of USMC-RL. This is because of the continuity of the model—it cannot jump from a very high output (20) to a very small output (2.5). Therefore, when it should produce a small output, it is still on its way down.


### 4.5.2. Sine Wave Target

Figures13–18 show the simulation result of the tracking control of a sine wave target zd,z˙d,z¨dT=2sint,2cost,−2sintT. The optimal η=6.05052635 for conventional SMC is given by PSO. The sampling time is 0.005 s and the simulation lasts for 2400 steps.The results analysis of the sine wave target is similar to that of the constant value target. For the sliding mode variables in Figure13, we can see that the USMC-RL renders a shorter settling time than the USMC and that the chattering issue is greatly alleviated in the vicinity of equilibrium. While the chattering amplitude of the USMC is about 0.03, the output of the USMC-RL remains smooth. Figure 14 shows the angle output. We can see that the outputs of both controllers follow the reference signal with some slight lagging. This lagging is caused by the invariant controller of the U-model. Similar lagging is also witnessed in Figure 15. The same overshoot is also shown in Figure 15, which is induced by the chattering of USMC-RL, as shown in Figures 16 and 18, illustrates a periodic pattern of the model outputs. The outputs are elevated when the absolute reference acceleration reaches the peaks.


## 4.5.1. Constant Value Target

Figures8–19 show the simulation result on the tracking control of the constant value target zd,z˙d,z¨dT=0,0,0T. The optimal η=7.60 for conventional SMC is given by PSO. The sampling time is 0.005 s and the simulation lasts for 2400 steps.Figure 8
Sliding mode variable profile.Figure 9
Angular profile.Figure 10
Velocity error profile.Figure 11
Torque profile.Figure 12
Actor network output.Figure 13
Sliding mode variable profile.Figure 14
Angular output profile.Figure 15
Velocity error profile.Figure 16
Torque profile.Figure 17
Actor network output.Figure 18
The 1st output of an actor network.Figure 19
The 2nd output of an actor network.For the sliding mode variable in Figure8, we can see that USMC-RL renders a shorter settling time than USMC, and that the chattering issue is greatly alleviated in the vicinity of equilibrium. While the chattering amplitude of the USMC is about 0.07, the output of the USMC-RL remains smooth. Figure 9 shows the angle output and shows that USMC-RL converges quicker than USMC. Figure 10 is the profile of velocity error. We can see that, at the beginning, the velocity error of the USMC-RL is greater than that of the USMC. This is because the faster response of adaptive SMC produces a faster speed to converge to equilibrium. This can also be seen in the torque profile in Figure 11 where the USMC-RL performs severe chattering upon arriving at equilibrium. Besides, it is obvious that the USMC renders a much higher chattering amplitude (around 20 Nm) near equilibrium. The output of the model in Figure 19 illustrates that when the error is considerable at the beginning, the outputs of the model are near the upper bound 20; however, when the system is near equilibrium, the outputs are small (about only 2.5). This means that the model successfully learns to speed up the convergence while alleviating chattering according to the current situation. Also, we can explain the chattering upon arriving at the equilibrium of USMC-RL. This is because of the continuity of the model—it cannot jump from a very high output (20) to a very small output (2.5). Therefore, when it should produce a small output, it is still on its way down.


## 4.5.2. Sine Wave Target

Figures13–18 show the simulation result of the tracking control of a sine wave target zd,z˙d,z¨dT=2sint,2cost,−2sintT. The optimal η=6.05052635 for conventional SMC is given by PSO. The sampling time is 0.005 s and the simulation lasts for 2400 steps.The results analysis of the sine wave target is similar to that of the constant value target. For the sliding mode variables in Figure13, we can see that the USMC-RL renders a shorter settling time than the USMC and that the chattering issue is greatly alleviated in the vicinity of equilibrium. While the chattering amplitude of the USMC is about 0.03, the output of the USMC-RL remains smooth. Figure 14 shows the angle output. We can see that the outputs of both controllers follow the reference signal with some slight lagging. This lagging is caused by the invariant controller of the U-model. Similar lagging is also witnessed in Figure 15. The same overshoot is also shown in Figure 15, which is induced by the chattering of USMC-RL, as shown in Figures 16 and 18, illustrates a periodic pattern of the model outputs. The outputs are elevated when the absolute reference acceleration reaches the peaks.


## 4.6. DDPG Output Visualization and Interpretation

Visualization is a simple but direct technique to interpret the deep learning model [41]. To justify the effectiveness of the learned policy, part of the output of the actor network is visualized. Figures 18 and 19 illustrate the 1st and 2nd output of DDPG when the state of the pendulum is θ,θ.T=0,0T. The x axis is the angular error, ranging from −2 rad to 2 rad, and the y axis is the velocity error, ranging from −2 rad/s to 2 rad/s. It is clear that both outputs present a similar shape of a valley. When the errors are small, the outputs are small; when the errors grow larger, the outputs are becoming larger. This fits the intuition because larger outputs mean larger control gains, and subsequently, quicker responses and higher accuracy. We think this indicates the correctness of the learned policy. However, we can also see that the outputs following the line y=x are smaller than y=−x, which is counterintuitive. If the angular error and velocity error have the same sign, then the error will keep increasing, and it is better to output larger values. We believe this is because, when y=−x, the velocity will force the absolute angular error to decrease, which renders a fake stimulation to the model, spurring the model to output larger values when y=−x. We believe this is caused by inadequate exploration.


## 4.7. DDPG Interpretation Using SHAP

While [42] visualization renders us a holistic picture of the model output, the information it can provide is only qualitative. To understand the model behaviour in a quantitative manner, we implemented the SHAP method for a model explanation, in terms of both global explanation and local explanation. First, an introduction of SHAP is given, and then, the global and local explanations are presented.SHAP (SHapley Additive exPlanations) is a model-agnostic, post-hoc explainable artificial intelligence (XAI) method. It explains the final output as the addition of attributions from all inputs. In this way, it can explain the influence of each input on the output, in a positive or negative direction. It borrows ideas from SHapley values [43] to calculate the marginal contribution of each input. To calculate SHapley values, we have to retrieve “background data” through sampling. Similar to the monte-carlo method, the more data points we sample, the more accurate the estimation is. In this paper, the sampling ranges of four input features are set as follows:(29)−2≤θ≤2,−2≤θ.≤2,−2≤e_θ≤2,−4≤e_θ.≤4.We took a uniform sample of 5,000 points from this domain, combined them with the respective model outputs, and formed our background data. Figure20 illustrates the distribution of our background data using a box plot. The mean values of four input features are −0.01758551,−0.01059006,0.02874098,0.05264564, and the corresponding standard deviations are 1.14712378,1.15875792,1.15254858,2.32362302.Figure 20
Distribution of background data.(Global explanation). The global explanation focuses on gauging how much influence each input feature has on each of the model outputs. This is carried out by calculating the expectation of the absolute SHapley value of each input feature with respect to the outputs. By using the background data, we have Figure21, which shows the estimated SHapley value of each input feature with respect to the two model outputs u1,u2. The feature names are listed and depicted in a descending order from top to bottom, according to the SHapley value. In other words, velocity error influences the outputs most, compared with other features. This fits with the intuition because the velocity range is larger than the angle range, and the velocity error accounts for the dynamic process. The second most influential feature is the angular error, and the last two features are angle and velocity. We can see that the first two features are related to the error information, which determines the robustness term of the SMC controller. The last two features are more related to the nominal control term. This result coincides with the rationale of the proposed method because the model outputs account for the robustness term of SMC, and it should take more consideration of the errors.Figure 21
Global explanation of the model.(Local explanation). The local explanation interprets quantitatively, figuring out why the model outputs certain values compared with the base value (the base value refers to the mean outputs among all the background data). Here, we take two extreme examples to show how our DDPG model can ease the chattering without compromising the settling time and accuracy. Figure22 and Figure 23 show the force plots of two DDPG outputs with the input features being θ,θ.,e_θ,e_θ.=0,0,0,0. We say this input is “under low error” because the errors are 0 and the system is in self-equilibrium. We can see from the figures that all four features are dragging the outputs from base values (13.87 and 14.13) to very low values (2.28 and 2.07). This behaviour fits with common sense since the system is now stable, so DDPG only needs small outputs to reject disturbance. In this way, the chattering issue is eased.Figure 22
Force plot ofu1 under low error.Figure 23
Force plot ofu2 under low error.Another extreme is when the errors are large. Consider the initial condition of our simulations, which isθ,θ.,e_θ,e_θ.=2,0,2,0. Figures 24 and 25 show the force plots under this situation. We can see that the velocity and velocity error drag the outputs to lower values, while the angle and angular error are dragging it higher. The reasons are twofold. Firstly, the low value of velocity and velocity error suggests that no large values are required. Secondly, both the high value of angle and the high value of angular error require larger outputs to stabilize. In this case, the system can converge quickly.Figure 24
Force plot ofu1 under high error.Figure 25
Force plot ofu2 under high error.In summary, the analysis results of the SHAP method show that our DDPG model can make rational decisions;when the errors are large, output large values to converge quickly; when the errors are small, output small values to reduce chattering.


## 5. Conclusion

An adaptive sliding mode controller based on a deep deterministic policy gradient (DDPG) is proposed and combined with U-model control in this paper, for the tracking control of uncertain nonlinear systems. The proposed methodology successfully integrates the simplicity of the U-model and the robustness of SMC. Besides, the online tuning of DDPG results in lower chattering without compromising settling time or accuracy compared with a conventional sliding mode controller. Simulation on the single-pendulum proves its superiority. We think the combination of RL and SMC complements each other. First, from the controller’s point of view, the implementation of RL is an alternative method to transform original controllers into adaptive controllers. Compared with typical adaptive controller design methods, RL can reduce manual efforts and find optimum settings automatically. Second, combining with SMC enables the reservation of some explainability and robustness for RL. On the one hand, while we cannot understand the DDPG output directly, we can have an intuitive understanding by combining it with the formula of SMC. On the other hand, the RL algorithm tends to overfit, and the robustness of SMC can help overcome this drawback. Even when the real environment and dynamics are not completely the same as in training (due to wear, tear, hysteresis, etc.), SMC maintains a certain range of stability, which prevents the RL from failing.However, compared with the other adaptive methods, there are some challenges existing for the RL-based SMC. First, while decreasing human craftsmanship, the RL algorithm requires a large amount of data for training, which may wear and tear the machines in practice. Many methods can be implemented to increase the efficiency of the data and accelerate convergence, e.g., model-based reinforcement learning and imitation learning to warm up. Second, is the generalization problem. The RL algorithms tend to overfit specific scenarios. How to bridge the gap between simulation and reality, as well as how to transfer the model to another scenario, are open questions.For future work, researchers may implement maximum entropy reinforcement learning to fully explore the potentially larger action space and avoid converging to a local optimum. Finally, to accommodate real-life problems, this methodology can be extended to multi-input-multi-output (MIMO) systems using multi-agent reinforcement learning (MARL). Last but not least, the implementation of DDPG in this paper is only a simple version. More delicate considerations need to be implemented in order to maximize the potential of reinforcement learning in future research. Also, the DDPG-based method should be compared with other typical adaptive SMC controllers.



---

*Source: 8980664-2022-10-07.xml*