# A Q-Learning-Based Parameters Adaptive Algorithm for Formation Tracking Control of Multi-Mobile Robot Systems

Authors: Chen Zhang; Wen Qin; Ming-Can Fan; Ting Wang; Mou-Quan Shen
Journal: Complexity
 (2022)

---

## Abstract

This paper proposes an adaptive formation tracking control algorithm optimized by Q-learning scheme for multiple mobile robots. In order to handle the model uncertainties and external disturbances, a desired linear extended state observer is designed to develop an adaptive formation tracking control strategy. Then an adaptive method of sliding mode control parameters optimized by Q-learning scheme is employed, which can avoid the complex parameter tuning process. Furthermore, the stability of the closed-loop control system is rigorously proved by means of matrix properties of graph theory and Lyapunov theory, and the formation tracking errors can be guaranteed to be uniformly ultimately bounded. Finally, simulations are presented to show the proposed algorithm has the advantages of faster convergence rate, higher tracking accuracy, and better steady-state performance.


---

## Body

## 1. Introduction

A multi-mobile robot system can present intelligent behaviours through mutual cooperation and achieve work efficiency and fault tolerance that a single individual cannot provide, so that it can complete some more difficult tasks. The coordinated formation control of multiple mobile robots has received extensive attention due to its important applications in the industrial and medical field [1]. The most existing control methods dealing with formation control problems of multiple mobile robots mainly include behaviour-based control [2], virtual structures [3], and leader-follower architecture [4–6]. As a decentralized control strategy, the leader-follower formation structure has become the preferred control strategy due to its simplicity and scalability and requires less computation and communication resources than other strategies [7]. The movement types of mobile robots are divided into omnidirectional mobile robot (OMR) (holonomic) and nonholonomic one [8]. Concerning the uncertainty and disturbance of robot docking, reference [9] proposed a novel robust containment architecture for nonholonomic mobile robot formations with docking capabilities, which realized multirobot formation maintenance/switching, docking, and collision avoidance. In [10], a dynamic control law was developed for the cooperative target encircling problem of multiple unicycle mobile robots subject to heterogeneous input disturbances generated by the linear exogenous system.Since nonholonomic mobile robots have fewer controllable degrees of freedom (DOFs) than holonomic mobile robots, geometric constraints are introduced on the robot's motion. Common examples with incomplete constraints are unicycles and car-like wheeled mobile robots. In contrast, holonomic mobile robots have the same number of controllable DOFs and total DOFs, which makes them very flexible and being able to move within the workspace without geometric constraints (e.g., they can perform both rotation and lateral translation). A typical example of this class of vehicles is the omnidirectional robot with mechanical wheels. More details on the types and configurations of mobile robots can be found in [11]. References [12, 13] studied the formation control problem of multiple omnidirectional mobile robots. Reference [12] developed a distributed adaptive control law for a multirobot system by obtaining information from moving targets through some mobile robots and using backstepping control technology in formation control. Reference [13] proposed an improved collision avoidance and formation control to configure a multirobot system optimized for omnidirectional visual simultaneous localization and mapping. However, the performance of the control schemes in [12, 13] will deteriorate when there exist uncertainties in the kinematics and dynamics of the omnidirectional mobile robots.In the research of formation control of complex nonlinear systems, sliding mode control (SMC) is an effective robust controller for suppressing disturbances because of its excellent characteristics of being insensitive to system parameter changes and external disturbance when the system enters the sliding mode. In [14], a nonsingular fast terminal SMC was proposed, which can drive the tracking error to zero in finite time. Reference [15] investigated the leader-follower formation control for multiwheel mobile robots by combining a motion controller with a dynamic controller based on sliding mode. Considering the bounded external disturbance and parameter uncertainty of mobile robots, reference [16] proposed a dual-loop attitude tracking robust controller for mobile robots, using SMC with modified arrival law to ensure that the actual speed converges within a finite time. The main disadvantage of SMC is chattering phenomenon due to the discontinuity of the control law. In order to alleviate chattering, adaptive SMC [17] and higher order SMC [18, 19] have been proposed. However, these control methods may bring serious chattering even leading to instability when the system is exposed to a dynamic environment with large uncertainties and disturbances. Moreover, the traditional SMC is conservative to some extent since it ignores the information of uncertainties and disturbances. An effective way to solve this problem is using the disturbance estimation and compensation to decrease the conservatism and improve the control performance. Reference [20] proposed a disturbance observer and super-twisting SMC for the multirobot formation. Reference [21] designed an adaptive high-gain observer for the robot to estimate the nonlinearity that appears in the dynamics of the wheeled robot. In [22], the active disturbance rejection control technology was used to estimate the external disturbance in the inner loop of the double closed-loop strategy. On the other hand, the velocities of omnidirectional mobile robots cannot often be measured due to the lack of sensors which are needed in controller design.Extended state observer (ESO) based active disturbance rejection control (ADRC), proposed by Han [23], is a powerful tool to cope with uncertainties and external disturbances. The key idea of ADRC is that the total disturbances (including internal uncertain dynamics, cross-coupling, and external disturbances), regarded as an extended state of the system, can be estimated by the ESO and then compensated in the control signal. As a matter of fact, the ESO is a state observer to estimate both the system states and the total disturbances. Considering the advantage of it, ESO is adopted to estimate the total disturbances and then followed by the ESO-based controller constructed to compensate it. Reference [24] employed a nonlinear extended state observer (NESO) to estimate unknown states as well as uncertainties and designed a robust finite-time tracking control scheme to handle wheeled mobile robots with parameter uncertainties and disturbances. Reference [25] used NESO-based estimation and compensation signals into the closed-loop control method; an NESO-based decoupling control method was then proposed to solve the attitude control problem of hypersonic gliding vehicles. In [26], an NESO was used to estimate the system uncertainties, and a saturation-resistant adaptive SMC was designed based on the estimated values to achieve robust trajectory tracking for a wheeled mobile robot. However, it is difficult to find appropriate nonlinear functions to design an NESO in practice. For convenience of theoretical analysis, Gao proposed a linearized and bandwidth-based linear ADRC (LADRC) to simplify parameter tuning and standardize controller tuning [27]. Linear ESO (LESO) with nested inner and outer loops was used in [28] to actively estimate and eliminate generalized interference and improve the estimation accuracy in various practical models. The proposal of LADRC makes the design and adjustment of the controller easier and more effective, and the tracking error will be more decreased than some classic control structures [16], which greatly promotes the engineering application of ADRC. However, the LADRC is loss of design flexibility because the LADRC parameters are adjusted based on bandwidth. A general LADRC with more tuning parameters was proposed in [29], and nowadays LADRC has been employed to various cases in engineering application and becomes much more popular [30, 31].Reinforcement learning (RL) has been developed rapidly in recent years. As one of the important algorithms of RL, Q-learning is off-policy, tabular, model-free, and based on temporal-difference methods [32]. It has the advantages of not relying on models and having good learning effects for complex systems. In order to improve the control performance, some scholars combined Q-learning with PID control and proposed many excellent control methods [33, 34]. In the autonomous underwater vehicle system, reference [35] proposed a Q-learning PID controller based on RBFNN to improve control performance, in which Q-learning neural network was used to adaptively optimize control parameters. Reference [36] combined model-based Q-learning into the predictive control setting to provide closed-loop stability in online learning and ultimately improve the performance of the limited range controller. Chen proposed an adaptive auto-disturbance-rejection controller parameter adaptation method based on Q-learning for ship heading control with multiple uncertainties due to wind, wave, and current interference [37].Inspired by the above statements, this paper investigates the formation tracking control of a multiple omnidirectional mobile robot system. The considered mobile robots have internal modelling uncertainties and external disturbances (considered as total disturbances). To handle the disturbances, an LESO is constructed, and the total disturbances can be effectively estimated through the ESO. Then, on the basis of distributed formation tracking control architecture, an LESO-based SMC (LESO-SMC) is designed for each OMR to ensure that the observer errors and the formation neighbourhood errors are uniformly ultimately bounded (UUB). However, LESO based control is not widely used in practice because there are not adequate methods for LESO parameter adjustment. In view of this, and furthermore for obtaining better control performance, Q-learning is employed to optimize the bandwidth parameters of LESO and the control parameters of SMC and to avoid the complex parameter tuning process. In addition, a simulation example is given to verify the effectiveness of the proposed method.The main features of the proposed methods are summarized as follows:(1)
An LESO is constructed to estimate the ‘total disturbances’ in real time, including both internal parameter uncertainties and external disturbances, and then an LESO-SMC based formation protocol is developed for the OMR system. The LESO provides distinctly better robustness against ‘total disturbances’ by providing accurate input variables to the control system, including the states of the mobile robot at each order, as well as the extended state representing the ‘total disturbances’. Then a corresponding improved strategy on the SMC is made to compensate the influence of the ‘total disturbances’, which ensures a faster convergence rate and decreases the conservatism of the traditional SMC.(2)
To take full advantages of the LESO-SMC, an adaptive method of LESO-SMC parameters optimized by Q-learning algorithm is proposed in the formation tracking control of the OMR system. Q-learning is introduced to perform online parameter adaptation (including the observer, sliding mode variables, and controller parameters), which exhibits better formation tracking performance and avoids the complex parameter tuning process.The organization of this paper is as follows. In Section2, the dynamic model and some preliminary knowledge are outlined. In Section 3, the proposed controller design based on LESO and SMC is presented. Both the Q-learning algorithm and the Q-learning parameter tuning process are also introduced. The results of numerical simulations are discussed in Section 4, followed by the conclusion of this paper.Notations:AT and A−1 represent the transpose and inverse of matrix A, respectively. Rn represents n-dimensional real column vector set. In denotes an n×n unit matrix. 0n denotes an n×n zero matrix. ⊗ stands for Kronecker product. sign⋅ is the sign function. ⋅ represents the Euclidean norm. diagx1,x2,…,xn denotes the diagonal matrix with its diagonal entries being x1,x2,…,xn. λmaxA and λminA represent the maximum and the minimum eigenvalues of matrix A, respectively.


## 2. Problem Formulation and Preliminaries

### 2.1. Dynamic Model

The Euler–Lagrange equation of motion can be used to describe the dynamic behaviour of an OMR. The dynamic model of the ith OMR can be described as [31](1)Miq¨i+Ciq˙i+Gi=τi,where i=1,…,n, qi=xi,yi,θiT∈R3 represents the position and orientation angle of the ith robot in the world coordinate frame. Mi is the inertia matrix, Ci is the Coriolis and centrifugal term, Gi is the gravitational force, and τi denotes the control input. It is assumed that the robots move on flat ground, where gravitational force Gi is 0.Considering the unknown dynamic disturbances and model uncertainties, a new dynamic model is obtained as follows:(2)Mi+ΔMiq¨i+Ci+ΔCiq˙i=τi+di,where ΔMi∈R3×3 and ΔCi∈R3×3 denote uncertainty terms, and di∈R3 denotes the unknown disturbance term.The above equation can be rewritten as(3)Miq¨i+Ciq˙i+Δi=τi+di,where Δi=ΔMiq¨i+ΔCiq˙i denotes the uncertain term.Assumption 1.
The unknown disturbance termdi satisfies dij≤λ1,j=0,1,2, where λ1 is an unknown positive constant.Assumption 2.
The uncertain termsΔMi and ΔCi are bounded. Thus, the uncertain term Δi satisfies Δij≤λ2,j=0,1,2, where λ2 is an unknown positive constant.
As for the LESO design in the following section, the total disturbance is extended as a new state. Definexi,1=qi, xi,2=q˙i, xi,3=fi=Mi−1di−Δi∈R3, where xi,3 is the extended state of the total disturbances fi. The dynamic model (3) can be transformed into the following system:(4)x˙i,1=xi,2,x˙i,2=xi,3+Mi−1τi−Mi−1Cixi,2,x˙i,3=f˙i.
The dynamic equation of the virtual leader is(5)x˙l=vl,v˙l=ul,where xl=xlx,xly,xlθT∈R3 and vl=vlx,vly,vlθT∈R3 indicate the position and velocity of the virtual leader, respectively. ul is the control input of the virtual leader which can be obtained by some followers.Assumption 3.
The derivative of the total disturbancesfi is bounded by an unknown constant ρi, i.e., f˙i≤ρi.Remark 1.
Note thatfi in the dynamic model (4) indicates an unknown term, such as external disturbances and model uncertainties for the mechanism of OMRs. In practical applications, the total disturbances mainly include wheel-ground sliding, modelling uncertainty due to robot load variation, etc.Remark 2.
In practice, both the speed of the DC motor which drives the OMR forward and its derivative have upper bounds, i.e.,qi, q˙i, and q¨i are all bounded. Δi is related to q˙i and q¨i, so we can conclude that Δi and its derivative are bounded. Similarly, fi and its time derivative are bounded as well. Therefore, the assumptions of Δi and fi in Assumptions 1 and 2 are reasonable.


### 2.2. Algebraic Graph Theory

Consider an omnidirectional mobile robot system consisting of one virtual leader andn followers. Assume that each robot is a node, and the information exchange among follower robots can be described by a directed graph G. The graph G=V,E,A is composed of the node set V=v1,…,vn, the edge set E=vi,vj∈V×V, and the adjacency matrix A=aij∈Rn×n. If there is an edge between agents i and j, i.e., vi,vj∈E, then aij>0; otherwise aij=0. The set of neighbours of node vi is denoted by N=j:vi,vj∈E. The in-degree of node vi is defined as di=∑j∈Naij. Then, the in-degree matrix of digraph G is D=diagd1,…,dn, and the Laplacian matrix of digraph G is L=D−A. A path in graph G from vi to vj is a sequence of distinct vertices starting with vi and ending with vj such that consecutive vertices are adjacent. Graph G is connected if there is a path between any two vertices. Graph G contains a directed spanning tree if there is a vertex (the root node) which can reach all the other vertices through a directed path. The virtual leader’s adjacency weight matrix B=diagb1,…,bn, where bi represents the topological weight of the communication between agent i and the virtual leader, bi>0, if there is communication between the agent i and the virtual leader; otherwise, bi=0.Assumption 4.
For the considered multiple mobile robot systems (2), graph G for the n follower robots contains a directed spanning tree; i.e., there is a vertex (the root node) which can reach all the other vertices through a directed path.Lemma 1 (see [38]).
IfG is a directed graph which contains a directed spanning tree and at least the root-node agent has access to the virtual leader, then the matrix L+B is of full rank.


## 2.1. Dynamic Model

The Euler–Lagrange equation of motion can be used to describe the dynamic behaviour of an OMR. The dynamic model of the ith OMR can be described as [31](1)Miq¨i+Ciq˙i+Gi=τi,where i=1,…,n, qi=xi,yi,θiT∈R3 represents the position and orientation angle of the ith robot in the world coordinate frame. Mi is the inertia matrix, Ci is the Coriolis and centrifugal term, Gi is the gravitational force, and τi denotes the control input. It is assumed that the robots move on flat ground, where gravitational force Gi is 0.Considering the unknown dynamic disturbances and model uncertainties, a new dynamic model is obtained as follows:(2)Mi+ΔMiq¨i+Ci+ΔCiq˙i=τi+di,where ΔMi∈R3×3 and ΔCi∈R3×3 denote uncertainty terms, and di∈R3 denotes the unknown disturbance term.The above equation can be rewritten as(3)Miq¨i+Ciq˙i+Δi=τi+di,where Δi=ΔMiq¨i+ΔCiq˙i denotes the uncertain term.Assumption 1.
The unknown disturbance termdi satisfies dij≤λ1,j=0,1,2, where λ1 is an unknown positive constant.Assumption 2.
The uncertain termsΔMi and ΔCi are bounded. Thus, the uncertain term Δi satisfies Δij≤λ2,j=0,1,2, where λ2 is an unknown positive constant.
As for the LESO design in the following section, the total disturbance is extended as a new state. Definexi,1=qi, xi,2=q˙i, xi,3=fi=Mi−1di−Δi∈R3, where xi,3 is the extended state of the total disturbances fi. The dynamic model (3) can be transformed into the following system:(4)x˙i,1=xi,2,x˙i,2=xi,3+Mi−1τi−Mi−1Cixi,2,x˙i,3=f˙i.
The dynamic equation of the virtual leader is(5)x˙l=vl,v˙l=ul,where xl=xlx,xly,xlθT∈R3 and vl=vlx,vly,vlθT∈R3 indicate the position and velocity of the virtual leader, respectively. ul is the control input of the virtual leader which can be obtained by some followers.Assumption 3.
The derivative of the total disturbancesfi is bounded by an unknown constant ρi, i.e., f˙i≤ρi.Remark 1.
Note thatfi in the dynamic model (4) indicates an unknown term, such as external disturbances and model uncertainties for the mechanism of OMRs. In practical applications, the total disturbances mainly include wheel-ground sliding, modelling uncertainty due to robot load variation, etc.Remark 2.
In practice, both the speed of the DC motor which drives the OMR forward and its derivative have upper bounds, i.e.,qi, q˙i, and q¨i are all bounded. Δi is related to q˙i and q¨i, so we can conclude that Δi and its derivative are bounded. Similarly, fi and its time derivative are bounded as well. Therefore, the assumptions of Δi and fi in Assumptions 1 and 2 are reasonable.


## 2.2. Algebraic Graph Theory

Consider an omnidirectional mobile robot system consisting of one virtual leader andn followers. Assume that each robot is a node, and the information exchange among follower robots can be described by a directed graph G. The graph G=V,E,A is composed of the node set V=v1,…,vn, the edge set E=vi,vj∈V×V, and the adjacency matrix A=aij∈Rn×n. If there is an edge between agents i and j, i.e., vi,vj∈E, then aij>0; otherwise aij=0. The set of neighbours of node vi is denoted by N=j:vi,vj∈E. The in-degree of node vi is defined as di=∑j∈Naij. Then, the in-degree matrix of digraph G is D=diagd1,…,dn, and the Laplacian matrix of digraph G is L=D−A. A path in graph G from vi to vj is a sequence of distinct vertices starting with vi and ending with vj such that consecutive vertices are adjacent. Graph G is connected if there is a path between any two vertices. Graph G contains a directed spanning tree if there is a vertex (the root node) which can reach all the other vertices through a directed path. The virtual leader’s adjacency weight matrix B=diagb1,…,bn, where bi represents the topological weight of the communication between agent i and the virtual leader, bi>0, if there is communication between the agent i and the virtual leader; otherwise, bi=0.Assumption 4.
For the considered multiple mobile robot systems (2), graph G for the n follower robots contains a directed spanning tree; i.e., there is a vertex (the root node) which can reach all the other vertices through a directed path.Lemma 1 (see [38]).
IfG is a directed graph which contains a directed spanning tree and at least the root-node agent has access to the virtual leader, then the matrix L+B is of full rank.


## 3. Main Results

In this section, in order to achieve better formation tracking control performance, the LESO-SMC scheme will be designed for system (3) in the presence of the unknown disturbances and model uncertainties, such that all follower OMRs can track the virtual leader with the given formation configuration in advance and maintain the same speed with the virtual leader. Furthermore, a parameter adaptation method based on Q-learning algorithm is involved in LESO-SMC to avoid the complex parameter tuning process, which displays a better formation tracking control performance.


### 3.1. Linear Extended State Observer Design

In this section, we use the LESO to estimate the OMR’s total disturbancesfi, which include unknown disturbances and model uncertainties.The LESO for system (4) is designed as follows:(6)x^˙i,1=x^i,2+β1xi,1−x^i,1,x^˙i,2=x^i,3+Mi−1τi−Mi−1Cix^i,2+β2xi,1−x^i,1,x^˙i,3=β3xi,1−x^i,1,where x^i,1,x^i,2,x^i,3, and h^i are the estimations of xi,1,xi,2,xi,3, and hi, respectively, and βkk=1,2,3 is the observer gain to be determined. β1=3ω0,β2=3ω02,β3=ω03, where ω0 is the ESO bandwidth [39].Define the estimation errors as follows:x˜i,j=xi,j−x^i,jj=1,2,3; then the estimation error equation is given by(7)x˜˙i,1=x˜i,2−3ω0x˜i,1,x˜˙i,2=x˜i,3+Mi−1Cix˜i,2−3ω02x˜i,1,x˜˙i,3=f˙i−ω03x˜i,1.Letξi=ξi,1,ξi,2,ξi,3T=x˜i,1x˜i,2/ω0x˜i,3/ω02T∈R9; equation (7) can be rewritten as(8)ξ˙i=ω0A1ξi+N2Mi−1Cix˜i,2ω0+N1f˙iω02,where A1=−3I3I303−3I303I3−I30303∈R9×9 is Hurwitz, N1=0303I3, N2=03I303.Slightly different to the proof in [40], the convergence analysis of LESO (6) is given below.Lemma 2.
Considering the estimation error dynamics (8), the LESO (6) is bounded stable if the observer bandwidth ω0 is designed to satisfy the condition ω0−2P⋅κω0/ω0−1>0.Proof.
Since matrixA1 is Hurwitz, there exists a positive definite matrix P1 which satisfies A1TP1+P1A1=−I9.(9)N2Mi−1Cix˜i,2≤κx˜i,2≤κω0ξi,2,where κ is an unknown constant.
Consider a Lyapunov function candidate for (8) as(10)Veso=∑i=1nξiTP1ξi.
Then we obtain the time derivative ofVeso as(11)V˙eso=∑i=1nξ˙iTP1ξi+ξiTP1ξ˙i=∑i=1nω0ξiTA1TP1ξi+N2Mi−1Cix˜i,2TP1ξiω0+N1f˙iTP1ξiω02+ω0ξiTP1A1ξi+ξiTP1N2Mi−1Cix˜i,2ω0+ξiTP1N1f˙iω02=∑i=1n−ω0ξiTξi+2ξiTP1N2Mi−1Cix˜i,2ω0+2ξiTP1N1f˙iω02=−ω0ξTξ+2ξTPN¯Mi−1Cix˜i,2ω0+2ξTPNf˙ω02≤−ω0−2P⋅κω0ω0ξ2+2ξTPNf˙ω02,where(12)ξ=ξ1T,…,ξnTT,P=1n⊗P1,N=1n⊗N1,N¯=1n⊗N2,1n=1,1,…,1T∈Rn,f˙=f˙1T,…,f˙nTT.
Using Young's inequality, one has(13)2ξTPNf˙ω02≤ξ2+PN2f˙2ω04.
From Assumption3, we have f˙≤ρ, where ρ is an unknown constant. It can be obtained from (11) that(14)V˙eso≤−ω0−2P⋅κω0ω0−1ξ2+PN2ρ2ω04.
Ifω0−2P⋅κω0/ω0−1>0, then PN2ρ2/ω04 is bounded; hence the proposed LESO is bounded stable. The proof is completed.
To achieve a better tracking control performance, in this section, an LESO-SMC-based formation control scheme will be proposed to ensure the tracking performance based on the dynamic models introduced in Section2.


### 3.2. Sliding Mode-Based Formation Controller Design

Define the system neighbourhood errors as(15)ei,1=∑j∈Naijx^i,1−δi−x^j,1−δj+bix^i,1−xl−δi,ei,2=∑j∈Naijx^i,2−x^j,2+bix^i,2−vl,where δi=δix,δiy,δiθT∈R3 denotes the desired relative position for the ith robot and the virtual leader.According to (15), we design the ESO-based sliding mode surface of the overall formation for the ith agent as(16)Si=μei,1+ei,2,where μ is a positive constant.The time derivative ofSi is(17)S˙i=μe˙i,1+e˙i,2=μei,2+∑j∈Naijx^˙i,2−x^˙j,2+bix^˙i,2−ul.The formation tracking controller based on ESO-SMC algorithm for the ith agent is designed as(18)τi=MiMi−1Cix^i,2−x^i,3+ul+(∑j∈Naij+bi)−1⋅∑j∈Naijτj−μei,2−c1Si−c2sgnSi,where c1 and c2 are positive constants, and ul is the control input of the virtual leader which can be obtained by some followers.The designed control structure block diagram is shown in Figure1.Figure 1
The proposed control block diagram.


### 3.3. Stability Analysis of State Tracking Error Dynamics

We have introduced LESO to estimate the states and uncertainties of each order of OMR; next we will propose the SMC-based formation controller.Theorem 1.
Consider the OMR system composed of (4) and (5) that satisfies Assumptions 1–4. Under the adaptive formation tracking control algorithm optimized by Q-learning scheme with the LESO (6), the SMC (18) and the Q-learning-based parameter adaptive algorithm (see Section 3.4), the estimation errors of the proposed LESO, and the formation tracking errors are guaranteed to be UUB.Similar to [41], we comprehensively consider the closed-loop system composed of the estimation errors of the observer (6) and the sliding variable (16). The stability analysis of the closed-loop system is given below.Proof.
Define the positive definite Lyapunov function as(19)V=12∑i=1nξiTP1ξi+12∑i=1nSiTSi,with the time derivative of the second term in the right-hand side of (19) being(20)∑i=1nSiTS˙i=∑i=1nSiTμe˙i,1+e˙i,2=∑i=1nSiTμei,2+∑j∈Naijx^˙i,2−x^˙j,2+bix^˙i,2−ul.
The system neighbourhood error (15) can be expressed in matrix form as(21)e1=L+B⊗I3⋅x^1−1n⊗xl−δ,e2=L+B⊗I3⋅x^2−1n⊗vl,where e1=e1,1T,…,en,1TT∈R3n, e2=e1,2T,…,en,2TT∈R3n, x^1=x^1,1T,…,x^n,1TT∈R3n, x^2=x^1,2T,…,x^n,2TT∈R3n, δ=δ1T,…,δnTT∈R3n, 1n=1,1,…,1T∈Rn.
The sliding surface (16) can be rewritten as(22)S=μe1+e2.
Then the formation tracking controller (18) can be rewritten as(23)τ=MM−1Cx^2−x^3+1n⊗ul+L+B−1⊗I3⋅−μe2−c1S−c2sgnS,where M=diagMi, C=diagCi.
Substituting (21)–(23) into (20), one can obtain(24)∑i=1nSiTS˙i=STμe2+L+B⊗I3⋅x^3+M−1τ−1n⊗ul=−c1STS−c2STsgnS=−c1S2−c2S<0.
Substituting (14) and (24) into (19), then we have(25)V˙≤−12ω0−2P⋅κω0ω0−1ξ2+PN2ρ22ω04−c1S2−c2S≤−φTΓφ+ς,where φ=ξT,ST, Γ=Γ100Γ2, ς=PN2ρ2/2ω04−c2S, with Γ1=diagη,…,η︸n, η=1/2ω0−2P⋅κω0/ω0−1, Γ2=diagc1,…,c1︸n. Since Γ1 and Γ2 are positive definite, one has(26)V˙≤−λminΓξ2+S2+ς≤−λminΓ1λmaxPξTPξ+STS+ς≤−χV+ς,where χ=2λminΓmin1,1/λmaxP.
Based on Gronwall's inequality, one has(27)Vt≤V0e−χt+ςχ1−e−χt.
SinceVt≥1/2ξTPξ≥1/2λminPξ2, one has 1/2λminPξ2≤V0e−χt+ς/χ1−e−χt.
Henceξ≤2V0χe−χt+ς1−e−χt/χλminp ; then we obtain the bound of the observer error as(28)ξ≤2ςχλminP,t⟶∞.
SinceVt≥1/2STS=1/2S2, one has ξ≤2V0χe−χt+ς1−e−χt/χ. Similarly, we can conclude that S≤2ς/χ,t⟶∞. Therefore ξ and S are UUB. By choosing appropriate parameters, the neighbourhood error can be attracted into a small stable region containing the origin. The proof is completed.


### 3.4. Q-Learning Based Parameter Optimization Process

The learning process of Q-learning is of continuous environmental interaction. First, at timet, select an action value A¯t. Then the agent will transfer from the original state S¯t to a new state S¯t+1 with a probability of PS¯t+1|S¯t,a. At this time, due to environmental interaction, the agent can get a feedback return R, then the time variable is updated, and the agent restarts the above steps in the new state until the optimal strategy is finally obtained. The Q-learning algorithm is shown below.Parameters:α∈0,1, small ε>0, γ∈0,1.Step 1.
For each state-action pairs,a, ∀s∈S¯,a∈A¯, initialize the table entry Qs,a to zero. Observe the current state s.Step 2.
Loop for each episode: Select an actiona for state s using ε-greedy policy and execute it, use the same method to choose at for state st, then receive immediate reward rt, and observe the new state st+1. Update the table entry for Qst,at as follows:
Qst,at←Qst,at+αrt+γmaxaQst+1,a−Qst,at, st←st+1, until s is the terminal state and all of Qs,a are convergent.Step 3.
Output the final policyπs=arg maxaQs,a.
In this paper, we consider the application of Q-learning ideas to the optimization of observers and controllers. Regarding the formation error and the derivative of the error as the state of Q-learning, dividing the controller parameters selection into a reasonable range and combining the divided values as an optional action, a Q-learning algorithm for observer and controller parameters optimization can be obtained. The specific steps of the algorithm are as follows:(a)
States set are composed ofS¯tεi,i=1,2 .(b)
Actions set are described asω0∈2,4 , c1∈5,7, c2∈5,7, μ∈3,5. Reasonably divide the interval of the four parameters and randomly select and combine them to obtain the actions set.(c)
In this paper, the stage performance index, i.e., the average of the squared errors of 10 sampling times, is used to design the reward function. Rewards are given as follows:(29)J=∑k=110ei,12k10,r=100,J≤σ,−100,J>σ.
Perform the learning of the value functionQ according to the algorithm flow mentioned above. Then we get the learned Q table and the optimal strategy πs=arg maxaQs,a for online parameter selection.
There are 3 termination conditions as follows. If any one of them is fulfilled then the Q-value training is terminated.(a)
It is not desired in practice that the formation error in the control is too large, which will make little sense to continue to iteratively calculate theQ table. So when ei,1>10 , the training is terminated and reinitialized.(b)
The control process has reached a steady state; then the training is terminated, i.e.,ei,1<0.001 and ei,2<0.01.(c)
The simulation timet=500s; then the training is terminated.
DenoteLt as the training times. When Lt=800, the training is terminated and the trained Q table can be obtained for online control.Remark 3.
Both the formation error and the error variation rate are divided into 7 linguistic values, {NB, NM, NL,Z, PL, PM, PB}, so that the number of state set elements is 49. Choose a reasonable range for each element in the parameter set Π=ω0c1c2μ. Based on the experience of adjusting the LESO-SMC controller parameters, set ω0∈2,4, c1∈5,7, c2∈5,7, μ∈3,5. Let p1=p2=p3=p4=3; i.e., each parameter has 3 possible values; then the number of equivalent selectable actions is A¯t=81. Therefore, the value function matrix Q∈R81×49 for the formation system.Remark 4.
The introduction of Q-learning avoids the process of selecting and optimizing the controller, the observer, and the sliding mode parameters. After Q-learning optimization, the convergence performance of the controller and observer can be guaranteed.


## 3.1. Linear Extended State Observer Design

In this section, we use the LESO to estimate the OMR’s total disturbancesfi, which include unknown disturbances and model uncertainties.The LESO for system (4) is designed as follows:(6)x^˙i,1=x^i,2+β1xi,1−x^i,1,x^˙i,2=x^i,3+Mi−1τi−Mi−1Cix^i,2+β2xi,1−x^i,1,x^˙i,3=β3xi,1−x^i,1,where x^i,1,x^i,2,x^i,3, and h^i are the estimations of xi,1,xi,2,xi,3, and hi, respectively, and βkk=1,2,3 is the observer gain to be determined. β1=3ω0,β2=3ω02,β3=ω03, where ω0 is the ESO bandwidth [39].Define the estimation errors as follows:x˜i,j=xi,j−x^i,jj=1,2,3; then the estimation error equation is given by(7)x˜˙i,1=x˜i,2−3ω0x˜i,1,x˜˙i,2=x˜i,3+Mi−1Cix˜i,2−3ω02x˜i,1,x˜˙i,3=f˙i−ω03x˜i,1.Letξi=ξi,1,ξi,2,ξi,3T=x˜i,1x˜i,2/ω0x˜i,3/ω02T∈R9; equation (7) can be rewritten as(8)ξ˙i=ω0A1ξi+N2Mi−1Cix˜i,2ω0+N1f˙iω02,where A1=−3I3I303−3I303I3−I30303∈R9×9 is Hurwitz, N1=0303I3, N2=03I303.Slightly different to the proof in [40], the convergence analysis of LESO (6) is given below.Lemma 2.
Considering the estimation error dynamics (8), the LESO (6) is bounded stable if the observer bandwidth ω0 is designed to satisfy the condition ω0−2P⋅κω0/ω0−1>0.Proof.
Since matrixA1 is Hurwitz, there exists a positive definite matrix P1 which satisfies A1TP1+P1A1=−I9.(9)N2Mi−1Cix˜i,2≤κx˜i,2≤κω0ξi,2,where κ is an unknown constant.
Consider a Lyapunov function candidate for (8) as(10)Veso=∑i=1nξiTP1ξi.
Then we obtain the time derivative ofVeso as(11)V˙eso=∑i=1nξ˙iTP1ξi+ξiTP1ξ˙i=∑i=1nω0ξiTA1TP1ξi+N2Mi−1Cix˜i,2TP1ξiω0+N1f˙iTP1ξiω02+ω0ξiTP1A1ξi+ξiTP1N2Mi−1Cix˜i,2ω0+ξiTP1N1f˙iω02=∑i=1n−ω0ξiTξi+2ξiTP1N2Mi−1Cix˜i,2ω0+2ξiTP1N1f˙iω02=−ω0ξTξ+2ξTPN¯Mi−1Cix˜i,2ω0+2ξTPNf˙ω02≤−ω0−2P⋅κω0ω0ξ2+2ξTPNf˙ω02,where(12)ξ=ξ1T,…,ξnTT,P=1n⊗P1,N=1n⊗N1,N¯=1n⊗N2,1n=1,1,…,1T∈Rn,f˙=f˙1T,…,f˙nTT.
Using Young's inequality, one has(13)2ξTPNf˙ω02≤ξ2+PN2f˙2ω04.
From Assumption3, we have f˙≤ρ, where ρ is an unknown constant. It can be obtained from (11) that(14)V˙eso≤−ω0−2P⋅κω0ω0−1ξ2+PN2ρ2ω04.
Ifω0−2P⋅κω0/ω0−1>0, then PN2ρ2/ω04 is bounded; hence the proposed LESO is bounded stable. The proof is completed.
To achieve a better tracking control performance, in this section, an LESO-SMC-based formation control scheme will be proposed to ensure the tracking performance based on the dynamic models introduced in Section2.


## 3.2. Sliding Mode-Based Formation Controller Design

Define the system neighbourhood errors as(15)ei,1=∑j∈Naijx^i,1−δi−x^j,1−δj+bix^i,1−xl−δi,ei,2=∑j∈Naijx^i,2−x^j,2+bix^i,2−vl,where δi=δix,δiy,δiθT∈R3 denotes the desired relative position for the ith robot and the virtual leader.According to (15), we design the ESO-based sliding mode surface of the overall formation for the ith agent as(16)Si=μei,1+ei,2,where μ is a positive constant.The time derivative ofSi is(17)S˙i=μe˙i,1+e˙i,2=μei,2+∑j∈Naijx^˙i,2−x^˙j,2+bix^˙i,2−ul.The formation tracking controller based on ESO-SMC algorithm for the ith agent is designed as(18)τi=MiMi−1Cix^i,2−x^i,3+ul+(∑j∈Naij+bi)−1⋅∑j∈Naijτj−μei,2−c1Si−c2sgnSi,where c1 and c2 are positive constants, and ul is the control input of the virtual leader which can be obtained by some followers.The designed control structure block diagram is shown in Figure1.Figure 1
The proposed control block diagram.


## 3.3. Stability Analysis of State Tracking Error Dynamics

We have introduced LESO to estimate the states and uncertainties of each order of OMR; next we will propose the SMC-based formation controller.Theorem 1.
Consider the OMR system composed of (4) and (5) that satisfies Assumptions 1–4. Under the adaptive formation tracking control algorithm optimized by Q-learning scheme with the LESO (6), the SMC (18) and the Q-learning-based parameter adaptive algorithm (see Section 3.4), the estimation errors of the proposed LESO, and the formation tracking errors are guaranteed to be UUB.Similar to [41], we comprehensively consider the closed-loop system composed of the estimation errors of the observer (6) and the sliding variable (16). The stability analysis of the closed-loop system is given below.Proof.
Define the positive definite Lyapunov function as(19)V=12∑i=1nξiTP1ξi+12∑i=1nSiTSi,with the time derivative of the second term in the right-hand side of (19) being(20)∑i=1nSiTS˙i=∑i=1nSiTμe˙i,1+e˙i,2=∑i=1nSiTμei,2+∑j∈Naijx^˙i,2−x^˙j,2+bix^˙i,2−ul.
The system neighbourhood error (15) can be expressed in matrix form as(21)e1=L+B⊗I3⋅x^1−1n⊗xl−δ,e2=L+B⊗I3⋅x^2−1n⊗vl,where e1=e1,1T,…,en,1TT∈R3n, e2=e1,2T,…,en,2TT∈R3n, x^1=x^1,1T,…,x^n,1TT∈R3n, x^2=x^1,2T,…,x^n,2TT∈R3n, δ=δ1T,…,δnTT∈R3n, 1n=1,1,…,1T∈Rn.
The sliding surface (16) can be rewritten as(22)S=μe1+e2.
Then the formation tracking controller (18) can be rewritten as(23)τ=MM−1Cx^2−x^3+1n⊗ul+L+B−1⊗I3⋅−μe2−c1S−c2sgnS,where M=diagMi, C=diagCi.
Substituting (21)–(23) into (20), one can obtain(24)∑i=1nSiTS˙i=STμe2+L+B⊗I3⋅x^3+M−1τ−1n⊗ul=−c1STS−c2STsgnS=−c1S2−c2S<0.
Substituting (14) and (24) into (19), then we have(25)V˙≤−12ω0−2P⋅κω0ω0−1ξ2+PN2ρ22ω04−c1S2−c2S≤−φTΓφ+ς,where φ=ξT,ST, Γ=Γ100Γ2, ς=PN2ρ2/2ω04−c2S, with Γ1=diagη,…,η︸n, η=1/2ω0−2P⋅κω0/ω0−1, Γ2=diagc1,…,c1︸n. Since Γ1 and Γ2 are positive definite, one has(26)V˙≤−λminΓξ2+S2+ς≤−λminΓ1λmaxPξTPξ+STS+ς≤−χV+ς,where χ=2λminΓmin1,1/λmaxP.
Based on Gronwall's inequality, one has(27)Vt≤V0e−χt+ςχ1−e−χt.
SinceVt≥1/2ξTPξ≥1/2λminPξ2, one has 1/2λminPξ2≤V0e−χt+ς/χ1−e−χt.
Henceξ≤2V0χe−χt+ς1−e−χt/χλminp ; then we obtain the bound of the observer error as(28)ξ≤2ςχλminP,t⟶∞.
SinceVt≥1/2STS=1/2S2, one has ξ≤2V0χe−χt+ς1−e−χt/χ. Similarly, we can conclude that S≤2ς/χ,t⟶∞. Therefore ξ and S are UUB. By choosing appropriate parameters, the neighbourhood error can be attracted into a small stable region containing the origin. The proof is completed.


## 3.4. Q-Learning Based Parameter Optimization Process

The learning process of Q-learning is of continuous environmental interaction. First, at timet, select an action value A¯t. Then the agent will transfer from the original state S¯t to a new state S¯t+1 with a probability of PS¯t+1|S¯t,a. At this time, due to environmental interaction, the agent can get a feedback return R, then the time variable is updated, and the agent restarts the above steps in the new state until the optimal strategy is finally obtained. The Q-learning algorithm is shown below.Parameters:α∈0,1, small ε>0, γ∈0,1.Step 1.
For each state-action pairs,a, ∀s∈S¯,a∈A¯, initialize the table entry Qs,a to zero. Observe the current state s.Step 2.
Loop for each episode: Select an actiona for state s using ε-greedy policy and execute it, use the same method to choose at for state st, then receive immediate reward rt, and observe the new state st+1. Update the table entry for Qst,at as follows:
Qst,at←Qst,at+αrt+γmaxaQst+1,a−Qst,at, st←st+1, until s is the terminal state and all of Qs,a are convergent.Step 3.
Output the final policyπs=arg maxaQs,a.
In this paper, we consider the application of Q-learning ideas to the optimization of observers and controllers. Regarding the formation error and the derivative of the error as the state of Q-learning, dividing the controller parameters selection into a reasonable range and combining the divided values as an optional action, a Q-learning algorithm for observer and controller parameters optimization can be obtained. The specific steps of the algorithm are as follows:(a)
States set are composed ofS¯tεi,i=1,2 .(b)
Actions set are described asω0∈2,4 , c1∈5,7, c2∈5,7, μ∈3,5. Reasonably divide the interval of the four parameters and randomly select and combine them to obtain the actions set.(c)
In this paper, the stage performance index, i.e., the average of the squared errors of 10 sampling times, is used to design the reward function. Rewards are given as follows:(29)J=∑k=110ei,12k10,r=100,J≤σ,−100,J>σ.
Perform the learning of the value functionQ according to the algorithm flow mentioned above. Then we get the learned Q table and the optimal strategy πs=arg maxaQs,a for online parameter selection.
There are 3 termination conditions as follows. If any one of them is fulfilled then the Q-value training is terminated.(a)
It is not desired in practice that the formation error in the control is too large, which will make little sense to continue to iteratively calculate theQ table. So when ei,1>10 , the training is terminated and reinitialized.(b)
The control process has reached a steady state; then the training is terminated, i.e.,ei,1<0.001 and ei,2<0.01.(c)
The simulation timet=500s; then the training is terminated.
DenoteLt as the training times. When Lt=800, the training is terminated and the trained Q table can be obtained for online control.Remark 3.
Both the formation error and the error variation rate are divided into 7 linguistic values, {NB, NM, NL,Z, PL, PM, PB}, so that the number of state set elements is 49. Choose a reasonable range for each element in the parameter set Π=ω0c1c2μ. Based on the experience of adjusting the LESO-SMC controller parameters, set ω0∈2,4, c1∈5,7, c2∈5,7, μ∈3,5. Let p1=p2=p3=p4=3; i.e., each parameter has 3 possible values; then the number of equivalent selectable actions is A¯t=81. Therefore, the value function matrix Q∈R81×49 for the formation system.Remark 4.
The introduction of Q-learning avoids the process of selecting and optimizing the controller, the observer, and the sliding mode parameters. After Q-learning optimization, the convergence performance of the controller and observer can be guaranteed.


## 4. Numerical Simulations

In this section, numerical simulation examples are used to illustrate the previous conclusions and the effectiveness of the proposed control scheme. Consider a scenario where a multiagent systems composed of three OMRs (followers) are simultaneously tracking a preset target (the virtual leader).The communication topologyG is given in Figure 2. The corresponding Laplacian matrix L and the adjacency weight matrix B can be described as(30)L=1−10−12−1−101,B=100000000.Figure 2
The communication topology of the systems. The root node 0 denotes the virtual leader.Each dynamics model of the three OMRs can be described by an Euler–Lagrange equation as follows:(31)Miq¨i+Ciq˙i+Δi=τi+di,i=1,2,3,where Mi=m0mdsinθi0m−mdcosθimdsinθi0Iv, Ci=00mdθ˙icosθi00mdθ˙isinθi000. Physical parameters and their values of the OMR are shown in Table 1. xi,yi and θi denote the positions and the orientation angle of the ith OMR in the x and y directions in the ground coordinate system, respectively. qi=xi,yi,θiT∈R3 represents the position and orientation angle of the ith robot in the world coordinate frame, q˙i=x˙i,y˙i,θ˙iT∈R3 represents the linear velocity and angular velocity of the ith robot in the world coordinate frame, and q¨i=x¨i,y¨i,θ¨iT∈R3 represents the linear acceleration and angular acceleration of the ith robot in the world coordinate frame.Table 1
Model parameters of the OMR.
ParametersDefinitionsValuesmMass of the OMR4.5 kgIvMoment of inertia0.25 kg∗m2dDistance between the center of mass of the OMR and the intersection point of the symmetry axis of the OMR and the wheel axis0.2 mrWheel radius0.0635 mbThe distance between an actuated wheel and the symmetry axis0.166 mWe assume the initial position states of each OMR are randomly chosen within−5,5×−5,5, the initial angle is 0, and the initial velocity is 0,0,0T. δ1=−2,0,0T, δ2=1,3,0T, and δ3=1,−3,0T. The comparison simulations are carried out by the proposed adaptive method of LESO-SMC parameters with and without Q-learning algorithm, respectively, denoted by 'Q-LESO-SMC′ and 'LESO-SMC′, respectively. The parameters of LESO-SMC are chosen as ω0=2 , c1=6, c2=7, μ=4, and the parameters of Q-LESO-SMC are obtained after online optimization by the Q-learning algorithm in Section 3.4. The other parameters used in the simulation are chosen as α=0.1,γ=0.8,σ=0.0001,ε=0.2.In order to observe the performance of the two different controllers, the robot formation is implemented in two cases, including the case with constant disturbances and the case with sinusoidal disturbances.Case A.
The case with constant disturbances.
Consider the robot formation subject to constant external disturbances, denoted by step functions. The corresponding results are shown in Figures3–9.
The estimation behaviours of the LESO in two dimensions for CaseA with constant disturbances are shown in Figures 3–5. Here x˜i,1x,x˜i,2x, and x˜i,3x (x˜i,1y,x˜i,2y,x˜i,3y) denote the position, velocity, and the extended state (total disturbances) estimation in x-direction (in y direction) for the ith agent, i = 1,2,3, respectively. In each figure, the error plots are locally enlarged to highlight the specific convergence time and steady errors. From the comparison results, the estimation errors of all three states converge to a small stable region, and the estimation error of Q-LESO has faster convergence time and smaller steady estimation error than that of LESO under the constant disturbances.
Figure6 shows the trajectories of three mobile robots and the virtual leader. ☆ and ○ represent the end positions of each follower robot and the virtual leader, respectively. It can be seen from the figure that the robots form a preset formation with the virtual leader as the center.
In order to make the comparison of control performance between LESO-SMC and Q-LESO-SMC, the formation tracking performance of each follower under these two controllers is depicted in Figures 7–8. Here ei,1x and ei,2x (ei,1y and ei,2y) denote the formation tracking position error and velocity error in x-direction (in y direction) for the ith agent, i = 1,2,3, respectively. It can be seen that the convergence speed of formation tracking error with Q-LESO-SMC is faster than that with LESO-SMC method, and Q-LESO-SMC method has stronger ability to suppress disturbance by comparing the magnitudes of the steady-state errors.
The position trajectories for three OMRs are shown in Figure9, where ○ represents the starting position of each robot. It can be seen that 3 followers quickly track the virtual leader under the proposed Q-LESO-SMC method.Figure 3
The position estimation errorx˜i,1 in Case A. (a) The position estimation error of the x-direction of the ith agent i = 1, 2, 3. (b) The position estimation error of the y‐direction of the ith agent i = 1, 2, 3.
(a)(b)Figure 4
The velocity estimation errorx˜i,2 in Case A. (a) The velocity estimation error of the x-direction of the ith agent i = 1, 2, 3. (b) The velocity estimation error of the y‐direction of the ith agent i = 1, 2, 3.
(a)(b)Figure 5
The extended state estimation errorx˜i,3 in Case A. (a) The disturbance estimation error of the x-direction of the ith agent i = 1, 2, 3. (b) The disturbance estimation error of the y-direction of the ith agent i = 1, 2, 3.
(a)(b)Figure 6
The formation trajectories of three followers and the virtual leader in CaseA.Figure 7
The formation tracking position errorei,1 in Case A. (a) The formation tracking position error of the x-direction of the ith agent i = 1, 2, 3. (b) The formation tracking position error of the y-direction of the ith agent i = 1, 2, 3.
(a)(b)Figure 8
The formation tracking velocity errorei,2 in Case A. (a) The formation tracking velocity error of the x-direction of the ith agent i = 1, 2, 3. (b) The formation tracking velocity error of the y-direction of the ith agent i = 1, 2, 3.
(a)(b)Figure 9
The tracking trajectories of the three followers and the virtual leader in CaseA. (a) The tracking trajectories of the 1st follower for the virtual leader. (b) The tracking trajectories of the 2nd follower for the virtual leader. (c) The tracking trajectories of the 3rd follower for the virtual leader.
(a)(b)(c)Case B.
The case with sinusoidal disturbances.
For simplicity of simulation, consider the robot formation subject to nonlinear external disturbances, such as sine functions, e.g.,fi=0.5e−0.02tcost,0.2sint,0.5costT. The corresponding results are shown in Figures 10–16.
Figures10–12 show the comparison of estimation performance between Q-LESO and LESO with sinusoidal disturbances. The comparison of disturbance estimation behaviour is shown in Figure 12, where the ‘total disturbances’ of each follower are approximated by Q-LESO in a shorter period. In addition, the amplitude of the steady state of the estimation error of Q-LESO is smaller than that of LESO as can be seen in the locally enlarged plot.
Similar to Figure6, Figure 13 shows the trajectories of the three mobile robots and the virtual leader in the same way. As can be seen from the figure, the target triangular formation is achieved with the virtual leader at the center.
Figures14–15 show the comparison of formation tracking error between Q-LESO-SMC and LESO-SMC with sinusoidal disturbances. As can be seen, the followers track the virtual leader faster with the Q-LESO-SMC-based formation controller. The error steady-state part is locally enlarged, and comparing the magnitude of the steady-state error before and after optimization shows that the error amplitude of Q-LESO-SMC is smaller than that of LESO-SMC.
Similar to Figure9 in Case A, the position trajectories for three OMRs are shown in Figure 16. It can be seen that 3 followers quickly track the virtual leader under the proposed Q-ESO-SMC method, and the tracking error is smaller than that under the LESO-SMC method.Figure 10
The position estimation errorx˜i,1 in Case B(a) The position estimation error of the x-direction of the ith agent i = 1, 2, 3. (b) The position estimation error of the y-direction of the ith agent i = 1, 2, 3.
(a)(b)Figure 11
The velocity estimation errorx˜i,2 in Case B(a) The velocity estimation error of the x-direction of the ith agent i = 1, 2, 3. (b) The velocity estimation error of the y-direction of the ith agent i = 1, 2, 3.
(a)(b)Figure 12
The disturbance estimation errorx˜i,3 in Case B(a) The disturbance estimation error of the x-direction of the ith agent i = 1, 2, 3. (b) The disturbance estimation error of the y-direction of the ith agent i = 1, 2, 3.
(a)(b)Figure 13
The formation trajectories of three followers and the virtual leader in CaseB.Figure 14
The formation tracking position errorei,1 in Case B(a) The formation tracking position error of the x-direction of the ith agent i = 1, 2, 3. (b) The formation tracking position error of the y-direction of the ith agent i = 1, 2, 3.
(a)(b)Figure 15
The formation tracking velocity errorei,2 in Case B(a) The formation tracking velocity error of the x-direction of the ith agent i = 1, 2, 3. (b) The formation tracking velocity error of the y-direction of the ith agent i = 1, 2, 3.
(a)(b)Figure 16
The tracking trajectories of the three followers and the virtual leader in CaseB(a) The tracking trajectories of the 1st follower for the virtual leader. (b) The tracking trajectories of the 2nd follower for the virtual leader. (c) The tracking trajectories of the 3rd follower for the virtual leader.
(a)(b)(c)Remark 5.
In CaseB, both LESO and Q-LESO methods can achieve well formation control performance. The comparison results show that the Q-LESO exhibited the advantages of faster convergence rate, smaller tracking error, and better disturbance suppression performance. In addition, to get good performance, the LESO needs larger observer gains, which may go beyond bandwidths of practical systems and make the required control energy infeasible. However, higher observer gains may lead to the bigger overshoot (the so-called peaking phenomenon); see Figures 10–12. Therefore, it is a trade-off between cost and performance.


## 5. Conclusions

In this paper, considering the uncertainties and external disturbances in the formation process, an LESO-SMC with Q-learning adaptive optimization are proposed to achieve formation tracking of multiple OMRs. The simulation results show that the proposed control method has the advantages of faster convergence rate, higher tracking accuracy, and good steady-state performance. However, the LESO brings large overshoot with increasing bandwidth. We will further investigate this problem in the future.



---

*Source: 5093277-2022-01-05.xml*