Model-Free Optimal Output Cluster Synchronization Control for Multiagent Systems

Authors: zongsheng  huang
Venue: IEEE_org_ICIST_2024_Conference
Published: 2024-08-21 07:42:05

Abstract: In this article, the model-free optimal output cluster
synchronization control problem is investigated for nonlinear
multiagent systems (MASs). First, in view of the unknown output
of leader, relying on practical prescribed-time performance
function, an observer is designed for each follower to estimate
the output of leader, and can achieve the desired accuracy
within prescribed time. Then, based on the designed observer, an
augmented system consisting of observer dynamics and follower
dynamics is constructed and the cost functin is built for each
follower. Accordingly, the optimal output cluster synchronization
control problem is transformed into a numerical solution to solve
the Hamilton-Jacobian-Bellman equation (HJBE). Subsequently,
the off-policy reinforcement learning (RL) algorithm is addressed
to learn the solution to HJBE without any knowledge of the
system dynamics. Meanwhile, to release computational burden,
the single critic neural network (NN) framework is employed to
implement the algorithm, where the least square method is used
for training the NN weights. Thus, the designed control algorithm
can minimize the cost functions and ensure the output cluster
synchronization of MASs with the unknown system dynamics
and unavailable leader output. Finally, the simulation examples
confirm the validity of the designed control scheme.

Main paper: 
 i. introduction: 
 Multiagent systems (MASs) are autonomous sensing systems containing multiple independent agents. Each agent shares information and collaborates with other agents to accomplish tasks via the communication network. Thanks to the robustness, extendibility and the ability to handle complex tasks, the synchronization control problem of MASs has received broad attention over the past decades, such as multiple vehicles [1] and multiple spacecrafts [2] . Due to the requirement of multiple tasks, MASs are partitioned into a few subgroups to accomplish different but cooperative tasks. Hence, the agents in the same cluster reach consistent goals, while the agents in the different clusters fulfil tasks variously, which is named cluster synchronization or group consensus [3] . It was proposed in [3] for the first time and extended to switching topologies and cooperation-competition network in [4] and [5] later, respectively. Recently, many scholars have made efforts for it. In [6] , a relative-output-based distributed control strategy was designed for MASs to achieve cluster synchronization. The cluster synchronization control problem via pinning control strategies was investigated for heterogeneous MASs in [7] . It should be acknowledged that in many performance-critical fields, such as missile guidance, MASs rendezvous, emergency braking and obstacle avoidance, achieving prescribed accuracy in given time is significant.Fortunately, the prescribed-time control (PTC) was firstly proposed by Song et al. to achieve the above target [8] . The PTC derives from the classical idea of strategic and tactical missile guidance application [9] , inherited the characteristics of finite-time control and fixed-time control. PTC came into being, which was independent on the initial values of the system and can achieve accurate prescribed settling time. Lots of controllers can be integrated with PTC algorithms that extend the application of PTC. Relying on [8] , in [10] , it has proved that the regular Lyapunov inequality was utilized for stability analysis and thus avoided the difficulty in stability analysis brought by PTC. Later, in [11] , the convergence rate can be preset as desired and a universal method for constructing the time-varying rate function was given which can be regarded as an extension of [8] , [10] . In [12] , the PTC was continued for observer with canonical form. In the above results, the size of the steady-state error depends on unknown parameters and the control accuracy after the settling time was uncertain, which was undesirable in practice. Therefore, the practical prescribed-time control was presented, where the settling time and control accuracy were directly assigned in advance by the designers [13] . In [13] , a new time-varying constraint function was designed to ensure that the system can operate even after the prescribed time and the global result was obtained. In [14] , the practical PTC-based timescale performance function was constructed to handle the case of arbitrary initial values and established the global consensus.As a popular control method, optimal control has attracted much attention. As we all know, for linear systems, with the aid of dynamic programming method, the optimal strategy is calculated associated with Ricatti equation. For nonlinear systems, the optimal solution can be produced related to Hamilton-Jacobian-Bellman equation (HJBE). Nevertheless, the HJBE can not be obtained via numerical methods. To break through this limitation, adaptive dynamic programming (ADP) method was firstly developed by Werbos [15] . The basic idea of ADP is to approximate the solution of HJBE by using the function approximation structure according to the optimality principle. Murray et al. [16] firstly proposed the value iteration (VI) algorithm for continuous systems and proved the convergence of algorithm, which was viewed as a major improvement in ADP area. On the basis of [16] , in [17] , the policy iteration (PI) algorithm was designed to approximate optimal saturation controller for nonlinear systems with saturation constraints. VI and PI are two important branches of ADP area, and also have important significance in output synchronization optimization for MASs, which mainly focuses on minimizing the cost function consisting of local output error between the follower and the leader. Currently, on the basis of PI/VI, many results associated with optimal output synchronization for MASs have been reported, e.g., see [18] and the references therein. It is worth noting that the above results require complete dynamical knowledge of the system and belong to model-based methods, which are hard to accomplish in complex industrial processes. Hence, model-free optimal approaches based on reinforcement learning (RL) have been exploited to overcome this weakness. RL is a fascinating machine learning technique in which optimal control strategy is learned relying on the interactions with the environment. In the optimal field, RL and ADP are interchangeable concepts [19] . As a typical model-free RL method, the off-policy algorithm was pioneered in [20] that learned the solution to the HJBE from the system data generated by an arbitrary training policy. This approach was continued in [21] for seeking the Nash equilibrium solution and solving H ∞ control problem in [22] , respectively. Later, due to the powerful feature about coping the unknown system dynamics, the off-policy RL algorithm was firstly extended to solve optimal synchronization problem for MASs with linear form in [23] . In [24] , the optimal synchronization problem for nonlinear MASs was investigated, which was viewed as an advance in off-policy algorithm. Recently, on the basis of [24] , in [25] , the asymmetric input-constraint was taken into account in model-free optimal control problem. The optimal synchronization of heterogeneous MASs in graphical games was handled in [26] , where the optimal control policy depending on not only the Nash equilibrium but the fixed control policies of neighbors. However, to the best of the authors' knowledge, there are no results about the HiTL optimal output cluster synchronization problem considering unknown output of leader, which motivates our research.Driven by these observations, in this paper, the prescribedtime observer-based HiTL optimal output cluster synchronization control strategy is built for MASs via off-policy RL algorithm. To improve the safety of MASs, the human operator is introduced to guide the whole MASs by sending the signal to the non-autonomous leader. A prescribed-time observer is designed for the situation that only part of the followers can access the output information of leader, which can achieve the desired accuracy in the prescribed time interval. Based on the constructed observer, the off-policy RL algorithm is applied to obtain the optimal controller, in which the requirement of completely known system dynamics is released. In addition, to implement the algorithm, the single critic NN is utilized. The main contributions are summarized as follows.(1) In this paper, to cope with the practical situation that the output of leader is unavailable to all followers, a prescribed-time observer is developed for each follower via the designed function. Compared with [27] , [18] , the observer error can converge to the required accuracy in prescribed time and the requirement of fullorder derivatives information of the leader is relaxed. More importantly, the prescribed time and accuracy are independent of the initial conditions, which are friendly to users and fit for practical engineering. (2) In this paper, in view of balancing the performance and cost of systems and achieving the optimal performance, the off-policy RL algorithm is applied to solve the optimal output cluster synchronization control problem without any knowledge of system dynamics. Compared with model-based RL algorithm in [18] , [28] , the modelfree RL algorithm relies only on measurement data of systems and has a broader development prospect in largescale complex systems. Furthermore, in contrast to the framework of critic-actor NNs in [25] , [28] , the single critic NN is utilized to implement off-policy algorithm.It leads to a simpler structure, less computation burden and eliminates the training errors caused by actor NN. The structure is listed as follows. In Section II, the considered system and some concepts are given. In Section III, the practical prescribed-time observer is presented. In Section IV, the process of designing optimal cluster synchronization controller is provided. Model-free RL algorithm and simulation results are presented in Section V and Section VI, respectively. Finally, the conclusion is given in Section VII.Notations: Throughout this article, R n×m represents the set of n × m real matrices; I p represents the identity matrix of dimension p; I N represents the identity matrix of dimension N ; I pN represents the identity matrix of dimension pN ; O p×1 denotes the zero matrix of dimension p×1; O p×m denotes the zero matrix of dimension p × m; ⊗ is the Kronecker product.The communication topology contains N followers is represented by a directed graph G = {V, ε}, whereV = {V 1 , V 2 , • • • , V N } and ε ⊆ V × V representthe vertex set and the edge set of N followers, respectively. If the information can be delivered to the ith follower directly from the jth follower, the edge (i, j) ∈ ε. Let A = [a ij ] ∈ R N ×N be the weighted adjacency matrix, where a ij > 0 if (i, j) ∈ ε, and , where b i = 1 indicates that the information of the leader in its cluster is available for the ith node. The ith node that can receive (send) information from (to) other clusters is called an in-node (out-node) of cluster ith. The sets of in-nodes and out-nodes of cluster m are respectively denoted as V in m ⊆ V m and V out m ⊆ V m .a ij = 0 if (i, j) / ∈ ε. 
 
 Appendix: