# Optimal Model-Free Mean-Square Consensus for Multi-Agents with Markov Switching Topology
## 1 Introduction  
The distributed control of multi-agent systems (MASs) has gained widespread application across various domains in the last several years, including smart grids [1], robotic control [2], and unmanned aerial vehicles [3]. Achieving consensus, where each agent aligns certain values, such as position or velocity, with its neighbors, is a fundamental challenge in MASs. However, in practical situations, not only is the achievement of consensus among agents a concern but it is also hoped that the control cost required for this process is as small as possible. Consequently, studying optimal consensus problems [4] is essential. As everyone is aware, the optimum consensus problem is a hard analytical problem that requires calculating the coupled Hamilton-Jacobi-Bellman (HJB) equation [5]. For example, in Ref. [6], a nonlinear protocol design rule was introduced to obtain the HJB equation, which solved the consensus problem of dynamic agent networks. Ref. [7] studied the consensus control for MASs in leadless and leader-follower modes using the method of solving nonlinear coupled PDE-ordinary differential equations, which are complex. Reinforcement learning [8] (RL) provides an effective approach for determining the approximate solution of the HJB equation. In Ref. [9], a novel MAS discrete-time (DT) dynamic game problem was proposed, which, under distributed control, can be regarded as an optimal tracking problem, and an online value iterative calculation was developed. Ref. [10] introduced an adaptive dynamic programming (ADP) algorithm to derive an optimal control strategy. It was assumed that the system in question involves state variables and control variables with time delays. A sliding mode control approach utilizing reinforcement learning was described in Ref. [11] to handle the optimal consensus problem of nonlinear MASs involving communication delay and uncertainty. To solve theconsensus of a heterogeneous MAS differential graph game, a cooperative policy iteration (PI) method was developed in Ref. [12]. Employing a heuristic dynamic programming iterative approach, a novel performance function was established in Ref. [13] to address the optimal tracking consensus problem over an indefinite time horizon. It is noteworthy that the aforementioned techniques often need a precise system model. In practice, the models of complex systems are often difficult to obtain, so it is more necessary to research the optimal model-free consensus. Apart from optimality, other significant aspects of the consensus problem include MAS model complexity (heterogeneity, latency, etc. ), an input constraint problem, and an unknown disturbance problem. In Ref. [14], an online policy iteration algorithm was presented that realizes adaptive control through the actor-critic framework. It also employed neural networks (NNs) to generate optimal control solutions for systems with unknown constraints on control inputs. This process eliminates the need for complete system dynamic knowledge. Referenced to as Ref. [15], they presented an online adaptive controller design approach utilizing integral RL for the optimal control of nonlinear systems. This method took into account input constraints and did not demand knowledge of the system's evolution. Ref. [16] presented an optimal synchronization control approach powered by RL to address the heterogeneous nonlinear MAS synchronization issue with asymmetric input constraints and unknown dynamic properties. An augmented system model was built by state-space transformation and the construction of a novel distributed observer, and a non-quadratic cost function was employed in a data-driven policy iteration algorithm. In the study referenced in [17], an RL-based approach was developed to optimize leader-follower consensus for certain second-order MASs characterized by unknown nonlinear systems. Ref. [18] proposed a distributed optimal consensus approach utilizing ADP, aimed at addressing input delays in unknown MASs.