# Optimal Model-Free Mean-Square Consensus for Multi-Agents with Markov Switching Topology
## 1 Introduction
This was achieved by transforming the MAS model with input delay into an equivalent non-delay model and developing an iterative policy calculation for a distributed asynchronous update mechanism. The prerequisites and techniques for linear interconnected systems to accomplish a consensus objective in finite time under internal and external delay circumstances were thoroughly examined in Ref. [19]. In the study mentioned [20], a policy-gradient-based distributed optimal control technique was introduced to tackle the control challenges in MASs that are subject to external disturbances. Note that all of the above articles examined MAS consensus in fixed topologies. But, in fact, the topology switching of MASs exists widely. For example, for a smart grid to remain stable, several control units must cooperate with one another. When some part of the grid goes offline for maintenance or failure, the topology of the system changes. In Ref. [21], a distributed gain scheduling management was proposed for load frequency control in a smart grid with varying communication topology. By modeling and analyzing the influence of changing communication topology on the dynamic performance, the robustness and computational efficiency of the system were improved. Ref. [22] presented a hybrid decentralized approach based on switching systems for the formation control and course consensus control of multi-robot systems, including collision avoidance functions, which can achieve stability control in the case of communication topology changes. The above two articles have explained the significance of studying switching topology in practical applications, but the current methods do not consider the realization of optimal performance. Compared with the RL algorithm, the traditional distributed algorithm is less adaptive and robust. Therefore, the theoretical analysis of practical scenarios such as smart grids and robots needs to be supplemented in order to facilitate better application. Ref. [23] explored the consensus issue in dynamic agent networks, both with fixed and switching topologies. It introduced a pair of consensus protocols, evaluated network efficiency under communication delay conditions, and linked the network's algebraic connectivity to the effectiveness of linear consensus protocols. Building upon the aforementioned research, [PERSON]. [24] delved deeper into the findings and proved the asymptotic consistency of discrete systems and continuous systems under the condition that the joint topology of dynamic network topology has spanning trees. In Ref. [25], the focus was on addressing the consensus challenge in heterogeneous MASs characterized by an undirected and connected network topology. [PERSON]. [26] studied the leader-follower consensus problem of DT MASs on the joint-connected switching directed graph. [PERSON]. [27] investigated mean-square consensus for heterogeneous MASs, encompassing agents of both first- and second-order dynamics, under Markovian switching topologies, concentrating on developing control policies and consensus criteria utilizing properties of stochastic matrices. All the above work [23; 24; 25; 26; 27] used the classical methods in the control field to study the problem of switching topology while ignoring the control cost, which does not meet the actual demand. Ref. [28] addressed the event-triggered-based prescribed-time optimal consensus problem for switched stochastic MASs under both random switching topologies and periodic switching topologies. The novel reconstruction mechanism handled the consensus error at switching instants, ensuring system stability and optimal performance. Ref. [29] studied optimal consensus control for DT MASs under conditions of input delay and variable topologies, and addressed Markovian topology changes through tracking error estimation.