# Optimal Model-Free Mean-Square Consensus for Multi-Agents with Markov Switching Topology
## 2 Preliminaries
### Value-Based Performance Function  
Define the local state-value function of agent \(i\) with fixed control policy \(\{u_{i}(\cdot),u_{-i}(\cdot)\}\) as  
\[V_{i}(e_{i}(k))=E[\sum_{l=k}^{\infty}\gamma^{l-k}r(e_{i}(l),u_{i}(e_{i}(l)),u_ {-i}(e_{-i}(l)))]=\sum_{\tau\in\mathcal{F}}V_{i}(e_{i}(k),\tau)\cdot Pr\{ \sigma(k)=\tau\}, \tag{16}\]which is the function of \(e_{i}(k)\) actually, where \(e_{-i}(k)=\{e_{j}(k):j\in\mathcal{N}_{i}\}\) denotes the LNTE of the neighbors of agent \(i\) and \(V_{i}(e_{i}(k),\tau)\) is  
\[V_{i}(e_{i}(k),\tau)=E\big{[}\sum_{l=k}^{\infty}\gamma^{l-k}r(e_{i}(l),u_{i}(e_ {i}(l)),u_{-i}(e_{-i}(l)))\big{]}\sigma(k)=\tau\big{]}. \tag{17}\]  
For convenience, we refer to \(u_{i}(e_{i}(k))\) as \(u_{i}(k)\) in the following. Then, the optimization problem is described by  
\[\min_{u_{i}(k)}V_{i}(e_{i}(k)), \tag{18}\]  
and the optimal control policy is presented as  
\[u_{i}^{*}(k)=\arg\min_{u_{i}(k)}V_{i}(e_{i}(k)). \tag{19}\]  
According to (16), the local state-value function satisfies the Bellman equation  
\[V_{i}(e_{i}(k))=E\big{[}r(e_{i}(k),u_{i}(k),u_{-i}(k))\big{]}+\gamma V_{i}(e_ {i}(k+1)). \tag{20}\]  
Similarly, (17) satisfies the Bellman equation  
\[V_{i}(e_{i}(k),\tau) =E\big{[}\tau(e_{i}(k),u_{i}(k),u_{-i}(k))\big{]}\sigma(k)=\tau \big{]}+\gamma E\big{[}\sum_{l=k+1}^{\infty}\gamma^{l-(k+1)}r(e_{i}(l),u_{i}(l),u_{-i}(l))\big{]}\sigma(k)=\tau\big{]} \tag{21}\] \[\triangleq r_{E}(e_{i}(k),\tau)+\gamma V_{i}(e_{i}(k+1),\tau),\]  
where  
\[V_{i}(e_{i}(k+1),\tau) =E\big{[}\sum_{l=k+1}^{\infty}\gamma^{l-(k+1)}r(e_{i}(l),u_{i}(l),u_{-i}(l))\big{]}\sigma(k)=\tau\big{]} \tag{22}\] \[=\sum_{\tau^{\prime}}E\big{[}\sum_{l=k+1}^{\infty}\gamma^{l-(k+1) }r(e_{i}(l),u_{i}(l),u_{-i}(l))\big{]}\sigma(k)=\tau,\sigma(k+1)=\tau^{\prime} \big{]}\] \[\cdot Pr\big{\{}\sigma(k+1)=\tau^{\prime}|\sigma(k)=\tau\big{\}}\] \[=\sum_{\tau^{\prime}}E\big{[}\sum_{l=k+1}^{\infty}\gamma^{l-(k+1) }r(e_{i}(l),u_{i}(l),u_{-i}(l))\big{]}\sigma(k+1)=\tau^{\prime}\big{]}\cdot p_ {\tau\tau^{\prime}}\] \[=\sum_{\tau^{\prime}}V_{i}(e_{i}(k+1),\tau^{\prime})\cdot p_{ \tau\tau^{\prime}}.\]  
Consequently, we are able to modify (21) as  
\[V_{i}(e_{i}(k),\tau)=r_{E}(e_{i}(k),\tau)+\gamma\sum_{\tau^{\prime}}V_{i}(e_{i }(k+1),\tau^{\prime})\cdot p_{\tau\tau^{\prime}}. \tag{23}\]  
Then, for agent \(i\), the Hamilton function is  
\[H_{i}(e_{i}(k),u_{i}(k),V_{i})=r_{E}(e_{i}(k),\tau)+\gamma\sum_{\tau^{\prime}} V_{i}(e_{i}(k+1),\tau^{\prime})\cdot p_{\tau\tau^{\prime}}-V_{i}(e_{i}(k), \tau). \tag{24}\]  
Definition 2: _[_9_]_ _A control set \(\{u_{i}(k),\forall i\}\) is considered as admissible with regard to (8) if the consensus can be achieved and guarantee (11) to be finite._  
The Bellman optimality principle states that the following HJB equation  
\[V_{i}^{*}(e_{i}(k))=\min_{u_{i}(k)}\{E\big{[}r(e_{i}(k),u_{i}(k),u_{-i}(k)) \big{]}+\gamma V_{i}^{*}(e_{i}(k+1))\} \tag{25}\]  
must be solved in order to obtain the optimal local state-value function.  
The Nash equilibrium definition [33] allows the coupled DT HJB equation to be rewritten as  
\[V_{i}^{*}(e_{i}(k))=E\big{[}r(e_{i}(k),u_{i}^{*}(k),u_{-i}^{*}(k))\big{]}+\gamma V _{i}^{*}(e_{i}(k+1)). \tag{26}\]