## 3 Multi-Agent Markov Game Model  
The MMG management system can be regarded as a decision-making model for multiple agents due to the differences in interest subjects and optimization objectives. The whole system can be divided into a DSO agent \(A^{D}\) and various MG agents \(A_{i}^{m}\), i = 1, 2,..., \(N_{m}\). The decision of each agent will be affected by the joint decision of other agents, and the Markov decision process is expanded into a Markov game in the multi-agent field. A Markov game of \(N\) agents can be represented as (\(S\), \(A_{1}\),..., \(A_{N}\),..., \(O_{1}\),..., \(O_{N}\), \(R_{1}\),..., \(R_{N}\), \(TR\)), where \(S\) represents the set of overall states of all agents, \(A_{i}\) represents the set of actions of agent \(i\), \(O_{i}\) represents the set of environmental states that agent \(i\) can observe, \(R_{i}\) represents the set of rewards of agent \(i\), and \(TR\) represents the state transition function. The Markov game process of agent \(i\) is as follows: agent \(i\) observes its current state \(o_{i}^{t}\in O_{i}\) at time step \(t\), and then selects actions \(a_{i}^{t}\in A_{i}\) according to the observed state to obtain corresponding rewards \(r_{i}^{t}\in R_{i}\) and the next observed state \(o_{i}^{t+1}\in O_{i}\). The strategy adopted by agent \(i\) is the mapping from state space to action space: \(O_{i}\times A_{i}\)\(\rightarrow\) [0,1]. The mapping of the state transition function \(TR\) is shown as follows: \(S\times A_{1}\times\ldots\times A_{N}\to S\). Reward is a function of the agent's state and action, \(S\times A_{i}\to R\). Each agent selects strategies according to its observation to maximize the expected discount reward:  
\[R_{i}^{t}=\sum_{n=0}^{T}\gamma^{n}r_{i}^{t+n} \tag{19}\]  
where \(\gamma\) is the discount factor and \(T\) is the time step of the observation interval. The Markov game of the DSO agent and MG agents are given below. The observation state space \(o_{t}^{D}\) of the DSO agent at time step \(t\) is shown in Equation (20):  
\[o_{t}^{D}:\left\{P_{t-1}^{ex},P_{t-2}^{ex},\ldots,P_{t-L}^{ex},\lambda_{t-1}^ {c},\lambda_{t-2}^{c},\ldots,\lambda_{t-L}^{c},h_{t}\right\} \tag{20}\]  
where \(P_{t-L}^{ex}\) is the total power exchange of MG at time step \(t\)\(-\)\(L\), \(\lambda_{t-L}^{c}\) is the wholesale electricity price at time step \(t\)\(-\)\(L\), and \(h_{t}\) is the time component. It can be seen that the DSO agent needs to obtain the power-exchange information and wholesale electricity price information of the last \(L\) time steps when making decisions. The action \(a_{t}^{D}:\lambda_{t}^{D}\) it takes at time step \(t\) is to set the retail electricity price of time step \(t\). The reward obtained by DSO agent in time step \(t\) is shown in (21):  
\[r_{t}^{d}=(1-\alpha)_{t}^{d}\ /\ f_{base}^{d}-\alpha\varphi_{t}^{d}\ /\ \varphi_{base}^{d} \tag{21}\]  
It is composed of electricity selling profits and penalty factors; \(\alpha\) is the weight factor. The state information of MG \(i\) is shown in Equation (22):  
\[o_{i,t}^{m}:\left\{\mathbf{\lambda_{t}^{d},L_{i,t},P_{i,t}^{re},SOC_{i,t},h_{ t}}\right\}\ \forall i\in\mathbf{N} \tag{22}\]  
where \(\mathbf{N}\) is the collection of MGs; \(\mathbf{\lambda_{t}^{d}}\) is the retail electricity price vector; \(\mathbf{L_{i,t}}\) is the load vector; \(\mathbf{P_{i,t}^{re}}\) is the renewable energy output vector, including wind power output and photovoltaic output; \(SOC_{i,t}\) is the state of charge of the energy storage battery in MG \(I\); and \(h_{t}\) is the time corresponding to the time step \(t\).