# Research on Wargame Decision-Making Method Based on Multi-Agent Deep Deterministic Policy Gradient
## 3 Improved MADDPG for Wargame Decision-Making
### Improved MAPDDPG
#### 3.2.2 Joint Action-Value Function  
MADDPG is an extension of DDPG in a multi-agent environment, which is suitable for collaborative, competitive, or mixed mission settings [40]. In MADDPG, a separate actor and critic are learned for each agent, and each agent is therefore allowed to have its own arbitrary reward function. In wargames, each agent \(x\) has a deterministic policy \(\mu_{x}(\tau_{x},\theta_{x})\), where \(\theta_{x}\) represents the parameter of \(\mu_{x}\), and \(\mu\) is the set of policies for all agents defined as \(\{\mu_{x}(\tau_{x},\theta_{x})\}_{x=1}^{n}\). In order to centralize the training of agents in the critic networks, it is necessary to estimate the Action-Value function \(Q_{x}^{\mu}(o,a_{1},a_{2},\ldots,a_{n};\phi_{x})\), which is learned for each agent \(x\) separately. In the Action-Value function \(Q_{x}^{\mu}\), \(o\) represents the observation set \(\Omega\), \(\{a_{1},a_{2},\ldots,a_{n}\}\) refers  
Figure 3: The POMDP framework. Figure 2: The MARL framework. to the actions of agents in wargame, and \(\phi_{x}\) is the parameter of the Action-Value function. It trains the critic network by minimizing the loss function:  
\[LOSS(\phi_{x})=\mathbb{E}_{D}\bigg{[}\Big{(}y^{x}-Q_{x}^{\mu}(o,a_{1},a_{2}, \ldots,a_{n};\phi_{x})\Big{)}^{2}\bigg{]}, \tag{1}\]  
where \(y^{x}=r_{x}+\gamma Q_{x}^{u}\Big{(}o^{\prime},a_{1}^{\prime},a_{2}^{\prime}, \ldots a_{n}^{\prime}|_{a_{n}^{\prime}=n_{x}(a_{n}^{\prime},a_{n}^{\prime})}; \phi_{x}^{\prime}\Big{)}^{2}\). In this equation, \(r_{x}\) denotes the reward received by each agent \(x\), \(\{a_{1}^{\prime},a_{2}^{\prime},\ldots a_{n}^{\prime}\}\) is the set of target policies, the delay parameters \(\theta_{x}^{\prime}\) and \(\phi_{x}^{\prime}\) are the parameter of the target critic network, and the Replay Buffer \(D\) consists of the set \([s,s^{\prime},a_{1},\ldots,a_{n},r_{1},\ldots,r_{n}]\). The following policy gradients are calculated separately to update each agent's policy:  
\[\
abla_{\theta_{x}}J(\mu_{x})=\mathbb{E}_{D}\Big{[}\
abla_{\theta_{x}}\mu_{x} (\tau_{x})\
abla_{a_{x}}Q_{x}^{\mu}(s,a_{1},\ldots,a_{n})|_{a_{x}=\mu_{x}(\tau _{x})}\Big{]}, \tag{2}\]  
where agent \(x\)'s current action \(a_{x}\) is sampled from its current policy \(\mu_{a}\) when evaluating the Action-Value function \(Q_{x}^{\mu}\), while all other agents' actions are sampled from the replay buffer \(D\). However, in the wargame environment, where the number of agents and actions is large, training a centralized critic using only a simple MADDPG is potentially difficult and may not converge. To meet this challenge, a Joint Action-Value function \(Q_{tot}^{\mu}\) is introduced to train a centralized critic. Specifically, the Action-Value function of MADDPG is factorized, and the \(Q_{x}^{\mu}\) of each agent is inputted into the mixing network, followed by setting a mixing network parameter, while the \(Q_{tot}^{\mu}\) is obtained after estimation. The following is the joint action value function \(Q_{tot}^{\mu}\):  
\[Q_{tot}^{\mu}(\mathbf{\tau},\mathbf{a},s;\mathbf{\phi},\psi)=g_{\psi}\Big{(}s,\Big{\{}Q_{ x}^{\mu_{x}}(\tau_{x},a_{x};\phi_{x})\Big{\}}_{x=1}^{n}\Big{)}, \tag{3}\]  
where \(\mathbf{\phi}\) is the parameter of the Joint Action-Value function \(Q_{tot}^{\mu}\), and \(\phi_{x}\) represents the parameter of the Action-Value function \(Q_{x}^{\mu_{x}}\). In a wargame, \(g_{\psi}\) denotes a linear monotonic function, where the parameter \(\psi\) is a mixing network parameter. In order to compute the policy accurately, the centralized critic network needs to be trained to minimize the loss function. The loss function is presented as follows  
\[LOSS(\mathbf{\phi},\psi)=\mathbb{E}_{D}\Big{[}\big{(}y^{tot}-Q_{tot}^{\mu}(\mathbf{ \tau},\mathbf{a},s;\mathbf{\phi},\psi)\big{)}^{2}\Big{]}, \tag{4}\]  
where \(D\) refers to the Replay Buffer, and \(y^{tot}=r+\gamma Q_{tot}^{\mu}(\mathbf{\tau}^{\prime},\mathbf{\mu}(\mathbf{\tau}^{\prime} ;\mathbf{\theta}^{\prime}),s^{\prime};\mathbf{\phi}^{\prime},\psi^{\prime})\).