SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Based on the above optimization problem, we have the following lemma for individual policy improvement.
Lemma 2 (Individual Policy Improvement). Let  $\pi_i^{\mathrm{new}}$  be the optimizer of the maximization problem in (7). Then, we have  $Q_{\mathrm{jt}}^{\pi_{\mathrm{jt}}^{\mathrm{new}}}(\boldsymbol{s},\boldsymbol{u})\geq Q_{\mathrm{jt}}^{\pi_{\mathrm{jt}}^{\mathrm{old}}}(\boldsymbol{s},\boldsymbol{u})$  for all  $(s,u)\in |S|\times |U|$  with  $|U| < \infty$ , where  $\pi_{\mathrm{jt}}^{\mathrm{old}}(\pmb {u}|\pmb {s}) = \prod_i\pi_i^{\mathrm{old}}(u_i|\pmb {s})$  and  $\pi_{\mathrm{jt}}^{\mathrm{new}}(\pmb {u}|\pmb {s}) = \prod_i\pi_i^{\mathrm{new}}(u_i|\pmb {s})$ .
Combining Lemma 1 and 2, we can have the following theorem which proves the convergence and optimality of multi-agent policy iteration with a fixed marginal distribution.
Theorem 1 (Multi-Agent Policy Iteration with a Fixed Marginal Distribution). For any joint policy  $\pi_{\mathrm{jt}}$ , if we repeatedly apply joint policy evaluation and individual policy improvement. Then the joint policy  $\pi_{\mathrm{jt}}(\boldsymbol{u}|\boldsymbol{s}) = \prod_{i=1}^{n}\pi_{i}(u_{i}|\boldsymbol{s})$  will eventually converge to  $\pi_{\mathrm{jt}}^{*}$ , such that  $Q_{\mathrm{jt}}^{\pi_{\mathrm{jt}}^{*}}(\boldsymbol{s},\boldsymbol{u}) \geq Q_{\mathrm{jt}}^{\pi_{\mathrm{jt}}}(\boldsymbol{s},\boldsymbol{u})$  for all  $\pi_{\mathrm{jt}}$ , assuming  $|U| < \infty$ .
4.3 Approximation for Dynamic Marginal Distribution
With the multi-agent policy iteration above, we can have  $Q_{\mathrm{jt}}$  and corresponding  $Q_{i}$  for each agent, however, under a fixed marginal distribution. In this section, we discuss how to approximate the dynamic marginal distribution to decouple  $\rho_{\pi_{\mathrm{jt}}} (s_{i,t}^{-}|s_{i,t}^{+})$  from  $\pi_{\mathrm{jt}}$ , and introduce the Blahut-Arimoto algorithm for the corresponding optimization.
Notice that the original objective (3) comes with a constraint (4). In Section 4.2, what we did is to remove this constraint for an easier optimization process. What we are going to do here, is to add a similar constraint back. First, consider the meaning of  $\rho_{\pi_{\mathrm{jt}}}(s_i^- | s_i^+)$ , it represents the potential team composition given team-unrelated information. Therefore, inspired by REFIL (Iqbal et al., 2021), we randomly partition team composition under  $s = \{s_i^+, s_i^-\}$  into different subgroups, yielding a set of imaginary team compositions and corresponding imaginary distribution  $\hat{\rho}(s_i^* | s_i^+)$  for imaginary team-related information. With this imaginary distribution, we can propose the approximate objective
Figure 1: Learning framework of MIPI, where each agent  $i$  has three modules: a utility function  $Q_{i}(s,u_{i};\theta_{i})$ , a policy  $\pi_i(u_i|s;\phi_i)$ , and a marginal policy  $\pi_i(u_i|s_i^+;\omega_i)$ .
for (3) as follows,
$$
\underset {\pi_ {\mathrm {j t}}} {\arg \max } \sum_ {t = 0} ^ {T} \mathbb {E} _ {\rho_ {\pi_ {\mathrm {j t}}} (\boldsymbol {s} _ {t})} \left[ \mathbb {E} _ {\pi_ {\mathrm {j t}} (\boldsymbol {u} _ {t} | \boldsymbol {s} _ {t})} \left[ \gamma^ {t} \left(r (\boldsymbol {s} _ {t}, \boldsymbol {u} _ {t}) - \alpha \sum_ {i} \log \frac {\pi_ {i} \left(u _ {i , t} | \boldsymbol {s} _ {t}\right)}{\pi_ {i} \left(u _ {i , t} | \boldsymbol {s} _ {i , t} ^ {+}\right)}\right) \right] \right] \tag {8}
$$
$$
\text {s . t .} \quad \pi_ {i} \left(u _ {i, t} \mid s _ {i, t} ^ {+}\right) = \sum_ {s _ {i, t} ^ {*}} \hat {\rho} \left(s _ {i, t} ^ {*} \mid s _ {i, t} ^ {+}\right) \pi_ {i} \left(u _ {i, t} \mid s _ {i, t} ^ {+}, s _ {i, t} ^ {*}\right). \tag {9}
$$
Therefore, we can have the approximate optimization problem for (7) as follows,
$$
\pi_ {i} ^ {\text {n e w}} = \arg \max  _ {\pi_ {i} ^ {\prime}} \mathbb {E} _ {\pi_ {i} ^ {\prime}} \left[ Q _ {i} ^ {\pi_ {i} ^ {\text {o l d}}} (s, u _ {i}) - \alpha \log \frac {\pi_ {i} ^ {\prime} \left(u _ {i} \mid s\right)}{\pi_ {i} ^ {\prime} \left(u _ {i} \mid s _ {i} ^ {+}\right)} \right] \tag {10}
$$
$$
\text {s . t .} \quad \pi_ {i} ^ {\prime} \left(u _ {i} \mid s _ {i} ^ {+}\right) = \sum_ {s _ {i} ^ {*}} \hat {\rho} \left(s _ {i} ^ {*} \mid s _ {i} ^ {+}\right) \pi_ {i} ^ {\prime} \left(u _ {i} \mid s _ {i} ^ {+}, s _ {i} ^ {*}\right). \tag {11}
$$
The objective above exhibits similarities with the rate-distortion problem (Cover, 1999), which could be solved using the Blahut-Arimoto algorithm. Although with the approximation above we break the convergence of Theorem 1, by using the Blahut-Arimoto algorithm we can have the following theorem, indicating the convergence of (10) as shown in Leibfried and Grau-Moya (2020).
Theorem 2 (Convergence of Constrained Individual Policy Improvement). The optimization problem induced by (10) can be solved by iterating in an alternate fashion through the following two equations:
$$
\pi_ {i} ^ {m} \left(u _ {i} \mid s _ {i} ^ {+}\right) = \sum_ {s _ {i} ^ {*}} \hat {\rho} \left(s _ {i} ^ {*} \mid s _ {i} ^ {+}\right) \pi_ {i} ^ {m} \left(u _ {i} \mid s _ {i} ^ {+}, s _ {i} ^ {*}\right) \tag {12}
$$
$$
\pi_ {i} ^ {m + 1} \left(u _ {i} \mid s _ {i} ^ {+}, s _ {i} ^ {-}\right) = \frac {\pi_ {i} ^ {m} \left(u _ {i} \mid s _ {i} ^ {+}\right) \exp \left(Q _ {i} \left(\boldsymbol {s} , u _ {i}\right) / \alpha\right)}{\sum_ {u _ {i}} \pi_ {i} ^ {m} \left(u _ {i} \mid s _ {i} ^ {+}\right) \exp \left(Q _ {i} \left(\boldsymbol {s} , u _ {i}\right) / \alpha\right)}, \tag {13}
$$
where  $m$  refers to the iteration index. Denoting the total number of iterations as  $M$ , the presented scheme converges at a rate of  $O(1 / M)$  to an optimal policy  $\pi_i^*$  for any given bounded utility function  $Q_{i}$  and any initial policy  $\pi_i^0$ .
4.4 MIPI Framework
In Section 4.2 and 4.3, we propose our learning algorithm in theory. In this section, we discuss how to implement our algorithm in practice, which can be summarized in Figure 1.
In MIPI, each agent has a utility function  $Q_{i}(s,u_{i};\theta_{i})$ , a policy  $\pi_i(u_i|s;\phi_i)$ , and a marginal policy  $\pi_i(u_i|s_i^+;\omega_i)$ . For joint policy evaluation, with the utilities of agents, we use a mixer network  $\mathrm{Mixer}(\cdot ,s;\Theta)$  to get the joint state-action value function  $Q_{\mathrm{jt}}$  as follows,
$$
\begin{array}{l} Q _ {\mathrm {j t}} (\boldsymbol {s}, \boldsymbol {u}) = \operatorname {M i x e r} \left(\left[ Q _ {i} (\boldsymbol {s}, u _ {i}; \theta_ {i}) \right] _ {i = 1} ^ {| A |}, \boldsymbol {s}; \Theta\right) (14) \\ = \sum_ {i = 1} ^ {| A |} w _ {i} (\boldsymbol {s}) Q _ {i} (\boldsymbol {s}, u _ {i}; \theta_ {i}) + b (\boldsymbol {s}), (15) \\ \end{array}
$$
Where  $w_{i}(s) \geq 0$  is a positive weight used to linearly decompose  $Q_{\mathrm{jt}}$  with the IGM principle being preserved. Same as REFIL,  $w_{i}(s)$  is computed via the attention mechanism to handle dynamic team composition. With  $Q_{\mathrm{jt}}$ , we can update the utilities and the mixer network by minimizing the following TD error:
$$
\mathcal {L} \left(\left[ \theta_ {i} \right] _ {i = 1} ^ {| A |}, \Theta\right) = \mathbb {E} _ {\mathcal {D}} \left[ Q _ {\mathrm {j t}} (\boldsymbol {s}, \boldsymbol {u}) - \left(r (\boldsymbol {s}, \boldsymbol {u}) + \gamma \left(\hat {Q} _ {\mathrm {j t}} \left(\boldsymbol {s} ^ {\prime}, \boldsymbol {u} ^ {\prime}\right) - \alpha \sum_ {i} ^ {| A |} \log \frac {\pi_ {i} \left(u _ {i} ^ {\prime} \mid \boldsymbol {s} ^ {\prime}\right)}{\pi_ {i} \left(u _ {i} ^ {\prime} \mid s _ {i} ^ {\prime +}\right)}\right)\right) \right], \tag {16}
$$
where  $\mathcal{D}$  is the replay buffer,  $\hat{Q}_{\mathrm{jt}}$  is the target network and  $u_{i}^{\prime}$  is sampled from the current policy  $\pi_i(u_i|s;\phi_i)$  of each agent. To accelerate the training with a fixed set of team compositions, we also incorporate the same imaginary objective based on random sub-group partitioning as REFIL for joint policy evaluation.
As we described in Section 4.3, the constrained individual policy improvement is achieved via an iterative update of  $\pi_i(u_i|s;\phi_i)$  and  $\pi_i(u_i|s_i^+;\omega_i)$ . For  $\pi_i(u_i|s_i^+;\omega_i)$ , we update it via the maximum likelihood estimation:
$$
\mathcal {L} \left(\omega_ {i}\right) = \mathbb {E} _ {\mathcal {D}} \left[ \mathbb {E} _ {s _ {i} ^ {*} \sim \hat {\rho} \left(s _ {i} ^ {*} \mid s _ {i} ^ {+}\right), u _ {i} \sim \pi_ {i} \left(u _ {i} \mid s _ {i} ^ {+}, s _ {i} ^ {*}\right)} \left[ \log \pi_ {i} \left(u _ {i} \mid s _ {i} ^ {+}\right) \right] \right]. \tag {17}
$$
For  $\pi_i(u_i|s;\phi_i)$ , we update it by minimizing the KL-divergence as follows,
$$
\mathcal {L} \left(\phi_ {i}\right) = \mathbb {E} _ {\mathcal {D}} \left[ \mathbb {E} _ {u _ {i} \sim \pi_ {i} \left(u _ {i} \mid \boldsymbol {s}\right)} \left[ \alpha \left(\log \frac {\pi_ {i} \left(u _ {i} \mid \boldsymbol {s}\right)}{\pi_ {i} \left(u _ {i} \mid s _ {i} ^ {+}\right)}\right) - Q _ {i} (\boldsymbol {s}, u _ {i}) \right] \right]. \tag {18}
$$
In this section, we evaluate MIPI in two different scenarios. One is a simple yet challenging matrix game, which we use to illustrate how mutual information may help to learn generalizable policies. Then, we evaluate MIPI in a complicated cooperative MARL scenario: StarCraft Multi-Agent Challenge (SMAC) (Samvelyan et al., 2019), comparing it against REFIL, AQMIX (Iqbal et al., 2021), CollaQ (Zhang et al., 2020) and MAPPO (Yu et al., 2022). More details about experiments, hyperparameters, and the learning curve of each algorithm are included in Appendix B and C. All results are presented using the mean and standard deviation of five runs with different random seeds.
5.1 An Illustrative Example: Matrix Game
We first use a matrix game to explain how mutual information works in solving generalization problems. In this game, we have two agents, and each of them can take two actions  $\{0,1\}$  and can take one of the two types  $\{A,B\}$ . During training, we train these two agents under team compositions  $(A,B)$  and  $(B,A)$ , where the corresponding payoff matrices are shown in Figure 2(a) and 2(b). However, during evaluation, we test their performance on team composition  $(B,B)$ , and have Figure 2(c) as the payoff matrix, which is different from training scenarios.
As we can see in Figure 2(a), 2(a), 2(c), we have different optimal joint actions in different team compositions. However, there exists a generalizable joint action  $(a_{1} = 0, a_{2} = 0)$  that can achieve consistent performance regardless of team compositions, even if it is not an optimal joint action in any team composition.
In Figure 2(d), we plot the evaluation results of three algorithms during training, which is evaluated on team composition  $(B, B)$ . These three algorithms are all the same except they receive different rewards: pure environmental reward, environmental reward combined with entropy, and environmental
(a) payoff matrix:  $(A,B)$
(b) payoff matrix:  $(B,A)$
(c) payoff matrix:  $(B,B)$
(d) learning curves
Figure 2: A matrix game with different team compositions: (a) (b) (c) payoff matrices for different team compositions; (d) learning curves of different methods on team composition  $(B, B)$ .
Table 1: Final performance on all SMAC maps. MIPI outperforms REFIL, AQMIX, and CollaQ in 8 out of 9 evaluation maps. We bold the best mean performance for each map.
#Agent Tasks	Algorithms	Training	Evaluation
3-5	6	7	8
SZ	MIPI	0.659±0.02	0.453±0.08	0.404±0.062	0.276±0.076
REFIL	0.674±0.038	0.441±0.103	0.352±0.078	0.236±0.103
AQMIX	0.528±0.044	0.343±0.105	0.291±0.084	0.182±0.058
CollaQ	0.588±0.03	0.366±0.086	0.314±0.076	0.198±0.097
MAPPO	0.256±0.01	0.129±0.019	0.148±0.031	0.036±0.015
CSZ	MIPI	0.548±0.032	0.42±0.102	0.297±0.112	0.261±0.09
REFIL	0.568±0.027	0.348±0.057	0.229±0.053	0.164±0.06
AQMIX	0.509±0.054	0.323±0.096	0.216±0.101	0.152±0.071
CollaQ	0.459±0.061	0.362±0.13	0.267±0.099	0.231±0.095
MAPPO	0.248±0.037	0.12±0.029	0.06±0.028	0.054±0.013
MMM	MIPI	0.548±0.023	0.495±0.054	0.447±0.041	0.467±0.067
REFIL	0.605±0.057	0.437±0.118	0.329±0.171	0.224±0.163
AQMIX	0.501±0.036	0.447±0.043	0.344±0.071	0.251±0.089
CollaQ	0.589±0.027	0.513±0.07	0.423±0.026	0.286±0.083
MAPPO	0.289±0.097	0.32±0.102	0.25±0.063	0.275±0.098
reward combined with mutual information. As we can see in Figure 2(d), with the help of mutual information, agents are able to resist the temptation of overfitting to the specific team composition and learn behavior that can generalize across different team compositions.
5.2 StarCraft Micromanagement Tasks
5.2.1 Performance
Further, we evaluate MIPI on SMAC with the map designed by Iqbal et al. (2021). We customize three different types of scenarios (SZ, CSZ, and MMM) based on this map for our experiments. In SZ scenarios, agents can take two different unit types, in CSZ and MMM, agents can take three different unit types. During training, the maps randomly initialize 3-5 agents and the same number of enemies at the start of each episode. During the evaluation, we use 6-8 agents and 6-8 enemies. Results are shown in Table 1. In general, MIPI outperforms the baselines in 8 out of 9 evaluation scenarios. When the evaluation scenario is similar to the training scenarios, the gap between MIPI and other baselines is relatively small, whereas, in the evaluation scenario that is very different from the training scenarios, the gap between MIPI and other baselines becomes larger. In terms of the training performance, REFIL achieves the best result in all three scenarios, as it does not consider the overfitting issue at all. However, the performance of MIPI is still at the same level as REFIL, which indicates that MIPI can achieve better zero-shot generalization without sacrificing the performance on the training set.
Table 2: Final performance on all SMAC maps. MIPI is compared with ablation baselines. We bold the best mean performance for each map.
#Agent Tasks	Algorithms	Training	Evaluation
3-5	6	7	8
SZ	MIPI	0.659±0.02	0.453±0.08	0.404±0.062	0.276±0.076
Value	0.621±0.042	0.336±0.075	0.275±0.114	0.154±0.052
Entropy	0.105±0.035	0.024±0.011	0.015±0.011	0.01±0.01
CSZ	MIPI	0.548±0.032	0.42±0.102	0.297±0.112	0.261±0.09
Value	0.542±0.059	0.368±0.083	0.207±0.076	0.172±0.112
Entropy	0.316±0.04	0.237±0.051	0.076±0.041	0.066±0.041
MMM	MIPI	0.548±0.023	0.495±0.054	0.447±0.041	0.467±0.067
Value	0.545±0.048	0.505±0.058	0.391±0.083	0.319±0.105
Entropy	0.265±0.034	0.2±0.085	0.16±0.064	0.075±0.044
5.2.2 Ablation
Although MIPI uses the random sub-group partitioning as in REFIL, it is an actor-critic structure, whereas REFIL uses only a critic. Therefore, one may question whether the improved generalization of MIPI is due to the introduction of mutual information, or simply due to the introduction of the actor. To eliminate such a concern, we build two ablation baselines, Value and Entropy, where all other perspectives are the same as MIPI, except they use pure environmental reward and entropy-augmented reward, respectively. As we can see in Table 2, MIPI also outperforms these two baselines, which demonstrates the importance of MI-augmented reward in MIPI.
In this paper, we propose MIPI, an MI-regularized multi-agent policy iteration algorithm to improve the generalization ability of agents under unseen team compositions. We first prove the convergence and optimality of our algorithm given a fixed marginal distribution, then we propose to use an imaginary distribution to approximate the dynamic marginal distribution to better approximate the original objective and incorporate the Blahut-Arimoto algorithm into the multi-agent policy iteration to optimize this approximate objective. We evaluate our algorithm in complex coordination scenarios, SMAC, and demonstrate that MIPI can achieve better zero-shot generalization results, without sacrificing the performance on the training set.
One potential limitation of this work is the introduction of approximation distorts the original mutual information augmented objective and breaks the convergences of multi-agent policy iteration. One possible solution to this problem is to seek alternative solutions using on-policy optimization methods (Schulman et al., 2015, 2017; Grudzien et al., 2022) to optimize the augmented objective.
This work was supported in part by NSF China under grant 62250068 and Tencent. The authors would like to thank the anonymous reviewers for their valuable comments.
Tong Wang, Jiahua Cao, and Azhar Hussain. Adaptive traffic signal control for large-scale scenario with cooperative group-based multi-agent reinforcement learning. Transportation research part C: emerging technologies, 125:103046, 2021.
Tonghan Wang, Jianhao Wang, Yi Wu, and Chongjie Zhang. Influence-based multi-agent exploration. arXiv preprint arXiv:1910.05512, 2019.
Zhaozhi Wang, Kefan Su, Jian Zhang, Huizhu Jia, Qixiang Ye, Xiaodong Xie, and Zongqing Lu. Multi-agent automated machine learning. arXiv preprint arXiv:2210.09084, 2022.
Muning Wen, Jakub Kuba, Runji Lin, Weinan Zhang, Ying Wen, Jun Wang, and Yaodong Yang. Multiagent reinforcement learning is a sequence modeling problem. Advances in Neural Information Processing Systems, 35:16509-16521, 2022.
Zifan Wu, Chao Yu, Deheng Ye, Junge Zhang, Hankz Hankui Zhuo, et al. Coordinated proximal policy optimization. Advances in Neural Information Processing Systems, 34:26437-26448, 2021.
Yiqin Yang, Xiaoteng Ma, Chenghao Li, Zewu Zheng, Qiyuan Zhang, Gao Huang, Jun Yang, and Qianchuan Zhao. Believe what you see: Implicit constraint approach for offline multi-agent reinforcement learning. Advances in Neural Information Processing Systems, 34:10299-10312, 2021.
Deheng Ye, Guibin Chen, Wen Zhang, Sheng Chen, Bo Yuan, Bo Liu, Jia Chen, Zhao Liu, Fuhao Qiu, Hongsheng Yu, et al. Towards playing full moba games with deep reinforcement learning. Advances in Neural Information Processing Systems, 33:621-632, 2020.
Chao Yu, Akash Velu, Eugene Vinitsky, Jiaxuan Gao, Yu Wang, Alexandre Bayen, and Yi Wu. The surprising effectiveness of ppo in cooperative multi-agent games. Advances in Neural Information Processing Systems, 35:24611-24624, 2022.
Tianhao Zhang, Yueheng Li, Chen Wang, Guangming Xie, and Zongqing Lu. Fop: Factorizing optimal joint policy of maximum-entropy multi-agent reinforcement learning. In International Conference on Machine Learning, pages 12491-12500. PMLR, 2021.
Tianjun Zhang, Huazhe Xu, Xiaolong Wang, Yi Wu, Kurt Keutzer, Joseph E Gonzalez, and Yuan-dong Tian. Multi-agent collaboration via reward attribution decomposition. arXiv preprint arXiv:2010.08531, 2020.
Lulu Zheng, Jiarui Chen, Jianhao Wang, Jiamin He, Yujing Hu, Yingfeng Chen, Changjie Fan, Yang Gao, and Chongjie Zhang. Episodic multi-agent reinforcement learning with curiosity-driven exploration. Advances in Neural Information Processing Systems, 34:3757-3769, 2021.
Tong Zhou, Dunbing Tang, Haihua Zhu, and Zequn Zhang. Multi-agent reinforcement learning for online scheduling in smart factories. Robotics and Computer-Integrated Manufacturing, 72: 102202, 2021.
Lemma 1 (Joint Policy Evaluation). Consider the modified Bellman backup operator  $\Gamma_{\pi_{\mathrm{jt}}} (5)$  and a mapping  $Q_{\mathrm{jt}}^{0}: S \times U \to \mathbb{R}$  with  $|U| < \infty$ , and define  $Q_{\mathrm{jt}}^{k+1} = \Gamma_{\pi_{\mathrm{jt}}} Q_{\mathrm{jt}}^{k}$ . Then, the sequence  $Q_{\mathrm{jt}}^{k}$  will converge to the joint  $Q$ -function of  $\pi_{\mathrm{jt}}$  as  $k \to \infty$ .
Proof. First, define the augmented reward $^2$  as:
$$
r _ {\pi_ {\mathrm {j t}}} (\pmb {s}, \pmb {u}) := r (\pmb {s}, \pmb {u}) - \alpha \mathbb {E} _ {\pmb {s ^ {\prime}}} \left[ \mathbb {E} _ {\pi_ {\mathrm {j t}}} \left[ \sum_ {i} \log \frac {\pi_ {i} (u _ {i} | \pmb {s ^ {\prime}})}{\pi_ {i} (u _ {i} | s _ {i} ^ {\prime +})} \right] \right].
$$
Then, rewrite the update rule as:
$$
Q _ {\mathrm {j t}} (\boldsymbol {s}, \boldsymbol {u}) \leftarrow r _ {\pi_ {\mathrm {j t}}} (\boldsymbol {s}, \boldsymbol {u}) + \gamma \mathbb {E} _ {\boldsymbol {s} ^ {\prime}, \boldsymbol {u} ^ {\prime} \sim \pi_ {\mathrm {j t}}} [ Q _ {\mathrm {j t}} (\boldsymbol {s} ^ {\prime}, \boldsymbol {u} ^ {\prime}) ].
$$
Last, apply the standard convergence results for policy evaluation (Sutton and Barto, 2018).
Lemma 2 (Individual Policy Improvement). Let  $\pi_i^{\mathrm{new}}$  be the optimizer of the maximization problem in (7). Then, we have  $Q_{\mathrm{jt}}^{\pi_{\mathrm{jt}}^{\mathrm{new}}}(s,\pmb {u})\geq Q_{\mathrm{jt}}^{\pi_{\mathrm{jt}}^{\mathrm{old}}}(s,\pmb {u})$  for all  $(s,u)\in |S|\times |U|$  with  $|U| < \infty$ , where  $\pi_{\mathrm{jt}}^{\mathrm{old}}(\pmb {u}|\pmb {s}) = \prod_i\pi_i^{\mathrm{old}}(u_i|\pmb {s})$  and  $\pi_{\mathrm{jt}}^{\mathrm{new}}(\pmb {u}|\pmb {s}) = \prod_i\pi_i^{\mathrm{new}}(u_i|\pmb {s})$ .
Proof. As  $\pi_i^{\mathrm{new}}$  optimizes (7), we can have:
$$
\mathbb {E} _ {\pi_ {i} ^ {\text {n e w}}} \left[ Q _ {i} ^ {\pi_ {i} ^ {\text {o l d}}} (\boldsymbol {s}, u _ {i}) - \alpha \log \frac {\pi_ {i} ^ {\text {n e w}} (u _ {i} | \boldsymbol {s})}{\pi_ {i} (u _ {i} | s _ {i} ^ {+})} \right] \geq \mathbb {E} _ {\pi_ {i} ^ {\text {o l d}}} \left[ Q _ {i} ^ {\pi_ {i} ^ {\text {o l d}}} (\boldsymbol {s}, u _ {i}) - \alpha \log \frac {\pi_ {i} ^ {\text {o l d}} (u _ {i} | \boldsymbol {s})}{\pi_ {i} (u _ {i} | s _ {i} ^ {+})} \right]. \tag {19}
$$
Since we assume that:
$$
Q _ {\mathrm {j t}} ^ {\pi_ {\mathrm {j t}}} (\boldsymbol {s}, \boldsymbol {u}) = \sum_ {i} w _ {i} (\boldsymbol {s}) * Q _ {i} ^ {\pi_ {i}} (\boldsymbol {s}, u _ {i}) + b (\boldsymbol {s}),
$$
$$
\begin{array}{l} \mathbb {E} _ {\boldsymbol {u} \sim \pi_ {\mathrm {j t}} ^ {\text {n e w}}} \left[ Q _ {\mathrm {j t}} ^ {\pi_ {\mathrm {j t}} ^ {\text {o l d}}} (\boldsymbol {s}, \boldsymbol {u}) - \alpha \sum_ {i} \log \frac {\pi_ {i} ^ {\text {n e w}} (u _ {i} | \boldsymbol {s})}{\pi_ {i} (u _ {i} | s _ {i} ^ {+})} \right] \\ = \mathbb {E} _ {\boldsymbol {u} \sim \pi_ {\mathrm {j t}} ^ {\text {n e w}}} \left[ \sum_ {i} w _ {i} (\boldsymbol {s}) * Q _ {i} ^ {\pi_ {i} ^ {\text {o l d}}} (\boldsymbol {s}, u _ {i}) + b (\boldsymbol {s}) - \alpha \sum_ {i} \log \frac {\pi_ {i} ^ {\text {n e w}} (u _ {i} | \boldsymbol {s})}{\pi_ {i} (u _ {i} | s _ {i} ^ {+})} \right] \\ = \sum_ {i} \mathbb {E} _ {u _ {i} \sim \pi_ {i} ^ {\mathrm {n e w}}} \left[ w _ {i} (\pmb {s}) * Q _ {i} ^ {\pi_ {i} ^ {\mathrm {o l d}}} (\pmb {s}, u _ {i}) - \alpha \log \frac {\pi_ {i} ^ {\mathrm {n e w}} (u _ {i} | \pmb {s})}{\pi_ {i} (u _ {i} | s _ {i} ^ {+})} \right] + b (\pmb {s}) \\ \geq \sum_ {i} \mathbb {E} _ {u _ {i} \sim \pi_ {i} ^ {\mathrm {o l d}}} \left[ w _ {i} (\boldsymbol {s}) * Q _ {i} ^ {\pi_ {i} ^ {\mathrm {o l d}}} (\boldsymbol {s}, u _ {i}) - \alpha \log \frac {\pi_ {i} ^ {\mathrm {o l d}} (u _ {i} | \boldsymbol {s})}{\pi_ {i} (u _ {i} | s _ {i} ^ {+})} \right] + b (\boldsymbol {s}) \\ = \mathbb {E} _ {\boldsymbol {u} \sim \pi_ {\mathrm {j t}} ^ {\mathrm {p l d}}} \left[ Q _ {\mathrm {j t}} ^ {\pi_ {\mathrm {j t}} ^ {\mathrm {o l d}}} (\boldsymbol {s}, \boldsymbol {u}) - \alpha \sum_ {i} \log \frac {\pi_ {i} ^ {\mathrm {o l d}} (u _ {i} | \boldsymbol {s})}{\pi_ {i} (u _ {i} | s _ {i} ^ {+})} \right] \\ = V _ {\mathrm {j t}} ^ {\pi_ {\mathrm {j t}} ^ {\mathrm {o l d}}} (\boldsymbol {s}), \tag {20} \\ \end{array}
$$
where the inequality is from plugging in (19).
2We assume  $\pi_i(u_i|s)$  and  $\pi_i(u_i|s_i^+)$  to be  $\epsilon$ -soft policy (Sutton and Barto, 2018) to avoid the log term being undefined.
Last, considering the modified Bellman equation, the following holds: