SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We have converted the relationship between global policies and values to the relationship between local policies and values. However, it is still unclear how to calculate the optimal local value functions in Eq. (9). Note that each local policy needs to satisfy  $\sum_{a_i\in \mathcal{A}}\pi_i^* (a_i|o_i) = 1$  in Eq. (9) in order to ensure it is well-defined, it is also worth noting that when each local policy  $\pi_i^*$  satisfies this self-normalization constraint, the optimal global policy  $\pi_{tot}^{*} = \prod_{i}\pi_{i}^{*}$  will also satisfy  $\sum_{a\in \mathcal{A}^n}\pi_{tot}^* (a|o) = 1$ . Thus, we only need to impose self-normalization constraints on local policies. Integrating the RHS of Eq. (9) over the local action space, and have:
$$
\mathbb {E} _ {a _ {i} \sim \mu_ {i}} \left[ \exp \left(\frac {1}{\alpha} w _ {i} (\boldsymbol {o}) \left(Q _ {i} ^ {*} \left(o _ {i}, a _ {i}\right) - V _ {i} ^ {*} \left(o _ {i}\right)\right)\right) \right] = 1 \tag {10}
$$
We have now established the global-to-local relationships among the optimal values and policies, however, one annoying issue is that the global and local behavior policies  $\mu_{tot}$  and  $\mu_{i}$  are typically unknown, and it is hard to estimate them accurately, especially when they are multi-modal. However, perhaps surprisingly, we show that it is possible to learn optimal local value functions in an implicit way without knowing either  $\mu_{tot}$  or  $\mu_{i}$ .
Proposition 4.4.  $V_{i}^{*}(o)$  can be obtained by solving the following convex optimization problem:
$$
\min  _ {V _ {i}} \mathbb {E} _ {a _ {i} \sim \mu_ {i}} \left[ \exp \left(\frac {w _ {i} (\boldsymbol {o})}{\alpha} \left(Q _ {i} ^ {*} \left(o _ {i}, a _ {i}\right) - V _ {i} \left(o _ {i}\right)\right)\right) + \frac {w _ {i} (\boldsymbol {o}) V _ {i} \left(o _ {i}\right)}{\alpha} \right] \tag {11}
$$
The proof follows that the first-order optimality condition of the above optimization objective (i.e., derivative with respect to  $V_{i}$  equals 0) is exactly the condition Eq. (10).
The above optimization problem provides a new learning objective for the local state-value function  $V_{i}$ . It is also worth noting that this objective actually corresponds to adding an equivalent implicit local value regularization on  $V_{i}$ . To see this, note that in addition to performing expected regression to let  $V_{i}$  fit  $Q_{i}$  by the first term, this objective also minimizes the second term  $w_{i}(\boldsymbol {o})V_{i}(o_{i}) / \alpha$  with respect to  $V_{i}$  on behavioral data  $\mu_{i}$ . This is similar to performing conservative optimal value estimation in existing offline RL literature [20, 27] that learns an underestimated value function to avoid the distributional shift.
Based on previous analysis, we are now ready to present our final algorithm, OMIGA. OMIGA consists of three supervised learning steps: learning local state-value function  $V_{i}$ , learning global and
Algorithm 1 Pseudocode of OMIGA
Require: Offline dataset  $\mathcal{D}$  . hyperparameter  $\alpha$
1: Initialize local state-value network  $V_{i}$ , local action-value network  $Q_{i}$  and its target network  $\bar{Q}_{i}$ , and policy network  $\pi_{i}$  for agent  $i = 1,2,\ldots n$ .
2: Initialize the weight function network  $w$  and  $b$ .
3: for  $t = 1, \dots,$  max-value-iteration do
4: Sample batch transitions  $(\pmb{o},\pmb{a},r,\pmb{o}^{\prime})$  from  $\mathcal{D}$
5: Update local state-value function  $V_{i}(o_{i})$  for each agent  $i$  via Eq. (12).
6: Compute  $V_{tot}(\pmb{o}')$ ,  $Q_{tot}(\pmb{o},\pmb{a})$  via Eq. (8).
7: Update local action-value network  $Q_{i}(o_{i},a_{i})$ , weight function network  $w(\pmb {o})$  and  $b(o)$  with objective Eq. (13).
8: Update local policy network  $\pi_{i}$  for each agent  $i$  via Eq. (14).
9: Soft update target network  $\bar{Q}_i(o_i, a_i)$  by  $\bar{Q}_i(o_i, a_i)$  for each agent  $i$ .
local Q-value functions, and learning local policies  $\pi_{i}$ . We can use Eq. (11) to learn the optimal local state-value function  $V_{i}^{*}$  as:
$$
\min  _ {V _ {i}} \mathbb {E} _ {\left(o _ {i}, a _ {i}\right) \sim \mathcal {D}} \left[ \exp \left(\frac {w _ {i} (\boldsymbol {o})}{\alpha} \left(Q _ {i} \left(o _ {i}, a _ {i}\right) - V _ {i} \left(o _ {i}\right)\right)\right) + \frac {w _ {i} (\boldsymbol {o}) V _ {i} \left(o _ {i}\right)}{\alpha} \right] \tag {12}
$$
With the learned local  $V_{i}^{*}$ , we can jointly optimize the local Q-value function  $Q_{i}$ , the weight and offset function  $w_{i}$  and  $b$  ( $i = 1, \dots, n$ ) with the following objective, where  $Q_{tot}$  and  $V_{tot}$  are computed based on the value decomposition in Eq. (8).
$$
\min  _ {\substack {Q _ {i}, w _ {i}, b \\ i = 1, \dots , n}} \mathbb {E} _ {(o, a, o ^ {\prime}) \sim \mathcal {D}} \left[ \left(r (o, a) + \gamma V _ {t o t} \left(o ^ {\prime}\right) - Q _ {t o t} (o, a)\right) ^ {2} \right] \tag{13}
$$
After obtaining the optimal local value functions, we can further learn the local policies  $\pi_{i}$  by minimizing the KL divergence between  $\pi_{i}$  and  $\pi_{i}^{*}$  in Eq. (9) [44], which yields the following learning objective:
$$
\left. \max  _ {\pi_ {i}} \mathbb {E} _ {\left(o _ {i}, a _ {i}\right) \sim \mathcal {D}} \left[ \exp \left(\frac {w _ {i} (\boldsymbol {o})}{\alpha} \left(Q _ {i} \left(o _ {i}, a _ {i}\right) - V _ {i} \left(o _ {i}\right)\right)\right) \cdot \log \pi_ {i} \left(a _ {i} \mid o _ {i}\right) \right] \right. \tag {14}
$$
Through the above three learning steps, we convert the initial intractable global behavior regularization in multi-agent POMDP to tractable implicit value regularizations at the local level. Moreover, all three steps are learned in a completely in-sample manner, which performs supervised learning only on dataset samples without the involvement of potentially OOD policy-generated actions, thus greatly improving the training stability. We summarize the psuedocode of OMIGA in Algorithm 1.
Discussion with prior works OMIGA draws connections with several prior works such as ICQ [11], DMAC [45] and IVR [10]. ICQ bears some similarities with OMIGA, however, ICQ needs to estimate the normalizing partition function  $Z(s) = \sum_{a} \mu(a|s) \exp \left( \frac{Q(s, a)}{\alpha} \right)$  to compute the weight in learning both  $J_{\pi}$  and  $J_{Q}$ .  $Z$  is estimated by learning an auxiliary behavior model or approximating with softmax operation over a mini-batch, which is hard to compute accurately, especially in the continuous action space. OMIGA uses Eq (11) to provide a new learning objective, it only needs to solve  $V$  in a simple and elegant way by imposing self-normalization constraints. The difference between OMIGA and DMAC is that DMAC considers the KL divergence between current policy and previous policy, so that it can compute the KL divergence explicitly. While OMIGA considers the KL divergence between current policy and behavior policy, the behavior policy is typically unknown and hard to estimate, hence OMIGA proposes a principled way to implicitly compute the KL divergence. Compared with IVR which proposes a general implicit regularized RL framework in the offline single-agent setting, OMIGA starts from a similar regularized framework but is designed for the offline multi-agent setting. Also, OMIGA transforms the regularization from global to local in the multi-agent setting, this provides valuable insights to the community, justifies why previous offline multi-agent methods that apply local behavior regularization can work, and also reveals their limitations.
Table 1: Average scores and standard deviations over 5 random seeds on the offline multi-agent MuJoCo tasks and offline SMAC tasks.
Multi-agent MuJoCo
Task	Dataset	BCQ-MA	CQL-MA	ICQ	OMAR	OMIGA(ours)
Hopper	expert	77.85±58.04	159.14±313.83	754.74±806.28	2.36±1.46	859.63±709.47
Hopper	medium	44.58±20.62	401.27±199.88	501.79±14.03	21.34±24.90	1189.26±544.30
Hopper	medium-replay	26.53±24.04	31.37±15.16	195.39±103.61	3.30±3.22	774.18±494.27
Hopper	medium-expert	54.31±23.66	64.82±123.31	355.44±373.86	1.44±0.86	709.00±595.66
Ant	expert	1317.73±286.28	1042.39±2021.65	2050.00±11.86	312.54±297.48	2055.46±1.58
Ant	medium	1059.60±91.22	533.90±1766.42	1412.41±10.93	-1710.04±1588.98	1418.44±5.36
Ant	medium-replay	950.77±48.76	234.62±1618.28	1016.68±53.51	-2014.20±844.68	1105.13±88.87
Ant	medium-expert	1020.89±242.74	800.22±1621.52	1590.18±85.61	-2992.80±6.95	1720.33±110.63
HalfCheetah	expert	2992.71±629.65	1189.54±1034.49	2955.94±459.19	-206.73±161.12	3383.61±552.67
HalfCheetah	medium	2590.47±1110.35	1011.35±1016.94	2549.27±96.34	-265.68±146.98	3608.13±237.37
HalfCheetah	medium-replay	-333.64±152.06	1998.67±693.92	1922.42±612.87	-235.42±154.89	2504.70±83.47
HalfCheetah	medium-expert	3543.70±780.89	1194.23±1081.06	2833.99±420.32	-253.84±63.94	2948.46±518.89
SMAC
Task	Dataset	BCQ-MA	CQL-MA	ICQ	OMAR	OMIGA(ours)
5m_vs_6m	good	7.76±0.15	8.08±0.21	7.87±0.30	7.40±0.63	8.25±0.37
5m_vs_6m	medium	7.58±0.10	7.78±0.10	7.77±0.3	7.08±0.51	7.92±0.57
5m_vs_6m	poor	7.61±0.36	7.43±0.10	7.26±0.19	7.27±0.42	7.52±0.21
2c_vs_64zg	good	19.13±0.27	18.48±0.95	18.82±0.17	17.27±0.78	19.15±0.32
2c_vs_64zg	medium	15.58±0.37	12.82±1.61	15.57±0.61	10.20±0.20	16.03±0.19
2c_vs_64zg	poor	12.46±0.18	10.83±0.51	12.56±0.18	11.33±0.50	13.02±0.66
6h_vs_8z	good	12.19±0.23	10.44±0.20	11.81±0.12	9.85±0.28	12.54±0.21
6h_vs_8z	medium	11.77±0.16	11.29±0.29	11.13±0.33	10.36±0.16	12.19±0.22
6h_vs_8z	poor	10.84±0.16	10.81±0.52	10.55±0.10	10.63±0.25	11.31±0.19
corridor	good	15.24±1.21	5.22±0.81	15.54±1.12	6.74±0.69	15.88±0.89
corridor	medium	10.82±0.92	7.04±0.66	11.30±1.57	7.26±0.71	11.66±1.30
corridor	poor	4.47±0.94	4.08±0.60	4.47±0.33	4.28±0.49	5.61±0.35
5.1 Offline Multi-Agent Datasets
We choose multi-agent MuJoCo [15] and StarCraft Multi-Agent Challenge (SMAC) [16] as our experiment environments. Multi-agent MuJoCo is a benchmark for continuous multi-agent robotic control, based on MuJoCo environment. Currently, there is still a lack of offline MARL datasets with continuous action. Thus, we provide an offline multi-agent MuJoCo dataset which is collected from HAPPO [46] algorithm. The dataset contains multiple different data types and the details of the dataset are shown in Appendix B. Another benchmark used in this paper is SMAC, a popular multi-agent cooperative environment for evaluating advanced MARL methods. The offline dataset we used is provided by Meng et al. [38], which is collected from the online trained MAPPO agents [47], and is the largest open offline dataset on SMAC. The offline dataset has three quality levels: good, medium, and poor. Besides, SMAC includes several StarCraft II multi-agent micromanagement maps. We consider 4 representative battle maps: 2 hard maps (5m_vs_6m, 2c_vs_64zg), and 2 super hard maps (6h_vs_8z, corridor).
We compare OMIGA with four recent offline MARL algorithms: the multi-agent version of BCQ [4] and CQL [20] (namely BCQ-MA and CQL-MA), ICQ [11] and OMAR [12]. BCQ-MA and CQL-MA use linear weighted value decomposition structure as Eq.(1) for the multi-agent setting. Details for the implementations of baseline algorithms and the hyperparameters used in OMIGA and baselines are shown in Appendix C.
5.3 Comparative Evaluation
Table 1 shows the mean and standard deviation of average returns for the offline Multi-agent MuJoCo and SMAC tasks. Each algorithm is evaluated by using 32 independent episodes and runs with 5 seeds for training. The results show that OMIGA outperforms all baselines and achieves state-of-the-art performance in most tasks.
For these multi-agent tasks, the credit assignment for multi-agent is complex and it is difficult to learn multi-agent cooperation in the offline setting. The global-to-local value regularization design of
(a) Analyses and ablations on weight function  $w_{i}$
Figure 1: Analyses and ablations.
(b) Analyses on different  $\alpha$
our method reflects the global regularization in the local value function learning implicitly, which leads to better performance than other baseline algorithms. Moreover, the state-value function and Q-value function in OMIGA are completely learned in an in-sample manner without the involvement of the agent policy, which also leads to better training stability and performance. Besides, compared with baseline algorithms, the local policy learning in OMIGA contains global information, and thus enjoys obvious advantages in tasks with continuous action space like multi-agent MuJoCo.
We also conduct experiments on more diverse datasets which are obtained by mixing datasets of various quality. The results are shown in Appendix D. On these mixed datasets, the behavior policies can be suboptimal and more heterogeneous. Therefore, it is difficult for algorithms such as BCQ-MA and OMAR to learn an accurate behavior policy, making implicit value learning with the regularization framework of OMIGA more appealing.
5.4 Analyses on Policy Learning with Global Information
As Eq. (12) and Eq. (14) show, the learning of both local state-value function  $V_{i}$  and local policy  $\pi_{i}$  in OMIGA contains global information, which is reflected in  $w_{i}(o)$ . Therefore, OMIGA can make the credit assignment during value and policy learning and leverage the global information to improve local policy. To verify the effect of this design, we transform OMIGA into two versions without global information, and then compare their performance on offline multi-agent MuJoCo and SMAC datasets. In OMIGA-w/o-w, local policy learning uses formula  $\max_{\pi_i}\mathbb{E}_{(o_i,a_i)\sim \mathcal{D}}[\exp (\frac{1}{\alpha} (Q_i(o_i,a_i) - V_i(o_i)))\cdot \log \pi_i(a_i|o_i)]$  and the weight function is not available during the policy learning. In OMIGA-local-w, the weight function is  $w_{i}(o_{i})$  that only contains local information, and local policy learning uses formula  $\max_{\pi_i}\mathbb{E}_{(o_i,a_i)\sim \mathcal{D}}[\exp (\frac{w_i(o_i)}{\alpha} (Q_i(o_i,a_i) - V_i(o_i)))\cdot \log \pi_i(a_i|o_i)]$ .
Figure 1(a) shows the experimental results on offline HalfCheetah datasets and 6h_vs_8z datasets. For these difficult multi-agent tasks, the cooperative relationships among agents are extremely complex, so global information is important to guide local policy learning. OMIGA-w/o-w and OMIGA-local-w lack global information, and policies are only learned at the local level, which causes worse performance and stability.
5.5 Analyses on the Regularization Hyperparameter
In OMIGA, hyperparameter  $\alpha$  is used to control the degree of regularization. The higher  $\alpha$  encourages the policy to stay closer to the behavioral distribution, and the lower  $\alpha$  makes the policy more aggressive and optimistic. Figure 1(b) shows the experiments about  $\alpha$  on the offline MuJoCo HalfCheetah task and SMAC 6h_vs_8z task. In the HalfCheetah task, too high and too low  $\alpha$  will lead to performance degradation. An appropriate  $\alpha$  can ensure that the policy is restricted near the dataset distribution and also does not lose performance due to being too conservative. In the 6h_vs_8z task, the dataset contains enough good transitions. Therefore, the degree of regularization should be higher to make the algorithm more conservative. Experimental results indicate that a higher  $\alpha$  brings better performance on such datasets. When  $\alpha$  is very small, the performance of the algorithm becomes very poor due to the lack of value regularization.
In this paper, we study the key challenge of the offline MARL problem. We start from the usage of global behavior constraints and smartly derive a global-to-local value regularization scheme under value decomposition. By doing so, we naturally get a new offline MARL algorithm, OMIGA, that in principle applies global-level regularization but actually imposes equivalent implicit local-level regularization. Our work reveals why the local-level regularization used by existing algorithms works and gives theoretical insights into their weaknesses and how to develop better algorithms. The global-to-local regularization design of OMIGA can capture global information and the impact of multi-agent credit assignment under the offline setting, which guarantees that the learned local policies are jointly optimal at the global level. One future work is to develop other offline MARL algorithms by using other choices of  $f$  functions.
This work is supported by National Key Research and Development Program of China under Grant (2022YFB2502904) and funding from Global Data Solutions Limited.
[1] Yongcan Cao, Wenwu Yu, Wei Ren, and Guanrong Chen. An overview of recent progress in the study of distributed multi-agent coordination. IEEE Transactions on Industrial Informatics, 9(1):427-438, 2013.
[2] Dayong Ye, Minjie Zhang, and Yun Yang. A multi-agent framework for packet routing in wireless sensor networks. Sensors, 15(5):10026-10047, 2015.
[3] Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemyslaw Debiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, et al. Dota 2 with large scale deep reinforcement learning. arXiv preprint arXiv:1912.06680, 2019.
[4] Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International conference on machine learning, pages 2052-2062. PMLR, 2019.
[5] Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. Advances in Neural Information Processing Systems, 32, 2019.
[6] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33:1179-1191, 2020.
[7] Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. International Conference on Learning Representations, 2021.
[8] Haoran Xu, Jiang Li, Jianxiong Li, and Xianyuan Zhan. A policy-guided imitation approach for offline reinforcement learning. In Advances in Neural Information Processing Systems, 2022.
[9] Jianxiong Li, Xianyuan Zhan, Haoran Xu, Xiangyu Zhu, Jingjing Liu, and Ya-Qin Zhang. When data geometry meets deep function: Generalizing offline reinforcement learning. In The Eleventh International Conference on Learning Representations, 2023.
[10] Haoran Xu, Li Jiang, Jianxiong Li, Zhuoran Yang, Zhaoran Wang, and Xianyuan Zhan. Offline with no ood actions: In-sample learning via implicit value regularization. In International Conference on Learning Representations, 2023.
[11] Yiqin Yang, Xiaoteng Ma, Li Chenghao, Zewu Zheng, Qiyuan Zhang, Gao Huang, Jun Yang, and Qianchuan Zhao. Believe what you see: Implicit constraint approach for offline multi-agent reinforcement learning. Advances in Neural Information Processing Systems, 34:10299-10312, 2021.
[12] Ling Pan, Longbo Huang, Tengyu Ma, and Huazhe Xu. Plan better amid conservatism: Offline multi-agent reinforcement learning with actor rectification. In International Conference on Machine Learning, pages 17221-17237. PMLR, 2022.
Proof. For any two global state-value functions  $V_{tot}^{1}$  and  $V_{tot}^{2}$ , let  $\pi_{tot}^{1}$  be the optimal global policy under  $\mathcal{T}_f^* V_{tot}^{1}$ , and we can get:
$$
\begin{array}{l} \left(\mathcal {T} _ {f} ^ {*} V _ {t o t} ^ {1}\right) (\boldsymbol {o}) - \left(\mathcal {T} _ {f} ^ {*} V _ {t o t} ^ {2}\right) (\boldsymbol {o}) \\ = \sum_ {\pmb {a}} \pi_ {t o t} ^ {1} (\pmb {a} | \pmb {o}) \left[ r + \gamma \mathbb {E} _ {\pmb {o} ^ {\prime}} \left[ V _ {t o t} ^ {1} (\pmb {o} ^ {\prime}) \right] - \alpha \log \left(\frac {\pi_ {t o t} ^ {1} (\pmb {a} | \pmb {o})}{\mu_ {t o t} (\pmb {a} | \pmb {o})}\right) \right] - \max _ {\pi_ {t o t}} \sum_ {\pmb {a}} \pi_ {t o t} (\pmb {a} | \pmb {o}) \left[ r + \gamma \mathbb {E} _ {\pmb {o} ^ {\prime}} \left[ V _ {t o t} ^ {2} (\pmb {o} ^ {\prime}) \right] - \alpha \log \left(\frac {\pi_ {t o t} (\pmb {a} | \pmb {o})}{\mu_ {t o t} (\pmb {a} | \pmb {o})}\right) \right] \\ \leq \sum_ {\boldsymbol {a}} \pi_ {t o t} ^ {1} (\boldsymbol {a} | \boldsymbol {o}) \left[ r + \gamma \mathbb {E} _ {\boldsymbol {o} ^ {\prime}} \left[ V _ {t o t} ^ {1} (\boldsymbol {o} ^ {\prime}) \right] - \alpha \log \left(\frac {\pi_ {t o t} ^ {1} (\boldsymbol {a} | \boldsymbol {o})}{\mu_ {t o t} (\boldsymbol {a} | \boldsymbol {o})}\right) \right] - \sum_ {\boldsymbol {a}} \pi_ {t o t} ^ {1} (\boldsymbol {a} | \boldsymbol {o}) \left[ r + \gamma \mathbb {E} _ {\boldsymbol {o} ^ {\prime}} \left[ V _ {t o t} ^ {2} (\boldsymbol {o} ^ {\prime}) \right] - \alpha \log \left(\frac {\pi_ {t o t} ^ {1} (\boldsymbol {a} | \boldsymbol {o})}{\mu_ {t o t} (\boldsymbol {a} | \boldsymbol {o})}\right) \right] \\ = \gamma \sum \pi_ {t o t} ^ {1} (\boldsymbol {a} | \boldsymbol {o}) \mathbb {E} _ {\boldsymbol {O} ^ {\prime}} \left[ V _ {t o t} ^ {1} \left(\boldsymbol {o} ^ {\prime}\right) - V _ {t o t} ^ {2} \left(\boldsymbol {o} ^ {\prime}\right) \right] \\ \leq \gamma \left\| V _ {t o t} ^ {1} - V _ {t o t} ^ {2} \right\| _ {\infty} \\ \end{array}
$$
$$
\left| \left| \mathcal {T} _ {f} ^ {*} V _ {t o t} ^ {1} - \mathcal {T} _ {f} ^ {*} V _ {t o t} ^ {2} \right| \right| _ {\infty} \leq \gamma \left| \left| V _ {t o t} ^ {1} - V _ {t o t} ^ {2} \right| \right| _ {\infty}
$$
Proof. For a behavior-regularized multi-agent POMDP with the behavior regularizer function  $f(\pi_{tot}, \mu_{tot}) = \log (\pi_{tot} / \mu_{tot})$ , its learning objective can be expressed as  $\max_{\pi_{tot}} \mathbb{E}\left[\sum_{t=0}^{\infty} \gamma^{t}(r(\pmb{o}_{t}, \pmb{a}_{t}) - \alpha \log(\pi_{tot}(\pmb{a}_{t} | \pmb{o}_{t}) / \mu_{tot}(\pmb{a}_{t} | \pmb{o}_{t})))\right]$ . Its Lagrangian function can obtain when the optimal global policy is written as follows:
$$
\begin{array}{l} L (\pi_ {t o t}, \beta , u) = \sum_ {\pmb {o}} d _ {\pi_ {t o t}} (\pmb {o}) \sum_ {\pmb {a}} \pi_ {t o t} (\pmb {a} | \pmb {o}) \left(Q _ {t o t} (\pmb {o}, \pmb {a}) - \alpha \log \left(\frac {\pi_ {t o t} (\pmb {a} | \pmb {o})}{\mu_ {t o t} (\pmb {a} | \pmb {o})}\right)\right) \\ - \sum_ {\boldsymbol {o}} d _ {\pi_ {t o t}} (\boldsymbol {o}) \left[ u (\boldsymbol {o}) \left(\sum_ {\boldsymbol {a}} \pi_ {t o t} (\boldsymbol {a} | \boldsymbol {o}) - 1\right) + \sum_ {\boldsymbol {a}} \beta (\boldsymbol {a} | \boldsymbol {o}) \pi_ {t o t} (\boldsymbol {a} | \boldsymbol {o}) \right], \\ \end{array}
$$
where  $d_{\pi_{tot}}$  is the stationary joint observation distribution of the global policy  $\pi_{tot}$ .  $u$  and  $\beta$  are Lagrangian multipliers for the equality and inequality constraints.
According to the Karush-Kuhn-Tucker (KKT) conditions where the derivative of the Lagrangian objective function with respect to the global policy is zero at the optimal solution, it follows that:
$$
Q _ {t o t} (\boldsymbol {o}, \boldsymbol {a}) - \alpha \left(\log \left(\frac {\pi_ {t o t} (\boldsymbol {a} | \boldsymbol {o})}{\mu_ {t o t} (\boldsymbol {a} | \boldsymbol {o})}\right) + 1\right) - u (\boldsymbol {o}) + \beta (\boldsymbol {a} | \boldsymbol {o}) = 0 \tag {15}
$$
$$
\sum_ {\pmb {a}} \pi_ {t o t} (\pmb {a} | \pmb {o}) = 1
$$
$$
\beta (\boldsymbol {a} | \boldsymbol {o}) \pi_ {t o t} (\boldsymbol {a} | \boldsymbol {o}) = 0
$$
$$
0 \leq \pi_ {t o t} (\boldsymbol {a} | \boldsymbol {o}) \leq 1 \text {a n d} 0 \leq \beta (\boldsymbol {a} | \boldsymbol {o})
$$
From Eq. (15), we can further solve the optimal global policy as:
$$
\pi_ {t o t} (\boldsymbol {a} | \boldsymbol {o}) = \mu_ {t o t} (\boldsymbol {a} | \boldsymbol {o}) \cdot \exp \left(\frac {Q _ {t o t} (\boldsymbol {o} , \boldsymbol {a}) - u (\boldsymbol {o}) + \beta (\boldsymbol {a} | \boldsymbol {o})}{\alpha} - 1\right)
$$
The above formula can be further simplified.  $\beta$  is the Lagrangian multiplier, and meets complementary slackness  $\beta (\pmb {a}|\pmb {o})\pi_{tot}(\pmb {a}|\pmb {o}) = 0$ . Considering the joint observation  $\pmb{o}$  is fixed,  $\exp \left(\frac{Q_{tot}(\pmb{o},\pmb{a}) - u(\pmb{o}) + \beta(\pmb{a}|\pmb{o})}{\alpha} -1\right)$  is always larger than 0. Therefore, for any positive probability action, its corresponding Lagrangian multiplier  $\beta (\pmb {a}|\pmb {o})$  is 0. Therefore,  $\pi_{tot}(\pmb {a}|\pmb {o})$  can be reformulated as: