SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
J _ {j} ^ {i} (\bar {\pi}) \leq J _ {j} ^ {i} (\pi) + E _ {s \sim \rho_ {\pi}, a \sim \bar {\pi}} \left[ A _ {j, \pi} ^ {i} \left(s, a ^ {i}\right) \right] + \frac {2 \gamma \max  _ {s , a ^ {i}} \left| A _ {j . \pi} ^ {i} \left(s , a ^ {i}\right) \right|}{(1 - \gamma) ^ {2}} D _ {K L} ^ {\max } (\bar {\pi}, \pi). \tag {13}
$$
It's to be noted that  $E_{s\sim \rho_{\pi},a\sim \bar{\pi}}[A_{j,\pi}^{i}(s,a^{i})] = E_{s\sim \rho_{\pi},a^{i}\sim \bar{\pi}^{i}}[A_{j,\pi}^{i}(s,a^{i})]$  as the actions of other agents than  $i$  do not change the value of the variable inside of the expectation. Furthermore,  $D_{KL}^{max}(\bar{\pi},\pi) = max_sD_{KL}(\bar{\pi} (\cdot |s),\pi (\cdot |s)) = max_s(\sum_{l = 1}^n D_{KL}(\bar{\pi}^l (\cdot |s),\pi^l (\cdot |s)))\leq \sum_{l = 1}^n max_sD_{KL}(\bar{\pi}^l (\cdot |s),\pi^l (\cdot |s))) = \sum_{l = 1}^n D_{KL}^{max}(\bar{\pi}^l,\pi^l)$ . Setting  $\nu_j^i = \frac{2\gamma max_{s,a^i}|A_{j,\pi}^i(s,a^i)|}{(1 - \gamma)^2}$ , we can finally obtain:
$$
J _ {j} ^ {i} (\bar {\pi}) \leq J _ {j} ^ {i} (\pi) + L _ {j, \pi} ^ {i} (\bar {\pi} ^ {i}) + \nu_ {j} ^ {i} \sum_ {l = 1} ^ {n} D _ {K L} ^ {\max } \left(\bar {\pi} ^ {l}, \pi^ {l}\right) \tag {14}
$$
The aforementioned equation is similar to Lemma 2 in MACPO, with the only difference being the order of policies in the Kullback-Leibler (KL) divergence term. However, this variation does not impact the subsequent derivations. To this end, we can establish the ultimate optimization problem presented in our work as follows:
$$
\underset {\pi_ {\theta} ^ {i _ {h}}} {\text {m a x i m i z e}} E _ {s \sim \rho_ {\pi_ {\theta_ {k}}}, a ^ {i _ {1: h - 1}} \sim \pi_ {\theta_ {k + 1}} ^ {i _ {1: h - 1}}, a ^ {i _ {h}} \sim \pi_ {i _ {h}}} \left[ A _ {\pi_ {\theta_ {k}}} ^ {i _ {h}} \left(s, a ^ {i _ {1: h - 1}}, a ^ {i _ {h}}\right) \right] \tag {15}
$$
$$
s. t. J _ {j} ^ {i _ {h}} \left(\boldsymbol {\pi} _ {\boldsymbol {\theta} _ {k}}\right) + E _ {s \sim \rho_ {\boldsymbol {\pi} _ {\boldsymbol {\theta} _ {k}}}, a ^ {i _ {h}} \sim \pi^ {i _ {h}}} \left[ A _ {j, \boldsymbol {\pi} _ {\boldsymbol {\theta} _ {k}}} ^ {i _ {h}} (s, a ^ {i _ {h}}) \right] \leq c _ {j} ^ {i _ {h}}, \forall j \in 1, \dots , m ^ {i _ {h}} \tag {16}
$$
$$
\bar {D} _ {K L} \left(\pi_ {\theta} ^ {i _ {h}}, \pi_ {\theta_ {k}} ^ {i _ {h}}\right) \leq \delta . \tag {17}
$$
where  $\bar{D}_{KL}(\pi_{\theta}^{i_h},\pi_{\theta_k}^{i_h})\triangleq E_{s\sim \rho_{\pi_{\theta_k}}}[D_{KL}(\pi_{\theta_k}^{i_h}(\cdot |s),\pi_\theta^{i_h}(\cdot |s))]$
We first demonstrate the optimization problem to be solved when finding optimization problem within nonparameterized policy space:
$$
\underset {\pi^ {i _ {h}}} {\text {m a x i m i z e}} E _ {s \sim \rho_ {\pi_ {\boldsymbol {\theta} _ {\boldsymbol {k}}}}, a ^ {i _ {1: h - 1}} \sim \pi_ {\theta_ {k + 1}} ^ {i _ {1: h - 1}}, a ^ {i _ {h}} \sim \pi_ {i _ {h}}} \left[ A _ {\pi_ {\boldsymbol {\theta} _ {\boldsymbol {k}}}} ^ {i _ {h}} \left(s, a ^ {i _ {1: h - 1}}, a ^ {i _ {h}}\right) \right] \tag {18}
$$
$$
s. t. J _ {j} ^ {i _ {h}} \left(\boldsymbol {\pi} _ {\boldsymbol {\theta} _ {k}}\right) + E _ {s \sim \rho_ {\boldsymbol {\pi} _ {\boldsymbol {\theta} _ {k}}}, a ^ {i _ {h}} \sim \pi^ {i _ {h}}} \left[ A _ {j, \boldsymbol {\pi} _ {\boldsymbol {\theta} _ {k}}} ^ {i _ {h}} (s, a ^ {i _ {h}}) \right] \leq c _ {j} ^ {i _ {h}}, \forall j \in 1, \dots , m ^ {i _ {h}} \tag {19}
$$
$$
\bar {D} _ {K L} \left(\pi^ {i _ {h}}, \pi_ {\theta_ {k}} ^ {i _ {h}}\right) \leq \delta \tag {20}
$$
Proof. We initiate our analysis by demonstrating the convexity of Problem (18-20) is convex w.r.t.  $\pi^{i_h}$ . Here we know that  $\pi_{\theta_k}$  and  $\theta_{k + 1}^{i_{1:h - 1}}$  are given and  $J_{j}^{i_{h}}(\pi_{\theta_{k}})$  remains constant w.r.t.  $\pi^{i_h}$ . In this
way,  $E_{s \sim \rho_{\pi_{\theta_k}}, a^{i_1:h-1} \sim \pi_{\theta_{k+1}}^{i_1:h-1}, a^{i_h} \sim \pi_{i_h}}[A_{\pi_{\theta_k}}^{i_h}(s, a^{i_1:h-1}, a^{i_h})]$  and  $E_{s \sim \rho_{\pi_{\theta_k}}, a^{i_h} \sim \pi^{i_h}}[A_{j, \pi_{\theta_k}}^{i_h}(s, a^{i_h})]$  are similar so that we only need to consider the latter one. We can divide this formula like this:  $E_{s \sim \rho_{\pi_{\theta_k}}, a^{i_h} \sim \pi^{i_h}}[A_{j, \pi_{\theta_k}}^{i_h}(s, a^{i_h})] = \sum_s \rho_{\pi_{\theta_k}}(s) \sum_{a^i h} \pi^{i_h}(a^{i_h}|s) A_{j, \pi_{\theta_k}}^{i_h}(s, a^{i_h})$ , where  $\rho_{\pi_{\theta_k}}(s)$  represents state visitation frequencies. We easily know that  $\rho_{\pi_{\theta_k}}(s)$  is not affected by  $\pi^{i_h}$ . Similarly, for each action of agent  $i_h$ , the  $A_{j, \pi_{\theta_k}}^{i_h}(s, a^{i_h})$  is also not relative to  $\pi^{i_h}$ . To this end, when  $\pi_{\theta_k}$  and  $\theta_{k+1}^{i_1:h-1}$  are given,  $E_{s \sim \rho_{\pi_{\theta_k}}, a^{i_h} \sim \pi^{i_h}}[A_{j, \pi_{\theta_k}}^{i_h}(s, a^{i_h})]$  is convex to  $\pi^{i_h}$ . Applying these analysis, the convexity of Equation 18 and 19 can be confirmed. Concerning constraint 20, it can be rewritten as  $\sum_s \rho_{\pi_{\theta_k}}(s) D_{KL}(\pi^{i_h}, \pi_{\theta_k}^{i_h})[s] \leq \delta$ . Notably, KL divergence is convex w.r.t. its first argument, hence constraint 20 can be represented as a linear combination of convex functions, confirming its convexity as well. As  $\pi_{\theta_k}^{i_h}$  fulfills constraint 19 and serves as an interior point within the set defined by constraint 20, therefore Slater's constraint qualification holds and strong duality holds.
Based on above discussion, we can solve for the optimal value for the problem (18 - 20)  $p^*$  by solving the corresponding dual problem. We define  $b_j^{i_h} = c_j^{i_h} - J_j^{i_h}(\pmb{\pi}_{\pmb{\theta}_k})$ , then
$$
\begin{array}{l} L \left(\pi , \lambda_ {j}, \nu_ {j}\right) = \lambda_ {j} \delta + \nu_ {j} b _ {j} ^ {i _ {h}} + E _ {s \sim \rho_ {\pi_ {\boldsymbol {\theta} _ {\boldsymbol {k}}}}} \left[ E _ {a ^ {i _ {1: h - 1}} \sim \pi_ {\theta_ {k + 1}} ^ {i _ {1: h - 1}}, a ^ {i _ {h}} \sim \pi^ {i _ {h}}} \left[ A _ {\pi_ {\boldsymbol {\theta} _ {\boldsymbol {k}}}} ^ {i _ {h}} \left(s, a ^ {i _ {1: h - 1}}, a ^ {i _ {h}}\right) \right] \right. \tag {21} \\ - \nu_ {j} E _ {a ^ {i _ {h}} \sim \pi^ {i _ {h}}} \left[ A _ {j, \pi_ {\theta_ {k}}} ^ {i _ {h}} (s, a ^ {i _ {h}}) \right] - \lambda_ {j} D _ {K L} \left(\pi^ {i _ {h}} | | \pi_ {\theta_ {k}} ^ {i _ {h}}\right) ] \\ \end{array}
$$
$$
p ^ {*} = \max  _ {\pi_ {i _ {h}} \in \Pi} \min  _ {\lambda_ {j}, \nu_ {j} \geq 0} L (\pi , \lambda_ {j}, \nu_ {j}) = \min  _ {\lambda_ {j}, \nu_ {j} \geq 0} \max  _ {\pi_ {i _ {h}} \in \Pi} L (\pi , \lambda_ {j}, \nu_ {j}) \tag {22}
$$
where we invoked strong duality in the second equality. According to the theory of convex optimization [44], if  $\pi^{i_h*},\lambda_j^*,\nu_j^*$  are optimal for 22,  $\pi^{i_h*}$  is also optimal for Problem 18-20.
Consider the inner maximization problem in 22, we can decompose this problem into separate problems, one for each  $s$ .
$$
\begin{array}{l} \underset {\pi_ {i _ {h}}} {\operatorname {m a x i m i z e}} E _ {a ^ {i _ {h}} \sim \pi^ {i _ {h}}} \left[ E _ {a ^ {i _ {1: h - 1}} \sim \pi_ {\theta_ {k + 1}} ^ {i _ {1: h - 1}}} \left[ A _ {\boldsymbol {\pi} _ {\boldsymbol {\theta} _ {\boldsymbol {k}}}} ^ {i _ {h}} \left(s, a ^ {i _ {1: h - 1}}, a ^ {i _ {h}}\right) \right] - \nu_ {j} A _ {j, \boldsymbol {\pi} _ {\boldsymbol {\theta} _ {\boldsymbol {k}}}} ^ {i _ {h}} \left(s, a ^ {i _ {h}}\right) \right. \tag {23} \\ - \lambda_ {j} \left(\log \pi^ {i _ {h}} (a | s) - \log \pi_ {\theta_ {k}} ^ {i _ {h}} (a | s)) \right], \sum \pi^ {i _ {h}} (a | s) = 1 \\ \end{array}
$$
As  $E_{a^{i_{1:h-1}} \sim \pi_{\theta_{k+1}}^{i_{1:h-1}}}\left[A_{\pi_{\theta_k}}^{i_h}(s, a^{i_{1:h-1}}, a^{i_h})\right]$  is irrelevant to  $\pi^{i_h}$ , we rename this term as  $\eta_{\pi_{\theta_k}}^{i_h}(s, a^{i_h})$  for simplicity. This is clearly a convex optimization problem which can be solved using a simple Lagrangian argument. We can then get
$$
G \left(\pi^ {i _ {h}}\right) = \sum_ {a} \pi^ {i _ {h}} (a | s) \left[ \eta_ {\pi_ {\theta_ {k}}} ^ {i _ {h}} (s, a ^ {i _ {h}}) - \nu_ {j} A _ {j, \pi_ {\theta_ {k}}} ^ {i _ {h}} (s, a ^ {i _ {h}}) - \lambda_ {j} \left(\log \pi^ {i _ {h}} (a | s) - \log \pi_ {\theta_ {k}} ^ {i _ {h}} (a | s)\right) + \zeta \right] - \zeta \tag {24}
$$
where  $\zeta$  is the Lagrange multiplier associated with the constraint  $\sum \pi^{i_h}(a|s) = 1$ . Differentiating  $G(\pi)$  w.r.t for some  $a$ :
$$
\frac {\partial G}{\partial \pi^ {i _ {h}} (a | s)} = \eta_ {\pi_ {\theta_ {k}}} ^ {i _ {h}} (s, a ^ {i _ {h}}) - \nu_ {j} A _ {j, \pi_ {\theta_ {k}}} ^ {i _ {h}} (s, a ^ {i _ {h}}) - \lambda_ {j} \left(\log \pi^ {i _ {h}} (a | s) - \log \pi_ {\theta_ {k}} ^ {i _ {h}} (a | s)\right) + \zeta \tag {25}
$$
Set 25 to 0 and similar to FOCOPS, we can know
$$
\pi^ {i _ {h} *} (a | s) = \frac {\pi_ {\theta_ {k}} ^ {i _ {h}} (a | s)}{Z _ {\lambda_ {j} , \nu_ {j}} (s)} \exp \left\{\frac {1}{\lambda_ {j}} \left(\eta_ {\pi_ {\theta_ {k}}} \left(s, a ^ {i _ {h}}\right) - \nu_ {j} A _ {j, \pi_ {\theta_ {k}}} ^ {i _ {h}} \left(s, a ^ {i _ {h}}\right)\right) \right\} \tag {26}
$$
where  $Z_{\lambda_j,\nu_j}(s)$  is the partition function that ensures  $\pi^{i_h*}$  to be a probability function, i.e.,  $\sum_{a}\pi^{i_h*}(a|s) = 1$ . Putting this  $\pi^*$  back into equation 22, we can get
$$
\begin{array}{l} p ^ {*} = \min  _ {\lambda_ {j}, \nu_ {j} \geq 0} \lambda_ {j} \delta + \nu_ {j} b _ {j} ^ {i _ {h}} + E _ {s \sim \rho_ {\boldsymbol {\pi} _ {\boldsymbol {\theta} _ {\boldsymbol {k}}}}, a ^ {i _ {h}} \sim \pi^ {i _ {h}} *} [ \eta_ {\boldsymbol {\pi} _ {\boldsymbol {\theta} _ {\boldsymbol {k}}}} ^ {i _ {h}} (s, a ^ {i _ {h}}) - \nu_ {j} A _ {j, \boldsymbol {\pi} _ {\boldsymbol {\theta} _ {\boldsymbol {k}}}} ^ {i _ {h}} (s, a ^ {i _ {h}}) - \lambda_ {j} (\log \pi^ {i _ {h} *} (a | s) - \log \pi_ {\theta_ {k}} ^ {i _ {h}} (a | s)) ] \\ = \min  _ {\lambda_ {j}, \nu_ {j} \geq 0} \lambda_ {j} \delta + \nu_ {j} b _ {j} ^ {i _ {h}} + E _ {s \sim \rho_ {\pi_ {\boldsymbol {\theta} _ {k}}}, a ^ {i _ {h}} \sim \pi^ {i _ {h}} *} [ \eta_ {\pi_ {\boldsymbol {\theta} _ {k}}} ^ {i _ {h}} (s, a ^ {i _ {h}}) - \nu_ {j} A _ {j, \pi_ {\boldsymbol {\theta} _ {k}}} ^ {i _ {h}} (s, a ^ {i _ {h}}) - \lambda_ {j} (\log \pi_ {\theta_ {k}} ^ {i _ {h}} (a | s) - \log Z _ {\lambda_ {j}, \nu_ {j}} \\ + \frac {1}{\lambda_ {j}} \left(\eta_ {\pi_ {\theta_ {k}}} \left(s, a ^ {i _ {h}}\right) - \nu_ {j} A _ {j, \pi_ {\theta_ {k}}} ^ {i _ {h}} \left(s, a ^ {i _ {h}}\right)\right) - \log \pi_ {\theta_ {k}} ^ {i _ {h}} (a | s)) ] \\ = \min  _ {\lambda_ {j}, \nu_ {j} \geq 0} \lambda_ {j} \delta + \nu_ {j} b _ {j} ^ {i _ {h}} + \lambda_ {j} E _ {s \sim \rho_ {\pi_ {\boldsymbol {\theta} _ {\boldsymbol {k}}}}, a ^ {i _ {h}} \sim \pi^ {i _ {h ^ {*}}}} [ \log Z _ {\lambda_ {j}, \nu_ {j}} (s) ] \\ \end{array}
$$
What's more, we give a simple description to show that for feasible policy  $\pi_{\theta_k}$ , the optimal policy update  $\pi^{i_h*}$  has an upper bound for worst-case guarantee for cost constraint satisfaction. For agent  $i_h$ , according to Equation 14, after getting the optimal joint update policy for all agents,  $J_j^i (\pmb{\pi}^*) \leq J_j^i (\pmb{\pi}_{\theta_k}) + L_j^i,\pi_{\theta_k}(\pi^{i_h*}) + \nu_j^{i_h}\sum_{l = 1}^n D_{KL}^{max}(\pi^{l*},\pi_{\theta_k}^l)$  can be obtained. According to the definition of  $L_{j,\pi}^{i}(\bar{\pi}^{i})$  and the constraint 16 in the optimization problem, we can know that  $J_{j}^{i}(\pmb{\pi}_{\theta_{k}}) + L_{j,\pi_{\theta_{k}}}^{i}(\pi^{i_{h}*}) \leq c_{j}^{i_{h}}$ , thus leading to  $J_{j}^{i}(\pmb{\pi}^{*}) \leq c_{j}^{i_{h}} + \nu_{j}^{i_{h}}\sum_{l = 1}^{n}D_{KL}^{max}(\pi^{l*},\pi_{\theta_k}^l)$ . In addition, we can know that the kl divergence between update policy and  $\pi_{\theta_k}$  for each agent  $l$  has an upper bound, which we call  $\delta^l$ . To this end, we achieve  $J_j^i (\pmb{\pi}^*) \leq c_j^{i_h} + \frac{2\gamma max_{s,a^i}|A_j^i_\pi(s,a^i)|}{(1 - \gamma)^2}\sum_{l = 1}^n\delta^l$  which is the upper bound for worst-case guarantee for cost constraint satisfaction. According to the result, we can know that with more agents, the upper bound for worst-case guarantee is higher, which means that optimization for more agents is more challenging, consistent with our intuition.
$$
\nabla_ {\theta} L (\theta) = E _ {s \sim \rho_ {\pi_ {\theta_ {k}}}} \left[ \nabla_ {\theta} D _ {K L} \left(\pi_ {\theta} ^ {i _ {h}} \right\lvert   | \pi^ {i _ {h} *}\right) [ s ] \tag {27}
$$
$$
\nabla_ {\theta} D _ {K L} \left(\pi_ {\theta} ^ {i _ {h}} \mid \mid \pi^ {i _ {h} *}\right) [ s ] = \nabla_ {\theta} D _ {K L} \left(\pi_ {\theta} ^ {i _ {h}} \mid \mid \pi_ {\theta_ {k}} ^ {i _ {h}}\right) - \frac {1}{\lambda_ {j}} E _ {a \sim \pi_ {\theta_ {k}} ^ {i _ {h}}} \left[ \frac {\nabla_ {\theta} \pi_ {\theta} ^ {i _ {h}} (a | s)}{\pi_ {\theta_ {k}} ^ {i _ {h}} (a | s)} \left(\eta_ {\pi_ {\theta_ {k}}} (s, a ^ {i _ {h}}) - \nu_ {j} A _ {j, \pi_ {\theta_ {k}}} ^ {i _ {h}} (s, a ^ {i _ {h}})\right) \right] \tag {28}
$$
Proof. Using the definition of KL divergence, we note that
$$
D _ {K L} \left(\pi_ {\theta} ^ {i _ {h}} \mid \mid \pi^ {i _ {h} *}\right) = - \sum_ {a} \pi_ {\theta} ^ {i _ {h}} (a | s) \log \pi^ {i _ {h} *} (a | s) + \sum_ {a} \pi_ {\theta} ^ {i _ {h}} (a | s) \log \pi_ {\theta} ^ {i _ {h}} (a | s) = H \left(\pi_ {\theta} ^ {i _ {h}}, \pi^ {i _ {h} *}\right) [ s ] - H \left(\pi_ {\theta} ^ {i _ {h}}\right) [ s ] \tag {29}
$$
where  $H(\pi_{\theta}^{i_h})[s]$  is the entropy and  $H(\pi_{\theta}^{i_h}, \pi^{i_h*})[s]$  is the cross-entropy. We expand the cross-entropy term which gives us:
$$
\begin{array}{l} H \left(\pi_ {\theta} ^ {i _ {h}}, \pi^ {i _ {h} *}\right) [ s ] = - \sum_ {a} \pi_ {\theta} ^ {i _ {h}} (a | s) \log \pi^ {i _ {h} *} (a | s) \\ = - \sum_ {a} \pi_ {\theta} ^ {i _ {h}} (a | s) * \log \left(\frac {\pi_ {\theta_ {k}} ^ {i _ {h}} (a | s)}{Z _ {\lambda_ {j} , \nu_ {j}}} e x p \left\{\frac {1}{\lambda_ {j}} \left(\eta_ {\pi_ {\theta_ {k}}} (s, a ^ {i _ {h}}) - \nu_ {j} A _ {j, \pi_ {\theta_ {k}}} ^ {i _ {h}} (s, a ^ {i _ {h}})\right) \right\}\right) \\ = - \sum_ {a} \pi_ {\theta} ^ {i _ {h}} (a | s) * \log \pi_ {\theta_ {k}} ^ {i _ {h}} (a | s) + \log Z _ {\lambda_ {j}, \nu_ {j}} (s) - \frac {1}{\lambda_ {j}} \sum_ {a} \pi_ {\theta} ^ {i _ {h}} (a | s) * \left(\eta_ {\pi_ {\theta_ {k}}} (s, a ^ {i _ {h}}) - \nu_ {j} A _ {j, \pi_ {\theta_ {k}}} ^ {i _ {h}} (s, a ^ {i _ {h}})\right) \\ \end{array}
$$
Then put this term back into Equation29:
$$
\begin{array}{l} D _ {K L} \left(\pi_ {\theta} ^ {i _ {h}} | | \pi^ {i _ {h} *}\right) [ s ] = - \sum_ {a} \pi_ {\theta} ^ {i _ {h}} (a | s) * \log \pi_ {\theta_ {k}} ^ {i _ {h}} (a | s) + \sum_ {a} \pi_ {\theta} ^ {i _ {h}} (a | s) \log \pi_ {\theta} ^ {i _ {h}} (a | s) + \log Z _ {\lambda_ {j}, \nu_ {j}} (s) \\ - \frac {1}{\lambda_ {j}} \sum_ {a} \pi_ {\theta} ^ {i _ {h}} (a | s) * (\eta_ {\pi_ {\theta_ {k}}} (s, a ^ {i _ {h}}) - \nu_ {j} A _ {j, \pi_ {\theta_ {k}}} ^ {i _ {h}} (s, a ^ {i _ {h}})) \\ = D _ {K L} \left(\pi_ {\theta} ^ {i _ {h}} \mid \mid \pi_ {\theta_ {k}} ^ {i _ {h}}\right) + \log Z _ {\lambda_ {j}, \nu_ {j}} (s) - \frac {1}{\lambda_ {j}} E _ {a \sim \pi_ {\theta_ {k}} ^ {i _ {h}}} \left[ \frac {\pi_ {\theta} ^ {i _ {h}} (a | s)}{\pi_ {\theta_ {k}} ^ {i _ {h}} (a | s)} \left(\eta_ {\pi_ {\theta_ {k}}} (s, a ^ {i _ {h}}) - \nu_ {j} A _ {j, \pi_ {\theta_ {k}}} ^ {i _ {h}} (s, a ^ {i _ {h}})\right) \right] \\ \end{array}
$$
In this way, take the gradient on both sides and we can get:
$$
\nabla_ {\theta} D _ {K L} \left(\pi_ {\theta} ^ {i _ {h}} \mid \mid \pi^ {i _ {h} *}\right) [ s ] = \nabla_ {\theta} D _ {K L} \left(\pi_ {\theta} ^ {i _ {h}} \mid \mid \pi_ {\theta_ {k}} ^ {i _ {h}}\right) - \frac {1}{\lambda_ {j}} E _ {a \sim \pi_ {\theta_ {k}} ^ {i _ {h}}} \left[ \frac {\nabla_ {\theta} \pi_ {\theta} ^ {i _ {h}} (a | s)}{\pi_ {\theta_ {k}} ^ {i _ {h}} (a | s)} \left(\eta_ {\pi_ {\theta_ {k}}} (s, a ^ {i _ {h}}) - \nu_ {j} A _ {j, \pi_ {\theta_ {k}}} ^ {i _ {h}} (s, a ^ {i _ {h}})\right) \right] \tag {30}
$$
D Proof of Corollary 2
$$
\frac {\partial L \left(\pi^ {i _ {h} *} , \lambda_ {j} , \nu_ {j}\right)}{\partial \nu_ {j}} = b _ {j} ^ {i _ {h}} - E _ {s \sim \rho_ {\boldsymbol {\pi} _ {\boldsymbol {\theta} _ {\boldsymbol {k}}}}, a ^ {i _ {h}} \sim \pi^ {i _ {h} *} (a | s)} \left[ A _ {j, \boldsymbol {\pi} _ {\boldsymbol {\theta} _ {\boldsymbol {k}}}} ^ {i _ {h}} (s, a ^ {i _ {h}}) \right] \tag {31}
$$
Proof. From the definition of  $L(\pi^{i_h*},\lambda_j,\nu_j)$  and above discussion, we can know that
$$
L \left(\pi^ {i _ {h} *}, \lambda_ {j}, \nu_ {j}\right) = \min  _ {\lambda_ {j}, \nu_ {j} \geq 0} \lambda_ {j} \delta + \nu_ {j} b _ {j} ^ {i _ {h}} + \lambda_ {j} E _ {s \sim \rho_ {\boldsymbol {\pi} _ {\boldsymbol {\theta} _ {\boldsymbol {k}}}}, a ^ {i _ {h}} \sim \pi^ {i _ {h} *}} [ \log Z _ {\lambda_ {j}, \nu_ {j}} (s) ] \tag {32}
$$
The first two terms is an affine function for  $\nu_{j}$  we focus on the expectation in the last term.
$$
\begin{array}{l} \frac {\partial \pi^ {i _ {h} *} (a | s)}{\partial \nu_ {j}} = \frac {\pi_ {\theta_ {k}} ^ {i _ {h}} (a | s)}{Z _ {\lambda_ {j} , \nu_ {j}} ^ {2} (s)} [ Z _ {\lambda_ {j}, \nu_ {j}} (s) * \frac {\partial e x p (\frac {1}{\lambda_ {j}} (\eta_ {\pi_ {\theta_ {k}}} (s , a ^ {i _ {h}}) - \nu_ {j} A _ {j , \pi_ {\theta_ {k}}} ^ {i _ {h}} (s , a ^ {i _ {h}})))}{\partial \nu_ {j}} \\ - \exp \left(\frac {1}{\lambda_ {j}} \left(\eta_ {\boldsymbol {\pi} _ {\boldsymbol {\theta} _ {k}}} (s, a ^ {i _ {h}}) - \nu_ {j} A _ {j, \boldsymbol {\pi} _ {\boldsymbol {\theta} _ {k}}} ^ {i _ {h}} (s, a ^ {i _ {h}})\right)\right) * \frac {\partial Z _ {\lambda_ {j} , \nu_ {j}} (s)}{\partial \nu_ {j}} ] \\ \end{array}
$$
For simplicity, we record  $\exp\left(\frac{1}{\lambda_j} (\eta_{\pi_{\theta_k}}(s,a^{i_h}) - \nu_jA_{j,\pi_{\theta_k}}^{i_h}(s,a^{i_h}))\right)$  as  $e(x)$ , so  $\pi^{i_h*(a|s)} = \frac{\pi_{\theta_k}^{i_h}(a|s)}{Z_{\lambda_j,\nu_j}(s)} * e(x)$ . In this way,
$$
\begin{array}{l} \frac {\partial \pi^ {i _ {h} *} (a | s)}{\partial v _ {j}} = \frac {\pi_ {\theta_ {k}} ^ {i _ {h}} (a | s)}{Z _ {\lambda_ {j} , \nu_ {j}} ^ {2} (s)} \left[ - \frac {A _ {j , \pi_ {\theta_ {k}}} ^ {i _ {h}} (s , a ^ {i _ {h}})}{\lambda_ {j}} Z _ {\lambda_ {j}, \nu_ {j}} (s) e (x) - e (x) \frac {\partial Z _ {\lambda_ {j} , \nu_ {j}} (s)}{\partial v _ {j}} \right] \tag {33} \\ = - \frac {A _ {j , \pi_ {\theta_ {k}}} ^ {i _ {h}} (s , a ^ {i _ {h}})}{\lambda_ {j}} \pi^ {i _ {h} *} (a | s) - \pi^ {i _ {h} *} (a | s) \frac {\partial \log Z _ {\lambda_ {j} , \nu_ {j}} (s)}{\partial \nu_ {j}} \\ \end{array}
$$
Therefore, the derivative of the expectation in the last term of  $L(\pi^{i_h*},\lambda_j,\nu_j)$  can be written as:
$$
\frac {\partial}{\partial \nu_ {j}} E _ {s \sim \rho_ {\pi_ {\boldsymbol {\theta} _ {k}}}, a ^ {i _ {h}} \sim \pi^ {i _ {h} *}} [ l o g Z _ {\lambda_ {j}, \nu_ {j}} (s) ] = E _ {s \sim \rho_ {\pi_ {\boldsymbol {\theta} _ {k}}}, a ^ {i _ {h}} \sim \pi^ {i _ {h}} _ {\theta_ {k}}} [ \frac {\partial}{\partial \nu_ {j}} (\frac {\pi^ {i _ {h} * (a | s)}}{\pi_ {\theta_ {k}} ^ {i _ {h}} (a | s)} l o g Z _ {\lambda_ {j}, \nu_ {j}} (s)) ]
$$
$$
\begin{array}{l} = E _ {s \sim \rho_ {\pi_ {\theta_ {k}}}, a ^ {i _ {h}} \sim \pi_ {\theta_ {k}} ^ {i _ {h}}} \left[ \frac {1}{\pi_ {\theta_ {k}} ^ {i _ {h}} (a | s)} \left(\frac {\partial \pi^ {i _ {h} *} (a | s)}{\partial \nu_ {j}} \log Z _ {\lambda_ {j}, \nu_ {j}} (s) + \pi^ {i _ {h} *} (a | s) \frac {\partial \log Z _ {\lambda_ {j} , \nu_ {j}} (s)}{\partial \nu_ {j}}\right) \right] \\ = E _ {s \sim \rho_ {\pi_ {\boldsymbol {\theta} _ {k}}}, a ^ {i _ {h}} \sim \pi_ {\boldsymbol {\theta} _ {k}} ^ {i _ {h}}} \left[ \frac {\pi^ {i _ {h} *} (a | s)}{\pi_ {\boldsymbol {\theta} _ {k}} ^ {i _ {h}} (a | s)} \left(- \frac {A _ {j , \pi_ {\boldsymbol {\theta} _ {k}}} ^ {i _ {h}} (s , a ^ {i _ {h}})}{\lambda_ {j}} \log Z _ {\lambda_ {j}, \nu_ {j}} (s) - \frac {\partial \log Z _ {\lambda_ {j} , \nu_ {j}} (s)}{\partial \nu_ {j}} \log Z _ {\lambda_ {j}, \nu_ {j}} (s)\right) + \frac {\partial \log Z _ {\lambda_ {j} , \nu_ {j}} (s)}{\partial \nu_ {j}} \right] \\ = E _ {s \sim \rho_ {\pi_ {\boldsymbol {\theta} _ {k}}}, a ^ {i _ {h}} \sim \pi^ {i _ {h}} * (a | s)} \left[ - \frac {A _ {j , \pi_ {\boldsymbol {\theta} _ {k}}} ^ {i _ {h}} (s , a ^ {i _ {h}})}{\lambda_ {j}} \log Z _ {\lambda_ {j}, \nu_ {j}} (s) - \frac {\partial \log Z _ {\lambda_ {j} , \nu_ {j}} (s)}{\partial \nu_ {j}} \log Z _ {\lambda_ {j}, \nu_ {j}} (s) + \frac {\partial \log Z _ {\lambda_ {j} , \nu_ {j}} (s)}{\partial \nu_ {j}} \right] \\ \end{array}
$$
In addition, according to the definition of  $Z_{\lambda_j,\nu_j}$ , we can get:
$$
\begin{array}{l} \frac {\partial Z _ {\lambda_ {j} , \nu_ {j} (s)}}{\partial \nu_ {j}} = \frac {\partial}{\partial \nu_ {j}} \left(\sum_ {a} \pi_ {\theta_ {k}} ^ {i _ {h}} (a | s) e x p \left\{\frac {1}{\lambda_ {j}} \left(\eta_ {\pi_ {\theta_ {k}}} \left(s, a ^ {i _ {h}}\right) - \nu_ {j} A _ {j, \pi_ {\theta_ {k}}} ^ {i _ {h}} (s, a ^ {i _ {h}})\right) \right\}\right. \\ = - \sum_ {a} \pi_ {\theta_ {k}} ^ {i _ {h}} (a | s) \frac {A _ {j , \pi_ {\theta_ {k}}} ^ {i _ {h}} (s , a ^ {i _ {h}}))}{\lambda_ {j}} e (x) = - \sum_ {a} \frac {A _ {j , \pi_ {\theta_ {k}}} ^ {i _ {h}} (s , a ^ {i _ {h}}))}{\lambda_ {j}} \frac {\pi_ {\theta_ {k}} ^ {i _ {h}} (a | s)}{Z _ {\lambda_ {j} , \nu_ {j}} (s)} e (x) Z _ {\lambda_ {j}, \nu_ {j}} (s) \\ = - \sum_ {a} \frac {A _ {j , \pi_ {\boldsymbol {\theta} _ {\boldsymbol {k}}}} ^ {i _ {h}} (s , a ^ {i _ {h}}))}{\lambda_ {j}} \pi^ {i _ {h} *} (a | s) Z _ {\lambda_ {j}, \nu_ {j}} (s) \\ = - \frac {Z _ {\lambda_ {j} , \nu_ {j}} (s)}{\lambda_ {j}} E _ {a ^ {i _ {h}} \sim \pi^ {i _ {h}} *} [ A _ {j, \pi_ {\theta_ {k}}} ^ {i _ {h}} (s, a ^ {i _ {h}})) ] \\ \end{array}
$$
What's more,
$$
\frac {\partial \log Z _ {\lambda_ {j} , \nu_ {j}} (s)}{\partial \nu_ {j}} = \frac {\partial Z _ {\lambda_ {j} , \nu_ {j}} (s)}{\partial \nu_ {j}} \frac {1}{Z _ {\lambda_ {j} , \nu_ {j}} (s)} = - \frac {1}{\lambda_ {j}} E _ {a ^ {i _ {h}} \sim \pi^ {i _ {h} *}} \left[ A _ {j, \pi_ {\theta_ {k}}} ^ {i _ {h}} \left(s, a ^ {i _ {h}}\right) \right] \tag {34}
$$
Putting this result to above equation, we can get
$$
\begin{array}{l} \frac {\partial}{\partial \nu_ {j}} E _ {s \sim \rho_ {\boldsymbol {\pi} _ {\boldsymbol {\theta} _ {\boldsymbol {k}}}; a ^ {i _ {h}} \sim \pi^ {i _ {h}} *}} [ l o g Z _ {\lambda_ {j}, \nu_ {j}} (s) ] \\ = E _ {s \sim \rho_ {\pmb {\pi} _ {\pmb {\theta} _ {k}}}, a ^ {i _ {h}} \sim \pi^ {i _ {h} *} (a | s)} [ - \frac {A _ {j , \pmb {\pi} _ {\pmb {\theta} _ {k}}} ^ {i _ {h}} (s , a ^ {i _ {h}})}{\lambda_ {j}} l o g Z _ {\lambda_ {j}, \nu_ {j}} (s) + \frac {A _ {j , \pmb {\pi} _ {\pmb {\theta} _ {k}}} ^ {i _ {h}} (s , a ^ {i _ {h}})}{\lambda_ {j}} l o g Z _ {\lambda_ {j}, \nu_ {j}} (s) - \frac {A _ {j , \pmb {\pi} _ {\pmb {\theta} _ {k}}} ^ {i _ {h}} (s , a ^ {i _ {h}})}{\lambda_ {j}} ] \\ = - \frac {1}{\lambda_ {j}} E _ {s \sim \rho_ {\boldsymbol {\pi} _ {\boldsymbol {\theta} _ {\boldsymbol {k}}}}, a ^ {i _ {h}} \sim \pi^ {i _ {h} *} (a | s)} \left[ A _ {j, \boldsymbol {\pi} _ {\boldsymbol {\theta} _ {\boldsymbol {k}}}} ^ {i _ {h}} (s, a ^ {i _ {h}}) \right] \\ \end{array}
$$
To sum up, the derivative of  $\nu_{j}$  to function  $L(\pi^{i_h*},\lambda_j,\nu_j)$  can be written:
$$
\frac {\partial L \left(\pi^ {i _ {h} *} , \lambda_ {j} , \nu_ {j}\right)}{\partial \nu_ {j}} = b _ {j} ^ {i _ {h}} - E _ {s \sim \rho_ {\pi_ {\boldsymbol {\theta} _ {k}}}, a ^ {i _ {h}} \sim \pi^ {i _ {h} *} (a | s)} \left[ A _ {j, \pi_ {\boldsymbol {\theta} _ {k}}} ^ {i _ {h}} (s, a ^ {i _ {h}}) \right] \tag {35}
$$
where  $b_{j}^{i_{h}} = c_{j}^{i_{h}} - J_{j}^{i_{h}}(\pmb{\pi}_{\pmb{\theta}_{k}})$ . In this way, we can update  $\nu_{j}$  by  $\nu_{j} \gets \text{proj}_{\nu_{j}}[\nu_{j} - \alpha(c_{j}^{i_{h}} - J_{j}^{i_{h}}(\pmb{\pi}_{\pmb{\theta}_{k}}))]$
E Procedure of MAFOCOPS
In this section, we describe the procedure of our algorithm, outlined in Algorithm 1. To be noted, hyperparameters for each agent are identical throughout the algorithm.
Algorithm 1 MAFOCOPS