SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} D _ {C Q L} ^ {C F} (s) = \Sigma_ {a} \boldsymbol {\pi} (\boldsymbol {a} | s) \left[ \Sigma_ {i = 1} ^ {n} \lambda_ {i} \frac {\mu^ {i} \left(a ^ {i} \mid s\right)}{\pi_ {\beta} ^ {i} \left(a ^ {i} \mid s\right)} - 1 \right] (A.3) \\ = \Sigma_ {i = 1} ^ {n} \lambda_ {i} \left[ \Sigma_ {a ^ {i}} \pi^ {i} \left(a ^ {i} | s\right) \left[ \frac {\mu^ {i} \left(a ^ {i} | s\right)}{\pi_ {\beta} ^ {i} \left(a ^ {i} | s\right)} - 1 \right] \right] (A.4) \\ = \Sigma_ {i = 1} ^ {n} \lambda_ {i} \left[ \Sigma_ {a ^ {i}} \left(\pi^ {i} \left(a ^ {i} | s\right) - \pi_ {\beta} ^ {i} \left(a ^ {i} | s\right) + \pi_ {\beta} ^ {i} \left(a ^ {i} | s\right)\right) \left[ \frac {\mu^ {i} \left(a ^ {i} | s\right)}{\pi_ {\beta} ^ {i} \left(a ^ {i} | s\right)} - 1 \right] \right] (A.5) \\ = \Sigma_ {i = 1} ^ {n} \lambda_ {i} \left[ \Sigma_ {a ^ {i}} \left(\pi^ {i} \left(a ^ {i} | s\right) - \pi_ {\beta} ^ {i} \left(a ^ {i} | s\right)\right) \left[ \frac {\pi^ {i} \left(a ^ {i} | s\right) - \pi_ {\beta} ^ {i} \left(a ^ {i} | s\right)}{\pi_ {\beta} ^ {i} \left(a ^ {i} | s\right)} \right] + \Sigma_ {a ^ {i}} \pi_ {\beta} ^ {i} \left(a ^ {i} | s\right) \left[ \frac {\mu^ {i} \left(a ^ {i} | s\right)}{\pi_ {\beta} ^ {i} \left(a ^ {i} | s\right)} - 1 \right] \right] (A.6) \\ = \Sigma_ {i = 1} ^ {n} \lambda_ {i} \left[ \Sigma_ {a ^ {i}} \left[ \frac {\left(\pi^ {i} \left(a ^ {i} | s\right) - \pi_ {\beta} ^ {i} \left(a ^ {i} | s\right)\right) ^ {2}}{\pi_ {\beta} ^ {i} \left(a ^ {i} | s\right)} \right] + 0 \right] \text {s i n c e ,} \forall i, \Sigma_ {a ^ {i}} \pi^ {i} \left(a ^ {i} | s\right) = \Sigma_ {a ^ {i}} \pi_ {\beta} ^ {i} \left(a ^ {i} | s\right) = 1 (A.7) \\ \geq 0 (A.8) \\ \end{array}
$$
As shown above, the  $D_{CQL}^{CF}(s) \geq 0$ , and  $D_{CQL}^{CF}(s) = 0$ , iff  $\pi^i(a^i|s) = \pi_\beta^i(a^i|s)$ . This implies that each value iterate incurs some underestimation, i.e.  $\hat{V}^{k+1}(s) \leq \mathcal{T}^\pi \hat{V}^k(s)$ .
We can compute the fixed point of the recursion in Equation A.2 and get the following estimated policy value:
$$
\hat {V} ^ {\pi} (s) = V ^ {\pi} (s) - \alpha \left[ \left(I - \gamma P ^ {\pi}\right) ^ {- 1} \Sigma_ {a} \pi (\boldsymbol {a} | s) \left[ \Sigma_ {i = 1} ^ {n} \lambda_ {i} \frac {\mu^ {i} \left(a ^ {i} | s\right)}{\tilde {\pi} _ {\beta} ^ {i} \left(a ^ {i} | s\right)} - 1 \right] \right] (s) \tag {A.9}
$$
Because the  $(I - \gamma P^{\pi})^{-1}$  is non negative and the  $D_{CQL}^{CF}(s)\geq 0$ , it's easily to prove that in the absence of sampling error, Theorem 4.1 gives a lower bound.
Incorporating sampling error. According to the conclusion in Kumar et al. [21], we can directly write down the result with sampling error as follows:
$$
\hat {V} ^ {\pi} (s) \leq V ^ {\pi} (s) - \alpha \left[ \left(I - \gamma P ^ {\pi}\right) ^ {- 1} \Sigma_ {a} \pi (\boldsymbol {a} | s) \left[ \Sigma_ {i = 1} ^ {n} \lambda_ {i} \frac {\mu^ {i} \left(a ^ {i} | s\right)}{\hat {\pi} _ {\beta} ^ {i} \left(a ^ {i} | s\right)} - 1 \right] \right] (s) + \left[ \left(I - \gamma P ^ {\pi}\right) ^ {- 1} \frac {C _ {r , T , \sigma} R _ {\max }}{(1 - \gamma) \sqrt {| D |}} \right] \tag {A.10}
$$
So, the statement of Theorem 4.1 with sampling error is proved. Please refer to the Sec.D.3 in Kumar et al. [21] For detailed proof. Besides, the choice of  $\alpha$  in this case to prevent overestimation is given by:
$$
\alpha \geq \max  _ {s, \boldsymbol {a} \in D} \frac {C _ {r , T , \sigma} R _ {\text {m a x}}}{(1 - \gamma) \sqrt {| D |}} \cdot \max  _ {s \in D} \left[ \Sigma_ {a} \boldsymbol {\pi} (\boldsymbol {a} | s) \left[ \Sigma_ {i = 1} ^ {n} \lambda_ {i} \frac {\mu^ {i} (a ^ {i} | s)}{\hat {\pi} _ {\beta} ^ {i} (a ^ {i} | s)} - 1 \right] \right] ^ {- 1} \tag {A.11}
$$
A.2 Proof of Theorem 4.2
Proof. According to the definition, we can get the formulation of  $D_{CQL}^{CF}(\pi, \beta)(s)$  and  $D_{CQL}(\pi, \beta)(s)$  as follow:
$$
\begin{array}{l} D _ {C Q L} ^ {C F} (\boldsymbol {\pi}, \boldsymbol {\beta}) (s) = \mathbb {E} _ {\boldsymbol {a} \sim \boldsymbol {\pi} (\cdot | s)} \left(\left[ \sum_ {i = 1} ^ {n} \lambda_ {i} \frac {\pi^ {i} \left(a ^ {i} | s\right)}{\beta^ {i} \left(a ^ {i} | s\right)} \right] - 1\right) (A.12) \\ = \sum_ {i = 1} ^ {n} \lambda_ {i} \left(\sum_ {a ^ {i}} \frac {\pi^ {i} \left(a ^ {i} | s\right) * \pi^ {i} \left(a ^ {i} | s\right)}{\beta^ {i} \left(a ^ {i} | s\right)}\right) - 1 \geq 0 (A.13) \\ \end{array}
$$
$$
\begin{array}{l} D _ {C Q L} (\boldsymbol {\pi}, \boldsymbol {\beta}) (s) = \mathbb {E} _ {\boldsymbol {a} \sim \boldsymbol {\pi} (\cdot | s)} \left(\left[ \frac {\boldsymbol {\pi} (\boldsymbol {a} | s)}{\boldsymbol {\beta} (\boldsymbol {a} | s)} \right] - 1\right) (A.14) \\ = \prod_ {i = 1} ^ {n} \left(\sum_ {a ^ {i}} \frac {\pi^ {i} \left(a ^ {i} | s\right) * \pi^ {i} \left(a ^ {i} | s\right)}{\beta^ {i} \left(a ^ {i} | s\right)}\right) - 1 \geq 0 (A.15) \\ \end{array}
$$
Then, by taking the logarithm of  $D_{CQL}(\pi ,\beta)(s)$  , we get:
$$
\ln \left(D _ {C Q L} (\boldsymbol {\pi}, \boldsymbol {\beta}) (s) + 1\right) = \sum_ {i = 1} ^ {n} \ln \left(\mathbb {E} _ {a ^ {i} \sim \pi^ {i (\cdot | s)}} \frac {\pi^ {i} \left(a ^ {i} | s\right)}{\beta^ {i} \left(a ^ {i} | s\right)}\right) \tag {A.16}
$$
As  $\Sigma_{i}\lambda_{i} = 1$ , it's obvious that
$$
\ln \left(D _ {C Q L} ^ {C F} (\boldsymbol {\pi}, \boldsymbol {\beta}) (s) + 1\right) \leq \ln \left(\sum_ {a ^ {j}} \frac {\pi^ {j} \left(a ^ {j} | s\right) * \pi^ {j} \left(a ^ {j} | s\right)}{\beta^ {j} \left(a ^ {j} | s\right)}\right), w h e r e j = \arg \max  _ {k} \mathbb {E} _ {\pi^ {k}} \frac {\pi^ {k}}{\beta^ {k}} \tag {A.17}
$$
By combining equation A.16 and inequation A.17, we get
$$
\begin{array}{l} \frac {D _ {C Q L} (\boldsymbol {\pi} , \boldsymbol {\beta}) (s) + 1}{D _ {C Q L} ^ {C F} (\boldsymbol {\pi} , \boldsymbol {\beta}) (s) + 1} \geq \exp \left(\sum_ {i = 1, i \neq j} ^ {n} \ln \left(\mathbb {E} _ {a ^ {i} \sim \pi^ {i (\cdot | s)}} \frac {\pi^ {i} \left(a ^ {i} | s\right)}{\beta^ {i} \left(a ^ {i} | s\right)}\right)\right) (A.18) \\ \geq \exp \left(\sum_ {i = 1, i \neq j} ^ {n} K L \left(\pi^ {i} (s) | | \beta^ {i} (s)\right)\right), w h e r e j = \arg \max  _ {k} \mathbb {E} _ {\pi^ {k}} \frac {\pi^ {k}}{\beta^ {k}} (A.19) \\ \end{array}
$$
the second inequality is derived from the Jensen's inequality. As the Kullback-Leibler Divergence is non-negative, it's obvious that  $D_{CQL}(\pi ,\beta)(s)\geq D_{CQL}^{CF}(\pi ,\beta)(s)$ , then we can simplify the left-hand side of this inequality:
$$
\frac {D _ {C Q L} (\boldsymbol {\pi} , \boldsymbol {\beta}) (s)}{D _ {C Q L} ^ {C F} (\boldsymbol {\pi} , \boldsymbol {\beta}) (s)} \geq \exp \left(\sum_ {i = 1, i \neq j} ^ {n} K L \left(\pi^ {i} (s) \mid \mid \beta^ {i} (s)\right)\right), w h e r e j = \underset {k} {\arg \max } \mathbb {E} _ {\pi^ {k}} \frac {\pi^ {k}}{\beta^ {k}} \tag {A.20}
$$
A.3 Proof of Equation 6
Proof. Similar to the proof of Lemma D.3.1 in CQL [21],  $Q$  is obtained by solving a recursive Bellman fixed point equation in the empirical MDP  $\hat{M}$ , with an altered reward,  $r(s,a) - \alpha \left[\sum_{i}\lambda_{i}\frac{\pi^{i}(a^{i}|s)}{\beta^{i}(a^{i}|s)} -1\right]$ , hence the optimal policy  $\pi^{*}(\pmb {a}|s)$  obtained by optimizing the value under the CFCQL Q-function equivalently is characterized via Eq. 6.
A.4 Proof of Theorem 4.3
Proof. Similar to Eq. 6,  $\pi_{MA}^{*}$  is equivalently obtained by solving:
$$
\boldsymbol {\pi} _ {M A} ^ {*} (\boldsymbol {a} | s) \leftarrow \arg \max  _ {\boldsymbol {\pi}} J (\boldsymbol {\pi}, \hat {M}) - \alpha \frac {1}{1 - \gamma} \mathbb {E} _ {s \sim d _ {\hat {M}} ^ {\boldsymbol {\pi}} (s)} [ D _ {C Q L} (\boldsymbol {\pi}, \boldsymbol {\beta}) (s) ]. \tag {A.21}
$$
Recall that  $\forall s,\pi ,\beta ,D_{CQL}(\pi ,\beta)(s)\geq 0$  . We have
$$
\begin{array}{l} J (\pmb {\pi} _ {M A} ^ {*}, \hat {M}) \geq J (\pmb {\pi} _ {M A} ^ {*}, \hat {M}) - \alpha \frac {1}{1 - \gamma} \mathbb {E} _ {s \sim d _ {\hat {M}} ^ {\pmb {\pi} _ {M A} ^ {*}} (s)} [ D _ {C Q L} (\pmb {\pi} _ {M A} ^ {*}, \beta) (s) ] \\ \geq J (\boldsymbol {\pi} ^ {*}, \hat {M}) - \alpha \frac {1}{1 - \gamma} \mathbb {E} _ {s \sim d _ {\hat {M}} ^ {\boldsymbol {\pi} ^ {*}} (s)} [ D _ {C Q L} (\boldsymbol {\pi} ^ {*}, \boldsymbol {\beta}) (s) ]. \\ \end{array}
$$
Then we give an upper bound of  $\mathbb{E}_{s\sim d_{M}^{\pi^{*}}(s)}[D_{CQL}(\pi^{*},\beta)(s)]$ . Due to the assumption that  $\beta^i$  is greater than  $\epsilon$  anywhere, we have
$$
\begin{array}{l} D _ {C Q L} (\boldsymbol {\pi}, \beta) (s) = \sum_ {\boldsymbol {a}} \boldsymbol {\pi} (\boldsymbol {a} | s) [ \frac {\boldsymbol {\pi} (\boldsymbol {a} | s)}{\beta (\boldsymbol {a} | s)} - 1 ] = \sum_ {\boldsymbol {a}} \boldsymbol {\pi} (\boldsymbol {a} | s) [ \frac {\boldsymbol {\pi} (\boldsymbol {a} | s)}{\prod_ {i = 1} ^ {n} \beta^ {i} (a ^ {i} | s)} - 1 ] \\ \leq \left(\frac {1}{\epsilon^ {n}} \sum_ {\boldsymbol {a}} \pi (\boldsymbol {a} | s) [ \pi (\boldsymbol {a} | s) ]\right) - 1 \leq \frac {1}{\epsilon^ {n}} - 1. \tag {A.23} \\ \end{array}
$$
Combining Eq. A.22 and Eq. A.23, we can get
$$
J \left(\boldsymbol {\pi} _ {M A} ^ {*}, \hat {M}\right) \geq J \left(\boldsymbol {\pi} ^ {*}, \hat {M}\right) - \frac {\alpha}{1 - \gamma} \left(\frac {1}{\epsilon^ {n}} - 1\right) \tag {A.24}
$$
Recall the sampling error proved in [21] and referred to above in (A.10), we can use it to bound the performance difference for any  $\pi$  on true and empirical MDP by
$$
| J (\pi , M) - J (\pi , \hat {M}) | \leq \frac {C _ {r , T , \delta} R _ {\max }}{(1 - \gamma) ^ {2}} \sum_ {s} \frac {\rho (s)}{\sqrt {| D (s) |}}, \tag {A.25}
$$
then let sampling error  $\coloneqq 2\cdot \frac{C_{r,T,\sigma}R_{max}}{(1 - \gamma)^2}\sum_s\frac{\rho(s)}{\sqrt{|D(s)|}}$ , and incorporate it into (A.24), we get
$$
J \left(\boldsymbol {\pi} _ {M A} ^ {*}, M\right) \geq J \left(\boldsymbol {\pi} ^ {*}, M\right) - \frac {\alpha}{1 - \gamma} \left(\frac {1}{\epsilon^ {n}} - 1\right) - \text {s a m p l i n g e r r o r} \tag {A.26}
$$
where sampling error is a constant dependent on the MDP itself and D. Note that during the proof we do not take advantage of the nature of  $\pi^{*}$ . Actually  $\pi^{*}$  can be replaced by any policy  $\pi$ . The reason we use  $\pi^{*}$  is that it can give that largest lower bound, resulting in the best policy improvement guarantee. Similarly,  $D_{CQL}^{CF}$  can be bounded by  $\frac{1}{\epsilon} - 1$ :
$$
\begin{array}{l} D _ {C Q L} ^ {C F} (\boldsymbol {\pi}, \boldsymbol {\beta}) (s) = \sum_ {i = 1} ^ {n} \lambda_ {i} \sum_ {a ^ {i}} \pi^ {i} (a ^ {i} | s) \left[ \frac {\pi^ {i} \left(a ^ {i} \mid s\right)}{\beta^ {i} \left(a ^ {i} \mid s\right)} - 1 \right] \\ \leq \left(\frac {1}{\epsilon} \sum_ {i = 1} ^ {n} \lambda_ {i} \sum_ {a ^ {i}} \pi^ {i} \left(a ^ {i} | s\right) \left[ \pi^ {i} \left(a ^ {i} | s\right) \right]\right) - 1 \tag {A.27} \\ \leq \frac {1}{\epsilon} \left(\sum_ {i = 1} ^ {n} \lambda_ {i}\right) - 1 = \frac {1}{\epsilon} - 1. \\ \end{array}
$$
We first show the theorem of safe policy improvement guarantee for MACQL and CFCQL, separately. Then we compare these two gaps.
MACQL has a safe policy improvement guarantee related to the number of agents  $n$ :
Theorem A.1. Given the discounted marginal state-distribution  $d_{\hat{M}}^{\pi}$ , we define  $\mathcal{B}(\pi, D) = \mathbb{E}_{s \sim d_{\hat{M}}^{\pi}}[\sqrt{D(\pi, \beta)(s) + 1}]$ . The policy  $\pmb{\pi}_{MA}^{*}(\pmb{a}|s)$  is a  $\zeta^{MA}$ -safe policy improvement over  $\beta$  in the actual MDP  $M$ , i.e.,  $J(\pmb{\pi}_{MA}^{*}, M) \geq J(\beta, M) - \zeta^{MA}$ , where  $\zeta^{MA} = 2\left(\frac{C_{r,\delta}}{1 - \gamma} + \frac{\gamma R_{\max} C_{T,\delta}}{(1 - \gamma)^2}\right)$ .
$$
\frac {\sqrt {| A |}}{\sqrt {| \mathcal {D} (s) |}} \mathcal {B} (\pmb {\pi} _ {M A} ^ {*}, D _ {C Q L}) + \frac {\alpha}{1 - \gamma} (\frac {1}{\epsilon^ {n}} - 1) - (J (\pmb {\pi} ^ {*}, \hat {M}) - J (\hat {\pmb {\beta}}, \hat {M})).
$$
Proof. We can first get a  $J(\pi_{MA}^{*}, \hat{M})$ -related policy improvement guarantee following the proof of Theorem 3.6 in Kumar et al. [21]:
$$
\begin{array}{l} J \left(\boldsymbol {\pi} _ {M A} ^ {*}, M\right) \geq J (\beta , M) - \left(2 \left(\frac {C _ {r , \delta}}{1 - \gamma} + \frac {\gamma R _ {\max } C _ {T , \delta}}{(1 - \gamma) ^ {2}}\right) \cdot \frac {\sqrt {| A |}}{\sqrt {| \mathcal {D} (s) |}} \mathcal {B} \left(\boldsymbol {\pi} _ {M A} ^ {*}, D _ {C Q L}\right) \right. \tag {A.28} \\ \left. - \left(J \left(\boldsymbol {\pi} _ {M A} ^ {*}, \hat {M}\right) - J (\hat {\boldsymbol {\beta}}, \hat {M})\right)\right) \\ \end{array}
$$
According to Eq. A.21,  $\pi_{MA}^{*}$  is obtained by optimizing  $J(\pi, \hat{M})$  with a  $D_{CQL}$ -related regularizer. And Theorem 4.3 shows that  $D_{CQL}$  can be extremely large when the team size expands, which may severely change the optimization objective and affects the shape of the optimization plane. Therefore,  $J(\pi_{MA}^{*}, \hat{M})$  may be extremely low, and keeping  $J(\pi_{MA}^{*}, \hat{M})$  in Eq. A.28 results in a mediocre policy improvement guarantee. To bound  $J(\pi_{MA}^{*}, \hat{M})$ , we introduce Eq. A.24 into Eq. A.28, we get the following:
$$
\begin{array}{l} J \left(\boldsymbol {\pi} _ {M A} ^ {*}, M\right) \geq J (\boldsymbol {\beta}, M) - \left(2 \left(\frac {C _ {r , \delta}}{1 - \gamma} + \frac {\gamma R _ {\max } C _ {T , \delta}}{(1 - \gamma) ^ {2}}\right) \cdot \frac {\sqrt {| A |}}{\sqrt {| \mathcal {D} (s) |}} \mathcal {B} \left(\boldsymbol {\pi} _ {M A} ^ {*}, D _ {C Q L}\right) \right. \tag {A.29} \\ \left. + \frac {\alpha}{1 - \gamma} \left(\frac {1}{\epsilon^ {n}} - 1\right) - \left(J \left(\boldsymbol {\pi} ^ {*}, \hat {M}\right) - J \left(\hat {\boldsymbol {\beta}}, \hat {M}\right)\right)\right) \\ \end{array}
$$
This complete the proof.
We can get a similar  $\zeta^{CF}$  satisfying  $J(\pi_{CF}^*, M) \geq J(\beta, M) - \zeta^{CF}$  for CFCQL, which is independent of  $n$ :
$$
\zeta^ {C F} = 2 \left(\frac {C _ {r , \delta}}{1 - \gamma} + \frac {\gamma R _ {\max } C _ {T , \delta}}{(1 - \gamma) ^ {2}}\right) \cdot \frac {\sqrt {| A |}}{\sqrt {| \mathcal {D} (s) |}} \mathcal {B} \left(\boldsymbol {\pi} _ {C F} ^ {*}, D _ {C Q L} ^ {C F}\right) + \frac {\alpha}{1 - \gamma} \left(\frac {1}{\epsilon} - 1\right) - (J (\boldsymbol {\pi} ^ {*}, \hat {M}) - J (\hat {\beta}, \hat {M})) \tag {A.30}
$$
Then we can prove Theorem 4.4.
Proof. Subtract  $\zeta^{CF}$  from  $\zeta^{MA}$ , and we get:
$$
\zeta^ {M A} - \zeta^ {C F} = 2 \left(\frac {C _ {r , \delta}}{1 - \gamma} + \frac {\gamma R _ {\max } C _ {T , \delta}}{(1 - \gamma) ^ {2}}\right) \frac {\sqrt {| A |}}{\sqrt {| \mathcal {D} (s) |}} \left(\mathcal {B} \left(\boldsymbol {\pi} _ {M A} ^ {*}, D _ {C Q L}\right) - \mathcal {B} \left(\boldsymbol {\pi} _ {C F} ^ {*}, D _ {C Q L} ^ {C F}\right)\right) + \frac {\alpha}{1 - \gamma} (\frac {1}{\epsilon^ {n}} - \frac {1}{\epsilon})
$$
Let the right side  $\geq 0$ , and we can get
$$
n \geq \log_ {\frac {1}{\epsilon}} \left[ \max  \left(1, \frac {1}{\epsilon} + \frac {2}{\alpha} \frac {\sqrt {| A |}}{\sqrt {| \mathcal {D} (s) |}} \left(C _ {r, \delta} + \frac {\gamma R _ {\max } C _ {T , \delta}}{1 - \gamma}\right) \cdot \left[ \mathcal {B} \left(\boldsymbol {\pi} _ {C F} ^ {*}, D _ {C Q L} ^ {C F}\right) - \mathcal {B} \left(\boldsymbol {\pi} _ {M A} ^ {*}, D _ {C Q L}\right) \right]\right) \right] \tag {A.32}
$$
According to Theorem 4.3,
$$
\mathcal {B} \left(\boldsymbol {\pi} _ {C F} ^ {*}, D _ {C Q L} ^ {C F}\right) = \mathbb {E} _ {s \sim d _ {\hat {M}} ^ {\boldsymbol {\pi} _ {C F} ^ {*}}} [ \sqrt {D _ {C Q L} ^ {C F} \left(\boldsymbol {\pi} _ {C F} ^ {*} , \boldsymbol {\beta}\right) (s) + 1} ] \leq \mathbb {E} _ {s \sim d _ {\hat {M}} ^ {\boldsymbol {\pi} _ {C F} ^ {*}}} [ \sqrt {\frac {1}{\epsilon} - 1 + 1} ] = \frac {1}{\sqrt {\epsilon}} \tag {A.33}
$$
In the meantime, we have
$$
\mathcal {B} \left(\pi_ {C F} ^ {*}, D _ {C Q L} ^ {C F}\right) = \mathbb {E} _ {s \sim d _ {\tilde {M}} ^ {\pi_ {M A} ^ {*}}} \left[ \sqrt {D _ {C Q L} \left(\pi_ {M A} ^ {*} , \beta\right) (s) + 1} \right] \geq \mathbb {E} _ {s \sim d _ {\tilde {M}} ^ {\pi_ {M A} ^ {*}}} \left[ \sqrt {D _ {C Q L} (\beta , \beta) (s) + 1} \right] = 1 \tag {A.34}
$$
Therefore, we can relax the lower bound of  $n$  to a constant that
$$
n \geq \log_ {\frac {1}{\epsilon}} \left(\frac {1}{\epsilon} + \frac {2}{\alpha} \frac {\sqrt {| A |}}{\sqrt {| \mathcal {D} (s) |}} \left(C _ {r, \delta} + \frac {\gamma R _ {\max } C _ {T , \delta}}{1 - \gamma}\right) \cdot \left(\frac {1}{\sqrt {\epsilon}} - 1\right)\right) \tag {A.35}
$$
B Implement Details
B.1 Derivation of the Update Rule
To utilize the Eq. 4 for policy optimization, following the analysis in the Section 3.2 in Kumar et al. [21], we formally define optimization problems over each  $\mu^i(a^i|s)$  by adding a regularizer  $R(\mu^i)$ . As shown below, we mark the modifications from the Eq. 4 in red.
$$
\begin{array}{l} \min  _ {Q} \max  _ {\mu} \alpha \left[ \sum_ {i = 1} ^ {n} \lambda_ {i} \mathbb {E} _ {s \sim \mathcal {D}, a ^ {i} \sim \mu^ {i}, \boldsymbol {a} ^ {- i} \sim \beta^ {- i}} [ Q (s, \boldsymbol {a}) ] - \mathbb {E} _ {s \sim \mathcal {D}, \boldsymbol {a} \sim \boldsymbol {\beta}} [ Q (s, \boldsymbol {a}) ] \right] \tag {B.36} \\ + \frac {1}{2} \mathbb {E} _ {s, \pmb {a}, s ^ {\prime} \sim \mathcal {D}} \left[ (Q (s, \pmb {a}) - \hat {\mathcal {T}} ^ {\pi} \hat {Q} _ {k} (s, \pmb {a})) ^ {2} \right] + \sum_ {i = 1} ^ {n} \lambda_ {i} R (\mu^ {i}), \\ \end{array}
$$
By choosing different regularizer, there are a variety of instances within CQL family. As recommended in Kumar et al. [21], we choose  $R(\mu^i)$  to be the KL-divergence against a Uniform distribution over action space, i.e.,  $R(\mu^i) = -D_{KL}(\mu^i, Unif(a^i))$ . Then we can get the following objective for  $\mu^i$ :
$$
\max  _ {\mu^ {i}} \mathbb {E} _ {x \sim \mu^ {i} (x)} [ f (x) ] + \mathcal {H} (\mu^ {i}), \quad s. t. \sum_ {x} \mu^ {i} (x) = 1, \mu^ {i} (x) \geq 0, \forall x, \tag {B.37}
$$
where  $\forall s, f(x) = Q(s,x,\pmb{a}^{-i})$ . The optimal solution is:
$$
\mu^ {i *} (x) = \frac {1}{Z} \exp (f (x)), \tag {B.38}
$$
where  $Z$  is the normalization factor, i.e.,  $Z = \sum_{x} \exp(f(x))$ . Plugging this back into Eq. B.36, we get:
$$
\begin{array}{l} \min  _ {Q} \alpha \mathbb {E} _ {s \sim \mathcal {D}} \left[ \sum_ {i = 1} ^ {n} \lambda_ {i} \mathbb {E} _ {\boldsymbol {a} ^ {- i} \sim \boldsymbol {\beta} ^ {- i}} \left[ \log \sum_ {a ^ {i}} \exp (Q (s, \boldsymbol {a})) \right] - \mathbb {E} _ {\boldsymbol {a} \sim \boldsymbol {\beta}} [ Q (s, \boldsymbol {a}) ] \right] \tag {B.39} \\ + \frac {1}{2} \mathbb {E} _ {s, \boldsymbol {a}, s ^ {\prime} \sim \mathcal {D}} \left[ \left(Q (s, \boldsymbol {a}) - \hat {\mathcal {T}} ^ {\pi_ {k}} \hat {Q} _ {k} (s, \boldsymbol {a})\right) ^ {2} \right]. \\ \end{array}
$$
B.2 Details for Computing  $\lambda$
To compute  $\lambda$ , we need an explicit expression of  $\pi^i$  and  $\beta^i$ . In the setting of discrete action space, as we use Q-learning,  $\pi^i$  can be expressed by the Boltzman policy, i.e.
$$
\pi^ {i} \left(a _ {j} ^ {i}\right) = \frac {\exp \left(\mathbb {E} _ {\boldsymbol {a} ^ {- i} \sim \boldsymbol {\beta} ^ {- i}} Q \left(s , a _ {j} ^ {i} , \boldsymbol {a} ^ {- i}\right)\right)}{\sum_ {k} \exp \left(\mathbb {E} _ {\boldsymbol {a} ^ {- i} \sim \boldsymbol {\beta} ^ {- i}} Q \left(s , a _ {k} ^ {i} , \boldsymbol {a} ^ {- i}\right)\right)} \tag {B.40}
$$
We use behaviour cloning to pre-train a parameterized  $\beta(s)$  with a three-level fully-connected network and MLE(Maximum Likelihood Estimation) loss.
With the explicit expression of  $\pi^i$  and  $\beta^i$ , we can directly compute  $\lambda$  with Eq. 8 and Eq. 9. While, in practice, we find the  $\mathbb{E}_{\pi^i}\frac{\pi^i(s)}{\beta^i(s)}$  may introduce extreme variance as its large scale and fluctuations,
which will hurt the performance. Instead, we take the logarithm of it and further reduced it to the Kullback-Leibler Divergence as follow:
$$
\forall i, s, \lambda_ {i} (s) = \frac {\exp \left(- \tau D _ {K L} \left(\pi^ {i} (s) \mid \mid \beta^ {i} (s)\right)\right)}{\sum_ {j = 1} ^ {n} \exp \left(- \tau D _ {K L} \left(\pi^ {j} (s) \mid \mid \beta^ {j} (s)\right)\right)}, \tag {B.41}
$$
For continuous action space, we use the deterministic policy like in MADDPG, whose policy distribution can be regarded as a Dirac delta function. Therefore, we approximate  $\mathbb{E}_{\pi^j}\frac{\pi^j(s)}{\beta^j(s)}$  by the following:
$$
\mathbb {E} _ {\pi^ {j}} \frac {\pi^ {j} (s)}{\beta^ {j} (s)} \approx \frac {1}{\beta^ {j} \left(\pi^ {j} (s) | s\right)} \tag {B.42}
$$
Then we need to obtain an explicit expression of  $\beta^i$ . We first train a VAE [17] from the dataset to obtain the lower bound of  $\beta^i$ . Let  $p_{\phi}(a,z|s)$  and  $q_{\varphi}(z|a,s)$  be the decoder and the encoder of the trained VAE, respectively. According to Wu et al. [48],  $\beta^j (a^j |s)$  can be explicitly estimated by (We omit the superscript  $j$  for brevity):
$$
\begin{array}{l} \log \beta_ {\phi} (a \mid s) = \log \mathbb {E} _ {q _ {\varphi} (z | a, s)} \left[ \frac {p _ {\phi} (a , z \mid s)}{q _ {\varphi} (z \mid a , s)} \right] \\ \approx \mathbb {E} _ {z ^ {(l)} q _ {\varphi} (z | a, s)} \left[ \log \frac {1}{L} \sum_ {l = 1} ^ {L} \frac {p _ {\phi} (a , z ^ {(l)} \mid s)}{q _ {\varphi} (z ^ {(l)} \mid a , s)} \right] \tag {B.43} \\ \stackrel {\mathrm {d e f}} {=} \widehat {\log \pi_ {\beta}} (a \mid s; \varphi , \phi , L). \\ \end{array}
$$
Therefore, we can sample from the VAE  $L$  times to estimate  $\beta^i$ . The sampling error reduces as  $L$  increases.
C.1 Tasks
Equal_Line is a multi-agent task which we design by simplify the space shape of Equal_Space to one-dimension. There are  $n$  agents and they are randomly initialized to the interval [0, 2]. The state space is a one-dimensional bounded region in  $[0, \max(10, 2 * n)]$  and the local action space is a discrete, eleven-dimensional space, i.e.  $[0, -0.01, -0.05, -0.1, -0.5, -1, 0.01, 0.05, 0.1, 0.5, 1]$ , which represents the moving direction and distance at each step. The reward is shared by the agents and formulated as  $10 * (n - 1) \frac{\text{min\_dis - last\_step\_min\_dis}}{\text{line\_length}}$ , which will spur the agents to cooperate to spread out and keep the same distance between each other.
For Multi-agent Particle Environment and Multi-agent Mujoco, we adopt the open-source implementations from Lowe et al. [27] and Peng et al. [35] respectively. And we use the datasets and the adversary agents provided by Pan et al. [34].
For StarCraft II Micromanagement Benchmark, we use the open-source implementation from Samvelyan et al.  $[40]^5$  and choose four maps with different difficulty and number of agents as the experimental scenarios, which is summarized in Table 4. We construct our own datasets with QMIX [37] by collecting training or evaluating data.
C.2 StarCraft II datasets collection
The datasets are made based on the training process or trained model of QMIX[37]. Specially, the Medium or Expert datasets are sampled by executing a partially-pretrained policy with a medium performance level or a fully-pretrained policy. The Medium - Replay datasets are exactly the replay buffer during training until the policy reaches the medium performance. The Mixed datasets are the equal mixture of Medium and Expert datasets. All datasets contain five thousand trajectories, except for the Medium - Replay.
<sup>3</sup>https://github.com/openai/multiagent-particle-envs
<sup>4</sup>https://github.com/schroederewitt/multiagent_mujoco
<sup>5</sup>https://github.com/oxwhirl/smac
Table 4: The details of tested maps in the StarCraft II micromanagement benchmark
Maps	Agents	Enemies	Difficulty
2s3z	2 Stalkers & 3 Zealots	2 Stalkers & 3 Zealots	Easy
3s_vs_5z	3 Stalkers	5 Zealots	Easy
5m_vs_6m	5 Marines	6 Marines	Hard
6h_vs_8z	6 Hydralisks	8 Zealots	Super Hard
C.3Baselines