SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Since  $D_{CQL}^{CF}$  can be written as  $\sum_{i}\lambda_{i}D_{CQL}(\pi^{i},\beta^{i})(s)$ , it can be regarded as a weighted average of each individual agents' policy deviations from its individual behavior policy. Therefore, the scale of  $D_{CQL}^{CF}$  is independent of the number of agents. Instead, Theorem 4.2 shows that both  $D_{CQL}$  and the ratio of  $D_{CQL}$  and  $D_{CQL}^{CF}$  explode exponentially as the number of agents  $n$  increases.
Then we discuss the influence of  $D_{CQL}^{CF}$  on the property of safe policy improvement. Define  $\hat{M}$  as the MDP induced by the transitions observed in the dataset. Let the empirical return  $J(\pi, \hat{M})$  be the discounted return for any policy  $\pi$  in  $\hat{M}$  and  $\hat{Q}$  be the fixed point of Equation 4. Then the optimal policy of CFCQL is equivalently obtained by solving:
$$
\pi_ {C F} ^ {*} (\boldsymbol {a} | s) \leftarrow \arg \max  _ {\boldsymbol {\pi}} J (\boldsymbol {\pi}, \hat {M}) - \alpha \frac {1}{1 - \gamma} \mathbb {E} _ {s \sim d _ {\hat {M}} ^ {\boldsymbol {\pi}} (s)} [ D _ {C Q L} ^ {C F} (\boldsymbol {\pi}, \boldsymbol {\beta}) (s) ]. \tag {6}
$$
See the proof in Appendix A.3. Eq. 6 can be regarded as adding a  $D_{CQL}^{CF}$ -related regularizer to  $J(\pi, \hat{M})$ . Replacing  $D_{CQL}^{CF}$  with  $D_{CQL}$  in the regularizer, we can get a similar form of  $\pi_{MA}^{*}$  for MACQL. We next discuss the performance bound of CFCQL and MACQL on the true MDP  $M$ . If we assume the full coverage of  $\beta^i$  on  $M$ , we have:
Theorem 4.3. Assume  $\forall s, a, i, \beta^i(a|s) \geq \epsilon$ , then with probability  $\geq 1 - \delta$
$$
J \left(\boldsymbol {\pi} _ {M A} ^ {*}, M\right) \geq J \left(\boldsymbol {\pi} ^ {*}, M\right) - \frac {\alpha}{1 - \gamma} \left(\frac {1}{\epsilon^ {n}} - 1\right) - \text {s a m p l i n g e r r o r},
$$
$$
J \left(\boldsymbol {\pi} _ {C F} ^ {*}, M\right) \geq J \left(\boldsymbol {\pi} ^ {*}, M\right) - \frac {\alpha}{1 - \gamma} \left(\frac {1}{\epsilon} - 1\right) - \text {s a m p l i n g e r r o r},
$$
where  $\pi^{*} = \arg \max_{\pi}J(\pi ,M)$ , i.e. the optimal policy, and sampling error is a constant dependent on the MDP itself and  $\mathcal{D}$ .
Theorem 4.3 shows that when sampling error and  $\alpha$  are small enough, the performances gap induced by  $\pi_{CF}^{*}$  can be small, but that induced by  $\pi_{MA}^{*}$  expands when  $n$  increases, making  $\pi_{MA}^{*}$  far away from the optimal. Upon examining the performance gap, we ultimately compare the safe policy improvement guarantees of the two methods on  $M$ :
Theorem 4.4. Assume  $\forall s, a, i, \beta^i(a|s) \geq \epsilon$ . The policy  $\pmb{\pi}_{MA}^{*}(\pmb{a}|s)$  is a  $\zeta^{MA}$ -safe policy improvement over  $\beta$  in the actual MDP  $M$ , i.e.,  $J(\pmb{\pi}_{MA}^{*}, M) \geq J(\beta, M) - \zeta^{MA}$ . And the policy  $\pmb{\pi}_{CF}^{*}(\pmb{a}|s)$  is a  $\zeta^{CF}$ -safe policy improvement over  $\beta$  in  $M$ , i.e.,  $J(\pmb{\pi}_{CF}^{*}, M) \geq J(\beta, M) - \zeta^{CF}$ . When  $n \geq \log_{\frac{1}{\epsilon}}\left(\frac{1}{\epsilon} + \frac{2}{\alpha} \frac{\sqrt{|A|}}{\sqrt{|\mathcal{D}(s)|}} (C_{r,\delta} + \frac{\gamma R_{\max} C_{T,\delta}}{1 - \gamma}) \cdot \left(\frac{1}{\sqrt{\epsilon}} - 1\right)\right)$ ,  $\zeta^{CF} \leq \zeta^{MA}$ .
Detailed formation of  $\zeta^{MA}$  and  $\zeta^{CF}$  is provided in Appendix A.5. Theorem 4.4 shows that with a large enough agent count  $n$ , CFCQL has better safe policy improvement guarantee than MACQL. The validation experiment of this theoretical result is presented in Section 5.1.
4.4 Practical Algorithm
The fixed point of Eq. 4 provides an underestimated  $Q$  function for any policy  $\mu$ . But it is computationally inefficient to solve Eq. 4 every time after one step policy update. Similar to CQL [21], we also choose a  $\mu(a|s)$  that would maximize the current  $\hat{Q}$  with a  $\mu$ -regularizer. If the regularizer aims to minimize the KL divergence between  $\mu^i$  and a uniform distribution,  $\mu^i(a^i|s) \propto \exp\left(\mathbb{E}_{a^{-i} \sim \beta^{-i}} Q(s, a^i, a^{-i})\right)$ , which results in the update rule of Eq. 7 (See Appendix B.1 for detailed derivation):
$$
\min  _ {Q} \alpha \mathbb {E} _ {s \sim \mathcal {D}} \left[ \sum_ {i = 1} ^ {n} \lambda_ {i} \mathbb {E} _ {\boldsymbol {a} ^ {- i} \sim \boldsymbol {\beta} ^ {- i}} [ \log \sum_ {\boldsymbol {a} ^ {i}} \exp (Q (\boldsymbol {s}, \boldsymbol {a})) ] - \mathbb {E} _ {\boldsymbol {a} \sim \boldsymbol {\beta}} [ Q (\boldsymbol {s}, \boldsymbol {a}) ] \right] + \hat {\mathcal {E}} _ {\mathcal {D}} (\boldsymbol {\pi}, Q). \tag {7}
$$
Finally, we need to specify each agent's weight of minimizing the policy Q function  $\lambda_{i}$ . Theoretically, any simplex of  $\lambda$  that satisfies  $\sum_{i=1}^{n} \lambda_{i} = 1$  can be used to induce an underestimated value function linearly increasing as agents number as we expect. Therefore, a simple way is to set  $\lambda_{i} = \frac{1}{n}, \forall i$  where each agent contributes penalty equally. Another way is to prioritize penalizing the agent that exhibits the greatest deviation from the dataset, which is the one-hot style of  $\lambda$ :
$$
\lambda_ {i} (s) = \left\{ \begin{array}{c c} 1. 0, & i = \arg \max  _ {j} \mathbb {E} _ {\pi^ {j}} \frac {\pi^ {j} (s)}{\beta^ {j} (s)} \\ 0. 0, & o t h e r s \end{array} \right. \tag {8}
$$
We assert that each agent's conservative contribution deserves to be considered and differed according to their degree of deviation. As a result, both the uniform and the one-hot treatment present some limitations. Consequently, we employ a moderate softmax variant of Eq. 8:
$$
\forall i, s, \lambda_ {i} (s) = \frac {\exp \left(\tau \mathbb {E} _ {\pi^ {i}} \frac {\pi^ {i} (s)}{\beta^ {i} (s)}\right)}{\sum_ {j = 1} ^ {n} \exp \left(\tau \mathbb {E} _ {\pi^ {j}} \frac {\pi^ {j} (s)}{\beta^ {j} (s)}\right)}, \tag {9}
$$
where  $\tau$  is a predefined temperature coefficient, controlling the influence of  $\mathbb{E}_{\pi^i}\frac{\pi^i}{\beta^i}$  on  $\lambda_i$ . When  $\tau \to 0$ ,  $\lambda_i \to \frac{1}{n}$ , and when  $\tau \to \infty$ , it turns into Eq. 8. To compute Eq. 9, we need an explicit expression of  $\pi^i$  and  $\beta^i$ . In discrete action space,  $\pi^i$  can be estimated by  $\exp \left(\mathbb{E}_{\pmb{a}^{-i} \sim \beta^{-i}} Q(s, a^i, \pmb{a}^{-i})\right)$ , and we use behavior cloning [29] to train a parameterized  $\beta(s)$  from the dataset. In continuous action space,  $\pi^i$  is parameterized by each agent's local policy. For  $\beta^i$ , we use the method of explicit estimation of behavior density in Wu et al. [48], which is modified from a VAE [17] estimator. Details for computing  $\lambda$  are deferred to Appendix B.2.
For policy improvement in continuous action space, we also take derivation of a counterfactual Q function for each agent, rather than updating all agents' policy together like in MADDPG. Specifically, the gradient of each agent  $i$ 's policy  $\pi^i$  is calculated by:
$$
\nabla_ {a ^ {i}} \mathbb {E} _ {s, \boldsymbol {a} ^ {- i} \sim \mathcal {D}, a ^ {i} \sim \pi^ {i} (s)} Q _ {\theta} (s, \boldsymbol {a}) \tag {10}
$$
The reason is that in CFCQL, we only minimize  $Q(s,\pi^i,\beta^{-i})$ , rather than  $Q(s,\pi)$ . Using the untrained  $Q(s,\pi)$  to directly guide PI like MADDPG may result in a bad policy.
We summarize CFCQL in discrete and continuous action space in Algorithm 1 as CFCQL-D and -C, separately.
Algorithm 1 CFCQL-D and CFCQL-C
1: Initialize  $Q_{\theta}$ , target network  $Q_{\hat{\theta}}$ , target update interval  $t_{tar}$ , replay buffer  $\mathcal{D}$ , and optionally  $\pi_{\psi}$  for CFCQL-C
2: for  $t = 1,2,\ldots ,{t}_{\max }$  do
3: Sample  $N$  transitions  $\{s, \pmb{a}, s', r\}$  from  $\mathcal{D}$
4: Compute  $Q_{\theta}(s, a)$  using the structure of QMIX for CFCQL-D or MADDPG for CFCQL-C.
5: Calculate  $\lambda$  according to Eq. 9
6: Update  $Q_{\theta}$  by Eq. 7 with sampled transitions. Using  $\hat{\mathcal{T}}_{\hat{\theta}}$  for CFCQL-D, and  $\hat{T}_{\hat{\theta}}^{\pi_{\psi_t}}$  for CFCQL-C
7: (Only for CFCQL-C) For each agent  $i$ , take one-step policy improvement for  $\pi_{\psi}^{i}$  according to Eq. 10
8: if  $t \mod t_{tar} = 0$  then
9: update target network  $\hat{\theta}\gets \theta$
10: end if
11: end for
Baselines. We compare our method CFCQL with several offline Multi-Agent baselines, where baselines with prefix  $MA$  adopt CTDE paradigm and the others adopt independent learning paradigm: BC: Behavior cloning. TD3-BC[11]: One of the state-of-the-art single agent offline algorithm, simply adding the BC term to TD3 [12]. MACQL: Naive extension of conservative Q-learning, as proposed in Sec.4.2. MAICQ[52]: Multi-agent version of implicit constraint Q-learning by decomposed multi-agent joint-policy under implicit constraint. OMAR[34]: Using zeroth-order optimization for better coordination among agents' policies, based on independent CQL (ICQL). MADTKD[45]: Using decision transformer to represent each agent's policy, trained with knowledge distillation. IQL[18] and AWAC[31]: variants of advantage weighted behaviour cloning, which are SOTA on single agent offline RL. Details for baseline implementations are in Appendix C.3.
Each algorithm is run for five random seeds, and we report the mean performance with standard deviation<sup>2</sup>. Four environments are adopted to evaluate our method, including both discrete action space and continuous action space. We have relocated additional experimental results to Appendix D to conserve space.
Figure 2: (a) The Equal_Line environment where  $n = 3$ . (b) Performance ratio of CFCQL and MACQL to the behaviour policy with a varying number of agents.
5.1 Equal Line
To empirically compare CFCQL and MACQL as agents number  $n$  increases, we design a multi-agent task called Equal_Line, which is a one-dimensional simplified version of Equal_Space introduced in Tseng et al. [45].
Details of the environment are in Appendix C.1. The  $n$  agents need to cooperate to disperse and ensure every agent is equally spaced. The datasets consist of 1000 trajectories sampled by executing a fully-pretrained policy of QMIX[37], i.e. Expert dataset. We plot the performance ratios of CFCQL and MACQL to the behavior policy for different agent number  $n$  in Fig.2(b). It can be observed that the performance of MACQL degrades dramatically as the increase of number of agents while the performance of CFCQL remains basically stable. The results strongly verify the conclusion we proposed in Sec.4.3, that CFCQL has better policy improvement guarantee than MACQL with a large enough number of agents  $n$ .
5.2 Multi-agent Particle Environment
In this section, we test CFCQL on Multi-agent Particle Environemnt with continuous action space. We use the dataset and the adversary agent provided by Pan et al. [34]. The performance of the trained model is measured by the normalized score  $100 \times (S - S_{Random}) / (S - S_{Expert})$  [10].
In Table 1, we only show the comparison of our method and the current state-of-the-art method OMAR and IQL to save space. For complete comparison with more baselines, e.g., TD3+BC and
Table 1: Results on Multi-agent Particle Environment. CN: Cooperative Navigation. PP:Predator-prey. World: World.
Env	Dataset	OMAR	MACQL	IQL	CFCQL
CN	Random	34.4±5.3	45.6±8.7	5.5±1.1	62.2±8.1
Med-Rep	37.9±12.3	25.5±5.9	10.8±4.5	52.2±9.6
Medium	47.9±18.9	14.3±20.2	28.2±3.9	65.0±10.2
Expert	114.9±2.6	12.2±31	103.7±2.5	112±4
PP	Random	11.1±2.8	25.2±11.5	1.3±1.6	78.5±15.6
Med-Rep	47.1±15.3	11.9±9.2	23.2±12	71.1±6
Medium	66.7±23.2	55±43.2	53.6±19.9	68.5±21.8
Expert	116.2±19.8	108.4±21.5	109.3±10.1	118.2±13.1
World	Random	5.9±5.2	11.7±11	2.9±4.0	68±20.8
Med-Rep	42.9±19.5	13.2±16.2	41.5±9.5	73.4±23.2
Medium	74.6±11.5	67.4±48.4	70.5±15.3	93.8±31.8
Expert	110.4±25.7	99.7±31	107.8±17.7	119.7±26.4
2Our code and datasets are available at: https://github.com/thu-rllab/CFCQL
AWAC, please refer to Appendix D.1. It can be seen that on 11 of the 12 datasets, CFCQL shows superior performance than current state-of-the-art. Note that we only report the results of  $\tau = 0$ . In Appendix D.2 of the ablation on  $\tau$ , we show that with carefully fine-tuned  $\tau$ , higher scores of CFCQL can be obtained. We also carry out ablations of  $\alpha$  in Appendix D.3.
5.3 Multi-agent MuJoCo
In this section we investigate the effect of our method on more complex continuous control task. We use the HalfCheetah-v2 setting from the multi-agent MuJoCo environment [35] and the datasets provided by Pan et al. [34]. Table 2 shows that CFCQL exceeds the current state-of-the-art on most datasets.
Except for the counterfactual  $Q$  function, we also analyze whether the counterfactual treatment in CFCQL can be incorporated in other components and help further improvement in Appendix D.4.
Table 2: Results on MaMuJoCo.
Dataset	Random	Med-rep	Medium	Expert
ICQ	7.4±0.0	35.6±2.7	73.6±5.0	110.6±3.3
TD3+BC	7.4±0.0	27.1±5.5	75.5±3.7	114.4±3.8
ICQL	7.4±0.0	41.2±10.1	50.4±10.8	64.2±24.9
OMAR	13.5±7.0	57.7±5.1	80.4±10.2	113.5±4.3
MACQL	5.3±0.5	37.0±7.1	51.5±26.7	50.1±20.1
IQL	7.4±0.0	58.8±6.8	81.3±3.7	115.6±4.2
AWAC	7.3±0.0	30.9±1.6	71.2±4.2	113.3±4.1
CFCQL	39.7±4.0	59.5±8.2	80.5±9.6	118.5±4.9
We find that the counterfactual policy improvement, i.e., the policy improvement by Eq. 10 rather than using MADDPG's PI, is critical for our method.
5.4 StarCraft II
Table 3: Averaged test winning rate of CFCQL and baselines in StarCraft II micromanagement tasks.
Map	Dataset	CFCQL	MACQL	MAICQ	OMAR	MADTKD	BC	IQL	AWAC
2s3z	medium	0.40±0.10	0.17±0.08	0.18±0.02	0.15±0.04	0.18±0.03	0.16±0.07	0.16±0.04	0.19±0.05
medium_replay	0.55±0.07	0.12±0.08	0.41±0.06	0.24±0.09	0.36±0.07	0.33±0.04	0.33±0.06	0.39±0.05
expert	0.99±0.01	0.58±0.34	0.93±0.04	0.95±0.04	0.99±0.02	0.97±0.02	0.98±0.03	0.97±0.03
mixed	0.84±0.09	0.67±0.17	0.85±0.07	0.60±0.04	0.47±0.08	0.44±0.06	0.19±0.04	0.14±0.04
3s_vs_5z	medium	0.28±0.03	0.09±0.06	0.03±0.01	0.00±0.00	0.01±0.01	0.08±0.02	0.20±0.05	0.19±0.03
medium_replay	0.12±0.04	0.01±0.01	0.01±0.02	0.00±0.00	0.01±0.01	0.01±0.01	0.04±0.04	0.08±0.05
expert	0.99±0.01	0.92±0.05	0.91±0.04	0.64±0.08	0.67±0.08	0.98±0.02	0.99±0.01	0.99±0.02
mixed	0.60±0.14	0.17±0.10	0.10±0.04	0.00±0.00	0.14±0.08	0.21±0.04	0.20±0.06	0.18±0.03
5m_vs_6m	medium	0.29±0.05	0.01±0.01	0.26±0.03	0.19±0.06	0.21±0.04	0.28±0.37	0.25±0.02	0.22±0.04
medium_replay	0.22±0.06	0.16±0.08	0.18±0.04	0.03±0.02	0.16±0.04	0.18±0.06	0.18±0.04	0.18±0.04
expert	0.84±0.03	0.01±0.01	0.72±0.05	0.33±0.06	0.58±0.07	0.82±0.04	0.77±0.03	0.75±0.02
mixed	0.76±0.07	0.01±0.01	0.67±0.08	0.10±0.10	0.21±0.05	0.21±0.12	0.76±0.06	0.78±0.02
6h_vs_8z	medium	0.41±0.04	0.01±0.01	0.19±0.04	0.04±0.03	0.22±0.07	0.40±0.03	0.40±0.05	0.43±0.06
medium_replay	0.21±0.05	0.08±0.04	0.07±0.04	0.00±0.00	0.12±0.05	0.11±0.04	0.17±0.03	0.14±0.04
expert	0.7±0.06	0.00±0.00	0.24±0.08	0.01±0.01	0.48±0.08	0.60±0.04	0.67±0.03	0.67±0.03
mixed	0.49±0.08	0.01±0.01	0.05±0.03	0.00±0.00	0.25±0.07	0.27±0.06	0.36±0.05	0.35±0.06
We further validate CFCQL's universality through complex experiments on the StarCraft II Micromanagement Benchmark [40], encompassing four maps with varying agent counts and difficulties: 2s3z, 3s_vs_5z, 5m_vs_6m, and 6h_vs_8z. As no pre-existing datasets exist for these tasks, we generate them ourselves, with dataset creation details provided in the Appendix C.2.
Figure 3: Hyperparameters examination on the temperature coefficient in different types of datasets.
Table 3 presents the average test winning rates of various algorithms on different datasets. MACQL's perfor
mance depends on agent count and environmental difficulty, only succeeding in 2s3z and 3s_vs_5z. MAICQ and OMAR perform well across many datasets but can not tackle all tasks and significantly
struggle in 6h_vs_8z, a highly challenging map. MADTKD, employing supervised learning and knowledge distillation, works well but seldom surpasses BC. IQL and AWAC are competitive baselines but they still fall short compared to CFCQL in most datasets. CFCQL significantly outperforms all baselines on most datasets, achieving state-of-the-art results, with its success attributed to moderate and appropriate conservatism compared to MACQL and other baselines.
Temperature Coefficient. To study the effect of temperature coefficient  $\tau$ , the key hyperparameter in computing the  $\lambda_{i}$ , Fig.3(a) shows the testing winning rate of CFCQL with different  $\tau$ s on each kind of dataset of map 6h_vs_8z. As shown, CFCQL is not sensitive to this hyperparameter and we find that the best value of  $\tau$  is usually greater than 0 while lower than infinity, showing that moderate imbalance can be more effective as expected in previous section.
6.1 Broader Impacts
Our proposed method holds potential for application in real-world multi-agent systems, such as intelligent warehouse management or medical treatment. However, directly implementing the derived policy might entail risks due to the domain gap between the training virtual datasets and real-world scenarios. To mitigate potential hazards, it is crucial for practitioners to operate the policy under human supervision, ensuring that undesirable outcomes are avoided by limiting the available options.
Here we discuss some limitations about CFCQL. In the case of discrete action space, since CFCQL uses QMIX as the backbone, it inherits the Individual-global-max principle [42], which means it cannot solve tasks that are not factorizable. On continuous action space, the counterfactual policy update used in CFCQL allows for updating only one agent's policy for each sample, which may lead to lower convergence speed compared to methods with independent learning.
In this paper, we study the offline MARL problem which is practical and challenging but lack of enough attention. We demonstrate from theories and experiments that naively extending conservative offline RL algorithm to multi-agent setting leads to over-pessimism as the exponentially explosion of action space with the increasing number of agents which hurts the performance. To address this issue, we propose a counterfactual way to make conservative value estimation while maintaining the CTDE paradigm. It is theoretically proved that the proposed CFCQL enjoys performance guarantees independent of number of agents, which can avoid overpessimism caused by the exponentially growth of joint policy space. The results of experiments with discrete or continuous action space show that our method achieve superior performance to the state-of-the-art. Some ablation study is also made to propose further understanding of CFCQL. The idea of counterfactual treatment may also be incorporated with other offline RL and multi-agent RL algorithms, which deserves further investigation both theoretically and empirically.
This work was supported by the National Key R&D Program of China under Grant 2018AAA0102801.
[1] Ackermann, J., Gabler, V., Osa, T., and Sugiyama, M. Reducing overestimation bias in multiagent domains using double centralized critics. arXiv preprint arXiv:1910.01465, 2019.
[2] Agarwal, R., Schuurmans, D., and Norouzi, M. An optimistic perspective on offline reinforcement learning. In International Conference on Machine Learning, pp. 104-114. PMLR, 2020.
[43] Sunehag, P., Lever, G., Gruslys, A., Czarnecki, W. M., Zambaldi, V. F., Jaderberg, M., Lanctot, M., Sonnerat, N., Leibo, J. Z., Tuyls, K., et al. Value-decomposition networks for cooperative multi-agent learning based on team reward. In International Conference on Autonomous Agents and MultiAgent Systems (AAMAS), 2018.
[44] Tan, M. Multi-agent reinforcement learning: Independent vs. cooperative agents. In International Conference on Machine Learning (ICML), 1993.
[45] Tseng, W.-C., Wang, T.-H., Lin, Y.-C., and Isola, P. Offline multi-agent reinforcement learning with knowledge distillation. In Advances in Neural Information Processing Systems, 2022.
[46] Wan, L., Liu, Z., Chen, X., Wang, H., and Lan, X. Greedy-based value representation for optimal coordination in multi-agent reinforcement learning. arXiv preprint arXiv:2112.04454, 2021.
[47] Wang, J., Ren, Z., Liu, T., Yu, Y., and Zhang, C. Qplex: Duplex dueling multi-agent q-learning. arXiv preprint arXiv:2008.01062, 2020.
[48] Wu, J., Wu, H., Qiu, Z., Wang, J., and Long, M. Supported policy optimization for offline reinforcement learning. In Advances in Neural Information Processing Systems, 2022.
[49] Wu, Y., Tucker, G., and Nachum, O. Behavior regularized offline reinforcement learning. arXiv preprint arXiv:1911.11361, 2019.
[50] Wu, Y., Zhai, S., Srivastava, N., Susskind, J., Zhang, J., Salakhutdinov, R., and Goh, H. Uncertainty weighted actor-critic for offline reinforcement learning. arXiv preprint arXiv:2105.08140, 2021.
[51] Xie, T., Cheng, C.-A., Jiang, N., Mineiro, P., and Agarwal, A. Bellman-consistent pessimism for offline reinforcement learning. Advances in neural information processing systems, 34, 2021.
[52] Yang, Y., Ma, X., Li, C., Zheng, Z., Zhang, Q., Huang, G., Yang, J., and Zhao, Q. Believe what you see: Implicit constraint approach for offline multi-agent reinforcement learning. Advances in Neural Information Processing Systems, 34:10299-10312, 2021.
[53] Yu, C., Velu, A., Vinitsky, E., Wang, Y., Bayen, A., and Wu, Y. The surprising effectiveness of mappo in cooperative, multi-agent games. arXiv preprint arXiv:2103.01955, 2021.
[54] Yu, T., Thomas, G., Yu, L., Ermon, S., Zou, J., Levine, S., Finn, C., and Ma, T. Mopo: Model-based offline policy optimization. arXiv preprint arXiv:2005.13239, 2020.
[55] Zanette, A., Wainwright, M. J., and Brunskill, E. Provable benefits of actor-critic methods for offline reinforcement learning. Advances in neural information processing systems, 34, 2021.
A Detailed Proof
Proof. Similar to the proof of Theorem 3.2 in Kumar et al. [21], we first prove this theorem in the absence of sampling error, and then incorporate sampling error at the end. By set the derivation of the objective in Eq. 4 to zero, we can compute the Q-function update induced in the exact, tabular setting  $(\mathcal{T}^{\pi} = \hat{\mathcal{T}}^{\pi}$  and  $\pi_{\beta}(\mathbf{a}|s) = \hat{\pi}_{\beta}(\mathbf{a}|s))$
$$
\forall s, \boldsymbol {a}, k, \hat {Q} ^ {k + 1} (s, \boldsymbol {a}) = \mathcal {T} ^ {\pi} \hat {Q} ^ {k} (s, \boldsymbol {a}) - \alpha \left[ \Sigma_ {i = 1} ^ {n} \lambda_ {i} \frac {\mu^ {i}}{\pi_ {\beta} ^ {i}} - 1 \right] \tag {A.1}
$$
Then, the value of the policy,  $\hat{V}^{k + 1}$  can be proved to be underestimated, since:
$$
\hat {V} ^ {k + 1} (s) = \mathbb {E} _ {\boldsymbol {a} \sim \boldsymbol {\pi} (\boldsymbol {a} | s)} \left[ \hat {Q} ^ {\boldsymbol {\pi}} (s, \boldsymbol {a}) \right] = \mathcal {T} ^ {\boldsymbol {\pi}} \hat {V} ^ {k} (s) - \alpha \mathbb {E} _ {\boldsymbol {a} \sim \boldsymbol {\pi} (\boldsymbol {a} | s)} \left[ \Sigma_ {i = 1} ^ {n} \lambda_ {i} \frac {\mu^ {i}}{\pi_ {\beta} ^ {i}} - 1 \right] \tag {A.2}
$$
Next, we will show that  $D_{CQL}^{CF}(s) = \Sigma_a\pi (\pmb {a}|s)\left[\Sigma_{i = 1}^n\lambda_i\frac{\mu^i(a^i|s)}{\hat{\pi}_\beta^i(a^i|s)} -1\right]$  is always positive, when  $\mu^i (a^i |s) = \pi^i (a^i |s)$ :