SlowGuess/NeurIPS_2023_OCR huggingface dataset:
BC: behavior cloning. In discrete action space, we train a three-level MLP network with MLE loss. In continuous action space, we use the method of explicit estimation of behavior density in Wu et al. [48], which is modified from a VAE [17] estimator. TD3-BC[11]: One of the SOTA single agent offline algorithm, simply adding the BC term to TD3 [12]. We use the open-source implementation and modify it to a CTDE version with centralised critic. IQL[18] and AWAC[31]: variants of advantage weighted behaviour cloning. We refer to the open-source implementation and implement a CTDE version similar to TD3-BC. MACQL:naive extension of conservative Q-learning, as proposed in Sec. 3.3. We implement it based on the open-source implementation. As the joint action space is enormous, we sample  $N$  actions for the logsumexp operation. MAICQ[52]:multi-agent version of implicit constraint Q-learning by propose the decomposed multi-agent joint-policy under implicit constraint. We use the open-source implementation in discrete action space and cite the experimental results in continuous action space from Pan et al. [34]. OMAR[34]:uses zeroth-order optimization for better coordination among agents' policies, based on independent CQL (ICQL). We cite the experimental results in continuous action space from Pan et al. [34] and implement a version in discrete action space based on the open-source implementation. MADTKD[45]:uses decision transformer to represent each agent's policy and trains with knowledge distillation. As lack of open-source implementation, We implement it based on the open-source implementation of another Decision Transformer based method MADT[28].
C.4 Resources
We use 2 servers to run all the experiments. Each one has  $8^{*}$  NVIDIA RTX 3090 GPUs, and  $2^{*}$  AMD 7H12 CPUs. Each setting is repeated for 5 seeds. For one seed in SC2, it takes about 1.5 hours. For MPE, 10 minutes is enough. The experiments on MaMuJoCo cost the most, about 5 hours for each seed.
C.5 Code, Hyper-parameters and Reproducibility
Please refer to this repository<sup>12</sup> for the code, datasets and the hyper-parameters of our method. For each dataset number 0, 1, 2, 3, 4, we use the seed 0, 1, 2, 3, 4, respectively.
D More results
D.1 Complete Results on MPE
Table 5 shows the complete results of our methods and more baselines on Multi-agent Particle Environment. Some results are cited from Pan et al. [34].
$^{6}$ https://github.com/sfujim/TD3_BC
<sup>7</sup>https://github.com/tinkoff-ai/CORL
<sup>8</sup>https://github.com/aviralkumar2907/CQL
$^{9}$ https://github.com/YiqinYang/ICQ
10https://github.com/ling-pan/OMAR
<sup>11</sup>https://github.com/ReinholdM/Offline-Pre-trained-Multi-Agent-Decision-Transformer
$^{12}$ https://github.com/thu-rllab/CFCQL
Table 5: Complete results on Multi-agent Particle Environment.
Env	Dataset	MAICQ	MATD3-BC	ICQL	OMAR	MACQL	IQL	AWAC	CFCQL
CN	Random	6.3±3.5	9.8±4.9	24.0±9.8	34.4±5.3	45.6±8.7	5.5±1.1	0.5±3.7	62.2±8.1
Medium-replay	13.6±5.7	15.4±5.6	20.0±8.4	37.9±12.3	25.5±5.9	10.8±4.5	2.7±3	52.2±9.6
Medium	29.3±5.5	29.3±4.8	34.1±7.2	47.9±18.9	14.3±20.2	28.2±3.9	25.7±4.1	65.0±10.2
Expert	104.0±3.4	108.3±3.3	98.2±5.2	114.9±2.6	12.2±31	103.7±2.5	103.3±3.5	112±4
PP	Random	2.2±2.6	5.7±3.5	5.0±8.2	11.1±2.8	25.2±11.5	1.3±1.6	0.2±1.0	78.5±15.6
Medium-replay	34.5±27.8	28.7±20.9	24.8±17.3	47.1±15.3	11.9±9.2	23.2±12	8.3±5.3	71.1±6
Medium	63.3±20.0	65.1±29.5	61.7±23.1	66.7±23.2	55±43.2	53.6±19.9	50.9±19.0	68.5±21.8
Expert	113.0±14.4	115.2±12.5	93.9±14.0	116.2±19.8	108.4±21.5	109.3±10.1	106.5±10.1	118.2±13.1
World	Random	1.0±3.2	2.8±5.5	0.6±2.0	5.9±5.2	11.7±11	2.9±4.0	-2.4±2.0	68±20.8
Medium-replay	12.0±9.1	17.4±8.1	29.6±13.8	42.9±19.5	13.2±16.2	41.5±9.5	8.9±5.1	73.4±23.2
Medium	71.9±20.0	73.4±9.3	58.6±11.2	74.6±11.5	67.4±48.4	70.5±15.3	63.9±14.2	93.8±31.8
Expert	109.5±22.8	110.3±21.3	71.9±28.1	110.4±25.7	99.7±31	107.8±17.7	107.6±15.6	119.7±26.4
Figure 4: Ablations of  $\tau$  on World.
D.2 Temperature Coefficient in Continuous Action Space
We carry out ablations of  $\tau$  on MPE's map World in Fig. 4. We find that although the best  $\tau$  differs in different datasets, the overall performance is not sensitive to  $\tau$ , which verifies the theoretical analysis that any simplex of  $\lambda$  that  $\sum_{i=1}^{n} \lambda_i = 1$  can induce an underestimated value function.
D.3 Ablation on CQL  $\alpha$
We carry out ablations of  $\alpha$  on MPE's map World in Fig. 5. We find that  $\alpha$  plays a more important role for team performance on narrow distributions (e.g., Expert and Medium) than that on wide distributions (e.g., Random and Medium - Replay).
D.4 Component Analysis on Counterfactual style
In the environment MaMuJo, except for the counterfactual Q function, we also analyze whether the conuterfactual treatment in CFCQL can be incorporated in other components and help further improvement in Table 6. We find that the counterfactual policy improvement is critical for this
Figure 5: Ablations of  $\alpha$  on World.
Table 6: Component Analysis on MaMuJoCo. CF_T: computing target Q by  $\mathbb{E}_{i\sim \mathrm{Unif}(1,n)}\mathbb{E}_{s',a^{-i}\sim \mathcal{D},a^i\sim \pi^i}Q_{\hat{\theta}}(s,a)$ . CF_P: the policy improvement (PI) by Eq. 10, otherwise using MADDPG's PI.
Dataset	Default	+CF_T	-CF_P	MACQL
Random	39.7±4.0	48.7±1.8	23.9±9.2	5.3±0.5
Med-Rep	59.5±8.2	58.9±9.6	43.5±5.6	36.7±7.1
Medium	80.5±9.6	76.2±12.1	43.8±7.8	51.5±26.7
Expert	118.5±4.9	118.1±6.9	3.7±3.1	50.1±20.1
environment. With CF_P, the method shows great performance gain on narrow data distribution, e.g., the Expert dataset.
Figure 6: Hyperparameters examination on the size of datasets.
D.5 Analysis on Size of Dataset
An additional study examines dataset size effects. We generate a full dataset with 50,000 trajectories for each type in map  $6h\_ vs\_ 8z$ , creating smaller datasets by sampling 5, 50, 500, 5000 trajectories. Fig. 6 displays CFCQL's testing winning rate for varying dataset sizes. It's notable that to ensure fairness, the maximum number of training steps for all datasets and algorithms on the SMAC environment is fixed at 1e7. In this additional study, however, we trained the CFCQL algorithm until convergence to eliminate the impact of underfitting. The results demonstrate that larger datasets contribute to improved convergence performances, thus confirming the scalability of CFCQL for larger data samples.
E.1 Overestimation in Offline RL
We offer an intuitive explanation for the phenomenon of overestimation caused by distribution shift in this section. For a rigorous proof, we refer readers to the related works.
In offline RL, a key challenge arises due to the distribution mismatch between the behavior policy—the policy responsible for generating the data—and the target policy, which is the policy one aims to improve. This mismatch can result in extrapolation errors during value function estimation, often leading to overestimation. Specifically, during the policy evaluation stage, the dataset may not encompass all possible state-action pairs in the Markov Decision Process (MDP), leading to inaccurate  $Q$  function estimates for unseen state-action pairs. These estimates may be either too high or too low compared to the actual  $Q$  values. Subsequently, in the policy improvement stage, the algorithm tends to shift towards actions that appear to offer higher rewards based on these overestimated values. Unlike online RL, which can implement the current training policy in the environment to obtain real feedback and thereby correct the policy's direction, offline RL lacks this corrective mechanism unless careful loss design is employed. When using common bootstrapping methods like temporal difference learning for training the  $Q$  function, these overestimation errors can propagate, affecting estimates
for other state-action pairs. This can result in a chain reaction of overestimations, potentially causing an exponential explosion of the  $Q$  function.