SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\pi_ {t o t} (\boldsymbol {a} | \boldsymbol {o}) = \mu_ {t o t} (\boldsymbol {a} | \boldsymbol {o}) \cdot \exp \left(\frac {Q _ {t o t} (\boldsymbol {o} , \boldsymbol {a}) - u (\boldsymbol {o})}{\alpha} - 1\right) \tag {16}
$$
Bringing Eq. (16) into  $\sum_{\pmb{a}}\pi_{tot}(\pmb {a}|\pmb {o}) = 1$  , we have:
$$
\mathbb {E} _ {\boldsymbol {a} \sim \mu_ {t o t}} \left[ \exp \left(\frac {Q _ {t o t} (\boldsymbol {o} , \boldsymbol {a}) - u (\boldsymbol {o})}{\alpha} - 1\right) \right] = 1 \tag {17}
$$
The left side of Eq. (17) can be seen as a continuous and monotonic function of  $u$ , so it has only one solution denoted as  $u^*$ , and we denote the corresponding policy  $\pi_{tot}$  as  $\pi_{tot}^*$ .
Integrating Eq. (16) into the expression of optimal global state value, we can get:
$$
\begin{array}{l} V _ {t o t} ^ {*} (\boldsymbol {o}) = \mathcal {T} _ {f} ^ {*} V _ {t o t} ^ {*} (\boldsymbol {o}) \\ = \sum_ {\boldsymbol {a}} \pi_ {t o t} ^ {*} (\boldsymbol {a} | \boldsymbol {o}) \left(Q _ {t o t} ^ {*} (\boldsymbol {o}, \boldsymbol {a}) - \alpha \log \left(\frac {\pi_ {t o t} ^ {*} (\boldsymbol {a} | \boldsymbol {o})}{\mu_ {t o t} (\boldsymbol {a} | \boldsymbol {o})}\right)\right) \\ = \sum_ {\boldsymbol {a}} \pi_ {t o t} ^ {*} (\boldsymbol {a} | \boldsymbol {o}) (u ^ {*} (\boldsymbol {o}) + \alpha) \\ = u ^ {*} (\boldsymbol {o}) + \alpha \\ \end{array}
$$
To summarize, we obtain the optimality condition of the behavior regularized MDP with Reverse KL divergence as follows:
$$
V _ {t o t} ^ {*} (\boldsymbol {o}) = u ^ {*} (\boldsymbol {o}) + \alpha
$$
$$
\pi_ {t o t} ^ {*} (\boldsymbol {a} | \boldsymbol {o}) = \mu_ {t o t} (\boldsymbol {a} | \boldsymbol {o}) \cdot \exp \left(\frac {Q _ {t o t} ^ {*} (\boldsymbol {o} , \boldsymbol {a}) - u ^ {*} (\boldsymbol {o})}{\alpha} - 1\right)
$$
where  $u(\pmb{o})$  is a normalization term and has a optimal value  $u^{*}$  that makes the corresponding optimal policy  $\pi_{tot}^{*}$  satisfy  $\sum_{\pmb{a}\in \mathcal{A}^n}\pi_{tot}^{*}(\pmb {a}|\pmb {o}) = 1$ .
B.1 Multi-Agent MuJoCo
Multi-agent Mujoco [15] is a benchmark framework developed for assessing and comparing the effectiveness of algorithms in continuous multi-agent robotic control. Within this framework, a robotic system is partitioned into independent agents, each tasked with controlling a specific set of joints. The agents collaborate harmoniously to accomplish shared objectives, such as acquiring the ability to walk through an environment, with the ultimate goal of maximizing the cumulative reward. Multi-agent MuJoCo environment consists of multiple different robot configurations, and it is often used for the study of novel MARL algorithms for decentralized coordination in isolation.
To generate the dataset transitions, we captured the interactions between the environment and trained online MARL algorithms. Specifically, we use HAPPO [46] algorithm to collect data. The expert dataset is generated by employing the converged HAPPO algorithm. This involves training the algorithm until it reaches a state of convergence, where the agents have learned optimal policies. The medium dataset is generated by first training a policy online using HAPPO, early-stopping the training, and collecting samples from this partially-trained policy. The medium-replay dataset consists of recording all samples in the replay buffer observed during training until the policy reaches the medium level of performance. The medium-expert dataset is constructed by mixing equal amounts of expert demonstrations and suboptimal data. For all datasets, the hyperparameter env_args(agent_obsk determines up to which connection distance agents will be able to form observations) is set to 1. The average returns of our datasets are listed in Table 2.
B.2 The StarCraft Multi-Agent Challenge
The StarCraft Multi-Agent Challenge (SMAC) benchmark is chosen as our testing environment. Due to its high control complexity, SMAC is a popular multi-agent cooperative control environment for evaluating advanced MARL methods. It consists of a collection of StarCraft II micro-management
Table 2: The multi-agent MuJoCo datasets.
Scenario	Quality	Average Return
2-Agent Ant	expert	2055.07
medium	1418.70
medium-expert	1736.88
medium-replay	1029.51
3-Agent Hopper	expert	2452.02
medium	723.57
medium-expert	1190.61
medium-replay	746.42
6-Agent HalfCheetah	expert	2785.10
medium	1425.66
medium-expert	2105.38
medium-replay	655.76
tasks in which two groups of units engage in combat. Agents based on the MARL algorithm control the first group's units, while a built-in heuristic game AI bot with different difficulties controls the second group's units. Scenarios vary in terms of the initial location, number and type of units, and elevated or impassable terrain. The available actions for each agent include no operation, move[direction], attack [enemy id], and stop. The reward that each agent receives is the same. The hit-point damage dealt and received determines the agents' share of the reward. SMAC consists of several StarCraft II multi-agent micromanagement maps. We consider 4 representative battle maps, including 2 hard maps (5m_vs_6m, 2c_vs_64zg), and 2 super hard maps (6h_vs_8z, corridor), as our experiment tasks. The task type and other details of the maps are listed in Table 3.
Table 3: SMAC maps for experiments.
Map Name	Ally Units	Enemy Units	Type
5m_vs_6m	5 Marines	6 Marines	homogeneous & asymmetric
2c_vs_64zg	2 Colossi	64 Zerglings	micro-trick: positioning
6h_vs_8z	6 Hydralisks	8 Zealots	micro-trick: focus fire
corridor	6 Zealots	24 Zerglings	micro-trick: wall off
The offline SMAC dataset used in this study is provided by [38], which is the largest open offline dataset on SMAC. Different from single-agent offline datasets, it considers the property of multi-agent POMDP, which owns local observations and available actions for each agent. The dataset is collected from the trained MAPPO agent and includes three quality levels: good, medium, and poor. For each original large dataset, we randomly sample 1000 episodes as our dataset. The average returns of SMAC datasets are listed in Table 4.
Table 4: The SMAC datasets.
Map Name	Quality	Average Return
5m_vs_6m	good	20.00
medium	11.03
poor	8.50
2c_vs_64zg	good	19.94
medium	13.00
poor	8.89
6h_vs_8z	good	17.84
medium	11.96
poor	9.12
corridor	good	19.88
medium	13.07
poor	4.93
C.1 Details of OMIGA
The local Q-value, state-value networks and policy networks of OMIGA are represented by 3-layer ReLU activated MLPs with 256 units for each hidden layer. For the weight network, we use 2-layer ReLU-activated MLPs with 64 units for each hidden layer. All the networks are optimized by Adam optimizer.
C.2 Details of baselines
We compare OMIGA against four recent offline MARL algorithms: ICQ [11], OMAR [12], BCQ-MA, and CQL-MA. For ICQ and OMAR, we implement them based on the algorithm described in their papers. BCQ-MA is the multi-agent version of BCQ [4], and CQL-MA is the multi-agent version of CQL [20]. BCQ-MA and CQL-MA use the linear weighted value decomposition structure as  $Q_{tot} = \sum_{i=1}^{n} w_i(o) Q_i(o_i, a_i) + b(o), w^i \geq 0$  for the multi-agent setting. The policy constraint of BCQ-MA and the value regularization of CQL-MA are both imposed on the local Q-value.
In this paper, all experiments are implemented with Pytorch and executed on NVIDIA V100 GPUs.
For multi-agent MuJoCo, the hyperparameters of OMIGA and baselines are listed in Table 5. An important hyperparameter of OMIGA is the regularization hyperparameter  $\alpha$ . The higher  $\alpha$  encourages OMIGA to stay near the behavioral distribution, and lower  $\alpha$  makes OMIGA more optimistic. On most tasks, we use  $\alpha = 10$  to ensure a good regularization effect. On the medium-quality dataset of the HalfCheetah task, we choose  $\alpha = 1$ .
Table 5: Hyperparameters of OMIGA and baselines for multi-agent MuJoCo.
Hyperparameter	Value
Shared parameters
Q-value network learning rate	5e-4
Policy network learning rate	5e-4
Optimizer	Adam
Target update rate	0.005
Batch size	128
Discount factor	0.99
Hidden dimension	256
Weight network hidden dimension	64
OMIGA
State-value network learning rate	5e-4
Regularization parameter α	1 or 10
Others
Lagrangian coefficient (ICQ)	10
Tradeoff factor α (OMAR, CQL-MA)	1
For SMAC, the hyperparameters of OMIGA and baselines are listed in Table 6. On most tasks, we use  $\alpha = 10$ . On the poor dataset of the  $6\mathrm{h\_vs\_8z}$  map, the quality of the dataset is relatively poor. It does not make much sense to make the policy close to the behavioral policy, so we choose  $\alpha = 2$  to make the algorithm more radical.
On most SMAC maps, the learning rate of all networks is set to 5e-4. The exception is the map  $2\mathrm{c\_vs\_64}\mathrm{zg}$ , on this map, the learning rate of all networks is set to 1e-4.
Table 6: Hyperparameters of OMIGA and baselines for SMAC.
Hyperparameter	Value
Shared parameters
Q-value network learning rate	5e-4 or 1e-4
Policy network learning rate	5e-4 or 1e-4
Optimizer	Adam
Target update rate	0.005
Batch size	128
Discount factor	0.99
Hidden dimension	256
Weight network hidden dimension	64
OMIGA
State-value network learning rate	5e-4 or 1e-4
Regularization parameter α	2 or 10
Others
Lagrangian coefficient (ICQ)	10
Threshold (BCQ-MA)	0.3
Tradeoff factor α (OMAR, CQL-MA)	1
Table 7: Average scores and standard deviations over 5 random seeds on the mixed offline SMAC datasets.
Map	Dataset	BCQ-MA	CQL-MA	ICQ	OMAR	OMIGA(ours)
6h_vs_8z	good-poor	11.41±0.44	9.56±0.25	11.00±0.36	9.17±0.19	11.88±0.27
6h_vs_8z	good-medium	11.79±0.29	10.08±0.26	11.18±0.25	10.02±0.16	12.05±0.47
6h_vs_8z	medium-poor	11.18±0.41	10.73±0.38	11.25±0.35	10.42±0.19	11.85±0.35
corridor	good-poor	12.37±1.36	4.88±0.35	11.78±1.53	5.54±0.75	13.01±0.89
corridor	good-medium	13.32±0.71	5.77±1.30	12.98±0.62	6.63±0.74	14.02±1.04
corridor	medium-poor	8.11±0.35	6.18±0.59	8.27±0.48	6.25 ±0.48	9.70±1.40
D.1 Results on mixed datasets
We also evaluated the performance of OMIGA on the multi-modal mixed datasets. Unlike BCQ-MA and OMR, OMIGA doesn't need to learn a behavior policy. We choose two original datasets on the SMAC super hard maps 6h_vs_8z and corridor and construct mixed datasets by combining these SMAC datasets of different quality, including good-poor, good-medium, and medium-poor datasets. Each mixed dataset is blended by  $50\%$  of each of the two original datasets. On these mixed suboptimal datasets, the behavior policy is heterogeneous. Therefore, it is more difficult for algorithms such as BCQ-MA and OMR to learn an accurate behavior policy, making implicit value learning with the regularization framework of OMIGA more appealing.
Table 7 shows the results that OMIGA consistently outperforms other offline MARL baselines under all different mixed dataset experiments. Compared with the results on the original datasets, the performance of OMIGA has become more leading, indicating the benefits of implicit value regularization of OMIGA.