Offline Multi-Agent Reinforcement Learning with Implicit Global-to-Local Value Regularization
Authors: Xiangsen Wang
Haoran Xu
Yinan Zheng
Xianyuan Zhan
Venue: NeurIPS 2023 poster (2023)
Keywords: Offline reinforcement learning; multi-agent reinforcement learning; multi-agent cooperation
TL;DR: A principled offline MARL approach that converts global-level regularization into equivalent implicit local-level regularizations with SOTA empirical results.

Abstract: Offline reinforcement learning (RL) has received considerable attention in recent years due to its attractive capability of learning policies from offline datasets without environmental interactions. Despite some success in the single-agent setting, offline multi-agent RL (MARL) remains to be a challenge. The large joint state-action space and the coupled multi-agent behaviors pose extra complexities for offline policy optimization. Most existing offline MARL studies simply apply offline data-related regularizations on individual agents, without fully considering the multi-agent system at the global level. In this work, we present OMIGA, a new offline multi-agent RL algorithm with implicit global-to-local value regularization. OMIGA provides a principled framework to convert global-level value regularization into equivalent implicit local value regularizations and simultaneously enables in-sample learning, thus elegantly bridging multi-agent value decomposition and policy learning with offline regularizations. Based on comprehensive experiments on the offline multi-agent MuJoCo and StarCraft II micro-management tasks, we show that OMIGA achieves superior performance over the state-of-the-art offline MARL methods in almost all tasks.

Offline Multi-Agent Reinforcement Learning with Implicit Global-to-Local Value Regularization
Xiangsen Wang!* Haoran Xu²* Yinan Zheng² Xianyuan Zhan 3:41
Beijing Jiaotong University UT Austin Tsinghua University
4 Shanghai Artificial Intelligence Laboratory
vangxiangsen@bjtu. edu. en, haoran xu@utexas........ haoram.xuuutexas. edu, zhengyn23@maile wet.imiissssssststeeeeeeee edu. cn, zhanxianyuan tsinghua, edu. cn
Abstract
Offline reinforcement learning (RL) has received considerable attention in recent years due to its attractive capability of learning policies from offline datasets without environmental interactions. Despite some success in the single-agent setting, offline multi-agent RI. (MARL.) remains joint state-action space and the coupled multi-agent behaviors pose extra com- plexities for offline policy optimization. Most existing offline MARL studies simply apply offline data-related regularizations on individual agents, without fully considering the multi-agent system at the global level. In this work, we present OMIGA,
a challenge. The large
offline multi-agent RL algorithm with implicit global-to-loce value regularization. OMIGA provides a principled framework to convert global- level value regularization into equivalent implicit local value regularizations and simultaneously enables in-sample learning, thus elegantly bridging multi-agent value decomposition and policy learning with offline regularizations. Based on comprehensive experiments on the offline multi-agent MuJoCo and StarCraft II micro-management tasks, we show that OMIGA achieves superior performance over the state-of-the-art offline MARL methods in almost all tasks. Our code is com Ttt///ZZengynnnnn AIR/OMMGA
available https: wtt.gtttbb..m//e////iaaaaaaeeeee
Introduction
Multi-agent reinforcement learning (MARL) an active research area tackle many real-world problems that involve multi-agent systems, such as autonomous vehicle coordination[]], network traffic routing [2], and multi-player strategy games 3 Although MARI. has made some impressive progress in recent years, most of its successes are restricted to simulation environments. In most real-world applications, building high-fidelity simulators can rather costly or even infeasible, and online interaction with the real system during policy learning is also expensive or risky. Inspired by the recently emerged offline RL methods 44 5
9 10,, equipping MARL with offline learning capability has become attractive direction to tackle real-world multi-agent tasks.
Offline RL focuses on learning optimal policies with pre-coll collected offline datasets with no further environmental interaction. Under the offline setting, evaluating value function on out-of-din distribbution (OOD) samples can cause extrapolation error accumulation on Q-values, leading erroneously over- estimated values and misguiding policy learning [5. This issuc, which is also called distributional shift (5), is one of the core challenges offline RI. Current offline RL methods tackle this issue primarily by incorporating various forms of data-related regularizations to restrict policy learning
Work done during the internships with Institute for Al Industry Research (AIR), Tsinghua University. Corresponding Author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

from deviating too much from the behavioral data. However, extending the same offline regularization technique to the aulti-agent setting poses some unique challenges. The joint state-action space grows exponentially with the number of agents, making the global-level regularization hard to compute and may also result in very sparse constraints under the huge joint state-action space, especially when the size and coverage of the offline dataset are limited.
To tackle the challenges, some recent works [1 12
13 attempt to design offline MARL algo- rithms heuristically by enforcing local-level regularizations. By using the Centralized Training with Decentralized Execution (CTDE) framework [4] to decompose the global value function into a combination of local value functions, these works turn the original offline multi- aagent RL problem into the offline single-agent RL problem and apply policy or value regularizations at the local level. While straightforward and easy to implement, simply enforcing local-level regularization cannot guarantee the induced regularization the global-level still remains valid, which can potentially lead to wver-conservative policy learning. Existing approaches offer no guarantee whether the optimized local policies are jointly optimal under a given value decomposition scheme. Furthermore, because the offline regularization of most existing methods is completely imposed from the local-level without considering the global information, these methods fail to capture coordinated agent behaviors and credit assignment in the multi-agent system.
In this paper, we aim to give a principled approach to the offline MARL problem. We start by imposing global-level regularizations to form a behaaiior-reullarized MDP. We then find that using some particular regularization choice in this MDP will have closed-form solutions, which naturally turn the global-level regularizations into local-level ones, under some mild assumptions. Finally, we show that the local-level regularization can be derived in an implicit way by using only samples from the offline datasets, without knowing the behavior policy or approximating the regularization term. We dub this new algorithm as offline multi-agent RL algorithm with implicit global-to-local value regularization (OMIGA). In OMIGA, the local-level value regularization is rigörously derived from the global regularization under value decomposition, thus also capturing global information and the impact of multi-agent credit assignment. Under this design, we can also guarantee that the learned local policies are jointly optimal at the global level. Also, OMIGA enables complete sampee learning without querying OOD action samples, which enjoys better stability during policy learning.
We evaluate our method using various types of offline datasets on both multi-agent MuloCo [15] and StarCraft Multi-Agent Challenge (SMAC) tasks [16). Under all settings, OMIGA achieves better performance and enjoys faster convergence compared with other strong baselines.
2 Related Work
Offline reinforcement learning. Due to the absence of online data collection, offline policy learning suffers from severe distributional shift and exploitation eπror accumulation issues when evaluating the value function on OOD actions [4. As a result, existing offline RL methods adopt various approaches to learn policies pessimistically and regularize policy learning close to data distribution. 18 19] add explicit or implicit policy constraints to regularize the policy from deviating too much from behavioral data. Value regularization methods 20 21 22
Policy constraint methods 4 S
17
23 learn conservatively estimated value functions on OOD data. Uncertainty-basee methods add
uncertainty penalization terms on value functions 24 25 or rewards 26
27
28 to enable pessimistic
provide new direction to avoid the distributional shift by learning value functions and policies completely within data. These methods eliminate the involvement of policy-generate OOD samples during policy evaluation, thus enjoying superior learning stability. Our method embeds multi-agent value decomposition within the in-sample learning paradigm, which fully exploits the multi-agent problem structure for offline policy optimization.
policy learning. Recently, in-sample learning methods 7 g
10
29
Multi-agent reinforcement learning. The complexity of multi-agent RL problems typically arises from the huge joint action space (30). In recent years, the CTDE framework [31] 32] has been proposed to decouple agents' learning and execution phases to handle the exploding action space issue. Under the CTDE framework, agents are trained in centralized manner with global information and make decisions with learned local policies during execution. Some representative works are the value decomposition methods
[ 34 35 36), which decompose the global Q-value function into a set of local valuee functions for scalable multi-agent policy optimization.
2

---

There have been some recent attempts to design MARL algorithms for the offline setting, which
can be broadly categorized as RL-based [ 13 37 and goal-conditioned supervised learning
Existing RI.-based methods are typically built upon the CTDR framework and perform offline policy regularization at the local level. For example, ICQ [1] uses importance sampling to implicitly constrain local policy learning on OOD samples. OMAR [2] adopts CQL (20) to learn local Q-value functions and adds zeroth-order optimization to avoid bad local optima. In these methods, the multi-agent modeling and offline RL ingredients are fragmented. which cannot guarantee that the local-level regularizations still remain valid at the global level, and also ignores the cooperative behavior among agents. Moreover, there is no consideration of the value decomposition' 's influence on the optimal local policies learned with offline regularizations. On the other hand, GCSL-based methods 38 39 leverage the sequential data modeling capability of transformer architecture to solve offline MARI. tasks. However, recent studies also show that GCSL-based methods can be over-conservative and cannot guarantee optimality under stochastic
(GCSL) based methods 38 39
environments 40
In this work, our proposed OMIGA is an RL-based method, which uses an implicit global-to-local value regularization scheme to tackle the aforementioned limitations, and offers an elegant solution to marry multi-agent modeling and offline learning.
Preliminaries
3.1 Notations
We focus the fully cooperative multi-agent tasks, which can be modeled as multi-agent Partially
Observable Markov Decision Process (POMDP) [42), defined by a tuple = (8, AA P,r, Z, O, n, 7) E S denotes the true state of the environment. A is the action set for each of the agents, At each
step, an agent E {1, 2, ....] chooses an action a, E A, forming a joint action a = (a1. 02 .....) E AT P (s'|s, |s. a): S x A" S + [0, 1] is the transition dynamics to the next state s' 7 [0. 1) is discount factor. In the partial observable environment, each agent receives an observation O2 E O at each step based All agents share the same global reward function r(o, a) O x A - R. In cooperative MARI,, all agents aim learn a of policies Toot = { Wn that jointly maximize the expected discounted returns EaETTor Deo D 7'r(Ot, a=)]. Under the offline setting, a pre-collected dataset D is obtained by sampling with the behavior policy Moot = {MI,... , f and the policy learning is conducted entirely with the data samples in D without any environment interactions.
observation function Z(s, i) S N + O, and we denote
(0), 02. ...),...
3.2 CTDE Framework and Value Decomposition
In MARL, the joint action space increases exponentially with the increase in the number of agents. Therefore, it is difficult to directly query an optimal joint action from the global Q-value function Qtor (o, a). and the global Q-value function could be negatively affected by the suboptimality of individual agents. To address these problems, the Centralized Training with Decentralized Execution (CTDE) framework [14 31 32] is proposed. In the training phase, agents can access the full environment information and share each other's experiences. In the execution phase, each agent chooses actions only according to its individual observation o;- Through the CTDE framework, optimization the individual level results in the optimization of the joint action space, which avoids the aforementioned problems. Value decomposition methods [33 34
35 36) are popular solutions under the CTDE framework to achieve the decomposition of the joint action space. Value decomposition is also used in recent offline MARL methods [2],2) which decomposes the global Q-value function (o, a) into a linear combination of local Q-value functions Q+(o,, a;):
Q0, = Σ (i(o)) a:) b()),
0. -i= ,n
(I)
where (o) and b(o) capture the weights and offset local Q-value functions.
3

---

3.3 Behavior-Regularizee MDP in Offline RL
To avoid the distributional shift in offline RL, existing approaches typically incorporate various data-related regularizations on rewards, value functions, or the policy optimization objective. In our work, we are specifically interested in the treatment that adds behavior regularizer the rewards r(s, a) to penalize OOD samples [1Q]. This framework leads to special behavior-regularized MDP, which optimizes a policy with the following objective:
max E x
Ex (r (st, of (ar (arlse) (adls)))
where a is scale parameter, and (() is the function that captures the divergence between и and J,, which is similar to the entropy regularization in SAC 43 In the above objective, we regularize the deviation between the learned policy and behavior policy 1e, so to avoid the distributional shift issue. This behavior-regularrzze MDP corresponds to the following modified policy evaluation operator Ty:
(Tf) Q(s a) r(s, a) 1 IV (s')
(77) V(8) = Enns (r(8,a) + + B. [V (s'Il,
where
v())= = Bawx (Q(s.a)= - af (- (arls) / (aels:))]
In the next section, we will derive our proposed method OMIGA based on the regulior- framework, and demonstrate the benefits and desired properties of this framework for the multi- aulti-agent setting.
4 Method
In this section, we formally present our implicit global-to-local value regularization approach OMIGA for offline MARL and explain how it can be integrated into effective offline learning- We begin with the multi-agent POMDP with a reverse KI. global value regularization added the rewards. We make equivalent reformulation that naturally converts the global-level
then show how regularization into local-level value regularizations. Lastly, we derive practical algorithm that implicitly enables local-level value regularizations via in-sample learning and provide a thorough discussion of it.
4.1 Multi-Agent POMDP with Global Value Regularization
When extending 2 to multi-agent POMDP, we can get the following learning objective:
ILIEX E
(r(,, - af (Toot (aslo:) phot (as (a.lo.)))
However, unlike the single-agent setting, it is difficult to directly compute the regularization term between the global policy tot and the global behavior policy Meot due to the huge state- action space in multi-agent RL. Can we choose a function f to allow certain forms of policy decomposition so as to simplify the calculations?
In this work, we find that choosing the regularization function I to be the reverse KI. divergence, which means f Ptot) log(Trot/ (tot), leads natural decomposition of the global regulariza- tion into the summation of a set of local regularizations, by only assuming the factorization structure of the behavior policy (i.e., Ptor (alo) II b4 (a.lo:)). Note that using reverse KL divergence as the behavior constraint has been widely studied in both online and offfine single-agent RL literature 45 17, and has been shown to produce the nice mode-seeking behavior to avoid OOD actions in the offline setting [18). Plug in the specific form of f. and we obtain the following global policy evaluation operator:
(Triee) (o a) a)= r(o, a) + + 7ooooo (Vot (o'))
(Tree Vio(())= = = EEanTin a) + 7oolo Vooe (o').]
(4)

---

where
Teot (alo)
Voo Bawtiu
Qhae a.) or log
Pood (alo)
Theorem 4.1. Define Tj the case where the global policy in T7u is the optimal policy Tlot then Tj a 7-contructiion.
The proof Appendix A. This theorem indicates global Q-value will converge to the Q-value under the optimal policy Tiot when applying the seed-point iteration with the policy evaluation operator.
We now aim analyze and derive the closed-form solution of the optimal value functions Qiot and Vist According to the Karush-Kuhn-T Kuhk-TTucerr (KKT) conditions where the derivative of Lagrangian objective function with respect to the global policy is zero at the optimal solution, we have the following proposition:
Proposition 4.2. For a behaaior-regularized multi-agent POMDP with f( Mtat) Log(TTod ( the optimal global policy Tiot and its optimal value functions Vio and Vist sat- isfy the following optimality condition:
(o - a) + yooooo M (o')]
fo (0) = u0) +
rooe (alo) = Prot (ror(alo)) exp (Qixlo, - ur -1)
where u(o) is normalization term and has optimal value u² that makes the corresponding optimal policy Tiot satisfy CnEAA Fot o)
The proof Appendix A. Note that Proposition 4.2 can further simplified. Since Vise u²(o) + u² and Vior can be converted to each other without any approximation. For the global policy, replacing u² (o) with V (o) a, we get the following formulation:
Ci a) Viaat (0)
non (alo) = Meor ( (alo) exp
Now we have the relationship among the optimal global policy Tiat behavior policy Piots Q-value function Qiot and state-value function V Under the multi-agent setting, due to the exponential growth of the joint state-action space with the number of agents, these global values are hard to be evaluated. We show that tractable relationship between local policies and value functions.
resort to the value decomposition strategy further derive
4.2 Global-to-Local Value and Policy Decomposition
In this work, we introduce the following value decomposition scheme for both the global Q-value function and the state-value function:
Qroi(o, Σ (o(o))((or.aa) b(o)
Vio((0) Σ (o. b())
(8)
wi 2 0, Vi = -....... 1....
Compared with existing offline MARL methods
12 3 that only decompose the global Q-value
function Qtat- we additionally decompose the global state-value function Viot- The decomposition of Qiot and Veot share common weight function w; (o). since the credit assignment on Q and V should be related. It should also be noted that Vios is free of the joint action space and thus not affected by OOD actions under offline learning.
If we incorporate the value decomposition scheme Eq. g into the optimal global policy ni. in Eq. (7) and utilize the property of the exponential function, we can naturally decompose the optimal global policy The into combination of optimal local policies xę.
5

---

Proposition 4.3. Under the value decomposition scheme specified in Eq. (8) and assume the global behavior policy is decomposable (i.e. (roo(alo) II P4 (a)(o:)), we have fattalo) II के (a))o:), where x3 defined as:
w;(o) (0) -V vV((o)))
m7 (aslos) m=(a))o;) exp
(9)
The result immediately follows by observing that:
-V; Fot (a 0) = (tto((alo) exp Ei wi(() (Q)(o) -
a
wi(o) -V
II (,,aoo))) exp
II m (a:lo)
TT can be perceived as the optimal local policy under the behavior regularization, since the RHS of Eq. (9) only depends on the optimal local value functions Q Vi", and the local behavior policy μ4 of each agent i. This policy decomposition structure has a number of attractive characteristics. First, this design naturally bridges global value regularization and value decomposition. Second, (o) explicitly appears in the optimal local policy nH. which is calculated from global observation o. This makes the local policy optimizable with global information and consistent with the credit assignment in the multi-agent system.
4.3 Equivalent Implicit Local Value Regularizations
We have converted the relationship between global policies and values to the relationship between local policies and values. However, is still unclear how to calculate the optimal local value functions in Eq. (). Note that each local policy needs to satisfy Ea,e, m, (a.lo)) 1 in Eq. (9) in order to ensure is well-defined, is also worth noting that when each local policy कद satisfies this self nof-lnommaizationn constraint, the optimal global policy niot = II. `. will also satisfy EasA" Tioe (a) o) = 1. Thus, we only need to impose self-nom uuff-nommlization constraints on local policies. Integrating the RHS of Eq. (9) over the local action space, and have:
exp (~ -uu(o) (Q) a.) - V1" (o.))) (c.)))
(10)
We have now established the global-to-local relationships among the optimal values and policies, however, one annoying issue is that the global and local behavior policies Ptoo and 14, are typically unknown, and it is hard to estimate them accurately, especially when they multi-modal. However, perhaps surprisingly, we show that is possible to learn optimal local value functions an implicit way without knowing either jxo p4.
Proposition 4.4. V (o) can be obtained by solving the following convex optimization problem:
(Q, a.) Vi(o.)) + (o(
min E. V
exp
(11)
The proof follows that the first-order optimality condition of the above optimization objective (i.e., derivative with respect to V; equals 0) is exactly the condition Eq. (10).
The above optimization problem provides V. It is also worth noting that this objective actually corresponds to adding an equivalent implicit local value regularization on V., To this, note that in addition performing expected regression to let V; fit Q: by the first term, this objective also minimizes the second term w; (o,)) with respect to V, behavioral data /4,- This is similar performing conservative optimal value estimation in existing offline RL literature 20 27 that learns an underestimated value function to avoid the distributional shift.
learning objective for the local state-value function
4.4 Algorithm Summary
Based on previous analysis, we are now ready to present our final algorithm, OMIGA. OMIGA consists of three supervised learning steps: learning local -alue function Vú, learning global and

---

Algorithm Pseudocode of OMIGA
Require:
Offline dataset D. hyperparameter
1: Initialize local state-value network V, local action-value network Q, and its target network Q, and policy network и; for agent i=1, 2, ...
2: Initialize the weight function network w and b.
3: for 1, max-value -iterationn do
4: Sample batch transitions (o, oo) from D
5: Update local state-value function Vo(0;) for each agent i via Eq. 12
6: Compute (o"), Qrar(os a) via Eq. (8)
7: Update local action- value network (1 a:), weight function network w(o) and b(o) with objective Eq. (13).
8: Update local policy network πT for each agent i via Eq. (14
9: Soft update target network Qi(o; a4) by Qi(o,, a:) for each agent i.
end for
local Q-value functions, and learning local policies Tú. We can use Eq. value function V as:
to learn the optimal local
(i
(Q)lor,a)) vi(ci)) + w;(o)V((o,)
min E (o
exp
(12)
With the learned local V,,", we can jointly optimize the local Q-value function Qi the weight and offset function W; and 6 (i = 1,... n)) with the following objective, where Qtot and Veot are computed based on the value decomposition in Eq. (
min ROoDoDD (r(o,a) = VVin (o') - ao))
(13)
1=1.-mm
After obtaining the optimal local value functions, we can further learn the local policies n; by minimizing the KL divergence between π. and पद in Eq. (9) [44], which yields the following learning objective:
(u((o) (Q.(or, a.) - a)) v((ai)
max E
(o exp
log m (a:
(14)
Through the above three learning steps, we convert the initial intractable global behavior regularization in multi-agent POMDP to tractable implicit value regularizations at the local level. Moreover, all three steps are learned in a completely in-sample manner, which performs supervised learning only on dataset samples without the involvement of potentially OOD improving the training stability. We summarize the psuedocode of OMIGA in Algorithm [
actions, thus greatly
Discussion with prior works OMIGA draws connections with several prior works such as ICQ [1I. DMAC 45) and IVR [10]. ICQ bears some similarities with OMIGA, however, ICQ needs to estimate the normalizing partition function Z(s) = >n, H{ass) cxp (Q(s, a)/aa) to compute the weight in learning both Jn and JQ- Z is estimated by learning approximating with softmax operation over a mini-batch, which is hard to compute accurately, especially in the continuous action space. OMIGA uses Eq (TT) to provide new learning objective, only needs to solve in simple and elegant way by imposing seff-normalization constraints. The difference between OMIGA and DMAC that DMAC considers the KL divergence between current policy and previous policy, so that it can compute the KL divergence explicitly. While OMIGA considers the KL divergence between current policy and behavior policy, the behavior policy is typically unknown and hard to estimate, hence OMIGA proposes principled way to implicitly compute the KL divergence. Compared with IVR which proposes general implicit regularized RL framework in the offline single-agens agent setting, OMIGA starts from similar regularized framework designed for the offline multi-agent setting. Also, OMIGA transforms the regularization from global to local in the multi-agent setting, this provides valuable insights to the community, justifies why previous offlin multi-agent methods that apply local behavior regularization can work, and also reveals their limitations.
auxiliary behavior model or
7

---

Table 1: Average and standard deviations over 5 random seeds on the offline multi-agent MuloCo tasks and offline SMAC tasks.
Multi-agent MuJoCo
Task
Dataset
BCQ.MA
CQL.MA
ICQ
OMAR
OMIGA((urr)
Hopper
expert
77.85258.04
139.14a 313.83
754.74 806.28
2.362 1.46
859.633409.447
Hopper
medium
44.58=20.62
401.27=199.88
501.79:14.03
21.34+2490
1189.264 544.30
Hopper
medium-replay
26.53:24.04
31.37=15.16
195.392103.6 .61
3.3043.22
774.18=494.27
Hopper
mediumn expert
54.31423.66
64.82:123.31
355,44
1,44a0.86
709.00=595.66
Ant
expert
1517.73=286.28
1042:992222..5
2050.00211.86
312,595297,6
2055,4621..88
Ant
medium
1059.601a91.22
533.90a1 1766.42
1412.41=10.93
11710.04 918
1418.44+5.36
Ant
medium replay
950,77=48.76
234.6241618.22
1016.68+53.5
-211,200-84,,
1105. 17:88877
Ant
medium-cxpert
1020.89=2440
800.22+102 .52
1590. 18485.60 61
-2992.80st 6,95
1720.33+ 10.63
HalrChoctah
expert
2992.71=629.65
1189.44=1034.99
2955,942259.10
-206. 7166613111112
338336115552.7
HallCheetah
medium
2590:47::111.335
1011.351016..4
2549.27296.34
-265 21..68114466 66.98
3608, 13:.33.337
HalrCheetah
mediumm replay
333.64#152.06
1998.67=693,92
1922,42:612.88
235.422 15488
25:4,700:3..47
HalfCheetah
medium- expert
3543.70=780.89
1194233-181
2233,99=420.32
-253.84+ 84+ 63.94
2948.46* 518.89
SMAC
Task
Dataset
BCQ-MA
COL-MA
ICQ
OMAR
OMIGA(ours)
good
7.76g0.15
8.0820.21
7.8720.30
7.4020.63
8.25=0.37
Smn, 6m
medium
7.58+0.10
7.78=0.10
7,77=0.3
7,0840.51
7.92=10.57
poor
7.61=0.36
7,3+0.110
7,26+0.19
7,27+0.42
7.52+0.21
2e 642g
good
19.13=0.27
18.48a0.95
18.82:0.17
17.2740.78
19.15=0.32
2e vs. 642g
medium
15.58=0.37
12.82+1.61
15.57=0.61
10,20+0.20
16.02=0.19
vs, 642g
poir
12.46a0.18
10.8300.51
12.56a0.18
65_ 8z
good
12.1920.23
10.4920.20
1181=0012
9.8320288
12.54:0.21
medium
11.77:0.16
11.29
11.13:0,33
10.3640.16
12.1920.22
10,84=0.16
10.51
10.55=0.10
10.6340.25
11.31=0.19
corridor
good
15.2421.21
5,2220.81
15.5421.12
6.7420.69
15.8820.89
vorridor corridor
mediumn
11.30=1.77
4.4720.94
4.0820.60
4,4720.33
4,2820.49
5.6120.35
5
Experiment
5.1 Offline Multi-Agent Datasets
We choose multi-agent MuJoCo [5] and StarCraft Multi-Agent Challenge (SMAC) as our experiment environments. Multi-agent MuloCo is a benchmark for continuous multi-agent robotic control, based on MuJoCo environment. Currently, there is still lack of offline MARL datasets with continuous action. Thus, provide an offline multi-agent MuJoCo dataset which is collected from HAPPO [46] algorithm. The dataset contains multiple different data types and the details of the dataset are shown in Appendix B. Another benchmark used in this paper is SMAC, popular multi-agent cooperative environment for evaluating advanced MARL methods. The offline dataset we used is provided by Meng et al. [38], which is collected from the online trained MAPPO agents (47), and is the largest open offline dataset on SMAC. The offline dataset has three quality levels: good, medium, and poor. Besides, SMAC includes several StarCraft II multi-agent micromanagemeet maps. We consider 4 representative battle maps: 2 hard maps (5m_vs. _am__v__6_____ 6m, 2c_vs (_sz), and 2 super hard
maps (6h vs_ 8z, corridor).
5.2 Baselines
We compare OMIGA with four recent offline MARL algorithms: the multi-agent version of BCQ and CQL 20 (namely BCQ-MA and CQL-MA). ICQ [TT] and OMAR [12]. BCQ-MA and CQL-MA use linear weighted value decomposition structure as (1) for the multi-agent setting. Details for the implementations of baseline algorithms and the hyperparameters used in OMIGA and baselines are shown in Appendix C.
5.3 Comparative Evaluation
Table I shows the mean and standard deviation of average returns for the offline Multi-agent MuJoCo and SMAC tasks. Each algorithm is evaluated by using 32 independent episodes and runs with 5 seeds for training. The results show that OMIGA outperforms all baselines and achieves otate-of-the-we performance in most tasks.
For these multi-agent tasks, the credit assignment for multi-agent is complex and difficult to learn multi-agent cooperation in the offline setting. The global-to-local value regularization design of

---

FaliCheetah medium
8h vs & medium
HallChestsh medium
8h Bz medium
13
4000
30.00
Avereta retumn
3000
1u880 dBusay
11
2000
10
20000
10
1000
OMISA
3
OMIGA
1900
OMIGHosphamC.11 1)
CM
OMIGA-Iccal-W
OMIGA-Koal-W
ONIGNOppeT1)
OMIGA-wwo-w
0MIGHoopha-10
CMIGAaaphe-10)
500
08
Timestope (millon)
Episocess (thousand)
Timesteps (milion)
Episades (thousand)
(a) Analyses and ablations on weight function W),
(b) Analyses on different
Figure 1: Analyses and ablations.
our method reflects the global regularization in the local value function learning implicitly, which leads to better performance than other baseline algorithms. Moreover, the state-value function and Q-value function in OMIGA are completely learned in an in-sample manner without the involvement of the agent policy, which also leads to better training stability and performance. Besides, compared with baseline algorithms, the local policy learning in OMIGA contains global information, and thus enjoys obvious advantages in tasks with continuous action space like multi-agent MuJoCo.
We also conduct experiments on more diverse datasets which are obtained by mixing datasets of various quality. The results are shown in Appendix D. On these mixed datasets, the behavior policies can be suboptimal and more heterogeneous. Therefore, it difficult for algorithms such as BCQ-MA and OMAR to learn an accurate behavior policy, making implicit value learning with the regularization framework of OMIGA more appealing.
5.4 Analyses on Policy Learning with Global Information
As Eq. (12 and Eq. (14) show, the learning of both local state-value function V, and local policy ny in OMIGA contains global information, which is reflected in w: (o). Therefore, OMIGA can make the credit assignment during value and policy learning and leverage the global information to improve local policy. To verify the effect of this design, we transform OMIGA into two versions without global information, and then compare their performance on offline multi-agent MuJoCo and SMAC datasets. In OMIGA- -WoGG- local policy learning uses formula max,, Eto, a>))> (exp( (Q, a,) aaaaa///lVvV Vi(o)))) log (a)))):)) and the weight function is not available during the policy learning. In OMIGA-local-w the weight function is w,(o:) that only contains local information, and local policy learning uses formula max,, B(o;,a))-Dl0 (expp( (o.(o)) (Qi(o,, a,) - Vi(o,))) log (a;lo))))
Figure aa) shows the experimental results on offline HalfCheetah datasets and 6h_ _h__s__7 datasets. For these difficult multi-agent tasks, the cooperative relationships among agents are extremely complex, so global information is important guide local policy learning. OMIGA-W/o-w and OMIGA- local-w lack global information, and policies are only learned at the local level, which causes worse performance and stability.
5.5 Analyses on the Regularization Hyperparameter
In OMIGA, hyperparameter is used control the degree of regularization. The higher a encourages the policy to stay closer to the behavioral distribution, and the lower makes the policy more aggressive and optimistic. Figure [Tb) shows the experiments about a on the offline MuJoCo HalfCheetah task and SMAC 6h vs_ 8z task. In the HalfCheetah task, too high and too low a will lead to performance degradation. An appropriate a can ensure that the policy is restricted near the dataset distribution and also does not lose performance due to being too conservative. In the 6h vs 8z task, the dataset contains enough good transitions. Therefore, the degree of regularization should be higher to make the algorithm more conservative. Experimental results indicate that higher brings better performance on such datasets. When a is very small, the performance of the algorithm becomes very poor due to the lack of value regularization.

---

6 Conclusion
In this paper, we study the key challenge of the offline MARL problem. We start from the usage of global behavior constraints and smartly derive a global-to-local value regularization scheme under value decomposition. By doing so, we naturally get new offline MARL algorithm, OMIGA, that in principle applies global-level regularization but actually imposes equivalent implicit local-level regularization. Our work reveals why the local-Ievel regularization used by existing algorithms works and gives theoretical insights into their weaknesses and how to develop better algorithms. The global-to-local regularization design of OMIGA can capture global information and the impact of multi-agen agent credit assignment under the offline setting, which guarantees that the Iearned local policies are jointly optimal at the global level. One future work is to develop other offline MARL algorithms by using other choices of f functions.
Acknowledgments
This work is supported by National Key Research and Development Program of China under Grant (2022YB3200904) and funding from Global Data Solutions Limited.
References
[1] Yongcan Cao, Wenwu Yu, Wei Ren, and Guanrong Chen. An overview of recent progress in the study of distributed multi-agent coordination. IEEE Trunsactions on Industrial Informatics, 91):4277-438 2013.
[2] Dayong Ye, Minjie Zhang. and Yun Yang. A multi-agent framework for packet routing in wireless sensor networks. Sensors, 15(5): 10026-1047, 2015.
[3] Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemyslaw Debiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, et al. Dota 2 with large scale deep reinforcement learning. arXiv preprint arXiv:
2019.
[4] Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International conference on machine learning, pages 2052-2062.
PMLR, 2019.
[5] Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off- policy q-learning via bootstrapping eror reduction. Advances in Neural Information Processing Systems, 32, 2019.
[6] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative lelearning for offline reinforcement learning. Advances in Neural Information Processing Systems, 139- 1191, 2020.
[7] Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-leariin. International Conference on Learning Representations, 2021.
[8] Haoran Xu, Jiang Li, Jianxiong Li, and Xianyuan Zhan. A policy-guided imitation approach for offline reinforcement learning. In Advances in Neural Information Processing Systems, 2022.
[9] Jianxiog Li, Xianyuan Zhan, Haoran Xu, Xiangyu Zhu, Jingjing Liu, and Ya-Qin Zhang. When data geometry meets deep function: Generalizing offline reinforcement learning. In The Eleventh International Conference on Learning Representations, 2023.
[10] Haoran Xu, Li Jiang, Jianxiong Li, Zhuoran Yang, Zhaoran Wang. and Xianyuan Zhan. Offline rl with ood actions: In- In-sample learning via implicit value regularization. In International Conference on Learning Representations, 2023.
[11] Yiqin Yang, Xiaoteng Ma, Li Chenghao, Zewu Zheng, Qiyuan Zhang, Gao Huang, Jun Yang, and Qianchuan Zhao. Believe what you see: Implicit constraint approach for offline multi-agent reinforcement learning. Advances in Neural Information Processing Systems, 14:10299-10312, 2021.
[12] Ling Pan, Longbo Huang, Tengyu Ma, and Huazhe Xu. Plan better amid conservatism: Offline multi-agent reinforcement learning with actor rectification. International Conference on Machine Learning, pages 17221-17237. PMLR, 2022.
10

---

[13] Jiechuan Jiang and Zongging Lu. Offline decentralized multi-agent reinforcement learning. arXiv preprint arXiv:2108.01832, 2021.
[14] Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, OpenAl Pieter Abbeel, and Igor Mordatch, Multi- agent actor-critic for mixed cooperative-oompettivve environments. In Advances in Neural Information Processing Systems, volume 30, pages 6379-6390, 2017.
[15] Christian Schroeder de Witt, Bei Peng, Pierre-Alcxande Kamienny, Philip Torr. Wendelin Böhmer, and Shimon Whiteson. Deep multi-agent reinforcement learning for decentralized continuous cooperative control. arXiv preprint arXiv:2003.06709 19, 2020.
[16] Mikayel Samvelyan, Tabish Rashid, Christian Schroeder de Witt, Gregory Farquhar, Nantas Nardelli, Tim G. J. Rudner, Chia-Man Hung, Philip H. S. Torr, Jakob Foerster, and Shimon Whiteson. The starcraft multi-agent challenge. In Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems, pages 2186-2188, 2019.
[17] Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint 1ttp::9911.1166.. 2019.
[18] Haoran Xu, Xianyuan Zhan, Jianxiong Li, and Honglei Yin. Offline reinforcement learning with soft behavior regularization. arXiv preprint arXiv:21 10.00395, 2021.
[19] Peng Cheng, Xianyuan Zhan, Zhihao Wu, Wenjia Zhang, Shoucheng Song, Han Wang. Youfang Lin, and Li Jiang. Look beneath the surface: Exploiting fundamental symmetry for sample- efficient offline rl. Advances in Newral Information Processing Systems, 2023.
[20] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33:1179
1191, 2020.
[21] Ilya Kostrikov, Rob Fergus, Jonathan Tompson, and Ofir Nachum. Offline reinforcement learning with fisher divergence critic regularization. In International Conference on Machine Learning, pages 5774 5783. PMLR, 2021.
[22] Haoran Xu, Xianyuan Zhan, and Xiangyu Zhu. Constraints penalized q-learning for safe offline reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 8753-8760, 2022.
[23] Haoyi NIu, Yiwen Qiu, Ming Li, Guyue Zhou, Jianming Hu, and Xianyuan Zhan. When to trust your simulator: Dynamics-aware hybrid offline-and fidiie--nd-onliee reinforcement learning. Advances in Neural Information Processing Systems, 1-36599-366111 2022.
[24] Yue Wu, Shuangfei Zhai, Nitish Srivastava, Joshua Susskind, Jian Zhang, Ruslan Salakhutdinov, and Hanlin Goh. Uncertainty weighted actor-critic for offline reinforcement learning. arXiv
preprint arXiv:2105. 08140, 2021.
[25] Chenjia Bai, Lingxiao Wang, Zhuoran Yang. Zhi- Zhi-Hong Deng, Animesh Garg, Peng Liu, and Zhaoran Wang. Pessimistic bootstrapping for uncertainty-drivee offline reinforcement learning. In International Conference on Learning Representations, 2021.
[26) Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. Advances in Neural Information Processing Systems, 33: 2444129-14442 2020.
[27] Xianyuan Zhan, Haoran Xu, Yue Zhang, Xiangyu Zhu, Honglei Yin, and Yu Zheng. Deepther- mal: Combustion optimization for thermal power generating units using offline reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, 2022.
[28] Xianyuan Zhan, Xiangyu Zhu, and Haoran Xu. Model-based offline planning with trajectory pruning. In Proceedings of the 31st International Joint Conference on Artificial Intelligence, IJCAI 2022, 2022.
[29] David Brandfonbrener, Will Whitney, Rajesh Ranganath, and Joan Bruna. Offline r1 without ofolpy evaluation. Advances in Neural Information Processing Systems, 34:4933 4946, 2021.
[30] Pablo Hermandez-Leal, Bilal Kartal, and Matthew E. Taylor. A survey and critique of multiagent deep reinforcement learning. Autonomous Agents and Multi-Agent Systems, 33(6)7550-777, 2019.
11

---

[31] Frans A. Oliehoek, Matthijs T. J. Spaan, and Nikos Vlassis. Optimal and approximate q-value functions for decentralized pomdps. Journal of Artificial Intelligence Research, 32(1):289-355 2008.
[32] Landon Kraemer and Bikramjit Banerjee. Multi-agent reinforcement learning as a rehearsal for decentralized planning. Neurocomputingg 190:82-94, 2016.
[33] Peter Sunchag, Guy Lever, Audrunas Gruslys, Wojciech Marian Czarnecki, Vinícius Flores Zambaldi, Max Jaderberg, Marc Lanctot, Nicolas Sonnerat, Joel Leibo, Karl Tuyls, and Thore Graepel. Value- cellp-ddcommooitionn networks for cooperative multi-agent learning. arXiv preprint (766:02706 2017.
[34] Tabish Rashid, Mikayel Samvelyan, Christian Schroeder, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning. In International Conference on Machine Learning, pages 4292-4301, 2018.
[35] Kyunghwan Son, Daewoo Kim, Wan Ju Kang, David Earl Hostallero, and Yung Yi. Qtran: Learning to factorize with transformation for cooperative multi-agent reinforcement learning. In International Conference on Machine Learning, pages 5887-5896, 2019.
[36] Jianhao Wang. Zhizhou Ren, Terry Liu, Yang Yu, and Chongjie Zhang. Qplex: Duplex dueling multi-agent q-learning. arXiv preprint arXiv:2008. 01062, 2020.
[37] Xiangsen Wang and Xianyuan Zhan. Offline multi-agent reinforcement learning with coupled value factorization. In Proceedings of the 2023 International Conference on Autonomous Agents and Multiagent Systems, pages
2023.
[38] Linghui Meng. Muning Wen, Yaodong Yang. Chenyang Le, Xiyun Li, Weinan Zhang, Ying Wen, Haifeng Zhang. Jun Wang, and Bo Xu. Offline pre-trained multi-agent decision transformer: One big sequence model conquers all starcraftii tasks. arXiv preprint arXiv:21 (t.i:22/2.002455 2021.
[39] Wei-Cheng Tseng, Tsun-Hsuan Wang, Yen-Chen Lin, and Phillip Isola. Offline multi-agent rein- forcement learning with knowledge distillation. In Advances in Neural Information P Processing Systems, 2022.
[40] David Brandfonbrener, Alberto Bietti, Jacob Buckman, Romain Laroche, and Joan Bruna. When does return-conii cenuii-cond supervised learning work for offline reinforcement learning? In Advances in Neural Information Processing Systems, 2022.
[41]
Benjamin Eysenbach, Soumith Udatha, Sergey Levine, and Ruslan Salakhutdinov. Imitating past successes can be very suboptimal. In Advances in Neural Information Processing Systems, 2022.
[42] Craig Boutilier. Planning, learning and coordination in multiagent decision processes. In TARK, volume 96, pages 195-210. Citeseer. 1996.
[43] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with stochastic actor. In International conference on machine learning, pages 1861-1870. PMLR, 2018.
[44] Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. wdvantage-weighted regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv:1910.0011
2019. 2019.
[45] Kefan Su and Zongging Lu. Conference on Machine Learning, pages 20580-20603. PMLR, 2022.
regulariee multi-agen agent actor-critic. In International
[46] Jakub Grudzien Kuba, Ruiqing Chen, Muning Wen, Ying Wen, Fanglei Sun, Jun Wang, and Yaodong Yang. Trust region policy optimisation in multi-agent reinforcement learning. arXiv preprint arXiv:2109.11255: 2021.
[47] Chao Yu, Akash Velu, Eugene Vinitsky, Yu Wang. Alexandre M. Bayen, and Yi Wu. The surprising effectiveness of mappo in cooperative, multi-agent games. arXiv: Learning, 2021.
12

---

A Proofs
A.1 Proof of Theorem 4.1
Proof. For any two global state-value functions Via and 1/2 let mioo be the optimal global policy under 7j Voo and we get:
tot:
(T) Vo.) (o)
(T/ vioi)
Mloaaal)) o [Voo (o')] log
(aor(a)o)
sevs(alo)
r- TR [V2s (o')]
Tele(alo) (eae(()o)
(alo(
*tol
$ >Σ (aalalo) r [Vibe (o')] log
(2na(alo)
Σ (oo((alo)
r+ r 18o [Ve (o')] log
(alo
pecollalo)
(aoc(alo)
= > (allaa))) (Vie (o') vior (o')]
$ M//M Vo
Therefore, it follows that:
17 Viot Tÿ -TViiolloc S
Vior Vi2
A.2 Proof of Proposition 4.2
Proof. For a behavior-regularizee multi-agent POMDP with the behavior regularizer log(Trot/ Mrot), its learning objective be expressed as maX..:: E 12 7 (r (o,, at) alog (Trot (a:loz) /Poot (arlo,)))]. Its Lagrangian function
function f(Ttott ((ttot-HHttt)
can obtain when the optimal global policy written follows:
L( 3,
Σ do (0) (alo)a)o)
Σ
(Qiaio, a log
toor
Meot (alo)
[ Toot (alo)
Σ drec (o) u(o)
+ Σ ((alo)
where dzw.. is the stationary joint observation distribution of the global policy Toot. u and are Lagrangian multipliers for the equality and inequality constraints.
According the Karush-Kuthe Kann--uckhere (KKT) conditions where the derivative of the Lagrangian objective function with respect to the global policy is zero at the optimal solution, it follows that:
Toot T (alo)
Qtoc a) log
+1) u(o) +(aa))) =0
(15)
Meot (a
Σ (a))(a =1
((a (alo) =
0 < Nest (alo) < and < B(alo)
From Eq. 15 we can further solve the optimal global policy as:
Toot (alo) Mroo(aaos exp Qiai(o,a) u(o) + 3(aoo) -1)
The above formula can be further simplified. 3 is the Lagrangian multiplier, and meets com- plementary slackness 3(a ((o))))))(((a)))
0. Considering the joint observation o is fixed, exp Quaa(o.a)-u(0)--(aal)) -1) is always larger than 0. Therefore, for any positive probability action, its corresponding Lagrangian multiplier ((alo) 0. Therefore, Tead (alo) can be reformulated as:
Qroe a) - u(O) -1)
To (a (a = (toe(alo) exp
(16)
13

---

Bringing Eq. (16 into Ea tooe (too(a 1, we have:
Qtoe u()) -1) -1
E Bar
exp
(17)
a
The left side of Eq- (17) can be seen continuous and monotonic function of u, it has only one solution denoted as v and we denote the corresponding policy Toot as Toot-
Integrating Eq. 16 into the expression of optimal global state value, we can get:
Vioc (0) = Tj Vio
Teoe
Σ (alo) Qior((a,
-a aloo
(ttt(a)
Σ Ahoe (u (0) +a)
u² (o) +a
To summarize, we obtain the optimality condition of the behavior regularized MDP with Reverse KI. divergence as follows:
Vio a) = r(o, a) + y.Boolooa Viio (o')]
Vint (0) u (o) +a
u -1)
iot (alo) Prod (a exp
a
where u(o) is normalization term and has optimal value u² that makes the corresponding optimal policy not satisfy EnE," T ( = 1.
B Experiment Settings
B.1 Multi-Agent MuJoCo
Multi-agent Mujoco [[5 is benchmark framework developed for assessing and comparing the effectiveness of algorithms continuous multi-agent robotic control. Within this framework, robotic system is partitioned into independent agents, each tasked with controlling a specific set of joints. The agents collaborate harmoniously to accomplish shared objectives, such as acquiring the ability to walk through environment, with the ultimate goal of maximizing the cumulative reward. Multi-agent MuJoCo environment consists of multiple different robot configurations, and it is often used for the study of novel MARL algorithms for decentralized coordination in isolation.
To generate the dataset transitions, we captured the interactions between the environment and trained online MARL algorithms. Specifically, we use HAPPO 46 algorithm to collect data. The expert dataset is generated by employing the converged HAPPO algorithm. This involves training the algorithm until it reaches state of convergence, where the agents have learned optimal policies. The medium dataset generated by first training policy online using HAPPO, early-stopping the training, and collecting samples from this partially-trained policy. The medium-replee dataset consists of recording all samples the replay buffer observed during training until the policy reaches the medium level of performance. The medium- dataset is constructed by mixing equal amounts of expert demonstrations and suboptimal data. For all datasets, the hyperparameter env args.agent obsk (determines up which connection distance agents will be able to form observations) is set to 1. The average returns of our datasets are listed in Table 2
B.2
The StarCraft Multi-Agent Challenge
The StarCraft Multi-Agent Challenge (SMAC) benchmark is chosen as our testing environment. Due to its high control complexity, SMAC is a popular multi-agent cooperative control environment for evaluating advanced MARL methods. It consists of collection of StarCraft II micro- nicro-managemmntt
14

---

Table 2: The multi-agent MuJoCo datasets.
Scenario
Quality
Average Return
expert
2055.07
medium
1418.70
2-Agent Ant
medium-expert
1736.88
medium-replay
1029.51
expert
2452.02
medium
723.57
3-Agent Hopper
medium-expert
1190.61
medium-replay
746.42
expert
2785.10
medium
1425.66
6-Agent HalfCheetah
medium-expert 2105.38
medium-replay 655.76
tasks in which two groups of units engage in combat. Agents based on the MARL algorithm control the first group's 's units, while a built-in heuristic game Al bot with different difficulties controls the second group's units. Scenarios vary in terms of the initial location, number and type of units, and elevated or impassable terrain. The available actions for each agent include no operation, move[direction) attack [enemy id], and stop. The reward that each agent receives is the same. The hit-point damage dealt and received determines the agents' share of the reward. SMAC consists of several StarCraft II multi-agent micromanagement maps. We consider 4 representative battle maps, including 2 hard maps (5m_vs, pan_vs__m____ 2c_vs_64__ vs_647g), and 2 super hard maps (6h vs 8z, corridor), as our experiment tasks. The task type and other details of the maps are listed in Table 3
Table 3: SMAC maps for experiments.
Map Name
Ally Units
Enemy Units
Type homogeneous & asymmetric
5m_ vs 6m
Marines
Marines
2c_ vs ss_44zg
2 Colossi
64 Zerglings
micro-trick: positioning
6h vs__r _z
6 Hydralisks
Zealots
micro-trick: focus fire micro-trick: wall off
corridor
Zealots
24 Zerglings
The offline SMAC dataset used in this study is provided by 388, which is the largest open offline dataset SMAC. Different from single-agent offline datasets, considers the property of multi-agent POMDP, which owns local observations and available actions for each agent. The dataset is collected from the trained MAPPO agent and includes three quality levels: good, medium, and poor. For each original large dataset, we randomly sample 1000 episodes our dataset. The average returns of SMAC datasets are listed in Table
Table 4: The SMAC datasets.
Map Name
Quality good medium
Average Return
20.00
5m_vs_ vm
11.03
poor
8.50
good
19.94
2c_ 2e_vs__440 647g
medium
13.00
poor
8.89
good
17.84
6h jh_
medium
11.96
poor
9.12
good
19.88
corridor
medium
13.07
poor
4.93
15

---

C Implementation Details
C.1 Details of OMIGA
The local Q-value, state-value networks and policy networks of OMIGA are represented by 3-layer ReLU activated MLPs with 256 units for each hidden layer. For the weight network, 2-layer ReL.U-activated MLPs with 64 units for each hidden layer. All the networks are optimized by Adam optimizer.
C.2 Details of baselines
We compare OMIGA against four recent offline MARL algorithms: ICQ [TI. OMAR (12). BCQ- MA, and CQL-MA. For ICQ and OMAR, we implement them based the algorithm described in their papers. BCQ-MA is the multi-agent version of BCQ 4.. and CQL-MA is the multi-agent version of CQL 20 BCQ-MA and CQL-MA use the linear weighted value decomposition structure as Qtot > i= wy (i)Qi (o,, a,) b(o). w 2 0 for the multi-agent setting. The policy constraint of BCQ-MA and the value regularization of CQL-MA are both imposed on the local Q-value.
In this paper, all experiments are implemented with Pytorch and executed on NVIDIA V100 GPUs.
C.3 Hyperparameters
For multi-agent MuJoCo, the hyperparameters of OMIGA and baselines are listed in Table 5 An important hyperparameter of OMIGA is the regularization hyperparameter encourages OMIGA to stay near the behavioral distribution, and lower a makes OMIGA more optimistic. On most tasks, we use = 10 to ensure a good regularization effect. On the medium- quality dataset of the HalfCheetah task, we choose a
The higher
Table 5: Hyperparameters of OMIGA and baselines for multi-agent MuJoCo.
Hyperparameter
Value
Shared parameters
Q-value network learning rate
5e-4
Policy network learning rate
5e-4
Optimizer
Adam
Target update rate
0.005
Batch size
128
Discount factor
0.99
Hidden dimension
256
Weight network hidden dimension
64
OMIGA
State-value network learning rate
5e-4
Regularization parameter
1 or 10
Others
Lagrangian coefficient (ICQ)
10
Tradeoff factor a (OMAR, CQL-MA) 1
For SMAC, the hyperparameters of OMIGA and baselines are listed in Table On most tasks, we 10. On the poor dataset of the 6h vs 8z map, the quality of the dataset is relatively poor. It does not make much sense to make the policy close to the behavioral policy, so we choose a 2 to make the algorithm
radical.
most SMAC maps, the learning rate of all networks set to 5e-4. The exception the map 2c_vs_ vs. 64zg, on this map, the learning rate of all networks is set to 1c-4.
16

---

Table 6: Hyperparameters of OMIGA and baselines for SMAC.
Hyperparameter
Value
Shared parameters
Q-value network learning rate
5e-4 or le-4
Policy network learning rate
5e-4 or le-4
Optimizer
Adam
Target update rate
0,005
Batch size
128
Discount factor
0,99
Hidden dimension
256
Weight network hidden dimension
64
OMIGA
State-value network learning rate Regularization parameter
5e-4 or le-4
2 or 10
Others
Lagrangian coefficient (ICQ) Threshold (BCQ-MA) Tradeoff factor a (OMAR, CQL-MA)
10
0.3
Table Average scores and standard deviations over random seeds on the mixed offline SMAC datasets.
Map
Dataset
BCQ-MA
CQL-MA
ICQ
OMAR
OMIGA(ours)
6h_vs_ 87
good-poor
11.41
9.56+0.25
11.00+0.36
9.17=0.1
11.88+0.27
6h vs 8z
good-medium
11.79=0.29
10.08+0.26
11.18+0.25
10.0220.16
12.05+0.47
6h vs. 8z
medium-poor
11.18=0.41
10.73+0.38
11.25+0.35
10.42=0.19
11.85=0.35
corridor
good-poor
12.37#1.36
4.88+0.35
11.78&1.53
5.5420.75
13.01
corridor
good-medium
13.3240.71
5.77+1.30
12.9840.62
6.63+0.74
14.02+1.04
corridor
medium-poor
8.1120.35
6.18+0.59
8.2740.48
6,25 +0.48
9,70+1.40
D Additional Results
D.1 Results on mixed datasets
We also evaluated the performance of OMIGA on the multi-modal mixed datasets. Unlike BCQ-MA and OMAR, OMIGA doesn't 't need to learn a behavior policy. We choose two original datasets on the SMAC super hard maps 6h vs 8z and corridor and construct mixed datasets by combining these SMAC datasets of different quality, including good-poor, good-mediumg and medium modiuum-poor datasets. Each mixed dataset is blended by 50% of each of the two original datasets. On these mixed suboptimal datasets, the behavior policy heterogeneous, Therefore, is more difficult for algorithms such BCQ-MA and OMAR learn accurate behavior policy, making implicit value learning with the regularization framework of OMIGA more appealing.
Table shows the results that OMIGA consistently outperforms other offline MARL baselines under different mixed dataset experiments. Compared with the results on the original datasets, the performance of OMIGA has become more leading, indicating the benefits of implicit value regularization OMIGA.
17


Reviews:
[{"review_id": "PfXOcZ5pSI", "reviewer": "Reviewer_trCY", "summary": "This work proposes OMIGA, a new offline multi-agent reinforcement learning (MARL) algorithm with implicit global-to-local value regularization. The policy regularization from offline reinforcement learning (RL), for avoiding distribution shifts between learned policy and behavioral policy, is first brought to centralized training in MARL. Then, the global value functions and policies are decomposed into local agent-wise counterparts. Finally, using the self-normalization constraint of local policies, the objective functions for learning local value functions are derived, which leads to the OMIGA. The proposed method shows good performance compared to relevant existing works in offline single-agent and multi-agent RL, on MuJoCo and SMAC benchmark tasks.", "questions": "I have listed below my questions regarding my confusion regarding the derivations or missing details in the experiments.\n\n- Could you explain the derivation of Eq. (5)? Substituting $f(\\pi_{tot}, \\mu_{tot}) = \\log (\\pi_{tot} / \\mu_{tot})$ into the equation under L153 yields \n  - $r({\\boldsymbol{o}}_0, {\\boldsymbol{a}}_0) - \\alpha \\log(\\pi_{tot}({\\boldsymbol{a}}_0 | {\\boldsymbol{o}}_0) / \\mu_{tot}({\\boldsymbol{a}}_0 | {\\boldsymbol{o}}_0)) + \\sum_{t=1}^{\\infty} (\\cdots)$, \n  - but I am not sure if Eq.(5) would cover the $-\\alpha \\log(\\pi_{tot}({\\boldsymbol{a}}_0 | {\\boldsymbol{o}}_0) / \\mu_{tot}({\\boldsymbol{a}}_0 | {\\boldsymbol{o}}_0))$ term.\n\n- In the derivation of Eq. (10), it seems that the $b(\\boldsymbol{o})$ term from Eq. (9) is omitted for simplicity. Will this leave the rest of the exposition unchanged, particularly during the derivation of Eq. (11) and (12)?\n\n- In L222-223, it is mentioned that \"The proof [of Proposition 4.4] follows [from the fact] that the first-order optimality condition of the above optimization objective (i.e., derivative with respect to $V_i$ equals 0) is exactly the condition Eq. (11).\" Yet I could not comprehend how the solution to Eq. (11) is unique. In other words, it would be appreciated if the authors could explain how satisfying Eq. (11) will necessarily mean that the solution of Eq. (12) is $V_i^*$.\n\n- In the “Discussion with prior works” paragraph (L247-265), OMIGA is compared to ICQ, DMAC, and IVR. Among these, only ICQ is tested in the experiments. If feasible within the rebuttal period, could you also provide the experimental results of DMAC and IVR?\n\n- In L262-265, it is mentioned that propagating regularization in a global-to-local direction \"justifies why previous offline multi-agent methods that apply local behavior regularization can work.\" Could you provide an example of this insight on existing works, such as ICQ to which OMIGA is compared experimentally?\n\n- The pseudocode of OMIGA (Algorithm 1) uses a target network counterpart $\\bar Q_i$ for local action-value function $Q_i$, rather than for local state-value function $V_i$. Considering that target networks are often used for calculating targets in the critic loss, and that this target is $r(\\boldsymbol o, \\boldsymbol a) + \\gamma V_{tot}(\\boldsymbol o')$ in Eq. (14), it seems that the target networks should be kept for state-value functions instead.\n\n- The overall scales of the scores (i.e. hundreds to thousands) in the MuJoCo tasks for Table 1 do not seem to match the scales of the reward distribution (i.e. ones) of the datasets outlined in Table 2 (Appendix B). Could you explain this discrepancy?\n\n- How long were the OMIGA and the baseline methods trained for? Could you also provide the learning curves for the main results?\n\n- How large are the datasets for MuJoCo?\n\n- It seems from Table 6 (Appendix) that despite the statement that \"on these mixed datasets, the behavior policy is suboptimal and more heterogeneous\" (L300-301), all the methods achieve surprisingly good stability in terms of variance. Could you explain how this could occur not only for OMIGA but also for the other baseline methods?\n\n- Will the offline datasets for MuJoCo be publicly released? This would be an immense contribution to the offline MARL community.\n", "limitations": "No limitations or potential negative societal impacts have been discussed in the submission. A limitation of the proposed work is that the underlying global value functions $Q_{tot}^*$ and $V_{tot}^*$ are assumed to be a linear combination of the local value functions $Q_i^*$ and $V_i^*$ in Eq. (9). Future work may investigate the importance of extending to nonlinear combination of local value functions, such as those satisfying monotonicity constraints [1] or IGM conditions [2,3].\n\n[1] Rashid, Tabish, et al. \"Monotonic value function factorization for deep multi-agent reinforcement learning.\" The Journal of Machine Learning Research 21.1 (2020): 7234-7284.\n\n[2] Son, Kyunghwan, et al. \"Qtran: Learning to factorize with transformation for cooperative multi-agent reinforcement learning.\" International conference on machine learning. PMLR, 2019.\n\n[3] Wang, Jianhao, et al. \"Qplex: Duplex dueling multi-agent q-learning.\" arXiv preprint arXiv:2008.01062 (2020).", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This work seamlessly combined the main idea/challenges from offline RL (i.e. regularization for tackling distribution shift) and MARL (i.e. value decomposition for tackling scalability). Particularly, I deem incorporating global information into local value/policy learning as a crucial matter in the field of MARL in general. In this context, the stance of this paper is well-positioned.\n\n- The motivations and the significance of each step of deriving the OMIGA algorithm were addressed well.\n\n- Ablation studies appropriately tackle the main sources of design choices that could occur within OMIGA: the incorporation of global observation $\\boldsymbol{o}$ and the regularization parameter $\\alpha$.\n", "weaknesses": "While it was difficult to find a critical weakness of the proposed algorithm within the scope of this paper, I have numerous questions regarding the derivation of the OMIGA algorithm as well as the presented results. These questions are listed in the \"Questions\" section.\n\nMeanwhile, I outline below some of the confusing typos or slight inconsistencies that might confuse the readers:\n\n- Notation mismatch between the joint policy $\\pi_{tot}$ (L112) and $\\boldsymbol\\pi$ (L113)\n\n- L136: It seems that the “derivation of $\\pi$ and $\\mu$” should actually be the “deviation between $\\pi$ and $\\mu$”.\n\n- The operator $\\mathcal T_f^*$ has been defined for the action-value function $Q$ in Eq. (5) but is applied on state-value functions $V_{tot}$’s throughout Appendix Section A.1 and in the equation under L492 (Appendix). While its definition is obvious in Section A.1, it would be appreciated if you could explicitly provide the definition.\n\n- The $\\beta(a|s)$ in L487 (Appendix A.2) should be $\\beta(\\boldsymbol{a}|\\boldsymbol{o})$.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review. We appreciate for minor writing typos you posted, we will correct those typos in our revision.\n\n1. **The derivation of Eq. (5).** The term, $ -\\alpha \\log( \\frac{\\pi\\_{tot}({\\boldsymbol{a}}_0 | {\\boldsymbol{o}}_0)} {\\mu\\_{tot} ({\\boldsymbol{a}}_0 | {\\boldsymbol{o}}_0)} ) $ is contained in the value function V (see Equation (6)). Note that our derivation is almost the same as SAC, see Equation (2) and (3) in [1]. Actually, if we put Equation (5) into Equation (6), we can get $ V(o) = E\\_{\\pi} \\left[ r - \\alpha \\log ( \\frac{\\pi}{\\mu} )) + V(o') \\right] $. When optimizing this \"modified\" value function, we are actually solving the origin MDP with a modified reward $ \\hat{r} = r - \\alpha \\log (\\pi / \\mu) $, which is exactly the Equation in line 153.\n\n2. **The derivation of Equation (10).**  We apologize that we make a typo in Equation 9, there should not have b(0) in it. $Q_{tot}(\\boldsymbol{o}, \\boldsymbol{a})=\\sum_i w_i(\\boldsymbol{o}) Q_i(o_i, a_i)+b(\\boldsymbol{o})$, $ V_{tot}(\\boldsymbol{o})=\\sum_i w_i(\\boldsymbol{o}) V_i(o_i)+b(\\boldsymbol{o})$,  $w_i \\geq 0, \\forall i=1 \\cdots, n$, so the global advantage should be expressed as $A_{tot}(\\boldsymbol{o}, \\boldsymbol{a})= Q_{tot}(\\boldsymbol{o}, \\boldsymbol{a})-V_{tot}(\\boldsymbol{o})=\\sum_i w_i(\\boldsymbol{o})(Q_i(o_i, a_i)-V_i(o_i))= \\sum_i w_i(\\boldsymbol{o}) A_i(o_i, a_i)$. Note that this typo does not affect the subsequent derivation, thank you for pointing out that.\n\n3. **The proof [of Proposition 4.4].** First we apologize for a typo in Equation (12): $Q$ should be $ Q^* $. The solution to Eq. (12) is unique because Eq. (12) is convex with respect to V. \nIn a convex optimization problem, the feasible region -- the intersection of convex constraint functions -- is a convex region. With a convex objective and a convex feasible region, there can be only one optimal solution, which is globally optimal, that is $ V^* $ in Eq. (11).\n\n4. **The experiments of DMAC and IVR.** Although we discuss the relationship between our method and DMAC and IVR, it is not necessary to add the experiments of DMAC and IVR. DMAC is an online MARL algorithm, and IVR is a single-agent offline RL algorithm. Neither DMAC nor IVR  can be directly applied to our offline multi-agent experiments. Our discussion with DMAC and IVR is only a theoretical explanation of the correlation between our method and these algorithms.\n\n5. **Example of this insight on existing works.** Some previous work, like MA-BCQ, MA-CQL, OMAR, apply local regularization per agent and achieve good results. However, this is a kind of ad-hoc choice by reusing techniques from offline single-agent RL to multi-agent setting, there's no guarantee that doing so is correctly applying global regularization. Our work starts from the correct thing, aka, global regularization and shows it could be decomposed into local ones under certain cases.\n\n6. **About the target network** Yes, target networks are often used for calculating targets in the critic loss. In our algorithm, state value function V are actually playing the role of Q target. However, note that V is learned by optimizing Equation (13), so we use Q target in Equation (13). In other words, we want to V to approximate $\\max\\_{a\\_i^{\\prime}  \\text { s.t. } \\mu\\_i\\left(a\\_i^{\\prime} \\mid o\\_i^{\\prime}\\right)>0 } \\bar{Q}\\_i (o'\\_i, a'\\_i)$.\n\n7. **The scales of the scores.**  Table 2 (Appendix B) shows the distribution of one-step rewards for each task. However, Table 1 shows the average returns for each task. The return is calculated from one episode and contains many timesteps.\n\n8. **Training time.** For multi-agent MuJoCo tasks, we train all methods for 200 million timesteps. For SMAC tasks, we train all methods for 50000 episodes. We will provide the learning curves in the subsequent versions of the paper.\n\n9. **The size of multi-agent MuJoCo datasets.**  Each task includes 1 million timesteps data.\n\n10.  **The good stability in terms of variance.**  It has a certain relationship with the experimental environment. In SMAC,  the environment will normalize the final return within a narrow range. Thus,  sometimes the final performance of the algorithm is relatively close, and the variance appears low. However, our algorithm still outperforms other baseline methods.\n\n11. **The question of whether to release the MuJoCo dataset.** Yes, we will publicly  release our  offline datasets and open source our code.\n\n[1] Soft Actor-Critic Algorithms and Applications, arXiv preprint 2018."}, {"review_id": "9QpVxqYCbp", "reviewer": "Reviewer_LQB9", "summary": "The paper proposed a novel offline MARL algorithm with named OMIGA. OMIGA transfers the global entropy regularization into local regularizations implicitly under the linear weighted value decomposition assumption. The empirical results on multi-agent MuJoCo and SMAC suggest that OMIGA almost outperforms current offline multi-agent SOTA baselines.", "questions": "1. Could you explain why you choose the hyper-parameter for the baselines as in the paper? The hyper-parameters seem different in their origin papers.\n2. Do you try the dataset collected from other MARL algorithms (other than HAPPO, MAPPO)? What is the performance of OMIGA?\n", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "Originality: OMIGA provides a novel principled framework to decompose global regularization into equivalent implicit local regularizations for offline MARL.\nQuality: The theoretical analyze supports the conversion of the global regulation and the empirical results shows that OMIGA outperforms current SOTA baselines under specific hyper-parameters.\nClarity: The paper is clearly written and well organized.\nSignificance: OMIGA almost outperforms current offline multi-agent SOTA baselines in the experiments.\n", "weaknesses": "The hyper-parameters seem unfair. For example, in the ICQ paper, the performance of ICQ is significantly improved when the Lagrangian coefficient changes from 10 to 100 for SMAC tasks, while this paper chooses 10 for ICQ.\nMirror: The variables $a_t$, $s_t$ in the RHS of Eq. (4) should be $a$, $s$ (below line 141).\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review.\n\n1.**The choice of hyper-parameter for baseline.**  The SMAC datasets used in the ICQ paper are generated by the authors, and the SMAC datasets in our paper come from publicly available datasets [1], which is the largest open offline dataset on SMAC.  These two datasets were collected by different algorithms (The datasets we use are collected by MAPPO, while the datasets for ICQ are collected by DOP), and the quality of these two datasets also varies  (The datasets we use include 3 different qualities, while the datasets for ICQ are mixing datasets). Due to the different datasets we use,  there are certain differences in the selection of hyperparameters. \n\nIn the ICQ paper, the Lagrangian coefficient for SMAC is set to 100 or 1000, and in our paper it is set to 10.  We also supplement the experimental results  of one seed for Lagrangian coefficient = 100 and Lagrangian coefficient = 1000 on the SMAC tasks.\n\n| task       | dataset | ICQ(1000) | **ICQ(100)** |  ICQ(10)   | **OMIGA**  |\n| ---------- | :-----: | :-------: | :----------: | :--------: | :--------: |\n| 5m_vs_6m   |  good   |   7.46    |     7.62     | 7.87±0.30  | 8.25±0.37  |\n| 5m_vs_6m   | medium  |   7.45    |     7.70     |  7.77±0.3  | 7.92±0.57  |\n| 5m_vs_6m   |  poor   |   7.43    |     7.45     | 7.26±0.19  | 7.52±0.21  |\n| 2c_vs_64zg |  good   |   17.99   |    19.30     | 18.82±0.17 | 19.15±0.32 |\n| 2c_vs_64zg | medium  |   14.89   |    16.01     | 15.57±0.61 | 16.03±0.19 |\n| 2c_vs_64zg |  poor   |   11.05   |    10.94     | 12.56±0.18 | 13.02±0.66 |\n| 6h_vs_8z   |  good   |   11.69   |    11.12     | 11.81±0.12 | 12.54±0.21 |\n| 6h_vs_8z   | medium  |   11.24   |    11.01     | 11.13±0.33 | 12.19±0.22 |\n| 6h_vs_8z   |  poor   |   9.99    |    10.25     | 10.55±0.10 | 11.31±0.19 |\n| corridor   |  good   |   15.94   |    17.06     | 15.54±1.12 | 15.88±0.89 |\n| corridor   | medium  |   10.66   |    10.37     | 11.30±1.57 | 11.66±1.30 |\n| corridor   |  poor   |   3.27    |     2.60     | 4.47±0.33  | 5.61±0.35  |\n\nIt can be seen that a higher Lagrangian coefficient does not bring significant improvement to ICQ algorithm. In many tasks, a higher Lagrangian coefficient leads to a decrease in ICQ performance.\n\nIn fact, we have tested the baseline algorithms before setting their hyperparameters, and there are no unfair hyperparameters.\n\n2.**The dataset collected from other MARL.**  Currently, we have only conducted experiments on the datasets in the paper. Both HAPPO and MAPPO are widely recognized MARL algorithms. The datasets that we use contain a variety of quality data, and also contain the data from training processes (such as the medium-replay dataset). Thus, the datasets we use in this paper are convincing. We will supplement experiments on other datasets in the future.\n\n[1] Offline pre-trained multi-agent decision transformer: One big sequence model conquers all starcraftii tasks, arXiv preprint 2021."}, {"review_id": "A5z69vHaGR", "reviewer": "Reviewer_xgpF", "summary": "This paper focuses on offline multi-agent reinforcement learning in the centralized training decentralized execution setting and proposes a SARSA-like approach that leverages value decomposition. Drawing from entropy-regularized and maxent RL, the authors propose a Behavior-Regularized formulation for Offline RL. The method is tested on offline MAMuJoCo datasets generated by the authors as well as SMAC datasets. The baselines are two single agent methods (BCQ and CQL) that the authors extend to the multi-agent setting and two multi-agent offline methods (ICQ and OMAR). Authors also investigate two ablations of their method.", "questions": "- the very poor performance of OMAR on Mujoco in table 1. is surprising as OMAR is reported to outperform MA-ICQ and MA-CQL in multi-agent continuous control [4].\n- how were the hyperparameters tuned and selected for OMIGA and the baselines on each task? \n- l.192 have authors investigated using different weights for V and Q?\n- l.51 \"some particular choice\" of what? \n\n### Summary of review\nDespite offline MARL being a challenge of significant importance, and the behavior-regularized formulation proposed here being interesting, I believe that the presented method and its evaluations have too many flaws to allow for publication. \n\n### references\n[1] Kostrikov, Ilya, Ashvin Nair, and Sergey Levine. \"Offline Reinforcement Learning with Implicit Q-Learning.\" International Conference on Learning Representations (2022).\n\n[2] Meng, Linghui, et al. \"Offline pre-trained multi-agent decision transformer: One big sequence model conquers all starcraftii tasks.\" arXiv preprint arXiv:2112.02845 (2021).\n\n[3] Barde, Paul, et al. \"A Model-Based Solution to the Offline Multi-Agent Reinforcement Learning Coordination Problem.\" arXiv preprint arXiv:2305.17198 (2023).\n \n[4] Pan, Ling, et al. \"Plan better amid conservatism: Offline multi-agent reinforcement learning with actor rectification.\" International Conference on Machine Learning. PMLR, 2022.", "limitations": "n/a", "rating": 5, "confidence": 4, "soundness": 1, "presentation": 2, "contribution": 3, "strengths": "- Offline multi-agent reinforcement learning is of significant interest. \n- Behavior-Regularized formulation of Offline RL is interesting. \n- The paper is overall easy to read and understand.", "weaknesses": "I believe there are major flaws with the method and the experimental procedure. I think that the method is quite similar to Behavioural Cloning (BC) and is unlikely to improve much on it or on the datasets' performance. This failed to be evidenced by the experimental results because 1) the method is not compared against BC and 2) the datasets' average and median scores are not reported. Additionally, I think that BCQ and CQL have been poorly extended into multi-agent baselines and that how results are presented in table 1 misleads the reader into believing that the presented method significantly outperforms the baselines. I detailed this point below\n \n\n- The method does not improve on BC: Eq. (15) would yield pi^star, but only if Q and V are Q^star and V^star. Which is not the case since they are learned from (14) which simply learns the Q value of the behaviour policy on the dataset (there is no greedification happening here) and V is just learned to normalize the policy to one. Therefore, the only greedification step is (15) (i.e. pi is a Boltzmann policy w.r.t. to the behaviour policy values). Yet, since the learned policy does not influence any of the value learning objectives, this does not greedify the value functions. Therefore, I believe that the presented method is very close to just doing BC. For comparison, consider the greedification steps of two other SARSA-like offline RL approaches, IQL [1] and ICQ: IQL explicitly greedifies the learned values from expectile-learning before doing AWR (AWR being equivalent to (15)).  In ICQ, the greedified policy greedifies the value learning through the importance sampling correction. \n\n    Therefore, I doubt that the presented method improves on the simple BC approach. Comparison to BC and reporting the datasets' performance is essential in order to show that the method can improve on the datasets and BC baseline. Regarding that last point, by quickly looking at the results in [2], it seems that OMIGA is not really able to improve on BC or match the average dataset performance on SMAC.  \n\n    I would also recommend comparing with a multi-agent version of IQL and independent-TD3+BC that explicitly regularizes the offline policy with BC (see for instance baselines description in [3]).  \n\n- Table 1 is misleading: bold values should denote values that are above others with statistical significance. Yet, in Table 1. Most bold values' confidence interval overlap with the ones of other values (at least for SMAC). This is quite misleading as it suggest that one method outperforms the others, while actually most methods are equivalent statistically speaking.\n\n- BCQ and CQL have been poorly extended into multi-agent baselines: the multi-agent extensions use centralized values (with value decomposition) but decentralized regularization, which seems to be specifically what authors have shown not to work in their ablation of OMIGA. I say \"seem\" because it is unclear from section 5.4 whether or not values are learned with a centralized value and value decomposition.\n    I believe that MA-BCQ and MA-CQL should be either fully decentralized ([4] shows that fully decentralized tends to be better for offline MARL) or with centralized regularization (like OMIGA). \n\n\nEDIT: I have raised my score to 5 after the discussion period", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the comments. However, I believe some of your understanding of our paper is wrong. See the reply as follows.\n\n**The method does not improve on BC.** Our method is totally different from BC. \n- From one perspective, our overall learning objective is Equation (2), which adds an behavior regularization term to the reward. In Equation (2), hyperparameter $\\alpha$ controls the strength between doing Q-learning and doing BC. If $\\alpha=0$, the learning objective return backs to the original RL objective. Due to the reason that $ V^* $, $ Q^* $ and $ \\pi^* $ (in Proposition 4.2) are closed-form solution of Equation (2) and $Q$, $V$ will converge to the $ Q^* $, $ V^* $ in Theorem 1, our method is definitely doing RL rather than BC.\n- From another perspective, in OMIGA, there exists one greedification step in **learning the value function V**. In Equation (13), the first term always pushes up V-values and the gradient becomes larger if Q − V is bigger than 0 while the second term always pushes down V-values, and $\\alpha$ trades off these two terms. This actually doing implicit maximization over actions in the dataset support, in other words, V is fitting $\\max\\_{a\\_i^{\\prime}  \\text { s.t. } \\mu\\_i\\left(a\\_i^{\\prime} \\mid o\\_i^{\\prime}\\right)>0 } \\bar{Q}\\_i (o'\\_i, a'\\_i)$. Please refer to Appendix A in [1] to get a more concrete understanding about the learning objective of V.\nWe also add the results of BC against OMIGA in the following table, it can be seen that OMIGA outperforms BC by a large margin.\n\n|           task            | **BC**  |       task        | **BC** |\n| :-----------------------: | :-----: | :---------------: | :----: |\n|       Hopper expert       |  96.92  |   5m_vs_6m good   |  7.13  |\n|       Hopper medium       | 514.95  |  5m_vs_6m medium  |  7.31  |\n|   Hopper medium-replay    | 192.22  |   5m_vs_6m poor   |  6.44  |\n|   Hopper medium-expert    |  90.18  |  2c_vs_64zg good  | 16.42  |\n|        Ant expert         | 2040.41 | 2c_vs_64zg medium | 14.80  |\n|        Ant medium         | 1418.06 |  2c_vs_64zg poor  | 11.12  |\n|     Ant medium-replay     | 999.88  |   6h_vs_8z good   | 11.91  |\n|     Ant medium-expert     | 1553.84 |  6h_vs_8z medium  |  9.67  |\n|    HalfCheetah expert     | 2932.75 |   6h_vs_8z poor   |  8.99  |\n|    HalfCheetah medium     | 2480.66 |   corridor good   | 10.14  |\n| HalfCheetah medium-replay | 2248.55 |  corridor medium  |  8.29  |\n| HalfCheetah medium-expert | 1846.96 |   corridor poor   |  3.44  |\n\n**The question of Table 1.** For multi-agent MuJoCo tasks, our algorithm performs significantly better than the baseline algorithms. For SMAC tasks, the environment will normalize the final return within a narrow range. Thus, in some tasks, the gap between algorithms may seem not very huge. Overall, our algorithm consistently outperforms other baseline methods.\n\n**The performance of OMAR.** Due to the different datasets used in our paper and OMAR, it is possible to obtain different results. Additionally, as OMAR is an improvement based on CQL-MA, its performance will be affected by CQL-MA. Our implementation of OMAR is based on the official code of OMAR, and our code implementation of OMAR is included in the supplementary materials.\n\n**The question of baselines.** Currently, the multi-agent versions of BCQ and CQL are still very popular offline multi-agent baselines. Papers [2] [3] [4]  all use the multi-agent versions of BCQ and CQL as baselines. They all use CTDE framework and value decomposition with local-level regularization as our baselines.  Due to the impact of partial observability, non-stationary environment and the problem of credit assignment, the fully decentralized algorithm cannot achieve good results in MARL. For most MARL algorithms, it is common to use CTDE or fully centralized framework to replace decentralized training.\n\n**The question of hyperparameters.** One advantage of the OMIGA algorithm is that only $\\alpha$ is the main hyperparameter. Due to the limited number of hyperparameters in OMIGA, parameter tuning is very easy, which means hyperparameters can be selected with a simple grid search. Besides, we have tested the baseline algorithms before setting their hyperparameters, and there are no unfair hyperparameters.\n\n**The question of  the weight function.** The reason why we use the same weight function for V and Q is that the credit assignment on the global Q and V should be related. If different weight functions are used for V and Q, it is difficult to guarantee that the algorithm will learn this correlation.  Another benefit of using the same weight function for V and Q is that the formula can be simplified in the subsequent derivation process. It also avoids learning too many parameters, which affects the convergence speed of the algorithm.\n\n**The problem of l.51.** It refers to the content in Eq. (7),  Eq. (8) and section 4.2.  After getting a behavior-regularized MDP with a global-level regularization,  we use Lagrangian objective function and  Karush-Kuhn-Tucker (KKT) conditions to get Proposition 4.2 (Appendix A.2).  Next, we utilize the value decomposition and the property of the exponential function, which enables the optimal global policy $\\pi^*_{tot}$ can be decomposed into a combination of optimal local policies $\\pi^*_{i}$ (Proposition 4.3). It naturally bridges global value regularization and value decomposition, and converts the relationship between global policies and values to the relationship between local policies and values.\n\n[1] Offline RL with No OOD Actions: In-Sample Learning via Implicit Value Regularization, ICLR 2023.\n\n[2] Believe What You See: Implicit Constraint Approach for Offline Multi-Agent Reinforcement Learning, NeurIPS 2021\n\n[3] Offline Multi-Agent Reinforcement Learning with Coupled Value Factorization, AAMAS 2023\n\n[4] Offline pre-trained multi-agent decision transformer: One big sequence model conquers all starcraftii tasks, arXiv preprint 2021."}, {"review_id": "yPmIcTusrr", "reviewer": "Reviewer_WVBc", "summary": "The paper investigates multi-agent offline reinforcement learning, and address some of the problems that arise with applying regularization at either the global or local level. To achieve this, a framework that allows to apply global regularization via an equivalent set of local regularizations is proposed. This way, the CTDE paradigm and value-decomposition scheme can be applied, and decentralized policy are easily learned, while at the same time retaining the positive effect and guarantees that global regularization has.", "questions": "- In the usual Dec-POMDP framework, the global Q-function does not condition on the joint observation $\\mathbf{o}$, but rather on the joint action-observation history $\\tau$ (and/or on the true environment state $s$). This is because the joint observation $\\mathbf{o}$ is not a sufficient statistics for the agents to learn an optimal policy $\\pi_{tot}^*$ (except for some particular subclasses of Dec-POMDP, like in Dec-MDP, where the joint observation $\\mathbf{o}$ forms the true underlying state $s$ itself, and thus reactive policies, i.e. policies conditioning on local observations $o_i$ rather than on local histories $\\tau_i$, can still achieve the optimal solution). Is your framework specifically for this very special kind of settings, or it also works on the more general, history-based, setting? Is it is the latter, please correct the notation to align it with the existing literature, otherwise please argument such limitation.\n- From Section 4.1 onward, you start writing the reward as a function of the joint observation $\\mathbf{o}_i$ rather than the true state $s_t$. This is not true in general (unless in a Dec-MDP, see my point above), and you should adjust this to reflect the actual environment you have introduced earlier on.\n- Where does the bias term $b(\\mathbf{o})$ goes in Equation (10) and the following?\n- Equation (11) is not immediate: please argument a bit more on why this is actually the case.\n- One thing that immediately comes to mind in this setting is that, in order to collect a properly various and useful dataset, a very good behavioural policy is generally needed. Indeed, in the Experiments section, you say that you use a trained HAPPO policy to generate the MA-MuJoCo dataset (similarly, the SMAC dataset comes from a trained MAPPO policy). While in actual supervised learning the dataset comes from whatever source of information, and has nothing to do with the final classifier that we aim to learn, here in order to learn a policy we need another (behavioural) policy to generate these data, and the policy itself has to be good enough in order to generate a dataset with also successful and good transitions that can improve the learned policy. I would like to see a bit of discussion with respect to this point: how good has the behavioural policy to be? How much of an improvement can we gain with offline learning over that (if there is no improvement, there would be no point in learning a new policy over the behavioural one that generated the dataset)? How difficult is this kind of approach in relation to generate the required datasets? This kind of questions are unique to the offline RL setting, and does not have a counterpart in standard supervised learning, and thus I think there deserves some discussion and clarification to properly assess the merits and eventual limitations of this family of algorithms. I appreciate that section 5.5 tries to go in that direction a bit, by showing how close or far the learned policy wants to be w.r.t. the behavioural policy. This is a nice ablation study, and worth a bit more investigation in my opinion.\n\nTypos:\n- Line 54: perhaps you mean dub (rather than dumb)?\n- Joint observations $\\mathbf{o}$ are never introduced explicitly, although used in the work.\n- Line 113: in the subscript of the expectation, shouldn't that be $\\pi_{tot}$?\n- Line 129: the weight is $w_i(\\mathbf{o})$ alone, not the entire $w_i(\\mathbf{o})Q_i(o_i,a_i)$.\n- Line 177: value functions\n- Line 206: optimizable (rather than \"can be optimized\")\n- Line 212: $\\pi_i^*$\n- Line 229: learns\n- Please expand a bit the caption of Figure 1 by describing a bit more in details what it represents.", "limitations": "No clear statement of possible limitations is provided in the paper.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper investigates an interesting area: offline reinforcement learning can be a viable way to learn in those real-world settings where actual environment interaction is not possible or too expensive. The application of the CTDE paradigm to effectively translate the global terms into local terms that are easier to learn and use is well aligned with the current research track on this line. The resulting algorithm is easy to follow and implement, and its performances are shown to be superior to other exiting offline RL algorithms with more complex regularization terms. Also, the paper is nicely written and clear to follow, and the empirical results are quite convincing.", "weaknesses": "However, the paper in my opinion is a bit sloppy when it comes to a proper characterization of the used setting, and the actual working mechanism of the various components (policies, value-functions...) that are used. A precise mathematical formulation is a very useful point for a paper, as it allows the reader to relate the present work to the previous literature. Please see my questions below for a more precise explanation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review.\n\n1.**Policies conditioning on local observations or local histories.**   Our framework is not designed for the special setting that only considers the local observation.  It can also be applied to the action-observation history based algorithms. There is no special reason for using observations to calculate the Q-value. Although in many MARL algorithms Q-value conditions on the action-observation history $\\tau$, there are still some algorithms where Q-value conditions on the observation $o$ [1] [2] [3] [4]. Thus, using the observation to calculate Q-values is not a special operation.  During the experiment, for our method OMIGA and baseline methods, we use the same input when calculating Q-values, so the effectiveness of our method is independent of whether we choose observation $o$ or history $\\tau$.  We will also supplement experiments based on action-observation history in the future.\n\n2.**The reward function based on the joint observation or the true state.**  The reward value is indeed given by the environment based on the global state. We directly use the reward  returned by the environment, which is not related to the design of the algorithm.\n\n3.**The question of Equation(10).** What we want to express in Equation (9) is $Q_{tot}(\\boldsymbol{o}, \\boldsymbol{a})=\\sum_i w_i(\\boldsymbol{o}) Q_i(o_i, a_i)+b(\\boldsymbol{o})$, $ V_{tot}(\\boldsymbol{o})=\\sum_i w_i(\\boldsymbol{o}) V_i(o_i)+b(\\boldsymbol{o})$,  $w_i \\geq 0, \\forall i=1 \\cdots, n$,  so the global advantage should be expressed as $A_{tot}(\\boldsymbol{o}, \\boldsymbol{a})= Q_{tot}(\\boldsymbol{o}, \\boldsymbol{a})-V_{tot}(\\boldsymbol{o})=\\sum_i w_i(\\boldsymbol{o})(Q_i(o_i, a_i)-V_i(o_i))= \\sum_i w_i(\\boldsymbol{o}) A_i(o_i, a_i)$. Equation (9) has a typo, and the corrected  Equation (9)  will derive  Equation (10).  Thank you for pointing out.\n\n4.**About Equation (11).** The solution to Eq. (12) is unique because Eq. (12) is convex with respect to V. In a convex optimization problem, the feasible region -- the intersection of convex constraint functions -- is a convex region. With a convex objective and a convex feasible region, there can be only one optimal solution, which is globally optimal, that is $ V^* $ in Eq. (11).\n\n5.**More discussion.** Thanks for pointing out that. That's a good question. In our paper, we follow those works in the offline single-agent RL setting that collected data using trained RL agents. I think that it is a general question about the offline RL setting: How good has the behavioral policy to be and how much of an improvement can we gain with offline learning. From our experience, the datasets may not need to be expert. Diversity and mediocre quality are needed for the agent to achieve a high score. For example, in hopper-medium datasets, some offline single RL methods can achieve near expert score. When the datasets’ quality is low or the diversity is limited, there may be not too much room for offline RL to improve over it. Simply using BC may achieve similar performance. Note that there are some theory papers trying to answer this question but beyond the scope of this paper. We will add some discussion about that in our revision.\n\n[1] Revisiting Some Common Practices in Cooperative Multi-Agent Reinforcement Learning, ICML 2022\n\n[2] Dealing with Non-Stationarity in MARL via Trust-Region Decomposition, ICLR 2022\n\n[3] Shared Experience Actor-Critic for Multi-Agent Reinforcement Learning, NeurIPS 2020\n\n[4] Towards Understanding Cooperative Multi-Agent Q-Learning with Value Factorization, NeurIPS 2021"}, {"review_id": "xHh9EV03D7", "reviewer": "Reviewer_uuTt", "summary": "The authors provide a principled framework to convert global-level value regularization into equivalent implicit local value regularizations and simultaneously enable in-sample learning, thus elegantly bridging multi-agent value decomposition and policy learning with offline regularizations. The method is evaluated through both theoretical analysis and experiments conducted on the offline multi-agent MuJoCo and StarCraft II micro-management tasks.\n", "questions": "Please answer the questions in the weaknesses.", "limitations": "N/A", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The methods are evaluated through both theoretical analysis and experiments.\n2. The writing is clear and easily comprehensible.", "weaknesses": "1. The paper's main contribution is to decompose the policy and value function based on the factorization of the behavior policies. However, the factorization assumption is not well justified. The author should provide more analysis and some simple examples to illustrate the assumption.\n2. Could the authors analyze whether the local Q-function Q_i adheres to the Individual-Global-Maximum (IGM)?\n3. Do all of the compared methods have access to global information? If not, the fairness of the results may be compromised.\n4. Another point worth considering is that in homogeneous scenarios, where agents exhibit consistent behavior, would there still be issues with simply enforcing local-level regularization?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review.\n\n1.**The factorization of the behavior policies.**  In multi-agent reinforcement learning,  the global policy is usually considered decomposable,  which means $\\pi_{tot}(\\boldsymbol{a} |\\boldsymbol{o})=\\prod_{i=1}^n \\pi_i\\left(a_i | o_i\\right)$, see [1] [2] [3] [4]. The global  behavior policy also belongs to a type of global policy, so it is reasonable to believe that the global behavior policy can be decomposed.\n\n2.**Whether the local Q-function Q_i adheres to the Individual-Global-Maximum.** Our decomposition of Q also satisfies $Q_{t o t}(\\boldsymbol{o}, \\boldsymbol{a})=\\sum_i w_i(\\boldsymbol{o}) Q_i\\left(o_i, a_i\\right)+b(\\boldsymbol{o}), w_i \\geq 0, \\forall i=1 \\cdots, n$ . Due to the nonnegativity of the weight function $w_i$ ,  when each local Q reaches the action that maximizes the local Q-value,  the global Q obtains the maximum joint action.\n\n3.**Do all of the compared methods have access to global information?** Yes, all the baseline methods have access to the global information. The baseline methods  adopt the same Q decomposition structure as OMIGA, and the global information is reflected in the weight $w_i(\\boldsymbol{o})$ , which is calculated based on the joint observation.\n\n4.**Simply enforcing local-level regularization in homogeneous scenarios.**  Even in homogeneous scenarios, the multi-agent system remains a whole. Thus,  global-level regularization is still better than local-level regularization, because it is difficult for the local-level offline regularization to consider the global information that captures coordinated agent behaviors and the credit assignment in the multi-agent system. Besides, for most homogeneous scenarios,  even if the action space of agents is the same, the behavior of agents is still diverse. Simply enforcing local-level regularization cannot guarantee the regularization still remains valid at the global level, and it also potentially leads to over-conservative policy learning.\n\n\n[1] Divergence-Regularized Multi-Agent Actor-Critic,  ICML 2022\n\n[2] Revisiting Some Common Practices in Cooperative Multi-Agent Reinforcement Learning, ICML 2022\n\n[3] More Centralized Training, Still Decentralized Execution: Multi-Agent Conditional Policy Factorization, ICLR 2023\n\n[4] Dealing with Non-Stationarity in MARL via Trust-Region Decomposition, ICLR 2022"}]

Author rebuttal:

Meta-review: The authors present OMIGA, a new offline multi-agent reinforcement learning (MARL) algorithm with implicit global-to-local value regularization. This work combines the main idea/challenges from offline RL of regularization for tackling distribution shift and MARL of using value decomposition for tackling scalability. Reviewers found the paper clear and algorithm well-motivated. Specifically, the motivations and the significance of each step of deriving the OMIGA algorithm were addressed and ablation studies appropriately tackle the main sources of design choices that could occur within OMIGA. There was much discussion among reviewers about optimality of the algorithm due to its closeness to BC, and acknowledgement of issues due to this. However, the derivation remains a good inspiration for devising the final algorithm OMIGA, and to address the gap between Eq. (12) and Eq. (13), the authors have provided explanations for "greedification" within Eq. (13), which a reviewer found intuitive, and this gap does not seem to be significant enough to affect the experimental results.

Due to the strength of the empirical results I vote for acceptance. However, I do believe that Reviewer xgpF's concerns are valid, and a discussion of the limitations of the derivation of the final algorithm necessary for the camera ready.
Decision: Accept (poster)