Title: Offline Multi-Agent Reinforcement Learning with Implicit Global-to-Local Value Regularization
Venue: NeurIPS 2023
Accepted: yes
Keywords: Offline reinforcement learning; multi-agent reinforcement learning; multi-agent cooperation
Abstract: Offline reinforcement learning (RL) has received considerable attention in recent years due to its attractive capability of learning policies from offline datasets without environmental interactions. Despite some success in the single-agent setting, offline multi-agent RL (MARL) remains to be a challenge. The large joint state-action space and the coupled multi-agent behaviors pose extra complexities for offline policy optimization. Most existing offline MARL studies simply apply offline data-related regularizations on individual agents, without fully considering the multi-agent system at the global level. In this work, we present OMIGA, a new offline multi-agent RL algorithm with implicit global-to-local value regularization. OMIGA provides a principled framework to convert global-level value regularization into equivalent implicit local value regularizations and simultaneously enables in-sample learning, thus elegantly bridging multi-agent value decomposition and policy learning with offline regularizations. Based on comprehensive experiments on the offline multi-agent MuJoCo and StarCraft II micro-management tasks, we show that OMIGA achieves superior performance over the state-of-the-art offline MARL methods in almost all tasks.

Offline Multi-Agent Reinforcement Learning with Implicit Global-to-Local Value Regularization

Abstract

Offline reinforcement learning (RL) has received considerable attention in recent years due to its attractive capability of learning policies from offline datasets without environmental interactions. Despite some success in the single-agent setting, offline multi-agent RL (MARL) remains to be a challenge. The large joint state-action space and the coupled multi-agent behaviors pose extra complexities for offline policy optimization. Most existing offline MARL studies simply apply offline data-related regularizations on individual agents, without fully considering the multi-agent system at the global level. In this work, we present OMIGA, a new offline multi-agent RL algorithm with implicit global-to-local value regularization. OMIGA provides a principled framework to convert globallevel value regularization into equivalent implicit local value regularizations and simultaneously enables in-sample learning, thus elegantly bridging multi-agent value decomposition and policy learning with offline regularizations. Based on comprehensive experiments on the offline multi-agent MuJoCo and StarCraft II micro-management tasks, we show that OMIGA achieves superior performance over the state-of-the-art offline MARL methods in almost all tasks. Our code is available at https://github.com/ZhengYinan-AIR/OMIGA .

1 Introduction

Multi-agent reinforcement learning (MARL) is an active research area to tackle many real-world problems that involve multi-agent systems, such as autonomous vehicle coordination[1], network traffic routing [2], and multi-player strategy games [3]. Although MARL has made some impressive progress in recent years, most of its successes are restricted to simulation environments. In most real-world applications, building high-fidelity simulators can be rather costly or even infeasible, and online interaction with the real system during policy learning is also expensive or risky. Inspired by the recently emerged offline RL methods [4, 5, 6, 7, 8, 9, 10], equipping MARL with offline learning capability has become an attractive direction to tackle real-world multi-agent tasks.

Offline RL focuses on learning optimal policies with pre-collected offline datasets with no further environmental interaction. Under the offline setting, evaluating value function on out-of-distribution (OOD) samples can cause extrapolation error accumulation on Q-values, leading to erroneously overestimated values and misguiding policy learning [4, 5]. This issue, which is also called distributional shift [5], is one of the core challenges in offline RL. Current offline RL methods tackle this issue primarily by incorporating various forms of data-related regularizations to restrict policy learning

from deviating too much from the behavioral data. However, extending the same offline regularization technique to the multi-agent setting poses some unique challenges. The joint state-action space grows exponentially with the number of agents, making the global-level regularization hard to compute and may also result in very sparse constraints under the huge joint state-action space, especially when the size and coverage of the offline dataset are limited.

To tackle the challenges, some recent works [11, 12, 13] attempt to design offline MARL algorithms heuristically by enforcing local-level regularizations. By using the Centralized Training with Decentralized Execution (CTDE) framework [14] to decompose the global value function into a combination of local value functions, these works turn the original offline multi-agent RL problem into the offline single-agent RL problem and apply policy or value regularizations at the local level. While straightforward and easy to implement, simply enforcing local-level regularization cannot guarantee the induced regularization at the global-level still remains valid, which can potentially lead to over-conservative policy learning. Existing approaches offer no guarantee whether the optimized local policies are jointly optimal under a given value decomposition scheme. Furthermore, because the offline regularization of most existing methods is completely imposed from the local-level without considering the global information, these methods fail to capture coordinated agent behaviors and credit assignment in the multi-agent system.

In this paper, we aim to give a principled approach to the offline MARL problem. We start by imposing global-level regularizations to form a behavior-regularized MDP. We then find that using some particular regularization choice in this MDP will have closed-form solutions, which naturally turn the global-level regularizations into local-level ones, under some mild assumptions. Finally, we show that the local-level regularization can be derived in an implicit way by using only samples from the offline datasets, without knowing the behavior policy or approximating the regularization term. We dub this new algorithm as offline multi-agent RL algorithm with implicit global-to-local value regularization (OMIGA). In OMIGA, the local-level value regularization is rigorously derived from the global regularization under value decomposition, thus also capturing global information and the impact of multi-agent credit assignment. Under this design, we can also guarantee that the learned local policies are jointly optimal at the global level. Also, OMIGA enables complete in-sample learning without querying OOD action samples, which enjoys better stability during policy learning.

We evaluate our method using various types of offline datasets on both multi-agent MuJoCo [15] and StarCraft Multi-Agent Challenge (SMAC) tasks [16]. Under all settings, OMIGA achieves better performance and enjoys faster convergence compared with other strong baselines.

2 Related Work

Offline reinforcement learning. Due to the absence of online data collection, offline policy learning suffers from severe distributional shift and exploitation error accumulation issues when evaluating the value function on OOD actions [4, 5]. As a result, existing offline RL methods adopt various approaches to learn policies pessimistically and regularize policy learning close to data distribution. Policy constraint methods [4, 5, 9, 17, 18, 19] add explicit or implicit policy constraints to regularize the policy from deviating too much from behavioral data. Value regularization methods [20, 21, 22, 23] learn conservatively estimated value functions on OOD data. Uncertainty-based methods add uncertainty penalization terms on value functions [24, 25] or rewards [26, 27, 28] to enable pessimistic policy learning. Recently, in-sample learning methods [7, 8, 10, 29] provide a new direction to avoid the distributional shift by learning value functions and policies completely within data. These methods eliminate the involvement of policy-generated OOD samples during policy evaluation, thus enjoying superior learning stability. Our method embeds multi-agent value decomposition within the in-sample learning paradigm, which fully exploits the multi-agent problem structure for offline policy optimization.

Multi-agent reinforcement learning. The complexity of multi-agent RL problems typically arises from the huge joint action space [30]. In recent years, the CTDE framework [31, 32] has been proposed to decouple agents' learning and execution phases to handle the exploding action space issue. Under the CTDE framework, agents are trained in a centralized manner with global information and make decisions with learned local policies during execution. Some representative works are the value decomposition methods [33, 34, 35, 36], which decompose the global Q-value function into a set of local Q-value functions for scalable multi-agent policy optimization.

There have been some recent attempts to design MARL algorithms for the offline setting, which can be broadly categorized as RL-based [11, 12, 13, 37] and goal-conditioned supervised learning (GCSL) based methods [38, 39]. Existing RL-based methods are typically built upon the CTDE framework and perform offline policy regularization at the local level. For example, ICQ [11] uses importance sampling to implicitly constrain local policy learning on OOD samples. OMAR [12] adopts CQL [20] to learn local Q-value functions and adds zeroth-order optimization to avoid bad local optima. In these methods, the multi-agent modeling and offline RL ingredients are fragmented, which cannot guarantee that the local-level regularizations still remain valid at the global level, and also ignores the cooperative behavior among agents. Moreover, there is no consideration of the value decomposition's influence on the optimal local policies learned with offline regularizations. On the other hand, GCSL-based methods [38, 39] leverage the sequential data modeling capability of transformer architecture to solve offline MARL tasks. However, recent studies also show that GCSL-based methods can be over-conservative [8] and cannot guarantee optimality under stochastic environments [40, 41].

In this work, our proposed OMIGA is an RL-based method, which uses an implicit global-to-local value regularization scheme to tackle the aforementioned limitations, and offers an elegant solution to marry multi-agent modeling and offline learning.

3 Preliminaries

3.1 Notations

We focus on the fully cooperative multi-agent tasks, which can be modeled as a multi-agent Partially Observable Markov Decision Process (POMDP) [42], defined by a tuple G = ⟨S , A , P , r, Z , O , n, γ ⟩ . s ∈ S denotes the true state of the environment. A is the action set for each of the n agents. At each step, an agent i ∈ { 1 , 2 , ...n } chooses an action a i ∈ A , forming a joint action a = ( a 1 , a 2 , ...a n ) ∈ A n . P ( s ′ | s , a ) : S × A n ×S → [0 , 1] is the transition dynamics to the next state s ′ . γ ∈ [0 , 1) is a discount factor. In the partial observable environment, each agent receives an observation o i ∈ O at each step based on an observation function Z ( s , i ) : S × N →O , and we denote o = ( o 1 , o 2 , ...o n ) . All agents share the same global reward function r ( o , a ) : O × A n → R . In cooperative MARL, all agents aim to learn a set of policies π tot = { π 1 , · · · , π n } that jointly maximize the expected discounted returns E a ∈ π tot , o ∈O [ ∑ ∞ t =0 γ t r ( o t , a t )] . Under the offline setting, a pre-collected dataset D is obtained by sampling with the behavior policy µ tot = { µ 1 , · · · , µ n } and the policy learning is conducted entirely with the data samples in D without any environment interactions.

3.2 CTDE Framework and Value Decomposition

In MARL, the joint action space increases exponentially with the increase in the number of agents. Therefore, it is difficult to directly query an optimal joint action from the global Q-value function Q tot ( o , a ) , and the global Q-value function could be negatively affected by the suboptimality of individual agents. To address these problems, the Centralized Training with Decentralized Execution (CTDE) framework [14, 31, 32] is proposed. In the training phase, agents can access the full environment information and share each other's experiences. In the execution phase, each agent chooses actions only according to its individual observation o i . Through the CTDE framework, optimization at the individual level results in the optimization of the joint action space, which avoids the aforementioned problems. Value decomposition methods [33, 34, 35, 36] are popular solutions under the CTDE framework to achieve the decomposition of the joint action space. Value decomposition is also used in recent offline MARL methods [11, 12], which decomposes the global Q-value function Q tot ( o , a ) into a linear combination of local Q-value functions Q i ( o i , a i ) :

where w i ( o ) and b ( o ) capture the weights and offset on local Q-value functions.

3.3 Behavior-Regularized MDP in Offline RL

To avoid the distributional shift in offline RL, existing approaches typically incorporate various data-related regularizations on rewards, value functions, or the policy optimization objective. In our work, we are specifically interested in the treatment that adds a behavior regularizer to the rewards r ( s, a ) to penalize OOD samples [10]. This framework leads to a special behavior-regularized MDP, which optimizes a policy with the following objective:

where α is a scale parameter, and f ( · , · ) is the function that captures the divergence between π and µ , which is similar to the entropy regularization in SAC [43]. In the above objective, we regularize the deviation between the learned policy π and behavior policy µ , so as to avoid the distributional shift issue. This behavior-regularized MDP corresponds to the following modified policy evaluation operator T f :

where

In the next section, we will derive our proposed method OMIGA based on the behavior-regularized framework, and demonstrate the benefits and desired properties of this framework for the multi-agent setting.

4 Method

In this section, we formally present our implicit global-to-local value regularization approach OMIGA for offline MARL and explain how it can be integrated into effective offline learning. We begin with the multi-agent POMDP with a reverse KL global value regularization added to the rewards. We then show how we can make an equivalent reformulation that naturally converts the global-level regularization into local-level value regularizations. Lastly, we derive a practical algorithm that implicitly enables local-level value regularizations via in-sample learning and provide a thorough discussion of it.

4.1 Multi-Agent POMDP with Global Value Regularization

When extending Eq.(2) to multi-agent POMDP, we can get the following learning objective:

However, unlike the single-agent setting, it is difficult to directly compute the regularization term between the global policy π tot and the global behavior policy µ tot due to the huge state-action space in multi-agent RL. Can we choose a function f to allow certain forms of policy decomposition so as to simplify the calculations?

In this work, we find that choosing the regularization function f to be the reverse KL divergence, which means f ( π tot , µ tot ) = log( π tot /µ tot ) , leads to natural decomposition of the global regularization into the summation of a set of local regularizations, by only assuming the factorization structure of the behavior policy (i.e., µ tot ( a | o ) = ∏ n i =1 µ i ( a i | o i ) ). Note that using reverse KL divergence as the behavior constraint has been widely studied in both online and offline single-agent RL literature [43, 17], and has been shown to produce the nice mode-seeking behavior to avoid OOD actions in the offline setting [18]. Plug in the specific form of f , and we obtain the following global policy evaluation operator:

where

Theorem 4.1. Define T ∗ f the case where the global policy in T π tot f is the optimal policy π ∗ tot , then T ∗ f is a γ -contraction.

The proof is in Appendix A. This theorem indicates global Q-value will converge to the Q-value under the optimal policy π ∗ tot when applying the fixed-point iteration with the policy evaluation operator.

We now aim to analyze and derive the closed-form solution of the optimal value functions Q ∗ tot and V ∗ tot . According to the Karush-Kuhn-Tucker (KKT) conditions where the derivative of a Lagrangian objective function with respect to the global policy is zero at the optimal solution, we have the following proposition:

Proposition 4.2. For a behavior-regularized multi-agent POMDP with f ( π tot , µ tot ) = log( π tot /µ tot ) , the optimal global policy π ∗ tot and its optimal value functions Q ∗ tot and V ∗ tot satisfy the following optimality condition:

where u ( o ) is a normalization term and has a optimal value u ∗ that makes the corresponding optimal policy π ∗ tot satisfy ∑ a ∈A n π ∗ tot ( a | o ) = 1 .

The proof is in Appendix A. Note that Proposition 4.2 can be further simplified. Since V ∗ tot ( o ) = u ∗ ( o ) + α , u ∗ and V ∗ tot can be converted to each other without any approximation. For the global policy, replacing u ∗ ( o ) with V ∗ tot ( o ) -α , we get the following formulation:

Now we have the relationship among the optimal global policy π ∗ tot , behavior policy µ tot , Q-value function Q ∗ tot and state-value function V ∗ tot . Under the multi-agent setting, due to the exponential growth of the joint state-action space with the number of agents, these global values are hard to be evaluated. We now show that we can resort to the value decomposition strategy to further derive a tractable relationship between local policies and value functions.

4.2 Global-to-Local Value and Policy Decomposition

In this work, we introduce the following value decomposition scheme for both the global Q-value function and the state-value function:

Compared with existing offline MARL methods [11, 12, 13] that only decompose the global Q-value function Q tot , we additionally decompose the global state-value function V tot . The decomposition of Q tot and V tot share a common weight function w i ( o ) , since the credit assignment on Q and V should be related. It should also be noted that V tot is free of the joint action space and thus not affected by OOD actions under offline learning.

If we incorporate the value decomposition scheme Eq. (8) into the optimal global policy π ∗ tot in Eq. (7) and utilize the property of the exponential function, we can naturally decompose the optimal global policy π ∗ tot into a combination of optimal local policies π ∗ i .

Proposition 4.3. Under the value decomposition scheme specified in Eq. (8) and assume the global behavior policy is decomposable (i.e., µ tot ( a | o ) = ∏ n i =1 µ i ( a i | o i ) ), we have π ∗ tot ( a | o ) = ∏ n i =1 π ∗ i ( a i | o i ) , where π ∗ i is defined as:

The result immediately follows by observing that:

π ∗ i can be perceived as the optimal local policy under the behavior regularization, since the RHS of Eq. (9) only depends on the optimal local value functions Q ∗ i , V ∗ i , and the local behavior policy µ i of each agent i . This policy decomposition structure has a number of attractive characteristics. First, this design naturally bridges global value regularization and value decomposition. Second, w i ( o ) explicitly appears in the optimal local policy π ∗ i , which is calculated from global observation o . This makes the local policy optimizable with global information and consistent with the credit assignment in the multi-agent system.

4.3 Equivalent Implicit Local Value Regularizations

We have converted the relationship between global policies and values to the relationship between local policies and values. However, it is still unclear how to calculate the optimal local value functions in Eq. (9). Note that each local policy needs to satisfy ∑ a i ∈A π ∗ i ( a i | o i ) = 1 in Eq. (9) in order to ensure it is well-defined, it is also worth noting that when each local policy π ∗ i satisfies this self-normalization constraint, the optimal global policy π ∗ tot = ∏ i π ∗ i will also satisfy ∑ a ∈A n π ∗ tot ( a | o ) = 1 . Thus, we only need to impose self-normalization constraints on local policies. Integrating the RHS of Eq. (9) over the local action space, and have:

We have now established the global-to-local relationships among the optimal values and policies, however, one annoying issue is that the global and local behavior policies µ tot and µ i are typically unknown, and it is hard to estimate them accurately, especially when they are multi-modal. However, perhaps surprisingly, we show that it is possible to learn optimal local value functions in an implicit way without knowing either µ tot or µ i .

Proposition 4.4. V ∗ i ( o ) can be obtained by solving the following convex optimization problem:

The proof follows that the first-order optimality condition of the above optimization objective (i.e., derivative with respect to V i equals 0 ) is exactly the condition Eq. (10).

The above optimization problem provides a new learning objective for the local state-value function V i . It is also worth noting that this objective actually corresponds to adding an equivalent implicit local value regularization on V i . To see this, note that in addition to performing expected regression to let V i fit Q i by the first term, this objective also minimizes the second term w i ( o ) V i ( o i ) /α with respect to V i on behavioral data µ i . This is similar to performing conservative optimal value estimation in existing offline RL literature [20, 27] that learns an underestimated value function to avoid the distributional shift.

4.4 Algorithm Summary

Based on previous analysis, we are now ready to present our final algorithm, OMIGA. OMIGA consists of three supervised learning steps: learning local state-value function V i , learning global and

Algorithm 1 Pseudocode of OMIGA

Require: Offline dataset D . hyperparameter α .

1: Initialize local state-value network V i , local action-value network Q i and its target network ¯ Q i , and policy network π i for agent i =1, 2, ... n .

2: Initialize the weight function network w and b .

3: for t = 1 , · · · , max-value-iteration do

4: Sample batch transitions ( o , a , r, o ′ ) from D

5: Update local state-value function V i ( o i ) for each agent i via Eq. (12).

6: Compute V tot ( o ′ ) , Q tot ( o , a ) via Eq. (8).

7: Update local action-value network Q i ( o i , a i ) , weight function network w ( o ) and b ( o ) with objective Eq. (13).

8: Update local policy network π i for each agent i via Eq. (14).

9: Soft update target network ¯ Q i ( o i , a i ) by Q i ( o i , a i ) for each agent i .

10: end for local Q-value functions, and learning local policies π i . We can use Eq. (11) to learn the optimal local state-value function V ∗ i as:

With the learned local V ∗ i , we can jointly optimize the local Q-value function Q i , the weight and offset function w i and b ( i = 1 , · · · , n ) with the following objective, where Q tot and V tot are computed based on the value decomposition in Eq. (8).

After obtaining the optimal local value functions, we can further learn the local policies π i by minimizing the KL divergence between π i and π ∗ i in Eq. (9) [44], which yields the following learning objective:

Through the above three learning steps, we convert the initial intractable global behavior regularization in multi-agent POMDP to tractable implicit value regularizations at the local level. Moreover, all three steps are learned in a completely in-sample manner, which performs supervised learning only on dataset samples without the involvement of potentially OOD policy-generated actions, thus greatly improving the training stability. We summarize the psuedocode of OMIGA in Algorithm 1.

Discussion with prior works OMIGA draws connections with several prior works such as ICQ [11], DMAC [45] and IVR [10]. ICQ bears some similarities with OMIGA, however, ICQ needs to estimate the normalizing partition function Z ( s ) = ∑ a µ ( a | s ) exp ( Q ( s, a ) /α ) to compute the weight in learning both J π and J Q . Z is estimated by learning an auxiliary behavior model or approximating with softmax operation over a mini-batch, which is hard to compute accurately, especially in the continuous action space. OMIGA uses Eq (11) to provide a new learning objective, it only needs to solve V in a simple and elegant way by imposing self-normalization constraints. The difference between OMIGA and DMAC is that DMAC considers the KL divergence between current policy and previous policy, so that it can compute the KL divergence explicitly. While OMIGA considers the KL divergence between current policy and behavior policy, the behavior policy is typically unknown and hard to estimate, hence OMIGA proposes a principled way to implicitly compute the KL divergence. Compared with IVR which proposes a general implicit regularized RL framework in the offline single-agent setting, OMIGA starts from a similar regularized framework but is designed for the offline multi-agent setting. Also, OMIGA transforms the regularization from global to local in the multi-agent setting, this provides valuable insights to the community, justifies why previous offline multi-agent methods that apply local behavior regularization can work, and also reveals their limitations.

5 Experiment

5.1 Offline Multi-Agent Datasets

We choose multi-agent MuJoCo [15] and StarCraft Multi-Agent Challenge (SMAC) [16] as our experiment environments. Multi-agent MuJoCo is a benchmark for continuous multi-agent robotic control, based on MuJoCo environment. Currently, there is still a lack of offline MARL datasets with continuous action. Thus, we provide an offline multi-agent MuJoCo dataset which is collected from HAPPO [46] algorithm. The dataset contains multiple different data types and the details of the dataset are shown in Appendix B. Another benchmark used in this paper is SMAC, a popular multi-agent cooperative environment for evaluating advanced MARL methods. The offline dataset we used is provided by Meng et al. [38], which is collected from the online trained MAPPO agents [47], and is the largest open offline dataset on SMAC. The offline dataset has three quality levels: good, medium, and poor. Besides, SMAC includes several StarCraft II multi-agent micromanagement maps. We consider 4 representative battle maps: 2 hard maps (5m_vs_6m, 2c_vs_64zg), and 2 super hard maps (6h_vs_8z, corridor).

5.2 Baselines

We compare OMIGA with four recent offline MARL algorithms: the multi-agent version of BCQ [4] and CQL [20] (namely BCQ-MA and CQL-MA), ICQ [11] and OMAR [12]. BCQ-MA and CQL-MA use linear weighted value decomposition structure as Eq.(1) for the multi-agent setting. Details for the implementations of baseline algorithms and the hyperparameters used in OMIGA and baselines are shown in Appendix C.

5.3 Comparative Evaluation

Table 1 shows the mean and standard deviation of average returns for the offline Multi-agent MuJoCo and SMAC tasks. Each algorithm is evaluated by using 32 independent episodes and runs with 5 seeds for training. The results show that OMIGA outperforms all baselines and achieves state-of-the-art performance in most tasks.

For these multi-agent tasks, the credit assignment for multi-agent is complex and it is difficult to learn multi-agent cooperation in the offline setting. The global-to-local value regularization design of

HalfCheetah_medium

6h_vs_8z_medium

HalfCheetah_medium

6h_vs_8z_medium

13

13

4000

3500

12

12

3000

Averagereturn

3000

Averagereturn

11

11

2500

2000

10

2000

10

1000

OMIGA

9

OMIGA

1500

OMIGA(alpha=0.1)

9

OMIGA(alpha=0.5)

OMIGA-local-w

OMIGA-local-w

1000

OMIGA(alpha=1)

OMIGA(alpha=2)

OMIGA-w/o-w

8

OMIGA-w/o-w

OMIGA(alpha=10)

8

OMIGA(alpha=10)

500

0.2

0.4

0.6

0.8

1.0

10

20

30

40

50

0.2

0.4

0.6

0.8

1.0

10

20

30

40

50

Timesteps (million)

Episodes (thousand)

Timesteps (million)

Episodes (thousand)

(a) Analyses and ablations on weight function w i .

(b) Analyses on different

our method reflects the global regularization in the local value function learning implicitly, which leads to better performance than other baseline algorithms. Moreover, the state-value function and Q-value function in OMIGA are completely learned in an in-sample manner without the involvement of the agent policy, which also leads to better training stability and performance. Besides, compared with baseline algorithms, the local policy learning in OMIGA contains global information, and thus enjoys obvious advantages in tasks with continuous action space like multi-agent MuJoCo.

We also conduct experiments on more diverse datasets which are obtained by mixing datasets of various quality. The results are shown in Appendix D. On these mixed datasets, the behavior policies can be suboptimal and more heterogeneous. Therefore, it is difficult for algorithms such as BCQ-MA and OMAR to learn an accurate behavior policy, making implicit value learning with the regularization framework of OMIGA more appealing.

5.4 Analyses on Policy Learning with Global Information

As Eq. (12) and Eq. (14) show, the learning of both local state-value function V i and local policy π i in OMIGA contains global information, which is reflected in w i ( o ) . Therefore, OMIGA can make the credit assignment during value and policy learning and leverage the global information to improve local policy. To verify the effect of this design, we transform OMIGA into two versions without global information, and then compare their performance on offline multi-agent MuJoCo and SMAC datasets. In OMIGA-w/o-w, local policy learning uses formula max π i E ( o i ,a i ) ∼D [exp( 1 α ( Q i ( o i , a i ) -V i ( o i ))) · log π i ( a i | o i )] and the weight function is not available during the policy learning. In OMIGA-local-w, the weight function is w i ( o i ) that only contains local information, and local policy learning uses formula max π i E ( o i ,a i ) ∼D [exp( w i ( o i ) α ( Q i ( o i , a i ) -V i ( o i ))) · log π i ( a i | o i )] .

Figure 1(a) shows the experimental results on offline HalfCheetah datasets and 6h_vs_8z datasets. For these difficult multi-agent tasks, the cooperative relationships among agents are extremely complex, so global information is important to guide local policy learning. OMIGA-w/o-w and OMIGAlocal-w lack global information, and policies are only learned at the local level, which causes worse performance and stability.

5.5 Analyses on the Regularization Hyperparameter

In OMIGA, hyperparameter α is used to control the degree of regularization. The higher α encourages the policy to stay closer to the behavioral distribution, and the lower α makes the policy more aggressive and optimistic. Figure 1(b) shows the experiments about α on the offline MuJoCo HalfCheetah task and SMAC 6h_vs_8z task. In the HalfCheetah task, too high and too low α will lead to performance degradation. An appropriate α can ensure that the policy is restricted near the dataset distribution and also does not lose performance due to being too conservative. In the 6h_vs_8z task, the dataset contains enough good transitions. Therefore, the degree of regularization should be higher to make the algorithm more conservative. Experimental results indicate that a higher α brings better performance on such datasets. When α is very small, the performance of the algorithm becomes very poor due to the lack of value regularization.

α

.

6 Conclusion

In this paper, we study the key challenge of the offline MARL problem. We start from the usage of global behavior constraints and smartly derive a global-to-local value regularization scheme under value decomposition. By doing so, we naturally get a new offline MARL algorithm, OMIGA, that in principle applies global-level regularization but actually imposes equivalent implicit local-level regularization. Our work reveals why the local-level regularization used by existing algorithms works and gives theoretical insights into their weaknesses and how to develop better algorithms. The global-to-local regularization design of OMIGA can capture global information and the impact of multi-agent credit assignment under the offline setting, which guarantees that the learned local policies are jointly optimal at the global level. One future work is to develop other offline MARL algorithms by using other choices of f functions.

Acknowledgments

This work is supported by National Key Research and Development Program of China under Grant (2022YFB2502904) and funding from Global Data Solutions Limited.

References

Yongcan Cao, Wenwu Yu, Wei Ren, and Guanrong Chen. An overview of recent progress in the study of distributed multi-agent coordination. IEEE Transactions on Industrial Informatics , 9(1):427-438, 2013.

Dayong Ye, Minjie Zhang, and Yun Yang. A multi-agent framework for packet routing in wireless sensor networks. Sensors , 15(5):10026-10047, 2015.

Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemysław Debiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, et al. Dota 2 with large scale deep reinforcement learning. arXiv preprint arXiv:1912.06680 , 2019.

Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International conference on machine learning , pages 2052-2062. PMLR, 2019.

Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing offpolicy q-learning via bootstrapping error reduction. Advances in Neural Information Processing Systems , 32, 2019.

Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems , 33:11791191, 2020.

Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. International Conference on Learning Representations , 2021.

Haoran Xu, Jiang Li, Jianxiong Li, and Xianyuan Zhan. A policy-guided imitation approach for offline reinforcement learning. In Advances in Neural Information Processing Systems , 2022.

Jianxiong Li, Xianyuan Zhan, Haoran Xu, Xiangyu Zhu, Jingjing Liu, and Ya-Qin Zhang. When data geometry meets deep function: Generalizing offline reinforcement learning. In The Eleventh International Conference on Learning Representations , 2023.

Haoran Xu, Li Jiang, Jianxiong Li, Zhuoran Yang, Zhaoran Wang, and Xianyuan Zhan. Offline rl with no ood actions: In-sample learning via implicit value regularization. In International Conference on Learning Representations , 2023.

Yiqin Yang, Xiaoteng Ma, Li Chenghao, Zewu Zheng, Qiyuan Zhang, Gao Huang, Jun Yang, and Qianchuan Zhao. Believe what you see: Implicit constraint approach for offline multi-agent reinforcement learning. Advances in Neural Information Processing Systems , 34:10299-10312, 2021.

Ling Pan, Longbo Huang, Tengyu Ma, and Huazhe Xu. Plan better amid conservatism: Offline multi-agent reinforcement learning with actor rectification. In International Conference on Machine Learning , pages 17221-17237. PMLR, 2022.

Jiechuan Jiang and Zongqing Lu. Offline decentralized multi-agent reinforcement learning. arXiv preprint arXiv:2108.01832 , 2021.

Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, OpenAI Pieter Abbeel, and Igor Mordatch. Multiagent actor-critic for mixed cooperative-competitive environments. In Advances in Neural Information Processing Systems , volume 30, pages 6379-6390, 2017.

Christian Schroeder de Witt, Bei Peng, Pierre-Alexandre Kamienny, Philip Torr, Wendelin Böhmer, and Shimon Whiteson. Deep multi-agent reinforcement learning for decentralized continuous cooperative control. arXiv preprint arXiv:2003.06709 , 19, 2020.

Mikayel Samvelyan, Tabish Rashid, Christian Schroeder de Witt, Gregory Farquhar, Nantas Nardelli, Tim G. J. Rudner, Chia-Man Hung, Philip H. S. Torr, Jakob Foerster, and Shimon Whiteson. The starcraft multi-agent challenge. In Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems , pages 2186-2188, 2019.

Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint arXiv:1911.11361 , 2019.

Haoran Xu, Xianyuan Zhan, Jianxiong Li, and Honglei Yin. Offline reinforcement learning with soft behavior regularization. arXiv preprint arXiv:2110.07395 , 2021.

Peng Cheng, Xianyuan Zhan, Zhihao Wu, Wenjia Zhang, Shoucheng Song, Han Wang, Youfang Lin, and Li Jiang. Look beneath the surface: Exploiting fundamental symmetry for sampleefficient offline rl. Advances in Neural Information Processing Systems , 2023.

Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems , 33:11791191, 2020.

Ilya Kostrikov, Rob Fergus, Jonathan Tompson, and Ofir Nachum. Offline reinforcement learning with fisher divergence critic regularization. In International Conference on Machine Learning , pages 5774-5783. PMLR, 2021.

Haoran Xu, Xianyuan Zhan, and Xiangyu Zhu. Constraints penalized q-learning for safe offline reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pages 8753-8760, 2022.

Haoyi Niu, Yiwen Qiu, Ming Li, Guyue Zhou, Jianming Hu, and Xianyuan Zhan. When to trust your simulator: Dynamics-aware hybrid offline-and-online reinforcement learning. Advances in Neural Information Processing Systems , 35:36599-36612, 2022.

Yue Wu, Shuangfei Zhai, Nitish Srivastava, Joshua Susskind, Jian Zhang, Ruslan Salakhutdinov, and Hanlin Goh. Uncertainty weighted actor-critic for offline reinforcement learning. arXiv preprint arXiv:2105.08140 , 2021.

Chenjia Bai, Lingxiao Wang, Zhuoran Yang, Zhi-Hong Deng, Animesh Garg, Peng Liu, and Zhaoran Wang. Pessimistic bootstrapping for uncertainty-driven offline reinforcement learning. In International Conference on Learning Representations , 2021.

Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. Advances in Neural Information Processing Systems , 33:14129-14142, 2020.

Xianyuan Zhan, Haoran Xu, Yue Zhang, Xiangyu Zhu, Honglei Yin, and Yu Zheng. Deepthermal: Combustion optimization for thermal power generating units using offline reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence , 2022.

Xianyuan Zhan, Xiangyu Zhu, and Haoran Xu. Model-based offline planning with trajectory pruning. In Proceedings of the 31st International Joint Conference on Artificial Intelligence, IJCAI 2022 , 2022.

David Brandfonbrener, Will Whitney, Rajesh Ranganath, and Joan Bruna. Offline rl without off-policy evaluation. Advances in Neural Information Processing Systems , 34:4933-4946, 2021.

Pablo Hernandez-Leal, Bilal Kartal, and Matthew E. Taylor. A survey and critique of multiagent deep reinforcement learning. Autonomous Agents and Multi-Agent Systems , 33(6):750-797, 2019.

Frans A. Oliehoek, Matthijs T. J. Spaan, and Nikos Vlassis. Optimal and approximate q-value functions for decentralized pomdps. Journal of Artificial Intelligence Research , 32(1):289-353, 2008.

Landon Kraemer and Bikramjit Banerjee. Multi-agent reinforcement learning as a rehearsal for decentralized planning. Neurocomputing , 190:82-94, 2016.

Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojciech Marian Czarnecki, Vinícius Flores Zambaldi, Max Jaderberg, Marc Lanctot, Nicolas Sonnerat, Joel Z. Leibo, Karl Tuyls, and Thore Graepel. Value-decomposition networks for cooperative multi-agent learning. arXiv preprint arXiv:1706.05296 , 2017.

Tabish Rashid, Mikayel Samvelyan, Christian Schroeder, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning. In International Conference on Machine Learning , pages 4292-4301, 2018.

Kyunghwan Son, Daewoo Kim, Wan Ju Kang, David Earl Hostallero, and Yung Yi. Qtran: Learning to factorize with transformation for cooperative multi-agent reinforcement learning. In International Conference on Machine Learning , pages 5887-5896, 2019.

Jianhao Wang, Zhizhou Ren, Terry Liu, Yang Yu, and Chongjie Zhang. Qplex: Duplex dueling multi-agent q-learning. arXiv preprint arXiv:2008.01062 , 2020.

Xiangsen Wang and Xianyuan Zhan. Offline multi-agent reinforcement learning with coupled value factorization. In Proceedings of the 2023 International Conference on Autonomous Agents and Multiagent Systems , pages 2781-2783, 2023.

Linghui Meng, Muning Wen, Yaodong Yang, Chenyang Le, Xiyun Li, Weinan Zhang, Ying Wen, Haifeng Zhang, Jun Wang, and Bo Xu. Offline pre-trained multi-agent decision transformer: One big sequence model conquers all starcraftii tasks. arXiv preprint arXiv:2112.02845 , 2021.

Wei-Cheng Tseng, Tsun-Hsuan Wang, Yen-Chen Lin, and Phillip Isola. Offline multi-agent reinforcement learning with knowledge distillation. In Advances in Neural Information Processing Systems , 2022.

David Brandfonbrener, Alberto Bietti, Jacob Buckman, Romain Laroche, and Joan Bruna. When does return-conditioned supervised learning work for offline reinforcement learning? In Advances in Neural Information Processing Systems , 2022.

Benjamin Eysenbach, Soumith Udatha, Sergey Levine, and Ruslan Salakhutdinov. Imitating past successes can be very suboptimal. In Advances in Neural Information Processing Systems , 2022.

Craig Boutilier. Planning, learning and coordination in multiagent decision processes. In TARK , volume 96, pages 195-210. Citeseer, 1996.

Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Offpolicy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning , pages 1861-1870. PMLR, 2018.

Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weighted regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv:1910.00177 , 2019.

Kefan Su and Zongqing Lu. Divergence-regularized multi-agent actor-critic. In International Conference on Machine Learning , pages 20580-20603. PMLR, 2022.

Jakub Grudzien Kuba, Ruiqing Chen, Muning Wen, Ying Wen, Fanglei Sun, Jun Wang, and Yaodong Yang. Trust region policy optimisation in multi-agent reinforcement learning. arXiv preprint arXiv:2109.11251 , 2021.

Chao Yu, Akash Velu, Eugene Vinitsky, Yu Wang, Alexandre M. Bayen, and Yi Wu. The surprising effectiveness of mappo in cooperative, multi-agent games. arXiv: Learning , 2021.

A Proofs

A.1 Proof of Theorem 4.1

Proof. For any two global state-value functions V 1 tot and V 2 tot , let π 1 tot be the optimal global policy under T ∗ f V 1 tot , and we can get:

Therefore, it follows that:

A.2 Proof of Proposition 4.2

Proof. For a behavior-regularized multi-agent POMDP with the behavior regularizer function f ( π tot , µ tot ) = log( π tot /µ tot ) , its learning objective can be expressed as max π tot E [ ∑ ∞ t =0 γ t ( r ( o t , a t ) -αlog ( π tot ( a t | o t ) /µ tot ( a t | o t )))] . Its Lagrangian function can obtain when the optimal global policy is written as follows:

where d π tot is the stationary joint observation distribution of the global policy π tot . u and β are Lagrangian multipliers for the equality and inequality constraints.

According to the Karush-Kuhn-Tucker (KKT) conditions where the derivative of the Lagrangian objective function with respect to the global policy is zero at the optimal solution, it follows that:

From Eq. (15), we can further solve the optimal global policy as:

The above formula can be further simplified. β is the Lagrangian multiplier, and meets complementary slackness β ( a | o ) π tot ( a | o ) = 0 . Considering the joint observation o is fixed, exp ( Q tot ( o , a ) -u ( o )+ β ( a | o ) α -1 ) is always larger than 0 . Therefore, for any positive probability action, its corresponding Lagrangian multiplier β ( a | o ) is 0 . Therefore, π tot ( a | o ) can be reformulated as:

Bringing Eq. (16) into ∑ a π tot ( a | o ) = 1 , we have:

The left side of Eq. (17) can be seen as a continuous and monotonic function of u , so it has only one solution denoted as u ∗ , and we denote the corresponding policy π tot as π ∗ tot .

Integrating Eq. (16) into the expression of optimal global state value, we can get:

To summarize, we obtain the optimality condition of the behavior regularized MDP with Reverse KL divergence as follows:

where u ( o ) is a normalization term and has a optimal value u ∗ that makes the corresponding optimal policy π ∗ tot satisfy ∑ a ∈A n π ∗ tot ( a | o ) = 1 .

B Experiment Settings

B.1 Multi-Agent MuJoCo

Multi-agent Mujoco [15] is a benchmark framework developed for assessing and comparing the effectiveness of algorithms in continuous multi-agent robotic control. Within this framework, a robotic system is partitioned into independent agents, each tasked with controlling a specific set of joints. The agents collaborate harmoniously to accomplish shared objectives, such as acquiring the ability to walk through an environment, with the ultimate goal of maximizing the cumulative reward. Multi-agent MuJoCo environment consists of multiple different robot configurations, and it is often used for the study of novel MARL algorithms for decentralized coordination in isolation.

To generate the dataset transitions, we captured the interactions between the environment and trained online MARL algorithms. Specifically, we use HAPPO [46] algorithm to collect data. The expert dataset is generated by employing the converged HAPPO algorithm. This involves training the algorithm until it reaches a state of convergence, where the agents have learned optimal policies. The medium dataset is generated by first training a policy online using HAPPO, early-stopping the training, and collecting samples from this partially-trained policy. The medium-replay dataset consists of recording all samples in the replay buffer observed during training until the policy reaches the medium level of performance. The medium-expert dataset is constructed by mixing equal amounts of expert demonstrations and suboptimal data. For all datasets, the hyperparameter env_args.agent_obsk (determines up to which connection distance agents will be able to form observations) is set to 1 . The average returns of our datasets are listed in Table 2.

B.2 The StarCraft Multi-Agent Challenge

The StarCraft Multi-Agent Challenge (SMAC) benchmark is chosen as our testing environment. Due to its high control complexity, SMAC is a popular multi-agent cooperative control environment for evaluating advanced MARL methods. It consists of a collection of StarCraft II micro-management

tasks in which two groups of units engage in combat. Agents based on the MARL algorithm control the first group's units, while a built-in heuristic game AI bot with different difficulties controls the second group's units. Scenarios vary in terms of the initial location, number and type of units, and elevated or impassable terrain. The available actions for each agent include no operation, move[direction], attack [enemy id], and stop. The reward that each agent receives is the same. The hit-point damage dealt and received determines the agents' share of the reward. SMAC consists of several StarCraft II multi-agent micromanagement maps. We consider 4 representative battle maps, including 2 hard maps (5m_vs_6m, 2c_vs_64zg), and 2 super hard maps (6h_vs_8z, corridor), as our experiment tasks. The task type and other details of the maps are listed in Table 3.

The offline SMAC dataset used in this study is provided by [38], which is the largest open offline dataset on SMAC. Different from single-agent offline datasets, it considers the property of multi-agent POMDP, which owns local observations and available actions for each agent. The dataset is collected from the trained MAPPO agent and includes three quality levels: good, medium, and poor. For each original large dataset, we randomly sample 1000 episodes as our dataset. The average returns of SMAC datasets are listed in Table 4.

C Implementation Details

C.1 Details of OMIGA

The local Q-value, state-value networks and policy networks of OMIGA are represented by 3-layer ReLU activated MLPs with 256 units for each hidden layer. For the weight network, we use 2-layer ReLU-activated MLPs with 64 units for each hidden layer. All the networks are optimized by Adam optimizer.

C.2 Details of baselines

We compare OMIGA against four recent offline MARL algorithms: ICQ [11], OMAR [12], BCQMA, and CQL-MA. For ICQ and OMAR, we implement them based on the algorithm described in their papers. BCQ-MA is the multi-agent version of BCQ [4], and CQL-MA is the multi-agent version of CQL [20]. BCQ-MA and CQL-MA use the linear weighted value decomposition structure as Q tot = ∑ n i =1 w i ( o ) Q i ( o i , a i ) + b ( o ) , w i ≥ 0 for the multi-agent setting. The policy constraint of BCQ-MA and the value regularization of CQL-MA are both imposed on the local Q-value.

In this paper, all experiments are implemented with Pytorch and executed on NVIDIA V100 GPUs.

C.3 Hyperparameters

For multi-agent MuJoCo, the hyperparameters of OMIGA and baselines are listed in Table 5. An important hyperparameter of OMIGA is the regularization hyperparameter α . The higher α encourages OMIGA to stay near the behavioral distribution, and lower α makes OMIGA more optimistic. On most tasks, we use α = 10 to ensure a good regularization effect. On the mediumquality dataset of the HalfCheetah task, we choose α = 1 .

For SMAC, the hyperparameters of OMIGA and baselines are listed in Table 6. On most tasks, we use α = 10 . On the poor dataset of the 6h_vs_8z map, the quality of the dataset is relatively poor. It does not make much sense to make the policy close to the behavioral policy, so we choose α = 2 to make the algorithm more radical.

On most SMAC maps, the learning rate of all networks is set to 5e-4. The exception is the map 2c_vs_64zg, on this map, the learning rate of all networks is set to 1e-4.

D Additional Results

D.1 Results on mixed datasets

We also evaluated the performance of OMIGA on the multi-modal mixed datasets. Unlike BCQ-MA and OMAR, OMIGA doesn't need to learn a behavior policy. We choose two original datasets on the SMAC super hard maps 6h_vs_8z and corridor and construct mixed datasets by combining these SMAC datasets of different quality, including good-poor, good-medium, and medium-poor datasets. Each mixed dataset is blended by 50% of each of the two original datasets. On these mixed suboptimal datasets, the behavior policy is heterogeneous. Therefore, it is more difficult for algorithms such as BCQ-MA and OMAR to learn an accurate behavior policy, making implicit value learning with the regularization framework of OMIGA more appealing.

Table 7 shows the results that OMIGA consistently outperforms other offline MARL baselines under all different mixed dataset experiments. Compared with the results on the original datasets, the performance of OMIGA has become more leading, indicating the benefits of implicit value regularization of OMIGA.

Peer reviews:
comments: SUMMARY:
The authors provide a principled framework to convert global-level value regularization into equivalent implicit local value regularizations and simultaneously enable in-sample learning, thus elegantly bridging multi-agent value decomposition and policy learning with offline regularizations. The method is evaluated through both theoretical analysis and experiments conducted on the offline multi-agent MuJoCo and StarCraft II micro-management tasks.

STRENGTHS:
1. The methods are evaluated through both theoretical analysis and experiments.
2. The writing is clear and easily comprehensible.

WEAKNESSES:
1. The paper's main contribution is to decompose the policy and value function based on the factorization of the behavior policies. However, the factorization assumption is not well justified. The author should provide more analysis and some simple examples to illustrate the assumption.
2. Could the authors analyze whether the local Q-function Q_i adheres to the Individual-Global-Maximum (IGM)?
3. Do all of the compared methods have access to global information? If not, the fairness of the results may be compromised.
4. Another point worth considering is that in homogeneous scenarios, where agents exhibit consistent behavior, would there still be issues with simply enforcing local-level regularization?

QUESTIONS:
Please answer the questions in the weaknesses.

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-06-15

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper investigates multi-agent offline reinforcement learning, and address some of the problems that arise with applying regularization at either the global or local level. To achieve this, a framework that allows to apply global regularization via an equivalent set of local regularizations is proposed. This way, the CTDE paradigm and value-decomposition scheme can be applied, and decentralized policy are easily learned, while at the same time retaining the positive effect and guarantees that global regularization has.\n\nSTRENGTHS:\nThe paper investigates an interesting area: offline reinforcement learning can be a viable way to learn in those real-world settings where actual environment interaction is not possible or too expensive. The application of the CTDE paradigm to effectively translate the global terms into local terms that are easier to learn and use is well aligned with the current research track on this line. The resulting algorithm is easy to follow and implement, and its performances are shown to be superior to other exiting offline RL algorithms with more complex regularization terms. Also, the paper is nicely written and clear to follow, and the empirical results are quite convincing.\n\nWEAKNESSES:\nHowever, the paper in my opinion is a bit sloppy when it comes to a proper characterization of the used setting, and the actual working mechanism of the various components (policies, value-functions...) that are used. A precise mathematical formulation is a very useful point for a paper, as it allows the reader to relate the present work to the previous literature. Please see my questions below for a more precise explanation.\n\nQUESTIONS:\n- In the usual Dec-POMDP framework, the global Q-function does not condition on the joint observation $\\mathbf{o}$, but rather on the joint action-observation history $\\tau$ (and/or on the true environment state $s$). This is because the joint observation $\\mathbf{o}$ is not a sufficient statistics for the agents to learn an optimal policy $\\pi_{tot}^*$ (except for some particular subclasses of Dec-POMDP, like in Dec-MDP, where the joint observation $\\mathbf{o}$ forms the true underlying state $s$ itself, and thus reactive policies, i.e. policies conditioning on local observations $o_i$ rather than on local histories $\\tau_i$, can still achieve the optimal solution). Is your framework specifically for this very special kind of settings, or it also works on the more general, history-based, setting? Is it is the latter, please correct the notation to align it with the existing literature, otherwise please argument such limitation.\n- From Section 4.1 onward, you start writing the reward as a function of the joint observation $\\mathbf{o}_i$ rather than the true state $s_t$. This is not true in general (unless in a Dec-MDP, see my point above), and you should adjust this to reflect the actual environment you have introduced earlier on.\n- Where does the bias term $b(\\mathbf{o})$ goes in Equation (10) and the following?\n- Equation (11) is not immediate: please argument a bit more on why this is actually the case.\n- One thing that immediately comes to mind in this setting is that, in order to collect a properly various and useful dataset, a very good behavioural policy is generally needed. Indeed, in the Experiments section, you say that you use a trained HAPPO policy to generate the MA-MuJoCo dataset (similarly, the SMAC dataset comes from a trained MAPPO policy). While in actual supervised learning the dataset comes from whatever source of information, and has nothing to do with the final classifier that we aim to learn, here in order to learn a policy we need another (behavioural) policy to generate these data, and the policy itself has to be good enough in order to generate a dataset with also successful and good transitions that can improve the learned policy. I would like to see a bit of discussion with respect to this point: how good has the behavioural policy to be? How much of an improvement can we gain with offline learning over that (if there is no improvement, there would be no point in learning a new policy over the behavioural one that generated the dataset)? How difficult is this kind of approach in relation to generate the required datasets? This kind of questions are unique to the offline RL setting, and does not have a counterpart in standard supervised learning, and thus I think there deserves some discussion and clarification to properly assess the merits and eventual limitations of this family of algorithms. I appreciate that section 5.5 tries to go in that direction a bit, by showing how close or far the learned policy wants to be w.r.t. the behavioural policy. This is a nice ablation study, and worth a bit more investigation in my opinion.\n\nTypos:\n- Line 54: perhaps you mean dub (rather than dumb)?\n- Joint observations $\\mathbf{o}$ are never introduced explicitly, although used in the work.\n- Line 113: in the subscript of the expectation, shouldn't that be $\\pi_{tot}$?\n- Line 129: the weight is $w_i(\\mathbf{o})$ alone, not the entire $w_i(\\mathbf{o})Q_i(o_i,a_i)$.\n- Line 177: value functions\n- Line 206: optimizable (rather than \"can be optimized\")\n- Line 212: $\\pi_i^*$\n- Line 229: learns\n- Please expand a bit the caption of Figure 1 by describing a bit more in details what it represents.\n\nLIMITATIONS:\nNo clear statement of possible limitations is provided in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-15"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on offline multi-agent reinforcement learning in the centralized training decentralized execution setting and proposes a SARSA-like approach that leverages value decomposition. Drawing from entropy-regularized and maxent RL, the authors propose a Behavior-Regularized formulation for Offline RL. The method is tested on offline MAMuJoCo datasets generated by the authors as well as SMAC datasets. The baselines are two single agent methods (BCQ and CQL) that the authors extend to the multi-agent setting and two multi-agent offline methods (ICQ and OMAR). Authors also investigate two ablations of their method.\n\nSTRENGTHS:\n- Offline multi-agent reinforcement learning is of significant interest. \n- Behavior-Regularized formulation of Offline RL is interesting. \n- The paper is overall easy to read and understand.\n\nWEAKNESSES:\nI believe there are major flaws with the method and the experimental procedure. I think that the method is quite similar to Behavioural Cloning (BC) and is unlikely to improve much on it or on the datasets' performance. This failed to be evidenced by the experimental results because 1) the method is not compared against BC and 2) the datasets' average and median scores are not reported. Additionally, I think that BCQ and CQL have been poorly extended into multi-agent baselines and that how results are presented in table 1 misleads the reader into believing that the presented method significantly outperforms the baselines. I detailed this point below\n \n\n- The method does not improve on BC: Eq. (15) would yield pi^star, but only if Q and V are Q^star and V^star. Which is not the case since they are learned from (14) which simply learns the Q value of the behaviour policy on the dataset (there is no greedification happening here) and V is just learned to normalize the policy to one. Therefore, the only greedification step is (15) (i.e. pi is a Boltzmann policy w.r.t. to the behaviour policy values). Yet, since the learned policy does not influence any of the value learning objectives, this does not greedify the value functions. Therefore, I believe that the presented method is very close to just doing BC. For comparison, consider the greedification steps of two other SARSA-like offline RL approaches, IQL [1] and ICQ: IQL explicitly greedifies the learned values from expectile-learning before doing AWR (AWR being equivalent to (15)).  In ICQ, the greedified policy greedifies the value learning through the importance sampling correction. \n\n    Therefore, I doubt that the presented method improves on the simple BC approach. Comparison to BC and reporting the datasets' performance is essential in order to show that the method can improve on the datasets and BC baseline. Regarding that last point, by quickly looking at the results in [2], it seems that OMIGA is not really able to improve on BC or match the average dataset performance on SMAC.  \n\n    I would also recommend comparing with a multi-agent version of IQL and independent-TD3+BC that explicitly regularizes the offline policy with BC (see for instance baselines description in [3]).  \n\n- Table 1 is misleading: bold values should denote values that are above others with statistical significance. Yet, in Table 1. Most bold values' confidence interval overlap with the ones of other values (at least for SMAC). This is quite misleading as it suggest that one method outperforms the others, while actually most methods are equivalent statistically speaking.\n\n- BCQ and CQL have been poorly extended into multi-agent baselines: the multi-agent extensions use centralized values (with value decomposition) but decentralized regularization, which seems to be specifically what authors have shown not to work in their ablation of OMIGA. I say \"seem\" because it is unclear from section 5.4 whether or not values are learned with a centralized value and value decomposition.\n    I believe that MA-BCQ and MA-CQL should be either fully decentralized ([4] shows that fully decentralized tends to be better for offline MARL) or with centralized regularization (like OMIGA). \n\n\nEDIT: I have raised my score to 5 after the discussion period\n\nQUESTIONS:\n- the very poor performance of OMAR on Mujoco in table 1. is surprising as OMAR is reported to outperform MA-ICQ and MA-CQL in multi-agent continuous control [4].\n- how were the hyperparameters tuned and selected for OMIGA and the baselines on each task? \n- l.192 have authors investigated using different weights for V and Q?\n- l.51 \"some particular choice\" of what? \n\n### Summary of review\nDespite offline MARL being a challenge of significant importance, and the behavior-regularized formulation proposed here being interesting, I believe that the presented method and its evaluations have too many flaws to allow for publication. \n\n### references\n[1] Kostrikov, Ilya, Ashvin Nair, and Sergey Levine. \"Offline Reinforcement Learning with Implicit Q-Learning.\" International Conference on Learning Representations (2022).\n\n[2] Meng, Linghui, et al. \"Offline pre-trained multi-agent decision transformer: One big sequence model conquers all starcraftii tasks.\" arXiv preprint arXiv:2112.02845 (2021).\n\n[3] Barde, Paul, et al. \"A Model-Based Solution to the Offline Multi-Agent Reinforcement Learning Coordination Problem.\" arXiv preprint arXiv:2305.17198 (2023).\n \n[4] Pan, Ling, et al. \"Plan better amid conservatism: Offline multi-agent reinforcement learning with actor rectification.\" International Conference on Machine Learning. PMLR, 2022.\n\nLIMITATIONS:\nn/a", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 1, "DATE": "2023-06-20"}

comments: SUMMARY:
The paper proposed a novel offline MARL algorithm with named OMIGA. OMIGA transfers the global entropy regularization into local regularizations implicitly under the linear weighted value decomposition assumption. The empirical results on multi-agent MuJoCo and SMAC suggest that OMIGA almost outperforms current offline multi-agent SOTA baselines.

STRENGTHS:
Originality: OMIGA provides a novel principled framework to decompose global regularization into equivalent implicit local regularizations for offline MARL.
Quality: The theoretical analyze supports the conversion of the global regulation and the empirical results shows that OMIGA outperforms current SOTA baselines under specific hyper-parameters.
Clarity: The paper is clearly written and well organized.
Significance: OMIGA almost outperforms current offline multi-agent SOTA baselines in the experiments.

WEAKNESSES:
The hyper-parameters seem unfair. For example, in the ICQ paper, the performance of ICQ is significantly improved when the Lagrangian coefficient changes from 10 to 100 for SMAC tasks, while this paper chooses 10 for ICQ.
Mirror: The variables $a_t$, $s_t$ in the RHS of Eq. (4) should be $a$, $s$ (below line 141).

QUESTIONS:
1. Could you explain why you choose the hyper-parameter for the baselines as in the paper? The hyper-parameters seem different in their origin papers.
2. Do you try the dataset collected from other MARL algorithms (other than HAPPO, MAPPO)? What is the performance of OMIGA?

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes OMIGA, a new offline multi-agent reinforcement learning (MARL) algorithm with implicit global-to-local value regularization. The policy regularization from offline reinforcement learning (RL), for avoiding distribution shifts between learned policy and behavioral policy, is first brought to centralized training in MARL. Then, the global value functions and policies are decomposed into local agent-wise counterparts. Finally, using the self-normalization constraint of local policies, the objective functions for learning local value functions are derived, which leads to the OMIGA. The proposed method shows good performance compared to relevant existing works in offline single-agent and multi-agent RL, on MuJoCo and SMAC benchmark tasks.\n\nSTRENGTHS:\n- This work seamlessly combined the main idea/challenges from offline RL (i.e. regularization for tackling distribution shift) and MARL (i.e. value decomposition for tackling scalability). Particularly, I deem incorporating global information into local value/policy learning as a crucial matter in the field of MARL in general. In this context, the stance of this paper is well-positioned.\n\n- The motivations and the significance of each step of deriving the OMIGA algorithm were addressed well.\n\n- Ablation studies appropriately tackle the main sources of design choices that could occur within OMIGA: the incorporation of global observation $\\boldsymbol{o}$ and the regularization parameter $\\alpha$.\n\nWEAKNESSES:\nWhile it was difficult to find a critical weakness of the proposed algorithm within the scope of this paper, I have numerous questions regarding the derivation of the OMIGA algorithm as well as the presented results. These questions are listed in the \"Questions\" section.\n\nMeanwhile, I outline below some of the confusing typos or slight inconsistencies that might confuse the readers:\n\n- Notation mismatch between the joint policy $\\pi_{tot}$ (L112) and $\\boldsymbol\\pi$ (L113)\n\n- L136: It seems that the “derivation of $\\pi$ and $\\mu$” should actually be the “deviation between $\\pi$ and $\\mu$”.\n\n- The operator $\\mathcal T_f^*$ has been defined for the action-value function $Q$ in Eq. (5) but is applied on state-value functions $V_{tot}$’s throughout Appendix Section A.1 and in the equation under L492 (Appendix). While its definition is obvious in Section A.1, it would be appreciated if you could explicitly provide the definition.\n\n- The $\\beta(a|s)$ in L487 (Appendix A.2) should be $\\beta(\\boldsymbol{a}|\\boldsymbol{o})$.\n\nQUESTIONS:\nI have listed below my questions regarding my confusion regarding the derivations or missing details in the experiments.\n\n- Could you explain the derivation of Eq. (5)? Substituting $f(\\pi_{tot}, \\mu_{tot}) = \\log (\\pi_{tot} / \\mu_{tot})$ into the equation under L153 yields \n  - $r({\\boldsymbol{o}}_0, {\\boldsymbol{a}}_0) - \\alpha \\log(\\pi_{tot}({\\boldsymbol{a}}_0 | {\\boldsymbol{o}}_0) / \\mu_{tot}({\\boldsymbol{a}}_0 | {\\boldsymbol{o}}_0)) + \\sum_{t=1}^{\\infty} (\\cdots)$, \n  - but I am not sure if Eq.(5) would cover the $-\\alpha \\log(\\pi_{tot}({\\boldsymbol{a}}_0 | {\\boldsymbol{o}}_0) / \\mu_{tot}({\\boldsymbol{a}}_0 | {\\boldsymbol{o}}_0))$ term.\n\n- In the derivation of Eq. (10), it seems that the $b(\\boldsymbol{o})$ term from Eq. (9) is omitted for simplicity. Will this leave the rest of the exposition unchanged, particularly during the derivation of Eq. (11) and (12)?\n\n- In L222-223, it is mentioned that \"The proof [of Proposition 4.4] follows [from the fact] that the first-order optimality condition of the above optimization objective (i.e., derivative with respect to $V_i$ equals 0) is exactly the condition Eq. (11).\" Yet I could not comprehend how the solution to Eq. (11) is unique. In other words, it would be appreciated if the authors could explain how satisfying Eq. (11) will necessarily mean that the solution of Eq. (12) is $V_i^*$.\n\n- In the “Discussion with prior works” paragraph (L247-265), OMIGA is compared to ICQ, DMAC, and IVR. Among these, only ICQ is tested in the experiments. If feasible within the rebuttal period, could you also provide the experimental results of DMAC and IVR?\n\n- In L262-265, it is mentioned that propagating regularization in a global-to-local direction \"justifies why previous offline multi-agent methods that apply local behavior regularization can work.\" Could you provide an example of this insight on existing works, such as ICQ to which OMIGA is compared experimentally?\n\n- The pseudocode of OMIGA (Algorithm 1) uses a target network counterpart $\\bar Q_i$ for local action-value function $Q_i$, rather than for local state-value function $V_i$. Considering that target networks are often used for calculating targets in the critic loss, and that this target is $r(\\boldsymbol o, \\boldsymbol a) + \\gamma V_{tot}(\\boldsymbol o')$ in Eq. (14), it seems that the target networks should be kept for state-value functions instead.\n\n- The overall scales of the scores (i.e. hundreds to thousands) in the MuJoCo tasks for Table 1 do not seem to match the scales of the reward distribution (i.e. ones) of the datasets outlined in Table 2 (Appendix B). Could you explain this discrepancy?\n\n- How long were the OMIGA and the baseline methods trained for? Could you also provide the learning curves for the main results?\n\n- How large are the datasets for MuJoCo?\n\n- It seems from Table 6 (Appendix) that despite the statement that \"on these mixed datasets, the behavior policy is suboptimal and more heterogeneous\" (L300-301), all the methods achieve surprisingly good stability in terms of variance. Could you explain how this could occur not only for OMIGA but also for the other baseline methods?\n\n- Will the offline datasets for MuJoCo be publicly released? This would be an immense contribution to the offline MARL community.\n\nLIMITATIONS:\nNo limitations or potential negative societal impacts have been discussed in the submission. A limitation of the proposed work is that the underlying global value functions $Q_{tot}^*$ and $V_{tot}^*$ are assumed to be a linear combination of the local value functions $Q_i^*$ and $V_i^*$ in Eq. (9). Future work may investigate the importance of extending to nonlinear combination of local value functions, such as those satisfying monotonicity constraints [1] or IGM conditions [2,3].\n\n[1] Rashid, Tabish, et al. \"Monotonic value function factorization for deep multi-agent reinforcement learning.\" The Journal of Machine Learning Research 21.1 (2020): 7234-7284.\n\n[2] Son, Kyunghwan, et al. \"Qtran: Learning to factorize with transformation for cooperative multi-agent reinforcement learning.\" International conference on machine learning. PMLR, 2019.\n\n[3] Wang, Jianhao, et al. \"Qplex: Duplex dueling multi-agent q-learning.\" arXiv preprint arXiv:2008.01062 (2020).", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-27"}