Title: AlberDICE: Addressing Out-Of-Distribution Joint Actions in Offline Multi-Agent RL via Alternating Stationary Distribution Correction Estimation
Venue: NeurIPS 2023
Accepted: yes
Keywords: Offline Reinforcement Learning

Multi-Agent Reinforcement Learning
Abstract: One of the main challenges in offline Reinforcement Learning (RL) is the distribution shift that arises from the learned policy deviating from the data collection policy. This is often addressed by avoiding out-of-distribution (OOD) actions during policy improvement as their presence can lead to substantial performance degradation. This challenge is amplified in the offline Multi-Agent RL (MARL) setting since the joint action space grows exponentially with the number of agents.
To avoid this curse of dimensionality, existing MARL methods adopt either value decomposition methods or fully decentralized training of individual agents. However, even when combined with standard conservatism principles, these methods can still result in the selection of OOD joint actions in offline MARL. To this end, we introduce AlberDICE,
an offline MARL algorithm that alternatively performs centralized training of individual agents based on stationary distribution optimization. AlberDICE circumvents the exponential complexity of MARL by computing the best response of one agent at a time while effectively avoiding OOD joint action selection. Theoretically, we show that the alternating optimization procedure converges to Nash policies. In the experiments, we demonstrate that AlberDICE significantly outperforms baseline algorithms on a standard suite of MARL benchmarks.

AlberDICE: Addressing Out-Of-Distribution Joint Actions in Offline Multi-Agent RL via Alternating Stationary Distribution Correction Estimation

Abstract

One of the main challenges in offline Reinforcement Learning (RL) is the distribution shift that arises from the learned policy deviating from the data collection policy. This is often addressed by avoiding out-of-distribution (OOD) actions during policy improvement as their presence can lead to substantial performance degradation. This challenge is amplified in the offline Multi-Agent RL (MARL) setting since the joint action space grows exponentially with the number of agents. To avoid this curse of dimensionality, existing MARL methods adopt either value decomposition methods or fully decentralized training of individual agents. However, even when combined with standard conservatism principles, these methods can still result in the selection of OOD joint actions in offline MARL. To this end, we introduce AlberDICE, an offline MARL algorithm that alternatively performs centralized training of individual agents based on stationary distribution optimization. AlberDICE circumvents the exponential complexity of MARL by computing the best response of one agent at a time while effectively avoiding OOD joint action selection. Theoretically, we show that the alternating optimization procedure converges to Nash policies. In the experiments, we demonstrate that AlberDICE significantly outperforms baseline algorithms on a standard suite of MARL benchmarks.

1 Introduction

Offline Reinforcement Learning (RL) has emerged as a promising paradigm to train RL agents solely from pre-collected datasets [13, 16]. Offline RL aims to address real-world settings in which further interaction with the environment during training is dangerous or prohibitively expensive, e.g., autonomous-car driving, healthcare operations or robotic control tasks [3, 4, 39]. One of the main challenges for successful offline RL is to address the distribution shift that arises from the difference between the policy being learned and the policy used for data collection. Conservatism is a commonly adopted principle to mitigate the distribution shift, which prevents the selection of OOD actions via conservative action-value estimates [11] or direct policy constraints [6].

However, avoiding the selection of OOD actions becomes very challenging in offline Multi-Agent RL (MARL) 2 , as the goal is now to stay close to the states and joint actions in the dataset. This is

not trivial since the joint action space scales exponentially with the number of agents, a problem known as the curse of dimensionality . Previous attempts to address these issues include decomposing the joint action-value function under strict assumptions such as the Individual-Global-Max (IGM) principle [30, 33, 36, 40], or decentralized training which ignores the non-stationarity caused by the changing policies of other agents [18, 25, 38]. While effective in avoiding the curse of dimensionality, these assumptions are insufficient in avoiding OOD joint action selection even when applying the conservatism principles.

To illustrate the problem of joint action OOD, consider the XOR game in Figure 1. In this game, two agents need to coordinate to achieve optimal joint actions, here, either ( A,B ) or ( B,A ) . Despite its simple structure, the co-occurrence of two global optima causes many existing algo- rithms to degenerate in the XOR game [5]. To see this, suppose we have an offline dataset D = { ( A,A, 0) , ( A,B, 1) , ( B,A, 1) } . In this situation, IGM-based methods [40] represent the joint Q ( a 1 , a 2 ) as a combination of individual Q 1 ( a 1 ) and Q 2 ( a 2 ) , where action B is incentivized over action A by both agents in the individual Q functions. As a consequence, IGM-based methods end up selecting ( B,B ) , which is the OOD joint action. Similarly,

A

B

A

0

1

B

1

0

decentralized training methods [25] also choose the OOD joint action ( B,B ) , given that each agent assumes that another agent is fixed with a data policy of selecting A with probability 2 3 . Furthermore, we can see that even behavior-cloning on the expert-only dataset, i.e., D = { ( A,B ) , ( B,A ) } , may end up selecting OOD joint actions as well: each individual policy π 1 ( a 1 ) and π 2 ( a 2 ) will be uniform over the two individual actions, leading to uniform action selection over the entire joint action space; thus, both ( A,A ) and ( B,B ) can be selected. Consequently, OOD joint actions can be hard to avoid especially in these types of environments with multiple global optima and/or when the offline dataset consists of trajectories generated by a mixture of data collection policies.

Our approach and results To address these challenges, we introduce AlberDICE ( AL ternate BE st R esponse Stationary DI stribution C orrection E stimation) , a novel offline MARL algorithm for avoiding OOD actions in the joint action space while circumventing the curse of dimensionality. We start by presenting a coordinate descent-like training procedure where each agent sequentially computes their best response policy while fixing the policies of others. In order to do this in an offline manner, we utilize the linear programming (LP) formulation of RL for optimizing stationary distribution, which has been adapted in offline RL [14] as a stable training procedure where value estimations of OOD actions are eliminated. Furthermore, we introduce a regularization term to the LP objective which matches the stationary distributions of the dataset in the joint action space . This regularization term allows AlberDICE to avoid OOD joint actions as well as the curse of dimensionality without any restrictive assumptions such as factorization of value functions via IGM or fully decentralized training. Overall, our training procedure only requires the mild assumption of Centralized Training and Decentralized Execution (CTDE), a popular paradigm in MARL [17, 28, 30] where we assume access to all global information such as state and joint actions during training while agents act independently during execution.

Theoretically, we show that our regularization term preserves the common reward structure of the underlying task and that the sequence of generated policies converges to a Nash policy (Theorem 4.2). We also conduct extensive experiments to evaluate our approach on a standard suite of MARL environments including the XOR Game, Bridge [5], Multi-Robot Warehouse [27], Google Research Football [12] and SMAC [31], and show that AlberDICE significantly outperforms baselines. To the best of our knowledge, AlberDICE is the first DICE-family algorithm successfully applied to the MARL setting while addressing the problem of OOD joint actions in a principled manner. 3

2 Background

Multi-Agent MDP (MMDP) We consider the fully cooperative MARL setting, which can be formalized as a Multi-Agent Markov Decision Process (MMDP) [24] 4 . An N -Agent MMDP is defined by a tuple G = ⟨N , S , A , r, P, p 0 , γ ⟩ where N = { 1 , 2 , . . . , N } is the set of agent indices, s ∈ S is the state, A = A 1 × · · · × A N is the joint action space, p 0 ∈ ∆( S ) is the initial state

distribution, and γ ∈ (0 , 1) is the discount factor. At each time step, every agent selects an action a i ∈ A i via a policy π i ( a i | s ) given a state s ∈ S , and receives a reward according to a joint (among all agents) reward function r : S × A → R . Then, the state is transitioned to the next state s ′ via the transition probability P : S × A → ∆( S ) which depends on the global state s ∈ S and the joint action a = { a 1 , . . . , a N } ∈ A and the process repeats. We also use a -i to denote the actions taken by all agents other than i .

Given a joint policy π = { π 1 , . . . , π N } ( or π = { π i , π -i } when we single out agent i ∈ N ) , we have that π ( a | s ) = π 1 ( a 1 | s ) · π 2 ( a 2 | s, a 1 ) · · · π N ( a N | s, a 1 , . . . , a N -1 ) . If the π i 's can be factorized into a product of individual policies, then π is factorizable , and we write π ( a | s ) = ∏ N j =1 π j ( a j | s ) . We denote the set of all policies as Π , and the set of factorizable policies as Π f . Given a joint policy π , a state value function V π and a action-value function Q π are defined by:

The goal is to find a factorizable policy π ∈ Π f that maximizes the joint rewards so that each agent can act in a decentralized manner during execution. We will consider solving MMDP in terms of optimizing stationary distribution. For a given policy π , its stationary distribution d π is defined by:

In offline MARL, online interaction with the environment is not allowed, and the policy is optimized only using the offline dataset D = { ( s, a , r, s ′ ) k } | D | k =1 collected by diverse data-collection agents. We denote the dataset distribution as d D and abuse the notation d D for s ∼ d D , ( s, a ) ∼ d D , and ( s, a, s ′ ) ∼ d D .

Nash Policy We focus on finding the factorizable policy { π i : S → ∆( A i ) } N i =1 ∈ Π f during centralized training, which necessitate the notions of Nash and ϵ -Nash policies.

Definition 2.1 (Nash and ϵ -Nash Policy) . A joint policy π ∗ = ⟨ π ∗ i ⟩ N i =1 is a Nash policy if it holds

Similarly, a joint policy π ∗ = ⟨ π ∗ i ⟩ N i =1 is an ϵ -Nash policy if there exists an ϵ > 0 so that for each agent i ∈ N , it holds that V π ∗ i , π ∗ -i i ( s ) ≥ V π i , π ∗ -i i ( s ) -ϵ , for all π i , s . In other words, π ∗ = ⟨ π ∗ i , π ∗ -i ⟩ is a Nash policy, if each agent i ∈ N has no incentive to deviate from π ∗ i to an alternative policy, π i , given that all other agents are playing π ∗ -i .

3 AlberDICE

In this section, we introduce AlberDICE, an offline MARL algorithm that optimizes the stationary distribution of each agent's factorizable policy while effectively avoiding OOD joint actions. AlberDICE circumvents the exponential complexity of MARL by computing the best response of one agent at a time in an alternating manner. In contrast to existing methods that adopt decentralized training of each agent [25] where other agents' policies are assumed to follow the dataset distribution, AlberDICE adopts centralized training of each agent, which takes into account the non-stationarity incurred by the changes in other agents' policies.

Regularized Linear Program (LP) for MMDP The derivation of our algorithm starts by augmenting the standard linear program for MMDP with an additional KL-regularization term, where only a single agent i ∈ N is being optimized while other agents' policies π -i are fixed:

where D KL ( p ( x ) ∥ q ( x )) := ∑ x p ( x ) log p ( x ) q ( x ) is the KL-divergence between probability distributions p and q , and α > 0 is a hyperparameter that controls the degree of conservatism, i.e., the amount of

penalty for deviating from the data distribution, which is a commonly adopted principle in offline RL [8, 14, 22, 42]. Satisfying the Bellman-flow constraints (3) guarantees that d ( s, a i , a -i ) := d i ( s, a i ) π -i ( a -i | s ) is a valid stationary distribution in the MMDP.

As we show in our theoretical treatment of the regularized LP in Section 4, the selected regularization term defined in terms of joint action space critically ensures that every agent i ∈ N optimizes the same objective function in (2). This ensures that when agents optimize alternately, the objective function always monotonically improves which, in turn, guarantees convergence (see Theorem 4.2). This is in contrast to existing methods such as [25], where each agent optimizes the different objective functions. Importantly, this is achieved while ensuring conservatism. As can be seen from (2), the KL-regularization term is defined in terms of the joint stationary distribution of all agents which ensures that the optimization of the regularized LP effectively avoids OOD joint action selection.

The optimal solution of the regularized LP (2-3), d ∗ i , corresponds to the stationary distribution for a best response policy π ∗ i against the fixed π -i , and π ∗ i can be obtained by π ∗ i = d ∗ i ( s,a i ) ∑ a ′ d ∗ i ( s,a i ) .

The (regularized) LP (2-3) can also be understood as solving a (regularized) reduced MDP ¯ M i = ⟨S , A i , ¯ P i , ¯ r i , γ, p 0 ⟩ for a single agent i ∈ N , where ¯ P i and ¯ r i are defined as follows 5 :

i

Then, d ∗ i is an optimal stationary distribution on the reduced MDP, ¯ M i , but the reduced MDP is non-stationary due to other agents' policy, π -i , updates. Therefore, it is important to account for changes in π -i during training in order to avoid selection of OOD joint actions.

Lagrangian Formulation The constrained optimization (2-3) is not directly solvable since we do not have a white-box model for the MMDP. In order to make (2-3) amenable to offline learning in a model-free manner, we consider a Lagrangian of the constrained optimization problem:

where ν i ( s ) ∈ R is the Lagrange multiplier for the Bellman flow constraints 6 . Still, (4) is not directly solvable due to its requirement of P ( s ′ | s, a i , a -i ) for ( s, a i ) ∼ d i that are not accessible in the offline setting. To make progress, we re-arrange the terms in (4) as follows

where e ν i ( s, a i ) is the advantage by ν i , and w i ( s, a i ) are the stationary distribution correction ratios between d i and d D . Finally, to enable every term in (7) to be estimated from samples in the offline dataset D , we adopt importance sampling, which accounts for the distribution shift in other agents' policies, π -i :

where ˆ e ν i ( s, a i , a -i , s ′ ) := r ( s, a i , a -i ) -α log π -i ( a -i | s ) π D -i ( a -i | s,a i ) + γν i ( s ′ ) -ν i ( s ) . Every term in (8) can be now evaluated using only the samples in the offline dataset. Consequently, AlberDICE aims to solve the unconstrained minimax optimization (8) for each agent i ∈ N . Once we compute the optimal solution ( ν ∗ i , w ∗ i ) of (8), we obtain the information about the optimal policy π ∗ i (i.e. the best response policy against the fixed π -i ) in the form of distribution correction ratios w ∗ i = d π ∗ i ( s,a i ) d D ( s,a i ) .

Pretraining autoregressive data policy To optimize (8), we should be able to evaluate π D -i ( a -i | s, a i ) for each ( s, a i , a -i ) ∈ D . To this end, we pretrain the data policy via behavior cloning, where we adopt an MLP-based autoregressive policy architecture, similar to the one in [43]. The input dimension of π D -i only grows linearly with the number of agents. Then, for each i ∈ N , we optimize the following:

̸

While, in principle, the joint action space grows exponentially with the number of agents, learning a joint data distribution in an autoregressive manner is known to work quite well in practice [7, 29].

Practical Algorithm: Minimax to Min Still, solving the nested minimax optimization (7) can be numerically unstable in practice. In this section, we derive a practical algorithm that solves a single minimization only using offline samples. For brevity, we denote each sample ( s, a i , a -i , s ′ ) in the dataset as x . Also, let ˆ E x ∈ D [ f ( x )] := 1 | D | ∑ x ∈ D f ( x ) be a Monte-Carlo estimate of E x ∼ p [ f ( x )] , where D = { x k } | D | k =1 ∼ p . First, we have an unbiased estimator of (7):

where ρ i ( x ) is defined as:

̸

Optimizing (10) can suffer from large variance due to the large magnitude of ρ ( x ) , which contains products of N -1 policies. To remedy the large variance issue, we adopt Importance Resampling (IR) [32] to (10). Specifically, we sample a mini-batch of size K from D with probability proportional to ρ ( x ) , which constitutes a resampled dataset D ρ i = { ( s, a i , a -i , s ′ ) k } K k =1 . Then, we solve the following optimization, which now does not involve the importance ratio:

where ¯ ρ i := ˆ E x ∈ D [ ρ i ( x )] . It can be proven that (12) is still an unbiased estimator of (7) thanks to the bias correction term of ¯ ρ [32]. The resampling procedure can be understood as follows: for each data sample x = ( s, a i , a -i , s ′ ) , if other agents' policy π -i selects the action a -i ∈ D with low probability, i.e., π -i ( a -i | s ) ≈ 0 , the sample x will be removed during the resampling procedure, which makes the samples in the resampled dataset D ρ i consistent with the reduced MDP ¯ M i 's dynamics. Finally, to avoid the numerical instability associated with solving a min-max optimization problem, we exploit the properties of the inner-maximization problem in (12), specifically, its concavity in w i , and derive its closed-form solution.

Proposition 3.1. The closed-form solution for the inner-maximization in (12) for each x is given by

By plugging equation (13) into (12), we obtain the following minimization problem:

As we show in Proposition B.1 in the Appendix, ˜ L ( ν i ) is an unconstrained convex optimization problem where the function to learn ν i is state-dependent . Furthermore, the terms in (14) are estimated only using the ( s, a i , a -i , s ′ ) samples in the dataset, making it free from the extrapolation error by bootstrapping OOD action values. Also, since ν i ( s ) does not involve joint actions, it is not required to adopt IGM-principle in ν i network modeling; thus, there is no need to limit the expressiveness power of the function approximator. In practice, we parameterize ν i using simple MLPs, which take the state s as an input and output a scalar value.

Policy Extraction The final remaining step is to extract a policy from the estimated distribution correction ratio w ∗ i ( s, a i ) = d π ∗ i ( s,a i ) d D ( s,a i ) . Unlike actor-critic approaches which perform intertwined optimizations by alternating between policy evaluation and policy improvement, solving (14) directly results in the optimal ν ∗ i . However, this does not result in an executable policy. We therefore utilize the I-projection policy extraction method from [14] which we found to be most numerically stable

In summary, AlberDICE computes the best response policy of agent i by: (1) resampling data points based on the other agents' policy ratios ρ (11) where the data policy π D -i ( a -i | s, a i ) can be pretrained, (2) solving a minimization problem to find ν ∗ i ( s ) (31) and finally, (3) extracting the policy using the obtained ν ∗ i by I-projection (15). In practice, rather than training ν i until convergence at each iteration, we perform a single gradient update for each agent ν i and π i alternatively. We outline the details of policy extraction (Appendix E.2) and the full learning procedure in Algorithm 1 (Appendix E).

4 Preservation of Common Rewards and Convergence to Nash Policies

In the previous sections, AlberDICE was derived as a practical algorithm in which agents alternately compute the best response DICE while avoiding OOD joint actions. We now prove formally that this procedure converges to Nash policies. While it is known that alternating best response can converge to Nash policies in common reward settings [1], it is not immediately clear whether the same result holds for the regularized LP (2-3), and hence the regularized reward function of the environment, preserves the common reward structure of the original MMDP. As we show in Lemma 4.1, this is indeed the case, i.e., the modified reward in (2-3) is shared across all agents. This directly implies that optimization of the corresponding LP yields the same value for all agents i ∈ N for any joint policy, π , with factorized individual policies, { π i } i ∈N .

Lemma 4.1. Consider a joint policy π = ( π i ) i ∈N , with factorized individual policies, i.e., π ( a | s ) = ∏ i ∈N π i ( a i | s ) for all ( s, a ) ∈ S ×A with a = ( a i ) i ∈ N . Then, the regularized objective in the LP formulation of AlberDICE, cf. equation (2), can be evaluated to

with ˜ r ( s, a i , a -i ) := r ( s, a i , a -i ) -α · log d π ( s ) π ( a | s ) d D ( s,a i , a -i ) , for all ( s, a ) ∈ S ×A . In particular, for any joint policy, π = ( π ) i ∈N , with factorized invdividual policies, the regularized objective in the LP formulation of AlberDICE attains the same value for all agents i ∈ N .

We can now use Lemma 4.1 to show that AlberDICE enjoys desirable convergence guarantees in tabular domains in which the policies, π i ( a i | s ) , can be directly extracted from d i ( s, a i ) through the expression π i ( a i | s ) = d i ( s,a i ) ∑ a j d i ( s,a j ) .

Theorem 4.2. Given an MMDP, G , and a regularization parameter α ≥ 0 , consider the modified MMDP ˜ G with rewards ˜ r as defined in Lemma 4.1 and assume that each agent alternately solves the regularized LP defined in equations (2-3). Then, the sequence of policy updates, ( π t ) t ≥ 0 , converges to a Nash policy, π ∗ = ( π ∗ i ) i ∈N , of ˜ G .

The proofs of Lemma 4.1 and Theorem 4.2 are given in Appendix D. Intuitively, Theorem 4.2 relies on the fact that the objectives in the alternating optimization problems (2-3) involve the same rewards for all agents for any value of the regularization parameter, α ≥ 0 , cf. Lemma 4.1. Accordingly, every update by any agent improves this common value function, ( ˜ V π ( s )) s ∈ S , and at some point the sequence of updates is bound to terminate at a (local) maximum of ˜ V . At this point, no agent can improve by deviating to another policy which implies that the corresponding joint policy is a Nash policy of the underlying (modified) MMDP. For practical purposes, it is also relevant to note that the process may terminate at an ϵ -Nash policy (cf. Definition 2.1), since the improvements in the common value function may become arbitrarily small when solving the LPs numerically.

A

B

A

0.45

0.22

B

0.22

0.11

BC

A

B

0.0

0.0

0.0

1.0

ICQ

A

B

0.0

0.0

0.0

1.0

OMAR

A

B

0.25

0.25

0.25

0.25

MADTKD

A

B

0.25

0.25

0.25

0.25

OptiDICE

A

B

0.0

1.0

0.0

0.0

AlberDICE

A direct implication from the construction of the AlberDICE algorithm, which is also utilized in the proof of Theorem 4.2, is that the AlberDICE algorithm maintains during its execution and, thus, also returns upon its termination, a factorizable policy, i.e., a policy that can be factorized.

Corollary 4.3. Let π ∗ be the extracted joint policy that is returned from the AlberDICE algorithm. Then, π ∗ is factorizable, i.e., there exist individual policies, ⟨ π ∗ i ⟩ i ∈N , one for each agent, so that π ∗ ( a | s ) = ∏ i ∈N π ∗ i ( a i | s ) for all a ∈ A,s ∈ S .

5 Experimental Results

We evaluate AlberDICE on a series of benchmarks, namely the Penalty XOR Game and Bridge [5], as well as challenging high-dimensional domains such as Multi-Robot Warehouse (RWARE) [27], Google Research Football (GRF) [12] and StarCraft Multi-Agent Challenge (SMAC) [31]. Our baselines include Behavioral Cloning (BC), ICQ [40], OMAR [25], MADTKD [35] and OptiDICE [14] 7 . For a fair comparison, all baseline algorithms use separate network parameters 8 for each agent and the same policy structure. Further details on the dataset are provided in Appendix F.

5.1 Penalty XOR Game

We first evaluate AlberDICE on a 2 × 2 Matrix Game called Penalty XOR shown in Figure 2. We construct four different datasets: (a) { AB } , (b) { AB,BA } , (c) { AA,AB,BA } , (d) { AA,AB,BA,BB } . The full results showing the final joint policy values are shown in Table 6 in the Appendix. We show the results for dataset (c) in Table 1.

AlberDICE is the only algorithm that converges to a deterministic optimal policy, BA or AB for all datasets.

On the other hand, OptiDICE and MATDKD converges to a stochastic policy where both agents choose A and B with equal probability. This is expected for both algorithms which optimize over joint actions during

A

B

A

0

1

B

1

-

2

centralized training which can still lead to joint action OOD if the joint policy is not factorizable. ICQ converges to AA for (b), (d) and BB for (c), which shows the tendency of the IGM constraint and value factorization approaches to converge to OOD joint actions. These results also suggest that the problem of joint action OOD becomes more severe when the dataset collection policy is diverse and/or the environment has multiple global optima requiring higher levels of coordination.

5.2 Bridge

Bridge is a stateful extension of the XOR Game, where two agents must take turns crossing a narrow bridge. We introduce a harder version (Figure 3) where both agents start 'on the bridge' rather than on the diagonal cells of the opponent goal states as in the original game.

This subtle change makes the task much harder because now there are only two optimal actions: (Left, Left) and (Right, Right) at the initial state. Conversely, the original game can be solved optimally as long as at least one agent goes on the bridge.

The optimal dataset (500 trajectories) was constructed by a mixture of deterministic optimal policies which randomizes between Agent 1 crossing the bridge first while Agent 2 retreats, and vice-versa. The mix dataset further adds 500 trajectories by a uniform random policy.

Dataset

BC

ICQ

OMAR

MADTKD

OptiDICE

AlberDICE

Optimal

-

1

.

26

-

2

.

21

±

0

.

90

-

1

.

81

±

0

.

12

-

6

.

01

±

0

.

00

-

4

.

31

±

0

.

27

-

2

.

71

±

0

.

69

-

1

.

27

±

0

.

03

Mix

-

4

.

56

-

5

.

88

±

0

.

49

-

6

.

01

±

0

.

00

-

6

.

01

±

0

.

00

-

6

.

58

±

0

.

26

-

1

.

76

±

0

.

17

-

1

.

29

±

0

.

00

(a) AlberDICE ( i = 1 ) (b) AlberDICE ( i = 2 )

(c) BC (

i

= 1

)

(d) BC (

i

= 2

)

Figure 4: Visualization of learned policies at the initial state for (from Left to Right) AlberDICE (agent 1, agent 2) and BC (agent 1, agent 2) , for the Bridge (Hard) on the optimal dataset. The blue arrows indicate agent 2's policies when agent 1 is at · and red arrows indicate agent 1's policy when agent 2 is at · .

The performance results in Table 2 show that AlberDICE can stably perform near-optimally in both the optimal and mix datasets. Also, the learned policy visualizations for the optimal dataset in Figure 4 show that AlberDICE is the only algorithm which converges to the optimal deterministic policy in the initial state, similar to the results in the Matrix game. We also include similar policy visualization results for all states and algorithms in Appendix H.

5.3 High-Dimensional MARL Benchmarks

We further evaluate AlberDICE on standard MARL benchmarks including RWARE [26], GRF [12] and SMAC [31]. For RWARE and GRF, we train an autoregressive policy using Multi-Agent Transformers (MAT) [37] in order to collect diverse trajectories for constructing offline datasets. For SMAC, we use the public dataset provided by Meng et al. [19].

RWARE simulates a real-world warehouse in which robots move and deliver requested goods in a partially observable environment (each agent can observe the 3 × 3 square centered on the agent). RWARE requires high levels of coordination, especially whenever the density of agents is high and there are narrow pathways where only a single agent can pass through (similar to Bridge).

The results in Table 3 show that AlberDICE performs on-par with OMAR in the Tiny ( 11 × 11 ) environment despite OMAR being a decentralized training algorithm. As shown in Figure 9(a) of [26], a large portion of the Tiny map contains wide passageways where agents can move around relatively freely without worrying about colliding with other agents. On the other hand, AlberDICE outperforms baselines in the Small ( 11 × 20 ) environment (shown in Figure 9(b) of [26]), where precise coordination among agents becomes more critical since there are more narrow pathways and the probability of a collision is significantly higher. We also note that the performance gap between AlberDICE and baselines is largest when there are more agents ( N = 6 ) in the confined space. This increases the probability of a collision, and thus, requires higher levels of coordination.

{

0.000

0.005

0.010

0.015

Uncertainty

0%

2%

4%

6%

8%

10%

12%

Percentage

8m_vs_9m

Data

BC

OptiDICE

MADTKD

AlberDICE (ours)

OMAR

ICQ

OOD threshold

0.000

0.005

0.010

0.015

Uncertainty

0%

2%

4%

6%

8%

Percentage

3s5z_vs_3s6z

Data

BC

OptiDICE

MADTKD

AlberDICE (ours)

OMAR

ICQ

OOD threshold

(a) Histogram of uncertainty estimates

8m_vs_9m

3s5z_vs_3s6z

(

N

= 8

)

(

N

= 8

)

BC

0.1%

0.3%

ICQ

54.7%

26.2%

OMAR

16.5%

5.7%

MADTKD

1.7%

0.8%

OptiDICE

0.1%

0.3%

AlberDICE

4.8%

1.6%

(b) Percentage of selecting OOD joint actions.

U

(

s,

a

) :

s

∈

D,a

∼

π

(

s

)

}

.

Our results for GRF and SMAC in Tables 4 and 5 show that AlberDICE performs consistently well across all scenarios, and outperforms all baselines especially in the Super Hard maps, Corridor and 6h8z. The strong performance by AlberDICE corroborates the importance of avoiding OOD joint actions in order to avoid performance degradation.

5.4 Does AlberDICE reduce OOD joint actions?

In order to evaluate how effectively AlberDICE prevents selecting OOD joint actions, we conducted additional experiments on two SMAC domains (8m_vs_9m and 3s5z_vs_3s6z) as follows. First, we trained an uncertainty estimator U ( s, a ) via fitting random prior [2] f : S ×A 1 ×··· × A N → R m using the dataset D = { ( s, a ) k } | D | k =1 . Then, U ( s, a ) = ∥ f ( s, a ) -h ( s, a ) ∥ 2 outputs low values for in-distribution ( s, a ) samples and outputs high values for out-of-distribution ( s, a ) samples. Figure 5(a) shows a histogram of uncertainty estimates U ( s, π 1 ( s ) , . . . , π N ( s )) for each s ∈ D and the joint action selected by each method. We set the threshold τ for determining OOD samples to 99.9%-quantile of { U ( s, a ) : ( s, a ) ∈ D } . Figure 5(b) presents the percentage of selecting OOD joint actions by each method. AlberDICE selects OOD joint actions significantly less often than ICQ (IGM-based method) and OMAR (decentralized training method) while outperforming them in terms of success rate (see Table 5).

6 Related Work

DICE for Offline RL Numerous recent works utilize the LP formulation of RL to derive DICE algorithms for policy evaluation [20-22]. OptiDICE [14] was introduced as the first policy optimization algorithm for DICE and as a stable offline RL algorithm which does not require value estimation of OOD actions. While OptiDICE can be naively extended to offline MARL in principle, it can still fail to avoid OOD joint actions since its primary focus is to optimize over the joint action space of the MMDPand does not consider the factorizability of policies. We detail the shortcomings of a naive extension of OptiDICE to multi-agent settings in Appendix C.

Value-Based MARL Apopular method in cooperative MARL is (state-action) value decomposition. This approach can be viewed as a way to model Q ( s, a ) implicitly by aggregating Q i in a specific manner, e.g., sum [34], or weighted sum [30]. Thus, it avoids modelling Q ( s, a ) explicitly over the joint action space. QTRAN [33] and QPLEX [36] further achieve full representativeness of IGM

9 . These approaches have been shown to perform well in high-dimensional complex environments including SMAC [31]. However, the IGM assumption and the value decomposition structure have been shown to perform poorly even in simple coordination tasks such as the XOR game [5].

Policy-Based MARL Recently, policy gradient methods such as MAPPO [41] have shown strong performance on many complex benchmarks including SMAC and GRF. Fu et al. [5] showed that independent policy gradient with separate parameters can solve the XOR game and the Bridge environment by converging to a deterministic policy for one of the optimal joint actions. However, it requires an autoregressive policy structure (centralized execution) to learn a stochastic optimal policy which covers multiple optimal joint actions. These empirical findings are consistent with theoretical results [15, 44] showing that running independent policy gradient can converge to a Nash policy in cooperative MARL. On the downside, policy gradient methods are trained with on-policy samples and thus, cannot be extended to the offline RL settings due to the distribution shift problem [16].

Offline MARL ICQ [40] was the first MARL algorithm applied to the offline setting. It proposed an actor-critic approach to overcome the extrapolation error caused by the evaluation of unseen state-action pairs, where the error is shown to grow exponentially with the number of agents. The centralized critic here uses QMIX [30] and thus, it inherits some of the weaknesses associated with value decomposition and IGM. OMAR [25] is a decentralized training algorithm where each agent runs single-agent offline RL over the individual Q-functions and treats other agents as part of the environment. As a consequence, it lacks theoretical motivation and convergence guaranteess in the underlying MMDP or Dec-POMDP. MADTKD [35] extends Multi-Agent Decision Transformers [19] to incorporate credit assignment across agents by distilling the teacher policy learned over the joint action space to each agent (student). This approach can still lead to OOD joint actions since the teacher policy learns a joint policy over the joint action space and the actions are distilled individually to students.

7 Limitations

AlberDICE relies on Nash policy convergence which is a well-established solution concept in Game Theory, especially in the general non-cooperative case where each agent may have conflicting reward functions. One limitation of AlberDICE is that the Nash policy may not necessarily correspond to the global optima in cooperative settings. The outcome of the iterative best response depends on the starting point (region of attraction of each Nash policy) and is, thus, generally not guaranteed to find the optimal Nash policy [1]. This is the notorious equilibrium selection problem which is an open problem in games with multiple equilibria, even if they have common reward structure (See Open Questions in [15]). Nonetheless, Nash policies have been used as a solution concept for iterative update of each agents as a way to ensure convergence to factorized policies in Cooperative MARL [10]. Furthermore, good equilibria tend to have larger regions of attraction and practical performance is typically very good as demonstrated by our extensive experiments.

8 Conclusion

In this paper, we presented AlberDICE, a multi-agent RL algorithm which addresses the problem of distribution shift in offline MARL by avoiding both OOD joint actions and the exponential nature of the joint action space. AlberDICE leverages an alternating optimization procedure where each agent computes the best response DICE while fixing the policies of other agents. Furthermore, it introduces a regularization term over the stationary distribution of states and joint actions in the dataset. This regularization term preserves the common reward structure of the environment and together with the alternating optimization procedure, allows convergence to Nash policies. As a result, AlberDICE is able to perform robustly across many offline MARL settings, even in complex environments where agents can easily converge to sub-optimal policies and/or select OOD joint actions. As the first DICE algorithm applied to offline MARL with a principled approach to curbing distribution shift, this work provides a starting point for further applications of DICE in MARL and a promising perspective in addressing the main problems of offline MARL.

Acknowledgments and Disclosure of Funding

This work was partly supported by the IITP grant funded by MSIT (No.2020-0-00940, Foundations of Safe Reinforcement Learning and Its Applications to Natural Language Processing; No.2022-000311, Development of Goal-Oriented Reinforcement Learning Techniques for Contact-Rich Robotic Manipulation of Everyday Objects; No.2019-0-00075, AI Graduate School Program (KAIST); No.2021-0-02068, AI Innovation Hub), NRF of Korea (NRF2019R1A2C1087634), Field-oriented Technology Development Project for Customs Administration through NRF of Korea funded by the MSIT and Korea Customs Service (NRF2021M3I1A1097938), KAIST-NAVER Hypercreative AI Center, the BAIR Industrial Consortium, and NSF AI4OPT AI Centre.

References

Dimitri Bertsekas. Multiagent value iteration algorithms in dynamic programming and reinforcement learning, 2020.

Kamil Ciosek, Vincent Fortuin, Ryota Tomioka, Katja Hofmann, and Richard Turner. Conservative uncertainty estimation by fitting prior networks. In International Conference on Learning Representations , 2020.

A. Dafoe, E. Hughes, Y. Bachrach, T. Collins, K. R. McKee, J. Z. Leibo, K. Larson, and T. Graepel. Open Problems in Cooperative AI. arXiv e-prints , December 2020.

A. Dafoe, Y. Bachrach, G. Hadfield, E. Horvitz, K. Larson, and T. Graepel. Cooperative AI: machines must learn to find common ground. Nature , 7857:33-36, 2021. doi: 10.1038/d41586-021-01170-0.

Wei Fu, Chao Yu, Zelai Xu, Jiaqi Yang, and Yi Wu. Revisiting some common practices in cooperative multi-agent reinforcement learning. In Proceedings of the 39th International Conference on Machine Learning , volume 162 of Proceedings of Machine Learning Research , pages 6863-6877. PMLR, 17-23 Jul 2022.

Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In Proceedings of the 36th International Conference on Machine Learning , volume 97 of Proceedings of Machine Learning Research , pages 2052-2062. PMLR, 09-15 Jun 2019.

Michael Janner, Qiyang Li, and Sergey Levine. Offline reinforcement learning as one big sequence modeling problem. In Advances in Neural Information Processing Systems , volume 34, pages 1273-1286. Curran Associates, Inc., 2021.

Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. MOReL : Model-based offline reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS) , 2020.

Geon-Hyeong Kim, Seokin Seo, Jongmin Lee, Wonseok Jeon, HyeongJoo Hwang, Hongseok Yang, and Kee-Eung Kim. DemoDICE: Offline imitation learning with supplementary imperfect demonstrations. In International Conference on Learning Representations , 2022.

Jakub Grudzien Kuba, Ruiqing Chen, Muning Wen, Ying Wen, Fanglei Sun, Jun Wang, and Yaodong Yang. Trust region policy optimisation in multi-agent reinforcement learning. In International Conference on Learning Representations , 2022.

Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. In Advances in Neural Information Processing Systems , volume 33, pages 11791191. Curran Associates, Inc., 2020.

K. Kurach, A. Raichuk, P. Sta´ nczyk, M. Zajac, O. Bachem, L. Espeholt, C. Riquelme, D. Vincent, M. Michalski, O. Bousquet, and S. Gelly. Google research football: A novel reinforcement learning environment, 2019.

Sascha Lange, Thomas Gabel, and Martin Riedmiller. Batch Reinforcement Learning , pages 45-73. Springer Berlin Heidelberg, Berlin, Heidelberg, 2012.

Jongmin Lee, Wonseok Jeon, Byungjun Lee, Joelle Pineau, and Kee-Eung Kim. OptiDICE: Offline policy optimization via stationary distribution correction estimation. In Proceedings of the 38th International Conference on Machine Learning , volume 139 of Proceedings of Machine Learning Research , pages 6120-6130. PMLR, 18-24 Jul 2021.

Stefanos Leonardos, Will Overman, Ioannis Panageas, and Georgios Piliouras. Global Convergence of Multi-Agent Policy Gradient in Markov Potential Games. In International Conference on Learning Representations , 2022.

Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems, 2020.

Ryan Lowe, YI WU, Aviv Tamar, Jean Harb, OpenAI Pieter Abbeel, and Igor Mordatch. Multi-agent actorcritic for mixed cooperative-competitive environments. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 30. Curran Associates, Inc., 2017.

Xueguang Lyu, Yuchen Xiao, Brett Daley, and Christopher Amato. Contrasting centralized and decentralized critics in multi-agent reinforcement learning. In Proceedings of the 20th International Conference on Autonomous Agents and MultiAgent Systems , AAMAS '21, page 844-852. International Foundation for Autonomous Agents and Multiagent Systems, 2021.

Linghui Meng, Muning Wen, Yaodong Yang, chenyang le, Xi yun Li, Haifeng Zhang, Ying Wen, Weinan Zhang, Jun Wang, and Bo XU. Offline pre-trained multi-agent decision transformer, 2022.

Ofir Nachum and Bo Dai. Reinforcement learning via fenchel-rockafellar duality, 2020.

Ofir Nachum, Yinlam Chow, Bo Dai, and Lihong Li. DualDICE: Behavior-agnostic estimation of discounted stationary distribution corrections. In Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019.

Ofir Nachum, Bo Dai, Ilya Kostrikov, Yinlam Chow, Lihong Li, and Dale Schuurmans. AlgaeDICE: Policy gradient from arbitrary experience. arXiv preprint arXiv:1912.02074 , 2019.

Allen Nie, Yannis Flet-Berliac, Deon R. Jordan, William Steenbergen, and Emma Brunskill. Data-efficient pipeline for offline reinforcement learning with limited data, 2023.

Frans A. Oliehoek and Christopher Amato. A Concise Introduction to Decentralized POMDPs . Springer Publishing Company, Incorporated, 1st edition, 2016. ISBN 3319289276.

Ling Pan, Longbo Huang, Tengyu Ma, and Huazhe Xu. Plan better amid conservatism: Offline multi-agent reinforcement learning with actor rectification. In Proceedings of the 39th International Conference on Machine Learning , volume 162 of Proceedings of Machine Learning Research , pages 17221-17237. PMLR, 17-23 Jul 2022.

Georgios Papoudakis, Filippos Christianos, Lukas Schäfer, and Stefano V Albrecht. Benchmarking multiagent deep reinforcement learning algorithms in cooperative tasks. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1) , 2021.

Georgios Papoudakis, Filippos Christianos, Lukas Schäfer, and Stefano V. Albrecht. Benchmarking multi-agent deep reinforcement learning algorithms in cooperative tasks. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks (NeurIPS) , 2021.

Bei Peng, Tabish Rashid, Christian Schroeder de Witt, Pierre-Alexandre Kamienny, Philip Torr, Wendelin Boehmer, and Shimon Whiteson. FACMAC: Factored multi-agent centralised policy gradients. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems , 2021.

Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. 2018.

Tabish Rashid, Mikayel Samvelyan, Christian Schroeder, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. QMIX: Monotonic value function factorisation for deep multi-agent reinforcement learning. In Proceedings of the 35th International Conference on Machine Learning , volume 80 of Proceedings of Machine Learning Research , pages 4295-4304. PMLR, 10-15 Jul 2018.

Mikayel Samvelyan, Tabish Rashid, Christian Schroeder de Witt, Gregory Farquhar, Nantas Nardelli, Tim G. J. Rudner, Chia-Man Hung, Philip H. S. Torr, Jakob Foerster, and Shimon Whiteson. The starcraft multi-agent challenge, 2019.

Matthew Schlegel, Wesley Chung, Daniel Graves, Jian Qian, and Martha White. Importance resampling for off-policy prediction. In Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019.

Kyunghwan Son, Daewoo Kim, Wan Ju Kang, David Earl Hostallero, and Yung Yi. QTRAN: Learning to factorize with transformation for cooperative multi-agent reinforcement learning. In Proceedings of the 36th International Conference on Machine Learning , volume 97 of Proceedings of Machine Learning Research , pages 5887-5896. PMLR, 09-15 Jun 2019.

Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojciech Marian Czarnecki, Vinicius Zambaldi, Max Jaderberg, Marc Lanctot, Nicolas Sonnerat, Joel Z Leibo, Karl Tuyls, et al. Value-decomposition networks for cooperative multi-agent learning. arXiv preprint arXiv:1706.05296 , 2017.

Wei-Cheng Tseng, Tsun-Hsuan Wang, Yen-Chen Lin, and Phillip Isola. Offline multi-agent reinforcement learning with knowledge distillation. In Advances in Neural Information Processing Systems , 2022.

Jianhao Wang, Zhizhou Ren, Terry Liu, Yang Yu, and Chongjie Zhang. QPLEX: Duplex dueling multiagent q-learning. In International Conference on Learning Representations , 2021.

Muning Wen, Jakub Grudzien Kuba, Runji Lin, Weinan Zhang, Ying Wen, Jun Wang, and Yaodong Yang. Multi-agent reinforcement learning is a sequence modeling problem. In Advances in Neural Information Processing Systems , 2022.

C. S. D. Witt, Tarun Gupta, Denys Makoviichuk, Viktor Makoviychuk, Philip H. S. Torr, Mingfei Sun, and Shimon Whiteson. Is independent learning all you need in the starcraft multi-agent challenge? ArXiv , abs/2011.09533, 2020.

Erfu Yang and Dongbing Gu. Multiagent reinforcement learning for multi-robot systems: A survey. 06 2004.

Yiqin Yang, Xiaoteng Ma, Chenghao Li, Zewu Zheng, Qiyuan Zhang, Gao Huang, Jun Yang, and Qianchuan Zhao. Believe what you see: Implicit constraint approach for offline multi-agent reinforcement learning. In Advances in Neural Information Processing Systems , volume 34, pages 10299-10312. Curran Associates, Inc., 2021.

Chao Yu, Akash Velu, Eugene Vinitsky, Jiaxuan Gao, Yu Wang, Alexandre Bayen, and Yi Wu. The surprising effectiveness of PPO in cooperative multi-agent games. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track , 2022.

Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. MOPO: Model-based offline policy optimization. In Advances in Neural Information Processing Systems (NeurIPS) , 2020.

Michael R Zhang, Thomas Paine, Ofir Nachum, Cosmin Paduraru, George Tucker, ziyu wang, and Mohammad Norouzi. Autoregressive dynamics models for offline policy evaluation and optimization. In International Conference on Learning Representations , 2021.

Runyu Zhang, Zhaolin Ren, and Na Li. Gradient play in stochastic games: stationary points, convergence, and sample complexity, 2021.

A Individual-Global-Max (IGM) and its Limitations

Centralized Training with Decentralized Execution (CTDE) refers to a paradigm in MARL where the training phase is permitted to utilize any global information such as the joint policy π = ⟨ π 1 , . . . , π N ⟩ and global state s . Practical CTDE algorithms avoid the combinatorial nature of the joint action space, and reduce the non-stationarity, which arises from agents' simultaneous policy updates during training.

However, an important challenge for CTDE algorithms is that during training they still need to learn a joint policy that can be factorized into individual policies. During the execution phase, agents take individual actions, a i , without conditioning on other agents' actions, a -i , or policies, π -i . This independence assumption poses a challenge for many pre-existing algorithms, especially for offline MARL when the dataset is generated by a mixture of different data collection policies and/or the environment contains multiple global optima.

One popular way to learn factorizable policies from centralized training is to impose an IndividualGlobal-Max (IGM) assumption.

Definition A.1 (Individual-Global-Max (IGM) [30, 33]) . Individual utility functions { Q i } N i =1 satisfies the IGM condition for a joint state-action value function Q : S ×A→ R if the following condition holds:

Intuitively, the IGM condition implies that the optimal Q-function Q ∗ for a given task or environment can be decomposed into individual utility functions which only condition on the individual actions a i . This assumption results in a space of MARL tasks where decentralized policies (i.e. greedy policies over Q i ) can be learned to collectively act optimally. The decomposed utility functions Q i can then be used for greedy action selection by individual agents [36].

It is easy to see that under the IGM assumption in Definition A.1, we cannot learn a set of individual utility functions { Q i } which can accurately represent the optimal Q-functions of many tasks with multiple global optima, including the XOR game. In fact, [5] showed formally that any algorithm under the IGM constraint cannot represent the underlying optimal Q-function in the XOR game.

B Proofs for AlberDICE

B.1 Proof of Proposition 3.1

Proposition 3.1. The closed-form solution for the inner-maximization in (12) for each x is given by

Proof. Let

Note that L ( ν i , w i ) is differentiable and strictly convex for w i . Therefore, we only need to find a point where the gradient becomes zero. For any x ∈ D ρ i ,

B.2 Convexity of L ( ν i )

Proposition B.1. Let L ( ν i ) be a function, defined as:

Then, L ( ν i ) is convex with respect to ν i

Proof. For any functions ν i , ν ′ i and constant λ ∈ [0 , 1] , we can derive the following equality:

Thus, e ν i is the linear function with respect to ν i . Furthermore, using the convexity of exp( · ) ,

Therefore, L ( ν i ) is convex function with respect to ν i .

We also show that our practical algorithm minimizes the upper bound of the original optimization problem in equation (7) restated here:

Corollary B.2. L ( ν i ) is an upper bound of L ( ν i , w ∗ i ) , where w ∗ i = arg max w i L ( ν i , w i ) , i.e., L ( ν i , w ∗ i ) ≤ L ( ν i ) always holds, where equality holds when the MDP transition and other agents' policy π -i are deterministic.

Proof. We first note that the closed-form solution for arg max w i L ( ν i , w i ) is as follows:

By plugging this into equation (7), we obtain

From Jensen's inequality, we get

Also, the inequality becomes tight when the transition model and the opponent policies are deterministic, since exp ( 1 α E a -i ∼ π -i s ′ ∼ P ( s,a i , a -i ) [ˆ e ( s, a i , a -i , s ′ )] ) = E a -i ∼ π -i s ′ ∼ P ( s,a i , a -i ) [ exp ( 1 α ˆ e ( s, a i , a -i , s ′ ) )] should always hold for the deterministic transition P and opponent policies π -i .

C Problems with Naive Extension of OptiDICE to Offline MARL

OptiDICE [14] is a (single-agent) offline policy optimization algorithm which is derived from the regularized Linear Programming (LP) formulation for RL. For a given MDP ⟨ S, A, P, R, γ ⟩ , the derivation of OptiDICE starts with the regularized dual of the LP:

Here, d ( s, a ) should be a stationary distribution of some policy π by the Bellman flow constraints (25) ( d π ( s, a ) := (1 -γ ) ∑ ∞ t =0 γ t Pr( s t = s, a t = a ; π ) ), and d D is the dataset distribution. The goal is to maximize the rewards while not deviating too much from the data distribution, following the conservatism principle in offline RL. Without the regularization term D KL ( d || d D ) , the optimal solution of (24-25) is the stationary distribution for the optimal policy, d ∗ = d π ∗ .

One of the main contributions of OptiDICE is a tractable re-formulation of the dual LP problem above into a single convex optimization problem,

Here, ν ( s ) ∈ R is the Lagrangian multiplier for the Bellman flow constraint (25), and it approaches the optimal state value function V ∗ ( s ) as α → 0 . Once we obtain the optimal solution of (26), ν ∗ , it was shown that the stationary distribution corrections of the optimal policy is given by w ∗ ν ∗ :

However, its extension to MARL can cause a number of subtle issues. While solving (26) does not suffer from the curse of dimensionality posed in MARL since ν ( s ) is a state-dependent function, ν ∗ itself is not an executable policy. We therefore should extract a policy from it. However, once we try to learn a parametric function for w ( s, a ) , we encounter a combinatorial space of joint actions. We thus should avoid learning any state-action dependent functions for policy extraction. One feasible way to do so is to perform policy extraction via Weighted Behavior-cloning (WBC):

which corresponds to behavior-cloning of the factorized policy on the state-action visits by the optimal (joint) policy. However, if the optimal joint policy (by ν ∗ ) is a multi-modal distribution, this WBC policy extraction step can result in an arbitrarily bad policy, selecting OOD joint actions. For example, consider the XOR matrix game in Figure 1 with a dataset D = { ( A,A ) , ( A,B ) , ( B,A ) } , where the optimal joint policy is given by π ∗ ( a 1 = A,a 2 = B ) = π ∗ ( a 1 = A,a 2 = B ) = 1 2 . In this situation, WBC of OptiDICE (29) obtains the factorized policies of π 1 ( a 1 = A ) = π 1 ( a 1 = B ) = 1 2 and π 2 ( a 2 = A ) = π 2 ( a 2 = B ) = 1 2 , which can select suboptimal (and OOD) joint actions:

This analysis is consistent with our experimental results in Section 5, which demonstrated the failure of OptiDICE in solving the Penalty XOR Game as well as Bridge by converging to sub-optimal OOD joint actions.

D Proofs for Section 4

To prove Theorem 4.2, we first need to show that the regularized objective in the LP formulation of AlberDICE, cf. equation (2), preserves the common reward structure of G .

Lemma 4.1. Consider a joint policy π = ( π i ) i ∈N , with factorized individual policies, i.e., π ( a | s ) = ∏ i ∈N π i ( a i | s ) for all ( s, a ) ∈ S ×A with a = ( a i ) i ∈ N . Then, the regularized objective in the LP formulation of AlberDICE, cf. equation (2), can be evaluated to

with ˜ r ( s, a i , a -i ) := r ( s, a i , a -i ) -α · log d π ( s ) π ( a | s ) d D ( s,a i , a -i ) , for all ( s, a ) ∈ S ×A . In particular, for any joint policy, π = ( π ) i ∈N , with factorized invdividual policies, the regularized objective in the LP formulation of AlberDICE attains the same value for all agents i ∈ N .

Proof. Recall that the KL-divergence, D KL ( p ( x ) ∥ q ( x )) , between probability distributions p and q is defined as D KL ( p ( x ) ∥ q ( x )) := ∑ x p ( x ) log p ( x ) q ( x ) . Thus, the D KL term in the objective of equation (2) can be written as

Since the π ′ i s are factorized by assumption, the decomposition d π i ( s, a i ) = d π ( s ) π i ( a i | s ) , implies that the numerator in the log -term of the previous expression can be written as

Substituting back in the initial expression of the objective function, we obtain that

Thus, by setting ˜ r ( s, a i , a -i ) := r ( s, a i , a -i ) -α · log d π ( s ) π ( a | s ) d D ( s,a i , a -i ) , for all ( s, a ) ∈ S × A , we obtain the claim. The equality of the last expression for all i ∈ N follows now immediately from application of the decomposition d π i ( s, a i ) π -i ( a -i | d ) = d π ( s ) π ( a | s ) , on the outer expectation for all agent i ∈ N .

Remark D.1 . In the LP formulation of AlberDICE, cf. equation (2) and (3), we used the notation d i ( s, a i ) rather than d π i ( s, a i ) , since, in this case, the variables are d i ( s, a i ) are decision variables that are not a-priori related to any particular policy π i . However, once the d i ( s, a i ) 's are fixed and translated to a policy, e.g., through the relation π i ( a i | s ) = d i ( s,a i ) ∑ a j d i ( s,a j ) that holds in tabular domains, then, we can apply Lemma 4.1. For the purposes of the alternating optimization procedure of the AlberDICE algorithm, Lemma 4.1 ensures that after each update by any agent i ∈ N , the value of the objective function is the same for each agent. To evaluate the modified utilities, ˜ r ( s, a i , a -i ) , during training, agents need to know both the action, a i , and the policy, π i ( a, s ) , from whichi this action drawn for each agent i ∈ N . Thus, this regularization terms exploits to the fullest the knowledge available to agents in the centralized training setting.

To proceed, we can define a modified game, ˜ G = ⟨ N,S, A , ˜ r, P, γ ⟩ which is the same as the original game in every respect, i.e., it has the same state space, agents, actions, transitions and discount factor, except for the rewards, r , which are replaced by the modified rewards ˜ r for any given value of the regularization parameter, α > 0 (for α = 0 , we simply have the rewards, r , of the original game, G ). Despite this modification, Lemma 4.1 implies that ˜ G still has a common reward structure. This can be used to prove that the AlberDICE algorithm has monotonic updates which eventually converge to a Nash equilibrium of the modified game, ˜ G . For this part, we focus on tabular domains, in which the policies, π i ( a i | s ) , can be directly extracted from d i ( s, a i ) as π i ( a i | s ) = d i ( s,a i ) ∑ d ( s,a ) .

Theorem 4.2. Given an MMDP, G , and a regularization parameter α ≥ 0 , consider the modified MMDP ˜ G with rewards ˜ r as defined in Lemma 4.1 and assume that each agent alternately solves the regularized LP defined in equations (2-3). Then, the sequence of policy updates, ( π t ) t ≥ 0 , converges to a Nash policy, π ∗ = ( π ∗ i ) i ∈N , of ˜ G .

Proof of Theorem 4.2. Let π 0 = ⟨ π 0 i ⟩ i ∈ N denote the initial joint policy and let π t = ⟨ π t i ⟩ i ∈ N denote the joint policy after iteration t ∈ N in an execution of the AlberDICE algorithm. For i = 1 , . . . , N , we will also write π t 1: i to denote the joint policy at time t after players 1 to i have updated their policies, i.e., π t 1: i = ( π t 1 , . . . , π t i , π t -1 i +1 , . . . , π t -1 N ) . Let d i ( s, a i ) := d π t -1 1: i -1 i ( s, a i ) denote the stationary distribution for agent i before the current optimization by player i , and let d ∗ i ( s, a i ) := d π t -1 1: i i ( s, a i ) , π ∗ i ( a i | s ) denote the stationary policy derived as the optimal solution of the LP and the corresponding extracted policy for agent i , respectively, after the optimization by agent i at time t . Then,

for each state s ∈ S , where we used Lemma 4.1 for the equality of the modified rewards among all agents in N . The inequality is strict unless agent i is already using an optimal policy, π i , against π -i . Letting ˜ V to denote the common value function, i.e., ˜ V i ≡ ˜ V for all agents i ∈ N , then, the previous inequality implies that after the update of agent i , all agents have a higher value with the current policy π t 1: i . Thus, the sequence, π t , of joint policies generated by the AlberDICE algorithm results in monotonic updates (increases) in the joint modified value function ˜ V ( s ) , s ∈ S . Since, V is bounded (rewards are bounded and discounted by assumption), this implies that also ˜ V is bounded and hence, at some point, the updates of the algorithm will reach a local maximum of V . Let π ∗ = ( π ∗ i , π ∗ -i ) denote the extracted policy at that point. Then, for all agents i ∈ N and any π i , it holds that ˜ V π ∗ i ( s ) = ˜ V π ∗ ≥ ˜ V ( π i ,π ∗ -i ) = ˜ V ( π i ,π ∗ -i ) i for all s ∈ S . Thus, π ∗ is a Nash policy as claimed.

An important property of the AlberDICE algorithm is that it maintains a sequence of factorizable joint policies, π t = ⟨ π k i ⟩ i ∈ N , for any t > 0 . Thus, after termination of the algorithm, the agents are guaranteed not only to reach an optimal state of the value function, but also an extracted joint policy that will be factorizable. This eliminates the problem of learning correlated policies during centralized training, which even if optimal, may not be useful during decentralized execution. This property of the AlberDICE is formally stated of Corollary 4.3. Its proof is immediate by the construction of the algorithm.

E Detailed Description of Practical Algorithm

E.1 Numerically Stable Optimization

The practical issue in optimizing (14) is that it is unstable due to its inclusion of exp( · ) , often causing exploding gradient problems:

where ˆ e ν i ( s, a i , a -i , s ′ ) := r ( s, a i , a -i ) -α log π -i ( a -i | s ) π D -i ( a -i | s,a i ) + γν i ( s ′ ) -ν i ( s ) . To address this issue, we use the following alternative, which can be optimized stably.

Note that the gradient ∇ x log ˆ E x [exp( h ( x ))] is given by ˆ E x [ exp( h ( x )) ˆ E x ′ [exp( h ( x ′ ))] ∇ x h ( x ) ] , which normalizes the value of exp( h ( x )) and thus it is numerically stable by preventing the exploding gradient issue. At first glance, it seems that optimizing (31) can result in a completely different solution to the solution of (14). However, as we will show in the followings, their optimal objective function values are the same and their optimal solutions only differ in a constant shift.

Proposition E.1. Let V ∗ = arg min ν i L ( ν i ) and ˜ V ∗ = arg min ˜ ν i ˜ L (˜ ν i ) be the sets of optimal solutions of (14) and (31). Then, L ( ν ∗ i ) = ˜ L (˜ ν ∗ i ) holds for any ν ∗ i ∈ V ∗ and ˜ ν ∗ i ∈ ˜ V ∗ . Also, for any ν ∗ i ∈ V and any C ∈ R , v ∗ i + C ∈ ˜ V ∗ .

We follow the proof steps in DemoDICE [9]. First, note that for any constant C , the advantage for ν i + C is:

Lemma E.2. For an arbitrary function ν i and any constant C , the following equality holds,

Proof. From the definition of ˜ L ( ν i ) ,

Lemma E.3. For any function ν i , the following inequality always holds:

Equality holds if and only if

Proof. For any y ≥ 0

holds.

and equality holds if and only if y = 1 . Thus,

and equality holds if and only if

Finally, we obtain the following results:

Lemma E.4. For any optimal solution ˜ ν ∗ i = arg min ν i ˜ L ( ν i ) , there is a constant C such that ˜ ν ∗ i + C is an optimal solution of min ν i L ( ν i ) .

Proof. Let ˜ ν ∗ i be an optimal solution of arg min ν i ˜ L ( ν i ) and

Then, ˆ ν i := ˜ ν ∗ i + C ∗ satisfies

Furthermore, ˆ ν i is also an optimal solution of min ν i ˜ L ( ν i ) by Lemma E.2 . Then, by the equality condition in Lemma E.3 ,

Thus, ˆ ν i is an optimal solution of min ν i L ( ν i ) .

Lemma E.5. An optimal solution ν ∗ i = arg min ν i L ( ν i ) is also an optimal solution of min ν i ˜ L ( ν i )

Proof. From Lemma E.3 ,

From Lemma E.4, min ν i L ( ν i ) and min ν i ˜ L ( ν i ) have the same minimum value, and thus,

The aforementioned Proposition E.1 can now be proved.

Proposition E.1. Let V ∗ = arg min ν i L ( ν i ) and ˜ V ∗ = arg min ˜ ν i ˜ L (˜ ν i ) be the sets of optimal solutions of (14) and (31). Then, L ( ν ∗ i ) = ˜ L (˜ ν ∗ i ) holds for any ν ∗ i ∈ V ∗ and ˜ ν ∗ i ∈ ˜ V ∗ . Also, for any ν ∗ i ∈ V and any C ∈ R , v ∗ i + C ∈ ˜ V ∗ .

Proof. This holds from combining Lemma E.4 and Lemma E.5

E.2 Policy Extraction

Finally, our practical AlberDICE optimizes (31) that yields ˜ ν ∗ i . However, ˜ ν ∗ i itself is not a directly executable policy, so we should extract a policy from it. To this end, first note that the optimal policy π ∗ i is encoded in w ∗ i as a form of stationary distribution correction ratios:

Then, w ∗ i ( s, a ) can be represented in terms of ˜ ν ∗ i as follows:

where e ˜ ν ∗ i ( s, a i ) = E a -i ∼ π -i ( s ) s ′ ∼ P ( s,a i , a -i ) [ ˆ e ˜ ν ∗ i ( s, a i , a -i , s ′ ) ] and ˆ e ˜ ν ∗ i ( s, a i , a -i , s ′ ) = r ( s, a i , a -i ) -

α log π -i ( a -i | s ) π D -i ( a -i | s,a i ) + γ ˜ ν ∗ i ( s ′ ) -˜ ν ∗ i ( s ) . Finally, we extract a policy from w ∗ i via I-projection policy extraction method introduced in equation (15).

where C 1 and C 2 denote some constants. π D i ( a i | s ) is a data policy for i -th agent, which is pretrained by maximizing the log-likelihood:

Eq. (41) can be understood as KL-regularized policy optimization, where we aim to maximize e ˜ ν ∗ i ( s, a i ) (analogous to critic value) while not deviating too much from the data policy, whose trade-off is controlled by the hyperparameter α . Finally, to enable e ˜ ν ∗ i to be evaluated at every action a i , we train an additional parametric function e i (implemented as an MLP that takes ( s, a i ) as an input and outputs a scalar value) by minimizing the mean squared error with a conservative regularization term R ( e i ) introduced in CQL [11] to penalize OOD action values:

where R CQL ( e i ) := α CQL ˆ E ( s,a i ) ∼ D ρ i [ log ∑ a i exp ( e i ( s, a i ) ) -E a i ∼ π i D [ e i ( s, a i )] ] . We used α CQL = 0 . 1 for all experiments.

E.3 Pseudocode of AlberDICE

To sum up, AlberDICE computes the best response of agent i by optimizing ν i , which corresponds to obtaining a stationary distribution correction ratios of the optimal policy. Then, we extract a policy by training e -network and performing I-projection as described in Section E.2.

We assume π D i , π D -i , ν i , e i , and π i are parameterized by β i , β -i , θ i , ψ i , and ϕ i , respectively 10 . Then, we optimize the parameters via stochastic gradient descent (SGD). The entire loss functions to optimize the parameters are summarized in the following:

̸

The pseudocode of AlberDICE is presented in Algorithm 1.

Algorithm 1 AlberDICE

Input: A dataset D := { ( s, a , r, s ′ ) k } | D | k =1 , a set of initial states D 0 := { s 0 ,k } | D 0 | k =1 , data policy networks { ( π D β i , π D β -i ) } N i =1 with parameters { ( β i , β -i ) } N i =1 , ν -networks { ν θ i } N i =1 with parameters { θ i } N i =1 , e -networks { e ψ i } N i =1 with parameters { ψ i } N i =1 , policy networks { π ϕ i } N i =1 with parameters { ϕ i } N i =1 , and a learning rate η

1: Pretrain (auto-regressive) data policies {( π D β i ( a i | s ) , π D β -i ( a -i | s, a i ) )} N i =1 by minimizing (4445).

2: for each iteration until convergence do

3: for each agent i ∈ N do

4: Sample mini-batches from s 0 ∼ D 0 and x ∼ D .

̸

5: Compute the importance ratio ρ i ( x ) = ∏ j = i π ϕ j ( a j | s ) π D β -i ( a -i | s,a i ) for each sample x . (Eq. (11))

6: Perform resampling with probability proportional to ρ i ( x ) , which constitutes the resampled dataset D ρ i .

7: Perform SGD updates using D 0 and D ρ i :

8: end for

9: end for

Output:

Factorized policies { π ϕ i ( a i | s ) } N i =1

F Dataset Details

F.1 Bridge

The optimal dataset (500 trajectories) was constructed by a hand-crafted (multi-modal) optimal policy which randomizes between Agent 1 crossing the bridge first while Agent 2 retreats, and vice-versa. The mix dataset is a mixture between 500 trajectories from the optimal dataset and 500 trajectories generated by a uniform random policy.

F.2 Multi-Robot Warehouse (RWARE)

For the data collection policy used to construct the dataset, we train Multi-Agent Transformers (MAT) [37] which takes an autoregressive policy structure and thus is able to generate diverse behavior. We further train MAT over 3 random seeds, and generate a expert dataset with a mixture of diverse behaviors.

F.3 Google Research Football

Similar to the dataset collection procedure in RWARE, we use MAT to generate a medium-expert dataset in order to ensure that agents score goals in different ways. Similar to Bridge, we construct a dataset of 2000 trajectories where 1000 trajectories have medium performance (roughly 60% performance of the expert policies) and another 1000 from fully trained "expert" MAT policies.

F.4 SMAC

We use the public dataset provided by [31].

G Matrix Game Results

In Table 6, we show results for the converged policies in the Matrix Game presented in Section 5 of the main text and shown again in Figure 6.

A

B

A

0

1

B

1

-

2

As expected, all algorithms converge to the optimal action AB for the dataset D ( a ) . However, AlberDICE is the only algorithm which can choose the optimal action deterministically for all 4 datasets, showing robustness even when the environment has multiple global optima and the dataset is generated by a mixture of diverse policies. Here we can consider any dataset containing both AB and BA as a mixture of diverse data collection policies where the two agents cooperate to select the optimal actions but in different ways. For OMAR, we see that it is able to learn the optimal policy for D ( b ) . However, as discussed in the Introduction of the main text, OMAR degenerates to the sub-optimal action BB for dataset D ( c ) because each agent acts independently and assumes the other agent chooses the individual action A with a 2 3 probability. Finally, we also note that BC can fail and converge to OOD joint actions even in D ( b ) where the dataset is optimal.

BC

A

B

A 0.00

1.00

B 0.00

0.00

D

(

a

)

A

B

0.25

0.24

0.26

0.25

D

(

b

)

A

B

0.45

0.22

0.22

0.11

D

(

c

)

A

B

0.25

0.26

0.24

0.25

D

(

d

)

ICQ

A

B

A 0.00

1.00

B 0.00

0.00

D

(

a

)

A

B

1.00

0.00

0.00

0.00

D

(

b

)

A

B

0.00

0.00

0.00

1.00

D

(

c

)

A

B

1.00

0.00

0.00

0.00

D

(

d

)

OMAR

A

B

A 0.00

1.00

B 0.00

0.00

D

(

a

)

A

B

0.00

0.00

1.00

0.00

D

(

b

)

A

B

0.00

0.00

0.00

1.00

D

(

c

)

A

B

1.00

0.00

0.00

0.00

D

(

d

)

MADTKD

A

B

A 0.00

1.00

B 0.00

0.00

D

(

a

)

A

B

0.25

0.24

0.26

0.25

D

(

b

)

A

B

0.25

0.25

0.25

0.25

D

(

c

)

A

B

0.25

0.26

0.24

0.25

D

(

d

)

OptiDICE

A

B

A 0.00

1.00

B 0.00

0.00

D

(

a

)

A

B

0.25

0.24

0.26

0.25

D

(

b

)

A

B

0.25

0.25

0.25

0.25

D

(

c

)

A

B

0.25

0.26

0.24

0.25

D

(

d

)

AlberDICE

A

B

A 0.00

1.00

B 0.00

0.00

D

(

a

)

A

B

0.00

0.00

1.00

0.00

D

(

b

)

A

B

0.00

1.00

0.00

0.00

D

(

c

)

A

B

0.00

1.00

0.00

0.00

D

(

d

)

H Bridge Policy Visualizations

The visualizations for learned policies of AlberDICE, OptiDICE, ICQ, OMAR, BC, and MADTKD are shown for all state possibilities. From Figure 7, it is clear that AlberDICE is the only algorithm which reliably chooses a deterministic action (Left, Left) at the initial state. However, since the visualizations are provided for the optimal dataset which has a small coverage of states, the policy values may still be sub-optimal at OOD states. The results for OMAR in Figure 10 shows that the agents are acting independently without regard for the other agents, and converges to (Left, Right) at the initial state which results in a collision. Finally, we note that the visualizations for MADTKD may not exactly correspond to the policy values used in the experimental results in Section 5. This is because MADTKD uses a history-dependent Transformer policy and it is not clear how to visualize the policy values depending on the history. The visualizations shown in Figure 11 assume that each agent is at the given state in the initial timestep, which is different from encountering that state after many timesteps. Nonetheless, the results for the true initial state shown in the central portions of the figure are consistent with the quantitative results in Section 5.

I Additional Ablation Results

Here we show ablation results for hyperparameter α which controls the degree of conservatism. Table 7 shows that AlberDICE is not too sensitive to the hyperparameter α as long as it is within a reasonable range.

J Implementation Details

J.1 AlberDICE for Dec-POMDP

While our paper focuses on MMDPs, our experimental results show that we can extend AlberDICE to Dec-POMDPs G = ⟨N , S , A , r, P, p 0 , γ, Ω , O ⟩ by using partial observations and a history-dependent policy in place of a state-dependent policy for each agent. In Dec-POMDP, each agent i observes individual observations o i ∈ Ω which is given by the observation function O : S × A → Ω . Each agent makes decision based on the observation-action history τ i ∈ (Ω × A ) t -1 × Ω , where each agent's (decentralized) policy is represented as π i ( a i | τ i ) .

For the Matrix Game, Bridge and GRF, we used an MLP policy since the partial observations for each agent correspond with the global state (i.e., each individual policy is conditioned only on its current observation o i , rather than the entire history). For Warehouse, each individual policy uses the partial observations as input, which is the 3x3 neighborhood surrounding each agent. We utilize a Transformer-based policy in order for the agent to condition on the history of local observations and actions, while speeding up training in comparison to Recurrent Neural Networks (RNNs). This same Transformer-based policy is used for all baselines as well.

During centralized training, AlberDICE uses the global state s for training π D β -i ( a -i | s, a i ) , ν θ i ( s ) , and e ψ i ( s, a ) for all environments, where its training procedure is identical to the MMDP training procedure described in Appendix E.3. Only the policy extraction step is different for Dec-POMDP, where each agent's history-dependent policies are trained by:

J.2 Hyperparameter Details

We conduct minimal hyperparameter tuning for all algorithms for fair comparisons. It is also worth noting that in offline RL, it is important to develop algorithms which require minimal hyperparameter tuning [23]. We chose the best values for α for both AlberDICE and OptiDICE between [0 . 1 , 1 , 5 , 10] on N=2 (Tiny) for RWARE and RPS (2 agents) for GRF. The best values were then used for all scenarios thereafter. The final values used were α = 0 . 1 (GRF) and α = 1 (RWARE) for AlberDICE and α = 1 (GRF, RWARE) for OptiDICE. We found that the performance gaps between different hyperparameters were minimal as long as they were within a reasonable range where training is numerically stable.

For ICQ [40], we found that the algorithm tends to become numerically unstable after a certain number of training epochs even with sufficient hyperparameter tuning due to the exploding Q values, especially in the GRF and SMAC environment.

K Computational Resources

For Warehouse experiments, we utilized a single NVIDIA Geforce RTX 3090 graphics processing unit (GPU). The experiments for running AlberDICE took 5H, 14H, and 29H for 2, 4, 6 agent environments, respectively.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nthe paper presents a novel approach, AlberDICE, an offline MARL algorithm that performs centralized training of individual agents based on stationary distribution optimization, to tackle the distribution shift challenge in offline MARL. The proposed method avoids OOD joint action selection by computing the best response of one agent at a time, thereby circumventing the exponential complexity of MARL. The authors provide theoretical proof of convergence to Nash policies and demonstrate through experiments that AlberDICE outperforms baseline algorithms on standard MARL benchmarks.\n\nSTRENGTHS:\n1.\tIt proposes a novel solution to address the distribution shift problem in offline reinforcement learning by performing centralized training of individual agents and optimizing the stationary distribution. This approach effectively mitigates the issue of excessive distribution shift in offline multi-agent reinforcement learning.\n2.\tThe authors avoid the exponential complexity challenge in multi-agent reinforcement learning by computing the best response of individual agents in a sequential manner, reducing the computational burden.\n3.\tThe paper provides theoretical proof of convergence to Nash policies through the alternating optimization process, offering a theoretical guarantee for the effectiveness of the algorithm.\n4.\tExperimental results demonstrate that AlberDICE outperforms baseline algorithms significantly on standard multi-agent reinforcement learning benchmarks, validating the performance and effectiveness of the algorithm.\n\nWEAKNESSES:\n1.\tThis paper considers the other agents’ policies are fixed and optimized based on the distribution correction technique. However, what if the data set contains multiple cooperation strategies?\n2.\tEven though there are many derivations in Section 3, many are derived from existing work [1-3]. Section 3 should be simplified to highlight the contribution of this work.\n3.\tThis works adopt BC(Behavioral Cloning) to learn \\pi^D_{-i}. When the quality of the data set is not high, such as the medium data set, the effect of BC is very poor. Will this affect the performance of the algorithm?\n4.\tThere is little discussion on the limitations and applicability of the algorithm, and there is a lack of in-depth exploration of special cases or problems, which may restrict the generalization and application of the algorithm.\n5.\tNecessary ablation experiments are required, such as the robustness of parameters, the selection of network models, etc.\n\nQUESTIONS:\nPlease refer to the weakness.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new algorithm called AlberDICE for addressing the challenge of out-of-distribution joint actions in offline Multi-Agent Reinforcement Learning (MARL). The proposed algorithm performs centralized training of individual agents based on stationary distribution optimization, which circumvents the exponential complexity of MARL by computing the best response of one agent at a time while effectively avoiding OOD joint action selection. The authors show that the alternating optimization procedure converges to Nash policies theoretically and demonstrate that AlberDICE significantly outperforms baseline algorithms on a standard suite of MARL benchmarks in experiments.\n\nSTRENGTHS:\n- Nice writing flow for solving the original LP problem. The paper provides a clear and concise explanation of the proposed approach and its implementation.\n- AlberDICE is shown to produce much stronger policies than baselines in experiments conducted in many multi-agent environments.\n\nWEAKNESSES:\n- Since $\\pi_{-i}$ is fixed, the process in Section 3 is quite similar to that of OptiDICE, including Lagrangian formation, the closed-form solution, and I-projection used in policy extraction. Also, the distribution shift/OOD problem is solved by the series of DICE. It weakens the contributions of this paper.\n- The key point for AlberDICE is Theorem 4.2 which NE can be achieved. However, it is theoretically unclear to me why after the optimization, the inequality between lines 591-592 in the appendix holds. Let's consider an extreme case in the Rock Paper Scissors game. If the dataset only contains draw cases, how does AlberDICE reach NE? Maybe Theorem 4.2 lacks the necessary assumptions.\n\nminors/typos\n- \"+\" before the second line of eq.5\n- In line 188, \"pretrained.\" -> \"pretrained,\"\n- \"=\" before eq.16 -> \"$min_{\\pi_i}$\"\n\nQUESTIONS:\n- $x$ in eq.10-14 is sample $(s, a, s')$, $e_v$ is also a function of $(s,a)$. How eq.14 is state-dependent instead of state-action-dependent?\n\nLIMITATIONS:\nThe paper does not explicitly mention any limitations of the proposed method. The paper has no potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

comments: SUMMARY:
In this paper, the authors propose an alternative multi-agent offline reinforcement learning algorithm, where each agent is optimized for the best response policy and avoiding the out-of- distribution joint actions. The theoretical part proves the nash policies convergence after the presented training paradigm. The empirical results show promissing results on several well- known environments over existing offline MARL algorithms.

STRENGTHS:
● The paper is well-written and easy to follow.

● The theoretical results show the Nash policy convergence which integrates the trackable theory, relaxing the conventional IGM regularization.

● The joint-action problem is novelly addressed in the offline MARL algorithm.

WEAKNESSES:
● The solution of the XOR game shown in the introduction for IGM-based and decentralized methods may not converge to (B,B) from the reviewer's perspective. These two categories of methods will connect the agents' policies by the mixing network for example, which is not
 totally individually be executed.

● The claim from the authors for circumventing the curse of dimensionality in MARL is somewhat overclaimed. The curse of dimensionality also includes the explosion in the joint policy space due to the increasing agent number.

QUESTIONS:
See the weakness part.

LIMITATIONS:
In the practical scenarios, the observation and action space may have some noise that is not computable and may obstacle to the presented method.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors consider the offline multi-agent reinforcement learning setting. They provide an example to demonstrate that decentralisation can lead to out of distribution actions at test time. They derive AlberDICE, an offline RL method based on computing alternating best responses to the other agents' policies. Starting from the linear program for an MMDP, they learn a policy by:\n\n1. Pretraining a data policy $\\pi^D_{-i}(a_{-i} | s, a_i)$\n2. Using the current policy and current data policy to resample the dataset\n3. Solve a minimisation problem to find $\\nu^*$ \n4. Extracting the policy from $\\nu^*$ using I-projection. \n\nThis method prevents out of distribution actions by optimising each agent alternately and assuming the other agents' policies are fixed. A KL penalty is used to ensure that the resulting learned policy stationary distribution stays faithful to the dataset's distribution. By alternately optimising each agent, issues of producing out-of-distribution actions due to simultaneous updates do not arise.\n\nSTRENGTHS:\nOverall I enjoyed reading this paper. In particular:\n* The paper evaluates their algorithm on a wide range of domains and performs strongly across most tasks.\n* The authors explain the problem setting clearly and point out some problems with naive approaches to the problem.\n* The paper is generally well-written and clearly presented.\n\nWEAKNESSES:\nWhile the paper is good overall, there are some weaknesses of the work.\n* It would have been nice to see an explicit demonstration that the proposed method reduces out-of-distribution actions in the more complex environments and therefore that the mechanism proposed is the reason for the better performance.\n* The evaluations mostly focus on a small number of agents ($\\leq 10$). \n* Although the authors evaluate on a wide range of domains, they only evaluate on a small number of SMAC maps. This is a well-documented problem with MARL evaluation [1].\n* The discussion in lines 125--131 doesn't seem to make clear the presence of the KL penalty term in (2). For example, $\\pi^*$ is only the optimal policy *for the MMDP with a penalty as the reward*. Adding a few words in key places clarifying this point would make this section easier to read -- I was a little confused on first reading.\n\n\n\n[1] Gorsane, Rihab, et al. \"Towards a standardised performance evaluation protocol for cooperative marl.\" Advances in Neural Information Processing Systems 35 (2022): 5510-5521.\n\nQUESTIONS:\nI have a few clarifying questions. \n\n- How does the method scale to larger numbers of agents? Is the relatively small number of units in each scenario because of computational constraints running this method for more units? \n\n- In equation (4), the $\\pi_{-i}(a'_{-i} | s')$ from equation (3) seems to have disappeared. It reappears later in equation (5) so I think this is just a typo. Is this correct or have I missed a detail here?\n\nLIMITATIONS:\nThe author discuss the limitations briefly in the appendix. The discussion there seems fine to me but of course more discussion of limitations is always appreciated.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-20"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces the AlberDICE framework that targets to address the out-of-distribution challenge (or distributional shift) in off-policy MARL. AlberDICE addresses this challenge by the alternating optimization procedure along with the regularization term. Also, AlberDICE addresses the curse of dimensionality in multi-agent settings with the autoregressive policy architecture. Empirical evaluations show that AlberDICE can find a more optimal solution than baselines in multiple multi-agent benchmark domains.\n\nSTRENGTHS:\n1. Theoretical contribution about showing how AlberDICE converges to a Nash equilibrium. \n2. Evaluations are performed across multiple domains, showing the empirical evidence of AlberDICE.\n\nWEAKNESSES:\n1. A possible concern about alternatively finding the best response policy while fixing others is that this approach effectively solves a static single-agent problem from each agent's perspective. As a result, in some cooperative settings, this method may learn some sub-optimal joint collaborative policy. \n2. Related to #1, the approach of alternatively finding the best response may not work or be effective in competitive/mixed-sum game settings, limiting the scope of the proposed method.\n\nQUESTIONS:\nIn addition to my concerns in the weaknesses section, I would like to ask the following questions:\n\n1. I appreciate the intuitive example (Figure 1) in the Introduction. If a centralized critic is employed with a decentralized actor, would the sub-optimality problem still happen?\n2. Section 3 states that \"when each agent is optimized alternatively, the objective function will always be monotonically improved and convergence is guaranteed.\" Would there be a proof or reference to this statement?\n3. Related to #2, while the monotonic improvement is guaranteed, would convergence to an optimal (Nash) equilibrium be guaranteed?\n4. According to Levine et al., 2020, the main challenge of off-policy learning is the distributional shift: a policy is trained under one distribution and it will be evaluated on a different distribution. In the evaluation section, it is unclear whether the learned cooperative policies are evaluated under a different distribution/environment/setting compared to the training dataset.\n5. Because the authors focus on cooperative settings, wouldn't it be better to target to converge to a correlated equilibrium instead of a Nash equilibrium as correlated equilibria are a superset of Nash equilibria (i.e., possible to converge to a more optimal solution). \n\nReference:\nLevine et al., Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems, 2020\n\nLIMITATIONS:\nYes, the authors have adequately addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-26"}