Offline Decentralized Multi-Agent Reinforcement Learning
Authors: Jiechuan Jiang
Zongqing Lu
Venue: ICLR 2022 Submitted (2022)
Keywords: multi-agent reinforcement learning

Abstract: In many real-world multi-agent cooperative tasks, due to high cost and risk, agents cannot continuously interact with the environment and collect experiences during learning, but have to learn from offline datasets. However, the transition probabilities calculated from the dataset can be much different from the transition probabilities induced by the learned policies of other agents, creating large errors in value estimates. Moreover, the experience distributions of agents' datasets may vary wildly due to diverse behavior policies, causing large difference in value estimates between agents. Consequently, agents will learn uncoordinated suboptimal policies. In this paper, we propose MABCQ, which exploits value deviation and transition normalization to modify the transition probabilities. Value deviation optimistically increases the transition probabilities of high-value next states, and transition normalization normalizes the biased transition probabilities of next states. They together encourage agents to discover potential optimal and coordinated policies. Mathematically, we prove the convergence of Q-learning under the non-stationary transition probabilities after modification. Empirically, we show that MABCQ greatly outperforms baselines and reduces the difference in value estimates between agents. 

Under review as a conference paper at ICLR 2022
OFFLINE DECENTRALIZED MULTI-AGENT FORCEMENT LEARNING
REIN-
Anonymous authors Paper under double-blind review
ABSTRACT
In many multi-agent cooperative tasks, due to high cost and risk, agents cannot continuously interact with the environment and collect experiences during learning, but have to learn from offline datasets. However, the transition probabil- ities calculated from the dataset can be much different from the transition prob- abilities induced by the learned policies of other agents, creating large errors in value estimates. Moreover, the experience distributions of agents' datasets may vary wildly due diverse behavior policies, causing large difference in value estimates between agents. Consequently, agents will learn uncoordinated subop- timal policies. In this paper, propose MABCQ. which exploits value deviation and transition normalization to modify the transition probabilities. Value devia- tion optimistically increases the transition probabilities of high-value next states, and transition normalization normalizes the biased transition probabilities of next states. They together encourage agents to discover potential optimal and coordi- nated policies. Mathematically. we prove the convergence of Q-learning under the non-stationary transition probabilities after modification. Empirically, we show that MABCQ greatly outperforms baselines and reduces the difference in value estimates between agents.
INTRODUCTION
In multi-agent cooperative tasks, agents learn from the experiences generated by continuously in- teracting with the environment to maximize the cumulative shared reward. Recently, multi-agent reinforcement learning (MARL. has been applied to real-world cooperative systems (Bhalla al., 2020; Xu et al., 2021). However, in many industrial applications, continuously interacting with the environment and collecting the experiences during learning is costly, risky, and time-consuming. One way to address this is offline RL, where the agent only access fixed dataset of experiences and learn the policy without further interacting with the environment. However, in multi-agent envi- ronments, the dataset of each agent is often pre-collected individually by different behavior policies, which are not necessary to be expert, and each dataset contains the individual action of the agent instead of the joint action of all agents, e.g., autonomous driving dataset. Therefore, the dataset does not satisfy the paradigm of centralized training, and the agent has to learn the coordinated policy in an offline and fully decentralized way.
The main challenge of offline RL is the extrapolation error, an error in value estimate incurred by the mismatch between the experience distributions of the learned policy and the dataset (Fujimoto al., 2019), e.g., the distance of the learned action distribution to the behavior action distribution, and the bias of the transition dynamies estimated from the dataset the true transition dynamics. Recently, almost all offline RL methods (Fujimoto et al., 2019; Levine et al., 2020; Jaques et al., 2019) focus on constraining the learned policy to close the behavior policy to avoid the overestimate of the values of out-of-distributio actions, but ignore to correct the transition bias since the deviation of estimated transition dynamics would not be too large the single-agent environment stationary.
However, in decentralized multi-agent environments, from the perspective of each agent, other agents are part of the environment, and the transition dynamics experienced by each agent de- pend the policies of other agents. Even in stationary environment, the experienced transition dynamics of each agent will change as other agents update their policies (Foerster et al., 2017). Since the behavior policies of other agents would be inconsistent with their learned policies which

---

Under review as a conference paper at ICLR 2022
unknowable in decentralized multi-agent environments, the transition dynamics estimated from the dataset by each agent would be different from the transition dynamics induced by the learned policies of other agents, causing large errors in value estimates. The extrapolation error would lead to suboptimal policies. Moreover, trained on different distributions of experiences collected by various behavior policies, the estimated values of the same state might be much different between agents, which causes that the learned policies cannot coordinate with each other.
To overcome the suboptimum and miscoordination caused by transition bias in decentralized learn- ing, we introduce value deviation and Iransition normalization to deliberately modify the estimated transition probabilities from the dataset. During data collection, if one agent takes an optimal action while other agents take suboptimal actions at a state, the transition probabilities of low-value next states will become large. Thus, the Q-value of the optimal action will be underestimated, and the agents will fall into suboptimum. Since the other agents are also trained, the learned policies of other agents would become better than the behavior policies. For each agent, the transition probabilities of high-value next states induced by the learned policies would be larger than those estimated from the dataset. Therefore, we let each agent be optimistic toward other agents and multiply the tran- sition probabilities by the deviation of the value of next state from the expected value over all next states, to make the estimated transition probabilities close to the transition probabilities induced by the learned policies of other agents.
Value deviation could decrease the extrapolation error and help the agents escape from the subopti- mum. However, in some cases, the behavior policies of other agents might be highly deterministic, which makes the distribution of experiences unbalanced. If the transition probabilities of high-value next states are extremely low, value deviation may not remedy the underestimate. Moreover, due to the diversity in experience distributions of agents, the value of the same state might be overes- timated by some agents while underestimated by others, which results in miscoordination of the leameed policies. To address these two problems, we normalize the transition probability estimated from the dataset to be uniform. Transition normalization balances the extremely biased distribu- tion of experiences and builds the consensus about value estimate. By combining value deviation and transition normalization, the agents would learn perforrfffrminggggg and coordinated policies in an offline and fully decentralized way.
Although value deviation and transition normalization make transition dynamics non-stationary, we mathematically prove the convergence of Q-learnin under such non-stationary transition dynamics. By importance sampling, value deviation and transition normalization take effect only as the weights of the objective function, which makes our method easy to implement. We instantiate the proposed method on BCQ, termed MABCQ, to additionally avoid -iitr--distribution actions. Nevertheless, other offline RL methods. We build the offline datasets and evaluate MABCQ in four multi-agent mujoco scenarios (Todorov et al., 2012). Experimental results show that MABCQ greatly outperforms BCQ, and ablation studies demonstrate the effectiveness of value deviation and transition normalization. To the best of our knowledge, MABCQ is the first method for offline and fully decentralized multi-agent reinforcement learning.
it can also be implemented
2 RELATED WORK
MARL. Many MARL methods have been proposed for learning to solve cooperative tasks in an online manner. Some methods (Lowe et al., 2017; Foerster et al., 2018; Iqbal & Sha, 2019) extend policy gradient into multi-agent cases. Value factorization methods (Sunehag et al., 2018: Rashid et al., 2018; Son et al., 2019) decompose the joint value function into individual value functions. Communication methods (Das et al.. 2019; Ding et al., 2020) share information between agents for better cooperation. All these methods follow centralized learning and decentralized execution, where the agents could access the information from other agents during centralized training. However, in our offline and decentralized setting, the datasets of agents different; each dataset contains individual actions instead of joint actions; and the agents cannot be trained in a centralized way.
For decentralized learning, the key challenge is the obsolete experiences in replay buffer. Finger- prints (Foerster et al., 2017) deals with obsolete experience problems by conditioning the value function on fingerprint that disambiguates the age of the sampled data. Lenient-DQN (Palmer et al., 2018) extends the leniency concept and introduces optimism in the value function update by forgiving suboptimal actions. Concurrent experience replay (Omidshafiei al., 2017) induces cor-
2

---

Under review as a conference paper at ICLR 2022
relations in local policy updates, making agents tend to converge to the same equilibrium. However, these methods require additional information, e.g. training iteration number, exploration rate, and timestamp, which often not provided by the offline dataset.
Offinne RL. Offline RL requires the agent to learn from a fixed batch of data {(s, a, s', r)}, con- sisting of single-step transitions without exploration. Unlike imitation learning, offline RL does not assume that the offline data is provided by a high-performing expert but has to handle the data generated by suboptimal or multi-modal behavior policies. Most offline RL methods consider the out-of-distributi action (Levine et al., 2020) as the fundamental challenge, which is the main cause of the extrapolation error (Fujimoto et al., 2019) in value estimate in the single-agent environ- ment. To minimize the extrapolation error, some recent methods introduce constraints to enforce the learned policy to close to the behavior policy, which could direct action constraint (Fujimoto et al., 2019), kernel MMD (Kumar et al., 2019), Wasserstein distance (Wu et al., 2019), and KI. divergence (Peng et al., 2019). Some methods train Q-function pessimistic to out-of-distribution actions to avoid overestimation by adding a reward penalty quantified by the learned environment model (Yu et al., 2020), by minimizing the Q-values of aut-of-distributiin actions (Kumar et al., 2020; Yu et al., 2021), or by weighting the update of Q-function via Monte Carlo dropout (Wu al., 2021). A finite-sample analysis for offline MARL (Zhang et al., 2018) has been studied, but the agents are assumed get individual rewards instead of the shared reward, and be connected by communication networks, which is not the fully decentralized setting. All these methods do not consider the extrapolation error introduced by the transition bias, which fatal problem in offline and decentralized MARL.
METHOD
3.1 PRELIMINARIES
We consider N agents in multi-agent MDP (Oliehoek & Amato, 2016) Mene =< S,A, R, ? > with the state space S and the joint action space A. At each timestep, each agent i gets state and performs an individual action a, and the environment transitions to the next state s' by taking the joint action ā with the transition probability Penv (s'|s, ā). The agents would get shared reward
R (s). which is simplified to just depending on state (Schulman et al., 2015). The agents learn to maximize the expected return E ΣT 7r,, where 7 is a discount factor and T is the time horizon of the episode. However, in the fully decentralized learning, Mene is partially observable to the agent since the agent cannot observe the joint action ã. During execution, from the perspective of each agent there is a viewed MDP ME. =< Ai R, PE, Y with the individual action space A, and the transition probability
PE. (s'1s,a.) Peno (s' σ) II n. (a,ls))
jfi
where a_, denotes the joint action of all agents except agent i, and т denotes the policy of the agent. As the transition probability depends on the policies of other agents, if other agents are also updating their policies, PE. becomes non-stationary. Moreover, if the agent cannot interact with the environment, PE, is unknown. Since only investigate the influence of other agents' policies PF,, we assume Penv to be deterministic.
In offline and decentralized settings, each agent i could only access fixed offline dataset B, which is pre-collected by behavior policies and contains the tuples (s, ai, r, 8'). As defined in BCQ (Fu- jimoto et al., 2019), the visible MDP MB. =< S,A,, R, PB,, ? constructed on B,, which has the transition probability
Ps. (s'1s, = num(s, aj s')
Σx num(s, a,, 3')
where num(s, d; 3') is the number of times the tuple (s, a;, 5') is observed in B, However, since the learned policies of other agents might be greatly different from the behavior policies, Ps. would be biased from PE., which creates large extrapolation errors and differences in value estimates between agents, and eventually leads uncoordinated suboptimal policies.
'The transition prohability in the following sections means the one calculated from B; unless otherwise stated.
3

---

Under review as a conference paper at ICLR 2022
Table 1: The matrix game.
Agent 2
(0.4)
(0.6)
Q1 (0.8)
5
Agent 1
02 (0.2)
Table 2: Transition probabilitics and expected returns calculated in the datasct.
action
transition
expected return
action
transition
expected return
p(1[a1) 0.4
p(1[az) 0.8
Agent I
3.4
Agent 2
2.0
p(5/01) = 0.6
p(6]a;) = 0.2
p(6)a2) = 0.4
p(5(az) a2) = 0.8
02
3.0
02
p(1|a2) 0.6
p( (2//22 0.2
To intuitively illustrate the miscoordination caused by the transition bias, devise offline datasets in a matrix game for two agents, with the payoff depicted in Table 1. The action distributions of the behavior polices of the two agents are [0.8, 0.2] and [0.4, 0.6). respectively. Table shows the transition probabilities and expected returns calculated by the independent agents in the datasets. Since the datasets are collected by poor behavior policies, when one agent chooses the optimal action, the other agent would choose the suboptimal action with high probability, which leads to low transition probabilities of high-value next states. Thus, the agents underestimate the optimal actions and converge to the suboptimal policies (41, a2), rather than the optimal policies (a2, a1).
3.2
IMPORTANCE WEIGHTS
3.2.
VALUE DEVIATION
If the behavior policies of some agents are low-performing during data collection, they usually take suboptimal actions to cooperate with the optimal actions of other agents, which leads high transition probabilities of low-value next states. When agent performs Q-learning with the dataset B, the Bellman operator T approximated by the transition probability Ps. (s (6'18, a:) to estimate the expectation over s':
TQi(s, a.) = Eg (a'ls,a)) r + Y max Qi (8', o.))
If Ps, of high-value lower than PE., the Q-value of this (s, a.) pair underestimated, which would cause large extrapolation error and guide the agent to the convergence of suboptimal policy.
As discussed before, during execution, the transition probability viewed by agent is Pg. and the en- vironment is deterministic, thus Pg. (s'ls, a;) only depends on the learned policies of other agents, which are unavailable. However, since the policies of other agents are also updating towards max- imizing the Q-values, Pr, of high-value next states would grow higher than Ps.- Based on this intuition, we let each agent optimistic towards other agents and modify Ps, as
PB, (s'1s,a,) (1+ (x) R. (3
1
(2('
gnd
value deviation
where the state value (s) = maxa, Q;(s,a;), + Vi'G''- is the deviation of the (."V"
value of next state from the expected value all next states, which increases the transition probabilities of the high-value next states and decreases those of the low-value next states, and 2dd Σ,, Ps; (5'1s,
V,,"), is normalization term to make sure the [.''
of the transition probabilities is one. Value deviation makes the transition probability to be close to Pg. and hence decreases the extrapolation enror. The optimism towards other agents helps the agents escape from local optima and discover potential optimal actions which are hidden by the poor behavior policies.

---

Under review as a conference paper at ICLR 2022
3.2.2 TRANSITION NORMALIZATION
In real-world applications, the action distribution of behavior policy might be unbalanced, which makes the transition probabilities Pt. biased, egg., the transition probabilities of Agent (i.e. action distribution of Agent 1) in Table the transition probability of a high-valus next state is extremely low, value deviation cannot correct the underestimate. Moreover, since B; of each agent is individu- ally collected by different behavior policies, the diversity in transition probabilities of agents leads to that the value of the same state will be overestimated by some agents, while be underestimated by others. Since the agents are trained to reach high-value states, the large divergences on state values will cause miscoordination of the learned policies. To overcome these problems, we normalize the biased transition probability PB, to be uniform over next states,
PB, (8'(8,0)) az) PB. (s' 8, O4
fn
transition normalization
where 2 is normalization term that is the number of different s' given (s, ( in B, Transition normalization enforces that each agent has the same PB. when acts the learned action oz on the same state s, and we have the following proposition.
Proposition 1. In episodic environments, if each agent performs Q-learning on Bi all agents will converge to the same V if they huve the same transition probubility on any state where each agent i acts the learned action o/.
Proof. The proof is provided in Appendix A.
However, to satisfy PB, (s'|s,aj) a)) Ps, =... PBx (3's8,0%) for all s' E S, the agents should have the same set of s' at (s,a"), which is a strong assumption. In practice, although the assumption not strictly satisfied, transition normalization could still normalize the biased transition distribution, encouraging the estimated state value V to be close each other.
3.2.3 OPTIMIZATION OBJECTIVE
We combine value deviation (1 V((d")-E which is denoted Aud; and transition VVV;
malization which is denoted as Xen, and modify Ps, as,
PB. (8'18, Ps, (8)18, Aod.
where 2, = E1+ V=( is the normalization term. a sense, P'S, makes the offline learning B, similar to the online decentralized MARL. In the initial stage, Aed, is close to I since Qi(s, a;) is not updated, and the transition probabilities are uniform, meaning other agents are act- ing randomly. During training, the transition probabilities of high-value states gradually grow under value deviation, which is an analogy of that other agents are improving their policies in the online learning. Starting from the normalized transition probabilities and changing the transition proba- bilities following the same optimism principle, the agents increase the values of potential optimal actions optimistically and unanimously, and build consensuses about value estimate. Therefore tran- sition normalization and value deviation encourage the agents to learn nigh-performing policies and improve coordination. Moreover, although Ps. is non-stationary (i.e., Xed, changes over updates of Q-value), have the following theorem about the convergence of Bellman operator T under Ps..
[
TQi(s, a.) = E. (1/18,4)) r Y max Qi (s',
d,
Theorem 1. Under the non-stationary transition probability Ps., the Bellman operator T is contraction and converges to a unique fixed point when y 2rmax Tiin if the reward is bounded by the positive region Tmin: Tmax].
Tuin
Proof. The proof is provided in Appendix A.
5

---

Under review as a conference paper at ICLR 2022
As positive affine transformation of the reward function does not change the optimal policy in the environments with fixed horizon (Zhang et al., 2021), Theorem holds in general. We could rescale the reward to make "min arbitrarily close to mmax as obtain a high upper bound of y.
In deep reinforcement learning, directly modifying the transition probability is infeasible. However, we could modify the sampling probability to achieve the same effect. The optimization objective of decentralized deep Q-learning Eps, (s.a,.s') (Q(s, a;) yil² is calculated by sampling the batch from B3 according to the sampling probability PB, (s, a4, s'). By factorizing PB, (s, s'), we have
PB, (8, as, 5 = PB, (s, az) Ps. (s's,,))
sampling probability
transition probubility
Therefore, we can modify the transition probability as with 2,. Then, the sampling probability can be re-written as
Arn, A Pis, (s'1s, ai)
and scale Ps. (s, a,)
Xtny Aldd (s, az, 5') 2,P8, (s, az) den, Aud: PB, (s'1s,
24
modifiidd sampling probability
modified transition probability
Since 2; is independent from s', it could be regarded as scale factor on pB, (s, a.). Scaling Ps, (s, a.) will not change the expected target value y/, so sampling batches for update accord- ing to the modified sampling probability could achieve the same effect of modifying the transition probability. Using importance sampling, the modified optimization objective is
Atn, Aed, DB, (s, Ci 8')
Exem, Pe, (s.as, [Qis, az) - yıl² = Ep, (s.ar, PE, (s, 0j 8¹)
Qi(s, - yil
Epe, (s.as.s Atn, Aed, [q (s. a,) - vil?
where Aen; and Aed, could be seen as the weights of the objective function.
3.3 IMPLEMENTATION
We implement our method on BCQ (Fujimoto et al., 2019), termed MABCQ. To make it adapt to high-dimensional continuous spaces, for each agent i, we train a Q-network Qu a perturbation net- work ç,, and conditional VAE G = {E: (n* , s, D (als, 2 ~ ( }. In execution, each agent generates actions by G adds small perturbations [-0, +) on the actions using E., and then selects the action with the highest value Q, The policy can be written as
(*) = argmax Qi of + (s,))))) where {at ( G: [s)) of+a:(e.a?)
Q: is updated by minimizing
Epa, (e.0.. den, Awd, (2,( - yil2, where yı - - (b()))
(I)
y is calculated by the target networks Q, and &. where मt is correspondingly the policy induced by Q and & is updated by maximizing
Eps, Aem, Aod, Qi (8, + & (s,a:))
(2)
To estimate Avd,. we need (y') = at and VV (s')l = -(),(s,))) - r), which can be estimated from the sample without actually going through all s'. We estimate Xud; using the target networks to stabilize Avd, along with the updates of Q2 and &,. To avoid extreme values, we clip Avd, to the region [1 e, + e), where is the optimism level.
To estimate Atn.. train VAE G² {E? (n² a, s') DD (als, 22
(p²,o²)) Since the latent variable of VAE follows the Gaussian distribution, we use the mean as the encoding of the input and estimate the probability density functions: pi(s, a) PN(o.)((tt) and pi(s, a,s')

---

Under review as a conference paper at ICLR 2022
Algorithm MABCQ
1: for E N do
2: Initialize the conditional VAEs:
G = { or,, DJ (als,=")) - {Ey D2 (als,
3: Initialize Q-network Q. perturbation network E,, and their target networks Q, and &..
4: Fit the VAEs G₁ and C using B,
5:
mnar_update do Sample mini- from B,
6:
7: Update Qi by minimizing (1).
8: Update by maximizing (2).
9: Update the target networks Q, and é.
10: end for
end for
(NT( where PNT(,.)) is the density of unit Gaussian distribution. The conditional density is -gés
()) ~ (x and the transition probability is PB, (5'|s, az) -2- pa(s' (((s'ss,add s,a)ds' Ro
(x
as ||ōş|| when the integral interval ||ós|| is small constant. Approximately, have
(N(0)))(4)) (NT(
Atn;
and the constant ||ó is considered in 2, In practice, we find that Amn, falls into the region [0.2, 1.4] for almost all samples. For completeness, we summarize the training procedure of MABCQ in Algorithm I.
EXPERIMENTS
4.1
MATRIX GAME
We perform MABCQ on the matrix game Table shown in Table 3, we only use And without considering transition normalization, as the transition probabilities of high-value next states have been increased, for agent the value of ag becomes higher than that of d1. However, due to the unbalanced action distribution of agent 1, the initial transition probabilities of agent are extremely biased. With Ayd. agent still underestimates the value of 01 and learns the action ag- The agents arrive the joint action (ag,az), which is worse solution than the initial one (Table 2). Normalizing the biased distribution by Aen the agents could learn the optimal solution (ag, a1) and build the consensus about the values of learned actions, shown in Table 4.
Table 3: Transition probabilitics and expected returns calculated in the datasct using only Aud-
action
transition
expected return
action
transition
expected return
p(1[a1) 0.12
p(1(a1) ) = 0.4
@1
4.52
4
Agent
p(5(a1) 0.88
Agent
p(6(a1) 0.6
p(6]az) 0.8
(K5)a2) 0.95
02
02
4.8
p(1]az) 0.2
p(1/a2) 0.05
Table 4: Transition probabilities and expected returns calculated in the dataset using Atn and Apd-
action
transition
expected return
action
transition
expected return
p(1|a)) 0.17
p(1/a1) = 0.14
@1
4.33
01
5,29
Agent
p(5(a1) 0.83
Agent
p(6)a1) = 0.86
p(6]a2) 0.86
p(5)a2) 0.83
02
5.29
U2
4.33
p(1/a2) 0.14
p(1[a2) 0.17

---

Under review as a conference paper at ICLR 2022
(a) HalfCheetah (b) Walker
(c) Hopper
(d) Ant
Figure I: Illustrations of scenarios. Different colors indicate different agents.
HABIO
DCD
DCG
(a) HalfCheetah
(b) Walker
(c) Hopper
(d) Ant
Figure 2: Learning curves of MABCQ and the baselines in HalfCheetah, Walker, Hopper, and Ant. The curves are plotted based the mean and standard deviation of five runs with difference random seeds.
4.2 ENVIRONMENTS AND DATASETS
To evaluate the effectiveness MABCQ in high-dimensional complex environments, we adopt multi-agent mujoco (de Witt et al., 2020), which splits the original action space of the mujoco tasks (Todorov et al., 2012: Brockman et al., 2016) into several sub-spaces. We consider four tasks, which HalfCheetah, Walker, Hopper, and Ant. As illustrated in Figure I, different colors indicate different agents. Each agent independently controls one or some joints of the robot and could get the state and reward of the robot, which are defined in the original tasks.
For each environment, we collect datasets for the N agents. Each dataset contains 1 million transitions (s, at, r, done). For data collection, train intermediate policy and expert policy for each agent using SAC algorithm (Haarnoja et al., 2018) provided by OpenAl Spinning Up (Achiam, 2018). The offline dataset B, is mixture of four parts: 20% transitions split from the experiences generated by the SAC agent at the early training. 35% transitions are generated from that the agent the intermediate policy while other agents the expert policies, 35% transitions are generated from that agent performs the expert policy while other agents act the intermediate policies, 10% transitions are generated from that all agents perform the expert policies. For the last three parts, we add small noise to the policies to increase the diversity of the dataset.
We compare MABCQ against the following methods:
MABCQ w/o Åtn Removing Am from MABCQ.
MABCQ w/o Aed- Removing Avd from MABCQ.
BCQ. Removing both Åtn and And from MABCQ.
DDPG (Lillicrap al., 2016). Each agent is trained using independent DDPG the offline B, without action constraint and transition probability modification.
Behavior. Each agent i takes the action generated from the VAE G..
The baselines have the same neural network architectures and hyperparameters as MABCQ. All the models are trained five runs with different random seeds. All the learning curves are plotted using mean and standard deviation. More details about hyperparameters are available in Appendix C.
4.3 PERFORMANCE AND ABLATION
Figure 2 shows the learning curves of all the methods in the four tasks. Without action constraint and transition probability modification, DDPG severely suffers from the large extrapolation error and hardly improve the performance throughout the training. BCQ outperforms the behavior policies
8

---

Under review as a conference paper at ICLR 2022
MADCO
MAOCO
MADGG
1
Upt
(a) HalfCheetah
(b) Walker
(c) Hopper
(d) Ant
Figure 3: Difference in value estimates among agents along with the training HalfChectah, Walker, Hopper,
and Ant. It is shown that transition normalization indeed reduces the difference value estimates.
Table Extrapolation emrors of MABCQ and BCQ.
HalfChectah
Walker
Hopper
MABCQ
98.4 31.3
55.0 +
28.1 3.4
180.2 22.2
BCQ
97.2 1 29.1
91.5 + 35.4
65.8 1 6.4
231.3 1 47
but only arrives at the mediocre performance. In Figure 2(a) and Figure 2(c), the performance of BCQ even descends in the later stage learning. During the collection of B, when agent takes "good" action, other agents usually take "bad" actions, making BCQ underestimate the "good" actions, especially in the latter stage. The learning curves of MABCQ w/o X are similar to those of BCQ in the first three tasks. That is because other agents' policies assumed to be random using only transition normalization, which is far from the learned policies and leads to large extrapolation errors. But in Ant, MABCQ w/o Avd outperforms BCQ the later stage, which is attributed the value consensus built by the normalized transition probabilities. By optimistically increasing the transition probabilities of high-value next states, MABCQ w/o Aen encourages the agents learn potential optimal actions and obviously boosts the performance. MABCQ combines the advantages of both value deviation and transition normalization and outperforms other baselines.
To interpret the effectiveness of transition normalization, we uniformly sample subset from the union of agents' states and calculate the difference in value estimates, max; V.* min, V.", on this subset, where V is calculated as (s (T)))) The results are illustrated in Figure 3. The max, V minn V of MABCQ is lower than that of MABCQ w/o A which verifies that transition normalization could decrease the difference in value estimates among agents. If there is consensus among agents about which states are high-value, the agents would select the actions that most likely lead to the common high-value states. This promotes the coordination of policies and helps MABCQ outperform MABCQ w/o Atn-
In Table 5, we present the extrapolation errors of MABCQ and BCQ, 1 Σ, Qi(s, R), where R is the true value evaluated by Monte Carlo return. Although MABCQ greatly outperforms BCQ (i.e., higher return), it still achieves smaller extrapolation errors than BCQ in Walker, Hopper, and Ant, which empirically verifies our claim that the proposed weights could decrease the extrapolation error.
N
We also investigate the effect of optimism level the computation efficiency of MABCQ, and the performance of the proposed weights on other datasets and another offline RL algorithm, i.e., CQL (Kumar et al., 2020). Due to the space limit, these results are given in Appendix B.
5 CONCLUSION
In this paper, we proposed MABCQ for offline and fully decentralized multi-agent reinforcement learning. MABCQ modifies the transition probability by value deviation that increases the transition probabilities of high-value next states, and by transition normalization that normalizes the biased transition probabilities. Mathematically, we show that under the ntn-stationary transition probability after modification, offline decentralized Q-learning converges to unique fixed point. Empirically, we show that MABCQ could help the agents escape from the suboptinum, learn coordinated poli- cies, and greatly outperform the baselines in variety of multi-agent offline datasets.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Joshua Achiam. Spinning Up in Deep Reinforcement Learning. 2018.
Sushrut Bhalla, Sriram Ganapathi Subramanian, and Mark Crowley, Deep multi agent reinforcement learning for autonomous driving. In Canadian Conference on Artificial Intelligence (Canadian AI), 2020.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym, 2016.
Abhishek Das, Théophile Gervet, Joshua Romoff, Dhruv Batra, Devi Parikh, Mike Rabbat, and Joelle Pineau. Tarmac: Targeted multi-agent communication. In International Conference on Machine Learning (ICML), 2019.
Christian Schroeder de Witt, Bei Peng, -lere-Alexandre Kamienny, Philip Torr, Wendelin Böhmer, and Shimon Whiteson. Deep multi-agent reinforcement learning for decentralized continuous cooperative control. arXiv preprint arXiv:2003.06700 06600, 2020.
Ziluo Ding, Tiejun Huang, and Zongqing Lu. Learning individually inferred communication for multi-agent cooperation. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
Jakob Foerster, Nantas Nardelli, Gregory Farquhar, Triantafyllos Afouras, Philip HS Torr, Pushmeet Kohli, and Shimon Whiteson. Stabilising experience replay for deep multi-agent reinforcement learning. In International Conference Machine Learning (ICML), 2017.
Jakob N Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon White- son. Counterfactual multi-agent policy gradients. In AAAI Conference on Artificial Intelligence (AAAI), 2018
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International Conference on Machine Learning (ICML). 2019.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International Confer- ence on Machine Learning (ICML), 2018.
Shariq Iqbal and Fei Sha. Actor-attention-ccittc for multi-agent reinforcement learning. In Interna- tional Conference on Machine Learning (ICML), 2019.
Natasha Jaques, Asma Ghandeharioun, Judy Hanwen Shen, Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Gu, and Rosalind Picard. Way off-policy batch deep reinforcement learning of implicit human preferences in dialog. arXiv preprint arXiv: arXiv:1907.004533 2019.
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off- off-policy q- learning via bootstrapping error reduction. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. arXiv preprint arXiv:2006.0477 2020.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tuto- rial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643, 2020.
Timothy P. Lillicrap. Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In Inter- national Conference on Learning Representations (ICLR), 2016.
Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, OpenAl Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cosperative-ooppttiin environments. In Advances in Neural Information Processing Systems (NeurIPS). 2017.
10

---

Under review as a conference paper at ICLR 2022
Frans A Oliehoek and Christopher Amato, A concise introduction to decentralized POMDPs. Springer, 2016.
Shayegan Omidshafici, Jason Pazis, Christopher Amato, Jonathan How, and John Vian. Deep decentralized multi-task multi-agent reinforcement learning under partial observability. In Inter- national Conference on Machine Learning (ICML), 2017.
Gregory Palmer, Karl Tuyls, Daan Bloembergen, and Rahul Savani. Lenient multi-agen deep rein- forcement learning. In International Conference on Autonomous Agents and MultiAgent Systems (AAMAS), 2018.
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advintage-weiighted regression: Simple and scalable off-policy reinforcement learning. arXiv preprint 1rX00019100.001777 2019.
Tabish Rashid, Mikayel Samvelyan, Christian Schroeder De Witt, Gregory Farquhar, Jakob Foer- ster, and Shimon Whiteson. Qmix: monotonic value function factorisation for deep multi-agent reinforcement learning. In International Conference on Machine Learning (ICML), 2018.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International Conference on Machine Learning (ICMI), 2015.
Kyunghwan Son, Daewoo Kim, Wan Ju Kang. David Earl Hostallero, and Yung Yi. Qtran: Learning to factorize with transformation for cooperative multi-agent reinforcement learning. In Interna- fional Conference on Machine Learning (ICML), 2019.
Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojeiech Marian Czarnecki, Vinicius Zambaldi, Max Jaderberg, Mare Lanctot, Nicolas Sonnerat, Joel Leibo, Karl Tuyls, et al. doooposecoooooitionn networks for cooperative multi-agent learning based on team reward. In International Conference on Autonomous Agents and Multiagent Systems (AAMAS). 2018.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), 2012.
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. 2019.
arXiv preprint
Yue Wu, Shuangfei Zhai, Nitish Srivastava, Joshua Susskind, Jian Zhang, Ruslan Salakhutdinov, and Hanlin Goh. Uncertainty weighted actor-critic for offline reinforcement learning. arXiv preprint arXiv:2105.088 2021.
Bingyu Xu, Yaowei Wang, Zhaozhi Wang, Huizhu Jia, and Zongqing Lu. Hierarchically and operatively learning traffic signal control. In AAAI Conference on Artificial Intelligence (AAAI), 2021.
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-bases based offline policy optimization. Advances in Neural Information Processing Systems (NeurIPS), 2020.
Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn. Combo: Conservative offline model-based policy optimization. arXiv preprint 1rXiv:2102.08366 2021.
Chi Zhang, Sanmukh Rao Kuppannagari, and Viktor Prasanna. Brac+: Going deeper with behavior regularized offline reinforcement learning. 2021.
Kaiqing Zhang, Zhuoran Yang, Han Liu, Tong Zhang, and Tamer Başar. Finite-sample analysis for decentralized batch multi-agent reinforcement learning with networked agents. arXiv preprint arXiv:1812.02783, 2018.
11

---

Under review as a conference paper at ICLR 2022
A PROOFS
In episodic environments, each agent i performs Q-learning on B all agents will converge to the same V* if they have the same transition probability on any state where each agent i acts the learned action az.
Proposition 1.
Proof. Considering the two-agent case, we define ((s) as the difference in the V*.
(s) (i) (s)
-Po Po. (s)s..))))++++ - EPs Poz (s)1
Σ (slls,a) (r (6') VV (6' - VV22 (5)) Σ (s)ss,)))) (r (5))
- (Ps, - Ps. (5'1s, 02)) (r VV² (s') + y PB. (s')
For the terminal state Send, we have ((send) = 0. If PB, (s'(s,at) = Ps. (5'(s, As' E S, recursively expanding the term, we arrive at 6(s) = 0 + 70 70 + ... +0 O. We can easily show that it also holds in the N-agent case.
Theorem 1.
Under the non-stationary transition probability PB.. the Bellman operator TT is contraction and converges to a unique fixed point when y < by the positive region T'in: !'min:Tmax! Tmax).
Drmaxe if the reward is bounded
Tinin
Proof. We initialize the Q-value to be Mjminn where denotes 1-y T+1 1-7 Since the reward is bounded by the positive region [rimin, mmax], the Q-value under the operator T is bounded to [yrmin mmmal- Based on the definition of Ps. (s'Is, a;), it can be written as Ee (a))
(x') where
(5') maxa, Qi (8', âz). Then, we have the following.
[TQ 1Q
Σ Pb. (s) r + + O (s' a.)
Pb. (s'1s,a,) + yama Of (s'.a.) s'ES
Σ
-max
5,9,
s'ES
Eves(Vit (8/)) Eves(V. (8/))
m
Eves Vi'l(s)) (s') Eves Vi")(")
8,09
Eves(Vi' (s'))-
())2-((1'" -(V." (s)))
Σ(v; (5))2
1
max ?
Eves Vi'(s')
Eves Vi*² EvesVi''s
ses
Eves(V.' (6) - -V, (s))) (8) Vr
Σ(v)' (s')) EvES Vi (5') Vi-('')
max
8,0,
Eves Vi"" (s')
EvV. EoEE (s))
YES
Eves (s') max (V: + vi(((')) (5')) EveS(V.' (8'))² (s)))
[vv (6)) (=)()'))
max 7
Eves Vi*((")
S'ES
< S>S1||[Q
Q
(n(2 max Timn)
|
2mmax
=7(
- 1) - 22
Tmin
Eves(V: (19) Eves Vi (y'(ss) (s))
The third term of the penultimate line is because: V:' (V;)+;;
(vE)(V;((0))² SV (s))+1 (5) (Ve (s')) MMmin Eves Vi'(
V (d))+V1² (8) ()))
< 21yr mi
V (s)
12

---

Under review as a conference paper at ICLR 2022
else,
(V; Eves (s')
[ves(V;"ss)) (s')) *apmmax
S 1p maax
Eves Vi
Since 21m
Minn 2 Mymx: have
(ves(V; (s'))² Eves (s"
IV (8) + V (s))
1 2n max min-
Therefore, if 27mas Tmin converges to unique fixed point.
the operator T is contraction. By contraction mapping theorem, T
B ADDITIONAL RESULTS
To clearly present the performance gain of MABCQ over BCQ, we summarize the mean value and standard deviation after x 10* updates in Table where we can see that MABCQ performs more than one standard deviation better than BCQ.
Table Comparison between MABCQ and BCQ.
HalfChectah
Walker
Hopper
Ant
MABCQ
1905 + 126
2498 + 256
1383 + 443
3027 + 178
BCQ
1384 360
1324 + 662
566 + 89
2573 + 637
To evaluate MABCQ in low-quality dataset, we carried out an additional experiment on 2-agent Swimmer with another data collection approach, where each dataset only contains the experiences at the early training and the experiences generated by intermediate policies. As shown in Figure 4, after removing the expert experiences, MABCQ greatly outperforms BCQ, and BCQ even cannot outperform the behavior policies.
MANECO
BOO
DOPP
Figure Learning of MABCQ and the baselines in Swimmer.
The optimism level e controls the strength of value deviation. If e is too small, value deviation has weak effects on the objective function. But if too large, the agent will be over optimistic about other agents' learned policies, which could result in large extrapolation errors and uncoordinated policies. Figure shows the learning curves of MABCQ with different e. is commonly observed that increasing elevates the performance, especially in HalfCheetah. However, in Walker and Ant,
(a) HalfCheetah
(b) Walker
(c) Hopper
(d) Ant
Figure 5: Learning curves of MABCQ with different in HalfCheetah. Walker, Hopper, and Ant. is shown that with any positive e, MABCQ does not underperform that without value deviation.
13

---

Under review as a conference paper at ICLR 2022
Table Average time taken by update.
HalfCheetah
Walker
Hopper
Ant
MABCQ BCQ
18
16 ms
20
10 ms
10 ms
9 ms
11 ms
we set large (0.8), performance slightly drops due to overoptimism. Even so, with any positive MABCQ does not underperform that without value deviation.
To demonstrate the computation efficiency of our method, we record the average time taken by one update in Table 7. The experiments carried out Intel i7-8700 CPU and NVIDIA GTX 1080Ti GPU. Since And and Atn could be calculated from the sampled experience without actually going through all next states, MABCQ additionally needs only two forward passes for computing A and Aen in the update and costs less than twice the training time of BCQ. Moreover, since MABCQ is fully decentralized, the learning of agents can be entirely parallelized. Therefore, MABCQ scales well with the number of agents.
Dowrt
1257
2010
2017
2050
1310
(a) HalfCheetah
(b) Hopper
(c) Walker
Figure Learning curves MACQL D4RL medium-replay datasets.
The two proposed weights Aed and Atn could also be extended other offline RL methods. We in- troduce the two weights to CQL (Kumar et al., 2020), as MACQL, and test them in D4RL medium- replay datasets (Fu al., 2020). CQL augments the standard Bellman error objective with servative regularizer, thus the learned Q-value may not represent the original meaning and thus not theoretically match value deviation. However, MACQL still outperforms CQL, as shown in Figure 6.
C EXPERIMENTAL SETTINGS AND HYPERPARAMETERS
The experimental settings and hyperparameters are summarized in Table 8.
Table Experimental settings and hyperparameters
Hyperparameter
HalfCheetah
Walker
Hopper
Ant
agent number (N)
2
2
3
4
state space
17
17
11
27
action space
3
2
horizon (I')
1000
discount (?)
0.99
B; size
10
batch size
1024
MI.P units ML.P activation
(64, 64)
ReLU
learning rate of Q learning rate of g learning rate of G
10
10-4
10-4
0.80
0.48
0.80
0.64
&
0.05
10
VAE hidden space
10
14

---

Under review as a conference paper at ICLR 2022
D LIMITATION AND FUTURE WORK
Although we consider the setting where each agent can get the state the environment, MABCQ could also be potentially applied partially observable environments. However, if the partial ob- servability is too limited, Q-value may not be accurately estimated from the observation. Many MARL methods adopt recurrent neural networks utilize the historical information, which how- ever is impractical if the timestamp is not included in the offline dataset. Moreover, if state is estimated as high-value state by an agent but not included in the datasets of other agents. the other agents cannot learn corresponding optimal actions to cooperate with that agent at that state, and thus miscoordination may occur. In this case, each agent should conservatively estimate the values of states that are absent from the datasets of other agents to avoid miscoordination. We leave the observation limitation and the state absence to future work.
15


Reviews:
[{"review_id": "svGa2EWpdn-", "reviewer": "Reviewer_N2SH", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "It is not clear to me when the setting described by the authors occurs in practical cases. I think that the paper lacks a clear applicative example in which the framework proposed by the authors occurs.\n\nThe assumption that the transition probabilities are deterministic for the environment seems to be a bit restrictive. Again, if the author could provide a practical example, it would help to motivate the setting described by the authors.\n\nIn the first formula of page 5, the first two terms cancel out. I suggest to provide a more direct formula for this probability, e.g., P_B_i(s'|s,a_i) = 1/|S|. Moreover, the presentation of these modifications of the probabilities should be more formally defined with a proper symbol for each one of them.\n\nProposition 1 is misleading. If I understood correctly the theorem holds if one applies Q-learning over the previously modified transitions. It should be mentioned in the theorem statement.\n\nThe title of Section 3.3 (Implementation) is misleading. First, I would have mentioned which are the characteristics and the assumptions required for a generic methodology to be compliant with your proposed approach. Second, this seems a specific case of the application of your method rather than its implementation.\n\nI think that the experiments, even if they present some difference in the expected values of the agents' objective, do not provide any statistical significance of the fact that the proposed method is somehow providing an improvement over the existing ones (other than DDPG).\n\nIt would also be interesting to see the performance of fully centralized learning, to understand how much we are losing from the fact that we are learning in a decentralised manner.", "summary_of_the_paper": "The authors propose a scheme to modify the transition probabilities in the learning process in a fully decentralised MARL setting. The authors show that the method is still converging to the optimal solution. They also provide some experiments on synthetic environments.", "summary_of_the_review": "The topic is interesting, but the setting lacks a strong applicative example and the formalism of the paper should be improved before it is worthed for publication. Moreover, I think that the statements sometimes are hard to understand. Therefore, I suggest the author should try their best to improve the readability of the paper. Moreover, the experiments should provide some statistical significance that the proposed method is better than the state of the art.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "qYntxq7bNY", "reviewer": "Reviewer_Hcfz", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n1. The problem studied in this paper is very interesting, with a clear motivating example emphasizing that in the decentralized offline MARL setting, the difference between transition probabilities in the dataset and other agents can be large and harm the performance. Overall, the paper is well-written.\n2. The theoretical analysis shows that the proposed operator is able to converge.\n\nCons:\n1. There is a recent paper about offline MARL, MA-ICQ (Yang et al., 2021) which should be compared or at least discussed in the paper.\n2. The assumption in Theorem that gamma<r_min/(2r_max-r_min) seems too strong in the case that r_max>>r_min. \n3. The experimental evaluations are not sufficient as it only compares MABCQ with baselines on a mixed dataset. How does MABCQ perform on other types of datasets?\n\n", "summary_of_the_paper": "The paper studies an offline MARL setting, with a focus on the decentralized case, and discovers that the difference between transitions in the dataset and other agents can be very large. Authors propose two techniques, value deviation and transition normalization, to modify the transition probabilities. Experimental results validate its performance improvement over BCQ on multi-agent MuJoCo tasks.\n", "summary_of_the_review": "This paper studies an interesting offline decentralized MARL setting. However, relevant works are not cited and discussed, and experimental evaluation can be improved.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "apHmidbdfjJ", "reviewer": "Reviewer_zmVr", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I appreciate the ideas proposed by the authors. However, a lot of descriptions in the submission are very vague and the statements are mathematically imprecise. I also have some concerns regarding the generality of the proposed framework. Please find my detailed comments below.\n\n(1) A lot of sentences are very vague and imprecise (without introducing the framework in a rigorous) way. To name a few examples:\n\n(1-A) It's stated in the abstract that \"However, the transition probabilities calculated from the dataset can be much different from the transition probabilities induced by the learned policies of other agents...\" This really depends on the information observability of the agents. If an individual agent could observe the states and actions of all agents in the system, then this won't be an issue.\n\n(1-B) In the first paragraph, the authors keep switching between \"agents\" and \"agent\" and sometimes mentioned \"offline RL\", it is difficult to understand if you are always talking about the multi-agent setting or moving to talk about the single-agent setting at some point.\n\n(1-C) The authors start with \"The main challenge of offline RL is the extrapolation error\" in the second paragraph, are you referring to the single-agent case, or multi-agent case, or both? Later in the paragraph, the authors mentioned \"... ignore to correct the transition bias\" without proper explanation of what you mean by transition bias.\n\n(1-D) The authors mentioned multiple times on \"decentralized multi-agent environment\" which is confusing to me. Based on my understanding of the paper, the proposed learning procedure is decentralized (choice of the agent or the coordinator) but this has nothing to do with the environment.\n\n(1-E) The second paragraph on page 2 does not make sense to me before talking about what is observable and what is not abservable to each individual agent.\n\n(1-F) I don't understand the sentence \"If the transition probabilities of high-value next states are extremely low...\"\n\n(2) Set up in Section 3.1:\n\n(2-A) The problem description is not mathematically rigorous. The authors should specify the dimension or the space.\n\n(2-B) Why the reward only depends on the state (not the action)?\n\n(2-C) It reads to me that the centralized training and decentralized execution scheme could still be applied to this setting. In this case, the bias issue mentioned in the motivation should be solved.\n\n\n", "summary_of_the_paper": "This paper proposes a method named MABCQ to utilize offline data in the MARL environment via (1) value deviation and (2) transition normalization. These two techniques are used to correct the bias in the individual observation of the transition probability and improve the discovery of optimal policy. The authors showed the convergence of the Q-learning under the non-stationary transition probabilities after modification.\n\n", "summary_of_the_review": "A lot of descriptions in the submission are very vague and the statements are mathematically imprecise. I do not think the submission is ready for acceptance with its current presentation.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Q-zAbFMoHNA", "reviewer": "Reviewer_WD4d", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: The problem the paper studies is important and interesting. \n\nQuality:\n1.Value deviation aims to increase the transition probabilities of the high-value next states, but it will increase the value estimate of the current state and action. It can worsen the performance in bad datasets, which is discussed in the CQL paper.\n2.The evaluation of MABCQ is not sufficient, the dataset seems to be \"replay\" in RL domain. How does  MABCQ perform in other datasets: Random, Medium, Expert?\n3.The interpretation of the effectiveness of transition normalization in Figure 3 is not solid. The difference between the maximum value and minimum value can not quantify the coordination of agents, it would be better to show a real example.\n\nSoundness:\n1. A recent MARL offline RL paper \"Believe What You See: Implicit Constraint Approach\nfor Offline Multi-Agent Reinforcement Learning\" is not discussed and compared.\n2. The weakness of the proposed heuristic method is not discussed, is there any other method that can improve the coordination of agents?\n", "summary_of_the_paper": "This paper studies a MARL offline setting, where each agent is trained independently. value deviation and transition normalization techniques are proposed to modify the transition probabilities. Experimental results on four benchmarks show MABCQ's performance improvement over BCQ.", "summary_of_the_review": "This paper studies a novel problem in MARL area. However, the proposed value deviation method is not well motivated, the experimental evaluation is only done in replay datasets, and other marl offline paper are not cited and discussed. Thus I recommend the rejection.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies the offline multi-agent RL problem. The finding is that the dataset collected by one agent could be very different for other agents. The authors provide two solutions to this problem. Although being interesting, the reviewers found that the there are many imprecise math statements, and some of the methods are not well motivated. Hence, the overall recommendation is a reject.
Decision: Reject