Plan Better Amid Conservatism: Offline Multi-Agent Reinforcement Learning with Actor Rectification
Authors: Ling Pan
Longbo Huang
Tengyu Ma
Huazhe Xu
Venue: ICLR 2022 Submitted (2022)
Keywords: Multi-Agent Reinforcement Learning (MARL)
Offline reinforcement learning (RL)
Offline Multi-Agent Reinforcement Learning

Abstract: The idea of conservatism has led to significant progress in offline reinforcement learning (RL) where an agent learns from pre-collected datasets. However, it is still an open question to resolve offline RL in the more practical multi-agent setting as many real-world scenarios involve interaction among multiple agents. Given the recent success of transferring online RL algorithms to the multi-agent setting, one may expect that offline RL algorithms will also transfer to multi-agent settings directly. Surprisingly, when conservatism-based algorithms are applied to the multi-agent setting, the performance degrades significantly with an increasing number of agents. Towards mitigating the degradation, we identify that a key issue that the landscape of the value function can be non-concave and policy gradient improvements are prone to local optima. Multiple agents exacerbate the problem since the suboptimal policy by any agent could lead to uncoordinated global failure. Following this intuition, we propose a simple yet effective method, \underline{O}ffline \underline{M}ulti-Agent RL with \underline{A}ctor \underline{R}ectification (OMAR), to tackle this critical challenge via an effective combination of first-order policy gradient and zeroth-order optimization methods for the actor to better optimize the conservative value function. Despite the simplicity, OMAR significantly outperforms strong baselines with state-of-the-art performance in multi-agent continuous control benchmarks.

Under review as a conference paper at ICLR 2022
PLAN BETTER AMID CONSERVATISM: OFFLINE MULTI-AGENT REINFORCEMENT LEARNING WITH ACTOR RECTIFICATION
Anonymous authors Paper under double-bling blind review
ABSTRACT
The idea of conservatism has led to significant progress in offline reinforcement learning (RL) where an agent learns from pre-collected datasets. However, it is still an open question to resolve offline RL in the more practical multi-agent setting as many real-world scenarios involve interaction among multiple agents, Given the recent success of transferring online RL algorithms to the multi-agent setting, one may expect that offline RL algorithms will also transfer to multi-agent settings directly. Surprisingly, when conservatism- -assdd algorithms the multi-agent setting, the performance degrades significantly with an increasing number of agents. Towards mitigating the degradation, that the landscape of the value function can be non-concave and policy gradient improvements are prone local optima. Multiple agents exacerbate the problem since the suboptimal policy by any agent could lead to uncoordinated global fail- ure. Following this intuition, we propose a simple yet effective method, Offline Multi-Agent RL with Actor Rectification (OMAR), to tackle this critical chal- lenge via an effective combination of first-order policy gradient and zeroth-order optimization methods for the actor to better optimize the conservative value fune- tion. Despite the simplicity, OMAR significantly outperforms strong baselines with state-of-the-ar performance in multi-agent continuous control benchmarks.
applied to
identify that key issue
INTRODUCTION
Offline reinforcement learning (RL) has shown great potential in advancing the deployment of RL in real-world tasks where interaction with the environment prohibitive, costly, or risky (Thomas) 2015). Since agent has learn from given pre-collected dataset in offline RL, it becomes chal-
lenging for regular online RL algorithms such as DDPG (Lillicrap et al.) 2016) and TD3 (Fujimoto al. 2018 due extrapolation error (Lee et al. 2021).
There has been recent progress in tackling the problem based on conservatism. Behavior regulariza- tion (Wu et al. 2019 Kumar al., 2019), e.g., TD3 with Behavior Cloning (TD3+BC) (Fujimoto & Gu 2021). compels the learning policy to stay close to the manifold of the datasets. Yet, its performance highly depends on the quality of the dataset. Another line of research investigates incorporating conservatism into the value function by critic regularization (Nachum et al., 2019
Kostrikov et al. (02T), egg., Conservative Q-Learning (Kumar et al. 2020), which usually learns conservative estimate of the value function to directly address the extrapolation eтror.
However, many practical scenarios involve multiple agents, e.g., multi-robot control (Amato 2018), autonomous driving (Pomerleau) 1989 Sadigh et al. (016).. Therefore, offline multi-agent rein- forcement learning (MARL.) Yang et al. 2021 Jiang Lu (02T) crucial for solving real-world tasks. Observing recent success of Independent PPO (de Witt et al. 2020) and Multi-Agent PPO (Yu ct al. 2021), both of which are based on the PPO (Schulman et al. 2017) algorithm, we find that on- line RL algorithms can be transferred to multi-agent scenarios through either decentralized training or a centralized value function without bells and whistles. Hence, we naturally expect that offline RL algorithms would also transfer easily when applied to multi-agent tasks.
the-hh art conservatism based CQL (Ku- Surprisingly, we observe that the performance of the state-of oftte-of-the-ee
mar al. 2020 algorithm in offline RI degrades dramatically with increasing number of agents

---

Under review as a conference paper at ICLR 2022
shown in Figure ()) in experiments. Towards mitigating the degradation, identify critical issue in CQL: solely regularizing the critic is insufficient for multiple agents to learn good policies for coordination in the offline setting. The primary cause is that first-order policy gradi- ent methods are prone to local optima (Nachum et al. 2016; Ge et al. 2017 Safran & Shamir) et al. 2019; Sun al. 2020p. or noisy gradient esti- mates (Such et al., 2017). As a result, this can lead to uncoordinated suboptimal learning behavior because the actor cannot leverage the global information in the critic well. The issue is exacerbated more in the multi-agent settings due to the exponentially-sizen joint action space (Yang et al.) 2021) as well as the nature of the setting that requires each of the agent to learn a good policy for suc- cessful joint policy. For example, in a basketball game, where there are two competing teams each consisting of five players. When one of the players passes the ball among them, is important for all teammates to perform their duties well in their roles to win the game. As result, if one of the agents in the team fails to learn a good policy, it can fail to cooperate with other agents for coordinated behaviors and lose the ball.
2017), saddle points Vlatakis-Gkaragkknn
In this paper, we propose a surprisingly simple yet effective method for offline multi-agent contin- uous control, Offline MARL with Actor Rectification (OMAR), better leverage the conservative value function via an effective combination of first-order policy gradient and zeroth-order optimiza- tion methods. Towards this goal, we add regularizer to the actor loss, which encourages the actor to mimic actions from the zeroth-order optimizer that maximizes Q-values so that we can combine the best of both first-order policy gradient and zeroth-order optimization. The sampling mechanism is motivated by evolution strategies (Such et al. 2017: Conti et al. 2017 Mania et al. (0188), which recently emerged as another paradigm for solving sequential decision making tasks (Salimans al., 2017]. Specifically, the zeroth-order optimization part maintains an iteratively updated and reñned Gaussian distribution to find better actions based on Q-values. Then, we rectify the policy towards this action to better leverage the conservative value function. We conduct extensive experiments in standard continuous control multi-agent particle environments and the complex multi-agent loco- motion task to demonstrate its effectiveness. On all the benchmark tasks, OMAR outperforms the multi-agent version of offline RL algorithms including CQL (Kumar et al. (0200) and TD3+BC (Fu- jimoto & Gu 2021). well as recent offline MARL algorithm MA-ICQ Yang et al. 2021 and achieves the state-of-the-art performance.
The main contribution of this work can be summarized as follows. We propose the OMAR al- gorithm that effectively leverages both first-order and zero-order optimization for solving offline MARL tasks. In addition, theoretically prove that OMAR leads to safe policy improvement. Finally, extensive experimental results demonstrate the effectiveness of OMAR, which significantly outperforms strong baseline methods and achieves state-of- thetth performance in datasets with dif- ferent qualities in both decentralized and centralized learning paradigms.
2 BACKGROUND
We consider the framework of partially observable Markov games (POMG) (Littman 1994 Hu et al. 1998), which extends Markov decision processes to the multi-agent setting. A POMG with N agents is defined by set of global states S, a set of actions o,,..., ON for each agent. At each timestep. each agent receive an observation og and chooses an action based on its policy Ti The environment transits the next state according to the state transition function P. S x A1 X x AN x S + [0, 1]. Each agent receives reward based on the reward function r; : x A1 x AN - RR and a private observation o, : - O, The initial state distribution is defined by p S > 1). The goal is to find set of optimal policies т = { J. where each agent aims to maximize its own discounted return >ie 7 with y denoting the discount factor. In the offline setting, agents learn from fixed dataset D generated from the behavior policy T3 without interaction with the environments.
and set of observations
2.1 MULTI-AGENT ACTOR CRITIC
Centralized critic. Lowe et al. (2117) propose Multi-Agent Deep Deterministic Policy Gradi- ents (MADDPG) under the centralized training with decentralized execution (CTDE) paradigm by extending the DDPG algorithm (Lillierap et al., 2016) to the multi-agent setting. In CTDE, agents are trained in centralized way where they access to extra global information dur-
2

---

Under review as a conference paper at ICLR 2022
ing training while they need to learn decentralized policies in order to act based only lo- cal observations during execution. In MADDPG, for an agent i, the centralized critic Q, is parameterized by 0,- It takes the global state action joint action inputs, and aims min- imize the temporal difference eπror defined by [(0.) Ep [[Q; (s, 01...... an)) y:)", where . = and Qi and म; denote target networks. To reduce
= r; + 7Qi(s', the overestimation problem in MADDPG, MATD3 (Ackermann et al. 2019) estimates the tar- get value using double estimators based on TD3 (Fujimoto et al., 2018), where Yi т; + y min (f di, dd)lef== Agents learn decentralized policies parameterized by Si. which take only local observations inputs, and are trained by multi-agent policy gradients cording to V (TT) = ED VV, ni(a: [oo))V., Va, Qi (s, d1,..., an) (o.= where aj is predicted from its policy while a_; sampled from the replay buffer.
Decentralized critic. Although using centralized critics is widely-adopted in multi-agent actor- critic methods, it introduces scalability issues due to the exponentially sized joint action space w.r.. the number of agents (Iqbal & Sha 2019). On the other hand, independent learning approaches train decentralized crities that take only the local observation and action as inputs.
shown in Witt et al.. (020); Lyu et al. (2021) that decentralized value functions can result in more robust performance and be beneficial in practice compared with centralized critic approaches, de Witt et al. (2020) propose Independent Proximal Policy Optimization (IPPO) based on PPO (Schulman al., 2017). and show that it can match or even outperform CTDE approaches in the challenging discrete control benchmark tasks (Samvelyan et al., 2019). We can also obtain the Independent TD3 (ITD3) algorithm based on decentralized critics, which is trained to minimize the temporal difference error defined by [(0,) = EE (Q: (oi, a.) 3 where y, = Ti + ming=1,2 (o (o()))
2.2 CONSERVATIVE Q- Q-LERNNING
Conservative Q-Learning (CQL) (Kumar al. 2020 adds regularizer the critic loss address the extrapolation error and learns lower-bounded Q-values. It penalizes Q-values of state-action pairs sampled from uniform distribution pairs in the dataset to be large. Specifically, when built upon decentralized critic methods in MARL, the critic loss is defined as in Eq. (I), where a denotes the regularization coefficient and n the empirical behavior policy of agent
policy while encouraging Q-values for state-action
Ep, [[Qi(0,, az) v1) + aEp;
log Σ exp(Qi(o,, a:)) - Barnse (11,)
(I)
3 PROPOSED METHOD
In this section, we first provide a motivating example where previous methods, such as CQL (Kumar ett al. 2020) and TD3+BC (Fujimoto & Gu 2021) be inefficient in the face of the multi-agent setting. Then, we propose a method called Offline Multi-Agent Reinforcement Learning with Actor Rectification (OMAR), where we effectively combine first-order policy gradients and zeroth-order optimization methods for the actor to better optimize the conservative value function.
3.1 THE MOTIVATING EXAMPLE
We design Spread environment as shown Figure T a) which involves n agents and n landmarks (n 2 1) with I-dimensional action space to demonstrate the problem and reveal interesting findings. For the multi-agent setting in the Spread task, n agents need to learn how to cooperate to cover all landmarks and avoid colliding with each other arriving at the same landmark by coordinating their actions. The experimental setup is the same as in Section 1.1.1
Figure bb) demonstrates the performance of the multi-agent version of TD3+BC (Fujimoto & Gu 2021). CQL (Kumar et al. 2020), and OMAR based on ITD3 in the medium--eplay dataset from the two-agent Spread environment. As MA-TD3+BC is based on behavior regularization that compels the learned policy to stay close to the behavior policy, its performance largely depends on the quality of the dataset. Moreover, it can be detrimental regularize policies to be close the dataset in
3

---

Under review as a conference paper at ICLR 2022
multi-agent settings due to decentralized training and the resulting partial observations. MA-CQL outperforms MA-TD3+BC, which pushes down Q-values of state- pairs that are sampled from random the current policy while pushing up Q-values for aate-action pairs in the dataset.
Figure (c) demonstrates the performance improvement percentage of MA-CQL over the behavior policy with increasing number of agents ranging from one five. From Figure T(c), observe that its performance degrades dramatically as there are more agents.
agent
landmark
Figure 1: Analysis of MA-TD3+BC, MA-CQL, and OMAR in the medium-replay dataset from Spread. (a) Spread. (b) Performance. (c) Performance improvement percentage of MA-CQI. over the behavior policy with varying number of agents. (d) Visualization of the Q-function landscape. The red circle represents the predicted action from the agent using MA-CQL. The green triangle and blue square represent the predicted action from the updated policy of MA-CQL and OMAR.
Towards mitigating the performance degradation, we identify key issue MA-CQL that solely regularizing the critic is insufficient for multiple agents to learn good policies for coordination. In Figure [d). we visualize the Q-function landscape of MA-CQL during training for an agent in a timestep, with the red circle corresponding to the predicted action from the actor. The green triangle represents the action predicted from the actor after the training step, where the policy gets stuck in bad local optimum. The first-order policy gradient method is prone to local optima (Dauphin) jet al., 2014, AAhmed et al., (219), where the agent can fail to globally leverage the conservative value function well and thus leading suboptimal, uncoordinated learning behavior. Note that the problem is exacerbated more in the offline multi-agent setting due to the exponentially sized joint action space w.r.t the number of agents (Yang al., 202I). In addition, usually requires each of the agent to learn a good policy for coordination to solve the task, and the suboptimal policy by any agent could result uncoordinated global failure.
Tables I and 2 show the performance of MA-CQL by increasing the learning rate or the number of updates for the actor. The results illustrate that, solve this challenging problem, we need better solution than blindly tuning hyperparameters, In the next section, we introduce how we tackle this problem by combining zeroth-order optimization with current RL algorithms.
Table 1:
Performance of MA-CQL with
Performance of MA-CQL with larger number of updates for the actor.
Table 2:
larger learning rate for the actor.
Learning rate le
5e - 2
le
# Updates
20
267.9
202.0
100.1
267.9
278.6
263.7
Performance
Performance
+19.0
+.38.
433.4
+19.0
+14.8
123.1
3.2 OFFLINE MULTI-AGENT REINFORCEMEN LEARNING WITH ACTOR RECTIFICATION
Our key identification as above is that policy gradient improvements are prone to local optima given bad value function landscape. It is important to note that this presents particularly critical chal- lenge in the multi-agent setting since it is sensitive to suboptimal actions. Zeroth-order optimization methods, e.g., evolution strategies (Rubinstein & Kroese 2013 Such et al. [017]] Conti al. 2017 Salimans et al. 2017; Mania et al. 2018). offer an alternative for policy optimization and are also robust local optima (Rubinstein & Kroese 2013).
We propose Offline Multi Agent Reinforcement Learning with Actor Rectification (OMAR) which incorporates sampled actions based on Q-values rectify the actor that it escape from bad
4

---

Under review as a conference paper at ICLR 2022
local optima. For simplicity of presentation, we demonstrate our method based on the decentralized training paradigm introduced Section 2.1 Note that can also be applied to centralized critics, as shown in Section 4.1.4) Specifically, we add regularizer to the policy objective:
Ep, (1 (i-r))((o, ((.)))
(T)().) 6.)²
(2)
where â; is the action provided by the zeroth-order optimizer and T E [o, 1| denotes the regulariza- tion coefficient. Note that TD3+BC (Fujimoto & Gu. [021) uses the seen action in the dataset for å, The distinction between optimized and seen actions enables OMAR to perform well even if the
dataset quality is from mediocre to low.
We borrow intuition for sampling actions from recent evolution strategy (ES) algorithms,, which show welcoming avenue towards using zeroth-order method for policy optimization. For example, the cross-entropy method (CEM) (Rubinstein & K [003), a popular ES algorithm, has shown great potential in RL (Lim et al. 2018). especially by sampling in the parameter space of the ac- tor (Pourchot& & Sigaud 2019). However, CEM does not scale to tasks with high-dimensional space well (Nagabandi et al. 2020 We therefore propose to sample actions in a softer way motivated by Williams et al. (2013); [ et (0118). Specifically, we sample actions according to an it- eratively refined Gaussian distribution N M(ui, o.). At each iteration j. we draw K candidate actions by af ~ [Np of) and evaluate their Q-values. The mean and standard deviation of the sampling distribution is updated and refined by Eq. (3) which produces a softer update and leverages more The OMAR algorithm is shown in Algorithm [
samples in the update (Nagabandi et al. 2020).
kk= exp(3()) ΣK exp(3QTT)
K Σ
(at-
1²)²
(3)
Besides the algorithmic design, we also prove that OMAR gives safe policy improvement guaran- tee. Let Jm;)) denote the discounted return of a policy n, in the empirical MDP M, which is induced
by transitions in the dataset D,, i..., M, = {(o, at, r,, of) D.). In Theorem | we give lower bound on the difference between the policy performance of OMAR over the empirical behavior policy oØ, in the empirical MDP Mr, The proof can be found in Appendix A
Theorem 1. Let क. be the policy obtained by optimizing Eq. ((TT) J(FR)) Z
(2) Then, we have that _ ornd (ox) [D( (ox)] + - E o (oi) [[T] à 0,)). where D(T,, (o.) ने (T.))))))
T-Baarad (o)).a the marginal discounted distribution of observations of policy nτ.
1, and d= (o:) is
As shown Theorem [ the difference between the second and third terms on the right-hand side is the difference between two expected distances. The former corresponds to the gap between the optimal action and the action from our zeroth-order optimizer, while the latter corresponds to the gap between the action from the behavior policy and the optimized action. Since both terms can be bounded, we find that OMAR gives a safe policy improvement guarantee over ft..
Discussion of the effect of OMAR in the Spread environment. We now investigate whether OMAR can address the identified problem and analyze its effect in the Spread environment introduced in Section 3.T] In Figure [d), the blue square corresponds to the action from the updated actor using OMAR according Eq. (2). In contrast to the policy update in MA-CQL, OMAR can better leverage the global information in the critic and help the actor to escape from the bad local optima. Figure further validates that OMAR significantly improves MA-CQL in terms of both performance and efficiency. Figure 2 shows the performance improvement percentage of OMAR over MA-CQL with varying number of agents, where OMAR always outperforms MA-CQL. We also notice that the performance improvement of OMAR over MA-CQL is much more significant in the multi-agent setting in the Spread task than the single-agent setting, which echoes with what is discussed above that the problem becomes more critical in scenarios with more agents that requires each of the agents to learn a good policy to cooperate for solving the task.
Figure 2: Performance improvement percentage of OMAR over MA- CQL with varying num- ber of agents.
5

---

Under review as a conference paper at ICLR 2022
Algorithm Offline Multi-Agent Reinforcement Learning with Actor Rectification (OMAR).
Initialize Q-networks Q/ O₂ and policy networks n; with random parameters 0i, 82 Φi and target networks with 0 +- 6,, 0z 0g and di t- Φs for each agent E [1,N] N]
2:
for training step / = to T do
3:
for agent i to NN do
4:
Sample a random minibatch of S samples (o,, a4 Ti, T(,0)) from
5:
Set - ri + 7 min (Qi(o, n. (o} + +))), (2 m.(o + +)))
6:
Update critics 0g to minimize Eq. ①
7:
Initialize N ((1,, o;)
8:
for iteration
to do
9:
Draw a population with K individuals A, = {āa ((u, )
10:
Estimate Q-values for K individuals in the population (Q) (oi, a
11:
Update 14, and Ø, of the distribution according to Eq. (3)
12:
Obtain the picked candidate action a; = arg max a
(0;, ai)
13:
Update the actor o; + maxs, 3 (1 T) (o,, Ti - (a:) a:)²
14:
Update target networks: 0f + p04 + (1 p)0² and di pbù (1 -p)Ø:
4 EXPERIMENTS
In this section, we conduct series of experiments to study the following key questions: i) How does OMAR compare against state-of-the-art offline RI. and offline MARL. methods? ii) What is the effect of critical hyperparameters and the sampling scheme? iii) Does the method help in both centralized training and decentralized training paradigms? iv) Can OMAR scale to the more complex continuous multi-agent locomotion tasks?
4.1 MULTI-AGENT PARTICLE ENVIRONMENTS
4.1.1 EXPERIMENTAL SETUP
We first conduct a series of experiments in the widely-adopted multi-agent particle tasks (Lowe al., (011)) as shown in Figure 5 Appendix B. The cooperative navigation task includes 3 agents and landmarks, where agents are rewarded based on the distance to the landmarks and penalized for colliding with each other. Thus, it is important for agents cooperate cover all landmarks without collision. In predator-pres prrtoorrppryy..ooo 3 predators aim to catch the prey. The predators need to cooperate to surround and catch the prey the predators are slower than the prey. The world task involves 4 slower cooperating agents that aim to catch 2 faster adversaries, where adversaries desire eat foods while avoiding being captured.
We construct a variety of datasets according to behavior policies with different qualities based on adding noises to the MATD3 algorithm increase diversity following previous work (Fu et al., 2020). The random dataset is generated by rolling out a randomly initialized policy for I million (M) steps. We obtain the medium-replay dataset by recording all samples the experience replay buffer during the training process until the policy reached the medium level of performance. The medium dataset consists 1M samples by unrolling a partaallypretrained policy in the online setting whose performance reaches a medium level of the performance. The expert dataset is constructed by 1M expert demonstrations from an online fully-trained policy.
We compare OMAR against state-of-the-ar offline RL algorithms including CQL (Kumar et al., 2020) and and TD3+BC (Fujimoto & Gu (022). We also compare with a recent offline MARL algorithm MA-ICQ YYang et al. 2021). We build all methods on independent TD3 based decentralized critics following de Witt et al. (0020). while we also consider centralized critics based on MATD3 following Yu et al. (2021) in Section 4.144 All baselines
implemented based the open-source code Each algorithm is run for five random seeds, and we report the mean performance with standard deviation. A detailed description of the construction of the datasets and hyperparameters can be found in Appendix B.I
https://github...
qba 1281 g

---

Under review as a conference paper at ICLR 2022
4.1.2 PERFORMANCE COMPARISON
Table summarizes the average normalized scores in different datasets in multi-agent particle envi- ronments, where the learning curves are shown in Appendix B.2 The normalized score is computed as 100 x (S Srandom) (Sepet Srandm) following Fu al. (020) As shown, the performance of MA--TD3BBC highly depends on the quality of the dataset. As the MA-ICQ method is based on only trusting seen state-action pairs in the dataset, does not perform well datasets with more diverse data distribution including random and medium-replay datasets, while generally matches the performance of MA-TD3+BC datasets with more narrow distribution including medium and ex- pert. MA-CQL matches or outperforms MA-TD3+BC in datasets with lower quality except for the expert dataset, as it does not rely on constraining the learning policy to stay close to the behavior pol- icy. Our OMAR method significantly outperforms all baseline methods and achieves state-of of the-art performance. We attribute the performance gain to the actor rectification scheme that is indepen- dent of data quality and improves global optimization. In addition, OMAR does incur much computation cost and only takes 4.7% more runtime on average compared with that of MA-CQL.
Table 3: Averaged normalized score of OMAR and baselines in multi-agent particle environments.
MA-ICQ
MA-TD3+BC
MA-CQL
OMAR
Random
Cooperative navigation
6.3 3.5
9.8
24.0 9.8
34.4 5.3
Predator-prey
2.2 2.6
3.7
8.2
11.1 2.8
World
1.0 3.2
2.8
0.6
5.9 5.2
Medium -replay
Cooperative navigation
1.7
5.6
2.4
37.9 12.3
Predator-prey
34.5 27.8
28.7 20.9
24.8 + 17.3
47.1 + 15.3
World
12.0
17.4 8.1
29.6 + 13.8
42.9 19.5
Medium
Cooperative navigation
29.3 5.5
29.3 4.8
3.21
47.9 + 18.9
Predator-prey
63.3 20.0
65.1 29.5
61.7 23.1
66.7 23.2
World
71.9 20.0
73.4 9.3
58.6 11.2
74.6 + 11.5
Expert
Cooperative navigation
104.0 3.4
108.3
9.2
114.9 2.6
Predator-prey
113.0 14.4
115.2 12.5
116.2 + 19.8
World
109.5 = 22.8
110.3 21.3
71.9 28.1
110.4 25.7
4.1.3 ABLATION STUDY
The effect of the regularization coefficient. We first investigate the effect of the regularization coefficient in the actor loss in Eq. (2). Figure 3 shows the averaged normalized score of OMAR over different tasks with different values of in each kind of dataset. As shown, the performance of OMAR is sensitive to this hyperparameter, which controls the exploitation level of the critic. We find the best value of r is neither close to nor O, showing that it the combination of both policy gradients and the actor rectification that performs well. We also notice that the optimal value of smaller for datasets with lower quality and more diverse data distribution including random and medium-replay. but larger for medium and expert datasets. In addition, the performance of OMAR with all values of matches or outperforms that of MA-CQL. This is the only hyperparameter that needs to be tuned in OMAR beyond MA-CQL.
21
Repreturio cootdam
Repfetizie costidet
C1 costidet
(a) Random.
(b) Medium-replay.
(c) Medium.
(d) Expert.
Figure 3: Ablation study on the effect of the regularization coefficient in different types of datasets.
7

---

Under review as a conference paper at ICLR 2022
The effect of key hyperparameters in the sampling scheme. Core hyperparameters for sam- pling mechanism involves the number of iterations, the number sampled actions, and the initial mean and standard deviation of the Gaussian distribution. Figures T (a)(d) show the performance comparison of OMAR with different values of these hyperparameters in the cooperative navigation task, where the grey dotted line corresponds the normalized score of MA-CQL.. As shown, our sampling mechanism is not sensitive to these hyperparameters, and we fix them to be the set with the best performance.
Plan
(a) Number of iterations.
(b) Number of samples.
(c) Mean.
(d) Standard deviation.
Figure 4: Ablation study the effect of key hyperparameters in the sampling mechanism averaged over different types of datasets.
The effect of the sampling mechanism. We now analyze the effect of the zeroth-order optimiza- tion methods in OMAR, and compare it against random shooting and the cross-entropy method (CEM) (De Boer et al. 2005) in the cooperative navigation task. As shown in Table 4 our sam- pling mechanism significantly outperforms the random sampling scheme and CEM, with a larger margin in datasets with lower quality including random and medium-replay. The proposed sampling technique incorporates more samples into the distribution updates more effectively.
Table Ablation study OMAR with different sampling mechanisms in different types of datasets.
Random
Medium-replay
Medium
Expert
OMAR (random)
24.3 7.0
23.5 + 5.3
41.2 + 11.1
101.0 + 5.2
OMAR (CEM)
25.8 + 7.3
32.6 5.1
45.0 13.3
106.4 + 13.8
OMAR
34.4 5.3
37.9 12.3
47.9 + 18.9
114.9 2.6
4.1.4 APPLICABILITY ON CENTRALIZED TRAINING WITH DECENTRALIZED EXECUTION
In this section, demonstrate the versatility of the method and show that it also be applied and beneficial to methods based on centralized critics under the CTDE paradigm. Specifically, all baseline methods are built upon the MATD3 algorithm (Ackermann et al. 2019) using centralized critics as detailed in Section 2 Table 5 summarizes the averaged normalized score of different algorithms in each kind of dataset. As shown, OMAR (centralized) also significantly outperforms MA-ICQ (centralized) and MA-CQL (centralized), and matches the performance of MA-TD3+BC (centralized) in the expert dataset while outperforming in other datasets.
Table 5: The average normalized score of different methods based MATD3 with centralized critics under the CTDE paradigm.
Random
Medium-reply
Medium
Expert
MA-ICQ
5.2 + 5.5
10.1 + 4.6
27.4 + 5.3
96.7 + 4.1
MA-TD3+BC
7.9 + 2.2
9.3 + 9.1
3.7
108.1 3.3
MA-CQL
12.8 4.9
11.2 6.6
26.3 + 13.3
69.5 + 15.7
OMAR
21.6 4.6
19.1 9.2
33.7 + 14.5
105.9 3.6
4.2 MULTI-AGENT MUJoCo
In this section, we investigate whether OMAR can scale to more complex continuous control multi- agent tasks. Peng et al. (2020) introduce multi-agent locomotion tasks which extends the high-
8

---

Under review as a conference paper at ICLR 2022
dimensional MuJoCo locomotion tasks in the singleagent setting to the multi-agent case. We con- sider the two-agent HalfCheetah task (Kim et al. 2021) as shown in Appendix [B.I] where the first and second agents control different parts of joints of the robot. Agents need to cooperate to make the robot run forward by coordinating the actions. We also construct different types of datasets follow- ing Fu et al. (2020) the same in Section 44..11] Jummarize the average normalized scores in each kind of dataset in multi-agent HalfCheetah. As shown, OMAR significantly outperforms baseline methods in random, medium-replay, and medium datasets, and matches the performance of MA-TD3+HD in expert, demonstrating its effectiveness to scale to more complex control tasks.
Table 6: Average normalized score of different methods in multi-agent HalfCheetah.
Random
Medium-reply
Medium
Expert
MA-ICQ MA-TD3+BC
7.4 0.0
35.6 +: 2.7
73.6 5.0
110.6 = 3.3
2.5
3.7
114.4 3.8
MA-CQL OMAR
7.4
41.2 10.1
50.4
64.2 24.9
15.4 12.3
57.7 5.1
80.7 + 10.2
113.5 + 4.3
RELATED WORK
Offline reinforcement learning. Many recent papers achieve improvements in offline RL (Wu et al. 2019; Kumar et al. 2019 Yu et al. 2020 Kidambi et al.) 2020) that address the extrapolation error. Behavior regularization typically compels the learning policy stay close to the behavior policy. Yet, its performance relies heavily on the quality of the dataset. Critic regularization ap- proaches typically add a regularizer to the critic loss that pushes down Q-values for actions sampled from a given policy (Kumar et al. 2020). As discussed above, it can be difficult for the actor to best leverage the global information the critic as policy gradient methods are prone to local optima, which is particularly important in the offline multi-agent setting.
Multi-agent reinforcement learning. number of multi-agent policy gradient algorithms train agents based on centralized value functions (Lowe et al. Foerster et al. 2018; Yu et al. (021]) while another line research focuses on decentralized training (de Witt et al. 2020). Yang et al. (22T)) show that the extrapolation eιror in offline RL can be more severe in the multi-agent setting than the single-agent case due to the exponentially sized joint action space w.r.t the number of agents. In addition, it presents a critical challenge in the decentralized setting when the datasets for each agent only consist of its own action instead of the joint action (Jiang & Lu 2021). Jiang & Lu (2021) address the challenges based on the behavior regularization BCQ (Fujimoto et al.) 2019) algorithm while Yang et al. (202T) propose to estimate the target value based on the next action from the dataset. As a result, both methods largely depend on the quality of the dataset.
Zeroth-order optimization method. It has been recently shown in (Such et al. 2 Conti et al., 2017 Mania et al.) 2018) that evolutionary strategies (ES) emerge as another paradigm for continu- ous control. Recent research shows that it is potential to combine RL with ES to reap the best of both worlds (Khadka & Tumer 2018 Pourchot& & ( in the high-dimensional parameter space for the actor. (Snn et al. (2020) replace the policy gradient update via supervised learning based on sampled noises from random shooting. KKlashhikkvv et al. (2018): Lim et al. (2018); [Simmons Edler et al. (2019); [Peng al. (2020) extend Q-learning based approaches to handle continuous action space based on the popular cross-entropy method (CEM) in ES.
6 CONCLUSION
In this paper, we identify the problem that when extending conservatism-base RL algorithms to of- fline multi-agent scenarios, the performance degrades along increasing number of agents. To tackle this problem, propose Offline Multi-Agent RL with Actor Rectification (OMAR) that combines first-order policy gradient with zeroth-order optimization. We find that OMAR can successfully help the actor escape from bad local optima and consequently find better actions. OMAR achieves state-of-the- thrt performance on multi-agent continuous control tasks empirically.

---

Under review as a conference paper at ICLR 2022
REPRODUCIBILITY STATEMENT
We include all details for our experiments in Appendix B. jncluding open-source implementations of environments and algorithms, with detailed description for the hyperparameters and network structures. For Theorem [ its proof can be found in Appendix [ The code will be open-sourced upon publication of work at htthtts:: //sttes...ss//eee///////ee/estes google. com www.wwwwwwwwt..tc......w..w..t.e icl httpiiewwirr2222mmaal
REFERENCES
Johannes Ackermann, Volker Gabler, Takayuki Osa, and Masashi Sugiyama. estimation bias in multi-agent domains using double centralized crities. arXiv:/910.01465, 2019.
Reducing over- arXiv preprint
Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. optimistic perspective on offline reinforcement learning. In International Conference on Machine Learning, pp. 104-114, PMLR, 2020.
Zffarali Ahmed, Nicolas Le Roux, Mohammad Norouzi, and Dale Schuurmans. Understanding the impact of entropy on policy optimization. International Conference on Machine Learning, pp. 151-160. PMLR, 2019.
Christopher Amato.
Decision-making under uncertainty in multi-agent and multi-robot systems: Planning and learning. In IJCAI, pp. 5662-5666, 2018.
Edoardo Conti, Vashisht Madhavan, Felipe Petroski Such, Joel Lehman, Kenneth O Stanley, and Jeff Clune. Improving exploration in evolution strategies for deep reinforcement learning via population of novelty-seeking agents. arXiv preprint arXiv: 2017.
Yann N Dauphin, Razvan Pascanu, Caglar Gulcehre, Kyunghyun Cho, Surya Ganguli, and Yoshua Bengio. Identifying and attacking the saddle point problem in high-dimensional non-convex op- timization. In Advances Neural Information Processing Systems, pp. 2933-2941, 2014.
Pieter-Tjerk De Boer, Dirk P Kroese, Shie Mannor, and Reuven Y Rubinstein. A tutorial on the cross-entropy method. Annals of operations research, 134(1):19-67, 2005.
Christian Schroeder de Witt, Tarun Gupta, Denys Makoviichuk, Viktor Makoviychuk, Philip HS Torr, Mingfei Sun, and Shimon Whiteson. Is independent learning all you need in the starcraft multi-agent challenge? arXiv preprint #rXiv:2011.09533 2020.
Jakob Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020.
Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. arXiv preprint 1rXXiv:216606688 2021.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor- critic methods. In International Conference on Machine Learning, pp. 1587-1596. PMLR, 2018.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. International Conference on Machine Learning, pp. 0052-2062. PMLR, 2019.
Rong Ge, Jason D Lee, and Tengyu Ma. Learning one- mme-hidden-layee neural networks with landscape design. arXiv preprint arXiv:1711.0001, 2017.
Caglar Gulcehre, Ziyu Wang, Alexander Novikov, Tom Le Paine, Sergio Gomez Colmenarejo, Kon- rad Zolna, Rishabh Agarwal, Josh Merel, Daniel Mankowitz, Cosmin Paduraru, et al. RI un- plugged: Benchmarks for offline reinforcement learning. arXiv e-prints, pp. arXiv-2006. 2020.
10

---

Under review as a conference paper at ICLR 2022
Junling Hu, Michael P Wellman, al. Multiagent reinforcement learning: theoretical framework and an algorithm. In ICML, volume 98, pp. 242-250. Citeseer, 1998.
Shariq Iqbal and Fei Sha. Actor-ateention-crttic for multi-agent reinforcement learning. In Interna- fional Conference on Machine Learning, pp. 2961-2970. PMLR, 2019.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameteriization with gumbel-softmax, arXiv preprint arXiv: 1.0/ 1.01 2016.
Jiechuan Jiang and Zongging Lu. Offline decentralized multi-agent reinforcement learning. arXiv preprint arXiv:2 000.000.00888 018332 2021.
Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, et al. Qt-opt: Scalable deep reinforcement learning for vision-based robotic manipulation. arXiv preprint arXiv: 10293, 2018.
Shauharda Khadka and Kagan Tumer. Evolution-guided policy gradient in reinforcement learning. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pp. 1196- 1208, 2018.
Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. Morel: Model- based offline reinforcement learning. arXiv preprint arXiv:2005.0595/ 2020.
Dong Ki Kim, Miao Liu, Matthew D Riemer, Chuangchuang Sun, Marwa Abdulhai, Golnaz Habibi, Sebastian Lopez-Cor Gerald Tesauro, and Jonathan How. A policy gradient algorithm for learning to learn in multiagent reinforcement learning. In International Conference Machine Learning, pp. 5541-5550. PMLR, 2021.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1rXiv:1412.6980 2014.
Ilya Kostrikov, Rob Fergus, Jonathan Tompson, and Ofir Nachum. Offline reinforcement learning with fisher divergence critic regularization. In International Conference on Machine Learning, pp. 5774-5783. PMLR, 2021.
Aviral Kumar, Justin Fu, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. arXiv preprint 100.00996600099, 2019.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. In Advances in Neural Information Processing Systems, volume 33, pp- 1179-1191, 2020.
Seunghyun Lee, Younggyo Seo, Kimin Lee, Pieter Abbeel, and Jinwoo Shin. Offline-to- online reinforcement learning via balanced replay and pessimistic q-ensemble. arXiv preprint arXiv:2107.0059)) 2021.
Timothy Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In ICLR (Poster), 2016.
Sungsu Lim, Ajin Joseph, Lei Le, Yangchen Pan, and Martha White. Actor-expert: A framework for using q-learning continuous action spaces. arXiv preprint arXiv:1810.09103. 2018.
Michael L Littman. Markov games as framework for multi-agent reinforcement learning. In
Machine learning proceedings 1994, pp. 157-163. Elsevier, 1994.
Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, OpenAl Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cooperative-compptiive environments. Advances Neural Information Processing Systems, 30:6379-63990 2017.
Kendall Lowrey, Aravind Rajeswaran, Sham Kakade, Emanuel Todorov, and Igor Mordatch. Plan online, leam offline: Efficient learning and exploration via model- based control. arXiv preprint arXiv: 18/ I. 1.01848, 2018.
11

---

Under review as a conference paper at ICLR 2022
Xueguang Lyu, Yuchen Xiao, Brett Daley, and Christopher Amato. Contrasting centralized and de- centralized critics in ageti- reinforcement learning. In Proceedings of the 20th International Conference Autonomous Agents and MultiAgent Systems, pp. 844-852, 2021.
Horia Mania, Aurelia Guy, and Benjamin Recht. Simple random search provides competitive approach reinforcement learning. arXiv preprini
2018.
Ofir Nachum, Mohammad Norouzi, and Dale Schuurmans. Improving policy gradient by exploring under-appreciated rewards. arXiv preprint arXiv:1611.0932,,, 2016.
Ofir Nachum, Bo Dai, Ily Kostrikov, Yinlam Chow, Lihong Li, and Dale Schuurmans. Algaedice: Policy gradient from arbitrary experience. arXiv preprint arXiv:19/2.02074 2019.
Anusha Nagabandi, Kurt Konolige, Sergey Levine, and Vikash Kumar. Deep dynamics models for learning dexterous manipulation. In Conference on Robot Learning, pp. 1101-1112. PMLR,
2020.
Bei Peng. Tabish Rashid, Christian A Schroeder de Witt, Pierre-Alexandre Kamienny, Philip HS Torr, Wendelin Böhmer, and Shimon Whiteson, Faemac: Factored multi-agent centralised policy gradients. arXii preprint arXiv:2003.06709 2020.
Dean Pomerleau. Alvinn: An autonomous land vehicle in a neural network. Technical report, CARNEGIE-MELLON UNIV PITTSBURGH PA ARTIFICIAL INTELLIGENCE AND PSY-
CHOLOGY 1989.
Pourchot and Sigaud. CEM-RL: Combining evolutionary and gradient-based methods for pol-
icy search. In International Conference on Learning Representations, 2019. URL https:
Reuven Y Rubinstein and Dirk Kroese. The cross-entropy method: a unified approach to combina- torial optimization, Monte-Carlo simulation and machine learning. Springer Science & Business Media, 2013.
Dorsa Sadigh, Shankar Sastry, Sanjit A Seshia, and Anca D Dragan, Planning for autonomous cars that leverage effects on human actions. In Robotics: Science and Systems, volume 2, pp. 1-9. Ann Arbor, MI, USA, 2016.
Itay Safran and Ohad Shamir. Spurious local minima common in two-layer relu neural networks. arXiv preprint arXiv: 1712.0898, 2017.
Tim Salimans, Jonathan Ho, Xi Chen, Szymon Sidor, and Ilya Sutskever. Evolution strategies scalable alternative to reinforcement learning. arXiv preprint arXiv:1703.03866 2017.
Mikayel Samvelyan, Tabish Rashid, Christian Schroeder de Witt, Gregory Farquhar, Nantas Nardelli, Tim GJ Rudner, Chia-Man Hung, Philip HS Torr, Jakob Foerster, and Shimon White- son. The starcraft multi-agent challenge. In Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems, pp. 2186-2188, 2019.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint
2017.
Riley Gimmons-Edllos Ben Eisner, Eric Mitchell, Sebastian Seung, and Daniel Lee. Q-learning for continuous actions with cross-entropy guided policies. arXiv preprint arXiv:/903 10605, 2019.
Felipe Petroski Such, Vashisht Madhavan, Edoardo Conti, Joel Lehman, Kenneth O Stanley, and Jeff Clune. Deep neuroevolution: Genetic algorithms are a competitive alternative for training deep neural networks for reinforcement learning. arXiv preprint arXiv: arXiv:1712.06567, 2017.
Hao Sun, Ziping Xu, Yuhang Song, Meng Fang, Jiechao Xiong, Bo Dai, Zhengyou Zhang, and Bolei Zhou. Zedoth-order supervised policy improvement. arXiv preprint S.ki:20000066600 2020.
Philip S Thomas. Safe reinforcement learning. PhD thesis, University of Massachusetts Libraries, 2015.
12

---

Under review as a conference paper at ICLR 2022
Emmanouil- Vasiileios Vlatakis Viatakis-Gkaragkouun Gkaragkounss, Lampros Flokas, and Georgios Piliouras. Efficiently avoiding saddle points with zero order methods: No gradients required. In Advances in Neural Information Processing Systems, volume 32, 2019.
Grady Williams, Andrew Aldrich, and Evangelos Theodorou. Model predictive path integral control using covariance variable importance sampling. arXiv preprint arXiv: 2015.
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint arX1v:1911.11300 2019.
Yiqin Yang, Xiaoteng Ma, Chenghao Li, Zewu Zheng, Qiyuan Zhang, Gao Huang, Jun Yang, and Qianchuan Zhao. Believe what you see: Implicit constraint approach for offline multi-agent reinforcement learning. arXiv preprint arXiv:2106.03400, 2021.
Chao Yu, Akash Velu, Eugene Vinitsky, Yu Wang, Alexandre Bayen, and Yi Wu. The surprising effectiveness of mappo in cooperative, multi-agent games. arXiv preprint arXiv:2103.cm 01955, 2021.
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. Advances Neural Information Processing Systems, 13:14129-14143 2020.
13

---

Under review as a conference paper at ICLR 2022
A PROOF OF THEOREM
Theorem 1. Let oi be the policy obtained by optimizing Eq.
12) Then, have
that [xx]] )+ B- (o.) [D(m] (o.) I- an i (a. [T]]] (o:) - a;)²]
onndd (o)), ( à,)²] where ( (o:) Fu, (F((o:)) 1, and d= (o,) is the
marginal discounted distribution of observations of policy n.
Proof. For OMAR, have the following iterative update for agent i:
Φ t- arg ming, aBo,-D; (Banii(atioon [Qi a:)l - E Maraa (Qi(os, ail]]
(0,(0) as) B (02, 27
(4)
B Boo
2
where (=:): 1 if and only if a; (i(o)).
Let +++1 be the fixed point of solving Equation by setting the derivative of Eq. 4 with respect to Qi to be O, then we have that
(ai=m1(or)) -1 (a.lo))
ao)
(5)
where I the indicator function.
Denote D(T,, at (o.) Ru (a;))))) 1. and obtain the difference between the value function (o,) and the original value function as:
Vi(o.) Vi(o,) (m,, (o:),
(6)
Then, the policy that minimizes the loss function defined in Eq. is equivalently obtained by maximizing
(1-+) (m(m. a - Bandy (os)
B
[D(T;, #g.) - r TE or (a)) [(T:(o))- - -,,) (7)
Therefore, we obtain that
(1- [J(xT) 1 E
and (0s) [D(xt tt Moill
TB
(T:) a,)²]
pp t (a.)
21 (()(())) TE
andd (a)) (ayor) [la: àz)
(8)
Then, from Eq. 8 we obtain the result.
B MORE DETAILS OF THE EXPERIMENTS
B. EXPERIMENTAL SETUP
Tasks. We adopt the open-source implementations for multi-agent particle environmenty P from (Lowe al. 2017) and Multi-Agent MuloCa from (Peng et al. 2020 Figure Jilluttaisss the tasks. The expert and random scores for cooperative navigation, predator-prey, world, and two-agent
HalfCheetah are (516.8, 159.8), (185.6, 4.1), f79.5, 6.8), and (3568.8, -284.0) respectively.
14

---

Under review as a conference paper at ICLR 2022
forest
landmark
landmark
X
landmark
landmark
food
landmark
(a) Cooperative navigation.
(b) Predator-prey.
(c) World.
(d)
Two-agent
HalfChee-
tah.
Figure Multi-agent particle environments and Agent HalfCheetah.
Baselines. All baseline methods are implemented based on an open-source implementatior from (Iqbal & Sha 2019), where we implement MA-TD3+BC MA-CQL and MA-ICQ based on authors' open-source implementations with fine-tuned hyperparameters. For MA-CQL, we tune best critic regularization coefficient from {0.1, 0.5, 1.0, 5.0} following (Kumar et al. 2020) for each task. Specifically, we use the discount factor y of 0.99. We sample minibatch of 1024 samples from the dataset for updating each agent's s actor and critic using the Adam (Kingma & Ba 2014) optimizer with the learning rate to be 0.01. The target networks for the actor and critic are soft updated with the update rate be 0.01. Both the actor and critic networks feedforward networks consisting of two hidden layers with 64 neurons per layer using ReLU activation. For OMAR, the only hyperparameter that requires tuning is the regularization coefficient X, where value for datasets with more diverse data distribution random and medium-replay with a value of 0.5, while we use larger value for datasets with more narrow data distribution in medium and expert with values of 0.7 and 0.9 respectively. As OMAR is insensitive to the hyperparameters of the sampling mechanism, where the number of iteration is 3, the number of samples is 10, the mean is 0.0, and the standard deviation is 2.0. The code will be released upon publication of the paper.
smaller
them fixed of values for all types of datasets in all tasks,
http://ggthubb.ccom////gaggagggg
15

---

Under review as a conference paper at ICLR 2022
B.2
LEARNING CURVES
Figure 6 demonstrates the learning curves of MA-ICQ, MA-TD3+BC, MA-CQL and OMAR in different types of datasets in multi-agent particle environments, where the solid line and shaded region represent mean and standard deviation, respectively.
MA-CC co MA-coL
MA4CO
MA4CO
MA-CO
NA-OO-
NA-OG-
MA-COL
(a) CN-random
(b) CN-medium-replay
(c) CN-medium
(d) CN-expert
MELTO NA
MA
(e) PP-random
(I) PP- re-meluu-rrpplay
PP-medium
(h) PP-expert
NA ICO
NO RCO
NEN CO
NN no
MA
NA CT
(i) W-random
W-medium-replay
(k) W-medium
(I W-expert
Figure 6: Learning curves of MA-ICQ, MA-TD3+BC, MA-CQL, and OMAR in multi-agent particle environments (CN, PP, and W is abbreviated for cooperative navigation, predator-prey. and world respectively).
B.3 STARCRAFT II MICROMANAGEMENT BENCHMARK
Setup. We investigate the effectiveness of OMAR larger-scale tasks based on the challenging StarCraft II micromanagement benchmark (Samvelyan et al. 2019) on maps with increasing number of agents and difficulties including 2s3z, 3s5z, 1c3s5z, and 2c_vs_642g, Details for the tasks shown in Table 7 We compare OMAR and MA-CQL based on the evaluation protocol in Kumar ctt al.] (2020); Agarwal al. (2020); Gulcehre et al. (020), where datasets are constructed following Agarwal al. (2020): Gulcehre et al. (2020) by recording samples observed during training. Each dataset consists of million samples. We use the Gumbel-Softmax reparameterization trick (Jang al. 2016) to generate discrete actions for MATD3 since it requires differentiable policies (Lowe et al.
2017 Iqbal & Shal 2019 Peng et al. 2020). All implementations are based on open-sourced and the same experimental setup Appendix B.1
code
https:://githu..com//
16

---

Under review as a conference paper at ICLR 2022
Table 7: Specs of tested maps in the StarCraft II micromanagement benchmark.
Name
Agents
Enemies
2s3z
2 Stalkers and 3 Zealots
Stalkers and Zealots
3s5%
3 Stalkers and 5 Zealots
3 Stalkers and 5 Zealots
Ic3s5z
1 Colossi, 3 Stalkers and Zealots
Colossi, 3 Stalkers and 5 Zealots
2c_vs_64zg
2 Colossi
64 Zerglings
Results. Figure 7 demonstrates the comparison result in test win rates. As shown, OMAR signif- icantly outperforms MA-CQL in performance and learning efficiency, and the average performance gain of OMAR compared to MA-CQL is 76. in all tested maps.
20v
0MM
OMA
0MN
Figure Comparison OMAR and MA-CQL in StarCraft II micromanagement tasks.
B.4 DISCUSSION ON OMAR IN ONLINE/OFFLINE, MULT-AGEETTSINGLL-AEENN SETTINGS
We now investigate the effectiveness of OMAR in the four following settings: (1) online multi-agent setting, (2) online single-agent setting, (3) offline single-agent setting, and (4) offline multi-agent setting in the Spread environment shown in Figure Ta).
Setup. For the online setting, we build our method upon the MATD3 algorithm with our proposed policy objective Eq. (2). and evaluate performance improvement percentage of our method over MATD3. The results for the online setting shown in the right part in Figure g where the x-axis corresponds to the performance improvement percentage and the y-axis corresponds to the number of agents indicating whether its single-agent multi-agent setting. We combine the results in Figure 2 for the offline setting which shows the performance improvement percentage OMAR over MA-CQL in the left part in Figure for better understanding of the effectiveness of our method in different settings.
Results. As shown in Figure 3 method is generally applicable in all the settings. However, the performance improvement is much more significant in the offline setting (left part) than the online case (right part), because the agents cannot explore and interact with the environment. Intuitively, in the online setting, the actor has not well exploited the global information in the value function, can still interact with the environment to collect better experiences for improving the estimation of the value function and provides better guidance for the policy. However, no exploration and interaction with the environment for new data collection are allowed the offline setting. Thus, is much harder for an agent to escape from bad local optimum and difficult for the actor to best leverage the global information in the critic. This presents an even more challenging problem in multi-agent RL because multiple agents result in exponentially-sized joint action space well as the nature of the setting that requires a coordinated joint policy. As expected, we also find that the significant in the offline ageti-agent domain, which requires each of the
performance gain is agents to learn a good policy for successful joint policy for coordination. Otherwise, it can lead to uncoordinated global failure.
17

---

Under review as a conference paper at ICLR 2022
Singl-agon
Oolive
of
Oniee It. MARL
Malll-ager
Office MARI
Parortage 262
Figure 8: Performance improvement percentage of our method over MATD3 in the online (right part) setting and MA-CQL in the offline setting (left part) with a varying number agents in the Spreak task. The first, second, thrid, and fourth quadrants correspond the online RL, offline RL, offline multi-agent RL, and online multi-agent RL settings.
B.5 THE EFFECT OF THE SIZE OF THE DATASET
In this section, conduct ablation study investigate the effect of the size of the dataset following the experimental protocol in Agarwal et al. (2020). We first generate a full replay dataset by recording all samples in the replay buffer encountered during the training course for million steps in the cooperative navigation task. Then, we randomly sample N% experiences from the full replay dataset and obtain several smaller datasets with the same data distribution, where (0.1,1, 1, 10, 20, 50, 100).
Figure 9 shows that the performance of MA-CQL increases given more data points for N {1, 10, 20). 20,
However, it does not further increase given even larger amount of data, which per- forms much worse than the fully-trained online agents and fails to recover their performance. On the contrary, OMAR always outperforms MA-CQL by large margin when N > 1%, whose per- formance is much closer to the fully-trained online agents given more data points. Therefore, the optimality issue still persists when dataset size becomes larger (c.g., escape from them the objective contains very flat regions (Ahmed et al. 2019)). In addition, the zeroth-order optimizer part in OMAR can better guide the actor given larger amount of data points with more accurate value function.
take very long time to
A5C
Figure 9: Normalized score of OMAR and MA-CQL trained using fraction of the entire replay dataset cooperative navigation.
B.6 ADDITIONAL RESULTS OF OMAR IN SINGLE-AGENT ENVIRONMENTS
Besides the single-agent setting the Spread task we have shown in Figure 2 we also evaluate the effectiveness of method in single-agent tasks by comparing it with CQL in the Maze2D domain from the D4RL benchmark (Fu et al. 2020). Table shows the results in an increasing order of complexity of the maze imaze2d-umazz maze2d maze2d-medium. maze2d-laree large).. Based on the results in Table 3 and Figure 2 we observe that OMAR performs much better than CQL, which indicates that OMAR is effective in the offline single-agent tasks.
18

---

Under review as a conference paper at ICLR 2022
Table Averaged normalized score of OMAR and CQL in the single-agent Maze2D domain from
D4RL.
maze2d-umaze
maze2d-medium
maze2d-large
CQL
109.8 + 23.9
106.4 + 11.0
94.6 + 44.6
OMAR
124.7 7.6
125.7 + 12.3
157.7 12.3
B.7 ANALYSIS OF HOW COOPERATION AFFECT THE PERFORMANCE OF CQL IN MULTI-AGENT TASKS
We consider non-cooperative version of the Spread task in Figure Ta) which involves agents and n. landmarks, where each the agents aims to navigate to its own unique target landmark. In contrast to the Spread task that requires cooperation, the reward function for each agent only depends on its distance to its target landmark. This is a variant of Spread that consists of multiple independent learning agents, and the performance is measured by the average return over all agents.
100
Percentage
40
20
Number of agants
Figure 10:
Performance improvement percentage of MA-CQL over the behavior policy with a vary- ing number of agents in non-cooperative version of the Spread task.
Figure shows the result of the performance improvement percentage of MA-CQL over the behav- ior policy the independent Spread task. As shown, the performance of CQL does not degrade with an increasing number of agents in this setting that does not require cooperation, unlike dramatic performance decrease in the cooperative Spread task in Figure lc). The result further confirms that the issue we discovered is due to the failure of coordination.
B.8 DISCUSSION ABOUT THE CENTRALIZED AND DECENTRALIZED CRITICS IN OFFLINE MULTI-AGENT RL
We attribute the lower performance in Table (based centralized value function) compared to Table 3 (based on a decentralized value function) due to the base algorithm, where Table 9 shows the performance comparison of offline independent TD3 and offline multi-agent TD3 in different types of dataset in cooperative navigation. As shown, utilizing centralized critics underperform forms decentralized critics in the offline setting. There has also been recent research (de Witt al. 2020
Lyu et al., (02TT) showing the benefits of decentralized value functions compared to centralized more robust performance. We attribute the performance loss of CTDE in the value function conditioning all agent's s actions and the global state that harder to learn well without exploration.
one, which leads
offline setting due to more complex and
Table 9: Averaged normalized score of ITD3 and MATD3 in cooperative navigation.
Random
Medium-replay
Medium
Expert
ITD3
18.7 8.0
19.9 + 4.7
18.6 + 4.4
75.5 - 7.9
MATD3
16.1 + 5.6
12.7 6.1
12.1 + 14.2
1.6 = 2.7
19


Reviews:
[{"review_id": "sWe804NpBo", "reviewer": "Reviewer_pt8q", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is overall clearly written. The intuition is discussed and the main idea is easy to follow with enough explanations. It is a bit surprising to see that such a simple combination of first-order and zeroth-order methods is able to deliver a good performance. The algorithm can be appealing given it seems quite easy to implement.\n\nI do however have some comments:\n(1) Theoretical: the authors state a theorem regarding the performance bound and argue that \"OMAR gives a safe policy improvement guarantee\". However, I do not see how \"safe\" the policy is. Can the authors provide more explanations about the bound? How large the lower bound could be? Is there easy example that one can get some concrete results? It's fine that the theorem is not strong since I consider this as primarily a practical paper, but some explanation is needed to avoid over-claiming the results..\n(2) Practical: I think the empirical evaluation is not adequate. In particular, in the beginning, it is motivated that standard methods fail when there are more and more agents. The experiments, while confirming the efficacy of the proposed method, largely focus on cases where there are few agents.I appreciate the detailed ablation study the authors provide for different parameters, but I think a curve showing how the method compares as the number of agents increase to a large number is useful. The proposed method also doesn't seem to have scalability issue to run such an experiment.", "summary_of_the_paper": "This work considers extending conservatism-based algorithms to offline RL with multi agents. The performance of standard algorithms often degrades significantly in this setting, especially when the number of agents increase. To resolve the issue, the authors propose a simple scheme which essentially combines first-order and zeroth-order policy optimization methods, with the goal to possibly extract the advantages of each method. Some empirical results demonstrate that the proposed algorithm can achieve better performance than standard baselines.\n", "summary_of_the_review": "Overall, I think this paper contains some interesting and useful ideas. Some parts can be further improved as mentioned before and I hope that the author response will address them.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Zp9SzlISTw-", "reviewer": "Reviewer_qETT", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I thought that the paper was pretty well written and did a pretty good job of motivating the fact multi-agent optimization is more likely to fall into bad local minima than single agent optimization. However, I am only so convinced by empirical examples and think the paper can be improved by motivating this fact more based on the theory of MARL i.e. the non-stationary underlying optimization process. I did appreciate, however, the theoretical contribution of Theorem 1 demonstrating that the approach leads to sound updates. Zeroth order optimization has been considered in the RL literature, so its application is not hugely novel in this context. Moreover, I can't shake the feeling that zeroth order optimization is somewhat indirectly related to the multi-agent/offline RL problems specifically. \n\nThe experiments were relatively small scale, but these things are harder in the multi-agent literature and the complexity considered is fairly respectable in comparison to related prior work. I found the experiments provided to be relatively thorough, addressing a lot of key questions I had about the approach. One question I had though was if the bolds in Table 3 are merely higher numbers or statistically significant.\n\n", "summary_of_the_paper": "This paper considers the offline multi-agent RL setting, first demonstrating that optimization is more likely to find bad local optima than in the single agent case. To deal with this problem, the authors propose to add zeroth order optimization to multi-agent training and provide a theorem guaranteeing that this approach leads to safe improvements. The authors conduct extensive experiments and ablations in comparison to relevant baselines on the multi-agent particle environments to demonstrate the efficacy of their approach as a function of the type of data used for offline training. The authors also provide experiments on a slightly larger scale tackling the multi-agent half-cheetah environment. ", "summary_of_the_review": "I generally like this paper and am in favor of acceptance. I think the paper presents a pretty good narrative and addresses the noted optimization issues with a somewhat novel and theoretically grounded approach. The experiments are pretty comprehensive although admittedly of very limited scale. What stops me from giving a higher score is mostly the somewhat disconnected nature of the proposed contribution to the theory of MARL, which is at this point mostly based on intuitive/empirical findings.  ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "RjwHJMsJ25U", "reviewer": "Reviewer_6gYu", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The authors proposed a simple but seemingly effective method for improving MARL with offline datasets. The paper is overall well written and easy to follow. I think the paper is looking at an important problem and contributing to the development of offline MARL, however, I have a few questions:\n\n1. OMAR uses a regularizer to encourage the actor policy to take action with a high Q-value. However, if the local Q-value gradient direction is opposite compared to the selected sampled action, would the actor policy get stuck in some suboptimal actions (e.g. in the middle of the two high Q-value points).\n2. Regarding using the Gaussian function sampling, I wonder if its performance would be affected by the dataset. Since CQL tries to reduce the Q-values of unknown actions, depending on the dataset, if the value function landscape has multiple picks (e.g. mix Gaussian), then the sampling method may result in suboptimal action. Can this issue be handled by this method?\n3. Does this value function saddle issue happens only in the offline setting? Can you provide some intuition regarding why the issue is more severe in an offline setting?\n4. It would be nice to also include the score of the online learning for the expert dataset.\n5. How does the size of the training dataset affects the benefit of this method. Does the low optimal issue go away when you have more data points?\n", "summary_of_the_paper": "This work looks at the problem of training multi-agent reinforcement learning with continuous action space in an offline setting. The authors identified a saddle point issue of the value function landscape in the existing offline MARL/RL methods, which causes the actor policy to be stuck in a bad local optimum. The proposed method samples and evaluates different actions based on a Gaussian function, and adds a regularizer to the actor loss to encourage the actor policy to take the action with a high Q-value. \n", "summary_of_the_review": "The authors identified a concrete problem in offline MARL training and proposed a method that is addressing the issue. The evaluation shows that overall the proposed method outperforms the existing baselines.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "DJ-jwEj8eJS", "reviewer": "Reviewer_cYK5", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## strengths\n\nThe local optimum problem is successfully solved using zero-order optimization for offline training in the multi-agent scenario for the first time.\n\nThe paper is clear and easy to understand, and the approach is simple and effective.\n\n## weaknesses\n\n\nAs mentioned in the paper, offline learning on multi-agent tasks is very common but poorly studied. Although the approach proposed in this paper achieves significant performance, the following issues are not explained or investigated very clearly.\n\n1. Regarding the motivation, although MAPPO is successful, it is undeniable that derictly applying the single-agent algorithm to multi-agent tasks may probabily perform not well (e.g. indepdent Q learning). So it is not surprising that MA-CQL performance is poor. The authors claim that this is due to the local optimum, but there is not sufficient experimental arguments to illustrate that. The simple illustration given in Figure 1.d may not appropriate to represent the relationship between the local optimum and the global optimum, because a higher Q value may be caused by overestimation.\n\n2. Regarding the method, zero-order optimization can be applied not just under multi-agent tasks. So I think a key ablation experiment to illustrate the contribution of zero-order optimization to offline reinforcement learning is performing zero-order optimization based on single-agent CQL.\n\n3. There are some details about the methodology that are worth discussing.\n3.1 Why does zero-order optimization not consider the minimum of multiple Q's?\n3.2 The authors argue that solely regularizing the critic is insufficient for multiple agents to learn good policies for coordination in the offline setting, but in fact, the authors do not analyze coorparation explitily. So is this really the key issue of CQL?\n\n4. There are also some details about the experiment that are worth discussing.\n4.1 Why the performance variance in Figure 3 is so large.\n4.2 Why is the learning rate set so large (0.01) in the training process? An explanation is needed, since the learning rate is usually set to 3e-4 in CQL or SAC.\n4.3 In Table 5, why is the performance worse under the CTDE paradigm? This is somehow contradictory to the conclusion of MADDPG, so a more sufficient explanation is needed.\n\nOverall, this paper addresses important question, but is not well researched. So, I consider the paper to be at the level of borderline for now. I would be happy to raise the score if the authors could more clearly verify the motivation through experiments, and discuss the advantages that zero-order optimization can bring under single-agent offline reinforcement learning tasks. I look forward to the follow-up of this work.\n", "summary_of_the_paper": "This paper proposes a simple yet effective multi-agent offline reinforcement learning algorithm, called Offline Multi-Agent RL with Actor Rectification, which combines CQL (first-order optimization) and evolutionary algorithms (zero-order optimization) to enhance the applicability of the CQL algorithm on multi-agent tasks.", "summary_of_the_review": "Overall, this paper addresses important question, but is not well researched. So, I consider the paper to be at the level of borderline for now, and I recommend marginally below the acceptance threshold (5).", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper makes a key observation that the gradient-based method gets more likely to suffer from poor local optima in multi-agent reinforcement learning (MARL) with more agents particularly in the offline setting.  The paper proposes the use of zeroth order optimization method to avoid local optima.  Specifically, it samples multiple actions and regularize the policy to get closer to the optimal action among those.  The use of such zeroth order method to avoid poor local optima is not particularly new, although finding its effective in MARL and the empirical support are valuable.  The main discussion point was the insufficiency of experimental support, and the additional experiments during the discussion have addressed the original concerns of the reviewers to some extent.  Overall, given the limited novelty and inefficiency of support (either theoretical or empirical), the paper is slightly below the borderline.
Decision: Reject