Plan Better Amid Conservatism: Offline Multi-Agent Reinforcement Learning with Actor Rectification

1 INTRODUCTION . Offline reinforcement learning ( RL ) has shown great potential in advancing the deployment of RL in real-world tasks where interaction with the environment is prohibitive , costly , or risky ( Thomas , 2015 ) . Since an agent has to learn from a given pre-collected dataset in offline RL , it becomes challenging for regular online RL algorithms such as DDPG ( Lillicrap et al. , 2016 ) and TD3 ( Fujimoto et al. , 2018 ) due to extrapolation error ( Lee et al. , 2021 ) . There has been recent progress in tackling the problem based on conservatism . Behavior regularization ( Wu et al. , 2019 ; Kumar et al. , 2019 ) , e.g. , TD3 with Behavior Cloning ( TD3+BC ) ( Fujimoto & Gu , 2021 ) , compels the learning policy to stay close to the manifold of the datasets . Yet , its performance highly depends on the quality of the dataset . Another line of research investigates incorporating conservatism into the value function by critic regularization ( Nachum et al. , 2019 ; Kostrikov et al. , 2021 ) , e.g. , Conservative Q-Learning ( Kumar et al. , 2020 ) , which usually learns a conservative estimate of the value function to directly address the extrapolation error . However , many practical scenarios involve multiple agents , e.g. , multi-robot control ( Amato , 2018 ) , autonomous driving ( Pomerleau , 1989 ; Sadigh et al. , 2016 ) . Therefore , offline multi-agent reinforcement learning ( MARL ) ( Yang et al. , 2021 ; Jiang & Lu , 2021 ) is crucial for solving real-world tasks . Observing recent success of Independent PPO ( de Witt et al. , 2020 ) and Multi-Agent PPO ( Yu et al. , 2021 ) , both of which are based on the PPO ( Schulman et al. , 2017 ) algorithm , we find that online RL algorithms can be transferred to multi-agent scenarios through either decentralized training or a centralized value function without bells and whistles . Hence , we naturally expect that offline RL algorithms would also transfer easily when applied to multi-agent tasks . Surprisingly , we observe that the performance of the state-of-the-art conservatism-based CQL ( Kumar et al. , 2020 ) algorithm in offline RL degrades dramatically with an increasing number of agents as shown in Figure 1 ( c ) in our experiments . Towards mitigating the degradation , we identify a critical issue in CQL : solely regularizing the critic is insufficient for multiple agents to learn good policies for coordination in the offline setting . The primary cause is that first-order policy gradient methods are prone to local optima ( Nachum et al. , 2016 ; Ge et al. , 2017 ; Safran & Shamir , 2017 ) , saddle points ( Vlatakis-Gkaragkounis et al. , 2019 ; Sun et al. , 2020 ) , or noisy gradient estimates ( Such et al. , 2017 ) . As a result , this can lead to uncoordinated suboptimal learning behavior because the actor can not leverage the global information in the critic well . The issue is exacerbated more in the multi-agent settings due to the exponentially-sized joint action space ( Yang et al. , 2021 ) as well as the nature of the setting that requires each of the agent to learn a good policy for a successful joint policy . For example , in a basketball game , where there are two competing teams each consisting of five players . When one of the players passes the ball among them , it is important for all teammates to perform their duties well in their roles to win the game . As a result , if one of the agents in the team fails to learn a good policy , it can fail to cooperate with other agents for coordinated behaviors and lose the ball . In this paper , we propose a surprisingly simple yet effective method for offline multi-agent continuous control , Offline MARL with Actor Rectification ( OMAR ) , to better leverage the conservative value function via an effective combination of first-order policy gradient and zeroth-order optimization methods . Towards this goal , we add a regularizer to the actor loss , which encourages the actor to mimic actions from the zeroth-order optimizer that maximizes Q-values so that we can combine the best of both first-order policy gradient and zeroth-order optimization . The sampling mechanism is motivated by evolution strategies ( Such et al. , 2017 ; Conti et al. , 2017 ; Mania et al. , 2018 ) , which recently emerged as another paradigm for solving sequential decision making tasks ( Salimans et al. , 2017 ) . Specifically , the zeroth-order optimization part maintains an iteratively updated and refined Gaussian distribution to find better actions based on Q-values . Then , we rectify the policy towards this action to better leverage the conservative value function . We conduct extensive experiments in standard continuous control multi-agent particle environments and the complex multi-agent locomotion task to demonstrate its effectiveness . On all the benchmark tasks , OMAR outperforms the multi-agent version of offline RL algorithms including CQL ( Kumar et al. , 2020 ) and TD3+BC ( Fujimoto & Gu , 2021 ) , as well as a recent offline MARL algorithm MA-ICQ ( Yang et al. , 2021 ) , and achieves the state-of-the-art performance . The main contribution of this work can be summarized as follows . We propose the OMAR algorithm that effectively leverages both first-order and zero-order optimization for solving offline MARL tasks . In addition , we theoretically prove that OMAR leads to safe policy improvement . Finally , extensive experimental results demonstrate the effectiveness of OMAR , which significantly outperforms strong baseline methods and achieves state-of-the-art performance in datasets with different qualities in both decentralized and centralized learning paradigms . 2 BACKGROUND . We consider the framework of partially observable Markov games ( POMG ) ( Littman , 1994 ; Hu et al. , 1998 ) , which extends Markov decision processes to the multi-agent setting . A POMG with N agents is defined by a set of global states S , a set of actions A1 , . . . , AN , and a set of observations O1 , . . . , ON for each agent . At each timestep , each agent i receive an observation oi and chooses an action based on its policy πi . The environment transits to the next state according to the state transition function P : S × A1 × . . . × AN × S → [ 0 , 1 ] . Each agent receives a reward based on the reward function ri : S × A1 . . . × AN → R and a private observation oi : S → Oi . The initial state distribution is defined by ρ : S → [ 0 , 1 ] . The goal is to find a set of optimal policies π = { π1 , . . . , πN } , where each agent aims to maximize its own discounted return ∑∞ t=0 γ trti with γ denoting the discount factor . In the offline setting , agents learn from a fixed dataset D generated from the behavior policy πβ without interaction with the environments . 2.1 MULTI-AGENT ACTOR CRITIC . Centralized critic . Lowe et al . ( 2017 ) propose Multi-Agent Deep Deterministic Policy Gradients ( MADDPG ) under the centralized training with decentralized execution ( CTDE ) paradigm by extending the DDPG algorithm ( Lillicrap et al. , 2016 ) to the multi-agent setting . In CTDE , agents are trained in a centralized way where they can access to extra global information dur- ing training while they need to learn decentralized policies in order to act based only on local observations during execution . In MADDPG , for an agent i , the centralized critic Qi is parameterized by θi . It takes the global state action joint action as inputs , and aims to minimize the temporal difference error defined by L ( θi ) = ED [ ( Qi ( s , a1 , . . . , an ) − yi ) 2 ] , where yi = ri + γQ̄i ( s ′ , a′1 , · · · , a′n ) |a′j=π̄j ( o′j ) and Q̄i and π̄i denote target networks . To reduce the overestimation problem in MADDPG , MATD3 ( Ackermann et al. , 2019 ) estimates the target value using double estimators based on TD3 ( Fujimoto et al. , 2018 ) , where yi = ri + γmink=1,2 Q̄ k i ( s ′ , a′1 , · · · , a′n ) |a′j=π̄j ( o′j ) . Agents learn decentralized policies πi parameterized by φi , which take only local observations as inputs , and are trained by multi-agent policy gradients according to ∇φiJ ( πi ) = ED [ ∇φiπi ( ai|oi ) ∇aiQi ( s , a1 , . . . , an ) |ai=πi ( oi ) ] , where ai is predicted from its policy while a−i are sampled from the replay buffer . Decentralized critic . Although using centralized critics is widely-adopted in multi-agent actorcritic methods , it introduces scalability issues due to the exponentially sized joint action space w.r.t . the number of agents ( Iqbal & Sha , 2019 ) . On the other hand , independent learning approaches train decentralized critics that take only the local observation and action as inputs . It is shown in de Witt et al . ( 2020 ) ; Lyu et al . ( 2021 ) that decentralized value functions can result in more robust performance and be beneficial in practice compared with centralized critic approaches . de Witt et al . ( 2020 ) propose Independent Proximal Policy Optimization ( IPPO ) based on PPO ( Schulman et al. , 2017 ) , and show that it can match or even outperform CTDE approaches in the challenging discrete control benchmark tasks ( Samvelyan et al. , 2019 ) . We can also obtain the Independent TD3 ( ITD3 ) algorithm based on decentralized critics , which is trained to minimize the temporal difference error defined by L ( θi ) = ED [ ( Qi ( oi , ai ) − yi ) 2 ] , where yi = ri + γmink=1,2 Q̄ki ( o ′ i , π̄i ( o ′ i ) ) . 2.2 CONSERVATIVE Q-LEARNING . Conservative Q-Learning ( CQL ) ( Kumar et al. , 2020 ) adds a regularizer to the critic loss to address the extrapolation error and learns lower-bounded Q-values . It penalizes Q-values of state-action pairs sampled from a uniform distribution or a policy while encouraging Q-values for state-action pairs in the dataset to be large . Specifically , when built upon decentralized critic methods in MARL , the critic loss is defined as in Eq . ( 1 ) , where α denotes the regularization coefficient and π̂βi is the empirical behavior policy of agent i. EDi [ ( Qi ( oi , ai ) − yi ) 2 ] + αEDi [ log ∑ ai exp ( Qi ( oi , ai ) ) − Eai∼π̂βi ( ai|oi ) [ Qi ( oi , ai ) ] ] ( 1 ) 3 PROPOSED METHOD . In this section , we first provide a motivating example where previous methods , such as CQL ( Kumar et al. , 2020 ) and TD3+BC ( Fujimoto & Gu , 2021 ) can be inefficient in the face of the multi-agent setting . Then , we propose a method called Offline Multi-Agent Reinforcement Learning with Actor Rectification ( OMAR ) , where we effectively combine first-order policy gradients and zeroth-order optimization methods for the actor to better optimize the conservative value function .

Summary: This paper considers the offline multi-agent RL setting, first demonstrating that optimization is more likely to find bad local optima than in the single agent case. To deal with this problem, the authors propose to add zeroth order optimization to multi-agent training and provide a theorem guaranteeing that this approach leads to safe improvements. The authors conduct extensive experiments and ablations in comparison to relevant baselines on the multi-agent particle environments to demonstrate the efficacy of their approach as a function of the type of data used for offline training. The authors also provide experiments on a slightly larger scale tackling the multi-agent half-cheetah environment.