Plan Better Amid Conservatism: Offline Multi-Agent Reinforcement Learning with Actor Rectification

1 INTRODUCTION . Offline reinforcement learning ( RL ) has shown great potential in advancing the deployment of RL in real-world tasks where interaction with the environment is prohibitive , costly , or risky ( Thomas , 2015 ) . Since an agent has to learn from a given pre-collected dataset in offline RL , it becomes challenging for regular online RL algorithms such as DDPG ( Lillicrap et al. , 2016 ) and TD3 ( Fujimoto et al. , 2018 ) due to extrapolation error ( Lee et al. , 2021 ) . There has been recent progress in tackling the problem based on conservatism . Behavior regularization ( Wu et al. , 2019 ; Kumar et al. , 2019 ) , e.g. , TD3 with Behavior Cloning ( TD3+BC ) ( Fujimoto & Gu , 2021 ) , compels the learning policy to stay close to the manifold of the datasets . Yet , its performance highly depends on the quality of the dataset . Another line of research investigates incorporating conservatism into the value function by critic regularization ( Nachum et al. , 2019 ; Kostrikov et al. , 2021 ) , e.g. , Conservative Q-Learning ( Kumar et al. , 2020 ) , which usually learns a conservative estimate of the value function to directly address the extrapolation error . However , many practical scenarios involve multiple agents , e.g. , multi-robot control ( Amato , 2018 ) , autonomous driving ( Pomerleau , 1989 ; Sadigh et al. , 2016 ) . Therefore , offline multi-agent reinforcement learning ( MARL ) ( Yang et al. , 2021 ; Jiang & Lu , 2021 ) is crucial for solving real-world tasks . Observing recent success of Independent PPO ( de Witt et al. , 2020 ) and Multi-Agent PPO ( Yu et al. , 2021 ) , both of which are based on the PPO ( Schulman et al. , 2017 ) algorithm , we find that online RL algorithms can be transferred to multi-agent scenarios through either decentralized training or a centralized value function without bells and whistles . Hence , we naturally expect that offline RL algorithms would also transfer easily when applied to multi-agent tasks . Surprisingly , we observe that the performance of the state-of-the-art conservatism-based CQL ( Kumar et al. , 2020 ) algorithm in offline RL degrades dramatically with an increasing number of agents as shown in Figure 1 ( c ) in our experiments . Towards mitigating the degradation , we identify a critical issue in CQL : solely regularizing the critic is insufficient for multiple agents to learn good policies for coordination in the offline setting . The primary cause is that first-order policy gradient methods are prone to local optima ( Nachum et al. , 2016 ; Ge et al. , 2017 ; Safran & Shamir , 2017 ) , saddle points ( Vlatakis-Gkaragkounis et al. , 2019 ; Sun et al. , 2020 ) , or noisy gradient estimates ( Such et al. , 2017 ) . As a result , this can lead to uncoordinated suboptimal learning behavior because the actor can not leverage the global information in the critic well . The issue is exacerbated more in the multi-agent settings due to the exponentially-sized joint action space ( Yang et al. , 2021 ) as well as the nature of the setting that requires each of the agent to learn a good policy for a successful joint policy . For example , in a basketball game , where there are two competing teams each consisting of five players . When one of the players passes the ball among them , it is important for all teammates to perform their duties well in their roles to win the game . As a result , if one of the agents in the team fails to learn a good policy , it can fail to cooperate with other agents for coordinated behaviors and lose the ball . In this paper , we propose a surprisingly simple yet effective method for offline multi-agent continuous control , Offline MARL with Actor Rectification ( OMAR ) , to better leverage the conservative value function via an effective combination of first-order policy gradient and zeroth-order optimization methods . Towards this goal , we add a regularizer to the actor loss , which encourages the actor to mimic actions from the zeroth-order optimizer that maximizes Q-values so that we can combine the best of both first-order policy gradient and zeroth-order optimization . The sampling mechanism is motivated by evolution strategies ( Such et al. , 2017 ; Conti et al. , 2017 ; Mania et al. , 2018 ) , which recently emerged as another paradigm for solving sequential decision making tasks ( Salimans et al. , 2017 ) . Specifically , the zeroth-order optimization part maintains an iteratively updated and refined Gaussian distribution to find better actions based on Q-values . Then , we rectify the policy towards this action to better leverage the conservative value function . We conduct extensive experiments in standard continuous control multi-agent particle environments and the complex multi-agent locomotion task to demonstrate its effectiveness . On all the benchmark tasks , OMAR outperforms the multi-agent version of offline RL algorithms including CQL ( Kumar et al. , 2020 ) and TD3+BC ( Fujimoto & Gu , 2021 ) , as well as a recent offline MARL algorithm MA-ICQ ( Yang et al. , 2021 ) , and achieves the state-of-the-art performance . The main contribution of this work can be summarized as follows . We propose the OMAR algorithm that effectively leverages both first-order and zero-order optimization for solving offline MARL tasks . In addition , we theoretically prove that OMAR leads to safe policy improvement . Finally , extensive experimental results demonstrate the effectiveness of OMAR , which significantly outperforms strong baseline methods and achieves state-of-the-art performance in datasets with different qualities in both decentralized and centralized learning paradigms . 2 BACKGROUND . We consider the framework of partially observable Markov games ( POMG ) ( Littman , 1994 ; Hu et al. , 1998 ) , which extends Markov decision processes to the multi-agent setting . A POMG with N agents is defined by a set of global states S , a set of actions A1 , . . . , AN , and a set of observations O1 , . . . , ON for each agent . At each timestep , each agent i receive an observation oi and chooses an action based on its policy πi . The environment transits to the next state according to the state transition function P : S × A1 × . . . × AN × S → [ 0 , 1 ] . Each agent receives a reward based on the reward function ri : S × A1 . . . × AN → R and a private observation oi : S → Oi . The initial state distribution is defined by ρ : S → [ 0 , 1 ] . The goal is to find a set of optimal policies π = { π1 , . . . , πN } , where each agent aims to maximize its own discounted return ∑∞ t=0 γ trti with γ denoting the discount factor . In the offline setting , agents learn from a fixed dataset D generated from the behavior policy πβ without interaction with the environments . 2.1 MULTI-AGENT ACTOR CRITIC . Centralized critic . Lowe et al . ( 2017 ) propose Multi-Agent Deep Deterministic Policy Gradients ( MADDPG ) under the centralized training with decentralized execution ( CTDE ) paradigm by extending the DDPG algorithm ( Lillicrap et al. , 2016 ) to the multi-agent setting . In CTDE , agents are trained in a centralized way where they can access to extra global information dur- ing training while they need to learn decentralized policies in order to act based only on local observations during execution . In MADDPG , for an agent i , the centralized critic Qi is parameterized by θi . It takes the global state action joint action as inputs , and aims to minimize the temporal difference error defined by L ( θi ) = ED [ ( Qi ( s , a1 , . . . , an ) − yi ) 2 ] , where yi = ri + γQ̄i ( s ′ , a′1 , · · · , a′n ) |a′j=π̄j ( o′j ) and Q̄i and π̄i denote target networks . To reduce the overestimation problem in MADDPG , MATD3 ( Ackermann et al. , 2019 ) estimates the target value using double estimators based on TD3 ( Fujimoto et al. , 2018 ) , where yi = ri + γmink=1,2 Q̄ k i ( s ′ , a′1 , · · · , a′n ) |a′j=π̄j ( o′j ) . Agents learn decentralized policies πi parameterized by φi , which take only local observations as inputs , and are trained by multi-agent policy gradients according to ∇φiJ ( πi ) = ED [ ∇φiπi ( ai|oi ) ∇aiQi ( s , a1 , . . . , an ) |ai=πi ( oi ) ] , where ai is predicted from its policy while a−i are sampled from the replay buffer . Decentralized critic . Although using centralized critics is widely-adopted in multi-agent actorcritic methods , it introduces scalability issues due to the exponentially sized joint action space w.r.t . the number of agents ( Iqbal & Sha , 2019 ) . On the other hand , independent learning approaches train decentralized critics that take only the local observation and action as inputs . It is shown in de Witt et al . ( 2020 ) ; Lyu et al . ( 2021 ) that decentralized value functions can result in more robust performance and be beneficial in practice compared with centralized critic approaches . de Witt et al . ( 2020 ) propose Independent Proximal Policy Optimization ( IPPO ) based on PPO ( Schulman et al. , 2017 ) , and show that it can match or even outperform CTDE approaches in the challenging discrete control benchmark tasks ( Samvelyan et al. , 2019 ) . We can also obtain the Independent TD3 ( ITD3 ) algorithm based on decentralized critics , which is trained to minimize the temporal difference error defined by L ( θi ) = ED [ ( Qi ( oi , ai ) − yi ) 2 ] , where yi = ri + γmink=1,2 Q̄ki ( o ′ i , π̄i ( o ′ i ) ) . 2.2 CONSERVATIVE Q-LEARNING . Conservative Q-Learning ( CQL ) ( Kumar et al. , 2020 ) adds a regularizer to the critic loss to address the extrapolation error and learns lower-bounded Q-values . It penalizes Q-values of state-action pairs sampled from a uniform distribution or a policy while encouraging Q-values for state-action pairs in the dataset to be large . Specifically , when built upon decentralized critic methods in MARL , the critic loss is defined as in Eq . ( 1 ) , where α denotes the regularization coefficient and π̂βi is the empirical behavior policy of agent i. EDi [ ( Qi ( oi , ai ) − yi ) 2 ] + αEDi [ log ∑ ai exp ( Qi ( oi , ai ) ) − Eai∼π̂βi ( ai|oi ) [ Qi ( oi , ai ) ] ] ( 1 ) 3 PROPOSED METHOD . In this section , we first provide a motivating example where previous methods , such as CQL ( Kumar et al. , 2020 ) and TD3+BC ( Fujimoto & Gu , 2021 ) can be inefficient in the face of the multi-agent setting . Then , we propose a method called Offline Multi-Agent Reinforcement Learning with Actor Rectification ( OMAR ) , where we effectively combine first-order policy gradients and zeroth-order optimization methods for the actor to better optimize the conservative value function .

Summary: This work looks at the problem of training multi-agent reinforcement learning with continuous action space in an offline setting. The authors identified a saddle point issue of the value function landscape in the existing offline MARL/RL methods, which causes the actor policy to be stuck in a bad local optimum. The proposed method samples and evaluates different actions based on a Gaussian function, and adds a regularizer to the actor loss to encourage the actor policy to take the action with a high Q-value.