Offline Decentralized Multi-Agent Reinforcement Learning

1 INTRODUCTION . In multi-agent cooperative tasks , agents learn from the experiences generated by continuously interacting with the environment to maximize the cumulative shared reward . Recently , multi-agent reinforcement learning ( MARL ) has been applied to real-world cooperative systems ( Bhalla et al. , 2020 ; Xu et al. , 2021 ) . However , in many industrial applications , continuously interacting with the environment and collecting the experiences during learning is costly , risky , and time-consuming . One way to address this is offline RL , where the agent can only access a fixed dataset of experiences and learn the policy without further interacting with the environment . However , in multi-agent environments , the dataset of each agent is often pre-collected individually by different behavior policies , which are not necessary to be expert , and each dataset contains the individual action of the agent instead of the joint action of all agents , e.g. , autonomous driving dataset . Therefore , the dataset does not satisfy the paradigm of centralized training , and the agent has to learn the coordinated policy in an offline and fully decentralized way . The main challenge of offline RL is the extrapolation error , an error in value estimate incurred by the mismatch between the experience distributions of the learned policy and the dataset ( Fujimoto et al. , 2019 ) , e.g. , the distance of the learned action distribution to the behavior action distribution , and the bias of the transition dynamics estimated from the dataset to the true transition dynamics . Recently , almost all offline RL methods ( Fujimoto et al. , 2019 ; Levine et al. , 2020 ; Jaques et al. , 2019 ) focus on constraining the learned policy to be close to the behavior policy to avoid the overestimate of the values of out-of-distribution actions , but ignore to correct the transition bias since the deviation of estimated transition dynamics would not be too large if the single-agent environment is stationary . However , in decentralized multi-agent environments , from the perspective of each agent , other agents are a part of the environment , and the transition dynamics experienced by each agent depend on the policies of other agents . Even in a stationary environment , the experienced transition dynamics of each agent will change as other agents update their policies ( Foerster et al. , 2017 ) . Since the behavior policies of other agents would be inconsistent with their learned policies which are unknowable in decentralized multi-agent environments , the transition dynamics estimated from the dataset by each agent would be different from the transition dynamics induced by the learned policies of other agents , causing large errors in value estimates . The extrapolation error would lead to suboptimal policies . Moreover , trained on different distributions of experiences collected by various behavior policies , the estimated values of the same state might be much different between agents , which causes that the learned policies can not coordinate with each other . To overcome the suboptimum and miscoordination caused by transition bias in decentralized learning , we introduce value deviation and transition normalization to deliberately modify the estimated transition probabilities from the dataset . During data collection , if one agent takes an optimal action while other agents take suboptimal actions at a state , the transition probabilities of low-value next states will become large . Thus , the Q-value of the optimal action will be underestimated , and the agents will fall into suboptimum . Since the other agents are also trained , the learned policies of other agents would become better than the behavior policies . For each agent , the transition probabilities of high-value next states induced by the learned policies would be larger than those estimated from the dataset . Therefore , we let each agent be optimistic toward other agents and multiply the transition probabilities by the deviation of the value of next state from the expected value over all next states , to make the estimated transition probabilities close to the transition probabilities induced by the learned policies of other agents . Value deviation could decrease the extrapolation error and help the agents escape from the suboptimum . However , in some cases , the behavior policies of other agents might be highly deterministic , which makes the distribution of experiences unbalanced . If the transition probabilities of high-value next states are extremely low , value deviation may not remedy the underestimate . Moreover , due to the diversity in experience distributions of agents , the value of the same state might be overestimated by some agents while underestimated by others , which results in miscoordination of the learned policies . To address these two problems , we normalize the transition probability estimated from the dataset to be uniform . Transition normalization balances the extremely biased distribution of experiences and builds the consensus about value estimate . By combining value deviation and transition normalization , the agents would learn high-performing and coordinated policies in an offline and fully decentralized way . Although value deviation and transition normalization make transition dynamics non-stationary , we mathematically prove the convergence of Q-learning under such non-stationary transition dynamics . By importance sampling , value deviation and transition normalization take effect only as the weights of the objective function , which makes our method easy to implement . We instantiate the proposed method on BCQ , termed MABCQ , to additionally avoid out-of-distribution actions . Nevertheless , it can also be implemented on other offline RL methods . We build the offline datasets and evaluate MABCQ in four multi-agent mujoco scenarios ( Todorov et al. , 2012 ) . Experimental results show that MABCQ greatly outperforms BCQ , and ablation studies demonstrate the effectiveness of value deviation and transition normalization . To the best of our knowledge , MABCQ is the first method for offline and fully decentralized multi-agent reinforcement learning . 2 RELATED WORK . MARL . Many MARL methods have been proposed for learning to solve cooperative tasks in an online manner . Some methods ( Lowe et al. , 2017 ; Foerster et al. , 2018 ; Iqbal & Sha , 2019 ) extend policy gradient into multi-agent cases . Value factorization methods ( Sunehag et al. , 2018 ; Rashid et al. , 2018 ; Son et al. , 2019 ) decompose the joint value function into individual value functions . Communication methods ( Das et al. , 2019 ; Ding et al. , 2020 ) share information between agents for better cooperation . All these methods follow centralized learning and decentralized execution , where the agents could access the information from other agents during centralized training . However , in our offline and decentralized setting , the datasets of agents are different ; each dataset contains individual actions instead of joint actions ; and the agents can not be trained in a centralized way . For decentralized learning , the key challenge is the obsolete experiences in replay buffer . Fingerprints ( Foerster et al. , 2017 ) deals with obsolete experience problems by conditioning the value function on a fingerprint that disambiguates the age of the sampled data . Lenient-DQN ( Palmer et al. , 2018 ) extends the leniency concept and introduces optimism in the value function update by forgiving suboptimal actions . Concurrent experience replay ( Omidshafiei et al. , 2017 ) induces cor- relations in local policy updates , making agents tend to converge to the same equilibrium . However , these methods require additional information , e.g. , training iteration number , exploration rate , and timestamp , which often are not provided by the offline dataset . Offline RL . Offline RL requires the agent to learn from a fixed batch of data { ( s , a , s′ , r ) } , consisting of single-step transitions without exploration . Unlike imitation learning , offline RL does not assume that the offline data is provided by a high-performing expert but has to handle the data generated by suboptimal or multi-modal behavior policies . Most offline RL methods consider the out-of-distribution action ( Levine et al. , 2020 ) as the fundamental challenge , which is the main cause of the extrapolation error ( Fujimoto et al. , 2019 ) in value estimate in the single-agent environment . To minimize the extrapolation error , some recent methods introduce constraints to enforce the learned policy to be close to the behavior policy , which could be direct action constraint ( Fujimoto et al. , 2019 ) , kernel MMD ( Kumar et al. , 2019 ) , Wasserstein distance ( Wu et al. , 2019 ) , and KL divergence ( Peng et al. , 2019 ) . Some methods train a Q-function pessimistic to out-of-distribution actions to avoid overestimation by adding a reward penalty quantified by the learned environment model ( Yu et al. , 2020 ) , by minimizing the Q-values of out-of-distribution actions ( Kumar et al. , 2020 ; Yu et al. , 2021 ) , or by weighting the update of Q-function via Monte Carlo dropout ( Wu et al. , 2021 ) . A finite-sample analysis for offline MARL ( Zhang et al. , 2018 ) has been studied , but the agents are assumed to get individual rewards instead of the shared reward , and be connected by communication networks , which is not the fully decentralized setting . All these methods do not consider the extrapolation error introduced by the transition bias , which is a fatal problem in offline and decentralized MARL . 3 METHOD . 3.1 PRELIMINARIES . We consider N agents in multi-agent MDP ( Oliehoek & Amato , 2016 ) Menv = < S , A , R , Penv , γ > with the state space S and the joint action space A . At each timestep , each agent i gets state s and performs an individual action ai , and the environment transitions to the next state s′ by taking the joint action ~a with the transition probability Penv ( s′|s , ~a ) . The agents would get a shared reward r = R ( s ) , which is simplified to just depending on state ( Schulman et al. , 2015 ) . The agents learn to maximize the expected return E ∑T t=0 γ trt , where γ is a discount factor and T is the time horizon of the episode . However , in the fully decentralized learning , Menv is partially observable to the agent since the agent can not observe the joint action ~a . During execution , from the perspective of each agent i , there is a viewed MDP MEi = < S , Ai , R , PEi , γ > with the individual action space Ai and the transition probability PEi ( s ′|s , ai ) = ∑ ~a−i Penv ( s ′|s , ~a ) N∏ j 6=i πj ( aj |s ) , where ~a−i denotes the joint action of all agents except agent i , and π denotes the policy of the agent . As the transition probability depends on the policies of other agents , if other agents are also updating their policies , PEi becomes non-stationary . Moreover , if the agent can not interact with the environment , PEi is unknown . Since we only investigate the influence of other agents ’ policies on PEi , we assume Penv to be deterministic . In offline and decentralized settings , each agent i could only access a fixed offline dataset Bi , which is pre-collected by behavior policies and contains the tuples ( s , ai , r , s′ ) . As defined in BCQ ( Fujimoto et al. , 2019 ) , the visible MDP MBi = < S , Ai , R , PBi , γ > is constructed on Bi , which has the transition probability1 PBi ( s ′|s , ai ) = num ( s , ai , s ′ ) ∑ ŝ′ num ( s , ai , ŝ ′ ) , where num ( s , ai , s′ ) is the number of times the tuple ( s , ai , s′ ) is observed in Bi . However , since the learned policies of other agents might be greatly different from the behavior policies , PBi would be biased from PEi , which creates large extrapolation errors and differences in value estimates between agents , and eventually leads to uncoordinated suboptimal policies . 1The transition probability in the following sections means the one calculated from Bi unless otherwise stated . To intuitively illustrate the miscoordination caused by the transition bias , we devise offline datasets in a matrix game for two agents , with the payoff depicted in Table 1 . The action distributions of the behavior polices of the two agents are [ 0.8 , 0.2 ] and [ 0.4 , 0.6 ] , respectively . Table 2 shows the transition probabilities and expected returns calculated by the independent agents in the datasets . Since the datasets are collected by poor behavior policies , when one agent chooses the optimal action , the other agent would choose the suboptimal action with a high probability , which leads to low transition probabilities of high-value next states . Thus , the agents underestimate the optimal actions and converge to the suboptimal policies ( a1 , a2 ) , rather than the optimal policies ( a2 , a1 ) .

Summary: This paper proposes a method named MABCQ to utilize offline data in the MARL environment via (1) value deviation and (2) transition normalization. These two techniques are used to correct the bias in the individual observation of the transition probability and improve the discovery of optimal policy. The authors showed the convergence of the Q-learning under the non-stationary transition probabilities after modification.