Offline Decentralized Multi-Agent Reinforcement Learning

1 INTRODUCTION . In multi-agent cooperative tasks , agents learn from the experiences generated by continuously interacting with the environment to maximize the cumulative shared reward . Recently , multi-agent reinforcement learning ( MARL ) has been applied to real-world cooperative systems ( Bhalla et al. , 2020 ; Xu et al. , 2021 ) . However , in many industrial applications , continuously interacting with the environment and collecting the experiences during learning is costly , risky , and time-consuming . One way to address this is offline RL , where the agent can only access a fixed dataset of experiences and learn the policy without further interacting with the environment . However , in multi-agent environments , the dataset of each agent is often pre-collected individually by different behavior policies , which are not necessary to be expert , and each dataset contains the individual action of the agent instead of the joint action of all agents , e.g. , autonomous driving dataset . Therefore , the dataset does not satisfy the paradigm of centralized training , and the agent has to learn the coordinated policy in an offline and fully decentralized way . The main challenge of offline RL is the extrapolation error , an error in value estimate incurred by the mismatch between the experience distributions of the learned policy and the dataset ( Fujimoto et al. , 2019 ) , e.g. , the distance of the learned action distribution to the behavior action distribution , and the bias of the transition dynamics estimated from the dataset to the true transition dynamics . Recently , almost all offline RL methods ( Fujimoto et al. , 2019 ; Levine et al. , 2020 ; Jaques et al. , 2019 ) focus on constraining the learned policy to be close to the behavior policy to avoid the overestimate of the values of out-of-distribution actions , but ignore to correct the transition bias since the deviation of estimated transition dynamics would not be too large if the single-agent environment is stationary . However , in decentralized multi-agent environments , from the perspective of each agent , other agents are a part of the environment , and the transition dynamics experienced by each agent depend on the policies of other agents . Even in a stationary environment , the experienced transition dynamics of each agent will change as other agents update their policies ( Foerster et al. , 2017 ) . Since the behavior policies of other agents would be inconsistent with their learned policies which are unknowable in decentralized multi-agent environments , the transition dynamics estimated from the dataset by each agent would be different from the transition dynamics induced by the learned policies of other agents , causing large errors in value estimates . The extrapolation error would lead to suboptimal policies . Moreover , trained on different distributions of experiences collected by various behavior policies , the estimated values of the same state might be much different between agents , which causes that the learned policies can not coordinate with each other . To overcome the suboptimum and miscoordination caused by transition bias in decentralized learning , we introduce value deviation and transition normalization to deliberately modify the estimated transition probabilities from the dataset . During data collection , if one agent takes an optimal action while other agents take suboptimal actions at a state , the transition probabilities of low-value next states will become large . Thus , the Q-value of the optimal action will be underestimated , and the agents will fall into suboptimum . Since the other agents are also trained , the learned policies of other agents would become better than the behavior policies . For each agent , the transition probabilities of high-value next states induced by the learned policies would be larger than those estimated from the dataset . Therefore , we let each agent be optimistic toward other agents and multiply the transition probabilities by the deviation of the value of next state from the expected value over all next states , to make the estimated transition probabilities close to the transition probabilities induced by the learned policies of other agents . Value deviation could decrease the extrapolation error and help the agents escape from the suboptimum . However , in some cases , the behavior policies of other agents might be highly deterministic , which makes the distribution of experiences unbalanced . If the transition probabilities of high-value next states are extremely low , value deviation may not remedy the underestimate . Moreover , due to the diversity in experience distributions of agents , the value of the same state might be overestimated by some agents while underestimated by others , which results in miscoordination of the learned policies . To address these two problems , we normalize the transition probability estimated from the dataset to be uniform . Transition normalization balances the extremely biased distribution of experiences and builds the consensus about value estimate . By combining value deviation and transition normalization , the agents would learn high-performing and coordinated policies in an offline and fully decentralized way . Although value deviation and transition normalization make transition dynamics non-stationary , we mathematically prove the convergence of Q-learning under such non-stationary transition dynamics . By importance sampling , value deviation and transition normalization take effect only as the weights of the objective function , which makes our method easy to implement . We instantiate the proposed method on BCQ , termed MABCQ , to additionally avoid out-of-distribution actions . Nevertheless , it can also be implemented on other offline RL methods . We build the offline datasets and evaluate MABCQ in four multi-agent mujoco scenarios ( Todorov et al. , 2012 ) . Experimental results show that MABCQ greatly outperforms BCQ , and ablation studies demonstrate the effectiveness of value deviation and transition normalization . To the best of our knowledge , MABCQ is the first method for offline and fully decentralized multi-agent reinforcement learning . 2 RELATED WORK . MARL . Many MARL methods have been proposed for learning to solve cooperative tasks in an online manner . Some methods ( Lowe et al. , 2017 ; Foerster et al. , 2018 ; Iqbal & Sha , 2019 ) extend policy gradient into multi-agent cases . Value factorization methods ( Sunehag et al. , 2018 ; Rashid et al. , 2018 ; Son et al. , 2019 ) decompose the joint value function into individual value functions . Communication methods ( Das et al. , 2019 ; Ding et al. , 2020 ) share information between agents for better cooperation . All these methods follow centralized learning and decentralized execution , where the agents could access the information from other agents during centralized training . However , in our offline and decentralized setting , the datasets of agents are different ; each dataset contains individual actions instead of joint actions ; and the agents can not be trained in a centralized way . For decentralized learning , the key challenge is the obsolete experiences in replay buffer . Fingerprints ( Foerster et al. , 2017 ) deals with obsolete experience problems by conditioning the value function on a fingerprint that disambiguates the age of the sampled data . Lenient-DQN ( Palmer et al. , 2018 ) extends the leniency concept and introduces optimism in the value function update by forgiving suboptimal actions . Concurrent experience replay ( Omidshafiei et al. , 2017 ) induces cor- relations in local policy updates , making agents tend to converge to the same equilibrium . However , these methods require additional information , e.g. , training iteration number , exploration rate , and timestamp , which often are not provided by the offline dataset . Offline RL . Offline RL requires the agent to learn from a fixed batch of data { ( s , a , s′ , r ) } , consisting of single-step transitions without exploration . Unlike imitation learning , offline RL does not assume that the offline data is provided by a high-performing expert but has to handle the data generated by suboptimal or multi-modal behavior policies . Most offline RL methods consider the out-of-distribution action ( Levine et al. , 2020 ) as the fundamental challenge , which is the main cause of the extrapolation error ( Fujimoto et al. , 2019 ) in value estimate in the single-agent environment . To minimize the extrapolation error , some recent methods introduce constraints to enforce the learned policy to be close to the behavior policy , which could be direct action constraint ( Fujimoto et al. , 2019 ) , kernel MMD ( Kumar et al. , 2019 ) , Wasserstein distance ( Wu et al. , 2019 ) , and KL divergence ( Peng et al. , 2019 ) . Some methods train a Q-function pessimistic to out-of-distribution actions to avoid overestimation by adding a reward penalty quantified by the learned environment model ( Yu et al. , 2020 ) , by minimizing the Q-values of out-of-distribution actions ( Kumar et al. , 2020 ; Yu et al. , 2021 ) , or by weighting the update of Q-function via Monte Carlo dropout ( Wu et al. , 2021 ) . A finite-sample analysis for offline MARL ( Zhang et al. , 2018 ) has been studied , but the agents are assumed to get individual rewards instead of the shared reward , and be connected by communication networks , which is not the fully decentralized setting . All these methods do not consider the extrapolation error introduced by the transition bias , which is a fatal problem in offline and decentralized MARL . 3 METHOD . 3.1 PRELIMINARIES . We consider N agents in multi-agent MDP ( Oliehoek & Amato , 2016 ) Menv = < S , A , R , Penv , γ > with the state space S and the joint action space A . At each timestep , each agent i gets state s and performs an individual action ai , and the environment transitions to the next state s′ by taking the joint action ~a with the transition probability Penv ( s′|s , ~a ) . The agents would get a shared reward r = R ( s ) , which is simplified to just depending on state ( Schulman et al. , 2015 ) . The agents learn to maximize the expected return E ∑T t=0 γ trt , where γ is a discount factor and T is the time horizon of the episode . However , in the fully decentralized learning , Menv is partially observable to the agent since the agent can not observe the joint action ~a . During execution , from the perspective of each agent i , there is a viewed MDP MEi = < S , Ai , R , PEi , γ > with the individual action space Ai and the transition probability PEi ( s ′|s , ai ) = ∑ ~a−i Penv ( s ′|s , ~a ) N∏ j 6=i πj ( aj |s ) , where ~a−i denotes the joint action of all agents except agent i , and π denotes the policy of the agent . As the transition probability depends on the policies of other agents , if other agents are also updating their policies , PEi becomes non-stationary . Moreover , if the agent can not interact with the environment , PEi is unknown . Since we only investigate the influence of other agents ’ policies on PEi , we assume Penv to be deterministic . In offline and decentralized settings , each agent i could only access a fixed offline dataset Bi , which is pre-collected by behavior policies and contains the tuples ( s , ai , r , s′ ) . As defined in BCQ ( Fujimoto et al. , 2019 ) , the visible MDP MBi = < S , Ai , R , PBi , γ > is constructed on Bi , which has the transition probability1 PBi ( s ′|s , ai ) = num ( s , ai , s ′ ) ∑ ŝ′ num ( s , ai , ŝ ′ ) , where num ( s , ai , s′ ) is the number of times the tuple ( s , ai , s′ ) is observed in Bi . However , since the learned policies of other agents might be greatly different from the behavior policies , PBi would be biased from PEi , which creates large extrapolation errors and differences in value estimates between agents , and eventually leads to uncoordinated suboptimal policies . 1The transition probability in the following sections means the one calculated from Bi unless otherwise stated . To intuitively illustrate the miscoordination caused by the transition bias , we devise offline datasets in a matrix game for two agents , with the payoff depicted in Table 1 . The action distributions of the behavior polices of the two agents are [ 0.8 , 0.2 ] and [ 0.4 , 0.6 ] , respectively . Table 2 shows the transition probabilities and expected returns calculated by the independent agents in the datasets . Since the datasets are collected by poor behavior policies , when one agent chooses the optimal action , the other agent would choose the suboptimal action with a high probability , which leads to low transition probabilities of high-value next states . Thus , the agents underestimate the optimal actions and converge to the suboptimal policies ( a1 , a2 ) , rather than the optimal policies ( a2 , a1 ) .

Summary: The authors propose a scheme to modify the transition probabilities in the learning process in a fully decentralised MARL setting. The authors show that the method is still converging to the optimal solution. They also provide some experiments on synthetic environments.