Containerized Distributed Value-Based Multi-Agent Reinforcement Learning

1 INTRODUCTION . Deep reinforcement learning ( DRL ) has been proved effective in various complex domains , including the game of Go ( Silver et al. , 2017 ) , Dota ( OpenAI , 2018 ) , and StarCraft II ( Vinyals et al. , 2017 ) . However , as problems increase in scale , the training of DRL models is increasingly timeconsuming ( Mnih et al. , 2016 ) , which is partly because complex tasks typically require large amounts of training samples . Distributed learning provides a promising direction to significantly reduce training costs by allowing the RL learner to effectively leverage massive experience collected by a large number of actors interacting with different environment instances . Compared to its single-agent counterparts , multi-agent tasks involve more learning agents and a larger search space consists of joint action-observation history . Therefore , multi-agent reinforcement learning ( MARL ) typically requires more samples ( Kurach et al. , 2020 ) and is more time-consuming . However , most of the impressive progress achieved by distributed DRL focuses on the single-agent setting and leaves the efficient training of MARL algorithms largely unstudied . To makeup for this shortage , in this paper , we study distributed value-based multi-agent reinforcement learning . We first pinpoint three unique challenges posed by the increasing number of agents in multi-agent settings listed as follows . ( 1 ) Demanding data transfer . Coming with the increased number of agents is a larger volume of experience – modern MARL algorithms typically require local observation history of all agents apart from global states . If we deploy environments on CPUs like state-of-the-art single-agent distributed DRL algorithms ( Espeholt et al. , 2019 ) , the data transfer between CPUs and GPUs will consume significant CPU computation utility . ( 2 ) Inter-process communication . Inter-process communication between environments , buffers , and learners is more likely to block in multi-agent settings . This is because the experience volume is large and reading/rewriting becomes more time-consuming . ( 3 ) Exploration . The search space of multi-agent problems grows exponentially with the number of agents . Learning performance of MARL algorithms largely rely on efficient exploration in such a large space . In this paper , we propose a containerized distributed value-based multi-agent reinforcement learning framework ( CMARL ) to address these problems . We pack several actors interacting with environments , a local learner , a local buffer , and a carefully designed multi-queue manager into a container . A container interacts with its environment instances , collects data , avoids block via its multi-queue manager , and actively updates its local policy . Data with high priority is transferred to a global learner that brings together the most talented and valuable experience to train the target policy . We further share the parameters of shallow layers for local and global learners to accelerate training while encouraging deep layers of local learners to be as diverse as possible to improve exploration . The advantages of the CMARL framework are as follows . ( 1 ) A container can be deployed on either CPUs or a GPU , making our method scalable and adaptive with the available computational resources . ( 2 ) Largely reduced data transfer . Data collection and prioritization happens within the container , and only highly prioritized data is transferred . ( 3 ) Multi-queue managers work asynchronously from policy learning , enabling efficient and unblocked inter-process communication when data collection . ( 4 ) Diverse behavior enables efficient exploration in the large search space . In this way , CMARL makes a scalable , time-efficient , and diverse distributed MARL framework with high system throughput . We empirically evaluate the performance of our architecture on both Google Research Football ( Kurach et al. , 2020 ) and StarCraft II micromanagement challenges ( Samvelyan et al. , 2019 ) . Our method is the only algorithm that can obtain a positive goal difference on the three GRF tasks , and is the first algorithm which can solve the challenging 5_v_5 full game . On the SMAC benchmark , our method obtains 4-18× better results compared against state-of-the-art non-distributed MARL algorithms . 2 METHODS . In this section , we present our novel containerized framework ( Fig . 1 ) for distributed MARL . We consider fully cooperative multi-agent tasks that can be modelled as a Dec-POMDP ( Oliehoek et al. , 2016 ) consisting of a tuple G=〈I , S , A , P , R , Ω , O , n , γ〉 , where I is the finite set of n agents , γ ∈ [ 0 , 1 ) is the discount factor , and s ∈ S is the true state of the environment . At each timestep , each agent i receives an observation oi ∈ Ω drawn according to the observation function O ( s , i ) and selects an action ai ∈ A , forming a joint action a ∈ An , leading to a next state s′ according to the transition function p ( s′|s , a ) , and observing a reward r = R ( s , a ) shared by all agents . Each agent has local action-observation history τi ∈ T ≡ ( Ω×A ) ∗ . Distributed deep reinforcement learning aims to provide a scalable and time-efficient computational framework . In the multi-agent setting , distributed RL encounters unique challenges . ( 1 ) The experience of MARL agents typically consists of local observations , and actions of all agents . Moreover , the centralized training with decentralized execution paradigm ( Foerster et al. , 2017 ; Rashid et al. , 2018 ) requires global states that contain information of all agents . Therefore , the size of experience grows quadratically with the number of agents . Consequently , transferring experience across different devices is very consuming , leading to a very high CPU usage . Meanwhile , large volume experience may also cause block in inter-process communication . In order to reduce such overhead , we pack several actors and environments into one container and design a multi-queue manager to avoid blocking . The container only sends a portion of experience to the centralized learner , and the centralized learner learns a global policy from these experiences . The whole architecture consists of several containers and one centralized learner . ( 2 ) Another challenge of multi-agent reinforcement learning is that the action-observation space grows exponentially with the number of agents , posing a great challenge to exploration . Multiple containers allow us to do diverse exploration in a way that each container learns an individual policy different from others and uses that policy to explore . Therefore , the proposed containerized framework ( Fig . 1 ) holds the promise to solve these unique challenges of distributed multi-agent reinforcement learning . However , to realize this goal , many structural details need to be designed for each framework component , including the container , the centralized learner , and the training scheme . We now describe them in greater detail . 2.1 CONTAINER . Inside each container , there are k actors interacting with k environment instances to collect experience , one container buffer storing experience , and one container learner training the container policy with batches of trajectories sampled from the local buffer . A critical design consideration of a successful distributed reinforcement learning framework is the uninterrupted learning of learners . To this end , we need to constantly sample batches from the container buffer . Meanwhile , the buffer needs to update itself with new experience constantly . In order to well manage these two operations and avoid read/write conflicts , we introduce a buffer manager , which is the only process controlling the buffer . The buffer manager repeatedly inserts new experience and samples batches . Directly sending new experience from actors to the buffer manager has two shortcomings . ( 1 ) The new experience would be stacked in the multi-process data-transfer queue when the buffer manager is sampling . In this case , actors have to wait , without collecting new trajectories until the buffer finishes sampling . Consequently , experience collection is slowed down . ( 2 ) Receiving trajectories one by one is slower than receiving trajectories in a batch . Moreover , receiving and inserting new experience frequently increases the waiting time between batch sampling . As sampling has to wait , the container learner has to wait when there is no batch to train . These two shortcomings become particularly problematic in multi-agent settings as trajectories consume more space in the multi-process data-transfer queue . In order to tackle these issues , we introduce a multi-queue manager . There is a shared signal between the multi-queue and the buffer manager . The multi-queue manager constantly gathers new experience together unless the signal indicates that the buffer manager requires new batches . When the signal is set , the multi-queue manager compacts all new experience it has gathered to a batch and send them to the buffer manager . Since we use prioritized experience replay , this batch goes through an initial priority calculator before the buffer manager where its priority is calculated . The priority of one trajectory is computed by pτ = Normalize ( ∑ t rt ) + , where Normalize ( X ) = X−L H−L , andL , H are the lower and upper bound of the sum of rewards in a whole trajectory , respectively . is a small constant that avoids a zero probability during sampling . 2.2 CENTRALIZER . The centralizer has an architecture similar to the container ( Fig . 1 ) , except that new experience comes from containers rather than actors . After containers ’ initial priority calculator receive a batch of new experience and compute their priority , containers will send η % of the experience to the centralizer ’ s experience receiver . Transferred experience is sampled with a probability proportional to the priority . Here , η is a real number between 0 and 100 , indicating the fraction of experience to be sent to the centralized learner . Although our architecture can be combined with any value-based MARL algorithms , in this paper , we use QMIX ( Rashid et al. , 2018 ) as the underlying algorithm . The centralized learner updates a QMIX network . Specifically , agents share a three-layer local Q-network , with a GRU ( Cho et al. , 2014 ) between two fully-connected layers , and the global Q value Qθ ( τ , a ) , parameterized by θ , is learned as a monotonic combination of local Q values . The centralized learner is updated by the following TD loss : LTD ( θ ) = EB∼Dcen [ ∑ τ∈B ∑Tτ−1 t=0 [ Qθ ( τt , at ) − ( rt + γmaxaQθ′ ( st+1 , a ) ) ] 2∑ τ∈B Tτ ] ( 1 ) where Tτ is the length of trajectory τ , Dcen is the centralized buffer , the expectation means that batches are sampled from Dcen according to the priority of trajectories , and θ′ is parameters of a target network copied from θ every C Q-network updates . 2.3 ENCOURAGING DIVERSITY AMONG CONTAINERS . Using multiple containers to interact with the environment leads to more time-efficient experience collection . Training will be boosted in large exploration spaces posed by multi-agent tasks when the collected experience is diverse . Such diversity can be achieved by letting containers act and explore differently from each other . In our architecture , containers maintain a Q-network with the same architecture as the centralized learner . Although the architecture is the same , container Q-networks are learned individually and encouraged to be different . To explicitly encourage diversity among containers , in addition to the local TD loss , every container includes a diversity objective in the loss function to maximize the mutual information between container id and its local experience I ( τ , id ) = Eτ , id [ log p ( τ |id ) p ( τ ) ] . ( 2 ) We expand the mutual information as follows : I ( τ , id ) = Eτ , id [ log p ( o0|id ) p ( o0 ) + ∑ t log p ( at|τt , id ) p ( at|τt ) + ∑ t log p ( ot+1|τt , at , id ) p ( ot+1|τt , at ) ] ( 3 ) Here , o0 is the initial observation , and its distribution is independent of container id . Therefore , p ( o0|id ) = p ( o0 ) . Similarly , p ( ot+1|τt , at ) is decided by the transition function and is the same for all containers . So we have p ( ot+1|τt , at , id ) = p ( ot+1|τt , at ) . It follows that I ( τ , id ) = Eτ , id [ ∑ t log p ( at|τt , id ) p ( at|τt ) ] . ( 4 ) However , p ( at|τt , id ) is typically a distribution induced by -greedy , distinguishing only the action with the highest probability and concealing most information about value functions . Therefore , we use the Boltzmann SoftMax distribution of local Q values to replace p ( at|τt , id ) and optimize a lower bound of Eq . 4 : I ( τ , id ) ≥ Eτ , id [ ∑ t log πid ( at|τt ) p ( at|τt ) ] , ( 5 ) where πid ( at|τt ) = Πni=1πiid ( ait|τ it ) . The inequality holds because DKL [ πid ( at|τt ) ‖p ( at|τt , id ) ] is non-negative . We approximate p ( at|τt ) = Πni=1pi ( ait|τ it ) by p ( at|τt ) ≈ Πni=1 ( 1N ∑N j=1 πj ( a i t|τ it ) ) , where N is the number of containers . Then we get the lower bound of Eq . 4 to optimize : I ( τ , id ) ≥ Eτ , id [ ∑ t n∑ i=1 log πid ( a i t|τ it ) 1 N ∑N j=1 πj ( a i t|τ it ) ] ( 6 ) = Eτ , id ∑ t n∑ i=1 DKL πid ( ·|τ it ) ‖ 1N N∑ j=1 πj ( ·|τ it ) ( 7 ) In practice , we minimize the following loss for training the learner of the id-th container : L ( θi ; B ) = LTD ( θi ) + β 1 |B| ∑ τ∈B ∑ t n∑ i=1 DKL πid ( ·|τ it ) ‖ 1N N∑ j=1 πj ( ·|τ it ) − λ 2 ( 8 ) where θi is the parameters of the container Q-function , β is a scaling factor , and λ is a factor controlling the value of the KL divergence . To balance diversity and learning sharing , we split agent network in containers into two parts . The lower two layers are shared among the global learner and all containers . Training data is sampled from the global buffer , and weights are periodically copied to the containers every tglobal_update_time seconds to avoid frequent transfer and unstable learning in containers . The last layer differs in each container and is updated locally , which enables containers to act differently under the same observation .

Summary: This paper focuses on an interesting and important question: how to perform distributed multi-agent deep reinforcement learning? The author first proposed three challenges to be considered: 1) Demanding data transfer. 2) Inter-process communication. 3) Effective Exploration. Further, the author proposed a container-based distributed marl framework, by placing the part that interacts with the environment in the container relieves the pressure of the cpu, and at the same time encourages different containers to have different policies and uses PER to select high-value samples, thereby improving training efficiency.