M^3RL: Mind-aware Multi-agent Management Reinforcement Learning

1 INTRODUCTION . As the main assumption and building block in economics , self-interested agents play a central roles in our daily life . Selfish agents , with their private beliefs , preferences , intentions , and skills , could collaborate ( ad-hoc teaming ) effectively to make great achievement with proper incentives and contracts , an amazing phenomenon that happens every day in every corner of the world . However , most existing multi-agent reinforcement learning ( MARL ) methods focus on collaboration when agents selflessly share a common goal , expose its complete states and are willing to be trained towards the goal . While this is plausible in certain games , few papers address the more practical situations , in which agents are self-interested and inclined to show off , and only get motivated to work with proper incentives . In this paper , we try to model such behaviors . We have multiple workers and a manager , together to work on a set of tasks . The manager gets an external reward upon the completion of some tasks , or one specific task . Each worker has a skill set and preference over the tasks . Note that their skills and preferences may not align with each other ( Fig . 1 ( a ) ) , and are not known to the manager ( Fig . 1 ( b ) ) . Furthermore , manager may not get any external reward until a specific task is complete , which depends on other tasks . By default , the self-interested workers simply choose the most preferred tasks , which is often unproductive from the perspective of the entire project . Therefore , the manager gives additional incentives in the form of contracts . Each contract assigns a goal and a bonus for achieving the goal to a worker . ∗Work done while interning at Facebook AI Research . 1Code is available at https : //github.com/facebookresearch/M3RL . With the external incentives , workers may choose different goals than their preferences . Upon completion of assigned goals , the manager receives the rewards associated with those goals and makes the promised payments to the workers . To generate optimal contracts , the manager must infer the workers ’ minds and learn a good policy of goal and reward assignment . Conventional approaches of mechanism design tackle similar problems by imposing strong assumptions ( e.g. , skill/preference distributions , task dependencies , etc ) to find an analytic solution . In contrast , we aim to train a manager using reinforcement learning to i ) assess minds of workers ( skills , preferences , intentions , etc . ) on the fly , ii ) to optimally assign contracts to maximize a collaborative reward , and iii ) is adapted to diverse and even evolving workers and environments . For this , we propose a novel framework – Mind-aware Multi-agent Management Reinforcement Learning ( M3RL ) , which entails both agent modeling for estimating workers ’ minds and policy learning for contract generation . For agent modeling , we infer workers ’ identities by their performance history , and track their internal states with a mind tracker trained by imitation learning ( IL ) . For contract generation , we apply deep reinforcement learning ( RL ) to learn goal and bonus assignment policies . To improve the learning efficiency and adaptation , we also propose high-level successor representation ( SR ) learning ( Kulkarni et al. , 2016 ) and agent-wise -greedy exploration . As a proof of concept , we evaluate our approach in two environments : Resource Collection and Crafting in 2D Minecraft , to simulate multi-agent management problems . The setup and underlying assumptions are designed to mimic real world problems , where workers are not compelled to reveal their true preferences and skills , and there may be dependency between tasks resulting in delayed and sparse reward signals . Workers may also be deceitful ( e.g. , accepting a contract even when the assigned goal is unreachable ) . Our experiments demonstrate that the manager trained by our approach can i ) estimate the mind of each worker from the recent behaviors , ii ) motivate the workers to finish less preferable or intermediate tasks by assigning the right bonuses , iii ) is adaptive to changing teams , e.g. , change of members and/or change of workers ’ skills and preferences , iv ) and has good generalization in different team sizes and novel environments . We have conducted substantial ablation studies by removing the key components , including IL , SR , agent-wise -greedy exploration , and performance history . Our approach shows a consistent performance in standard settings as well as in more challenging ones where workers ’ policies are stochastic and sub-optimal , or there are multiple levels of bonuses required to motivate workers . 2 RELATED WORK . Multi-agent reinforcement learning . For collaboration problems , common multi-agent reinforcement learning ( Littman , 1994 ; Busoniu et al. , 2008 ) usually trains agents ( Oliehoek et al. , 2008 ; Foerster et al. , 2016 ; Peng et al. , 2017 ; Omidshafiei et al. , 2017 ; Lowe et al. , 2017 ) so that they will jointly maximize a shared reward . There also have been work on contributing different credits to agents by factorized value functions ( Koller & Parr , 1999 ; Guestrin et al. , 2001 ; Sunehag et al. , 2018 ; Rashid et al. , 2018 ) , but the spontaneous collaboration assumption is still required . In contrast , we instead train a manager to manage multiple self-interested workers for an optimal collaboration . Principal-agent problems . Our problem setup is closely related to principal-agent problems ( Laffont & Martimort , 2002 ) ( or moral hazard problems ( Hlmstrom , 1979 ) ) in economics . Our manager and workers can be considered as the principal and agents respectively , where agents and principal have different objectives , and the principal needs to provide the right incentives to ensure that the agents make the best choices for what the principal delegates . These problems face similar technical challenges as our problem setup , e.g. , information asymmetry between principals and agents , how to setup incentive cost , how to infer agents types , how to monitor their behaviors , etc . Traditional approaches in economics ( Myerson , 1982 ; Hlmstrom & Milgrom , 1991 ; Sannikov , 2008 ) build mathematical models to address these issues separately in stateless games , often with the assumption that the utility functions and the behavior patterns of the agents are known , leading to complicated models with many tunable parameters . In comparison , our paper provides a practical end-to-end computational framework to address this problem in a data-driven way without any assumption about the agents ’ utilities and their decision making processes . Moreover , this framework is adaptive to changes of agents preferences and capabilities , which very few papers in economics have addressed . We also evaluate our approach in more complex game settings than the ones in the current economics literature . Mechanism design . Similar to our problem setting , mechanism design also tackles problems where agents have different and private preferences ( Myerson , 1981 ; Conitzer & Sandholm , 2002 ) . Its core idea is to set up rules so that the agents will truthfully reveal their preferences for their own interests , and ultimately an optimal collective outcome can be achieved . Our work differs from mechanism design in several ways . First , in addition to preferences , we also acknowledge the fact that agents may have different skills . Second , mechanism design does not consider sequential decision problems , whereas we have to dynamically change the contracts over time . Optimal reward design . The contract generation in our work can be seen as reward design . Some prior work has proposed optimal reward design approaches ( Zhang et al. , 2009 ; Zhang & Parkes , 2008 ; Sorg et al. , 2010 ; Ratner et al. , 2018 ) , where a teacher designs the best reward so that the student will learn faster or alter its policy towards the target policy . In contrast , we try to use deep RL to train optimal reward design policies to manage multi-agents in more complex tasks . Meta-learning . Our work also resembles meta-learning ( Wang et al. , 2016 ; Finn et al. , 2017 ) , which typically aims at learning a meta strategy for multiple tasks ( Maclaurin et al. , 2015 ; Duan et al. , 2017 ; Hariharan & Girshick , 2017 ; Wichrowska et al. , 2017 ; Yu et al. , 2018 ; Baker et al. , 2017 ) with good sample efficiency , or for a fast adaptation ( Al-Shedivat et al. , 2018 ) . The meta-learning in this paper is for addressing the problem of ad-hoc teaming ( Bowling & McCracken , 2005 ; Stone et al. , 2010 ) by training from a limited set of worker population . Theory of Mind . Our agent modeling is inspired by the prior work on computational theory of mind , where both Bayesian inference ( Baker et al. , 2009 ) and end-to-end training ( Rabinowitz et al. , 2018 ) have been applied to understand a single agent ’ s decision making by inferring their minds . In this work , we extend this to optimal multi-agent management by understanding agents ’ minds . 3 PROBLEM SETUP . In an environment , there is a set of goals G corresponding to several tasks , N self-interested workers with different minds , and a manager which can observe workers ’ behaviors but is agnostic of their true minds . Different from the common Markov game setting for MARL in prior work ( Littman , 1994 ) , we use an independent Markov Decision Process ( MDP ) , i.e. , 〈Si , Ai , Ri , Ti〉 , ∀i ∈ N , to model each worker , where Si and Ai are the state space and action space , Ri : Si × Gi → R is the reward function , and Ti : Si × Ai → Si is the state transition probabilities . For achieving goals , a worker has its own policy πi : Si × Gi → Ai . We define the key concepts in this work as follows . Contract . A contract is a combination of goal and bonus assignment initiated by the manager to a specific worker . For simplicity , we consider discrete bonuses sampled from a finite set B . Thus , for worker i at time t , it will receive a contract defined as ( gti , b t i ) , where g t i ∈ G is the goal and bti ∈ B is the corresponding bonus for achieving the goal . Note that the contract will change over time . Worker ’ s mind . We model a worker ’ s mind by its preferences , intentions , and skills . We do not study worker agents ’ beliefs in this paper , which we leave as future work . Preference . A worker ’ s preference is formally defined as its bounded internal utilities of achieving different goals , ui = ( uig : g ∈ G ) , where 0 ≤ uig ≤ umax . Combined with received contract , the worker agent ’ s reward function can be defined as rtig = Ri ( s t i , g ) = ( uig + 1 ( g = g t i ) b t i ) 1 ( s t i = sg ) , g ∈ G. ( 1 ) where sg is the goal state . Intention . The intention of a worker is the goal it is pursuing at any time , i.e. , Iti ∈ G , which is not fully revealed to the manager . Based on the reward defined in Eq . ( 1 ) , there are multiple ways to choose the goal . For a rational worker who is clear about its skills , it will choose the goal by maximizing expected return . I.e. , Iti = arg maxg E [ ∑∞ t=0 γ t ir t ig ] , where 0 < γi ≤ 1 is its discount factor . However , this requires a worker to have a good estimate of its skills and to be honest , which is not always true . E.g. , a worker may want to pursue some valuable goal that it can not reach . So an alternative way is to maximize the utility instead : Iti = arg maxg uig + 1 ( g = gti ) bti . This will make a worker ’ s behavior more deceptive as it may agree to pursue a goal but will rarely produce a fruitful result . In this work , we focus on the second way to achieve a more realistic simulation . After determine which goal to pursue , a worker will decide whether to sign the assigned contact . We denote this by dti ∈ { 0 , 1 } , where dti = 1 means that worker i signs the contract given at time t. Skill . The skill of a worker is jointly determined by its state transition probabilities Ti and its policy conditioned on its intention , i.e. , πi ( ·|sti , Iti ) . Manager ’ s objective . The manager in our setting has its own utility v = ( vg : g ∈ G ) , where vg ≥ 0 is the utility of achieving goal g. To maximize its gain , the manager needs to assign contracts to workers optimally . For the sake of realism , we do not assume that the manager knows for sure if a worker agent is really committed to the assignment . The only way to confirm this is to check whether the goal achieved by the worker is consistent with its last assigned goal . If so , then the manager will gain certain reward based on its utility of that goal and pay the promised bonus to the worker . Thus , we may define the manager ’ s reward function as : rt = RM ( St+1 ) = ∑ g∈G N∑ i=1 1 ( st+1i = sg ) 1 ( g = g t i ) ( vg − bti ) , ( 2 ) where St+1 = { st+1i : i = 1 , · · · , N } is the collective states of all present worker agents at time t+ 1 . The objective of the manager is to find optimal contract generation to maximize its expected return E [ ∑∞ t=0 γ trt ] , where 0 < γ ≤ 1 is the discount factor for the manager . Note that the manager may get the reward of a goal for multiple times if workers reach the goal respectively . Population of worker agents . The trained manager should be able to manage an arbitrary composition of workers rather than only specific teams of workers . For this , we maintain a population of worker agents during training , and sample several ones from that population in each episode as the present workers in each episode . The identities of these workers are tracked across episodes . In testing , we will sample workers from a new population that has not been seen in training .

Summary: This paper proposes a way to train a manager agent which would manage a bunch of worker agents to achieve a high-level goal. Each worker has its own set of skills and preferences and the manager tries to assign sub-tasks to these agents along with bonuses such that the agents can even perform tasks that are not preferred by them. Authors achieve this by training a manager which tracks the skills and preferences of the agents on the fly. Authors have done an extensive analysis of the proposed approach in two simple domains: resource collection and crafting.