On the Critical Role of Conventions in Adaptive Human-AI Collaboration

1 INTRODUCTION . Humans collaborate well together in complex tasks by adapting to each other through repeated interactions . What emerges from these repeated interactions is shared knowledge about the interaction history . We intuitively refer to this shared knowledge as conventions . Convention formation helps explain why teammates collaborate better than groups of strangers , and why friends develop lingo incomprehensible to outsiders . The notion of conventions has been studied in language ( Clark & Wilkes-Gibbs , 1986 ; Clark , 1996 ; Hawkins et al. , 2017 ; Khani et al. , 2018 ) and also alluded to in more general multiagent collaborative tasks ( Boutilier , 1996 ; Stone et al. , 2010 ; Foerster et al. , 2019 ; Carroll et al. , 2019 ; Lerer & Peysakhovich , 2019 ; Hu et al. , 2020 ) . For example , Foerster et al . ( 2019 ) trained agents to play the card game Hanabi , and noted the emergent convention that “ hinting for red or yellow indicates that the newest card of the other player is playable ” . One established characterization of a convention that is commonly used ( Boutilier , 1996 ; Hawkins et al. , 2017 ; Lerer & Peysakhovich , 2019 ) is an arbitrary solution to a recurring coordination problem ( Lewis , 1969 ) . A convention is thus one of many possible solutions that a group of partners happens to converge to . This is in contrast to problem solutions that are enforced by the rule constraints , and would have arisen no matter how the partners collaborated and what behavior they converged to . Success in a collaborative task typically involves learning both types of knowledge , which we will refer to as convention-dependent and rule-dependent behavior . The distinction between these two types of behavior has been studied extensively in the linguistic literature ( Franke et al . ; Brochhagen , 2020 ) . In this work , we focus on the less-explored setting of implicit communication , and provide a concrete approach to learn representations for these two different types of behavior . In the context of multi-agent or human-AI collaboration , we would like our AI agents to adapt quickly to new partners . AI agents should be able to flexibly adjust their partner-specific convention- dependent behavior while reusing the same rule-dependent behavior to simplify the learning problem – just like how humans quickly adapt when playing basketball with a new friend , without the need to re-learn the rules of the game . We would also like our AI agents to coordinate well on similar tasks when paired with the same partners – just like how humans can coordinate better when playing a new sport but with an old friend . Although many existing works similarly recognize conventions as an important factor of successful collaboration , they do not focus on separating conventions from rules of the task . This means that adapting to a new partner can be as difficult as learning a new task altogether . For example , existing techniques that emphasize modeling the partner ’ s policy , such as theory of mind ( Simon , 1995 ; Baker et al. , 2017 ; Brooks & Szafir , 2019 ) or multi-agent reinforcement learning ( Foerster et al. , 2018 ) , attempt to model everything about the agent ’ s belief of the partner ’ s state and policies . Such belief modeling approaches very quickly become computationally intractable , as opposed to solely focusing on the relevant conventions developed with a partner . To address the challenges above , we propose a framework that explicitly separates conventiondependent representations and rule-dependent representations through repeated interactions with multiple partners . After many rounds of solving a task ( e.g . playing basketball ) with different partners , an AI agent can learn to distinguish between conventions formed with a specific partner ( e.g . pointing down signals bounce pass ) and intrinsic complexities of the task ( e.g . dribbling ) . This enables us to leverage the representations separately for fast adaptation to new interactive settings . In the rest of this paper , we formalize the problem setting , and describe the underlying model of partners and tasks . Next , we present our framework for learning separations between rule and convention representations . We show that , under some conditions , our rule representation learns a sufficient statistic of the distribution over best-response strategies . We then run a study on humanhuman interactions to test if our hypothesis – that partners can carry over the same conventions across tasks – indeed holds for human subjects . Finally , we show the merits of our method on 3 collaborative tasks : contextual bandit , block placing , and a small scale version of Hanabi ( Bard et al. , 2020 ) . 2 RELATED WORK . Convention formation has been studied under the form of iterated reference games ( Hawkins et al. , 2017 ; 2019 ) , and language emergence ( Mordatch & Abbeel , 2018 ; Lazaridou et al. , 2017 ) . In these works , partners learn to reference objects more efficiently by forming conventions through repeated interactions . But in these tasks , there is little intrinsic task difficulty beyond breaking symmetries . In multi-agent reinforcement learning , techniques such as self-play , cross-training , or opponent learning ( Foerster et al. , 2018 ) have been used to observe emergent behavior in complex , physical settings ( Nikolaidis & Shah , 2013 ; Liu et al. , 2019 ; Baker et al. , 2020 ) . In addition , convention formation has been shown in tasks like Hanabi ( Foerster et al. , 2019 ; Hu & Foerster , 2019 ) , Overcooked ( Carroll et al. , 2019 ; Wang et al. , 2020 ) , and negotiation tasks ( Cao et al. , 2018 ) , where agents learn both how to solve a task and how to coordinate with a partner . But , these works qualitatively analyze emergent conventions through post-hoc inspection of the learned policies , and do not learn representations that separate conventions from rule-dependent behavior . This makes it difficult to leverage learned conventions when adapting to new partners . More recently , an approach was proposed to design agents that avoid relying on conventions altogether ; instead , the agents aim for unambiguous but potentially sub-optimal strategies ( Hu et al. , 2020 ) . Other approaches for studying conventions include game theory ( Lerer & Peysakhovich , 2019 ) and theory of mind ( Simon , 1995 ; Rabinowitz et al. , 2018 ) . Pragmatic reasoning ( Goodman & Frank , 2016 ) is another framework that aims to estimate beliefs over the partner ’ s states and policies , but all these frameworks that rely on modelling beliefs quickly become intractable . Instead of approximating these belief modelling frameworks ( Monroe & Potts , 2015 ) , we instead focus on learning representations for rules and conventions . Our work shares similarities with the paradigms of meta-learning ( Schmidhuber , 1987 ; Finn et al. , 2017 ) or multi-task learning ( Caruana , 1997 ) , if we view collaboration with different partners as new but related tasks . However , we are also interested in how conventions persist across tasks with similar symmetries . More related to our work is perhaps modular policy networks ( Devin et al. , 2017 ) , which learns policy modules and composes them together to transfer across robot arms with different degrees of freedom and across different tasks . However , their approach relies on input cleanly split between background observations of the task , and robot ’ s internal observations . We do not assume such a split in the input ; our approach tries to learn the separation between rules and partners with one input channel . 3 PRELIMINARIES . We begin by formally defining our problem setting as a two-player Markov Decision Process ( MDP ) . Two-Player MDP We consider a two-agent MDP with identical payoffs , which is a tuple { S , { Ae , Ap } , P , R } . S is a set of states , Ae is a set of actions for agent e , Ap is a set of actions for agent p. In general , e represents the ego agent ( that we control ) , and p represents agents who partner with e. P : S × Ae × Ap × S → [ 0 , 1 ] is the probability of reaching a state given the current state and actions of all agents , and R : S × S → R is the real-valued reward function . Since we are interested in repeated interactions , we consider finite-horizon MDPs . A policy π is a stochastic mapping from a state to an action . For our setting , our policy π = ( πe , πp ) has two parts : πe : S → Ae and πp : S → Ap , mapping the state into actions for agent e and p respectively . We also consider collaborative tasks that are partially observable in our experiments . In addition , some tasks are turn-based , which can be handled by including the turn information as part of the state , and ignoring the other player ’ s actions . Here , we focus on tasks with discrete state and action spaces . Running Example : Collaborative Contextual Bandit We describe a collaborative task to ground our discussion around adapting to new partners , and adapting to new tasks with old partners . Consider a contextual bandit setting with 2 contexts , 4 actions , and a prize for each action . The two players each independently pick an arm to pull , and they score prize ( a ) points if they both picked the same arm a ; otherwise , they score 0 points . An example is depicted in Figure 2 , where green boxes represent arms with prize 1 , the rest have prize 0 , and red and blue circles show the player ’ s actions . In Task 1 of Figure 2 , context A only has one good arm a1 while the context B has two good arms a2 and a4 . After repeated rounds of playing , two agents can eventually converge to coordinating on a convention that chooses one of the two good arms in context B ( e.g . selecting the leftmost good arm a2 ) . When the task shifts slightly as shown in Task 2 but context B remains the same across the two tasks , we can reasonably expect the partners to adhere to the same convention they developed for context B of Task 1 when playing context B of Task 2 . There are two axes of generalization at play . First , for a fixed task we would like to learn the underlying structure of the task ( e.g . green arm locations ) to quickly adapt to a new partner when playing the same task . Second , for a fixed partner we would like to keep track of developed conventions to quickly coordinate with this partner on a new task . In the next sections , we define the notion of a new task and a new partner , and propose a framework to effectively generalize across both dimensions . 4 MODEL OF PARTNERS AND TASKS We consider a family of tasks that share the same state space , action space , and transition dynamics ( all of which we refer to as the domain ) , but can differ in the reward function of the underlying MDP . We also consider a family of partners that our ego agent would like to coordinate with . The underlying model governing the actions taken by a partner at a state of a given task can be seen in Figure 3 . For a new task , a new reward function is sampled from θR , while the domain of the MDP is fixed . For a new partner , we sample from θρ a new function ρ , which maps the Q-function at a state of the task to an action . We refer to ρ as the convention of a partner . In other words , the action aits of a partner i at state s of task t depends on the partner ’ s conventions and ( indirectly through the Q-function at state s ) the reward of the task . θρ is the distribution over conventions and can correspond to , for example , the distribution over emergent conventions from AI-agents trained from selfplay , or can be derived from human partners . More formally , for a fixed domain let QRs ( ap ) = maxae Q R ( s , ae , ap ) , where QR is the optimal Q-function for the two player MDP with reward R ( Boutilier , 1996 ; Oliehoek et al. , 2008 ) . In other words , QRs is the Q-function from the partner ’ s perspective at state s in the task with reward R assum- ing best response by the ego agent . Then , the convention of a partner ρ : S×R|Ap| → Ap determines the partner ’ s action at state s , given by ρi ( s , QRs ) . For example , we might expect the distribution of actions across different partners at a state s to follow Boltzmann rationality ( Ziebart et al. , 2008 ) : Eρ∼θρ [ 1 [ ρ ( s , QRs ) = ap ] ] ∝ exp ( QRs ( ap ) ) . Lastly , we assume that behavior at different states are uncorrelated : a choice of action at one state tells us nothing about actions at other states . Given this formulation , we can hope to learn the underlying structure of a task by playing with a number of partners on the same task . For example , if many sampled partners all make the same action ap at a state s , it is likely that QRs ( ap ) is much higher than the second best action , i.e. , the optimal action is dependent on the rules of the task as opposed to the arbitrariness of partner strategies . Therefore a new partner will likely take action ap as well . Additionally , when coordinating with the same partner across different tasks , we can expect to see developed conventions persist across states with similar Q-functions . In particular , if QR1s = Q R2 s for two different tasks with reward functions R1 and R2 , then ρi ( s , QR1s ) = ρ i ( s , QR2s ) , so a partner will take the same action at state s across the two tasks ( e.g . context B in Figure 2 across Task 1 and 2 ) . We note that the roles of partners and tasks in our model are asymmetrical , assuming rational partners . A task can completely determine the actions of all partners at states with one good action ( e.g . context A in Figure 2 ) , whereas a partner can not blindly pick the same action across all tasks . This asymmetry is reflected in our learning framework and our setup of adapting to new partners and new tasks .

Summary: The paper proposes an interesting model to study multi-agent interactions, in uncertain environments. In s nutshell, the model proposed consists of a MDP (Finite state, action, horizon) and two players playing simultaneously (the turn-by-turn model can be subsumed by the simultaneous move model as stated in the paper). In the absence of learning, the finite MDP has an optimal solution. The key contribution of the paper is to focus on instances when the optimal solution is not unique. In a two-player model, this requires symmetry breaking in order for a sample path of the MDP to track the optimal trajectory.