On the Critical Role of Conventions in Adaptive Human-AI Collaboration

1 INTRODUCTION . Humans collaborate well together in complex tasks by adapting to each other through repeated interactions . What emerges from these repeated interactions is shared knowledge about the interaction history . We intuitively refer to this shared knowledge as conventions . Convention formation helps explain why teammates collaborate better than groups of strangers , and why friends develop lingo incomprehensible to outsiders . The notion of conventions has been studied in language ( Clark & Wilkes-Gibbs , 1986 ; Clark , 1996 ; Hawkins et al. , 2017 ; Khani et al. , 2018 ) and also alluded to in more general multiagent collaborative tasks ( Boutilier , 1996 ; Stone et al. , 2010 ; Foerster et al. , 2019 ; Carroll et al. , 2019 ; Lerer & Peysakhovich , 2019 ; Hu et al. , 2020 ) . For example , Foerster et al . ( 2019 ) trained agents to play the card game Hanabi , and noted the emergent convention that “ hinting for red or yellow indicates that the newest card of the other player is playable ” . One established characterization of a convention that is commonly used ( Boutilier , 1996 ; Hawkins et al. , 2017 ; Lerer & Peysakhovich , 2019 ) is an arbitrary solution to a recurring coordination problem ( Lewis , 1969 ) . A convention is thus one of many possible solutions that a group of partners happens to converge to . This is in contrast to problem solutions that are enforced by the rule constraints , and would have arisen no matter how the partners collaborated and what behavior they converged to . Success in a collaborative task typically involves learning both types of knowledge , which we will refer to as convention-dependent and rule-dependent behavior . The distinction between these two types of behavior has been studied extensively in the linguistic literature ( Franke et al . ; Brochhagen , 2020 ) . In this work , we focus on the less-explored setting of implicit communication , and provide a concrete approach to learn representations for these two different types of behavior . In the context of multi-agent or human-AI collaboration , we would like our AI agents to adapt quickly to new partners . AI agents should be able to flexibly adjust their partner-specific convention- dependent behavior while reusing the same rule-dependent behavior to simplify the learning problem – just like how humans quickly adapt when playing basketball with a new friend , without the need to re-learn the rules of the game . We would also like our AI agents to coordinate well on similar tasks when paired with the same partners – just like how humans can coordinate better when playing a new sport but with an old friend . Although many existing works similarly recognize conventions as an important factor of successful collaboration , they do not focus on separating conventions from rules of the task . This means that adapting to a new partner can be as difficult as learning a new task altogether . For example , existing techniques that emphasize modeling the partner ’ s policy , such as theory of mind ( Simon , 1995 ; Baker et al. , 2017 ; Brooks & Szafir , 2019 ) or multi-agent reinforcement learning ( Foerster et al. , 2018 ) , attempt to model everything about the agent ’ s belief of the partner ’ s state and policies . Such belief modeling approaches very quickly become computationally intractable , as opposed to solely focusing on the relevant conventions developed with a partner . To address the challenges above , we propose a framework that explicitly separates conventiondependent representations and rule-dependent representations through repeated interactions with multiple partners . After many rounds of solving a task ( e.g . playing basketball ) with different partners , an AI agent can learn to distinguish between conventions formed with a specific partner ( e.g . pointing down signals bounce pass ) and intrinsic complexities of the task ( e.g . dribbling ) . This enables us to leverage the representations separately for fast adaptation to new interactive settings . In the rest of this paper , we formalize the problem setting , and describe the underlying model of partners and tasks . Next , we present our framework for learning separations between rule and convention representations . We show that , under some conditions , our rule representation learns a sufficient statistic of the distribution over best-response strategies . We then run a study on humanhuman interactions to test if our hypothesis – that partners can carry over the same conventions across tasks – indeed holds for human subjects . Finally , we show the merits of our method on 3 collaborative tasks : contextual bandit , block placing , and a small scale version of Hanabi ( Bard et al. , 2020 ) . 2 RELATED WORK . Convention formation has been studied under the form of iterated reference games ( Hawkins et al. , 2017 ; 2019 ) , and language emergence ( Mordatch & Abbeel , 2018 ; Lazaridou et al. , 2017 ) . In these works , partners learn to reference objects more efficiently by forming conventions through repeated interactions . But in these tasks , there is little intrinsic task difficulty beyond breaking symmetries . In multi-agent reinforcement learning , techniques such as self-play , cross-training , or opponent learning ( Foerster et al. , 2018 ) have been used to observe emergent behavior in complex , physical settings ( Nikolaidis & Shah , 2013 ; Liu et al. , 2019 ; Baker et al. , 2020 ) . In addition , convention formation has been shown in tasks like Hanabi ( Foerster et al. , 2019 ; Hu & Foerster , 2019 ) , Overcooked ( Carroll et al. , 2019 ; Wang et al. , 2020 ) , and negotiation tasks ( Cao et al. , 2018 ) , where agents learn both how to solve a task and how to coordinate with a partner . But , these works qualitatively analyze emergent conventions through post-hoc inspection of the learned policies , and do not learn representations that separate conventions from rule-dependent behavior . This makes it difficult to leverage learned conventions when adapting to new partners . More recently , an approach was proposed to design agents that avoid relying on conventions altogether ; instead , the agents aim for unambiguous but potentially sub-optimal strategies ( Hu et al. , 2020 ) . Other approaches for studying conventions include game theory ( Lerer & Peysakhovich , 2019 ) and theory of mind ( Simon , 1995 ; Rabinowitz et al. , 2018 ) . Pragmatic reasoning ( Goodman & Frank , 2016 ) is another framework that aims to estimate beliefs over the partner ’ s states and policies , but all these frameworks that rely on modelling beliefs quickly become intractable . Instead of approximating these belief modelling frameworks ( Monroe & Potts , 2015 ) , we instead focus on learning representations for rules and conventions . Our work shares similarities with the paradigms of meta-learning ( Schmidhuber , 1987 ; Finn et al. , 2017 ) or multi-task learning ( Caruana , 1997 ) , if we view collaboration with different partners as new but related tasks . However , we are also interested in how conventions persist across tasks with similar symmetries . More related to our work is perhaps modular policy networks ( Devin et al. , 2017 ) , which learns policy modules and composes them together to transfer across robot arms with different degrees of freedom and across different tasks . However , their approach relies on input cleanly split between background observations of the task , and robot ’ s internal observations . We do not assume such a split in the input ; our approach tries to learn the separation between rules and partners with one input channel . 3 PRELIMINARIES . We begin by formally defining our problem setting as a two-player Markov Decision Process ( MDP ) . Two-Player MDP We consider a two-agent MDP with identical payoffs , which is a tuple { S , { Ae , Ap } , P , R } . S is a set of states , Ae is a set of actions for agent e , Ap is a set of actions for agent p. In general , e represents the ego agent ( that we control ) , and p represents agents who partner with e. P : S × Ae × Ap × S → [ 0 , 1 ] is the probability of reaching a state given the current state and actions of all agents , and R : S × S → R is the real-valued reward function . Since we are interested in repeated interactions , we consider finite-horizon MDPs . A policy π is a stochastic mapping from a state to an action . For our setting , our policy π = ( πe , πp ) has two parts : πe : S → Ae and πp : S → Ap , mapping the state into actions for agent e and p respectively . We also consider collaborative tasks that are partially observable in our experiments . In addition , some tasks are turn-based , which can be handled by including the turn information as part of the state , and ignoring the other player ’ s actions . Here , we focus on tasks with discrete state and action spaces . Running Example : Collaborative Contextual Bandit We describe a collaborative task to ground our discussion around adapting to new partners , and adapting to new tasks with old partners . Consider a contextual bandit setting with 2 contexts , 4 actions , and a prize for each action . The two players each independently pick an arm to pull , and they score prize ( a ) points if they both picked the same arm a ; otherwise , they score 0 points . An example is depicted in Figure 2 , where green boxes represent arms with prize 1 , the rest have prize 0 , and red and blue circles show the player ’ s actions . In Task 1 of Figure 2 , context A only has one good arm a1 while the context B has two good arms a2 and a4 . After repeated rounds of playing , two agents can eventually converge to coordinating on a convention that chooses one of the two good arms in context B ( e.g . selecting the leftmost good arm a2 ) . When the task shifts slightly as shown in Task 2 but context B remains the same across the two tasks , we can reasonably expect the partners to adhere to the same convention they developed for context B of Task 1 when playing context B of Task 2 . There are two axes of generalization at play . First , for a fixed task we would like to learn the underlying structure of the task ( e.g . green arm locations ) to quickly adapt to a new partner when playing the same task . Second , for a fixed partner we would like to keep track of developed conventions to quickly coordinate with this partner on a new task . In the next sections , we define the notion of a new task and a new partner , and propose a framework to effectively generalize across both dimensions . 4 MODEL OF PARTNERS AND TASKS We consider a family of tasks that share the same state space , action space , and transition dynamics ( all of which we refer to as the domain ) , but can differ in the reward function of the underlying MDP . We also consider a family of partners that our ego agent would like to coordinate with . The underlying model governing the actions taken by a partner at a state of a given task can be seen in Figure 3 . For a new task , a new reward function is sampled from θR , while the domain of the MDP is fixed . For a new partner , we sample from θρ a new function ρ , which maps the Q-function at a state of the task to an action . We refer to ρ as the convention of a partner . In other words , the action aits of a partner i at state s of task t depends on the partner ’ s conventions and ( indirectly through the Q-function at state s ) the reward of the task . θρ is the distribution over conventions and can correspond to , for example , the distribution over emergent conventions from AI-agents trained from selfplay , or can be derived from human partners . More formally , for a fixed domain let QRs ( ap ) = maxae Q R ( s , ae , ap ) , where QR is the optimal Q-function for the two player MDP with reward R ( Boutilier , 1996 ; Oliehoek et al. , 2008 ) . In other words , QRs is the Q-function from the partner ’ s perspective at state s in the task with reward R assum- ing best response by the ego agent . Then , the convention of a partner ρ : S×R|Ap| → Ap determines the partner ’ s action at state s , given by ρi ( s , QRs ) . For example , we might expect the distribution of actions across different partners at a state s to follow Boltzmann rationality ( Ziebart et al. , 2008 ) : Eρ∼θρ [ 1 [ ρ ( s , QRs ) = ap ] ] ∝ exp ( QRs ( ap ) ) . Lastly , we assume that behavior at different states are uncorrelated : a choice of action at one state tells us nothing about actions at other states . Given this formulation , we can hope to learn the underlying structure of a task by playing with a number of partners on the same task . For example , if many sampled partners all make the same action ap at a state s , it is likely that QRs ( ap ) is much higher than the second best action , i.e. , the optimal action is dependent on the rules of the task as opposed to the arbitrariness of partner strategies . Therefore a new partner will likely take action ap as well . Additionally , when coordinating with the same partner across different tasks , we can expect to see developed conventions persist across states with similar Q-functions . In particular , if QR1s = Q R2 s for two different tasks with reward functions R1 and R2 , then ρi ( s , QR1s ) = ρ i ( s , QR2s ) , so a partner will take the same action at state s across the two tasks ( e.g . context B in Figure 2 across Task 1 and 2 ) . We note that the roles of partners and tasks in our model are asymmetrical , assuming rational partners . A task can completely determine the actions of all partners at states with one good action ( e.g . context A in Figure 2 ) , whereas a partner can not blindly pick the same action across all tasks . This asymmetry is reflected in our learning framework and our setup of adapting to new partners and new tasks .

Summary: This paper makes the observation that when performing cooperative tasks with partners, there are two components to learn: how to perform the task, and how to coordinate with the partner according to conventions. Therefore, it proposes to separate these two components via a modular architecture, which learns a task-specific action distribution that attempts to marginalize out all possible partner conventions, and a series of partner-specific action distributions. The agent's own policy is the product of these two distributions. When coordinating with a new partner, the partner-specific component is learned from scratch using a pre-trained task-specific component, and vice versa.