Learning to Guide and to be Guided in the Architect-Builder Problem

1 INTRODUCTION . Humans are notoriously successful at teaching – and learning from – each others . This enables skills and knowledge to be shared and passed along generations , being progressively refined towards mankind ’ s current state of proficiency . People can teach and be taught in situations where there is no shared language and very little common ground , such as a parent teaching a baby how to stack blocks during play . Experimental Semiotics ( Galantucci & Garrod , 2011 ) , a line of work that ∗Equal contribution . †Work conducted while at Inria . ‡Canada CIFAR AI Chair . studies the forms of communication that people develop when they can not use pre-established ones , reveals that humans can even teach and learn without direct reinforcement signal , demonstrations or a shared communication protocol . Vollmer et al . ( 2014 ) for example investigate a co-construction ( CoCo ) game experiment where an architect must rely only on arbitrary instructions to guide a builder toward constructing a structure . In this experiment , both the task of building the structure and the meanings of the instructions – through which the architect guides the builder – are simultaneously learned throughout interactions . Such flexible teaching – and learning – capabilities are essential to autonomous artificial agents if they are to master an increasing number of skills without extensive human supervision . As a first step toward this research direction , we draw inspiration from the CoCo game and propose the Architect-Builder Problem ( ABP ) : an interactive learning setting that models agents ’ interactions with Markov Decision Processes ( Puterman , 2014 ) ( MDPs ) . In the ABP learning has to occur in a social context through observations and communication , in the absence of direct imitation or reinforcement ( Bandura & Walters , 1977 ) . Specifically , the constraints of the ABP are : ( 1 ) the builder has absolutely no knowledge about the task at hand ( no reward and no prior on the set of possible tasks ) , ( 2 ) the architect can only interact with the builder through communication signals ( can not interact with the environment or provide demonstrations ) , and ( 3 ) the communication signals have no pre-defined meanings ( nor belong to a set of known possible meanings ) . ( 1 ) sets this work apart from Reinforcement Learning ( RL ) and even Multi-Agent RL ( MARL ) where explicit rewards are available to all agents . ( 2 ) implies the absence of tele-operation or third-person demonstrations and thus distinguishes the ABP from Imitation and Inverse Reinforcement Learning ( IRL ) . Finally , ( 3 ) prevents the architect from relying on a fixed communication protocol since the meanings of instructions must be negotiated . These constraints make ABP an appealing setting to investigate Human-Robot Interaction ( HRI ) ( Goodrich & Schultz , 2008 ) problems where “ a learner tries to figure out what a teacher wants them to do ” ( Grizou et al. , 2013 ; Cederborg & Oudeyer , 2014 ) . Specifically , the challenge of Brain Computer Interfaces ( BCI ) , where users use brain signals to control virtual and robotic agents in sequential tasks ( Katyal et al. , 2014 ; deBettencourt et al. , 2015 ; Mishra & Gazzaley , 2015 ; Muñoz-Moldes & Cleeremans , 2020 ; Chiang et al. , 2021 ) , is well captured by the ABP . In BCIs , ( 3 ) is identified as the calibration problem and is usually tackled with supervised learning to learn a mapping between signals and meanings . As this calibration phase is often laborious and impractical for users , current approaches investigate calibration-free solutions where the mapping is learned interactively ( Grizou et al. , 2014 ; Xie et al. , 2021 ) . Yet , these works consider that the user ( i.e . the architect ) is fixed , in the sense that it does not adapt to the agent ( i.e . the builder ) and uses a set of pre-defined instructions ( or feedback ) meanings that the agent must learn to map to signals . In our ABP formulation however , the architect is dynamic and , as interactions unfold , must learn to best guide a learning builder by tuning the meanings of instructions according to the builder ’ s reactions . In that sense , ABP provides a more complete computational model of agent-agent or human-agent interaction . With all these constraints in mind , we propose Architect Builder Iterated Guiding ( ABIG ) , an algorithmic solution to ABP when both agents are AIs . ABIG is inspired by the field of experimental semiotics and relies on two high-level interaction priors : shared intent and interaction frames . Shared intent refers to the fact that , although the builder ignores the objective of the task to fulfill , it will assume that its objective is aligned with the architect ’ s . This assumption is characteristic of cooperative tasks and shown to be a necessary condition for the emergence of communication both in practice ( Foerster et al. , 2016 ; Cao et al. , 2018 ) and in theory ( Crawford & Sobel , 1982 ) . Specifically , the builder should assume that the architect is guiding it towards a shared objective . Knowing this , the builder must reinforce the behavior it displays when guided by the architect . We show that the builder can efficiently implement this by using imitation learning on its own guided behavior . Because the builder imitates itself , we call it self-imitation . The notion of interaction frames ( also called pragmatic frames ) states that agents that interact in sequence can more easily interpret the interaction history ( Bruner , 1985 ; Vollmer et al. , 2016 ) . In ABIG , we consider two distinct interaction frames . These are stationary which means that when one agent learns , the other agent ’ s behavior is fixed . During the first frame ( the modelling frame ) , the builder is fixed and the architect learns a model of the builder ’ s message-conditioned behavior . During the second frame ( the guiding frame ) , the architect is fixed and the builder learns to be guided via self-imitation learning . We show that ABIG results in a low-level , high-frequency , guiding communication protocol that not only enables an architect-builder pair to solve the task at hand , but can also be used to solve unseen tasks . Our contributions are : • The Architect-Builder Problem ( ABP ) , an interactive learning setting to study how artificial agents can simultaneously learn to solve a task and derive a communication protocol . • Architect-Builder Iterated Guiding ( ABIG ) , an algorithmic solution to the ABP . • An analysis of ABIG ’ s key learning mechanisms . • An evaluation of ABIG on a construction environment where we show that ABIG agents evolve communication protocols that generalize to unseen harder tasks . • A detailed analysis of ABIG ’ s learning dynamics and impact on the mutual information between messages and actions ( in the Supplementary Material ) . 2 PROBLEM DEFINITION . The Architect-Builder Problem . We consider a multi-agent setup composed of two agents : an architect and a builder . Both agents observe the environment state s but only the architect knows the goal at hand . The architect can not take actions in the environment but receives the environmental reward r whereas the builder does not receive any reward and has thus no knowledge about the task at hand . In this asymmetrical setup , the architect can only interact with the builder through a communication signal m sampled from its policy π A ( m|s ) . These messages , that have no a priori meanings , are received by the builder which acts according to its policy π B ( a|s , m ) . This makes the environment transition to a new state s′ sampled from P E ( s′|s , a ) and the architect receives reward r′ . Messages are sent at every time-step . The CoCo game that inspired ABP is sketched in Figure 1 ( a ) while the overall architect-builder-environment interaction diagram is given in Figure 1 ( b ) . The differences between the ABP setting and the MARL and IRL settings are illustrated in Figure 8 . BuildWorld . We conduct our experiments in BuildWorld . BuildWorld is a 2D construction gridworld of size ( w × h ) . At the beginning of an episode , the agent and Nb blocks are spawned at different random locations . The agent can navigate in this world and grasp blocks by activating its gripper while on a block . The action space A is discrete and include a “ do nothing ” action ( |A| = 6 ) . At each time step , the agent observes its position in the grid , its gripper state as well as the position of all the blocks and if they are grasped ( |S| = 3 + 3Nb ) . Tasks . BuildWorld contains 4 different training tasks : 1 ) ‘ Grasp ’ : The agent must grasp any of the blocks ; 2 ) ‘ Place ’ : The agent must place any block at a specified location in the grid ; 3/4 ) ‘ H-Line/Vline ’ : The agent must place all the blocks in a horizontal/vertical line configuration . BuildWorld also has a harder fifth testing task , ‘ 6-blocks-shapes ’ , that consists of more complex configurations and that is used to challenge an algorithm ’ s transfer abilities . For all tasks , rewards are sparse and only given when the task is completed . This environment encapsulates the interactive learning challenge of ABP while removing the need for complex perception or locomotion . In the RL setting , where the same agent acts and receives rewards , this environment would not be very impressive . However , it remains to be shown that the tasks can be solved in the setting of ABP ( with a reward-less builder and an action-less architect ) . Communication . The architect guides the builder by sending messages m which are one-hot vectors of size |V| ranging from 2 to 72 , see Suppl . Section B.3 for the impact of this parameter . Additional Assumptions . In order to focus on the architect-builder interactions and the learning of a shared communication protocol , the architect has access to P E ( s′|s , a ) and to the reward function r ( s , a ) of the goal at hand . This assumes that , if the architect were to act in the environment instead of the builder , it would be able to quickly figure out how to solve the task . This assumption is compatible with the CoCo game experiment ( Vollmer et al. , 2014 ) where humans participants , and in particular the architects , are known to have such world models . 3 ABIG : ARCHITECT-BUILDER ITERATED GUIDING . 3.1 ANALYTICAL DESCRIPTION . Agents-MDPs . In the Architect-Builder Problem , agents are operating in different , yet coupled , MDPs . Those MDPs depend on their respective point of view ( see Figure 2 ) . From the point of view of the architect , messages are actions that influence the next state as well as the reward ( see Figure 2 ( a ) ) . The architect knows the environment transition function P E ( s′|s , a ) and r ( s , a ) , the true reward function associated with the task that does not depend explicitly on messages . It can thus derive the effect of its messages on the builder ’ s actions that drive the reward and the next states ( see Figure 2 ( b ) ) . On the other hand , the builder ’ s state is composed of the environment state and the message , which makes estimating state transitions challenging as one must also capture the message dynamics ( see Figure 2 ( c ) ) . Yet , the builder can leverage its knowledge of the architect picking messages based on the current environment state . The equivalent transition and reward models , when available , are given below ( see derivations in Suppl . Section A ) . P A ( s′|s , m ) = ∑ a∈A π̃ B ( a|s , m ) P E ( s′|a , s ) r A ( s , m ) = ∑ a∈A π̃ B ( a|s , m ) r ( s , a ) with π̃B ( a|s , m ) ≜ P ( a|s , m ) ( 1 ) P B ( s′ , m′|s , m , a ) = π̃ A ( m′|s′ ) P E ( s′|s , a ) with π̃ A ( m′|s′ ) ≜ P ( m′|s′ ) ( 2 ) where subscripts A and B refer to the architect and the builder , respectively . x̃ denotes that x is unknown and must be approximated . From the builder ’ s point of view , the reward – denoted r̃ – is unknown . This prevents the use of classical RL algorithms . Shared Intent and Interaction Frames . It follows from Eq . ( 1 ) that , provided that it can approximate the builder ’ s behavior , the architect can compute the reward and transition models of its MDP . It can then use these to derive an optimal message policy π∗ A that would maximize its objective : π∗ A = argmax π A G A = argmax π A E [ ∑ t γtr A , t ] ( 3 ) γ ∈ [ 0,1 ] is a discount factor and the expectation can be thought of in terms of π A , P A and the initial state distribution . However , the expectation can also be though in terms of the corresponding trajectories τ ≜ { ( s , m , a , r ) t } generated by the architect-builder interactions . In other words , when using π∗ A to guide the builder , the architect-builder pair generates trajectories that maximizes G A . The builder has no reward signal to maximize , yet , it relies on a shared intent prior and assumes that its objective is the same as the architect ’ s one : G B = G A = Eτ [ ∑ t γtr A , t ] = Eτ [ ∑ t γtr̃t ] ( 4 ) where the expectations are taken with respect to trajectories τ of architect-builder interactions . Therefore , under the shared intent prior , architect-builder interactions where the architect uses π∗ A to maximize G A also maximize G B . This means that the builder can interpret these interaction trajectories as demonstrations that maximize its unknown reward function r̃ . Consequently , the builder can reinforce the desired behavior – towards which the architect guides it – by performing self-Imitation Learning1 on the interaction trajectories τ . Note that in Eq . ( 1 ) , the architect ’ s models can be interpreted as expectations with respect to the builder ’ s behavior . Similarly , the builder ’ s objective depends on the architect ’ s guiding behavior . This makes one agent ’ s MDP highly non-stationary and the agent must adapts its behavior if the other agent ’ s policy changes . To palliate to this , agents rely on interaction frames which means that , when one agent learns , the other agent ’ s policy is fixed to restore stationarity . The equivalent MDPs for the architect and the builder are respectively M A = ⟨S , V , P A , r A , γ⟩ and M B = ⟨S × V , A , P B , ∅ , γ⟩ . Finally , π A : S 7→ V , P A : S×V 7→ [ 0 , 1 ] , r A : S×V 7→ [ 0 , 1 ] , π B : S×V 7→ A and P B : S ×V ×A 7→ [ 0 , 1 ] where S , A and V are respectively the sets of states , actions and messages .

Summary: The paper proposes an approach for interactive learning between a so-called architect and builder agent, which is a different, but related protocol to RL or imitation learning. Under the assumption that the architect knows the target dynamics and reward function, the authors focus on learning the communication between the two parties, such that the builder can solve the MDP.  After formalizing the setting in the multi-agent paradigm, where two individual MDPs are defined, the authors propose an algorithm with so-called modelling- and guiding phases. In each phase, architect and builder gather datasets, from which policies are extracted via behaviour cloning and planning.  The paper is evaluated on a proposed block environment for the problem and uses a random building agent and combination of random modelling phase with proposed guiding phase as baselines. The results show that the proposed approach is superior wrt the baselines for solving the block environment. In addition, experiments show that the learned communication channel can potentially be reused for solving other tasks.