Learning to Guide and to be Guided in the Architect-Builder Problem

1 INTRODUCTION . Humans are notoriously successful at teaching – and learning from – each others . This enables skills and knowledge to be shared and passed along generations , being progressively refined towards mankind ’ s current state of proficiency . People can teach and be taught in situations where there is no shared language and very little common ground , such as a parent teaching a baby how to stack blocks during play . Experimental Semiotics ( Galantucci & Garrod , 2011 ) , a line of work that ∗Equal contribution . †Work conducted while at Inria . ‡Canada CIFAR AI Chair . studies the forms of communication that people develop when they can not use pre-established ones , reveals that humans can even teach and learn without direct reinforcement signal , demonstrations or a shared communication protocol . Vollmer et al . ( 2014 ) for example investigate a co-construction ( CoCo ) game experiment where an architect must rely only on arbitrary instructions to guide a builder toward constructing a structure . In this experiment , both the task of building the structure and the meanings of the instructions – through which the architect guides the builder – are simultaneously learned throughout interactions . Such flexible teaching – and learning – capabilities are essential to autonomous artificial agents if they are to master an increasing number of skills without extensive human supervision . As a first step toward this research direction , we draw inspiration from the CoCo game and propose the Architect-Builder Problem ( ABP ) : an interactive learning setting that models agents ’ interactions with Markov Decision Processes ( Puterman , 2014 ) ( MDPs ) . In the ABP learning has to occur in a social context through observations and communication , in the absence of direct imitation or reinforcement ( Bandura & Walters , 1977 ) . Specifically , the constraints of the ABP are : ( 1 ) the builder has absolutely no knowledge about the task at hand ( no reward and no prior on the set of possible tasks ) , ( 2 ) the architect can only interact with the builder through communication signals ( can not interact with the environment or provide demonstrations ) , and ( 3 ) the communication signals have no pre-defined meanings ( nor belong to a set of known possible meanings ) . ( 1 ) sets this work apart from Reinforcement Learning ( RL ) and even Multi-Agent RL ( MARL ) where explicit rewards are available to all agents . ( 2 ) implies the absence of tele-operation or third-person demonstrations and thus distinguishes the ABP from Imitation and Inverse Reinforcement Learning ( IRL ) . Finally , ( 3 ) prevents the architect from relying on a fixed communication protocol since the meanings of instructions must be negotiated . These constraints make ABP an appealing setting to investigate Human-Robot Interaction ( HRI ) ( Goodrich & Schultz , 2008 ) problems where “ a learner tries to figure out what a teacher wants them to do ” ( Grizou et al. , 2013 ; Cederborg & Oudeyer , 2014 ) . Specifically , the challenge of Brain Computer Interfaces ( BCI ) , where users use brain signals to control virtual and robotic agents in sequential tasks ( Katyal et al. , 2014 ; deBettencourt et al. , 2015 ; Mishra & Gazzaley , 2015 ; Muñoz-Moldes & Cleeremans , 2020 ; Chiang et al. , 2021 ) , is well captured by the ABP . In BCIs , ( 3 ) is identified as the calibration problem and is usually tackled with supervised learning to learn a mapping between signals and meanings . As this calibration phase is often laborious and impractical for users , current approaches investigate calibration-free solutions where the mapping is learned interactively ( Grizou et al. , 2014 ; Xie et al. , 2021 ) . Yet , these works consider that the user ( i.e . the architect ) is fixed , in the sense that it does not adapt to the agent ( i.e . the builder ) and uses a set of pre-defined instructions ( or feedback ) meanings that the agent must learn to map to signals . In our ABP formulation however , the architect is dynamic and , as interactions unfold , must learn to best guide a learning builder by tuning the meanings of instructions according to the builder ’ s reactions . In that sense , ABP provides a more complete computational model of agent-agent or human-agent interaction . With all these constraints in mind , we propose Architect Builder Iterated Guiding ( ABIG ) , an algorithmic solution to ABP when both agents are AIs . ABIG is inspired by the field of experimental semiotics and relies on two high-level interaction priors : shared intent and interaction frames . Shared intent refers to the fact that , although the builder ignores the objective of the task to fulfill , it will assume that its objective is aligned with the architect ’ s . This assumption is characteristic of cooperative tasks and shown to be a necessary condition for the emergence of communication both in practice ( Foerster et al. , 2016 ; Cao et al. , 2018 ) and in theory ( Crawford & Sobel , 1982 ) . Specifically , the builder should assume that the architect is guiding it towards a shared objective . Knowing this , the builder must reinforce the behavior it displays when guided by the architect . We show that the builder can efficiently implement this by using imitation learning on its own guided behavior . Because the builder imitates itself , we call it self-imitation . The notion of interaction frames ( also called pragmatic frames ) states that agents that interact in sequence can more easily interpret the interaction history ( Bruner , 1985 ; Vollmer et al. , 2016 ) . In ABIG , we consider two distinct interaction frames . These are stationary which means that when one agent learns , the other agent ’ s behavior is fixed . During the first frame ( the modelling frame ) , the builder is fixed and the architect learns a model of the builder ’ s message-conditioned behavior . During the second frame ( the guiding frame ) , the architect is fixed and the builder learns to be guided via self-imitation learning . We show that ABIG results in a low-level , high-frequency , guiding communication protocol that not only enables an architect-builder pair to solve the task at hand , but can also be used to solve unseen tasks . Our contributions are : • The Architect-Builder Problem ( ABP ) , an interactive learning setting to study how artificial agents can simultaneously learn to solve a task and derive a communication protocol . • Architect-Builder Iterated Guiding ( ABIG ) , an algorithmic solution to the ABP . • An analysis of ABIG ’ s key learning mechanisms . • An evaluation of ABIG on a construction environment where we show that ABIG agents evolve communication protocols that generalize to unseen harder tasks . • A detailed analysis of ABIG ’ s learning dynamics and impact on the mutual information between messages and actions ( in the Supplementary Material ) . 2 PROBLEM DEFINITION . The Architect-Builder Problem . We consider a multi-agent setup composed of two agents : an architect and a builder . Both agents observe the environment state s but only the architect knows the goal at hand . The architect can not take actions in the environment but receives the environmental reward r whereas the builder does not receive any reward and has thus no knowledge about the task at hand . In this asymmetrical setup , the architect can only interact with the builder through a communication signal m sampled from its policy π A ( m|s ) . These messages , that have no a priori meanings , are received by the builder which acts according to its policy π B ( a|s , m ) . This makes the environment transition to a new state s′ sampled from P E ( s′|s , a ) and the architect receives reward r′ . Messages are sent at every time-step . The CoCo game that inspired ABP is sketched in Figure 1 ( a ) while the overall architect-builder-environment interaction diagram is given in Figure 1 ( b ) . The differences between the ABP setting and the MARL and IRL settings are illustrated in Figure 8 . BuildWorld . We conduct our experiments in BuildWorld . BuildWorld is a 2D construction gridworld of size ( w × h ) . At the beginning of an episode , the agent and Nb blocks are spawned at different random locations . The agent can navigate in this world and grasp blocks by activating its gripper while on a block . The action space A is discrete and include a “ do nothing ” action ( |A| = 6 ) . At each time step , the agent observes its position in the grid , its gripper state as well as the position of all the blocks and if they are grasped ( |S| = 3 + 3Nb ) . Tasks . BuildWorld contains 4 different training tasks : 1 ) ‘ Grasp ’ : The agent must grasp any of the blocks ; 2 ) ‘ Place ’ : The agent must place any block at a specified location in the grid ; 3/4 ) ‘ H-Line/Vline ’ : The agent must place all the blocks in a horizontal/vertical line configuration . BuildWorld also has a harder fifth testing task , ‘ 6-blocks-shapes ’ , that consists of more complex configurations and that is used to challenge an algorithm ’ s transfer abilities . For all tasks , rewards are sparse and only given when the task is completed . This environment encapsulates the interactive learning challenge of ABP while removing the need for complex perception or locomotion . In the RL setting , where the same agent acts and receives rewards , this environment would not be very impressive . However , it remains to be shown that the tasks can be solved in the setting of ABP ( with a reward-less builder and an action-less architect ) . Communication . The architect guides the builder by sending messages m which are one-hot vectors of size |V| ranging from 2 to 72 , see Suppl . Section B.3 for the impact of this parameter . Additional Assumptions . In order to focus on the architect-builder interactions and the learning of a shared communication protocol , the architect has access to P E ( s′|s , a ) and to the reward function r ( s , a ) of the goal at hand . This assumes that , if the architect were to act in the environment instead of the builder , it would be able to quickly figure out how to solve the task . This assumption is compatible with the CoCo game experiment ( Vollmer et al. , 2014 ) where humans participants , and in particular the architects , are known to have such world models . 3 ABIG : ARCHITECT-BUILDER ITERATED GUIDING . 3.1 ANALYTICAL DESCRIPTION . Agents-MDPs . In the Architect-Builder Problem , agents are operating in different , yet coupled , MDPs . Those MDPs depend on their respective point of view ( see Figure 2 ) . From the point of view of the architect , messages are actions that influence the next state as well as the reward ( see Figure 2 ( a ) ) . The architect knows the environment transition function P E ( s′|s , a ) and r ( s , a ) , the true reward function associated with the task that does not depend explicitly on messages . It can thus derive the effect of its messages on the builder ’ s actions that drive the reward and the next states ( see Figure 2 ( b ) ) . On the other hand , the builder ’ s state is composed of the environment state and the message , which makes estimating state transitions challenging as one must also capture the message dynamics ( see Figure 2 ( c ) ) . Yet , the builder can leverage its knowledge of the architect picking messages based on the current environment state . The equivalent transition and reward models , when available , are given below ( see derivations in Suppl . Section A ) . P A ( s′|s , m ) = ∑ a∈A π̃ B ( a|s , m ) P E ( s′|a , s ) r A ( s , m ) = ∑ a∈A π̃ B ( a|s , m ) r ( s , a ) with π̃B ( a|s , m ) ≜ P ( a|s , m ) ( 1 ) P B ( s′ , m′|s , m , a ) = π̃ A ( m′|s′ ) P E ( s′|s , a ) with π̃ A ( m′|s′ ) ≜ P ( m′|s′ ) ( 2 ) where subscripts A and B refer to the architect and the builder , respectively . x̃ denotes that x is unknown and must be approximated . From the builder ’ s point of view , the reward – denoted r̃ – is unknown . This prevents the use of classical RL algorithms . Shared Intent and Interaction Frames . It follows from Eq . ( 1 ) that , provided that it can approximate the builder ’ s behavior , the architect can compute the reward and transition models of its MDP . It can then use these to derive an optimal message policy π∗ A that would maximize its objective : π∗ A = argmax π A G A = argmax π A E [ ∑ t γtr A , t ] ( 3 ) γ ∈ [ 0,1 ] is a discount factor and the expectation can be thought of in terms of π A , P A and the initial state distribution . However , the expectation can also be though in terms of the corresponding trajectories τ ≜ { ( s , m , a , r ) t } generated by the architect-builder interactions . In other words , when using π∗ A to guide the builder , the architect-builder pair generates trajectories that maximizes G A . The builder has no reward signal to maximize , yet , it relies on a shared intent prior and assumes that its objective is the same as the architect ’ s one : G B = G A = Eτ [ ∑ t γtr A , t ] = Eτ [ ∑ t γtr̃t ] ( 4 ) where the expectations are taken with respect to trajectories τ of architect-builder interactions . Therefore , under the shared intent prior , architect-builder interactions where the architect uses π∗ A to maximize G A also maximize G B . This means that the builder can interpret these interaction trajectories as demonstrations that maximize its unknown reward function r̃ . Consequently , the builder can reinforce the desired behavior – towards which the architect guides it – by performing self-Imitation Learning1 on the interaction trajectories τ . Note that in Eq . ( 1 ) , the architect ’ s models can be interpreted as expectations with respect to the builder ’ s behavior . Similarly , the builder ’ s objective depends on the architect ’ s guiding behavior . This makes one agent ’ s MDP highly non-stationary and the agent must adapts its behavior if the other agent ’ s policy changes . To palliate to this , agents rely on interaction frames which means that , when one agent learns , the other agent ’ s policy is fixed to restore stationarity . The equivalent MDPs for the architect and the builder are respectively M A = ⟨S , V , P A , r A , γ⟩ and M B = ⟨S × V , A , P B , ∅ , γ⟩ . Finally , π A : S 7→ V , P A : S×V 7→ [ 0 , 1 ] , r A : S×V 7→ [ 0 , 1 ] , π B : S×V 7→ A and P B : S ×V ×A 7→ [ 0 , 1 ] where S , A and V are respectively the sets of states , actions and messages .

Summary: This paper presents an approach — Architect-Builder Iterated Guiding — a method that tackles what is presented as an “Architect-Builder” problem: a scenario in which two actors, an Architect, with knowledge of a high-level goal, or reward function, must communicate over a discrete channel with a Builder who can take actions in the environment based on the Architect’s message. In many ways this resembles a **hierarchical reinforcement learning** setup, specifically reminiscent of Feudal Reinforcement Learning (“Feudal Reinforcement Learning,” Dayan et. al. NeurIPS 1992). The paper presents a motivated, easy-to-understand algorithm for training both the Architect and the Builder, and evaluate on a series of “construction” based grid-world tasks (resembling GridLU, MiniGrid, or Mazebase).   Namely, the proposed approach takes inspiration from Experimental Semiotics and separates learning into separate **interaction frames** (similar again to multi-phase approaches in the Hierarchical RL literature, e.g. in HIRO — “Data Efficient Hierarchical Reinforcement Learning,” Nachum et. al. 2018; https://arxiv.org/abs/1703.01161). These interaction frames consist of a modeling frame, in which the architect learns a model of the builder after “rolling out” and sending messages/watching the builder’s actions, followed by a guiding frame where the architect exploits its model of the builder to produce the “optimal” actions via a heuristic driven Monte Carlo Tree Search (rather than explicitly estimate a value function, which is costly and noisy). Making this all possible is that the architect has full knowledge of the high-level reward, **in addition to the ground-truth state transition function**. This lets the architect explicitly search over messages to send to the builder (as it fully observes the builder’s rollout), then improve the builder by “self-imitating” over this new data, mirroring a bi-phase optimization setup.  The evaluation focuses on the proposed model, and two simple ablations: one where the architect has “no-intent” at training, sending random messages, and one in which the builder takes random actions. The paper also presents (in the main body and appendix) a meaningful, thoughtful intuitive explanation of the learning dynamics of the architecture and the builder — more papers should dedicate portions of the main body to explanations such as this!