Learning to Guide and to be Guided in the Architect-Builder Problem
Authors: Paul Barde
Tristan Karch
Derek Nowrouzezahrai
Clément Moulin-Frier
Christopher Pal
Pierre-Yves Oudeyer
Venue: ICLR 2022 Poster (2022)
Keywords: Social Learning
Interactive Learning
Teacher-Student Learning
Computational Experimental Semiotics
Socially Supervised Learning

Abstract: We are interested in interactive agents that learn to coordinate, namely, a $builder$ -- which performs actions but ignores the goal of the task, i.e. has no access to rewards -- and an $architect$ which guides the builder towards the goal of the task. 
We define and explore a formal setting where artificial agents are equipped with mechanisms that allow them to simultaneously learn a task while at the same time evolving a shared communication protocol.  
Ideally, such learning should only rely on high-level communication priors and be able to handle a large variety of tasks and meanings while deriving communication protocols that can be reused across tasks.
The field of Experimental Semiotics has shown the extent of human proficiency at learning from a priori unknown instructions meanings. Therefore, we take inspiration from it and present the Architect-Builder Problem (ABP): an asymmetrical setting in which an architect must learn to guide a builder towards constructing a specific structure. The architect knows the target structure but cannot act in the environment and can only send arbitrary messages to the builder. The builder on the other hand can act in the environment, but receives no rewards nor has any knowledge about the task, and must learn to solve it relying only on the messages sent by the architect. Crucially, the meaning of messages is initially not defined nor shared between the agents but must be negotiated throughout learning.
Under these constraints, we propose Architect-Builder Iterated Guiding (ABIG), a solution to the Architect-Builder Problem where the architect leverages a learned model of the builder to guide it while the builder uses self-imitation learning to reinforce its guided behavior. To palliate to the non-stationarity induced by the two agents concurrently learning, ABIG structures the sequence of interactions between the agents into interaction frames. We analyze the key learning mechanisms of ABIG and test it in a 2-dimensional instantiation of the ABP where tasks involve grasping cubes, placing them at a given location, or building various shapes. In this environment, ABIG results in a low-level, high-frequency, guiding communication protocol that not only enables an architect-builder pair to solve the task at hand, but that can also generalize to unseen tasks. 

equipped with Published as a conference paper at ICLR 2022
LEARNING TO GUIDE AND TO BE GUIDED IN THE SCCHTTCT-BUILDER PROBLEM
Paul B
Tristan Karch*
Derek Nowrouzezahrai
Québec AT institute (Mila) McGill University
Inria Flowers team Université de Bordeaux tristan, karch@inria.fr htgeeeeeeittta..rkeeeii......... fr
Québec Al institute (Mila)
McGill University
ba bardepauEmi la. quebec
Clément Moulin-Frier
Christopher Pal
Pierre-Y Oudeyer
Inria Flowers team Université de Bordeaux ENSTA ParisTech
Québec Al institute (Mila)
Inria Flowers team
Polythechnique Montréal ServiceNow Element AI
Univ. Bordeaux
Microsoft Research Montreal
ABSTRACT
We are interested interactive agents that learn to coordinate, namely, a builder which performs actions but ignores the goal of the task, i.e. has no access rewards - and an architect which guides the builder towards the goal of the task. We define and explore a formal setting where artificial agents mechanisms that allow them to simultancously Iearn a task while at the same time evolving shared communication protocol. Ideally, such learning should only rely on high-level communication priors and be able to handle a large variety tasks and meanings while deriving communication protocols that can be reused across tasks. The field of Experimental Semiotics has shown the extent of human proficiency learning from a priori unknown instructions meanings. Therefore, we take inspiration from it and present the Architect- BrchitettBBiilder Problem (ABP): an asymmetrical setting in which an architect must learn to guide a builder towards constructing specific structure. The architect knows the target structure but can- not act the environment and can only send arbitrary messages to the builder. The builder the other hand can act in the environment, but receives no rewards nor has any knowledge about the task, and must learn to solve it relying only on the messages sent by the architect. Crucially, the meaning of messages is initially not defined nor shared between the agents but must be negotiated throughout learning. Under these constraints, we propose Sechitect-Buildes Iterated Guiding (ABIG), a solution to the Architect-Builde Problem where the architect leverages a learned model of the builder to guide it while the builder reinforce its guided behavior. To palliate to the non-stationarity induced by the two agents concurrently learning, ABIG structures the sequence of interactions be- tween the agents into interaction frames. We analyze the key learning mechanisms of ABIG and test in a 2-dimensional instantiation of the ABP where tasks involve grasping cubes, placing them given location, or building various shapes. In this environment, ABIG results low-level, high-frequency, guiding communi- cation protocol that not only enables an cechitctt-buildee pair to solve the task at hand, but that can also generalize to unseen tasks.
self-imitation learning
INTRODUCTION
Humans are notoriously successful at teaching and learning from each others. This enables skills and knowledge to be shared and passed along generations, being progressively refined to- wards mankind's current state of proficiency. People teach and be taught in situations where there is no shared language and very little common ground, such as a parent teaching a baby how to stack blocks during play. Experimental Semioties (Galantucci & Garrod 2011). line of work that
"qqual contribution.
Work conducted while Inria. Canada CIFAR AT Chair.

---

Published as a conference paper at ICLR 2022
studies the forms of communication that people develop when they cannot use pre-established ones, reveals that humans can even teach and learn without direct reinforcement signal, demonstrations or shared communication protocol. Vollmer et al. (2014) for example investigate co-construction (CoCo) game experiment where an architect must rely only on arbitrary instructions to guide a builder toward constructing a structure. In this experiment, both the task of building the structure and the meanings of the instructions- - through which the architect guides the builder- are simul- taneously learned throughout interactions. Such flexible teaching and learning capabilities are essential to autonomous artificial agents if they are to master an increasing number of skills without extensive human supervision. As a first step toward this research direction, we draw inspiration from the CoCo game and propose the Architect-Builde Problem (ABP), an interactive learning setting that models agents' interactions with Markov Decision Processes (Puterman 2014) (MDPs). In the ABP learning has to occur in a social context through observations and communication, in the absence of direct imitation or reinforcement (Bandura & Walters 1977). Specifically, the constraints of the ABP are: (1) the builder has absolutely no knowledge about the task at hand (no reward and no prior on the set of possible tasks), (2) the architect can only interact with the builder through communication signals (cannot interact with the environment provide demonstrations), and (3) the communica- tion signals have no pre-defined meanings (nor belong to a set of known possible meanings). (1) sets this work apart from Reinforcement Learning (RL) and even Multi-Agent RL (MARL) where explicit rewards are available to all agents. (2) implies the absence of tele-operation or third-person demonstrations and thus distinguishes the ABP from Imitation and Inverse Reinforcement Learning (IRL), Finally, (3) prevents the architect from relying on fixed communication protocol since the meanings of instructions must be negotiated.
These constraints make ABP an appealing setting to investigate Sluman-Robor Interaction (HRI) (Goodrich & Schultz, 2008) problems where "a learner tries to figure out what a teacher wants them to do (Grizou et al. 2013] Cederborg & Oudeyer, (2014). Specifically, the challenge of Bruin Com- puter Interfaces (BCI), where users use brain signals to control virtual and robotic agents in sequen-
tial tasks (Katyal et al.) 2014 BeBeetencourt et al.) 2015 Mishra & Gazzaley 2015 MuldzsMMlles
& Cleeremans 2020; Chiang et al. (021). is well captured by the ABP. In BCIs, (3) is identified as the calibration problem and is usually tackled with supervised learning to learn a mapping between signals and meanings. As this calibration phase is often laborious and impractical for users, current approaches investigate calibration -free solutions where the mapping is learned interactively (Grizou let al. 2014 Xie et al., 2021). Yet, these works consider that the user (i.e. the architect) is fixed, in the sense that it does not adapt to the agent (i.e. the builder) and uses a set of pre-defined instructions (or feedback) meanings that the agent must learn to map to signals. In our ABP formulation however, the architect dynamic and, as interactions unfold, must learn to best guide a learning builder by tuning the meanings of instructions according to the builder's 's reactions. In that sense, ABP provides a more complete computational model of agent-agent or human-agent interaction.
With all these constraints in mind, we propose Architect Builder Iterated Guiding (ABIG), an al- gorithmic solution to ABP when both agents are Als. ABIG is inspired by the field of experimen- tal semiotics and relies on two high-level interaction priors: shared intent and interaction frames. Shared intent refers to the fact that, although the builder ignores the objective of the task to fulfill, it will assume that its objective is aligned with the architect's. s. This assumption is characteristic of cooperative tasks and shown to be a necessary condition for the emergence of communication both in practice (Foerster et al.) 2016 Cao et al. 2018) and in theory (Crawford & Sobel 1982). Specifi- cally, the builder should assume that the architect is guiding it towards a shared objective. Knowing this, the builder must reinforce the behavior it displays when guided by the architect. We show that the builder can efficiently implement this by using imitation learning on its own guided behavior. Because the builder imitates itself, we call called pragmatic,frame frames) states that agents that interact in sequence can more easily interpret the in- teraction history (Bruner, 1985; Vollmer et al. 2016). In ABIG, we consider two distinct interaction frames. These are stationary which means that when one agent learns, the other agent's behavior fixed. During the first frame (the modelling frame), the builder is fixed and the architect learns a model of the builder's ss cressage-cnnditoons behavior. During the second frame (the guiding frame), the architect is fixed and the builder learns to be guided via self-imitation learning.
ceff-imitttion. The notion of interaction frames (also
We show that ABIG results in low-level, high-frequency, guiding communication protocol that not only enables an architect- builder pair to solve the task at hand, but can also be used to solve unseen tasks. Our contributions are:
2

---

Published as a conference paper at ICLR 2022
The Sechitctt-Biidder Problem (ABP), interactive learning setting to study how artificial
agents can simultaneously learn to solve a task and derive a communication protocol.
Architect- Builder Iterated Guiding (ABIG), algorithmic solution to the ABP.
An analysis of ABIG's is key learning mechanisms.
construction environment where show that ABIG agents evolve communication protocols that generalize to unseen harder tasks.
An evaluation of ABIG
A detailed analysis of ABIG's learning dynamics and impact on the mutual information between messages and actions (in the Supplementary Material).
2 PROBLEM DEFINITION
The Sehitect--uuider Problem. We consider a multi-agent setup composed of two agents: an architect and builder. Both agents observe the environment state 8 but only the architect knows the goal at hand. The architect cannot take actions in the environment but receives the environmental reward r whereas the builder does not receive any reward and has thus no knowledge about the task at hand. In this asymmetrical setup, the architect can only interact with the builder through communication signal m sampled from its policy n (m]s). These messages, that have no priori meanings, received by the builder which acts according to its policy n, (als, m). This makes the environment transition to a new state s' sampled from Pj (s') s, a) and the architect receives reward r'. Messages are sent every time-step. The CoCo game that inspired ABP sketched in Figure [ while the overall differences between the ABP setting and the MARL and IRL settings are illustrated in Figure
interaction diagram is given in Figure [ The
Video
Pictacel
Dusas Sugee
Architect
A
Duilder
Ps
Builder
Buttonn
Architect
Scaon duplacing
Savnn dedeee
ypolads
RuildWold
(b)
Figure (a) Schematic view the CoCo Game (the inspiration for ABP). The architect and builder should collaborate in order to build the construction target while located in different rooms. The architecture has picture of the target while the builder has access to the blocks. The architect monitors the builder workspace via camera (video stream) and communicate with the builder only through the of 10 symbols (button events). (b) Interaction diagram between agents and the environment in our proposed ABP. The architect communicates messages (m) to the builder. Only the builder can act (a) in the environment. The
builder conditions its action on the message sent hy the huilder (Tg (a|s, m)). The builder never perceives any reward from the environment. A schematic view of the equivalent ABP problem is provided in Figure b))
BorldWorld. We conduct our experiments in BuildWorid. BuildWorld is 2D construction grid- world of size (w x h). At the beginning of an episode, the agent and N₂ blocks are spawned at different random locations. The agent can navigate in this world and grasp blocks by activating its gripper while on a block. The action space A is discrete and include a "do nothing" action (l.A) = 6). At each time step, the agent observes its position the grid, its gripper state as well as the position of all the blocks and if they are grasped ([S) = 3 3.².).
Tasks. BuildWorld contains different training tasks: 1) 'Grasp': The agent must grasp any of the blocks; 2) 'Placc': The agent must place any block at specified location in the grid; 3/4) 'H-Line/V. line': The agent must place all the blocks a horizomtal/vertical line configuration. BuildWorld also has harder fifth testing task, "6-blocks-shapp" that is used to challenge an algorithm's 's transfer abilities. For all tasks, rewards are sparse and only given when the task is completed.
that consists of more complex configurations and
This environment encapsulates the interactive learning challenge of ABP while removing the need for complex perception or locomotion. In the RL. setting, where the same agent acts and receives
3

---

Published as a conference paper at ICLR 2022
rewards, this environment would not be very impressive. However, it remains to be shown that the tasks can solved in the setting of ABP (with reward-less builder and an action-less architect).
Communication. The architect guides the builder by sending messages which one-hot vectors of size [V] rangiing from 2 to 72, see Suppl. Section B.3 for the impact of this parameter.
Additional Assumptions. In order focus the urchitect-buildes interactions and the learning of a shared communication protocol, the architect has access to P, (s'ls, a) and to the reward function r(s, of the goal at hand. This assumes that, if the architect were to act in the enviromment instead of the builder, would be able to quickly figure out how to solve the task. This assumption is compatible with the CoCo game experiment (Vollmer et al. 2014) where humans participants, and in particular the architects, are known to have such world models.
3 ABIG: ARCHITECT-BUILDER ITERATED GUIDING
3.1 ANALYTICAL DESCRIPTION
R
(a) Architect MDP
(b) Implicit Architect MDP
R
M
M
*8
(c) Builder MDP
(d) Implicit Builder MDP
Figure 2: Agent's Markov Decision Processes. Highlighted regions refer to MDP coupling. (a) The archi- lect's transitions and rewards conditioned by the builder's policy Tg- (b) Architect's MDP where transition and reward models implicitly account for builder's behavior. (c-d) The builder's transition model depends on the architect's 's message policy π The builder's ' learning signal r is unknown.
MDPs In the Architect-Builder Problem, agents are operating in different, yet coupled, MDPs. Those MDPs depend on their respective point of view (see Figure 2 From the point of view of the architect, messages Figure 2 (a)). The architect knows the environment transition function P. (s'|s, a) and r(s,a), the true reward function associated with the task that does not depend explicitly on messages. It can thus derive the effect of its messages on the builder's actions that drive the reward and the next states (see Figure (b)). On the other hand, the builder's state is composed of the environment state and the message, which makes estimating state transitions challenging as one must also capture the message dynamics (see Figure (c)). Yet, the builder can leverage its knowledge of the architect picking messages based on the current environment state. The equivalent transition and reward models, when available, are given below (see derivations Suppl. Section A).
actions that influence the next state as well as the reward (see
(861)), Σ (als, (8'1a,
aed
with
(als, m) P(als,m)
r2 (s, # (als, Σ acd
(2) where subscripts A and B refer the architect and the builder, respectively. & denotes that is unknown and must be approximated. From the builder's point of view, the reward denoted 7 is unknown. This prevents the use of classical RI. algorithms.
(s', m'ls m) (m')s (s's (s' s) with (m')s') & P(m'ls')
4

---

Published as a conference paper at ICLR 2022
Shared Intent and Interaction Frames. It follows from Eq. ① that, provided that it approxi- mate the builder's behavior, the architect can compute the reward and transition models of its MDP. It can then use these to derive an optimal message policy Ti that would maximize its objective:
argmax GA argmax E
(3)
y E [0.1] discount factor and the expectation can be thought of in terms of T Pa and the initial state distribution. However, the expectation can also be though in terms of the corresponding trajectories 7 & {(s. m. a,r), generated by the sechitect-builde interactions. In other words, when using n to guide the builder, the schitect-builder pair generates trajectories that maximizes G. The builder has reward signal to maximize, yet, it relies shared intent prior and assumes that objective the same as the architect's one:
G -
G = Σ - Σ 7'rr]
(4)
where the expectations taken with respect trajectories of architect-builder interactions. Therefore, under the shared intent prior, architect-builder interactions where the architect uses πr to maximize G2 also maximize G3. This means that the builder can interpret these interaction trajecto- ries as demonstrations that maximize its unknown reward function F. Consequently, the builder can reinforce the desired behavior towards which the architect guides by performing self-Imitation Learning on the interaction trajectories т.
Note that in Eq. (I]) the architect's models can be interpreted as expectations with respect to the builder's behavior. Similarly, the builder's objective depends the architect's guiding behavior. This makes one agent's MDP highly non-stationary and the agent must adapts behavior the other agent's policy changes. To palliate to this, agents rely interaction frames which means that, when one agent learns, the other agent's policy is fixed to restore stationarity. The equivalent MDPs for the architect and the builder
respectively M, (S.V. TA. 7) and Ma (S x V.A. P, P,,0, r). Finally, T, S + S V, P. S x ~> [0, 1]. r, SxV [0, 1]. Th S ト A and : SxVxA [0, 1] where S, and V are respectively the sets of states, actions and messages.
3.2 PRACTICAL ALGORITHM
Modelling Frame
Architect
Vnifocua))
Architect
((s m...a. (1
BC ng
Builder
Beloctor modellnn
Collect
Builder
BC
F.A. Planring
Z
Inputs
Replace model
Self-iiitatio
BulldWorld
Train model
Gaiding Frame
Figure 3: Architect-Buidde Iterated Guiding, Agents iteratively interact through the modelling and guiding frames. In each frame, one agent collects data and improves its policy while the other agent's 's hehavior is fixed.
ABIG iteratively structures the interactions between builder-archiine pair into interaction frames. Each iteration starts with modelling frame during which the architect learns model of the builder. Directly after, during the guiding frame, the architect leverages this model to produce messages that
not to be confused with Oh et al. 2018) which is off-policy actor-critic algorithm promoting exploration single-agent RI.
5

---

Published as a conference paper at ICLR 2022
guide the builder. On its side, the builder stores the guiding interactions to train and refine its policy Tn. The interaction frames are described below. The algorithm is illustrated in Figure and the pseudo-code is reported in Algorithm in Suppl. Section A.3
Modelling Frame. The architect records a data-set of interactions Dj _ {(s, m, a, ))) by sending random messages m to the builder and observing its reaction. After collecting enough interactions, the architect learns model of the builder 7e using Behavioral Cloning (BC) (Pomerleau 1991).
Guiding Frame. During the guiding frame, the architect observes the environment states s and pro- duces messages so maximize its return (see Eq. 3). The policy of the architect is Monte Carlo Tree Search Algorithm (MCTS) (Kocsis & Szepesvári 2006) that searches for the best message by simulating the reaction of the builder using "n (,,)) alongside the dynamics and reward models. During this frame, the builder stores the interactions in buffer DB - {(s, m. a, s')) )') At the end of the guiding frame, the builder self-imitates by updating its policy Tr with BC on D,,
Practical Considerations. All models are parametrized by two-hidden layer 126-units feedforward ReLu networks. BC minimizes the cross-entropy loss with Adam optimizer (Kinmma & Ba 2015).
Networks are re-initialized before each BC training. The architect's MCTS uses Upper-Confidence bound for Trees and relies on heuristics rather than Monte-Carlo rollouts to estimate the value of states. For more details about training, MCTS and hyper-parameters please see Suppl. Section A..3
The resulting method (ABIG) is general and can handle variety of tasks while not restricting the kind of communication protocol that can emerge. Indeed, it only relies on a few high-level priors, namely, the architect's access to environment models, shared intent and interaction framess
In addition to ABIG we also investigate two control settings: ABIG -no-intent - the builder interacts with architect that disregards the goal and therefore sends random messages during training. At evaluation, the architect has access to the exact model of the builder (म = n₂) and leverages to guide it towards the evaluation goal (the architect no longer disregards the goal). And random the builder takes random actions. The comparison between ABIG and ABIG-no-inten measures the impact of doing self-imitation on guiding versus non-guiding trajectories. The random baseline is used to provide a performance lower bound that indicates the task's difficulty.
3.3 UNDERSTANDING THE LEARNING DYNAMICS
Architect-Builde Iterated Guiding relies on two steps. First, the architect selects favorable mes- sages, i.c. messages that maximize the likelihood of the builder picking optimal actions with respect to the architect's reward. Then, the builder does self-imitation and reinforces the guided behav- ior by maximizing the likelihood of the corresponding messages-actions sequence under its policy. The message-to-actte Maximum likelihood assumes that actions are initially equiprobable for a given message. Therefore, actions under a message that is not present in the data-set (D,,) remains so. In other words, the builder never observes message, it assumes that this message is equally associated with all the possible actions. This enables the builder to forget past message-to-action associations that are not used and thus not reinforced by the architect. In practice, initial uniform likelihood is ensured by resetting the builder's policy network before each self-imitation. The architect can leverage the forget mechanism to erase unfavorable associations until a favorable one emerges. Such favorable associations can then be reinforced by the architect-builde pair until it is made deterministic. The reinforcement process of favorable associations is also enabled by the self-imitation phase. Indeed, for given message m, the self- imitation objective for π on data-set D collected using is:
associations (or preferences) are encoded in the builder's policy Tg (als, m).
J (m. T) = Σ log (a/m) Re Ea log π (alm)) ~ [[[(-|m)]
(5)
where H stands for the entropy of distribution. Therefore, maximizing the likelihood in this case results in minimizing the entropy of rr([m) and thus reinforces the associations between messages and actions. Using these mechanisms the architect can adjust the policy of the builder until it be- comes controllable, i.e. deterministic (strong preferences over actions for given message) and flexible (varied preferences across messages). Conversely, in the case of ABIG-no-intenl the archi- tect does not guide the builder and simply sends messages random. Favorable and unfavorable messages are thus sampled alike which prevents the forget mechanism to undo unfavorable message- to-action associations. Consequently in that case, self-imitation tends to simply reinforce initial

---

Published as a conference paper at ICLR 2022
builder's preferences over actions making the controllability of the builder policy depend heavily on the initial preferences. We illustrate the above learning mechanisms in Suppl. Section A.4 by ap- plying ABIG to a simple instantiation of the ABP. Figure Mand Figure [ confirm that ABIG uses the forget and reinforcement mechanisms to circumvent the unfavorable initial conditions while ABIG- no-intent simply reinforces them. Eventually, Figure TT reports that ABIG always reaches 100% success rate regardless of the initial conditions while ABIG-no-intent success rate depends on the initial preferences (only 3% when they are unfavorable).
Interestingly, the emergent learning mechanisms discussed here are reminiscent of the amplification and ooff-enforeement of random fluctuations in naming games (Steels) 1995). In naming games how- ever, the self oreaansgannntnonn of vocabularies are driven by each agent maximizing its communicative success whereas our case the builder has no external learning signal and simply self-imitates.
4 RELATED WORK
This work is inspired by experimental semiotics (Galantucci & Garrod Vollmer et al. (2014) that studied the CoCo game with human subjects as a key step towards under- standing the underlying mechanisms of the emergence of communication. Here we take comple- mentary approach by defining and investigating solutions to the ABP, a general formulation of the CoCo game where both agents are Als.
2011) and in particular
Recent MARL work (Lowe et al. 2017 Woodward et al. 2020 Roy et al. 2020; Ndousse et al,, 2021), investigate how RL agents trained in the presence of other agents leverage the behaviors they observe to improve learning. In these settings, the other agents are used to build useful representation or gain information but the main learning signal of every agent remains ground truth reward.
Feudal Learning (Dayan & Hinton 1992 Kulkarni et al. 2016; Vezhnevets et al. 2017; Nachum al. 2018 Ahilan & Dayan 2019) investigate setting where manager sets the rewards of workers to maximize its own return. In this Hierarchical setting, the manager interacts by directly tweaking the workers' learning signal. This would be unfeasible for physically distinct agents, hence those methods are restricted to singlc-agent learning. On the other hand, ABP considers separate agents, that must hence communicate by influencing each other's observations instead of rewards signals.
Inverse Reinforcement Learning (IRL) (Ng et al. 2000) and Imitation Learning (IL) (Pomerleau 1991] have been investigated for HRI when it challenging to specify a reward function. Instead of defining rewards, IRI. and II. rely on expert demonstrations,
Maddield-Menelle et al. (2016) argue that learning from expert demonstrations not always optimal and investigate how to produce instructive demonstrations to best teach an apprentice. Crucially, the expert is aware of the mechanisms by which the apprentice learns, namely RL on top of IRL. This allows the expert to assess how its demonstrations influence the apprentice policy, effectively reducing the problem to a single agent POMDP. In our case however, the architect and the builder do not share the same action space which prevents the architect from producing demonstrations. In addition, the architect ignores the builder's learning process which makes the simplification to a single agent teacher problem impossible.
In essence, the ABP is closest works tackling the calibration-fres BCI control problem (Grizou let al. 2014] Xie et al., 202T). Yet, these works both consider that the architect sends messages after the builder's s actions and thus enforce that the feedback conveys a reward. Crucially, the architect does not learn and communicates with a fixed mapping between feedback and pre-defined meanings ("correct" vs. "wrong"). Those meanings are known the builder and simply has to learn the mapping between feedback and meaning. In our case however, the architect communicates before the builder's s action and thus rather gives instructions than feedback. Additionally, the builder has no priori knowledge of the set of possible meanings and the architect adapts those to the builder's reaction. Finally, Grizzu et al.) (2013) handles both feedback and instruction communications but relies on known task distribution and set of possible meanings. In terms of motivations, previous works are interested in one robot figuring out a fixed communication protocol while we train two agents to collectively emerge one.
Our World resembles GridLU proposed by Bahdanau et al. (009 to analyze reward modelling in maaguage-conditione learning. However, their setting is fundamentally different to ours as it in- vestigates single agent goal-conditioned IL where goals are predefined episodic linguistic instruc- tions labelling expert demonstrations. Nguyen et al. (222) alleviate the need for expert demonstra-
7

---

Published as a conference paper at ICLR 2022
tions by introducing interactive teacher that provides descriptions of the learning agent's trajecto- ries. In this HRI setting, the teacher still follows fixed pre-defined communication protocol known by the learner: messages Collahorative Building Task (Narayan- et al., 2019) and the IGLJU competition (Kiseleva et al.) 202T]; however, they do not consider emergent communication. Rather, they focus generating chitect utterances by leveraging human-humm dialogues corpus to learn pre-established meanings expressed in natural language. Conversely, in ABP both agents learn and must evolve the meanings of messages while solving the task without relying on any form of demonstration.
activity descriptions. Our ABP formulation relates to the Minecraft
5 RESULTS
In the following sections, success rates (sometimes referred as scores) are averaged over 10 random seeds and error bars are 22SEM with SEM the Standard Error of the Mean. If not stated otherwise, the grid size is (5 x 6), contains three blocks (N, 3) and the vocabulary size is [V] = 18.
ABIG'S learning performances. We apply ABIG the four learning tasks of BuildWorld and compare it with the two control settings: ABIG-no-intent (no guiding during training) and random (builder takes random actions). Figure reports the mean success rate on the four tasks defined in Section
2 First, we observe that ABIG significantly outperforms the control conditions Second, we notice that on the simpler "grasp" task ABIG-no-intent achieves a satisfactory mean score of 0.77=0.03. This is consistent with the learning dynamic analysis provided in Sup. Section A.4 that shows that, in favorable settings, a self- iifittiitt builder can develop a reasonably controllable policy (defined in Section 3.3) even if it learns on non-guiding trajectories. Nevertheless, when the tasks get more complicated and involve placing objects or drawing lines, the performances of ABIG- no-intent drop significantly whereas ABIG continues achieve high success rates (> 0.8). This demonstrates that ABIG enables builder-architect pair to successfully agree on communication protocol that makes the builder's policy controllable and enables the architect to efficiently guide it.
all tasks.
Mosn score (10 seeds)
grosp
place
line
Viiro
Figure 4: Methods performances (stars indicate significance with respect to ABIG model according to Welch's (-test with null hypothesis μ1 μ2, at level a 0.05). ABIG outperforms control baselines all goals.
ABIG's transfer performances. Building upon previous results, we propose study whether learned communication protocol can transfer to new tasks. The architect-builder pairs are trained on single task and then evaluated without retraining on the four tasks. In addition, we include 'all-goals', control setting in which the builder learns a single policy by being guided on all four goals during training. Figure shows that, on all training tasks except 'grasp', ABIG enables transfer performance above 0.65 on all testing tasks. Notably, training on 'place' results in robust communication protocol that can be used to solve the other tasks with a success rate above 0.85, being effectively equivalent training 'all-goals' directly. This might be explained by the fact that placing blocks at specified locations an atomic operation required to build lines.
lesting
gropp
Vilnn
Mesn scoe [1 seeds)
1.0
08
0.
grapp
300
lie
Vinne
al gogs
Training Goal
Figure 5: ABIG transfer performances without retraining depending the training goal. ABIG agents learn communication protocol that transfers new tasks. Highest performances reached when training on 'place".
8

---

Published as a conference paper at ICLR 2022
Challenging ABIG's transfer abilities. Motivated by ABIG's transfer performances, propose to train it on the 'place' task in a bigger grid (6 x 6) with N₂ = 6 and | = 72. Then, with- out retraining, given Figure [ The training performance on 'place' is 0.96 + 0.02 and the transfer performance on the "6-block-shapes" is 0.85 + 0.03. This further demonstrates ABIG's s ability derive robust communication protocols that can solve more challenging unseen tasks.
evaluate it the '6-block-shapes task that consists in constructing the shapes
C
Figure 6: 6-block-shapes that ABIG can construct in transfer mode when trained on the 'place' task.
Additional experiments. The Supplementary Materials contains the following experiments:
Figures 9 10 and II analyse the builder's message- cassageet--actton preferences. They illustrate
ABIG's learning mechanisms (forget and reinforce) and compare them to ABIG-no-intentss Figure shows that, as the communication protocol settles, the message-action mutual information becomes greater than the state-action mutual information which desirable feature for the emergence of communication.
Figure reports ABIG outperforming complementary baselines.
Figure 14 shows ABIG's performance increasing with the vocabulary size, suggesting that with more messages available, the architect can more efficiently refer to the desired action.
DISCUSSION AND FUTURE WORK
This work formalizes the ABP reinforcement, demonstrations or a shared language. To tackle ABP, we propose ABIG: an algorithm allowing learn how to guide and to be guided. ABIG is based only on two high-level priors to com- munication emergence (shared intent and interactions frames). ABP's general formulation allows us to formally enforce those priors during learning. We study their influence through ablation studies, highlighting the importance of shared intent achieved by doing self-imitation on guiding trajectories. When performed in interaction frames, this mechanism enables agents evolve communication protocol that allows them to solve all the tasks defined in BuildWorld. More impressively, we find that communication protocols derived on simple task can be used solve harder, never-seen goals.
interactive setting where learning must occur without explicit
Our approach has several limitations which open different opportunities for further work. First, ABIG trains agents a stationary configuration which implies doing several interaction frames. Each interaction frame involves collecting numerous transitions. Thus, ABIG is not data efficient. A challenging avenue would to relax this stationarity constraint and have agents learn from buffers containing non-stationary data with obsolete agent behaviors. Second, the builder remains depen- dent on the architect's messages even convergence. Using Vygotskian approach (Colas et al.) 2020 2021), the builder could internalize the guidance from the architect become autonomous in the task. This could, for instance, be achieved by having the builder learn model of the architect's message policy once the communication protocol has converged.
Because we present the first step towards interactive agents that learn in the ABP, our method uses simple tools (feed-forward networks and self-imitation learning). It is however important note that our proposed formulation of the ABP can support many different research directions. Experimenting with agents' models could allow for the investigation of other forms of communication. One could, for instance, include memory mechanisms in the models of agents in order to facilitate the emergence of retrospective feedback, form of emergent communication observed in Vollmer et al. (2014). ABP is also compatible with low-frequency feedback. As further experiment in this direction, one could penalize the architect for sending messages and assess whether a pair can converge to legeer-level meanings. Messages could also be composed of several tokens in order to allow for the emergence of compositionality. Finally, our proposed framework can serve as testbed to study the fundamental mechanisms of emergent communication by investigating the impact of high level communication priors from experimental semiotics.
For rollouts see wttttttt////tte http://///.....oooo..o.//////aaa http://ttt//ao///oooooo//ooooo/t 1ww.buuudderbbllllleeee

---

Published as a conference paper at ICLR 2022
7 ETHICS STATEMENT
This work investigates novel interactive approach to autonomous agent learning and proposes fundamental learning setting. This work does not directly present sensitive applications data. While the implications of autonomous agents learning are not trivial we do not aim here at dis- cussing the general impact of autonomous agents. Rather, we focus on discussing how the proposed learning setting contrasts with more classical supervisions such as reward signals and demonstra- tions. By proposing an iterative and interactive learning setting, the Architect-Builde Problem (ABP) promotes finer control over learned behavior than designing rewards or demonstrations. In- deed, the behavior is constantly evaluated and refined throughout learning as interactions unfold. Still, in this process it is essential to keep in mind the importance of the architect it is the agent that judges if the learned behavior is satisfactory.
REPRODUCIBILITY STATEMENT
We ensure the reproducibility of the experiments presented in this work by providing our code 3 Additional information regarding the methods and hyper-parameters can be found in Section 3.2 and in the Suppl. Section A.3 We ensure the statistical significance of our experimental results by using 10 random seeds, reporting the standard error of the mean and using Welch's t-test. Finally, we propose complete analytical derivations in Suppl. Section
ACKNOWLEDGMENT
The authors thank Erwan Lecarpentier for valuable advice on Monte-Carlo Tree Search as well as Compute Canada and GENCI-IDRIS (Grant 2020-A009100 1996) for providing computing re- sources. Derek and Paul acknowledge support from the NSERC Industrial Research Chair pro- gram. Tristan Karch is partly funded by the French Ministère des Armées Direction Générale de I' l'Armement.
REFERENCES
Sanjeevan Ahilan and Peter Dayan. Feudal multi-agent hierarchies for cooperative reinforcement learning. arXiv preprint arXiv: (0t:::901.0892, 2019.
Dzmitry Bahdanau, Felix Hill, Jan Leike, Edward Hughes, Pushmeet Kohli, and Edward Grefen- stette. Learning to understand goal specifications by modelling reward. International Confer- ence on Learning Representations, 2019.
Albert Bandura and Richard H Walters. Social learning theory, volume Englewood cliff's Prentice Hall, 1977.
Jerome Bruner. Child's talk: Learning to use language. Child Language Teaching and Therapy, (1):111-114, 1985.
Kris Cao, Angeliki Lazaridou, Mare Lanctot, Joel Leiboo Karl Tuyls, and Stephen Clark. Emergent communication through negotiation. In International Conference Learning Representations, 2018.
Thomas Cederborg and Pierre-Y Oudeyer. A social learning formalism for learners trying to figure out what a teacher wants them to do. Paladyn: Journal of Behavioral Robotics, 5:64-99, 2014.
Kuan-Jung Chiang, Dimitra Emmanouilidou, Hannes Gamper, David Johnston, Mihai Jalobeanu, Edward Cutrell, Andrew Wilson, Winko W. An, and Ivan Tashev. A closed-loop adaptive brain- computer interface framework: Improving the classifier with the use of error-related potentials. In 2021 10th International JEEE/EMBS Conference Neural Engineering (NER), pp. 487-490, 2021. doi: 10.1109/NER449883.221.....
com http:///ttttot..ooooooeeeeeettte bbb
10

---

Published as a conference paper at ICLR 2022
Cédric Colas, Tristan Karch, Clément Moulin-Frier, and Pierre-Yver Oudeyer. Cognitive Tool: Dall-E, Humans and Vygotskian RL Agents, March 2021. URL https:
Language
Cédric Colas, Tristan Karch, Nicolas Lair, Jean-Michel Dussoux, Clément Moulin-Free Frier, F Peter Dominey, and Pierre-Yves Oudeyer. Language cognitive tool to imagine goals in curiosity driven exploration. NeurlPS, 2020.
Vincent P Crawford and Joel Sobel. Strategic information transmission. Econometrica: Journal of the Econometric Sociery, pp. 1431-1451, 1982.
Peter Dayan and Geoffrey E. Hinton. Feudal reinforcement learning. In Advances in Neural Infor- mation Processing Systems 5, [NIPS ConferenceJ, pp. 271-278, San Francisco, CA, USA, 1992.
Morgan Kaufmann Publishers Inc. ISBN 1558602747.
Megan T. deBettencourt, Jonathan D. Cohen, Ray F. Lee, Kenneth A. Norman, and Nicholas B. Turk-Browne. Closed-loop training of attention with real-time brain imaging. Nature neuro- science, 18:470 475, 2015.
Jakob Foerster, Ioannis Alexandros Assael, Nando de Freitas, and Shimon Whiteson. Learning to communicate with deep multi-agent reinforcement learning. In Advances in Neural Information Processing Systems, pp. 2137-2145, 2016.
Bruno Galantucci and Simon Garrod. Experimental semiotics: review. Frontiers in human neuro- science, 5:11, 2011.
Michael A Goodrich and Alan C Schultz. Human-robot interaction: a survey, Now Publishers Inc, 2008
Jonathan Grizou, Manuel Lopes, and Pierre Yves Oudeyer. Robot learning simultancously task and how to interpret human instructions. In 2013 IEEE Third Joint International Conference on Development and Learning and Epigenetic Robotics (ICDL), pp. 1-8, 2013. doi: 10.1
DevL.m.200 20...66525622..
Jonathan Grizou, Iñaki Iturrate, Luis Montesano, Pierre-Yver Oudeyer, and Manuel Lopes. Calibration-Free BCI Based Control. In Twenty-Fighth AAAI Conference on Artificial Intelli- gence, pp. 1-8, Quehec, Canada, July 2014. URL Er/hal 00984068
Dylan Hadfield-Menell, Stuart Russell, Pieter Abbeel, and Anca Dragan. Cooperative inverse reinforcement learning. Advances in neural information processing systems, 29:3909-3917, 2016.
Kapil D. Katyal, Matthew S. Johannes, Spencer Kellis, Tyson Aflalo, Christian Klaes, Timo- thy McGee, Matthew P. Para, Ying Shi, Brian Lee, Kelsie Pejsa, Charles Liu, Brock A. Wester, Francesco Tenore, James D. Beaty, Alan D. Ravitz, Richard A. Andersen, and Michael P. McLoughlin. A collaborative bei approach to autonomous control a prosthetic limb system. In 2014 IEEE International Conference on Systems, Man, and Cybernetics (SMC), pp. 1479-1482, 2014. doi: 10. 11110///MC..201469941111
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015.
Julia Kiseleva, Ziming Li, Mohammad Aliannejadi, Shrestha Mohanty, Maartje ter Hoeve, Mikhail Burtsev, Alexey Skrynnik, Artem Zholus. Aleksandr Panov, Kavya Srinet, et al. Neurips 2021 competition iglu: Interactive grounded language understanding in a collaborative environment. arXiv preprint 20 2021.
Levente Koesis and Csaba Szepesvári. Bandit based monte-carlo planning, In European conference on machine learning, pp. 282-293. Springer, 2006.
Tejas D Kulkamn, Karthik Narasimhan, Ardavan Saeedi, and Josh Tenenbaum. Hierarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation. Advances in neural information processing systems, 29:3675-3683, 2016.

---

Published as a conference paper at ICLR 2022
Ryan Lowe, YI WU, Aviv Tamar, Jean Harb, OpenAl Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cooperative-compettii environments. Advances Neural Information Processing Systems, 30:6379-6333 6390, 2017.
Jyoti Mishra and Adam Gazzaley. Closed-loop cognition: the next frontier arrives. Trends in Cog- nitive Sciences, 19:242-243, 2015.
Santiago Muñoz-Moldes and Axel Cleeremans. Delineating implicit and explicit processes in
neurofeedback learning. Neuroscience & Biobehavioral Reviews, 118:681-688 2020. ISSN
0149-7634. doi: http://///...a.a.a.ttta////////. sciencedirect. com/
003. URL https::/wwwt //www.
Ofrr Nachum, Shixiang Shane Gu, Honglak Lee, and Sergey Levine. Data-efficient hierarchical reinforcement learning. Advances in Neural Information Processing Systems, 31:3303-3313, 2018.
Anjali Narayan-Chen, Chen, Prashant Jayannavar, and Julia Hockenmaier. Collaborative dialogue in minecraft. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 5405-5415, 2019.
Kamal K Ndousse, Douglas Eck, Sergey Levine, and Natasha Jaques. Emergent social learning via multi-agent reinforcement learning. In International Conference on Machine Learning. pp- 7991-8004. PMLR, 2021.
Andrew Y Ng. Stuart Russell, et al. Algorithms for inverse reinforcement learning. In International Conference on Machine Learning, volume 1, pp. 2, 2000.
Khanh Nguyen, Dipendra Misra, Robert Schapire, Miro Dudík, and Patrick Shafto. Interactive learning from activity description, 2021.
Junhyuk Oh, Yijie Guo, Satinder Singh, and Honglak Lee. Self-imitation learning. In ICML, 2018.
Dean A Pomerlcau. Efficient training of artificial neural networks for autonomous navigation. Neu- ral computation, 3(1):88-97, 1991.
Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming. John Wiley Sons, 2014.
Julien Roy, Paul Barde, Félix Harvey, Derek Nowrouzezahrai, and Chris Pal. Promoting coordi- nation through policy regularization in multi-agent deep reinforcement learning. In Advances in Neural Information Processing Systems, volume 33, pp. 15774-15785, 2020.
Lue Steels. A self-organizing spatial vocabulary. Artificial life. 2(3):319-3333 1995.
Alexander Sasha Vezhnevets, Simon Osindero, Tom Schaul, Nicolas Heess, Max Jaderberg, David Silver, and Koray Kavukcuoglu. Feudal networks for hierarchical reinforcement learning. In International Conference Machine Learning, pp. 3540-3549, PMLR, 2017.
Anna-Lisa Vollmer, Jonathan Grizou, Manuel Lopes, Katharina Rohlfing, and Pierre-Yvel Oudeyer. Studying the co-construction interaction protocols collaborative tasks with humans. In 4th International Conference on Development and Learning and on Epigenetic Robotics, pp. 208- 215. IEEE, 2014.
Anna-Lisa Vollmer, Britta Wrede, Katharina J Rohlfing, and Pierre-Y Oudeyer. Pragmatic frames for teaching and learning in human-robot interaction: Review and challenges. Frontiers in neu- rerobotics, 10:10, 2016.
Mark Woodward, Chelsea Finn, and Karol Hausman. Learning to interactively learn and assist. In Proceedings of the AAAI conference artificial intelligence, volume 34, pp.
2020.
Tengyang Xie, John Langford, Paul Mineiro, and Ida Momennejad. Interaction-groundee learning. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Ma- chine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 11414-11423. PMLR. 18-24 Jul 2021. URL [ntpps: httaas//ppcceeeding.. mlr. cxress/139/xiee.
html
12

---

Published as a conference paper at ICLR 2022
SUPPLEMENTARY MATERIAL
This Supplementary Material provides additional derivations, implementation details and results. More specifically:
Section A proposes derivations, implementation details and analysis related to our method.
Subsection A.
provides additional diagrams illustrating the ABP problem and its position with respect to related settings.
Subsection A.2 proposes the full derivation of the agents' MDP.
Subsection A.
proposes our methods pseudo-code, algorithmic implementation de-
tails (for BC and MCTS), hyper-parameters and compute resources.
Subsection A.4 proposes analysis that explore method's learning mechanisms.
Subsection A.5 discusses the differences between ABP and Hierarchical/Feudal Rein-
forcement Learning.
Section BattiionnBprovvidee additional results.
Subsection B.1
monitors the builder's behavior properties training progresses.
Subsection 2
compares our method to additional baselines.
Subsection B analyses the impact of the vocabulary size the learning perfor-
mance.
A SUPPLEMENTARY METHODS
A.1 SUPPLEMENTARY SKETCHES
Video stream
Pitur creamactio sarge
wollppo
Builder
Rutton
Architect
Screnn dapbeing
digiaying
Architect
10.00..0
0
W
Architect
0D
Pr
Builder
BuildWorld
(b)
(c)
Figure (a) Schematic view the CoCo Game. The architect the builder should collaborate order to build the construction target while located in different rooms. The architecture has a picture of the target while the builder has access the blocks. The architect monitors the builder workspace via camera (video stream) and communicate with the builder only through the of 10 symbols (button events). (b) Schematic view architect must
how to use messages guide builder while the builder needs to learn to make sense of the messages in order to be guided by the architect. (c) Interaction diagram between the agents and the environment in our proposed ABP. The architect communicates mes- sages (m) to the builder. Only the builder can (a) in the environment. The builder conditions its action the message sent by the builder ( (a|s, m)). The builder never perceives any reward from the environment
of Architect-Builde Problem.
13

---

Published as a conference paper at ICLR 2022
Architect
Architect AA
AA
Architect
Builder
m
Builder
P's
Ps
BuilWorld
BuilWorld
BuildWordd
(a)
(b)
(c)
Figure (a) Vertical view of the interaction diagram between the agents and the environment in our proposed ABP. Only the architect perceives a reward signal r: (b) Interaction diagram for a standard MARL modelization. Both the architect and the builder have access to environmental rewards r and rg. Which would contradict the fact that the builder ignores everything about the task at hand; (c) Inverse Reinforcement Learning modelization of the ABP. The architect needs provide demonstrations. The architect does not exchange messages with builder. The builder relies on the demonstrations ((x, a, 8() to learn the desired behavior.
A.2 ANALYTICAL DESCRIPTION
Transition Probabilites from the architect point of view and conditional probabilities we have:
Using the laws of total probabilities
P. (s) = Σ P(s' als,
acA
Σ (m)
(6)
acA
> (s a, s)))s(()))
aEA
Where the final equality uses the knowledge that next-states only depends on states and builder's actions.
Reward function from the architect point of view
rg m. E[R m, s']
r (( m. s')dr
/
P(r, als, m, 8'rrr
aEA
/
Σ
P(rls, m, a, 8¹)
P(als, 8 m, m, 8'dr
JR
(7)
aEA
Σ
P(rls, (a)o
aEA
/ rP(rlo,a,
Σ #n(als,m)
aEA
Σ
m))ss,,))
aEA
14

---

Published as a conference paper at ICLR 2022
Transition function from the builder point of view
P(s', ms m. a) = P(m'lk fomnls',s,, s. P
P(m'l (
(8)
= (m' (
A.3. PRACTICAL ALGORITHM
Algorithm 1: Architect- Builder Iterated Guiding (ABIG)
Require: randomly initialized builder policy "g. reward function r, transition function Pe. BC algorithm, MCTS algorithm for è in range(NN Nateration)) do MODELLING FRAME:
for e in range(Neattect) Neotlect do Architect populates DA using m Uniformo and observing a Tg (-)s,m) end for
Architect learns 7g m) on DA with BC
Architect sets T, (m]s) A MCTS(r, aa. Pa)
Architect flushes D,
GUIDING FRAME:
for in range(/Veottect/ do Builder populates D3 using Te while guided by Architect, i.c. m ~ T (-1s)
end for
Builder learns n, (als, m) on D, with BC Builder flushes DB
end for Architect last Modelling Frame Result: TA Te
Behavioral Cloning The data-set is split into training (70%) and validation (30%) sets. If the validation accuracy does not improve during a wait for number of epochs the training early stopped. For training data-set D ((s, m)a)} of size N the BC loss minimize for policy me parametrized by 0 is given by:
1 N
J(0)
Σ log me (a|s, m)
Monte-Carlo Tree Search In the architect's MCTS, nodes labeled by environment's states and they are expanded by selecting messages. Selecting message m from a node with label yields a builder action according the architect's builder model #, [a] s, m)), this sampled action in turn yields the label of the child node according to the environment's transition model s' ~ Pa (s' ss a). We repeat this process until select message that was never selected from the current node or we sample a next state that does not correspond to child node yet. In both of these cases new node has be created. We estimate the value of the new node using an engineered heuristic that estimates the return of an optimal policy x =*(a|s) (s) from state 8. This value is scaled down by factor to avoid overestimation: the builder's policy may not allow the architect to have follow nº This estimated value for newly created node at depth / is back-propaggan as return to parents node at depth according to:
1-1-k G+
Σ 7=0
Tk+414 + -kkd k k-,..,
(10)
15

---

Published as a conference paper at ICLR 2022
where ry is the reward collected from node at depth child node at depth j 1. From node with label s we select messages according to the Upper Confidence Bound rule:
N(,
m = argmax (ss, m) + VV
(s)
T/
(11)
Q(s, G.(,s,
m))
where (s, m) is the number of times message m was selected from the node, Gi(s, m) are the returns obtained from the node when selecting m and is constant set v2. When the architect must choose a message from the environment state its policy T, (m|s) runs the above procedure from a root node labeled with the current environment state 8. After expanding a budget b of nodes the architect picks the best message to send according to Eq. (TT) applied to the root node. It is then possible to reuse the tree for the next action selection or to discard it, a tree is reused its maximal depth should be constrained.
Hyper-parameeees
sampling temperature 0.5
samples per iteration
learning rate
number of epochs
batch size
100
0.1
1000
50
Table 1: Toy experiment hyper-parameten
budget
reuse tree
max tree depth
100
true
500
Table 2: MCTS parameters
episode len
grid size
reward sparse
message one-hot
40
3x6/(6 x 6)
discount factor
episodes per iteration
vocab size 18/ (72)
evaluation episode len
0.95
600
40 / (60)
Table 3: BuildWorld parameters for 3 blocks (for 6 blocks if different)
learning rate
number epochs
batch-size
wait for
x 10-1
1000
256
300
Table 4: Architect's BC parameters BuildWorld for blocks / (for 6 blocks if different)
learning rate
number of epochs
batch-size
wait for
1 x 10-1
1000
256
300
Table 5: Builder's BC parameters on BuildWorld for 3 blocks / (for 6 blocks different)
Sparse reward means that the architect receives if the goal is achieved and 0 otherwise. Episodes per iterations are equally divided into the modelling and guiding frames. Only the learning rates on BuildWorld were searched over with grid-searches. For BuildWorld with 3 blocks the searched range [5 x 10 -4 x 10
x 10-5] for both architect and builder (vocabulary size was fixed at 6). For "grasp" with blocks the searched range is [1 10-3, x 10 1 10-41 for the architect and [5 x 10-4, 1 x 10 4 5 x 10 5] for the builder (vocabulary size was fixed at 72). The other hyper-parameters do not seem have major impact the performance provided that:
16

---

Published as a conference paper at ICLR 2022
enable an agent that has access to the reward to solve the task. there is enough BC epochs to approach convergence.
the MCTS
Regarding the vocabulary size, the bigger the better (see experiments in Figure 14
Computing resources A complete ABIG training can take up to 48 hours on a single modern CPU (Intel E5-2683 v4 Broadwell
1GHz. The presented results require approximately 700 CPU hours. For each training, the main computation cost comes from the MCTS planning during the guiding frames. The self self-imitation and behavior modelling steps only account for a small fraction of the computation.
A.4 INTUITIVE EXPLANATION OF THE LEARNING DYNAMICS
To illustrate the learning mechanisms of ABIG we propose look at the simplest instantiation of the Architect-Builde Problem: there is one state (thus it can be ignored). two messages mi and m2 and two possible actions 0₁ and dg If the builder chooses a is a loss (r(a,) -1) but choosing 02 results in a win (r(02) = 1). Figure 9 displays several iterations of ABIG on this problem when the initial builder's policy is unfavorable (a1 is more likely than 02 for all the messages). During each iteration the architect selects messages in order to maximize the likelihood of the builder pick- ing action d2 and then the builder does self-Imitation Learning by maximizing the likelihood of the corresponding messages- actions sequence under its policy. Figure 9 shows that this process leads to forgetting unfavorable associations until a favorable association emerges and can be reinforced. On the other hand, for ABIG-no-intent in Figure T0 favorable and unfavorable messages are pled alike which prevents the forget mechanism to undo unfavorable message-to-action associations. Consequently, initial preferences reinforced.
(loose)
(win)
4V
11
2
Figure 9: ABIG-driven evolution of message-conditionee action probabilities (builder's policy) for a simple problem where the builder must learn produce action da. Even under unfavorable initial condition the architect-builder pair eventually manages to associate message (here mi)) with the winning action (a2). Initial conditions are unfavorable since a1 is more likely than ag for both messages. (i = 0) Given the initial condi- tions, the architect only sends message m2 since it is the most likely to result in action Og. (i = 1) the builder guiding data only consisted of m, message therefore it cannot learn preference
actions for and both actions are equally likely under m2. The architect now only sends message m2 since is more likely than mi at triggering d2. (i = 2) Unfortunately, the sampling of my resulted during the guiding frame and the builder thus associates m2 with 0₁ The architect tries its luck again but now with ma. (i = 3) Eventually, the sampling results in more actions being sampled in the guiding data and the builder now associates mi d2. (i = 4) and (i The architect now keep sending mi messages reinforce this association.
builder doing more aj than aq
(comm)
(wm)
Figure 10: ABIG-no-intent driven evolution message-conditioned action probabilities for a simple problem where builder must learn to produce action dz. Initial conditions are unfavorable since a1 is more likely than 02 for both messages. Without an architect's guiding messages during training, a self-imitating builder rein- forces the action preferences of the initial conditions and fails (even when evaluated alongside knowledgeable architect both messages can only yield a1).
17

---

Published as a conference paper at ICLR 2022
To further assess how the architect's message choices impact the performance of self-imitating builder, we compare the distribution of the builder's preferred actions obtained after using ABIG and ABIG-no-intent. We consider three different initial conditions (favorable, unfavorable, intermediate) that are each ran to convergence (meaning that the policy does not change anymore across iterations) for 100 different seeds. Figure[TT dissplas the resulting distributions of preferred -ie. most likely- action for each message. When applying ABIG on the toy problem, the pair always reaches a success rate of 100/100 no matter the initial condition. We also observe that at convergence the builder never prefers action d1, yet when an action is preferred for given message, the other message yields no preference over action (p(a) (m) p(azl m)). This is due the forget mechanism discussed in Section 3.3 The results when applying ABIG-no-intent the toy problem on the initial condition. In the unfavorable scenario, ABIG-no-intent fails heavily with only seeds succeeding over the 100 experiments. This is due to the fact that, in absence of message guidance from the architect, the builder has high chances to continually reinforce the association between the two messages and d1, therefore losing. However, in rare cases, the builder inverse the initial nessage-conditioned probabilities by 'luckily' sampling more often 02 when receiving mı and win. This only happened times over the 100 seeds. Finally, when initial conditions more favorable, the self-imitation steps reinforce the association between the messages and 02 which makes the builder prefer d2 for least message and enables high success rates (100/100 for favorable and 98/1 for intermediate).
much more dependent
Unfavorablc
Favorable
Intermediate
(a) Initial probabilities
P(as [mm) 0.8, P(az/mu)
P(a: [ms) 0.2, P(az/mu)
P(a: mms) 0.9, P(azlm))
P(ai (m2) 0.9, P(o2[m2) 0.1
P(e (m2) 0.1, P(02(m2) 0.9
P(e. m2) 0.1, P(22(m2) 0.9
(b) ABIG: Distributions of final preferred action for each message calculated over 100 seeds
Jonee
Plajeet -Pajett
1.00
1.00
3.00
1.00
0.63
0.63
0.00
mh)
(0)
difficult
easy
varied
Success Rate= 100/100
Success Rate = 100/100
Success Rate 100/100
(c) ABIG-no-intent: Distributions final preferred action for each message calculated over 100 seeds
0.97
1.00
0.95
0,00 0.03
0,00
0.00
6
0.08
0.00
0.00
0.02 0,00
0.00 0.00
easy
varied
Staccess Rate 3/100
Success Rate 100/100
Success Rate 9R/100
Figure 11: Toy experiment analysis (a) Initial conditions: initial probability for each action given message m; distributions final builder's preferred actions for each message after applying (b) ABIG and ABIG- no-intent on toy problem; distributions are calculated over 100 seeds. each method and each initial condition, we report the success rate obtained by knowledgeable architect guiding the builder. At evaluation, the architect has access to the builder's 's model and does not ignore the goal. ABIG always succeeds while ABIG-no CoG--o-inee no-intent's success depends on the initial conditions.
18

---

Published as a conference paper at ICLR 2022
A.5 RELATED WORK
In this section we develop the differences between ABP and Hierarchical/Feuual Reinforcement Learning more in detail.
Kulkarni et al. (2016) proposes to decompose a RI. agent into a two-stage hierarchy with a meta- controller (or manager) setting the goals of controller (or worker). The meta-controller trained to select sequences of goals that maximize the environment reward while the controller is trained to maximize goal-conditioned intrinsic rewards. The definition of the goal-space well the corre- sponding hard-coded goal-conditioned reward functions are task-related design choices. In Vezhn- evets et al. (2017). the authors propose a more general approach by defining goals embeddings that directly modulate the worker's policy. Additionally, the authors define intrinsic rewards as the cosine distance between goals and embedded-state deltas (difference between the embedded-stale -tatee at the moment the goal was given and the current embedded-state) Thus, goals can be interpreted as directions in embedding space. Nachum et al. (2018) build on this idea but let go the embed- ding transformation by considering goals directions to reach and rewards distances between state deltas and goals. These works tackle the single-agent learning problem and therefore allow the manager to directly influence the learning signal of the workers. However, in the multi-agent setting where agents are physically distinct, it is not possible for an agent to explicitly tweak another agent's 's learning algorithm. Instead, agents must communicate by influencing each other's observa- tions instead of intrinsic rewards. Since it is designed to investigate the emergence of communication between agents, ABP lies in this latter auett--agent setting where agents can interact with one-another only through observations. This makes applying Feudal or Hierarchical methods to the ABP unfea- sible as they are restricted to worker agents that directly receive rewards. In contrast, in ABP, the reward-less builder observes communication messages that, initially, have arbitrary meaning.
19

---

Published as a conference paper at ICLR 2022
B SUPPLEMENTARY RESULTS
B.1 LEARNING ANALYSIS
The main document shows performances (success rates) builder-architec pairs convergence. In this supplementary section we propose to thoroughly study the evolution of the builder's 's policy in order to provide a deeper analysis of ABIG.
Metric definition. We define three metrics that characterize the builder behavior. We compute these metrics on constant Meusurement Sel M made of 6000 randomly sampled states, for each of these states we sample all the possible messages ~ Uniform()) where V is the set of possible messages. Therefore, | = 6000 x [ The set of possible actions A and we denote by á the indicator function.
We also define the following distributions:
(s)
((s'=s))
M
SEM
1
Per (m) & P(mls)
Pas (8, (8.m) - Ps (s) P(p P(mls) = Ps (s)Pp
PAMA (s, ",mmaal a) - Pan (s, m) P(alm))apaaaaaaaaa m) = PaN (8, m) (als, m)
Pa (a)
Σ
PSMA (8, m, a)
(s.m)EM
Pass (n, a)
Σ
Pasra (8, a)
BEM
Ps (8,
Pssaa (8, m,
meM
From this can define the monitoring metries:
Mean Entropy:
H(T)= M
Σ
Σ
(a.m)EM
aEAA
Mulual Information between messages and actions
Im = Σ. Prea (m, a) log Para (m, a) Ps (a)pss (m)
mEV aEA
Mutual Information between states and actions
I. = ΣΣ Psa (s, log Po (s, a) D, (a)p,
SEMoEA
Analysis. Figure 12 displays the evolution of these metrics after each iteration as well as the evolution of the success rate (a). As indicated by Eq. (5). doing self-imitatio learning results in a decay of the mean entropy (b). This decay similar for ABIG and ABIG-no-intent. The most interesting result is provided by the evolution of the mutual information (c). For ABIG-no-intent, we see that 1, and Im slowly increase with La > Im over all iterations. This indicates that the builder policy (a(a]s, m) relies more states than on messages to compute the actions. In this scenario the builder, therefore, tends to ignore messages. On the other hand, I, and I,, evolve differently for ABIG. Both metrics first increase with I, > Im until they cross around iteration 25. Then I, starts decreasing and Im grows. This shows that ABIG results a builder policy that strongly selects actions based on the messages it receives which is desirable feature of emergent communication.
20

---

Published as a conference paper at ICLR 2022
1.00
ABIG
ABIG-no-intent
Success Rale
0.75
0.50
0.25
0.00
20
40
80
iteration
(a) Evolution of the success rate
ABIG
1.6
ABIG-no-Intent
Builder Entropy
1.2
0.8
044
0.0
20
60
Iteration
(b) Evolution of the builder policy mean entropy Hap
ABIG I
ABIG-no-intent 1, ABIG-no-intent
1.00
Mutual Information
ABIG
0.50
0.25
0.00
20
iteration
(c) Evolution the mutual information I, and Im
Figure 12: Comparison of the evolution of builder policy properties when applying ABIG and ABIG-no-intent on the "place" task in BuildWorld. (a) ABIG enables much higher performance that BBIG-noo-itenn (b) Both methods use self-imitation and thus reduce the entropy of the policy. (c) ABIG promotes the mutual information between messages and action which indicates successful communication protocols.
B.2 ADDITIONAL BASELINE COMPARISON
We define two extra baselines:
Stochastic:
where the builder policy is fixed softmax policy parameterized by a randomly initialized network;
Deterministic: where the builder policy is a fixed argmax policy parameterized by ran- domly initialized network.
In the performances reported in Figure 13 the architect has direct access to the exact policy of the builder (#g = Ty and uses it to plan and guide the builder during evaluation. We observe that the stochastic condition exhibits similar performances as the random builder. This indicates that, even the architect tries to guide the builder, the stochastic policy is not controllable and performances not improved. Finally, the architect. Yet, as pointed out in Figure B the initial deterministic policies lack flexibility and fail. This shows that the builder must iteratively evolve its policy in order to make it controllable.
would expect a deterministic policy to be more easily controllable by
21

---

Published as a conference paper at ICLR 2022
Mathod
A-lii-no-iddrrt
Fandom
stochsticc
dederministic
Moan scare (10 seedss)
0.8
0.8
0.4
0.2
0.0
grasp
Vline
Goal
Figure 13: Baseline performance depending on the goal: stochastic policy bchaves on par with random builder. Self-imitation with ABIG-no-intent remains the controllable bascline.
B.3
IMPACT OF THE VOCABCLARY SIZE
Vocabu ary
Meen (10 6.0
place
Goal
Figure 14: Influence of the Vocabulary size for ABIG on the 'place' task. Performance increases with the
vocabulary size.
22


Reviews:
[{"review_id": "zdy78mB5Dv", "reviewer": "Reviewer_hMeT", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, I find the setting and the proposed algorithm novel and interesting. I really like that authors carefully distinguish their setting from RL and IL, and give very nice intuitions about the learning dynamics of the proposed algorithm. Although the architect-builder relationship has been explored in previous work, preventing the builder from knowing the rewards is, to my best knowledge, not explored. However, I feel like this setting makes more sense if the architect were a human, as they may have implicit intent that the builder has to manage to interpret. I'd encourage the authors to give a concrete example where this setting is useful in the agent-agent setting. In general, what is the motivation of this work? Is it a computational model for studying human-human communication? Or is it offering a learning framework that aim for specific practical scenarios? The motivation should be stated more clear in the introduction. \n\nAnother suggestion is to rewrite the problem formulation using the MDP formalism (i.e. <S, A, T, R, gamma>), clearly defining the MDPs and policies of the architect and the builder. In addition, the description of the interaction needs to be more precise: right now, it does not specify whether the architect sends a message to the builder after every step or the builder can take multiple steps after a message. \n\nThe main limitations of this work are (1) the architecture requires access to a simulator of the environment (which may not be a problem for a human) and (2) the simplicity of the experimented environment. The authors adequately acknowledge these limitations. \n\nSuggested related work (on more complex environments):\n\n[1] Collaborative Dialogue in Minecraft. https://aclanthology.org/P19-1537.pdf\n\n[2] Hierarchical Decision Making by Generating and Following Natural Language Instructions. https://arxiv.org/pdf/1906.00744.pdf\n\n[3] Interactive Learning from Activity Description. https://arxiv.org/pdf/2102.07024.pdf\n\n[4] Neural Abstructions: Abstractions that Support Construction for Grounded Language Learning. https://arxiv.org/pdf/2107.09285.pdf\n\n[5] Incorporating Pragmatic Reasoning Communication into Emergent Language. https://arxiv.org/pdf/2006.04109.pdf", "summary_of_the_paper": "\n**After rebuttal**: I am keeping my current score. I am positive about this framework as it presents a better model for multi-agent communication, especially enriching the communication among agents over the fixed, restricted reward-based communication protocol in traditional RL.\n\n**Before rebuttal**: The paper proposes an architect-builder problem setting where the architecture guides the builder to accomplish a goal by sending messages. Only the architecture knows the goal and has access to rewards, whereas only the builder can act in the environment. This setting is distinct from traditional reinforcement learning and imitation learning. \n\nDrawing inspiration from cognitive science theories, the authors devise an algorithm for learning a communication protocol between the architect and the builder. On grid-world tasks, they show that learned the communication protocol can generalize to previously unseen tasks. \n", "summary_of_the_review": "The paper presents a novel and interesting setting and algorithm. They nicely implement ideas from cognitive science in an MDP setting. Even though the formulation needs to be more rigorous and the experiment environments are simplistic, I think the contributions are interesting enough to draw attention of the community in the future. I recommend acceptance. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "jAXEKYaNJ98", "reviewer": "Reviewer_FMPZ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The modelling of the problem is sensible (i.e. using two distinct, connected, MDPs). It would support clarity if this design decision was explained/backed with respect to established research on multi-agent modelling. Are any reductions possible? It seems that several challenges of the proposed framework have been solved in different MAS works, so it would be userful to be clear here.\n\nThe goal of the paper is relevant for the conference, as a communication channel is directly learned and the framework somewhat works towards the bigger goal of AGI. However, it would be useful to add more real world applications of the approach. What would intermediate applications be before a perfect version of the approach is available?\n\nThe related work is informative, but I am missing comparison to recent works on such learning protocols - e.g. [1]. There are differences between the frameworks and [1] might work towards HRI (as also mentioned in the paper), but the direction seems quite related. It would be also interesting to dwell on the reusability of the proposed algorithm for the problem at hand.\n\nThe authors themselves note in the discussion of the paper that the employed methods are suboptimal for the defined learning problem (i.e. using behavioral cloning and MCTS). It would be beneficial to clearly state the learning challenges when defining the protocol, as it is a main part of the contribution. \n\nI am also wondering to what extent it might be possible to reuse established approaches for emerging communication, as already cited in the paper (e.g. [2]).\n\nThe empirical evaluation is informative, although the random building agent does not give too much additional insight. It would be good to show relevant ablations in the main part of the paper. \n\nA minor comment with respect to the figure of the approach: I find the pseudocode of the algorithm quite helpful (in my opinion, more helpful), but it is in the suppl. Material.\n\n[1] Nguyen, K., Misra, D., Schapire, R., Dudík, M. and Shafto, P., 2021. Interactive Learning from Activity Description. arXiv preprint arXiv:2102.07024.\n[2] Foerster, J.N., Assael, Y.M., De Freitas, N. and Whiteson, S., 2016. Learning to communicate with deep multi-agent reinforcement learning. arXiv preprint arXiv:1605.06676.\n", "summary_of_the_paper": "The paper proposes an approach for interactive learning between a so-called architect and builder agent, which is a different, but related protocol to RL or imitation learning. Under the assumption that the architect knows the target dynamics and reward function, the authors focus on learning the communication between the two parties, such that the builder can solve the MDP.\n\nAfter formalizing the setting in the multi-agent paradigm, where two individual MDPs are defined, the authors propose an algorithm with so-called modelling- and guiding phases. In each phase, architect and builder gather datasets, from which policies are extracted via behaviour cloning and planning.\n\nThe paper is evaluated on a proposed block environment for the problem and uses a random building agent and combination of random modelling phase with proposed guiding phase as baselines. The results show that the proposed approach is superior wrt the baselines for solving the block environment. In addition, experiments show that the learned communication channel can potentially be reused for solving other tasks.  ", "summary_of_the_review": "The paper proposes an interesting and relevant framework for interactive learning / teaching between two agents. The model choice and solution is sensible, and the empirical evaluation can shows that the initial approach for the problem works sufficiently well for the toy problem. The learning challenges with respect to other works in MAS could be clarified in more detail.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "ezHtOYiCiuU", "reviewer": "Reviewer_BHGy", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strengths\n* An interesting setting and learning paradigm which seems to promote the emergence of generalizable communication protocols.\n\n## Weaknesses\n* Several assumptions seem to be made which limit the approach's direct applicability to more complex environments\n  * Giving the architect access to the ground-truth environment model seems to be a very strong assumption.\n  * The heuristic used in MCTS is not described in detail, but, presumably, it uses a significant amount of domain knowledge.\n* It's unclear how the self-imitation learning works (see \"Questions\" below for more detail). My main concern is that this only works due to the fact that the architect has access to ground truth environment transition models such that it can exploit spurious correlations between messages and behavior in an untrained builder model. This seems unlikely to work at all even in this simple environment without access to the environment model.\n* Lacking in experiments which attempt to understand the nature of the learned communication protocol. From section B.3 it seems like the agents simply learn to map messages directly to actions which is somewhat disappointing. If this is the case, then the generalization results may simply be down to the effectiveness of the MCTS procedure. In other words, the architect-builder pair learns messages that map to all possible actions, so the architect's job is reduced to standard MCTS with the original action space.\n\n## Questions\n* How is it possible for the builder to have any sort of coherent predictable behavior at the beginning of training? It seems this would be crucial for the self-imitation learning to work. In other words, an untrained builder agent will have no ability to interpret messages and modulate its behavior through them. As a result, the trajectories produced by the architect-builder pair will be no better than random and self-imitation of these trajectories shouldn't produce any meaningful results. It seems section 3.3 attempts to address this, but a more intuitive description would be appreciated.", "summary_of_the_paper": "The authors propose a setting in which two agents with asymmetric information collaborate to complete a goal. They demonstrate that learning in this setting results in the emergence of communication protocols that generalize to new tasks.", "summary_of_the_review": "This paper presents an interesting setting; however, it appears to rely too heavily on strong assumptions, and the communication protocols that emerge do not seem to exhibit any interesting properties (i.e. the communicating agent simply learns to output messages that correspond to the desired action for the acting agent).", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "aewOiZcqq-W", "reviewer": "Reviewer_rTG5", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I believe that this is a well-written paper, with solid motivation, and thoughtful care in crafting the approach. However, the bulk of my criticism focuses on the assumptions underlying the approach, and it’s situation relative to prior work in hierarchical, and specifically Feudal RL, an entire body of work that is omitted in the paper.\n\nConcretely, I  believe that providing the architect access to the transition function is an incredibly strong assumption that undercuts this work. Existing work in Hierarchical RL that learns separate high- and low-level policies, under similar assumptions as the Architecture-Builder setup (though possibly without the restriction of the discrete message channel) *do not* make this perfect transition assumption, and instead focus on alternative means of learning the “architect.” Some examples are HIRO (https://arxiv.org/abs/1805.08296), Feudal Networks (https://arxiv.org/abs/1703.01161), h-DQN (https://arxiv.org/abs/1604.06057), amongst many many more (there are many other Feudal RL approaches, including more modern ones that appear in the skill discovery literature).\n\nThis leads into my second key weakness, which is around the evaluation. I understand that full system evaluations are hard, but in this specific case, I think the current ablations trivially fail, and don’t provide much insight into the proposed approach. Instead, I would love to see other work that looks at more traditional HRL approaches, or that relaxes the some of the assumptions made in this work. Specifically, relaxing the “discrete communication channel” assumption would allow for out-of-the-box comparisons to HIRO and h-DQN, as well as more recent work. Other ablations I’d like to see in future versions of this paper would look at versions of the Architect learning without access to transition rewards.\n\nTypos/Style/Questions:\n\nBroad Positioning Question: I love the Architect-Builder problem statement, but it’s not immediately clear to me why we need to restrict the communication mechanism between the architect and the builder/what that gives us more broadly. If the goal is to see how we might emerge a “language” (discrete tokens, similar to work on emergent communication work) it’d be nice to see how well an Architect or Builder learned with this approach transfers to “real-language” settings (e.g., adapt to natural language instructions!). If the goal is broadly coordination, I definitely think falling back on the Feudal RL literature is important; looking at latent continuous representations of “intent” is pretty pervasive in HIRO and newer work such as VALOR (https://arxiv.org/abs/1807.10299). ", "summary_of_the_paper": "This paper presents an approach — Architect-Builder Iterated Guiding — a method that tackles what is presented as an “Architect-Builder” problem: a scenario in which two actors, an Architect, with knowledge of a high-level goal, or reward function, must communicate over a discrete channel with a Builder who can take actions in the environment based on the Architect’s message. In many ways this resembles a **hierarchical reinforcement learning** setup, specifically reminiscent of Feudal Reinforcement Learning (“Feudal Reinforcement Learning,” Dayan et. al. NeurIPS 1992). The paper presents a motivated, easy-to-understand algorithm for training both the Architect and the Builder, and evaluate on a series of “construction” based grid-world tasks (resembling GridLU, MiniGrid, or Mazebase). \n\nNamely, the proposed approach takes inspiration from Experimental Semiotics and separates learning into separate **interaction frames** (similar again to multi-phase approaches in the Hierarchical RL literature, e.g. in HIRO — “Data Efficient Hierarchical Reinforcement Learning,” Nachum et. al. 2018; https://arxiv.org/abs/1703.01161). These interaction frames consist of a modeling frame, in which the architect learns a model of the builder after “rolling out” and sending messages/watching the builder’s actions, followed by a guiding frame where the architect exploits its model of the builder to produce the “optimal” actions via a heuristic driven Monte Carlo Tree Search (rather than explicitly estimate a value function, which is costly and noisy). Making this all possible is that the architect has full knowledge of the high-level reward, **in addition to the ground-truth state transition function**. This lets the architect explicitly search over messages to send to the builder (as it fully observes the builder’s rollout), then improve the builder by “self-imitating” over this new data, mirroring a bi-phase optimization setup.\n\nThe evaluation focuses on the proposed model, and two simple ablations: one where the architect has “no-intent” at training, sending random messages, and one in which the builder takes random actions. The paper also presents (in the main body and appendix) a meaningful, thoughtful intuitive explanation of the learning dynamics of the architecture and the builder — more papers should dedicate portions of the main body to explanations such as this!", "summary_of_the_review": "In general, I believe this is a well-written, well-motivated paper that draws some great ideas from Experimental Semiotics. Furthermore, the discussion and analysis of the learning dynamics of the various components of the approach should be a mainstay of future systems driven work in ML.\n\nHowever, I am deeply concerned with the assumptions made with this approach, stemming from the architect’s access to the ground-truth state transition function. This coupled with a missing discussion of related work in hierarchical RL and Feudal RL specifically, and a more thorough evaluation including comparison to these methods and other relevant ablations inform my decision to lean towards rejecting this paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a cognitive science-inspired interaction setting between two agents, an "architect" and "builder", in which the architect must produce messages to guide the builder to achieve a task. Unlike other related settings (such as typical approaches in MARL, HRL, or HRI), the builder does not have access to the architect's reward function, and must learn to interpret the architect's messages by assuming the architect is telling it to do something sensible. At the same time, the architect determines what is "sensible" by building a model of the builder's behavior and planning over it. This setting is common particularly in human-agent interactions, where humans may not be able to either (1) accurately communicate a scalar reward or (2) provide demonstrations, but can still provide information that the agent ought to be able to learn from. The paper demonstrates that the learned communication protocol generalizes well to new settings.

While this paper generated a lot of discussion, the reviewers did not come to a consensus on whether the paper should be accepted or rejected, with those in favor of the paper maintaining it should be accepted and those not in favor maintaining that it needs work. I have therefore done a particularly close read of both the paper and the discussion in order to weigh the pros and cons brought up by the reviewers.

The positive reviews clearly indicate that this work is insightful and of interest to researchers in the ICLR community (in fact, all reviewers mentioned they found the work interesting and well-written). In particular, Reviewer hMeT wrote: "I am positive about this framework as it presents a better model for multi-agent communication, especially enriching the communication among agents over the fixed, restricted reward-based communication protocol in traditional RL." I am inclined to agree with this assessment and find the communication setting studied in this paper to be much more ecologically valid for human-agent interaction settings than having humans communicate scalar rewards or provide demonstrations: humans are typically poor at the former and may not have the same embodiment to achieve the latter.

The negative reviews focused on a few cons: (1) the assumption that the architect has access to a ground-truth environment model, (2) confusion about differences from other related fields (e.g. feudal RL, MARL), and (3) lack of analysis of the communication protocol. I have considered these points, but do not feel any of them are fatal flaws: (1) From the perspective of human-agent interaction, I think it is very reasonable to assume that a human architect would have a good model of the world and would be generally proficient at solving tasks in the world. Making this approach work in the setting where the architect is *also* learning how the world works seems squarely in the domain of future work. (2) The authors have done an extensive job of clarifying the differences between these related areas, and as discussed above, other reviewers found the way in which AGP is different to be insightful and ecologically valid. (3) This is potentially the most serious con: as the discussion with Reviewer BHGy brought up, the learned communication protocol may just be a simple mapping between messages and environment interactions. After further discussion in which the authors argued that learning a simple mapping is not a problem---the main question is how to even induce such a mapping in the first place---the reviewer acknowledged that this is not a fatal flaw but that makes the results somewhat less interesting.

In summary, the positive reviews highlighted the interestingness and insightful nature of the questions studied in this paper and have convinced me that this paper will be of interest to the ICLR community as it has provides a new perspective on the problem of agent-agent interaction (particularly for the special case of human-agent interaction). The negative reviews did highlight a few limitations of the paper, but I expect these can be addressed by future work and do not feel they outweigh the interestingness of the problem. In light of this, I recommend acceptance as a poster.

Suggestion for the authors: I found the discussion with Reviewer BHGy to be particularly insightful and helpful in understanding the aims of the paper. I would encourage you to incorporate some of this into the camera-ready version of the paper, and perhaps to lean more heavily on the special case of human-agent interaction as motivation of this work (as also hinted at by Reviewer hMeT).
Decision: Accept (Poster)