Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper presents an approach — Architect-Builder Iterated Guiding — a method that tackles what is presented as an “Architect-Builder” problem: a scenario in which two actors, an Architect, with knowledge of a high-level goal, or reward function, must communicate over a discrete channel with a Builder who can take actions in the environment based on the Architect’s message. In many ways this resembles a **hierarchical reinforcement learning** setup, specifically reminiscent of Feudal Reinforcement Learning (“Feudal Reinforcement Learning,” Dayan et. al. NeurIPS 1992). The paper presents a motivated, easy-to-understand algorithm for training both the Architect and the Builder, and evaluate on a series of “construction” based grid-world tasks (resembling GridLU, MiniGrid, or Mazebase). 

Namely, the proposed approach takes inspiration from Experimental Semiotics and separates learning into separate **interaction frames** (similar again to multi-phase approaches in the Hierarchical RL literature, e.g. in HIRO — “Data Efficient Hierarchical Reinforcement Learning,” Nachum et. al. 2018; https://arxiv.org/abs/1703.01161). These interaction frames consist of a modeling frame, in which the architect learns a model of the builder after “rolling out” and sending messages/watching the builder’s actions, followed by a guiding frame where the architect exploits its model of the builder to produce the “optimal” actions via a heuristic driven Monte Carlo Tree Search (rather than explicitly estimate a value function, which is costly and noisy). Making this all possible is that the architect has full knowledge of the high-level reward, **in addition to the ground-truth state transition function**. This lets the architect explicitly search over messages to send to the builder (as it fully observes the builder’s rollout), then improve the builder by “self-imitating” over this new data, mirroring a bi-phase optimization setup.

The evaluation focuses on the proposed model, and two simple ablations: one where the architect has “no-intent” at training, sending random messages, and one in which the builder takes random actions. The paper also presents (in the main body and appendix) a meaningful, thoughtful intuitive explanation of the learning dynamics of the architecture and the builder — more papers should dedicate portions of the main body to explanations such as this!
main_review: I believe that this is a well-written paper, with solid motivation, and thoughtful care in crafting the approach. However, the bulk of my criticism focuses on the assumptions underlying the approach, and it’s situation relative to prior work in hierarchical, and specifically Feudal RL, an entire body of work that is omitted in the paper.

Concretely, I  believe that providing the architect access to the transition function is an incredibly strong assumption that undercuts this work. Existing work in Hierarchical RL that learns separate high- and low-level policies, under similar assumptions as the Architecture-Builder setup (though possibly without the restriction of the discrete message channel) *do not* make this perfect transition assumption, and instead focus on alternative means of learning the “architect.” Some examples are HIRO (https://arxiv.org/abs/1805.08296), Feudal Networks (https://arxiv.org/abs/1703.01161), h-DQN (https://arxiv.org/abs/1604.06057), amongst many many more (there are many other Feudal RL approaches, including more modern ones that appear in the skill discovery literature).

This leads into my second key weakness, which is around the evaluation. I understand that full system evaluations are hard, but in this specific case, I think the current ablations trivially fail, and don’t provide much insight into the proposed approach. Instead, I would love to see other work that looks at more traditional HRL approaches, or that relaxes the some of the assumptions made in this work. Specifically, relaxing the “discrete communication channel” assumption would allow for out-of-the-box comparisons to HIRO and h-DQN, as well as more recent work. Other ablations I’d like to see in future versions of this paper would look at versions of the Architect learning without access to transition rewards.

Typos/Style/Questions:

Broad Positioning Question: I love the Architect-Builder problem statement, but it’s not immediately clear to me why we need to restrict the communication mechanism between the architect and the builder/what that gives us more broadly. If the goal is to see how we might emerge a “language” (discrete tokens, similar to work on emergent communication work) it’d be nice to see how well an Architect or Builder learned with this approach transfers to “real-language” settings (e.g., adapt to natural language instructions!). If the goal is broadly coordination, I definitely think falling back on the Feudal RL literature is important; looking at latent continuous representations of “intent” is pretty pervasive in HIRO and newer work such as VALOR (https://arxiv.org/abs/1807.10299). 
summary_of_the_review: In general, I believe this is a well-written, well-motivated paper that draws some great ideas from Experimental Semiotics. Furthermore, the discussion and analysis of the learning dynamics of the various components of the approach should be a mainstay of future systems driven work in ML.

However, I am deeply concerned with the assumptions made with this approach, stemming from the architect’s access to the ground-truth state transition function. This coupled with a missing discussion of related work in hierarchical RL and Feudal RL specifically, and a more thorough evaluation including comparison to these methods and other relevant ablations inform my decision to lean towards rejecting this paper.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes an approach for interactive learning between a so-called architect and builder agent, which is a different, but related protocol to RL or imitation learning. Under the assumption that the architect knows the target dynamics and reward function, the authors focus on learning the communication between the two parties, such that the builder can solve the MDP.

After formalizing the setting in the multi-agent paradigm, where two individual MDPs are defined, the authors propose an algorithm with so-called modelling- and guiding phases. In each phase, architect and builder gather datasets, from which policies are extracted via behaviour cloning and planning.

The paper is evaluated on a proposed block environment for the problem and uses a random building agent and combination of random modelling phase with proposed guiding phase as baselines. The results show that the proposed approach is superior wrt the baselines for solving the block environment. In addition, experiments show that the learned communication channel can potentially be reused for solving other tasks.  
main_review: The modelling of the problem is sensible (i.e. using two distinct, connected, MDPs). It would support clarity if this design decision was explained/backed with respect to established research on multi-agent modelling. Are any reductions possible? It seems that several challenges of the proposed framework have been solved in different MAS works, so it would be userful to be clear here.

The goal of the paper is relevant for the conference, as a communication channel is directly learned and the framework somewhat works towards the bigger goal of AGI. However, it would be useful to add more real world applications of the approach. What would intermediate applications be before a perfect version of the approach is available?

The related work is informative, but I am missing comparison to recent works on such learning protocols - e.g. [1]. There are differences between the frameworks and [1] might work towards HRI (as also mentioned in the paper), but the direction seems quite related. It would be also interesting to dwell on the reusability of the proposed algorithm for the problem at hand.

The authors themselves note in the discussion of the paper that the employed methods are suboptimal for the defined learning problem (i.e. using behavioral cloning and MCTS). It would be beneficial to clearly state the learning challenges when defining the protocol, as it is a main part of the contribution. 

I am also wondering to what extent it might be possible to reuse established approaches for emerging communication, as already cited in the paper (e.g. [2]).

The empirical evaluation is informative, although the random building agent does not give too much additional insight. It would be good to show relevant ablations in the main part of the paper. 

A minor comment with respect to the figure of the approach: I find the pseudocode of the algorithm quite helpful (in my opinion, more helpful), but it is in the suppl. Material.

[1] Nguyen, K., Misra, D., Schapire, R., Dudík, M. and Shafto, P., 2021. Interactive Learning from Activity Description. arXiv preprint arXiv:2102.07024.
[2] Foerster, J.N., Assael, Y.M., De Freitas, N. and Whiteson, S., 2016. Learning to communicate with deep multi-agent reinforcement learning. arXiv preprint arXiv:1605.06676.

summary_of_the_review: The paper proposes an interesting and relevant framework for interactive learning / teaching between two agents. The model choice and solution is sensible, and the empirical evaluation can shows that the initial approach for the problem works sufficiently well for the toy problem. The learning challenges with respect to other works in MAS could be clarified in more detail.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: 
**After rebuttal**: I am keeping my current score. I am positive about this framework as it presents a better model for multi-agent communication, especially enriching the communication among agents over the fixed, restricted reward-based communication protocol in traditional RL.

**Before rebuttal**: The paper proposes an architect-builder problem setting where the architecture guides the builder to accomplish a goal by sending messages. Only the architecture knows the goal and has access to rewards, whereas only the builder can act in the environment. This setting is distinct from traditional reinforcement learning and imitation learning. 

Drawing inspiration from cognitive science theories, the authors devise an algorithm for learning a communication protocol between the architect and the builder. On grid-world tasks, they show that learned the communication protocol can generalize to previously unseen tasks. 

main_review: Overall, I find the setting and the proposed algorithm novel and interesting. I really like that authors carefully distinguish their setting from RL and IL, and give very nice intuitions about the learning dynamics of the proposed algorithm. Although the architect-builder relationship has been explored in previous work, preventing the builder from knowing the rewards is, to my best knowledge, not explored. However, I feel like this setting makes more sense if the architect were a human, as they may have implicit intent that the builder has to manage to interpret. I'd encourage the authors to give a concrete example where this setting is useful in the agent-agent setting. In general, what is the motivation of this work? Is it a computational model for studying human-human communication? Or is it offering a learning framework that aim for specific practical scenarios? The motivation should be stated more clear in the introduction. 

Another suggestion is to rewrite the problem formulation using the MDP formalism (i.e. <S, A, T, R, gamma>), clearly defining the MDPs and policies of the architect and the builder. In addition, the description of the interaction needs to be more precise: right now, it does not specify whether the architect sends a message to the builder after every step or the builder can take multiple steps after a message. 

The main limitations of this work are (1) the architecture requires access to a simulator of the environment (which may not be a problem for a human) and (2) the simplicity of the experimented environment. The authors adequately acknowledge these limitations. 

Suggested related work (on more complex environments):

[1] Collaborative Dialogue in Minecraft. https://aclanthology.org/P19-1537.pdf

[2] Hierarchical Decision Making by Generating and Following Natural Language Instructions. https://arxiv.org/pdf/1906.00744.pdf

[3] Interactive Learning from Activity Description. https://arxiv.org/pdf/2102.07024.pdf

[4] Neural Abstructions: Abstractions that Support Construction for Grounded Language Learning. https://arxiv.org/pdf/2107.09285.pdf

[5] Incorporating Pragmatic Reasoning Communication into Emergent Language. https://arxiv.org/pdf/2006.04109.pdf
summary_of_the_review: The paper presents a novel and interesting setting and algorithm. They nicely implement ideas from cognitive science in an MDP setting. Even though the formulation needs to be more rigorous and the experiment environments are simplistic, I think the contributions are interesting enough to draw attention of the community in the future. I recommend acceptance.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors propose a setting in which two agents with asymmetric information collaborate to complete a goal. They demonstrate that learning in this setting results in the emergence of communication protocols that generalize to new tasks.
main_review: ## Strengths
* An interesting setting and learning paradigm which seems to promote the emergence of generalizable communication protocols.

## Weaknesses
* Several assumptions seem to be made which limit the approach's direct applicability to more complex environments
  * Giving the architect access to the ground-truth environment model seems to be a very strong assumption.
  * The heuristic used in MCTS is not described in detail, but, presumably, it uses a significant amount of domain knowledge.
* It's unclear how the self-imitation learning works (see "Questions" below for more detail). My main concern is that this only works due to the fact that the architect has access to ground truth environment transition models such that it can exploit spurious correlations between messages and behavior in an untrained builder model. This seems unlikely to work at all even in this simple environment without access to the environment model.
* Lacking in experiments which attempt to understand the nature of the learned communication protocol. From section B.3 it seems like the agents simply learn to map messages directly to actions which is somewhat disappointing. If this is the case, then the generalization results may simply be down to the effectiveness of the MCTS procedure. In other words, the architect-builder pair learns messages that map to all possible actions, so the architect's job is reduced to standard MCTS with the original action space.

## Questions
* How is it possible for the builder to have any sort of coherent predictable behavior at the beginning of training? It seems this would be crucial for the self-imitation learning to work. In other words, an untrained builder agent will have no ability to interpret messages and modulate its behavior through them. As a result, the trajectories produced by the architect-builder pair will be no better than random and self-imitation of these trajectories shouldn't produce any meaningful results. It seems section 3.3 attempts to address this, but a more intuitive description would be appreciated.
summary_of_the_review: This paper presents an interesting setting; however, it appears to rely too heavily on strong assumptions, and the communication protocols that emerge do not seem to exhibit any interesting properties (i.e. the communicating agent simply learns to output messages that correspond to the desired action for the acting agent).
review_title: null

Meta-review: This paper proposes a cognitive science-inspired interaction setting between two agents, an "architect" and "builder", in which the architect must produce messages to guide the builder to achieve a task. Unlike other related settings (such as typical approaches in MARL, HRL, or HRI), the builder does not have access to the architect's reward function, and must learn to interpret the architect's messages by assuming the architect is telling it to do something sensible. At the same time, the architect determines what is "sensible" by building a model of the builder's behavior and planning over it. This setting is common particularly in human-agent interactions, where humans may not be able to either (1) accurately communicate a scalar reward or (2) provide demonstrations, but can still provide information that the agent ought to be able to learn from. The paper demonstrates that the learned communication protocol generalizes well to new settings.

While this paper generated a lot of discussion, the reviewers did not come to a consensus on whether the paper should be accepted or rejected, with those in favor of the paper maintaining it should be accepted and those not in favor maintaining that it needs work. I have therefore done a particularly close read of both the paper and the discussion in order to weigh the pros and cons brought up by the reviewers.

The positive reviews clearly indicate that this work is insightful and of interest to researchers in the ICLR community (in fact, all reviewers mentioned they found the work interesting and well-written). In particular, Reviewer hMeT wrote: "I am positive about this framework as it presents a better model for multi-agent communication, especially enriching the communication among agents over the fixed, restricted reward-based communication protocol in traditional RL." I am inclined to agree with this assessment and find the communication setting studied in this paper to be much more ecologically valid for human-agent interaction settings than having humans communicate scalar rewards or provide demonstrations: humans are typically poor at the former and may not have the same embodiment to achieve the latter.

The negative reviews focused on a few cons: (1) the assumption that the architect has access to a ground-truth environment model, (2) confusion about differences from other related fields (e.g. feudal RL, MARL), and (3) lack of analysis of the communication protocol. I have considered these points, but do not feel any of them are fatal flaws: (1) From the perspective of human-agent interaction, I think it is very reasonable to assume that a human architect would have a good model of the world and would be generally proficient at solving tasks in the world. Making this approach work in the setting where the architect is *also* learning how the world works seems squarely in the domain of future work. (2) The authors have done an extensive job of clarifying the differences between these related areas, and as discussed above, other reviewers found the way in which AGP is different to be insightful and ecologically valid. (3) This is potentially the most serious con: as the discussion with Reviewer BHGy brought up, the learned communication protocol may just be a simple mapping between messages and environment interactions. After further discussion in which the authors argued that learning a simple mapping is not a problem---the main question is how to even induce such a mapping in the first place---the reviewer acknowledged that this is not a fatal flaw but that makes the results somewhat less interesting.

In summary, the positive reviews highlighted the interestingness and insightful nature of the questions studied in this paper and have convinced me that this paper will be of interest to the ICLR community as it has provides a new perspective on the problem of agent-agent interaction (particularly for the special case of human-agent interaction). The negative reviews did highlight a few limitations of the paper, but I expect these can be addressed by future work and do not feel they outweigh the interestingness of the problem. In light of this, I recommend acceptance as a poster.

Suggestion for the authors: I found the discussion with Reviewer BHGy to be particularly insightful and helpful in understanding the aims of the paper. I would encourage you to incorporate some of this into the camera-ready version of the paper, and perhaps to lean more heavily on the special case of human-agent interaction as motivation of this work (as also hinted at by Reviewer hMeT).
Decision: Accept (Poster)