Venue: ICLR 2022 Track Blog

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: The blogpost looks at the work on rules and conventions by Shih et al (henceforth Rules&Conventions), and additionally describes in good detail these 4 other works: BAD, SAD, Other-Play, and Modular Policy Networks in Robots (MPN).

Its description of Rules&Conventions is detailed and accurate, noting the main takeaway of learning modular representations that can transfer to different partners and tasks. The blogpost then tries to "discuss how [Rules&Conventions] is used in contrast to other approaches" by putting it in the context of the other 4 works.

The broad aim of the blogpost in clarifying the relationship between these recent works is indeed useful, since these works can all seem intertwined to someone unfamiliar with the field. A successful blogpost in this regard can help organize the related literature for researchers looking to contribute to the field of multi-agent coordination.

However, the blogpost falls short on presenting the comparisons in an easily digestable form. The comparisons are sprinkled throughout the discussion, e.g. "[SAD] is different from [Rules&Conventions] in that it does not save this information for transfer between similar tasks or new partners", "[Other-Play's] approach to adaptive Human-AI collaboration doesn’t rely on the use of conventions at all", and are buried in the text which is spent largely on describing the low-level details each method. In addition, much of the low-level details of the other methods are being discussed in isolation, without much clarity on how they are relevant to the high-level discussion of Rules&Conventions.


In my opinion, I would restructure the blogpost in the following way:
- Have a big front figure depicting three stages in the lifecycle of a multi-agent collaboration.
1. Training: learning the representations for developing conventions and abiding by the rules
2. Zero-Shot/Instantaneous Coordination: deploying our model to coordinate with new partners/tasks
3. Few-Shot/Adaptive Coordination: tweaking our learned representations to improve adaptaion with new partners/tasks

- Discuss which stages the five works are aiming to solve:
- BAD/SAD are improving Stage 1 by learning better conventions for solving difficult games (5-player Hanabi)
- Other-Play is improving Stage 2 by learning strategies that are agnostic to conventions.
- MPN/Rules&Conventions are improving Stage 3 by learning modular representations for better adaptation.

- Dive more deeply into each work, but only focusing on details that provide additional insight into the high-level comparisons described above.
- For example, we can highlight that learning conventions is beneficial in general (Stage 1 & 3), but is actually harmful in Stage 2 for Zero-Shot Coordination, since there is no time to learn/adapt to new conventions. This is the main insight of Other-Play, which proposes to learn convention-agnostic strategies. Then we can provide details on how exactly that is done.
- It's clear how BAD/SAD differs from MPN/Rules&Conventions, since they target different stages in the lifecycle of a multi-agent collaboration. But, it's not clear how those works within the same stage differ from each other. We can then describe the differences between BAD/SAD (SAD came after BAD, and proposed an algorithmic simplification), and the differences between MPN/Rules&Conventions (one focuses on robot/environment, the other on partner/task).

- Forward Looking (room for improvement) section
- Currently the blogpost describes additional works and conceptual axes (self-play / scale / human-subjects), but this dilutes the main message of the blogpost.
- I would suggest re-emphasizing the stages in the lifecycle of a multi-agent collaboration, and thinking about potential future directions that can unify / organize these stages.
- For example, can we have one algorithm/framework that addresses all stages? If not (since Stage 2 may be incompatible with Stage 1&3), what are the tradeoffs, or are there any principled statements we can make about their incompatibility.
- What are other stages in the coordination lifecycle that are interesting? Are the stages always so clean cut, or are they blurred in real applications people care about? What techniques can be transferred across different stages?

In summary, the content of the blogpost is detailed and accurate, but can benefit from having a more focused message.
review_title: Need improvements

final_score: rating: 4: Ok but not good enough - rejection
review_content: This blog post looks at several 5 works on rule and convention transfer published in the last few years. 

I appreciate some of the information presented in this blog post, especially the explanation of the concepts of "convention" and "rule". I also like the illustrative example, although, given the familiarity of American football being restricted mostly to North American audiences, I would suggest using a game that has a broader appeal. In addition, any real-time dynamic game is much too far-fetched to serve as a running example for existing rule and convention transfer algorithms -- these algorithms are a long way from being able to handle such games. 

My main issue with this post though is that it leaves the reader highly confused as to what the takeaway or even the main intent of this post is. At first, I thought it was going to focus on the article by Shih et al, 2021. The post briefly discusses this approach and then shifts to another approach, BAD, supposedly in an attempt to provide a context for Shih et al, 2021. It describes BAD in as much detail as Shih et al, 2021 itself, but I'm still not sure how the two are related, other than using the same Hanabi benchmark. The the blog post goes on to describe SAD. Its relationship to Shih et al, 2021 is clearer, but its description seems unnecessarily long. Then the post discusses Other-Play and MPNs. What confuses me about their discussion is that Other-Play and MPNs are specific algorithms, each with a section dedicated to it, whereas SAD and BAD don't have their own sections -- instead, they are both in the section Learning Conventions. I would expect either each section to be about a high-level idea or each section to be about a concrete algorithm. The post ends with a section "Improvements to Human-AI Collaboration. This is also bizarre, because the post started with a very specific paper -- Shih et al, 2021 -- so I don't understand why it wraps up with discussing possible improvements to HAI collaboration in general rather than improvements to Shih et al, 2021 in particular. Overall, it's unclear why the post claimed at the beginning that it was trying to provide context specifically for Shih et al, 2021; it seems to simply describe 5 works on a broad topic, without strongly emphasizing any of them.

It's difficult to pinpoint exactly what makes this blog post so confusing to me, but I believe it's verbosity and lack of structure. The blog's narrative meanders too much and constantly mixes high-level insights with low-level details. 
 
The same issues apply at the level of individual sentences: many sentences are verbose and/or poorly structured. A number of phrases are used incorrectly despite remaining comprehensible, e.g., "is catered to" instead of "caters to". I started compiling a list of error corrections but stopped after noting down a few, because I realized that there are too many low-level things in need of fixing. The main aspect that needs to change is the writeup's high-level structure though. The corrections I've noted down are at the end of the review.

Unfortunately, these problems mean that the post falls short of the blog track's main objective -- provide clarity and/or insights about prior works. Therefore, I can't recommend it for acceptance in its present form.


============================




"the bounds they are allowed to stand within while holding the football during play when they are allowed to pass the line of scrimmage and the requirements for earning points" 
--> 
"the bounds they are allowed to stand within while holding the football during play, when they are allowed to pass the line of scrimmage, and the requirements for earning points" (Note the commas.)



", and as such faster adaptation..."
-->
"; thus, faster adaptation..."



"Intuitively, learning separation representations between tasks and partners allows them to specifically generalize to new tasks or new partners of their choice."
-->
This sentence is broken in multiple ways, I'm not sure what it's trying to say. What are "separation representations"? Did you mean "separate representations"? Also, what does it mean to "specifically generalize"?



"...work, before discussing..."
-->
"...work before discussing..."



"improvements in this approach"
-->
"improvements to this approach"



"The mechanics of Shih’s approach are going to be important in distinguishing it"
-->
"The mechanics of Shih’s approach are important for distinguishing it"

review_title: Good intentions but poor structure

Decision: Reject