Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: The authors present a fun and effective idea to translate a peer's message in terms of one agent's own experience. The benefit of doing so makes sense intuitively and is verified to be effective empirically.


Strengths:

+ The motivation is clear, and the key idea is well-presented. Paper is positioned well in relevant works of communication-aided MARL research.

+ Evaluation is thorough and indicative of the authors' claims.


Major Concerns:

- The reviewer has yet to discover a major issue with the paper with regard to its correctness, contribution, novelty, and effectiveness.
review_title: Intriguing and effective novel approach to off-policy learning

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: Summary:

This paper considers communication games when agents use experience replay. The agents' communication protocol may change over time, leaving outdated symbols in the replay buffer which are then trained on. This paper proposes replacing the old communication actions with up-to-date actions as the transitions are sampled, and shows that this leads to greatly improved convergence speed and higher performance plateaus.

--------------------

Positives:

- The problem of multiagent communication and how to learn it is important and relevant to the ICLR community. The solution presented here seems like a natural fit with the problem and popular agent architectures and is well presented.

- The paper is well motivated and well written.  Overall it was an enjoyable and easy read!

- The experiments in Figures 4, 5, and 6 seem like great choices to show the strengths of the approach.  They're simple, well described, and well targeted.

--------------------

Negatives:

- I feel like there's a pretty obvious question about "What happens in richer domains?" that (unless I missed it) isn't addressed in the paper - I'll expand on that in my 'Questions to clarify recommendation' section below. While the technique seems to work very well in the experiments chosen for the paper, I wish the paper touched a bit more on upcoming challenges, possible foreseen problems, and next steps.

--------------------

Recommendation and Justification:

Overall, I feel like this was a strong paper and should be accepted. My only real negative was that I am excited to know more about what comes next.

--------------------

Questions to clarify recommendation:

The three environments presented in the paper, if I've understood them correctly, are pretty straightforward in that 1) the speaker only has communication actions (and no environment actions), and 2) seems to only have one consistent message to communicate during the entire episode (after perhaps waiting to receive a message from others, in Hierarchical Communication). But right from the abstract onwards, I was wondering about possible problems in richer domains, where it seems like this technique could be harmful. Specifically, what if by updating the old communication action to one chosen by the current policy, we present a communication action that no longer aligns with the old environment action, which we do not update?  I felt like this was a pretty natural question, but unless I missed it, the paper doesn't mention possible problems like this.

I'll ground this in an example, similar to Cooperative Communication.  Imagine a two-player gridworld where the players cannot see each other, but are rewarded for arriving at the same map location.  Similar to Bach and Stravinsky / Battle of the Sexes, each player has a different preference over locations, but being at the same location is most important. Let's call the locations Left and Right.  To enable coordination, let one player be a Speaker that can take communication actions to signal the other player as to where they should meet in that episode. While that permits greedy Speaker policies (always announce their preferred location and then go there) and greedy Listener policies (always go to their preferred location, regardless of Speaker's announcement), it would also allow the speaker to arrange a correlated equilibrium: announce a randomly chosen location in each episode and then go there, to maximize joint reward beyond any greedy Nash equilibrium policy.

Here's where I see a possible failure with the technique in this paper.  Assume that the replay buffer contains an episode where the speaker emitted symbol L (for left) and then took environment actions to move to the Left location.  Later in training, using the technique presented here, we might sample this experience, update the symbol to L', and still move Left.  As described in this paper, I would expect that should work, and converge faster than by using the out-of-date symbol L.  However, it seems possible that the newer Speaker policy might prefer to move Right on that episode instead.  Updating the symbol would change it from old L to new R', but since the technique does not (and cannot, without a world model) update the environment actions, it seems like the listener and speaker would then train on this misaligned tuple of communication action R' and environment actions to move Left.  I would expect this confusing example to be much worse than training on the original example with the outdated but still aligned communication actions.

More generally: how can we make sure that the updated communication actions still align in intent with the agent's environment actions that we cannot change? It seems like conditioning the communication action on the agent's environment action for that timestep might help, but would only be a partial solution: if an agent must speak now but take their first significant environment action in the future, we would have the same problem.

My questions regarding this point are:
- Do you agree that this could be a problem in richer environments than those presented in the paper?
- If so, do you foresee an easy solution, or will this be a challenge for future work?

I think the paper is strong enough as-is, and does not need an experiment in this paper to investigate richer games like this.  However, if the authors agree that the technique could fail to help or could harm convergence in richer settings than those presented in the paper to support the technique, then I think a couple of sentences about future challenges and future work are warranted.

-------------------

Issues and Suggestions:

- Nit: Pg2, Experience Replay. The first sentence describes the agent as receiving (s_t, a_t, ..., s_t+1) at each time step. Should this be (o_t, ..., o_t+1), since the agent receives observations and not environment states?

- Typo: Pg2, MADDPG.  'uses deterministic polices' --> policies

- Suggestion: Pg3, Methods section and Equation 4. Equation 4 describes the tuple as containing r^e_t+1, r^m_t+1, but the text in the paragraph above only mentions r_t'+1, and the text below only indirectly clarifies what r^e and r^m are when it describes the cheap talk setting where r^m=0.  This threw me for a while when I read the equation, and scanned back up the page to try to see where r^e and r^m were defined, and they aren't. I suggest changing the sentence in the previous paragraph from "receives rewards r_t'+1" to something like "receives rewards r_t'+1 (split into an environmental reward r^e and a messaging cost r^m)..." to clarify this before the symbols are used.

- Typo: Pg4, Ordered Relabelling. "may themselves by conditioned" --> "may themselves be conditioned"

- Clarify: Pg4, under equation 6. The sentence "...we sample an extra o^m_t-1 in order to determine (using the other agents' policies) the new \^{o}^m_t, which allows us to relabel...". I don't understand what this sentence is trying to say. Which player is this for? The symbol \^{o}^m_t doesn't appear in equations 5 or 6, so I don't understand what sampling an extra o^m_t-1 would do, since it's to compute a symbol that doesn't connect with the equations being discussed. Maybe I'm just missing something obvious, but I spent a couple of minutes trying to figure this out, before giving up and moving on.

- Nit: Pg5, Implementation. Extremely minor, but the phrasing "we can therefore only relabel..." suggests a limitation of the approach (e.g., we are only able to do this...) whereas I think you're suggesting a performance win (we can do this using only...). I feel like flipping the words to "we can therefore relabel only a single..." better communicates that.

- Typo: Pg9 and 10, References. In both of the references including Pieter Abbeel, his affiliation is prefixed (OpenAI Pieter Abbeel). No other authors' affiliations are listed, so this just seems like a .bib typo.
review_title: Accept: Correcting experience replay for multi-agent communication

final_score: rating: 7: Good paper, accept
review_content: ---- Summary ----
The paper proposes a method for modifying an experience replay when learning in communication environments, by relabelling messages using the latest policy.

---- Reasons for score ----
The paper addresses the problem of non-stationarity in an important class of multiagent environment. The correction proposed is simple, and effective in the domains it is tested in. My main concern is how broadly the method applies, which I am uncertain of from the paper.

---- Pros ----
The paper provides a simple way to better leverage replay data for communication environments, addressing non-stationarity in those environments. This is an important problem in multiagent environments.

The experiments show improvements in learning speed and final reward in some communication domains. These cover a few important cases for the algorithm, including hierarchical communication and communication with an adversarial listener.

The paper is well situated in the literature on emergent communication, and it is clear and well written throughout.

---- Cons ----
My main worry is that the paper leaves me uncertain on when the method can be applied. The OCC algorithm suggests that a necessary condition is that the message graph is acyclic. However, this limitation is not explicitly discussed, and I am also unsure whether an acyclic message graph is a sufficient condition. For example, does the method apply with multiple listeners acting in the same environment, or when the speaker also acts? It would strengthen the paper to be more precise about the settings where these algorithms can be applied and can be expected to help.

The discussion on Covert Communication in the appendix significantly changed how I understood the results of this experiment; in particular, the agents have not solved the task intended in the environment (using the key to communicate), but instead appear to be constantly changing strategy to outpace the listener - with the key being irrelevant. This is hinted at in the main text, but I think it is central enough to the interpretation of the experiment that it should be moved there.
review_title: Interesting method, could use more discussion of when it applies.

final_score: rating: 7: Good paper, accept
review_content: The paper considers a multi-agent reinforcement learning (MARL) scenario where agents take actions based on the current observation alone. The paper proposes a communication correction mechanism where, during the centralized training, messages there were received in the past from other agents are reevaluated according to the updated policy. This way old messages can be updated instead of discarded, which is more efficient overall. 

The paper is clearly written and easy to follow. The experiments are enough to convince that the communication correction idea is effective. The idea is not very deep or insightful, so the significance of the contribution depends on how useful this trick will be in practice. This is completely fine, but since I'm not doing research in this area, it's hard for me to judge if this idea will be of value to other researchers, given the existing alternatives in the literature. Therefore my review focuses on the presentation of the idea. 

I'm missing some intuition regarding why applying this correction is always "safe". Couldn't it somehow drift since it always gives so much importance to the new learned actions? what if for some period of time the new actions are worse than the old ones? Have you encountered any scenarios where this correction (at least if not properly tuned) degraded the performance? 

Please elaborate on the assumption of actions that only depend on current observation. This seems very restrictive. How common is this assumption in the relevant literature? As a Markov strategy, it makes more sense if the state summarized the history of the game in some intelligent way. Is this the case for the scenarios you tested? 

I think it can be insightful to also experiment with a scenario where communication is noisy. Specifically,  the scenario when there is a probability of p that a message is not received looks interesting to me. The reason is that this kind of randomness seems to be essentially different than the noiseless case, as opposed to a weak Gaussian noise. With this kind of noise, the sampling part of the algorithm will often sample a communication error when trying to relabel a message, effectively discarding the old message. Is this desirable or acceptable? Would one need to prevent this possibility if communication is random? Since questions like this arise (if it makes sense), I'd be careful not to make statements about how easily the method can be applied for more general scenarios (like with noise), especially since the paper is 100% empirical so we only see that what was tested works. Also, please define the ~ sign explicitly to avoid confusion. 

Is this a realistic assumption that p(o|a) is known? it requires a model for the type of noise and communication failures that will occur in real-time, making the method not entirely model-free. Please discuss. 

I didn't find Fig.1 very helpful. What does it contribute over (5)? 

If you used the correction in Section A.3, why is (1) presented in the paper? seems a bit misleading to clarify that only in the appendix. 

Please add a reference for "cheap talk". 

The abstract should what is the basis for the claim that "it substantially improves...", so at least mention this is based on experiments. 

"We analyze the performance of this task in Figure 3" - seems to be an overstatement, there is no analysis here, just reporting the results of the experiment. 



review_title: Possibly a useful idea supported by interesting experiments, but the presentation lacks depth and detail

Meta-review: This work proposes a simple and intuitive way to improve how to learn a communication protocol off-policy in the non-stationary situation in which messages received in the past do not reflect an agent's current policy. The authors introduce a communication correction that relabels the received message adjusting it to the current policy. The authors show that this method, besides being simple, is effective in a number of experiments. As observed by some reviewers, an issue with the method is that it is not clear how it would scale up to more complex environments than those considered. However, the authors addressed the concerns during the response phase, both adding new experiments, and with a clear statement of what are the outstanding issues. The paper is certainly a clever and solid contribution to the area of multi-agent communication learning, and I am strongly in favour of accepting it.

Decision: Accept (Spotlight)