Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: ##########################################################################

Summary:
 
The paper provides an interesting way to add structure to MARL problems that have delay in the communication of state information. By explicitly building a predictive module for the future latent state of the agent and including that predicted state in the passed messages, it is possible that the agent will appropriately pass information that removes the effect of the delay in message passing across the network. They then apply this model to some interesting traffic light and cooperative vehicle control tasks.
##########################################################################

Reasons for score: 
 
Overall, I vote for rejecting. The idea is a nice contribution but the paper is quite confusingly written and the experiments do not adequately show the impact of the approach. In particular, the deviation between the method and the baseline does not appear to be particularly significant and the authors do not outline their hyperparameter search grid for both method and baselines which makes it quite difficult to tell whether a fair comparison was made. I really enjoy the applications and the idea but feel more careful experimentation is needed to demonstrate that the approach is valuable.
 ##########################################################################

Pros: 
 
1. The paper tackles a practical issue, namely that traffic control architectures are often necessarily decentralized and delayed. This is a good problem to study and a useful add on to existing work studying communication architectures in MARL settings.

2. The idea of adding a predictive forwards model whose output is passed to other agents is a clever way of adding structure to the problem to get around delays. Technically, this does not add anything as an LSTM could internally learn to do this type of prediction as it constructs its messages but structure can often be useful.
3. This partial model based approach may make the controller more robust to changing dynamics conditions or evolving agents.

 
##########################################################################

Cons: 

1. While you tune the value of alpha for your method, it is not clear that you have also given equal amounts of hyperparameter optimization efforts to the baselines. This makes it unclear if your method is actually better or whether you have simply found better hyperparameters. 
2. It would be worth expanding on the details of your baselines in the appendix more. It’s not necessary and doesn’t affect my review, but it would make for a better paper! 
3. The authors suggest that the world model may “reduce the impacts of nonstationarity” but do not run any experiments that demonstrate this. Additionally, since the world model is implicitly a function of the other agents in the system, the opposite may be true and experiments are not run to investigate this possibility.

 
A few ablations / changes that might lead to a more insightful paper but are not strictly necessary:
(1) Many of the environments you describe are many-hop and many time-step delayed. It would be interesting to see how your results evolve with longer imagination horizons. 
(2) It would be useful for the reader to see the evolution of the world model accuracy over time, to see if it is in fact stationary or convergent. Without this graph, it’s unclear if you are even able to learn a good future predictive model. One way of demonstrating this (though by no means the only way) would be to learn a decoder on the hidden state and see if you can reconstruct the relevant world state variables.
(3) Please add a hyperparameter grid to the appendix so we can see if fair comparison was made to the baselines. Since there do not appear to be existing benchmarks for the baseline methods on these tasks, it is probably necessary to give them equal amounts of tuning effort.

#########################################################################

Things that would improve readability:

- What does it mean to share the policy with the agents around you? What is the representation that is shared?
- The description of environments is a wall of text that is hard to read. Maybe describe the environments at a higher level and move detailed descriptions to the appendix?
- Grammar: grammar throughout the paper is pretty inconsistent and there are too many grammar issues to point them out individually. It may be helpful to pass the paper through an automatic grammar checker and do several more proofreads.
- There is a \varphi in section 4.4 whose purpose I don’t understand; it does not seem to be used anywhere.
- It would be useful to motivate the lower bound that appears in equation (2) 
- Including the architecture of the module for a 2-agent case directly in the body of the paper rather than in the appendix would probably make the notation a lot easier to read. As it was, I am still not 100% certain that I understood the notation correctly.
- Figures are small and low-res; I can’t see them very well even by zooming in.
review_title: Interesting idea but further experimentation is needed to understand the impact of the proposed approach

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper proposes to communicate predicted local states between neighboring agents to address the problem of delayed information in networked multi-agent reinforcement learning. To enable agents to predict future states, a world model is learned at each agent. It is empirically demonstrated that the proposed method has good performance in traffic signal control and cooperative adaptive cruise control. 
+ The idea of communicating predicted local states is interesting and could mitigate the problem of delayed information.
+ The proposed method is empirically verified in two scenarios and outperforms other communication methods in MARL.
+ The tuning of hyperparameter (alpha) is also verified by experiments.

Concerns:
- The main concern about this paper is lack of novelty. It seems just applying the world model for single agent directly to MARL without any adaptation. Equations 2, 3, 4 are taken directly from (Luo at el, 2018) without any consideration of the characteristics of MARL. For example, why do these still hold in multi-agent case? Without rigorous mathematical analysis, it is hard to say it is valid. The proofs given in Appendix are not informative at all. 
- As in MARL, the state transition depends on the joint action of agents, why does equation 3 hold in practice?  It is just not clear. Any assumption is made for that? I am wondering how well the world model is learned in the experiments. 
- In the experiments, only one-step state prediction is used. It is claimed that two-hop information already leads to good performance. However, it is not supported by experiments. What if each agent directly uses two-hop information without prediction (it is easy to implement in simulation). This can serve as a baseline so as to illustrate how the learned model affects the performance empirically. 
-  The communication baselines, i.e., CommNet and DIAL, are pretty old. Why not use more recent methods such as ATOC, IC3Net, TarMAC, which perform much better than CommNet and DIAL.

---
**After rebuttal**

The authors' responses do not address my major concerns (the first two). I do not think the responses directly answer my questions.
So, I keep my score unchanged.   

review_title: Review 1

final_score: rating: 4: Ok but not good enough - rejection
review_content: To reduce the delay of global information, the authors propose an imagination module to predict future information for communication. The agent predicts its future state and shares that with its neighbors. The information delay is an important problem in real-world applications. The proposed ImagComm outperforms other baselines on a range of NSC tasks.

However, I have some concerns:

What are the difference between predicting the future information at the sender end and the receiver end? Since the sender makes a prediction based on its history information, and the history information has been communicated to the receiver, the receiver could make the same prediction using its collected information, which could reduce the communication cost.

$f_i$, $g_i$, and $W_i$ are trained by Eq 6c. The Eq 6c would make the outputs of $W_i$ and $g_i$ close to each other. Would it cause mode collapse, which means that the outputs of $W_i$ and $g_i$ become a constant representation?

In experiments, the ablation study that removes the optimization objective Eq 6c in Algorithm 1 could help to verify that the performance gain is caused by the predictive information rather than the additional information.

review_title: Reviews

Meta-review: This paper proposes a technique of communicating predicted local states between agents in multi-agent reinforcement learning to deal with the delay in communication.  While the paper addresses an important practical problem, the reviewers have concerns about the insufficiency of novelty and experimental validation.
Decision: Reject