Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper proposes a multiagent learning algorithm for networked system control. The main contributions are 1) a spatial discount factor that can stablize the learning process, 2) a differentiable communication protocol NeurComm. The paper was evaluated thoroughly by comparing against recent MARL baselines on two realistic environments.

I voted for "Weak Accept" because this paper can have important real-world applications such as traffic control, autonomous driving and power grid. I really like its evaluations on the realistic environments. In addition, the paper is clearly written, the algorithm seems reasonable and the evaluations are comprehensive. 

I did not give it a higher rating because the proposed algorithm only outperforms the baselines in two out of four test cases. It is not a very strong result. In this situation, I would hope that see a more detailed discussion why in CACC experiments, NeurComm does not perform as well. And I would also suggest the paper adding more test cases to show that the proposed algorithm indeed can dominate in most of the cases.
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: 1. Summary

The authors use decentralized MARL for networked system control. Each agent might control a traffic light (exp 1) or a car in traffic (exp 2). Some features of their approach are a spatial Markov assumption (only neighborhood matters), a spatial discount factor, and NeurComm: a general message passing scheme between agent policies. The authors compare their method with CommNet (averages messages before broadcast), DIAL (small-scale direct communication), etc.

2. Decision (accept or reject) with one or two key reasons for this choice.

Weak accept.

3. Supporting arguments

The experiments and analysis of the more general communication scheme are nice and the assumptions used make sense for the environments considered (spatial interactions and dynamics).
The 2 environments used are nice, but it would be nice to see non-traffic applications as well.
The methodological contributions make sense in the spatial domain, but it would be interesting to see how neighborhood-based assumptions can be used in non-spatial / non-Euclidean domains.

4. Additional feedback with the aim to improve the paper. Make it clear that these points are here to help, and not necessarily part of your decision assessment.

It would be nice to explain why / what communication strategy enables FPrint to do better than NeurComm.

5. Questions
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: This paper is concerned with network multi-agent RL (N-MARL), where agents need to update their policy based on messages obtained only from neighboring nodes. This is done under sensible restrictions on the state transition distribution, which can be claimed to hold true in realistic networked settings. The authors argue that introducing a spatial discount factor (along a temporal one), where neighboring nodes have a small distance, stabilizes learning. Also, they provide a way of learning a networked communication protocol. Experiments are done on somewhat realistic simulations of traffic.

I am not an expert in this type of distributed MARL, and what the SotA there is. What is proposed here makes sense to me, even though it also does not look very surprising to me. Spatial discounting seems to make sense. The authors do not comment on the relationship between the d_{ij} (or how to choose them) with the neighborhood graph, I suppose that neighbors have short distances, while non-neighbors have not. The NeurComm protocol is a bit vague to me, and I cannot say how it relates to other proposals. I find it a bit strange (unless I misunderstood something) that complete state and policy information is transmitted to every neighbor, and on top some latent vector. This sounds expensive to me, surely these communication channels are limited? It'd be more interesting to consider bandwidth limitations here. Without knowing related work on other protocols in detail, I suspect they probably work less well than NeurComm, simply because they allow for smaller messages only (f.ex., whereas they talk about "policy fingerprints", they seem to submit the complete policy parameters to neighbors -- that is not just a fingerprint). Maybe I am wrong, but if so, the paper does not explain things properly (there is some encoding and decoding going on, but just to propagate the hidden states at each node).

To me as an outsider, this looks like interesting work with well-simulated experiments. Of course, these are simulated and allow for no real-world conclusions, but then much of RL work does neither. It is just I'd be hard pressed to say what is really surprising in this paper, and what could be learned from it.

review_title: Official Blind Review #2

Meta-review: The paper focuses on multi-agent reinforcement learning applications in network systems control settings. A key consideration is the spatial layout of such systems, and the authors propose a problem formulation designed to leverage structural assumptions (e.g., locality). The authors derive a novel approach / communication protocol for these settings, and demonstrate strong performance and novel insights in realistic applications. Reviewers particularly commended the realistic applications explored here. Clarifying questions about the setting, experiments, and results were addressed in the rebuttal, and the resulting paper is judged to provide valuable novel insights.
Decision: Accept (Poster)