Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: The paper proposes a novel algorithm for encouraging synergistic behavior in multi-agent setups with an intrinsic reward that promotes the agents to work together to achieve states that they cannot achieve individually without cooperation. The paper focuses on a two-agent environment where an approximate forward dynamics model is learnt for each agent, and can be composed sequentially to predict the next environment state given each agent’s action. However, this prediction will be inaccurate if the agent’s affected the environment state in such a way that individual dynamics model cannot predict i.e. synergistic behavior was produced. This prediction error is used as extrinsic reward by the proposed approach, while also having a variant where the true next state is replaced by another approximation of a joint forward model which allows for differentiability of actions with respect to the intrinsic reward. Empirical analysis shows that this intrinsic reward promotes synergetic behavior on two-agent robotic manipulation tasks and achieves better performance that baselines and ablations.

I vote for weak accept as the paper proposes a novel intrinsic reward for promoting synergetic behavior in multi-agent systems, while also demonstrating that such an intrinsic reward can be differentiable if a joint forward dynamics model is approximated in addition to individual forward dynamics models given each agent’s actions. The paper does not show experiments beyond 2 agents and the four robotic manipulation tasks have been shown to work when provided with generic skills as an action space, which requires hand-defining or learning by demonstration.

From the description of the random policy, it is stated that a random policy over skills serves as a sanity check to ensure that the skills do not trivialize the task. This seems to suggest that the extrinsic reward only baseline did not use these skills and was disadvantaged. Some clarification is required here - did the extrinsic reward only baseline use the same skills as the proposed method? If it did, it would obviate the need to have a random policy sanity check. 

The paper suggests a baseline for separate arm surprise. In a similar vein, why wasn’t a joint-arm surprise baseline employed, which can basically treat both arms as a single agent?

Synergistic behavior seems hard to achieve for a large number of agents, but the paper does not give insights into whether such an algorithm will work for more than 2 agents. Typically multi-agent systems in prior work have worked with a large number of agents in environments other than robotic manipulation - such experiments may help in strengthening the proposed method.

review_title: Official Blind Review #2

final_score: rating: 8: Accept
review_content: The paper is technically sound and easy to read. I very much welcome that.

The authors address the important issue of exploration in reinforcement learning. In this case, they propose to use reward shaping to encourage joint-actions whose outcomes deviate from the sequential counterpart. Although the proposed intrinsic reward is targeted at a particular family of two-agent robotic tasks, one can imagine generalizing some of the ideas here to other multi-agent learning tasks.

The authors did mention testing with lambda=0 (intrinsic only), which did not aim to solve the 'task' but would be informative in terms of understanding the results of this particular bias. It would be interesting to include a plot on the effect of lambda across a range, say from 0 to 10.

review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: The paper focuses on using intrinsic motivation to improve the exploration process of reinforcement learning agents in tasks with sparse-reward and that require multi-agent to achieve. The authors proposed to encourage the agents toward the actions which changed the world in the ways that "would not be achieved if the agents were acting alone". The experiments are done with dual-arm manipulation.

The idea of guiding the agents toward the actions that they cannot do without concurrent cooperation is interesting. In this paper, it is presented by two types of intrinsic rewards: compositional prediction error and prediction disparity. The core component is the composition of these single-agent prediction model (f^{composed}). Although the formulation proposed is only based on intuition, the authors did enough ablation study to highlight the advantage of this loss function. 

Areas to improve:
+ The objects used in the experiment are symmetric, it is good to open your study to the task in which the objects are asymmetric or even deformable.
+ It is good to extend to the problem of multiple-agents (>2), while the order of agents who acts is important to compute "the expected outcome with individual agents acting sequentially" (for now, you only assume that the A will act first then the B acts later). 


review_title: Official Blind Review #3

Meta-review: The authors address the important issue of exploration in reinforcement learning. In this case, they propose to use reward shaping to encourage joint-actions whose outcomes deviate from the sequential counterpart. Although the proposed intrinsic reward is targeted at a particular family of two-agent robotic tasks, one can imagine generalizing some of the ideas here to other multi-agent learning tasks.

The reviewers agree that the paper is of interest to the ICLR audience.
Decision: Accept (Poster)