Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: The paper presents a new algorithm for maximizing the diversity of different policies learned for a given task. The diversity is quantified using a metric, where in this case the total variation is used. A policy is different from a set of other policy if its minimum distance to all the other policies is high. The authors formulate a new constraint optimization problem where the diversity to previous policies is lower bounded in order to avoid a tedious search for combining task reward and diversity reward. The algorithm is evaluated on different Mojoco locomotion tasks. 

Positive Points:
- The idea of maximizing the minimum total variation is novel and interesting
- The approach seems to work better than current SOTA approaches for generating diverse behavior

Negative Points:
- The paper needs to be improved in terms of writing as in particular some of the main parts of the algorithm are unclear
- The definition of Eq 7 does not make too much sense to me (see below)
- The results have high variance and some conclusion drawn from it are hard to verify given the plots

More comments:
- Eq 7 does not seem to be a very good choice to me. Why does the total variation needs to be different at *every* time step? We can certainly generate very diverse behavior even if the policy is exactly the same for some states. It could even be the case that for some states, only one action does not lead to a failure. In this case, Eq 7 would completely fail to produce any valid policy (?)
- In general the writing is clear, it gets however quite unclear for the main part of the algorithm (after Eq. 7). It is unclear how equation 8 is obtained and why the limit of alpha going to 0 should lead to the same solution as Eq 7 (if alpha is 0 than it should be the same as optimizing just the task reward??). While this might be obvious for experts of the interior point method, it needs to be explained in much more detail in this paper. I think it is always a good strategy to make a paper self-contained, in particular for the main parts of the algorithm.
- Also the termination mechanism needs to be much better explained. What reward is given in this case? The current formulation sounds quite heuristic to me, but maybe a better explanation can fix that.
- while Fig 3 shows a clear advantage of the method, the section about better policy discovery would need better data to verify their claims. Fig 4 shows very noisy results and while for the hopper there might be a clear improvement of performance for number of policies > 2, this does not seem to be very significant for half cheetah. Given the amount of noise in the results many more trials would be needed to really make such statements.

review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: This paper proposes a new way to incentivize diverse policy learning in RL agents: the key idea is that each agent receives an implicit negative reward (in the form of an early episode termination signal) from previous agents when an episode begins to resemble prior agents too much (as measured by the total variational distance measure between the two policy outputs). 

Results on three Mujoco tasks are mixed: when PPO is combined with the proposed objective for training diverse policies, it results in very strong performance boosts on Hopper and HalfCheetah, but falls significantly short of standard PPO on Walker 2D. I would have liked to see a deeper analysis of what makes the approach work in some environments and not in others. 

Experimental comparisons in the paper are only against alternative approaches to optimize the same diversity objective as the proposed approach (with weighted sum of rewards (WSR) or task novel bisection(TNB)). Given that this notion of diversity is itself being claimed as a contribution, I would expect to see comparisons against prior methods, such as in DIAYN. There are other methods that have been proposed before in similar spirit to induce diversity in the policies learned. Aside from the evolutionary approaches covered in related work, within RL too, there have been methods such as the max-entropy method proposed in Eysenbach et al, "Diversity is All You Need...". These methods, evolutionary and RL, could be compared against to make a more convincing experimental case for the proposed approach.

The experimental setting is also not fully clear to me: throughout experiments, are the diversity methods being evaluated for the average performance over all the policies learned in sequence to be different from prior policies? Or only the performance of the last policy? Related, I would be curious to know, if K policies are trained, the reward vs the training order k of the K policies. This is close to, but not identical to the study in Fig 4, to my understanding.

Aside from the above points being unclear, the paper in general could overall be better presented. While I am not an expert in this area, I would still expect to be able to understand and evaluate the paper better than I did. 
- Sec 3.1 makes a big deal of metric distance, but never quite explains how this is key to the method.
- The exact baselines used in experiments are unhelpfully labeled "TNB" (with no nearby expansion) and "weighted sum of rewards (WSR)", with further description moved to appendix. In general, there are a few too many references to appendices.
- The results in Fig 2 are difficult to assess for diversity, and this is also true for the video in the authors' comment.
- There is an odd leap in the paper above Eq 7, where it claims that "social uniqueness motivates people in passive ways", which therefore suggests that "it plays more like a constraint than an additional target". 
- Sec 5.1 at one point points to Table 1 for "detailed comparison on task related rewards" but says nothing about any important conclusions from the table.
- There are grammar errors throughout. 
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: This paper proposes a new method for learning diverse policies in RL environments, with the ultimate goal of increasing reward. The paper develops a novel method, called interior policy differentiation (IPD), that constrains trained policy to be sufficiently different from one another. They test on 3 Mujoco domains, showing improved diversity in all of them and improved performance in 2 of them.

Overall, this paper is very well executed. The explanation of the method is thorough, and the paper is well-written and polished. I like the idea of enforcing a constraint on the policy diversity via manipulating the transitions that the agents can learn from.  The experiments section compares to two other methods of increasing policy diversity and IPD outperforms both of them. I think this is a solid contribution to the literature on improving policy diversity.

That being said, I have some concerns about the paper:
1) The motivation for explicitly encouraging diverse policies is a bit confusing, and isn’t very convincing. The paper draws inspiration from social influence in animal society, and say they formulate social influence in RL. First, the term social influence already has an established meaning in RL (see e.g. Jaques et al. (2018)) and refers to agents explicitly influencing others in a causal way in a multi-agent environment. Second, I think calling policy diversity a form of social influence is a bit of a stretch (and anthropomorphizes the agents unnecessarily). I think the paper should scrap the ‘social influence’ angle and instead frame it as ‘increasing policy diversity’. 

The paper also motivates itself in comparison to Heess et al. (2017), which uses a set of environments to get diverse policies. However, the goal of these works are different: in Heess et al., the goal is to train agents that can exhibit complex behaviours in relatively simple environments (the focus is more on complexity of behaviours vs. the fact that agents in the same environment learn diverse policies). In this work, the goal is not to develop any more complex policies, but to have different agents on the same task learn diverse policies (and since the experiments are in Mujoco, the degree of diversity is limited). Thus, while the works are related, I don’t think the Heess et al. paper is good motivation for this work. 

I think the primary motivation that makes sense for explicitly encouraging diversity is to improve final performance on the task. Thus, I think it would be best for the paper to clarify the introduction by focusing on this. The paper could also give some reasons why having diverse policies is inherently a good thing (maybe for some applications with humans-in-the-loop it could be helpful?), but currently this is absent. 

2) Given that improving the final reward of an RL agent is the main goal, it’s not clear that the experiments (in 3 simple Mujoco settings) are enough to show this reliably. Specifically, it’s unclear whether encouraging diversity in this way will generalize to more complex tasks or domains (e.g. tasks in Mujoco with sparser reward, or environments with larger state spaces). It is possible that the success of the technique is most prevalent when there is only a small observation space. 

3) I’d like to see more discussion / analysis of *why* we’d expect diverse policies to lead to better rewards. In work on intrinsic motivation / curiosity for better exploration, it’s clear that encouraging agents to visit unseen states will lead to a better exploration of the state space, and thus will make them more likely to stumble upon rare rewards. But is this also true for policy diversity? Currently, the paper speculates that encouraging diversity could help agents not all fall into the same failure mode. But I could also imagine that it could lead agents to avoid a successful strategy that another agent learned. For example, if a certain sequence of moves is necessary at the beginning to avoid termination, the first agent could find this sequence of moves, but the other agents might avoid this sequence for the sake of diversity (depending on the threshold). Does something like this happen in practice? In my opinion, the environments considered aren’t rich enough to know. 

4) There are also some inconsistencies in results section. Specifically:
- There seems to be a disagreement between the results in Table 1 (which uses 10 peers) and the ablation over number of peers in Figure 4, which shows that the performance with 10 agents is roughly the same as it is with 1 agent (and overall shows little positive trend between the number of peers and performance). 
- If ‘success rate’ means ‘percentage of time beating average PPO policy’, why does PPO sometimes get 100% in Table 1? 

Given the concerns above, I’d assess the paper as being borderline for accept. I’m currently erring on the side of rejection, but I’d consider changing my score if some of the above points are addressed. 


Smaller concerns and questions:
- There are a couple of instances where I found the claims of the paper with respect to related work to be over-stated. For example:
‘Yet designing a complex environment requires a huge amount of manual efforts’ -> not necessarily. There is an initial engineering overhead, but it’s possible to generate environments programmatically with different properties, resulting in different agent behaviours. 
Also:
On the Task-Novelty Bisector method of (Zhang et al., 2019): ‘the foundation of such joint optimization is not solid’. This is given without any explanation --- how is it not solid?

- In the TNB and WSR implementation, what metric is being used? Is it the same as is defined in Section 3?

- It would be nice to have some videos of the agents behavior to be able to more easily assess the learned policy diversity. 


Small fixes:
‘and similar results can be get’ -> and get similar results. 

review_title: Official Blind Review #1

Meta-review: The paper proposes a mechanism for obtaining diverse policies for solving a task by posing it as a multi-agent problem, and incentivizing the agents to be different from each other via maximizing total variation.

The reviewers agreed that this is an interesting idea, but had issues with the placement and exact motivations -- precisely what kind of diversity is the work after, why, and what accordingly related approaches does it need to be compared to.
Some reviewers also found the technical and exposition clarity to be lacking.

Given the consensus, I recommend rejection at this time, but encourage the authors to take the reviewers' feedback into account and resubmit to another venue.
Decision: Reject