Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: RL in environments with deceptive rewards can produce sub-optimal policies. To remedy this, the paper proposes a method for population-based exploration. Multiple actors, each parameterized with policies based on Normalizing Flows (radial contractions), are optimized over iterations using the off-policy SAC algorithm. To encourage diverse-exploration as well as high-performance, the SAC-policy-gradient is supplemented with gradient of an “attraction” or “repulsion” term, as defined using the KL-divergence of current policy to another policy from an online archive. When applying the KL-gradient, the authors find it crucial to only update the flow layers, and not the base Gaussian policy.

I have 2 issues with this paper:

1.	Lack of novelty – Most of the components of the proposed algorithm have been researched in other works. SAC with NF was contributed by Mazoure et. al., and this paper seems to use the exact same architecture.  The diversity-enforcing objective (Equation 1.) was proposed by Hong et. al., albeit not with a separate archive of policies, but using the experience replay (for off-policy) and recent policies (for on-policy). The objective is Section 3.2 is also the same as that in Hong et. al. 

2.	I have major concerns about how the results have been reported:
     a.	The authors have omitted SAC and SAC-NF from Figure 4, and therefore the figure gives the impression that ARAC is significantly better than the presented baselines in some of the environments. For example, take Humanoid-v2. The original SAC paper reports reaching close to 5k score in about 0.5 million timesteps. It therefore seems that most of the benefit of ARAC (over the presented baselines) comes just from using SAC. Another example is Humanoid-sparse, where SAC-NF achieves ~550 (from Table 1), but is not shown in the figure, making ARAC (score ~600) look awesome. The authors also incorrectly mention in text that “ARAC is the only method that can achieve non-zero performance” on this.

     b.	Table 1 reports “maximum” average return. This is very misleading and non-standard in RL. Take Humanoid-sparse for instance. The number reported for ARAC is 816 which is the peak performance during training. As we see in Figure 4, ARAC is unstable and the perf. drops to ~600 at end of training. The peak performance during training is an irrelevant metric.

     c.	Humanoid-rllab range on y-axis looks incorrect


Minor points:

For creating a diverse archive, I’m not sure if k-means on the episodic-returns is the most effective mechanism. As explored in Conti et al., and also mentioned in the introduction of this paper, behavioral-diversity is a more useful characterization, and episodic-returns may not be aligned to that (especially in sparse reward environments).

Some of the missing related work (on population-based diversity): DIAYN, Learning Self-Imitating Diverse Policies, Divide and Conquer RL.

review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: The paper proposes an ensemble method for reinforcement learning in which the policy updates are modulated with a loss which encourages diversity among all experienced policies. It is a combination of SAC, normalizing flow policies, and an approach to diversity considered by Hong et al. (2018). The work seems rather incremental and the experiments have some methodological flaws. Specifically the main results (Fig. 4) are based on a comparison between 4 different codebases which makes it impossible to make meaningful conclusions as pointed out e.g. by [1]. The authors mention that their work is built on the work of Hong et al. (2018) yet the comparisons do not seem to include it as a baseline. I'm also concerned about how exactly are environment steps counted: in Algorithm 1 on line 27, it seems that the fitness which is used for training is evaluated by interacting with the environment yet these interactions are not counted towards total_step.

Further comments: 
1. I don't agree with the first sentence in the abstract: there's nothing special about "continuous control" when it comes to deceptive rewards. Perhaps you should be more specific and refer to robotics-inspired environments or to "commonly used Mujoco environments".
2. I don't know what "non-convex continuous action spaces" refers to. All the environments studied in the paper have action space [-1, 1]^n which is a convex set.
3. I'm also not convinced that the existing environments have "deceptive" rewards--they were likely tuned so that learning on them is feasible.
4. The discussion regarding exploration completely ignores well studied methods based on e.g. upper confidence bounds or Thompson sampling. Unlike diversity based approaches, these methods are theoretically motivated.
5. In Section 2 you start with finite horizon MDPs but then present infinite horizon discounted set up.
6. I think the point about having only a single critic would deserve more discussion: what policy is this Q-value of? Would having an independent critic for each agent help or make things worse?
 
[1] Henderson et al. Deep Reinforcement Learning that Matters. (2017)

-------------------------------------------------------------------------------------------------------------------------------------------------------
Thanks for your comments. I'm still concerned about comparing to different codebases hence I'm keeping my score as is. Also it is still unclear to me whether you count the evaluation rollouts towards the total number of steps--fitness and hence learning is based on these rollouts so they should be included.
review_title: Official Blind Review #3

final_score: rating: 8: Accept
review_content: [Note to authors: I made profit of a small edit to add a new important comment, which starts with a star below]

This paper builds on SAC-NF, an extension of SAC with Normalizing Flows which is shown to improve exploration at a low cost (Mazoure et al., 2019). In this paper, the authors build a population of SAC-NF agents with different parameters for the NF, and use an attraction-repulsion approach to ensure diversity and performance of the population. The resulting combination is shown to be competitive to or better than state-of-the-art algorithms.

This is a nice paper with nice ideas, and the resulting ARAC algorithm shows state-of-the-art performance on difficult continuous control tasks. Thus I'm in favor of accepting it. However, it should be improved before final acceptance.

Here are a few random remarks:

About related work, "balancing the objective and diversity" is also the central concern of Quality-Diversity (QD) algorithms (see e.g. Cully&Demiris for a survey).
* Actually, in QD as well as Novelty Search (NS), the algorithms look for diversity in a so called "behavior space", which is also called "outcome space in Goal Exploration Processes (GEPs), see e.g. "O Sigaud, F Stulp (2108) Policy Search in Continuous Action Domains: an Overview, Neural Networks", Section 4 for a unifying view. According to eq. (8) and (9), ARAC is looking for diversity in "fitness space", which in my opinion is weaker. I would be glad to see a comment on that.

I had to have a look at (Rezende&Mohamed, 2015) and to read (Mazoure et al., 2019) to get the normalizing flow part. An effort could be made in the beginning of Section 2.3 to make the paper more self-contained.

The account of SAC corresponds to an early version of the algorithm. In the most recent one, the value function approximator has been removed (see "Soft actor-critic algorithms and applications").

I had a hard time to figure out whether the \beta of the NF had something to do with the \beta_\pi of the AR loss.

Fig. 4 is of much interest. I would be curious to see the performance of ARAC on Swimmer, as this benchmark has been shown to suffer from a deceptive gradient effect.

Reproducibility: "See github.com" => Can you be more specific ? :)

The caption of Fig 6 describes 3 things, but I can only see two curves.

Fig 7 seems to be a mere repetition of Fig 4. Late edit before submitting? ;)

Fig.8 suffers from a poor choice of colors: even magnifying a lot the pdf, I cannot tell which is TD3 and which is SAC.

Again, Fig.9 is the same as Fig. 2.

It would be nice to move Alg 1 to the main part if possible.

typos:

p3: without risk (of) loss of information
review_title: Official Blind Review #1

Meta-review: The reviewers generally expressed considerable reservation about the novelty of the proposed method. After reading the reviews in detail and looking at the paper, I'm inclined to agree that the contribution is rather incremental. Using normalizing flows for representing policies in RL has been studied in a number of prior works, including with soft actor-critic, and I think the novelty in this work is limited in relation to prior work. Therefore, I cannot recommend acceptance at this time.
Decision: Reject