Attraction-Repulsion Actor-Critic for Continuous Control Reinforcement Learning
Authors: Thang Doan
Bogdan Mazoure
Audrey Durand
Joelle Pineau
R Devon Hjelm
Venue:  (2020)
Keywords: reinforcement learning
continuous control
multi-agent
mujoco

Abstract: Continuous control tasks in reinforcement learning are important because they provide an important framework for learning in high-dimensional state spaces with deceptive rewards, where the agent can easily become trapped into suboptimal solutions.
One way to avoid local optima is to use a population of agents to ensure coverage of the policy space, yet learning a population with the ``best" coverage is still an open problem. In this work, we present a novel approach to population-based RL in continuous control that leverages properties of normalizing flows to perform attractive and repulsive operations between current members of the population and previously observed policies. Empirical results on the MuJoCo suite demonstrate a high performance gain for our algorithm compared to prior work, including Soft-Actor Critic (SAC). 

Under review as a conference paper at ICLR 2020
ACTOR-CRITIC ArTRACTION-REPULSION C FOR CON- TINUOUS CONTROL REINFORCEMENT LEARNING
Anonymous authors Paper under double-bling review
ABSTRACT
In reinforcement learning, robotic control tasks are often useful for understanding how agents perform in environments with deceptive rewards where the agent easily become trapped into suboptimal solutions. One way to avoid these local optima is use population of agents ensure coverage of the policy space form of exploration), yet learning a population with the "best" coverage is still an open problem. In this work, we present novel approach to population- -aased RL in continuous control that leverages properties of normalizing flows to perform attractive and repulsive operations between current members of the population and previously observed policies. Empirical results on the MuloCo suite demonstrate a high performance gain for our algorithm compared to prior work, including Soft-Actor Critic (SAC).
INTRODUCTION
Many important reinforcement learning (RL) tasks, such as those in robotics and self-driving cars, challenging due large action and state spaces (Lee al. 2018). In particular, environments with large continuous action spaces are prone to deceptive rewards, i.e. fall into local optima in learning (Conti et al. 2018). Applying traditional policy optimization algorithms to these domains often leads to locally optimal, yet globally sub- ootimattm policies. The agent should then explore the reward landscape more thoroughly in order to avoid falling into these local optima.
Not all RL domains that require exploration are suitable for understanding how to train agents that are robust to deceptive rewards. For example, Montezuma's Revenge, a game in the Atari Learning Environment (Bellemare et al., (013). has sparse rewards; algorithms that perform the best on this task encourage exploration by providing denser intrinsic reward to the agent to encourage exploration (Tang et al. (017)) On the other hand, many robotic control problems, such as those found in MuJoCo (Todorov et al. 2012). provide the agent with dense reward signal, yet their high-dimensional action spaces induce multimodal, often deceptive, reward landscape. For example, in the biped environments, coordinating both arms and legs is crucial for performing well on even simple tasks such forward motion. However, simply learning maximize the reward can be detrimental across training: agents will tend to run and fall further away from the start point rather than discovering stable and efficient walking motion. In this setting, exploration serves provide more reliable learning signal for the agent by covering more different types of actions during learning.
One way to maximize action space coverage is the maximum entropy RL framework (Ziebart 2010), which prevents variance collapse by adding a policy entropy auxiliary objective. One such prominent algorithm, Soft Actor-Critic (Harnoja al.) (2018)), has been shown excel in large continuous action spaces. To further improve on exploration properties of SAC, one can maintain a population of agents that cover non-identical sections of the policy space. To prevent premature convergence, a diversity-presse preserving mechanism is typically put in place: balancing the objective and the diversity term becomes key to converging to global optimum (Hong et al. 2018). This paper studies a particular family of population-based exploration methods, which conduct coordinated local search in the policy space. Prior work on population- based strategies improves performance on robotic control domains through stochastic perturbation 2019) or a set of actor's parameters (Conti et al. 2018 Khadka & Tumer 2018; Liu et al. (017). We hypothesize that exploring directly in the policy space will be more effective than perturbing
single actor's parameter (Pourchot & Sigaud
Liu

---

Under review as a conference paper at ICLR 2020
the parameters of the policy, the latter does not guarantee diversity (i.e., different neural network parameterizations can approximately represent the same function).
Given population of RL agents, enforce local exploration using Attraction Repulsion (AR) mechanism. The later consists in adding an auxiliary loss to encourage pairwise attraction or repulsion between members of population, measured by divergence term. We make use of the Kullback- Leibler (KL) divergence because of its desirable statistical properties and its easiness of computation. However, naively maximizing the KL term between two Gaussian policies can be detrimental (e.g. drives both means apart). Because of this, we parametrize the policy with a general family of distributions called Normalizing Flows LNFs.RRezena & Mohamed] 2015); this modification allows to improve upon AR+Gaussian (see Appendix Figure(o. NFs are shown to improve the expressivity of the policies using invertible mappings while maintaining entropy guarantees (Mazoure et al.] 2019) Tang & Agrawal 2018). Nonlinear density estimators have also been previously used for deep RL. problems contexts of distributional RL. (Doan et al. 2018) and reward shaping (Tang et al. 2017).
AttractionReepllsion
The AR objective blends particularly well with SAC, since computing the KL requires stochastic policies with tractable densities for each agent.
2 PRELIMINARIES
We first formalize the RL setting in a Markov decision process (MDP). A discrete-time, finite-horizon, MDP [Belman 1957 Puterman 2014) is described by a state space S, an action space A, a transition function P : S x A x S RT. and reward function S x A + E On each round t, an agent interacting with this MDP observes the current state & E S, selects an action de E A. and observes a
reward r(st, a.) E IR upon transitioning to new state St+1 ~ (st,, ae). Let y [0. 1] be discount factor. The goal of an agent evolving in a discounted MDP is to learn a policy S x A + [0, I] such taking action ar ~ (-1(s) would maximize the expected sum of discounted returns,
(*( = E, Σ (z)
In the following, we use Prr to denote the trajectory distribution induced by following policy л. If or A are vector spaces, action and space vectors are respectively denoted by a and s.
2.1 DISCOVERING NEW SOLUTIONS THRROUGH POPULATION-BASED ATTRACTION- REPUSIINN
Consider evolving a population of M agents, also called individuals, { LM m= each agent corre- sponding to a policy with its own parameters. order to discover new solutions, we aim to generate agents that can mimic some target policy while following path different from those of other policies.
Let G denote an archive of policies encountered in previous generations of the population. natural way of enforcing to be different from similar to the policies contained in G is by augmenting the loss of the agent with an Attraction-Repulae Repulsion (AR) term:
LAR = EE [3 (Ta[[T|[T] x'~g
(1)
where T' is an archived policy and B,, is a coefficient weighting the relative importance of the Kullback-Leible (KL) divergence between and n', which we will choose to be function of the average reward (see Sec. [3. beloow. Intuitively, Fq. [ddds to the agent objective a weighted average distance between the current and the archived policies. For Brr 0, the agent tends move away from the archived policy's behavior (i.e. repulsion, see Figure a). On the other hand, Brr < encourages the agent क to imitate x' (i.e. attraction).
Requirements for AR In order for agents within a population to be trained using the proposed AR-based loss (Eq. []] we have the following requirements:
1. Their policies should be stochastic, so that the KL-divergence between two policies is well-defined.
AA and S be either discrete or continuous.
2

---

Under review as a conference paper at ICLR 2020
2.
Their policies should have tractable distributions, that the KL-divergence can be computed easily, either with closed-form solution or Monte Carlo estimation.
Several RI algorithms enjoy such properties (Haarnoja al., 2018 Schulman et al. 2015 2017). In particular, the soft actor-critic (SAC, Haarnoja et 2018))) is straighttorward choice, it currently outperforms other candidates and is off-policy, thus maintains single critic shared among all agents (instead of one critic per agent), which reduces computation costs.
2.2 SOFT ACTOR-CRITIC
SAC (Haarnoja et al., 2018) is off-policy learning algorithm which finds the information projection of the Boltzmann Q- Q-fuctioon onto the set diagonal Gaussian policies I1:
(x((ss)) exp (T
arg min DKL r'lll
Z Rou (St)
where a E (0, 1) controls the temperature, i.e. the peakedness of the distribution. The policy æ, critic Q, and value function V are optimized according to the following loss functions:
Sn,SAC = EB Ear (a log T (arls:) Q(s a
(2)
LQ
E
([((s,)) (r 7V,
(3)
(s,a,
Lv = ES
2 (s:)
Ea, -m (( a.) a log ( [s.]]]
(4)
where B is the replay buffer. The policy used in SAC as introduced in Haarnoja et al. (2018) is Gaussian, which both stochastic and tractable, thus compatible with our AR loss function in Eq.
Together with the AR loss in Eq. [ the final policy loss becomes:
Lx Lx, + CAR
However, Gaussian policies are arguably of limited expressibility; we can improve on the family of policy distributions without sacrificing qualities necessary for AR or SAC by using Normalizing Flows (NFs, Rezende & Mohamed, 2015).
2.3 NORMALIZING FLOWS
NFs (Rezende & Mohamed 2015) were introduced as means of transforming simple distributions into more complex distributions using learnable and invertible functions. Given random variable no with density qu, they define set of differentiable and invertible functions, [fi] which generate
sequence of d-dimensional random variables, {z, 1
Because SAC uses explicit, yet simple parametric policies, NFs can be used transform the SAC policy into a richer one (e.g.. multimodal) without risk loss of information. For example, Mazoure al. 2019) enhanced SAC using a family of radial contractions around point 2g IR.".
3
f(x) a
zo)
zoll2
for a R and IR. This results rich set of policies comprised an initial noise sample ag. state-noise embedding ha st), and flow (for)N of arbitrary length N, parameterized by [o.). Sampling from the policy (a)e(a))st) can be described by the following set of equations:
ag NN(0 I);
ha St);
a, SÆx Sox-1 .... f0, (z),
where hn = agrI+ + p(s;) depends on the state and the noise variance o > 0. Different SAC policies thus be crafted by parameterizing their NFs layers.
3

---

Under review as a conference paper at ICLR 2020
Figure I: a) Augmenting the loss function with AR constraints allows an agent to reach a target policy by following different paths. Attractive and Repulsive policies represent any other agent's s policy. b) General flow of the proposed ARAC strategy.
ARAC: ATTRACTION-REPULSII ACTOR-CRITIC
We now detail the general procedure for training population of agents using the proposed diversity- seeking AR mechanism. More specifically, we consider here SAC agents enhanced with NFs (Ma zoure et al., 2019). Figure displays the general flow of the procedure. Algorithm [ (Appendixx provides the pseudo-code of the proposed ARAC strategy, where sub-procedures for rollout and archive update can be found in the Appendix.
Overview ARAC works by evolving population of M SAC agents { "18.8/==1 with radial NFs
policies (Eq. 7 and shared critic Qu. and by maintaining archive of policies encountered in
previous generations of the population. After performing I steps per agent on the environment (Alg. [
12). individuals are evaluated by performing R rollouts on the environment (Alg. I 16 28).
This allows to identify the top-K best agents (Alg. [ also called elites, which will be used to update the critic as they provide the most meaningful feedback (Alg. 1 17.. The archive finally updated in diversity-secking fashion using the current population (Alg. 1[30
The core component of the proposed approach lies within the update of the agents (Alg. D 25).
During this phase, elite individuals are updated using AR operations w.r.t. policies sampled from the archive (Eq. 5). whereas non-elites are updated regularly (Eq. (
3.1 ENHANCING DIVERSITY IN THE ARCHIVE
Throughout the training process, we maintain an archive G of maximum capacity G. which contains some previously encountered policies. The process goes follow: until reaching full capacity, the archive saves a copy of the parameters of every individual in the population after the evaluation step. However, by naively adding all individuals as if the archive were just heap, the archive could end up filled with policies leading to similar rewards, which would result in a loss of diversity (Mauldin 1984). We mitigate this issue by keeping track of two fitness clusters (low and high) using the partition formed by running a k-means algorithm on the fitness value. Hence, when | = G is reached and a new individual is added the archive, randomly replaces an archived policy from its respective cluster. This approach, also known as niching, has proved itself effective at maintaining high diversity levels (Gupta & Ghanr 2012
Mahfoud 1995).
3.2 DISCOVERING NEW POLICIES THROUGH IETRACTIOO-REPULL.SS
The crux of this work lies in the explicit search for diversity in the policy space achieved using the AR mechanism. Since the KL between two base policies (i.e. input of the first flow layer) can be trivially maximized by driving their means apart, we apply intractionrrepulsion only on the flow layers, while holding the mean of the base policy constant. This ensures that the KL term doesn't 't depend on the difference in means and hence controls the magnitude of the AR mechanism. Every time the AR operator is applied (Alg. 1 21p. policies sampled from the archive and are used for estimating the AR loss (Fq. D. As in Hong et al. 2018)), 1), we consider two possible strategies
"These steps can be performed in parallel.
4

---

Under review as a conference paper at ICLR 2020
to dictate the value of Snr' coefficients for policies x' G:
A = - f( - fmin - 1
(proactive)
(8)
Smax Smin
f(TT) smin
A = 1 Jmax J'in
(reactive)
(9)
where f(T represents the fitness function of policy π (average reward in our case), and fmin and fmax are estimated based the sampled archived policies. The proactive strategy aims to mimic high reward archived policies, while the reactive strategy is more cautious, only repulsing away the current policy from low fitness archived policies. Using this approach, the current agent policy will be attracted to some sampled policies (3zr < 0) and will be repulsed from others (3,r 2 0) in more or less aggressive way, depending the strategy.
Unlike Hong et al. (2018) who applied proactive and reactive strategies on policies up to 5 timesteps back, we maintain an archive consisting of two clusters seen so far: policies with low and high fitness, respectively. Having this cluster allows to attract/repulse from a set of diverse agents, replacing high-reward policies by policies with similar performance. Indeed, without this process, elements of the archive would collapse on the most frequent policy, from which all agents would attract/repulse. To avoid performing AR against a single "average policy". we separate low-reward and high-reward agents via clustering.
4 RELATED WORK
The challenges of exploration are well studied in the RL literature. Previously proposed approaches for overcoming hard exploration domains tend to either increase the capacity the state-action value function (Gal & Ghahramani 2016 Henderson et al.) (017) the policy expressivity (Mazoure
ctt al.) 2019 Tang& & Agrawal 2018 Touati et al. 2018). This work rather tackles exploration from a diverse multi-agent perspective. Unlike prior nopulation-based approaches for exploration (Conti Pourchot Sigaud 2019), which seek diversity through the parameters space, we directly promote diversity in the policy space.
ct al. 2018 Khadka & Tumer 2018;
The current work was inspired by Hong et al. (2018), who relied on the KL divergence to at- solutions. However, in their work, the archive is time-based (they restrict themselves to the most recent policies), while our archive is built following diversity-seeking strategy (i.e., niching and policies come from multiple agents). Notably, ARAC is different of previously discussed works in that it explores the action space in multiple regions simultaneously, a property enforced through the AR mechanism.
tract/repulse from set of previous policies to discover
The proposed approach bears some resemblance with Liu et al. (201)), who took advantage of a multi-agent framework in order to perform repulsion operations among agents using of similarity kernels between parameters of the agents. The AR mechanism gives rise to exploration through structured policy rather than randomized policy. This strategy has also been employed in multi-task learning (Gupta et al. 2018), where experience previous tasks was used to explore on new tasks.
5
EXPERIMENTS
5.1 DIDACTIC EXAMPLE
Consider 2-dimensional multi-armed bandit problem where the actions lie in the real square [-6, 67. We illustrate the example of using proactive strategy where SAC agent with radial flows policy imitates a desirable (expert) policy while simultaneously repelling from a less desirable policy. The task consists in matching the expert's policy (blue density) while avoiding taking actions from repulsive policy (red). We illustrate the properties of radial flows in Figure by increasing the number of flows (where 0 flow corresponds
Gaussian distribution).
We observe that increasing the number of flows (bottom to top) leads to more complex policy's shapes and multimodality unlike the Gaussian policy which has its variance shrinked (the KL divergence
We overload notation for both the normalizing flow and the fitness depending the context
5

---

Under review as a conference paper at ICLR 2020
Figure 2: Agent trained to imitate target while avoiding a repulsive policy using proactive strategy.
Increasing the number flows leads to more complex policy's shape.
is proportional to the ratio of the two variances, hence maximizing it can lead to reduction in the variance which can detrimental for exploration purpose). Details are provided in Appendix.
Hopper
Thenstope
Temeslags 13e00
Theestops (led)
Temestepa 1264)
Figure 3: Average return and one standard deviation on 5 random seeds across 7 MuJoCo tasks for ARAC against single SAC agents (with and without NFs). Curves are smoothed using Savitzky-Golay filtering with window size 7.
5.2 MuJoCo LOCOMOTION BENCHMARKS
We now compare ARAC against the CEM-TD3
Pourrhoot & Sigaud 2019). ERL Khadka & 2019) multi-agent baselines on seven continuous control
Tumer 2018) and CERL (Kaakka al.
tasks from the Muloco suite (Duan et al. Humano aSt anmaanidSttaduu-r22 Hopper-v2, Walker2d-v2 and Humanoid (rllab). We also designed sparse reward environment Sparseffummnoiddee
2016): Ant-v2, Hal Ehheetah-v,, Humanoid-v2,
All algorithms are run over 1M time steps on each environment, except Humanoid (rllab) which gets 2M time steps and sparseHiuaanoddy on 0.6M time steps. We also include comparison against single-agent baselines.
ARAC performs Rt = 10 rollouts for evaluation steps every 10, 000 interaction steps with the environment. We consider a small population of = 5 individuals with K = as elites. Every SAC agent has one feedforward hidden layer of 256 units acting as state embedding, followed by radial flow of length {3, 4). A temperature of a = 0.05 or 0.2 is used across all the environments (See appendix for more details). AR operations carried out by sampling uniformly archived

---

Under review as a conference paper at ICLR 2020
policies from G. Parameters details are provided in the Appendix (Table 4
All networks trained
with Adam optimizer (Kingma & Ba 2015) using a learning rate of 3E
Baselines CEM-TDE?
ERL
CERI use the code contained in their respective repositories.
ARAC
CEM-TD3
CERL
ERL
SAC NF
SAC
TD3
Ant
6044
4239
1639
1442
4912
4370
4372
HC
10 264
10659
5703
6746
900
9543
Hopper
3587
3655
2970
1149
3538
2794
3564
Hu
5965
212
4756
551
5506
5504
71
Standup
175 000
29 000
117 000
12 900
116 000
149
54 000
Hu (rllab)
230
1334
3340
5531
1963
286
Walker2d
4704
1710
4386
1107
5196
3783
4682
Hu (Sparse) 816
0
1.32
8.65
5417
88
Table I: Maximum average return after 1M (2M for Humanoid (rllab) and 600k for SparseHumanoid- time steps random seeds. Bold: best methods when the gap less than 100 units. See appendix for average return with standard deviation. Environment short names: HC: Hal Fhhh Hu: Humanoid-ve Standup: SommaniddStandapn
Figure 4 displays the performance of all algorithms on three environments over time steps (see Appendix Figure 7 for all environments). Results are averaged over random seeds. Table reports the best observed reward for each method.
HraftCheutab-42
Walkarrd--
Thenslege Gelt
Thams Pays Dal)
Timestags Caen
Tirescpps (an)
Figure 4: Average return and one standard deviation on 5 random seeds across MuloCo tasks. Curves smoothed using Savitzy-Golay filtering with window size of 7.
Small state space environments Hal FChfetah--v2, Hopper-v2, and Walker2d-v2 are low-dimensional state space environments (d < 17). Except for HalfCheetah-ve the proposed approach shows comparable results with its concurrent. Those results match the findings of (Plappert et al. 2018) that some environments with well-structured dynamics require little exploration. Full learning curves can be found in the Appendix.
Deceptive reward and Large state space environments Humanoid-v2, HumanoidStandupp-o... and Humanoid (rllab) belong to bipedal environments with high-dimensional state space (d = 376 and d 147), and known to trap algorithms into suboptimal solutions. In addition to the legs, the agent also needs to control the arms, which may influence the walking way and hence induce deceptive rewards (Conti al. (2018). Figure shows the learning curves on MuloCo tasks. We observe that ARAC beats both baselines in performance as well as in convergence rate.
Ant v2 is another eigh--imensional state space environment (d 2 100). In an unstable setup, naive algorithm implementing an unbalanced fast walk could still generate high reward, the reward
hutee://gtthubl.co//ooooccc/////
httpsss//iitbbbb.cm////aaaaa/ Bactleee___pp_______ paper_nips18
http:://iithbb.com///teea//////

---

Under review as a conference paper at ICLR 2020
taking into account the distance from start, instead of learning to stand, stabilize, and walk (as expected).
Sparse reward environment To test ARAC in sparse reward environment, created SearseHumanoid-ee The dynamic is the same as Humanoid-v2 but rewards of +1 granted only given is the center of mass of the agent is above threshold (set to 0.6 unit in our case). The challenge not only lies in the sparse reward property but also on the complex body dynamic that can make the agent falling down and terminating the episode. As shown in Figure 4 ARAC is the only method that can achieve non zero performance. A comparison against single agent methods in the Appendix also shows better performance for ARAC.
Sample efficiency compared with single agent methods Figure 3 (in Appendix) also shows that the sample efficiency of the population-bast baased ARAC compares to a single SAC agent (with and without NEs) and other baselines methods (SAC, TD3). On Humanoid v and Ant v ARAC converges faster, reaching the 6k (4k, respectively) milestone performance after only 1M steps, while single SAC agent requires 4M (3M, respectively) steps according to (Haarnoja et al., 2018). In general, ARAC achieves competitive results (no flat curves) and makes the most difference (faster convergence and better performance) in the biped environments.
Attraction-repulsion ablation study To illustrate the impact of repulsive forces, we introduce a hyperparameter in the overall loss (Fq.5)
Co.0.1 La. + ALo.AR
(10)
We ran an ablation analysis on Humanoid-v2 by varying that coefficient. For two random states, we sampled 500 actions from all agents and mapped these actions onto two-dimensional space (via t-SNE). Appendix Figure 5 Shows that without repulsion (A = 0), actions from all agents are
entangled, while repulsion (A forces agents to behave differently and hence explore different regions of the action space.
The second ablation study is dedicated to highlight the differences between Gaussian policy (similar to Hong et al. (0188) and an NF policy under AR operators. As one can observe in Figure using a Gaussian policy deteriorates the solution the repulsive KL term drives apart the means of agents and blows up/ shrinks the variance of the Gaussian policy. On the other hand, applying the AR term on the NF layers maximizes the KL conditioned on the mean and variance of both base policies, resulting in solution which allows sufficient exploration. More details are provided in the Appendix.
Finally, through a toy example subject to AR, we characterize the policy's shape when increasing the number of the radial flow policy in Figure (experimental setup in Appendix). Unlike the diagonal Gaussian policy (SAC) that has symmetry constraints, increasing the number of flows allows the radial policy to adopt more complex shapes (from bottom to top).
6 CONCLUSION
In this paper, we addressed the issue of RL. domains with deceptive rewards by introducing a population-based search model for optimal policies using attraction-repulsion operators. Our method relies on powerful density estimators (normalizing flows), to let policies exploit the reward landscape under AR constraints. Our ablation studies showed that (1) the strength of AR and (2) the number of flows are the two factors which predominantly affect the shape of the policy. Selecting the correct AR coefficient is therefore important obtain good performance, while the same time preventing premature convergence.
Empirical results on the MuJoCo suite demonstrate high performance of the proposed method in most settings, including with sparse rewards. Moreover, in biped environments that are known to trap algorithms into suboptimal solutions, ARAC enjoys higher sample efficiency and better performance compared to its competitors which confirms our intuitions using AR with normalizing flows. As future steps, borrowing from multi-objective optimization literature methods could allow one to combine other diversity metries with the performance objective, in turn improve the coverage of the solution space among the individuals by working with the corresponding Pareto front (Hom et al. 1994).
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environ- ment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47: 253-279, 2013.
Richard Bellman. A markovian decision process. Journal of Mathematics and Mechanics, pp. 679-684, 1957.
Edoardo Conti, Vashisht Madhavan, Felipe Petroski Such, Joel Lehman, Kenneth Stanley, and Jeff Clune. Improving exploration in evolution strategies for deep reinforcement learning via a population of novelty-seeking agents. In Advances in Neural Information Processing Systems (NeurIPS), pp. 5027-5038, 2018.
Thang Doan, Bogdan Mazoure, and Clare Lyle. Gan q-learning. arXiv preprint arXiv:1805.04874, 2018.
Yan Duan, Xi Chen, Rein Houthooft, John Schulman, and Pieter Abbeel. Benchmarking deep reinforcement learning for continuous control. International Conference on Machine Learning (ICML), pp. 1329-1338, 2016.
Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In International conference on machine learning (ICML), pp. 1050- 1059, 2016.
Abhishek Gupta, Russell Mendonca, YuXuan Liu, Pieter Abbeel, and Sergey Levine. Meta- reinforcement learning of structured exploration strategies. In Advances in Neural Information Processing Systems (NeurIPS), pp. 5302-5311, 2018.
Deepti Gupta and Shabina Ghafir. An overview of methods maintaining diversity in genetic algorithms. International journal of emerging technology and advanced engineering, 2(5):56-66 60, 2012.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft Actor Cctor-Critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. International Conference Machine Learning (ICML), pp. 1856-1865, 2018.
Peter Henderson, Thang Doan, Riashat Islam, and David Meger. Bayesian policy gradients via alpha divergence dropout inference. NIPS Bayesian Deep Learning Workshop. 2017.
Zhang-Wer Wei Hong, Tzu- Yun Shann, Shih-Ya Shih-Yang Su, Yi-Hsiang Chang, Tsu-Jui Fu, and Chun Yi Lee. Diversity-driven exploration strategy for deep reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), pp. 10489-10500, 2018.
Jeffrey Horn, Nicholas Nafpliotis, and David H. Goldberg. A niched pareto genetic algorithm for multiobjective optimization. In Proceedings of the 1st IEEE Conference Evolutionary Computation, IEEE World Congress on Computational Intelligence, pp. 82-87, 1994.
Shauharda Khadka and Kagan Tumer. Evolution Evolution-guided policy gradient in reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), pp. 1 1188-12000 1000, 2018.
Shauharda Khadka, Somdeb Majumdar, Tarek Nassar, Zach Dwiel, Evren Tumer, Santiago Miret, Yinyin Liu, and Kagan Tumer. Collaborative evolutionary reinforcement learning. CoRR, abs/
2019, URL http::/arxiv..tt www.:orggggbbbbbboeee 0tt.//1110000066656 00976
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. In International Conference on Learning Representations (ICLR), 2015.
Kyowoon Lee, Sol-A Kim, Jaesik Choi, and Seong- Whan Lee. Deep reinforcement learning in con- tinuous action spaces: case study in the game of simulated curling. In International Conference on Machine Learning. pp. 2943-2952, 2018
Yang Liu, Prajit Ramachandran, Qiang Liu, and Jian Peng. Stein variational policy gradient. In Conference Uncertainty Artificla Intelligence (UAI), 2017.

---

Under review as a conference paper at ICLR 2020
Samir W Mahfoud. Niching methods for genetic algorithms. PhD thesis, University of Illinois at Urbana- ChaamaChamp Champaign, USA, 1995.
Michael L Mauldin. Maintaining diversity in genetic search. In AAAI Conference on Artificial Intelligence (AAAI), pp. 247-250, 1984.
Bogdan Mazoure, Thang Doan, Audrey Durand, Devon Hjelm, and Joelle Pineau. Leveraging exploration in off-policy algorithms via normalizing flows. Proceedings of the 3rd Conference on Robot Learning (CoRL 20/9), 2019.
Joelle Pineau. The machine learning reproducibility checklist. 2018.
Matthias Plappert, Rein Houthooft, Prafulla Dhariwal, Szymon Sidor, Richard Y Chen, Xi Chen, Tamim Asfour, Pieter Abbeel, and Marcin Andrychowicz. Parameter space noise for exploration. arXiv preprint
2017.
Matthias Plappert, Rein Houthooft, Prafulla Dhariwal, Szymon Sidor, Richard Y. Chen, Xi Chen, Tamim Asfour, Pieter Abbeel, and Marcin Andrychowicz. Parameter space noise for exploration. In International Conference on Learning Representations (ICI.), 2018.
Alois Pourchot and Olivier Sigaud. CEM-RL: Combining evolutionary and gradient- based methods for policy search. In International Conference on Learning Representations (ICLR), 2019.
Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014.
Danilo Jimenez Rezende and Shakir Mohamed. Variational inference with normalizing flows. In International Conference Machine Learning (ICML), pp. 1530-1538, 2015.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International Conference on Machine Learning (ICML), pp. 1889-1897, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint: 1707.06347, 2017.
Haoran Tang, Rein Houthooft, Davis Foote, Adam Stooke, OpenAl Xi Chen, Yan Duan, John Schulman, Filip DeTurck, and Pieter Abbeel. # Exploration: A study of count-based exploration for deep reinforcement learning. In Advances in neural information processing systems (NeurIPS), pp. 2753-2762, 2017.
Yunhao Tang and Shipra Agrawal. Boosting trust region policy optimization by normalizing flows policy. arXiv preprint: 10326, 2018.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 5026-5033. IEEE, 2012.
Ahmed Touati, Harsh Satija, Joshua Romoff, Joelle Pineau, and Pascal Vincent. Randomized value functions via multiplicative normalizing tlows. arXiv preprint: 1806.02315, 2018.
Brian D Ziebart. Modeling purposeful adaptive behavior with the principle of maximum causal entropy. PhD thesis, figshare, 2010.
10

---

Under review as a conference paper at ICLR 2020
APPENDIX
REPRODUCIBILITY CHECKLIST
We follow the reproducibility checklist (Pineau 2018) and point to relevant sections explaining them here.
For all algorithms presented, check you include:
A clear description of the algorithm, see main paper and included codebase. The proposed approach is completely deseribed by Alg. [ (main paper). 2 (Appendix), and (Appendix). The proposed population- based method uses attraction-eepllsio operators in order to enforce a better policy space coverage by different agents.
3
An analysis of the complexity (time, space, sample size) of the algorithm. See Appendix Figure 7 and 3
discussed in Experimentally, we demonstrate improvement sample complexity as main paper. In term of computation time, the proposed method scales linearly with the population size if agents are evaluated sequentially (as presented in Alg, [ for clarity). However, this as mentioned in the paper, can be parallelized. All our results are obtained using M small network architectures with 1 x 256-units hidden layer followed by f layers of | + units each (f being the number of radial flows and |A| being the action space dimension).
A link to downloadable source code, including all dependencies. The code is included with the Appendix as zip file; all dependencies can be installed using Python's 's package manager. Upon publication, the code would be available on Github.
For all figures and tables that present empirical results, check if you include:
A complete description of the data collection process, including sample size. We use standard benchmarks provided in OpenAl Gym (Brockman et al., 2016).
A link to downloadable version of the dataset or simulation environment. See: https://gtthu..com
An explanation of how samples were allocated for training / validation / testing. We do not use a training-validation-- standard deviation) of the policy at evaluation time, openai/gym for OpenAl Gym benchmarks and https:/wwww.rbbot.uuiiiio
split, but instead report the mean performance (and
html for MuJoCo suite. obtained with
random seeds.
An explanation of any data that were excluded. We did not compare easy environ- ments (e.g. Reacher-v2) because all existing methods perform well on them. In that case, the improvement of our method upon baselines is incremental and not worth mentioning.
The exact number of evaluation runs. 5 seeds for MuJoCo experiments, IM, 2M or 3M environment steps depending on the domain.
A description of how experiments were run. See Section 5 in the main paper and didactic example details in Appendix.
A clear definition of the specific measure or statistics used to report results. Undis- counted returns across the whole episode are reported, and turn averaged across seeds.
Clearly defined error bars. Confidence intervals and table values always mean 1 standard deviation over seeds.
A description results with central tendency (e.g, mean) and variation (e.g. stddev).
All results use the mean and standard deviation.
A description of the computing infrastructure used. All runs used 1 CPU for all experi- ments (toy and MuJoCo) with 8Gb of memory.
11

---

Under review as a conference paper at ICLR 2020
IMPACT OF REPULSIVE FORCE
A= (Independent)
A=1 (ARAC)
A=
State 1
State 2
Figure 5: Mapping in two-dimension space (t-SNE) of agents' actions for two arbitrary states. Each color represents a different agent.
To illustrate the impact of the repulsive force coefficient X, that coefficient (recall that the overall loss function is La La,SAC + ACAR where À = 1 in our experiment).
ablation analysis by varying
For two random states, we sampled 500 actions from all agents and mapped theses actions in common 2-dimensional space (t-SNE).
As shown in the Figure above, policies trained without AR (A 0) result in entangled actions, while increasing the repulsive coefficient forces agents to have different actions and hence explore different regions of the policy space. Note that due to the specific nature of t-SNE, the policies are shown as Gaussians lower-dimensional embedding, while is not necessarily the case in the true space.
12

---

Under review as a conference paper at ICLR 2020
STABILIZING ATTRACTION- ALTRAACTIONRRPULLIION WITH NORMALIZING FLOW
In this section, we illustrate the consequence of the AR operators with a Gaussian policy (as in Hong al. (2018) and our Normalizing flow policy for Ant-v2, Humanoid-v2 and Ha Hal IfCheetth v2. As shown in the figure below, AR with Gaussian policies yield worse re- sults. One reason is that the KL term drives apart the mean and variance of the Gaussian policy which deteriorates the main objective of maximizing the reward. On the other side, our method applies the AR only on the NF layers allows enough exploration by deviating sufficiently from the main objective function.
Ant-v2
Humanoldiv22
HolfCheetah--2 V2
Awerage
Timestopaa (Tet))
Tenesloga
Timestogo (1e8)
Figure 6: Comparison of ARAC agents using (1) AR with radial flows, (2) AR with only the base (Gaussian) policy and (3) no AR with radial flows.
COMPARING ARAC AGAINST BASELINES ON MUJOCO TASKS
Figure shows the performance of ARAC and baselines (CEM-TD3, CERL and ERL) over time steps. Learning curves are averaged over random seeds and displayed with one standard deviation. Evaluation is done every 10, 000 environment steps using 10 rollouts per agent. Overall, ARAC has reasonable performance on all tasks (no flat curves) and demonstrates high performance. especially in humanoid tasks.
HaifCheetahh-v2 +2
Humanold-a x2
Humanoid (rilab)
Humanold (sparse)
Ant
Timestaga
Timestaps (1ad)
Theastope
Figure Average return and one standard deviation on random seeds across MuJoCo tasks for ARAC against baselines. Curves are smoothed using Savitzky-oolay filtering with window size of 7.
13

---

Under review as a conference paper at ICLR 2020
BENEFITS OF POPULATION BASED STRATEGIES: ARAC AGAINST SINGLE AGENTS
In this section, we highlight the benefits of the proposed population-based strategy by comparing with single agents. Figure 3 shows the performance of ARAC against single SAC agent (with and without normalizing flows). Learning curves are averaged over 5 random seeds and displayed with one standard deviation. Evaluation done every 10, 000 environment steps using 10 rollouts per agent. We observe a high beneficial impact on the convergence rate as well as on the perfor- mance. ARAC outperforms single agents in almost all tasks (except for Fhh and Walker-v2) with large improvement. Note the high sample efficiency on humanoid environments (Humanoid-ie v2 and Humanoid (rllab)) ), where shows faster convergence addition to better performance. Indeed, on Humanoid (rllab) single SAC agent reaches the 4k milestone after 4M steps (Haamoja et al. 2018) while ARAC achieves this performance less than 2M steps. Also, in Searseeumanoidee i-- due to its better coordinated exploration, ARAC could find good solution faster than SAC-NE.
14

---

Under review as a conference paper at ICLR 2020
OVERALL PERFORMANCES ON MUJOCO TASKS
ARAC
CEM-TD3
CHRI.
HRI.
SAC
SAC
TD33
Ant-v2
6,044 + 216
4, + 0,48
1, 639 + 564
1,442 + 819
4,912 = 954
4,370 = 173
4,372 = 900
HalfCheetah-
10, 204 + 271
10,659 + 1,473
5,3 + 831
6,746 295
8,429 = 818
11,896 574
9,543 = 978
Hopper-v2 v2
3,587
3,655 82
2,970
1
3, = 108
2, 7974 729
3,564
Humanoid-v2
5,965
212
4,756 454
551
5,506 147
5,504 1:04
71 54k + 24k
HumanoidStundup-v2
175k + 38k
29k + 4k
117k = 8k
1294
116h 9k:
149K 7k
Humanoid (rllab)
14,234 = 7251
1,334 = 551
3,340 3,340
57
5, 531 434
106+11. 1,963
286 + 151
Walker2d-v2
4,704 1 261
4,710 1 320
4,3860 615
1. 1,7
5,196 527
3,783 1 366
4,682 T 539
Sumanoolu
816
010
1.32 2.64
8.65 1 15.90
159
010
Table Maximum average return after 1M (2M for Humanoid (r1lab) and 600k for SparseHlmaanodd-22 time steps + one standard deviation on 5 random seeds. Bold: best methods when the gap less than 100 units.
ARAC
TRPO
PPO
Trust-PCL
Plappert et al. (2017)
Touati ct al. (2018)
Hong et al. (2018)
HalfCheetah-v2
10, 264
-15
2, 600
2, 200
5, 000
7, 700
4, 200
Walker-v2
4, 764
2, 400
4, .50
400
850
500
N/A
Hopper-v2
3,
600
280
500
N/A
Ant-v2
6,
1, 1,000
1,0000
N/A
N/A
Humanoid-v2
5, 9399
400
400
N/A
N/A
N/A
1, 250
HumanoidStandup-v2
1 163, 884
80, 000
N/A
N/A
N/A
N/A
N/A
Humanoid (rllab)
4, 117
23
200
N/A
N/A
N/A
N/A
Table 3: Performance after 1M (except for rllab which is 2M) timesteps on 5 seeds. Values taken
from their corresponding papers. N/A means the values were not available in the original paper.
15

---

Under review as a conference paper at ICLR 2020
EXPERIMENTAL PARAMETERS
Table 4 provides the hyperparameters of ARAC used to obtain results in the MuJoCo domains. The noise input for normalizing flows in SAC policies (see Sec. 2.3 sampled from N (00 o), where the variance a is function of the state (either fixed at given value or learned).
ARAC
parameters
# flows
G
P
alpha
strategy
Ant-v2
0.2
10
proactive
HaltCheetah--2
20
2
proactive
Hopper-v2
0.8
20
1
0.05
proactive
Walker2d-v2
0.6
10
3
0.05
pruactive
Humancid-v2
3
0.6
10
1
0.05
reactive
HumanoidStanduup
20
1
0.2
reactive
Humanoid (rllab)
10
1
0.05
pruactive
SparscHumanoid-12
0.6
20
proactive
Adam Optimizer parameters
0x
aw: de
3.10-1
3.10
0g
3.10
Algorithm parameters
Batch size Buffer size B
256
10
Archive sample size
Table 4: ARAC parameters.
16

---

Under review as a conference paper at ICLR 2020
IMPACT OF NUMBER OF FLOWS THE POLICY SHAPE
We used a single SAC agent with different radial flows numbers and randomly initialized weights, starting with actions centered (0,0). All flow parameters are & regularized with hyperparameter 2. The agent is trained with the classical evidence lower bound (ELBO) objective augmented with the AR loss (Eq. I where the coefficient of the repulsive policy x' given by 8 =+I* Fig. g shows how both the NF and learned variance Gaussian policies manage to recover the target policy. We see that NF takes advantage of its flexible parametrization to adjust its density and can show asymmetric properties unlike the Gaussian distribution. This indeed can have advantage in some non symmetric environment where the Gaussian policy would be trapped into a suboptimal behavior. Finally, increasing the number of flows (from bottom top) can lead more complex policy's 's shape.
10
Target policy
Repuiites pallcy
Agent's polccy
Nr policy (8 flows)
dimenttonn
Increasing number of flows
Action direnstoon
policy
Addion drsansaon 2
Action
Iberation 0
Theration 50
Heratton 250
Hametton 2008
Increasing number of iterations
Figure 8: Single state didactic illustration of attraction attraction-repulsion operators. Comparing behavior of NF policy against Gaussian policy with learned variance under repulsive constraint.
17

---

Under review as a conference paper at ICLR 2020
6.1 VARIANCE FITNESS THE ARCHIVE
Due to the high computation time for behavioral-diverse baselines such as DIYAN, we propose to the agent's fitness (i.e. undiscounted returns) candidate to repulse/attract from.
Average variance archive fitness across steps
Humanoid
HumanoidStandup
248
WMy
20003
1500
20000
1000
3000
5000
Teme
Figure ?? shows the variance of the archive across three MuloCo domains: Ant, Humanoid and HumanoidStandup. training progresses, the clustering approach allows to maintain a high variance in the archive, preventing mode collapse to single, "average" fitness.
18

---

Under review as a conference paper at ICLR 2020
6.2 PSEUDO-CODE FOR ARAC
Algorithm ARAC:
Attraction-Repuliii Actor-Critic
Input:
population size M; number of elites K; maximum archive capacity G; archive sample size n; number of evaluation rollouts ft; actor coefficient p: strategy (either proactive or reactive).
2: Initialize value function network Vú and critic network Qw
Initialize population of policy networks {T 1M
4: Initialize empty archive & and randomly assign K individuals top-K
total step f-
6: while total step I max step do
7: step f 0
8:
for agent m .... .M do
9:
(_, Steps) 5) rollout (n with noise, over episode)
Collect samples
10:
step + step +
11:
total_sted total step + end for
total_step
12:
13:
C= K
14:
for policy = in top-K do
15:
Update critic with n" for C mini-batcher (Eq. Update value function (Eq.
Update critic
17:
end for
18:
for agent 1...MMMMM .M do
19:
policy nhhh is in top- K then
20:
Sample n archived policies uniformly from G
21:
Update actor π for step Mf mini-batches (Eq. and or
Update actors
22:
else
23:
Update actor nh for step mini-batches (Eq. 2 end if
M
24:
25:
end for
26: for agent m = 1... ......M. do
(Fitnessm: ) rollout( without noise, over R episodes) end for
Evaluate actors
28:
29:
Rank population { - and identify top-K
30:
apdate_archivell ( ( 1 G)
31: end while
19

---

Under review as a conference paper at ICLR 2020
COMPLEMENTARY PSEUDO-COLD FOR ARAC
Algorithms 2 and respectively provide the pseudo-code of functions rollout and
update_archi used in Algorithm [
Algorithm 2 rollout
Input: actor и, noise status; number of episodes E; replay buffer B; Fitness +-
for episode 1, . do f Initial state So from the environment
for step O... termination do if with noise then
Sample noise
else
Set
end if
a, ~ (--|5t: 2)
Observe St+1 P(- ax) and obtain reward
Fitness < Fitness + rt
Store transition (1,411 in B
end for
end for
Fitness + Fitness/
return Average fitness per episode and number of steps performed
Algorithm 3
Input: archive G; population of size M; maximal archive capacity G. if [9] G then
Add all agents of current population to g
else
02 < 2-means fitness of individuals in G)
for agent ... .M do
Assign agent m to closest cluster E {ci,e2} based on its fitness Sample an archived agent (niform(c)
Replace archived individual by m
end for
end if
return Updated archive G
20


Reviews:
[{"review_id": "SkxWSu2K_B", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "[Note to authors: I made profit of a small edit to add a new important comment, which starts with a star below]\n\nThis paper builds on SAC-NF, an extension of SAC with Normalizing Flows which is shown to improve exploration at a low cost (Mazoure et al., 2019). In this paper, the authors build a population of SAC-NF agents with different parameters for the NF, and use an attraction-repulsion approach to ensure diversity and performance of the population. The resulting combination is shown to be competitive to or better than state-of-the-art algorithms.\n\nThis is a nice paper with nice ideas, and the resulting ARAC algorithm shows state-of-the-art performance on difficult continuous control tasks. Thus I'm in favor of accepting it. However, it should be improved before final acceptance.\n\nHere are a few random remarks:\n\nAbout related work, \"balancing the objective and diversity\" is also the central concern of Quality-Diversity (QD) algorithms (see e.g. Cully&Demiris for a survey).\n* Actually, in QD as well as Novelty Search (NS), the algorithms look for diversity in a so called \"behavior space\", which is also called \"outcome space in Goal Exploration Processes (GEPs), see e.g. \"O Sigaud, F Stulp (2108) Policy Search in Continuous Action Domains: an Overview, Neural Networks\", Section 4 for a unifying view. According to eq. (8) and (9), ARAC is looking for diversity in \"fitness space\", which in my opinion is weaker. I would be glad to see a comment on that.\n\nI had to have a look at (Rezende&Mohamed, 2015) and to read (Mazoure et al., 2019) to get the normalizing flow part. An effort could be made in the beginning of Section 2.3 to make the paper more self-contained.\n\nThe account of SAC corresponds to an early version of the algorithm. In the most recent one, the value function approximator has been removed (see \"Soft actor-critic algorithms and applications\").\n\nI had a hard time to figure out whether the \\beta of the NF had something to do with the \\beta_\\pi of the AR loss.\n\nFig. 4 is of much interest. I would be curious to see the performance of ARAC on Swimmer, as this benchmark has been shown to suffer from a deceptive gradient effect.\n\nReproducibility: \"See github.com\" => Can you be more specific ? :)\n\nThe caption of Fig 6 describes 3 things, but I can only see two curves.\n\nFig 7 seems to be a mere repetition of Fig 4. Late edit before submitting? ;)\n\nFig.8 suffers from a poor choice of colors: even magnifying a lot the pdf, I cannot tell which is TD3 and which is SAC.\n\nAgain, Fig.9 is the same as Fig. 2.\n\nIt would be nice to move Alg 1 to the main part if possible.\n\ntypos:\n\np3: without risk (of) loss of information", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "H1xXQ2i6Fr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "RL in environments with deceptive rewards can produce sub-optimal policies. To remedy this, the paper proposes a method for population-based exploration. Multiple actors, each parameterized with policies based on Normalizing Flows (radial contractions), are optimized over iterations using the off-policy SAC algorithm. To encourage diverse-exploration as well as high-performance, the SAC-policy-gradient is supplemented with gradient of an “attraction” or “repulsion” term, as defined using the KL-divergence of current policy to another policy from an online archive. When applying the KL-gradient, the authors find it crucial to only update the flow layers, and not the base Gaussian policy.\n\nI have 2 issues with this paper:\n\n1.\tLack of novelty – Most of the components of the proposed algorithm have been researched in other works. SAC with NF was contributed by Mazoure et. al., and this paper seems to use the exact same architecture.  The diversity-enforcing objective (Equation 1.) was proposed by Hong et. al., albeit not with a separate archive of policies, but using the experience replay (for off-policy) and recent policies (for on-policy). The objective is Section 3.2 is also the same as that in Hong et. al. \n\n2.\tI have major concerns about how the results have been reported:\n     a.\tThe authors have omitted SAC and SAC-NF from Figure 4, and therefore the figure gives the impression that ARAC is significantly better than the presented baselines in some of the environments. For example, take Humanoid-v2. The original SAC paper reports reaching close to 5k score in about 0.5 million timesteps. It therefore seems that most of the benefit of ARAC (over the presented baselines) comes just from using SAC. Another example is Humanoid-sparse, where SAC-NF achieves ~550 (from Table 1), but is not shown in the figure, making ARAC (score ~600) look awesome. The authors also incorrectly mention in text that “ARAC is the only method that can achieve non-zero performance” on this.\n\n     b.\tTable 1 reports “maximum” average return. This is very misleading and non-standard in RL. Take Humanoid-sparse for instance. The number reported for ARAC is 816 which is the peak performance during training. As we see in Figure 4, ARAC is unstable and the perf. drops to ~600 at end of training. The peak performance during training is an irrelevant metric.\n\n     c.\tHumanoid-rllab range on y-axis looks incorrect\n\n\nMinor points:\n\nFor creating a diverse archive, I’m not sure if k-means on the episodic-returns is the most effective mechanism. As explored in Conti et al., and also mentioned in the introduction of this paper, behavioral-diversity is a more useful characterization, and episodic-returns may not be aligned to that (especially in sparse reward environments).\n\nSome of the missing related work (on population-based diversity): DIAYN, Learning Self-Imitating Diverse Policies, Divide and Conquer RL.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BJx3HfFoYB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes an ensemble method for reinforcement learning in which the policy updates are modulated with a loss which encourages diversity among all experienced policies. It is a combination of SAC, normalizing flow policies, and an approach to diversity considered by Hong et al. (2018). The work seems rather incremental and the experiments have some methodological flaws. Specifically the main results (Fig. 4) are based on a comparison between 4 different codebases which makes it impossible to make meaningful conclusions as pointed out e.g. by [1]. The authors mention that their work is built on the work of Hong et al. (2018) yet the comparisons do not seem to include it as a baseline. I'm also concerned about how exactly are environment steps counted: in Algorithm 1 on line 27, it seems that the fitness which is used for training is evaluated by interacting with the environment yet these interactions are not counted towards total_step.\n\nFurther comments: \n1. I don't agree with the first sentence in the abstract: there's nothing special about \"continuous control\" when it comes to deceptive rewards. Perhaps you should be more specific and refer to robotics-inspired environments or to \"commonly used Mujoco environments\".\n2. I don't know what \"non-convex continuous action spaces\" refers to. All the environments studied in the paper have action space [-1, 1]^n which is a convex set.\n3. I'm also not convinced that the existing environments have \"deceptive\" rewards--they were likely tuned so that learning on them is feasible.\n4. The discussion regarding exploration completely ignores well studied methods based on e.g. upper confidence bounds or Thompson sampling. Unlike diversity based approaches, these methods are theoretically motivated.\n5. In Section 2 you start with finite horizon MDPs but then present infinite horizon discounted set up.\n6. I think the point about having only a single critic would deserve more discussion: what policy is this Q-value of? Would having an independent critic for each agent help or make things worse?\n \n[1] Henderson et al. Deep Reinforcement Learning that Matters. (2017)\n\n-------------------------------------------------------------------------------------------------------------------------------------------------------\nThanks for your comments. I'm still concerned about comparing to different codebases hence I'm keeping my score as is. Also it is still unclear to me whether you count the evaluation rollouts towards the total number of steps--fitness and hence learning is based on these rollouts so they should be included.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers generally expressed considerable reservation about the novelty of the proposed method. After reading the reviews in detail and looking at the paper, I'm inclined to agree that the contribution is rather incremental. Using normalizing flows for representing policies in RL has been studied in a number of prior works, including with soft actor-critic, and I think the novelty in this work is limited in relation to prior work. Therefore, I cannot recommend acceptance at this time.
Decision: Reject