Attraction-Repulsion Actor-Critic for Continuous Control Reinforcement Learning

In reinforcement learning , robotic control tasks are often useful for understanding how agents perform in environments with deceptive rewards where the agent can easily become trapped into suboptimal solutions . One way to avoid these local optima is to use a population of agents to ensure coverage of the policy space ( a form of exploration ) , yet learning a population with the “ best ” coverage is still an open problem . In this work , we present a novel approach to population-based RL in continuous control that leverages properties of normalizing flows to perform attractive and repulsive operations between current members of the population and previously observed policies . Empirical results on the MuJoCo suite demonstrate a high performance gain for our algorithm compared to prior work , including Soft-Actor Critic ( SAC ) . 1 INTRODUCTION . Many important reinforcement learning ( RL ) tasks , such as those in robotics and self-driving cars , are challenging due to large action and state spaces ( Lee et al. , 2018 ) . In particular , environments with large continuous action spaces are prone to deceptive rewards , i.e . fall into local optima in learning ( Conti et al. , 2018 ) . Applying traditional policy optimization algorithms to these domains often leads to locally optimal , yet globally sub-optimal policies . The agent should then explore the reward landscape more thoroughly in order to avoid falling into these local optima . Not all RL domains that require exploration are suitable for understanding how to train agents that are robust to deceptive rewards . For example , Montezuma ’ s Revenge , a game in the Atari Learning Environment ( Bellemare et al. , 2013 ) , has sparse rewards ; algorithms that perform the best on this task encourage exploration by providing a denser intrinsic reward to the agent to encourage exploration ( Tang et al. , 2017 ) . On the other hand , many robotic control problems , such as those found in MuJoCo ( Todorov et al. , 2012 ) , provide the agent with a dense reward signal , yet their high-dimensional action spaces induce a multimodal , often deceptive , reward landscape . For example , in the biped environments , coordinating both arms and legs is crucial for performing well on even simple tasks such as forward motion . However , simply learning to maximize the reward can be detrimental across training : agents will tend to run and fall further away from the start point rather than discovering stable and efficient walking motion . In this setting , exploration serves to provide a more reliable learning signal for the agent by covering more different types of actions during learning . One way to maximize action space coverage is the maximum entropy RL framework ( Ziebart , 2010 ) , which prevents variance collapse by adding a policy entropy auxiliary objective . One such prominent algorithm , Soft Actor-Critic ( SAC , Haarnoja et al . ( 2018 ) ) , has been shown to excel in large continuous action spaces . To further improve on exploration properties of SAC , one can maintain a population of agents that cover non-identical sections of the policy space . To prevent premature convergence , a diversity-preserving mechanism is typically put in place ; balancing the objective and the diversity term becomes key to converging to a global optimum ( Hong et al. , 2018 ) . This paper studies a particular family of population-based exploration methods , which conduct coordinated local search in the policy space . Prior work on population-based strategies improves performance on robotic control domains through stochastic perturbation on a single actor ’ s parameter ( Pourchot & Sigaud , 2019 ) or a set of actor ’ s parameters ( Conti et al. , 2018 ; Khadka & Tumer , 2018 ; Liu et al. , 2017 ) . We hypothesize that exploring directly in the policy space will be more effective than perturbing the parameters of the policy , as the latter does not guarantee diversity ( i.e. , different neural network parameterizations can approximately represent the same function ) . Given a population of RL agents , we enforce local exploration using an Attraction-Repulsion ( AR ) mechanism . The later consists in adding an auxiliary loss to encourage pairwise attraction or repulsion between members of a population , as measured by a divergence term . We make use of the KullbackLeibler ( KL ) divergence because of its desirable statistical properties and its easiness of computation . However , naively maximizing the KL term between two Gaussian policies can be detrimental ( e.g . drives both means apart ) . Because of this , we parametrize the policy with a general family of distributions called Normalizing Flows ( NFs , Rezende & Mohamed , 2015 ) ; this modification allows to improve upon AR+Gaussian ( see Appendix Figure 6 ) . NFs are shown to improve the expressivity of the policies using invertible mappings while maintaining entropy guarantees ( Mazoure et al. , 2019 ; Tang & Agrawal , 2018 ) . Nonlinear density estimators have also been previously used for deep RL problems in contexts of distributional RL ( Doan et al. , 2018 ) and reward shaping ( Tang et al. , 2017 ) . The AR objective blends particularly well with SAC , since computing the KL requires stochastic policies with tractable densities for each agent . 2 PRELIMINARIES . We first formalize the RL setting in a Markov decision process ( MDP ) . A discrete-time , finite-horizon , MDP ( Bellman , 1957 ; Puterman , 2014 ) is described by a state space S , an action spaceA , a transition function P : S ×A× S 7→ R+ , and a reward function r : S ×A 7→ R.1 On each round t , an agent interacting with this MDP observes the current state st ∈ S , selects an action at ∈ A , and observes a reward r ( st , at ) ∈ R upon transitioning to a new state st+1 ∼ P ( st , at ) . Let γ ∈ [ 0 , 1 ] be a discount factor . The goal of an agent evolving in a discounted MDP is to learn a policy π : S × A 7→ [ 0 , 1 ] such as taking action at ∼ π ( ·|st ) would maximize the expected sum of discounted returns , V π ( s ) = Eπ [ ∞∑ t=0 γtr ( st , at ) |s0 = s ] . In the following , we use ρπ to denote the trajectory distribution induced by following policy π . If S or A are vector spaces , action and space vectors are respectively denoted by a and s . 2.1 DISCOVERING NEW SOLUTIONS THROUGH POPULATION-BASED ATTRACTION-REPULSION . Consider evolving a population of M agents , also called individuals , { πθm } Mm=1 , each agent corresponding to a policy with its own parameters . In order to discover new solutions , we aim to generate agents that can mimic some target policy while following a path different from those of other policies . Let G denote an archive of policies encountered in previous generations of the population . A natural way of enforcing π to be different from or similar to the policies contained in G is by augmenting the loss of the agent with an Attraction-Repulsion ( AR ) term : LAR = − E π′∼G [ βπ′DKL [ π||π′ ] ] , ( 1 ) where π′ is an archived policy and βπ′ is a coefficient weighting the relative importance of the Kullback-Leibler ( KL ) divergence between π and π′ , which we will choose to be a function of the average reward ( see Sec . 3.2 below ) . Intuitively , Eq . 1 adds to the agent objective a weighted average distance between the current and the archived policies . For βπ′ ≥ 0 , the agent tends to move away from the archived policy ’ s behavior ( i.e . repulsion , see Figure 1 ) a ) . On the other hand , βπ′ < 0 encourages the agent π to imitate π′ ( i.e . attraction ) . Requirements for AR In order for agents within a population to be trained using the proposed AR-based loss ( Eq . 1 ) , we have the following requirements : 1 . Their policies should be stochastic , so that the KL-divergence between two policies is well-defined . 1A and S can be either discrete or continuous . 2 . Their policies should have tractable distributions , so that the KL-divergence can be computed easily , either with closed-form solution or Monte Carlo estimation . Several RL algorithms enjoy such properties ( Haarnoja et al. , 2018 ; Schulman et al. , 2015 ; 2017 ) . In particular , the soft actor-critic ( SAC , Haarnoja et al. , 2018 ) is a straightforward choice , as it currently outperforms other candidates and is off-policy , thus maintains a single critic shared among all agents ( instead of one critic per agent ) , which reduces computation costs . 2.2 SOFT ACTOR-CRITIC . SAC ( Haarnoja et al. , 2018 ) is an off-policy learning algorithm which finds the information projection of the Boltzmann Q-function onto the set of diagonal Gaussian policies Π : π = arg min π′∈Π DKL ( π′ ( .|st ) ∥∥∥∥exp ( 1αQπold ( st , . ) ) Zπold ( st ) ) , where α ∈ ( 0 , 1 ) controls the temperature , i.e . the peakedness of the distribution . The policy π , critic Q , and value function V are optimized according to the following loss functions : Lπ , SAC = Est∼B [ Eat∼π [ α log π ( at|st ) −Q ( st , at ) ] ] ( 2 ) LQ = E ( s , a , r , s′ ) ∼B [ { Q ( s , a ) − ( r + γV πν ( s′ ) ) } 2 ] ( 3 ) LV = Est∼D [ 1 2 { V πν ( st ) − Eat∼π [ Q ( st , at ) − α log π ( at|st ) ] } 2 ] , ( 4 ) where B is the replay buffer . The policy used in SAC as introduced in Haarnoja et al . ( 2018 ) is Gaussian , which is both stochastic and tractable , thus compatible with our AR loss function in Eq . 1 . Together with the AR loss in Eq . 1 , the final policy loss becomes : Lπ = Lπ , SAC + LAR ( 5 ) However , Gaussian policies are arguably of limited expressibility ; we can improve on the family of policy distributions without sacrificing qualities necessary for AR or SAC by using Normalizing Flows ( NFs , Rezende & Mohamed , 2015 ) . 2.3 NORMALIZING FLOWS . NFs ( Rezende & Mohamed , 2015 ) were introduced as a means of transforming simple distributions into more complex distributions using learnable and invertible functions . Given a random variable z0 with density q0 , they define a set of differentiable and invertible functions , { fi } Ni=1 , which generate a sequence of d-dimensional random variables , { zi } Ni=1 . Because SAC uses explicit , yet simple parametric policies , NFs can be used to transform the SAC policy into a richer one ( e.g. , multimodal ) without risk loss of information . For example , Mazoure et al . ( 2019 ) enhanced SAC using a family of radial contractions around a point z0 ∈ Rd , f ( z ) = z + β α+ ||z− z0||2 ( z− z0 ) ( 6 ) for α ∈ R+ and β ∈ R. This results in a rich set of policies comprised of an initial noise sample a0 , a state-noise embedding hθ ( a0 , st ) , and a flow { fφi } Ni=1 of arbitrary length N , parameterized by φ = { φi } Ni=1 . Sampling from the policy πφ , θ ( at|st ) can be described by the following set of equations : a0 ∼ N ( 0 , I ) ; z = hθ ( a0 , st ) ; at = fφN ◦ fφN−1 ◦ ... ◦ fφ1 ( z ) , ( 7 ) where hθ = a0σI + µ ( st ) depends on the state and the noise variance σ > 0 . Different SAC policies can thus be crafted by parameterizing their NFs layers .

Summary: RL in environments with deceptive rewards can produce sub-optimal policies. To remedy this, the paper proposes a method for population-based exploration. Multiple actors, each parameterized with policies based on Normalizing Flows (radial contractions), are optimized over iterations using the off-policy SAC algorithm. To encourage diverse-exploration as well as high-performance, the SAC-policy-gradient is supplemented with gradient of an “attraction” or “repulsion” term, as defined using the KL-divergence of current policy to another policy from an online archive. When applying the KL-gradient, the authors find it crucial to only update the flow layers, and not the base Gaussian policy.