Attraction-Repulsion Actor-Critic for Continuous Control Reinforcement Learning

In reinforcement learning , robotic control tasks are often useful for understanding how agents perform in environments with deceptive rewards where the agent can easily become trapped into suboptimal solutions . One way to avoid these local optima is to use a population of agents to ensure coverage of the policy space ( a form of exploration ) , yet learning a population with the “ best ” coverage is still an open problem . In this work , we present a novel approach to population-based RL in continuous control that leverages properties of normalizing flows to perform attractive and repulsive operations between current members of the population and previously observed policies . Empirical results on the MuJoCo suite demonstrate a high performance gain for our algorithm compared to prior work , including Soft-Actor Critic ( SAC ) . 1 INTRODUCTION . Many important reinforcement learning ( RL ) tasks , such as those in robotics and self-driving cars , are challenging due to large action and state spaces ( Lee et al. , 2018 ) . In particular , environments with large continuous action spaces are prone to deceptive rewards , i.e . fall into local optima in learning ( Conti et al. , 2018 ) . Applying traditional policy optimization algorithms to these domains often leads to locally optimal , yet globally sub-optimal policies . The agent should then explore the reward landscape more thoroughly in order to avoid falling into these local optima . Not all RL domains that require exploration are suitable for understanding how to train agents that are robust to deceptive rewards . For example , Montezuma ’ s Revenge , a game in the Atari Learning Environment ( Bellemare et al. , 2013 ) , has sparse rewards ; algorithms that perform the best on this task encourage exploration by providing a denser intrinsic reward to the agent to encourage exploration ( Tang et al. , 2017 ) . On the other hand , many robotic control problems , such as those found in MuJoCo ( Todorov et al. , 2012 ) , provide the agent with a dense reward signal , yet their high-dimensional action spaces induce a multimodal , often deceptive , reward landscape . For example , in the biped environments , coordinating both arms and legs is crucial for performing well on even simple tasks such as forward motion . However , simply learning to maximize the reward can be detrimental across training : agents will tend to run and fall further away from the start point rather than discovering stable and efficient walking motion . In this setting , exploration serves to provide a more reliable learning signal for the agent by covering more different types of actions during learning . One way to maximize action space coverage is the maximum entropy RL framework ( Ziebart , 2010 ) , which prevents variance collapse by adding a policy entropy auxiliary objective . One such prominent algorithm , Soft Actor-Critic ( SAC , Haarnoja et al . ( 2018 ) ) , has been shown to excel in large continuous action spaces . To further improve on exploration properties of SAC , one can maintain a population of agents that cover non-identical sections of the policy space . To prevent premature convergence , a diversity-preserving mechanism is typically put in place ; balancing the objective and the diversity term becomes key to converging to a global optimum ( Hong et al. , 2018 ) . This paper studies a particular family of population-based exploration methods , which conduct coordinated local search in the policy space . Prior work on population-based strategies improves performance on robotic control domains through stochastic perturbation on a single actor ’ s parameter ( Pourchot & Sigaud , 2019 ) or a set of actor ’ s parameters ( Conti et al. , 2018 ; Khadka & Tumer , 2018 ; Liu et al. , 2017 ) . We hypothesize that exploring directly in the policy space will be more effective than perturbing the parameters of the policy , as the latter does not guarantee diversity ( i.e. , different neural network parameterizations can approximately represent the same function ) . Given a population of RL agents , we enforce local exploration using an Attraction-Repulsion ( AR ) mechanism . The later consists in adding an auxiliary loss to encourage pairwise attraction or repulsion between members of a population , as measured by a divergence term . We make use of the KullbackLeibler ( KL ) divergence because of its desirable statistical properties and its easiness of computation . However , naively maximizing the KL term between two Gaussian policies can be detrimental ( e.g . drives both means apart ) . Because of this , we parametrize the policy with a general family of distributions called Normalizing Flows ( NFs , Rezende & Mohamed , 2015 ) ; this modification allows to improve upon AR+Gaussian ( see Appendix Figure 6 ) . NFs are shown to improve the expressivity of the policies using invertible mappings while maintaining entropy guarantees ( Mazoure et al. , 2019 ; Tang & Agrawal , 2018 ) . Nonlinear density estimators have also been previously used for deep RL problems in contexts of distributional RL ( Doan et al. , 2018 ) and reward shaping ( Tang et al. , 2017 ) . The AR objective blends particularly well with SAC , since computing the KL requires stochastic policies with tractable densities for each agent . 2 PRELIMINARIES . We first formalize the RL setting in a Markov decision process ( MDP ) . A discrete-time , finite-horizon , MDP ( Bellman , 1957 ; Puterman , 2014 ) is described by a state space S , an action spaceA , a transition function P : S ×A× S 7→ R+ , and a reward function r : S ×A 7→ R.1 On each round t , an agent interacting with this MDP observes the current state st ∈ S , selects an action at ∈ A , and observes a reward r ( st , at ) ∈ R upon transitioning to a new state st+1 ∼ P ( st , at ) . Let γ ∈ [ 0 , 1 ] be a discount factor . The goal of an agent evolving in a discounted MDP is to learn a policy π : S × A 7→ [ 0 , 1 ] such as taking action at ∼ π ( ·|st ) would maximize the expected sum of discounted returns , V π ( s ) = Eπ [ ∞∑ t=0 γtr ( st , at ) |s0 = s ] . In the following , we use ρπ to denote the trajectory distribution induced by following policy π . If S or A are vector spaces , action and space vectors are respectively denoted by a and s . 2.1 DISCOVERING NEW SOLUTIONS THROUGH POPULATION-BASED ATTRACTION-REPULSION . Consider evolving a population of M agents , also called individuals , { πθm } Mm=1 , each agent corresponding to a policy with its own parameters . In order to discover new solutions , we aim to generate agents that can mimic some target policy while following a path different from those of other policies . Let G denote an archive of policies encountered in previous generations of the population . A natural way of enforcing π to be different from or similar to the policies contained in G is by augmenting the loss of the agent with an Attraction-Repulsion ( AR ) term : LAR = − E π′∼G [ βπ′DKL [ π||π′ ] ] , ( 1 ) where π′ is an archived policy and βπ′ is a coefficient weighting the relative importance of the Kullback-Leibler ( KL ) divergence between π and π′ , which we will choose to be a function of the average reward ( see Sec . 3.2 below ) . Intuitively , Eq . 1 adds to the agent objective a weighted average distance between the current and the archived policies . For βπ′ ≥ 0 , the agent tends to move away from the archived policy ’ s behavior ( i.e . repulsion , see Figure 1 ) a ) . On the other hand , βπ′ < 0 encourages the agent π to imitate π′ ( i.e . attraction ) . Requirements for AR In order for agents within a population to be trained using the proposed AR-based loss ( Eq . 1 ) , we have the following requirements : 1 . Their policies should be stochastic , so that the KL-divergence between two policies is well-defined . 1A and S can be either discrete or continuous . 2 . Their policies should have tractable distributions , so that the KL-divergence can be computed easily , either with closed-form solution or Monte Carlo estimation . Several RL algorithms enjoy such properties ( Haarnoja et al. , 2018 ; Schulman et al. , 2015 ; 2017 ) . In particular , the soft actor-critic ( SAC , Haarnoja et al. , 2018 ) is a straightforward choice , as it currently outperforms other candidates and is off-policy , thus maintains a single critic shared among all agents ( instead of one critic per agent ) , which reduces computation costs . 2.2 SOFT ACTOR-CRITIC . SAC ( Haarnoja et al. , 2018 ) is an off-policy learning algorithm which finds the information projection of the Boltzmann Q-function onto the set of diagonal Gaussian policies Π : π = arg min π′∈Π DKL ( π′ ( .|st ) ∥∥∥∥exp ( 1αQπold ( st , . ) ) Zπold ( st ) ) , where α ∈ ( 0 , 1 ) controls the temperature , i.e . the peakedness of the distribution . The policy π , critic Q , and value function V are optimized according to the following loss functions : Lπ , SAC = Est∼B [ Eat∼π [ α log π ( at|st ) −Q ( st , at ) ] ] ( 2 ) LQ = E ( s , a , r , s′ ) ∼B [ { Q ( s , a ) − ( r + γV πν ( s′ ) ) } 2 ] ( 3 ) LV = Est∼D [ 1 2 { V πν ( st ) − Eat∼π [ Q ( st , at ) − α log π ( at|st ) ] } 2 ] , ( 4 ) where B is the replay buffer . The policy used in SAC as introduced in Haarnoja et al . ( 2018 ) is Gaussian , which is both stochastic and tractable , thus compatible with our AR loss function in Eq . 1 . Together with the AR loss in Eq . 1 , the final policy loss becomes : Lπ = Lπ , SAC + LAR ( 5 ) However , Gaussian policies are arguably of limited expressibility ; we can improve on the family of policy distributions without sacrificing qualities necessary for AR or SAC by using Normalizing Flows ( NFs , Rezende & Mohamed , 2015 ) . 2.3 NORMALIZING FLOWS . NFs ( Rezende & Mohamed , 2015 ) were introduced as a means of transforming simple distributions into more complex distributions using learnable and invertible functions . Given a random variable z0 with density q0 , they define a set of differentiable and invertible functions , { fi } Ni=1 , which generate a sequence of d-dimensional random variables , { zi } Ni=1 . Because SAC uses explicit , yet simple parametric policies , NFs can be used to transform the SAC policy into a richer one ( e.g. , multimodal ) without risk loss of information . For example , Mazoure et al . ( 2019 ) enhanced SAC using a family of radial contractions around a point z0 ∈ Rd , f ( z ) = z + β α+ ||z− z0||2 ( z− z0 ) ( 6 ) for α ∈ R+ and β ∈ R. This results in a rich set of policies comprised of an initial noise sample a0 , a state-noise embedding hθ ( a0 , st ) , and a flow { fφi } Ni=1 of arbitrary length N , parameterized by φ = { φi } Ni=1 . Sampling from the policy πφ , θ ( at|st ) can be described by the following set of equations : a0 ∼ N ( 0 , I ) ; z = hθ ( a0 , st ) ; at = fφN ◦ fφN−1 ◦ ... ◦ fφ1 ( z ) , ( 7 ) where hθ = a0σI + µ ( st ) depends on the state and the noise variance σ > 0 . Different SAC policies can thus be crafted by parameterizing their NFs layers .

Summary: The paper proposes an ensemble method for reinforcement learning in which the policy updates are modulated with a loss which encourages diversity among all experienced policies. It is a combination of SAC, normalizing flow policies, and an approach to diversity considered by Hong et al. (2018). The work seems rather incremental and the experiments have some methodological flaws. Specifically the main results (Fig. 4) are based on a comparison between 4 different codebases which makes it impossible to make meaningful conclusions as pointed out e.g. by [1]. The authors mention that their work is built on the work of Hong et al. (2018) yet the comparisons do not seem to include it as a baseline. I'm also concerned about how exactly are environment steps counted: in Algorithm 1 on line 27, it seems that the fitness which is used for training is evaluated by interacting with the environment yet these interactions are not counted towards total_step.