Composing Entropic Policies using Divergence Correction

1 INTRODUCTION . Reinforcement learning algorithms coupled with powerful function approximators have recently achieved a series of successes ( Mnih et al. , 2015 ; Silver et al. , 2016 ; Lillicrap et al. , 2015 ; Kalashnikov et al. , 2018 ) . Unfortunately , while being extremely powerful , deep reinforcement learning ( DRL ) algorithms often require a large number of interactions with the environment to achieve good results , partially because they are often applied “ from scratch ” rather than in settings where they can leverage existing experience . This reduces their applicability in domains where generating experience is expensive , or learning from scratch is challenging . The data efficiency of DRL algorithms is affected by various factors and significant research effort has been directed at achieving improvements ( e.g . Popov et al. , 2017 ) . At the same time the development of basic locomotor behavior in humans can , in fact , require large amounts of experience and practice ( Adolph et al. , 2012 ) , and it can take significant effort and training to master complex , high-speed skills ( Haith & Krakauer , 2013 ) . Once such skills have been acquired , however , humans rapidly put them to work in new contexts and to solve new tasks , suggesting transfer learning as an important mechanism . Transfer learning has been explored extensively in multiple fields of the machine learning community ( see e.g . Weiss et al. , 2016 , for a recent review ) . In RL and robotics the transfer of knowledge from one task to another has been studied from a variety of angles . For the purpose of this paper we are interested in methods that are suitable for transfer in the context of high-dimensional motor control problems . We further focus on model-free approaches , which are evident in human motor control ( Haith & Krakauer , 2013 ) , and have recently been used by a variety of scalable deep RL methods ( e.g . Lillicrap et al. , 2015 ; Mnih et al. , 2015 ; Schulman et al. , 2017 ; Kalashnikov et al. , 2018 ) . Transfer may be especially valuable in domains where a small set of skills can be composed , in different combinations , to solve a variety of tasks . Different notions of compositionality have been considered in the RL and robotics literature . For instance , ‘ options ’ are associated with discrete units of behavior that can be sequenced , thus emphasizing composition in time ( Precup et al. , 1998 ) . In this paper we are concerned with a rather distinct notion of compositionality , namely how to combine and blend potentially concurrent behaviors . This form of composition is particularly relevant in high-dimensional continuous action spaces , where it is possible to achieve more than one task simultaneously ( e.g . walking somewhere while juggling ) . One approach to this challenge is via the composition of task rewards . Specifically , we are interested in the following question : If we have previously solved a set of tasks with similar transition dynamics but different reward functions , how can we leverage this knowledge to solve new tasks which can be expressed as a convex combination of those rewards functions ? This question has recently been studied in two independent lines of work : by Barreto et al . ( 2017 ; 2018 ) in the context of successor feature ( SF ) representations used for Generalized Policy Improvement ( GPI ) with deterministic policies , and by Haarnoja et al . ( 2018a ) ; van Niekerk et al . ( 2018 ) in the context of maximum entropy policies . These approaches operate in distinct frameworks but both achieve skill composition by combining the Q-functions associated with previously learned skills . We clarify the relationship between the two approaches and show that both can perform well in some situations but achieve poor results in others , often in complementary ways . We introduce a novel method of behavior composition that that can consistently achieve good performance . Our contributions are as follows : 1 . We introduce succcessor features ( SF ) in the context of maximum entropy and extend the GPI theorem to this case ( max-ent GPI ) . 2 . We provide an analysis of when GPI , and compositional “ optimism ” ( Haarnoja et al. , 2018a ) of entropy-regularized policies transfer . We construct both tabular and continuous action tasks where both fail to transfer well . 3 . We propose a correction term – which we call Divergence Correction ( DC ) – based on the Rényi divergence between policies which allows us , in principle , to recover the optimal policy for transfer for any convex combination of rewards . 4 . We demonstrate a practical implementation of these methods in continuous action spaces using adaptive importance sampling and compare the approaches introduced here : max-ent GPI and DC with optimism ( Haarnoja et al. , 2018a ) and Conditional Q functions ( Schaul et al. , 2015 ) in a variety of non-trivial continuous action transfer tasks . 2 BACKGROUND . 2.1 MULTI-TASK RL . We consider Markov Decision Processes defined by the tupleM containing : a state space S , action spaceA , a start state distribution p ( s1 ) , a transition function p ( st+1|st , at ) , a discount γ ∈ [ 0 , 1 ) and a reward function r ( st , at , st+1 ) . The objective of RL is to find a policy π ( a|s ) : S → P ( A ) which maximises the discounted expected return from any state J ( π ) = Eπ , M [ ∑∞ τ=t γ τ−trτ ] where the expected reward is dependent on the policy π and the MDPM . We formalize transfer as in Barreto et al . ( 2017 ) ; Haarnoja et al . ( 2018a ) , as the desire to perform well across all tasks in a setM ∈ T ′ after having learned policies for tasksM ∈ T , without additional experience . We assume that T and T ′ are related in two ways : all tasks share the same state transition function , and tasks in T ′ can be expressed as convex combinations of rewards associated with tasks in set T . So if we write the reward functions for tasks in T as the vectorφ = ( r1 , r2 , . . . ) , tasks in T ′ can be expressed as rw = φ ·w . We focus on combinations of two policies rb = bri + ( 1− b ) rj but the methods can be extended to more than two tasks . We refer to a transfer method as optimal , if it achieves optimal returns on tasks in T ′ , using only experience on tasks T . 2.2 SUCCESSOR FEATURES . Successor Features ( SF ) ( Dayan , 1993 ) and Generalised Policy Improvement ( GPI ) ( Barreto et al. , 2017 ; 2018 ) provide a principled solution to transfer in the setting defined above . SF make the additional assumption that the reward feature φ is fully observable , that is , the agent has access to the rewards of all tasks in T but not T ′ during training on each individual task . The key observation of SF representations is that linearity of the reward rw with respect to the features φ implies the following decomposition of the value policy of π : Qπw ( st , at ) = Eπ [ ∞∑ τ=t γτ−tφτ ·w|at ] = Eπ [ ∞∑ i=t γτ−tφτ |at ] ·w ≡ ψπ ( st , at ) ·w , ( 1 ) where ψπ is the expected discounted sum of features φ induced by policy π . This decomposition allows us to compute the action-value for π on any task w by learning ψπ . If we have a set of policies π1 , π2 , ... , πn indexed by i , SF and GPI provide a principled approach to transfer on task πw . Namely , we act according to the deterministic GPI policy πGPIw ( st ) ≡ argmaxat Q GPI w ( st , at ) ) where QGPIw ( st , at ) ≡ maxiQπiw ( st , at ) = maxiψπi ( s , a ) ·w ( 2 ) The GPI theorem guarantees the GPI policy has a return at least as good as any component policy , that is , V π GPI w ( s ) ≥ maxi V πiw ( s ) ∀s ∈ S . 2.3 MAXIMUM ENTROPY RL . The maximum entropy ( max-ent ) RL objective augments the reward to favor entropic solutions J ( π ) = Eπ , M [ ∑∞ i=τ γ τ−t ( rτ + αH [ π ( ·|sτ ) ) ] ] ( 3 ) where α is a parameter that determines the relative importance of the entropy term . This objective has been considered in a number of works including Kappen ( 2005 ) ; Todorov ( 2009 ) ; Haarnoja et al . ( 2017 ; 2018a ) ; Ziebart et al . ( 2008 ) ; Fox et al . ( 2015 ) . We define the action-value Qπ associated with eq . 3 as Qπ ( st , at ) ≡ rt + Eπ [ ∑∞ τ=t+1 γ τ−t ( rτ + αH [ π ( ·|sτ ) ] ) ] ( 4 ) ( notice Qπ ( st , at ) does not include any entropy terms for the state st ) . Soft Q iteration Q ( st , at ) ← r ( st , at , st+1 ) + γEp ( st+1|st , at ) [ V ( st+1 ) ] ( 5 ) V ( st ) ← Eπ [ Q ( st , at ) ] + αH [ π ( ·|st ) ] = α log ∫ A exp ( 1 α Q ( st , at ) ) da ≡ α logZ ( st ) ( 6 ) where π ( at|st ) ∝ exp ( 1αQ ( st , at ) ) converges to the optimal policy with standard assumptions ( Haarnoja et al. , 2017 ) . 3 COMPOSING POLICIES IN MAX-ENT REINFORCEMENT LEARNING . In this section we present two novel approaches for max-ent transfer learning . In section 4 we then outline a practical method for making use of these results . 3.1 MAX-ENT SUCCESSOR FEATURES AND GENERALIZED POLICY IMPROVEMENT . We introduce max-ent SF , which provide a practical method for computing the value of a maximum entropy policy under any convex combination of rewards . We then show the GPI theorem ( Barreto et al. , 2017 ) holds for maximum entropy policies . We define the action-dependent SF to include the entropy of the policy , excluding the current state , analogous to the max-entropy definition of Qπ in ( 4 ) : ψπ ( st , at ) ≡ φt + Eπ [ ∑∞ τ=i+1 γ τ−t ( φτ + α1 ·H [ π ( ·|s ) ] ) ] = φt + γEp ( st+1|st , at ) [ Υ ( st+1 ) ] ( 7 ) where 1 is a vector of ones of the same dimensionality as φ and we define the state-dependent successor features as the expected ψπ in analogy with V π ( s ) : Υπ ( s ) ≡ Ea∼π ( ·|s ) [ ψπ ( s , a ) ] + α1 ·H [ π ( ·|s ) ] . ( 8 ) The max-entropy action-value of π for any convex combination of rewards w is then given by Qπw ( s , a ) = ψ π ( s , a ) ·w . Max-ent SF allow us to estimate the action-value of previous policies on a new task . We show that , as in the deterministic case , there is a principled way to combine multiple policies using their action-values on task w. Theorem 3.1 ( Max-Ent Generalized Policy Improvement ) Let π1 , π2 , ... , πn be n policies with α-max-ent action-value functions Q1 , Q2 , ... , Qn and value functions V 1 , V 2 , ... , V n. Define π ( a|s ) ∝ exp ( 1 α maxiQ i ( s , a ) ) . Then , Qπ ( s , a ) ≥ max i Qi ( s , a ) for all s ∈ S and all a ∈ A , ( 9 ) V π ( s ) ≥ max i V i ( s ) for all s ∈ S , ( 10 ) where Qπ ( s , a ) and V π ( s ) are the α-max-ent action-value and value function respectively of π . Proof : See appendix A.1 . In our setup , we learn ψπi ( s , a ) , the SFs of policies πi for each task in T , we define the max-ent GPI policy for task w ∈ T ′ as πGPIw ( a|s ) ∝ exp ( 1α maxiQ πi w ( s , a ) ) = exp ( 1α maxiψ πi ( s , a ) ·w ) .

Summary: This paper proposes using Divergence Correction to compose max ent policies. Based on successor features, this method corrects the optimistic bias of Haarnoja 2018. The motivation for composing policies is sound. This paper addresses the problem statement where policies must accomplish different linear combinations of different reward functions. This method does not require observation the reward weights.