Composing Entropic Policies using Divergence Correction

1 INTRODUCTION . Reinforcement learning algorithms coupled with powerful function approximators have recently achieved a series of successes ( Mnih et al. , 2015 ; Silver et al. , 2016 ; Lillicrap et al. , 2015 ; Kalashnikov et al. , 2018 ) . Unfortunately , while being extremely powerful , deep reinforcement learning ( DRL ) algorithms often require a large number of interactions with the environment to achieve good results , partially because they are often applied “ from scratch ” rather than in settings where they can leverage existing experience . This reduces their applicability in domains where generating experience is expensive , or learning from scratch is challenging . The data efficiency of DRL algorithms is affected by various factors and significant research effort has been directed at achieving improvements ( e.g . Popov et al. , 2017 ) . At the same time the development of basic locomotor behavior in humans can , in fact , require large amounts of experience and practice ( Adolph et al. , 2012 ) , and it can take significant effort and training to master complex , high-speed skills ( Haith & Krakauer , 2013 ) . Once such skills have been acquired , however , humans rapidly put them to work in new contexts and to solve new tasks , suggesting transfer learning as an important mechanism . Transfer learning has been explored extensively in multiple fields of the machine learning community ( see e.g . Weiss et al. , 2016 , for a recent review ) . In RL and robotics the transfer of knowledge from one task to another has been studied from a variety of angles . For the purpose of this paper we are interested in methods that are suitable for transfer in the context of high-dimensional motor control problems . We further focus on model-free approaches , which are evident in human motor control ( Haith & Krakauer , 2013 ) , and have recently been used by a variety of scalable deep RL methods ( e.g . Lillicrap et al. , 2015 ; Mnih et al. , 2015 ; Schulman et al. , 2017 ; Kalashnikov et al. , 2018 ) . Transfer may be especially valuable in domains where a small set of skills can be composed , in different combinations , to solve a variety of tasks . Different notions of compositionality have been considered in the RL and robotics literature . For instance , ‘ options ’ are associated with discrete units of behavior that can be sequenced , thus emphasizing composition in time ( Precup et al. , 1998 ) . In this paper we are concerned with a rather distinct notion of compositionality , namely how to combine and blend potentially concurrent behaviors . This form of composition is particularly relevant in high-dimensional continuous action spaces , where it is possible to achieve more than one task simultaneously ( e.g . walking somewhere while juggling ) . One approach to this challenge is via the composition of task rewards . Specifically , we are interested in the following question : If we have previously solved a set of tasks with similar transition dynamics but different reward functions , how can we leverage this knowledge to solve new tasks which can be expressed as a convex combination of those rewards functions ? This question has recently been studied in two independent lines of work : by Barreto et al . ( 2017 ; 2018 ) in the context of successor feature ( SF ) representations used for Generalized Policy Improvement ( GPI ) with deterministic policies , and by Haarnoja et al . ( 2018a ) ; van Niekerk et al . ( 2018 ) in the context of maximum entropy policies . These approaches operate in distinct frameworks but both achieve skill composition by combining the Q-functions associated with previously learned skills . We clarify the relationship between the two approaches and show that both can perform well in some situations but achieve poor results in others , often in complementary ways . We introduce a novel method of behavior composition that that can consistently achieve good performance . Our contributions are as follows : 1 . We introduce succcessor features ( SF ) in the context of maximum entropy and extend the GPI theorem to this case ( max-ent GPI ) . 2 . We provide an analysis of when GPI , and compositional “ optimism ” ( Haarnoja et al. , 2018a ) of entropy-regularized policies transfer . We construct both tabular and continuous action tasks where both fail to transfer well . 3 . We propose a correction term – which we call Divergence Correction ( DC ) – based on the Rényi divergence between policies which allows us , in principle , to recover the optimal policy for transfer for any convex combination of rewards . 4 . We demonstrate a practical implementation of these methods in continuous action spaces using adaptive importance sampling and compare the approaches introduced here : max-ent GPI and DC with optimism ( Haarnoja et al. , 2018a ) and Conditional Q functions ( Schaul et al. , 2015 ) in a variety of non-trivial continuous action transfer tasks . 2 BACKGROUND . 2.1 MULTI-TASK RL . We consider Markov Decision Processes defined by the tupleM containing : a state space S , action spaceA , a start state distribution p ( s1 ) , a transition function p ( st+1|st , at ) , a discount γ ∈ [ 0 , 1 ) and a reward function r ( st , at , st+1 ) . The objective of RL is to find a policy π ( a|s ) : S → P ( A ) which maximises the discounted expected return from any state J ( π ) = Eπ , M [ ∑∞ τ=t γ τ−trτ ] where the expected reward is dependent on the policy π and the MDPM . We formalize transfer as in Barreto et al . ( 2017 ) ; Haarnoja et al . ( 2018a ) , as the desire to perform well across all tasks in a setM ∈ T ′ after having learned policies for tasksM ∈ T , without additional experience . We assume that T and T ′ are related in two ways : all tasks share the same state transition function , and tasks in T ′ can be expressed as convex combinations of rewards associated with tasks in set T . So if we write the reward functions for tasks in T as the vectorφ = ( r1 , r2 , . . . ) , tasks in T ′ can be expressed as rw = φ ·w . We focus on combinations of two policies rb = bri + ( 1− b ) rj but the methods can be extended to more than two tasks . We refer to a transfer method as optimal , if it achieves optimal returns on tasks in T ′ , using only experience on tasks T . 2.2 SUCCESSOR FEATURES . Successor Features ( SF ) ( Dayan , 1993 ) and Generalised Policy Improvement ( GPI ) ( Barreto et al. , 2017 ; 2018 ) provide a principled solution to transfer in the setting defined above . SF make the additional assumption that the reward feature φ is fully observable , that is , the agent has access to the rewards of all tasks in T but not T ′ during training on each individual task . The key observation of SF representations is that linearity of the reward rw with respect to the features φ implies the following decomposition of the value policy of π : Qπw ( st , at ) = Eπ [ ∞∑ τ=t γτ−tφτ ·w|at ] = Eπ [ ∞∑ i=t γτ−tφτ |at ] ·w ≡ ψπ ( st , at ) ·w , ( 1 ) where ψπ is the expected discounted sum of features φ induced by policy π . This decomposition allows us to compute the action-value for π on any task w by learning ψπ . If we have a set of policies π1 , π2 , ... , πn indexed by i , SF and GPI provide a principled approach to transfer on task πw . Namely , we act according to the deterministic GPI policy πGPIw ( st ) ≡ argmaxat Q GPI w ( st , at ) ) where QGPIw ( st , at ) ≡ maxiQπiw ( st , at ) = maxiψπi ( s , a ) ·w ( 2 ) The GPI theorem guarantees the GPI policy has a return at least as good as any component policy , that is , V π GPI w ( s ) ≥ maxi V πiw ( s ) ∀s ∈ S . 2.3 MAXIMUM ENTROPY RL . The maximum entropy ( max-ent ) RL objective augments the reward to favor entropic solutions J ( π ) = Eπ , M [ ∑∞ i=τ γ τ−t ( rτ + αH [ π ( ·|sτ ) ) ] ] ( 3 ) where α is a parameter that determines the relative importance of the entropy term . This objective has been considered in a number of works including Kappen ( 2005 ) ; Todorov ( 2009 ) ; Haarnoja et al . ( 2017 ; 2018a ) ; Ziebart et al . ( 2008 ) ; Fox et al . ( 2015 ) . We define the action-value Qπ associated with eq . 3 as Qπ ( st , at ) ≡ rt + Eπ [ ∑∞ τ=t+1 γ τ−t ( rτ + αH [ π ( ·|sτ ) ] ) ] ( 4 ) ( notice Qπ ( st , at ) does not include any entropy terms for the state st ) . Soft Q iteration Q ( st , at ) ← r ( st , at , st+1 ) + γEp ( st+1|st , at ) [ V ( st+1 ) ] ( 5 ) V ( st ) ← Eπ [ Q ( st , at ) ] + αH [ π ( ·|st ) ] = α log ∫ A exp ( 1 α Q ( st , at ) ) da ≡ α logZ ( st ) ( 6 ) where π ( at|st ) ∝ exp ( 1αQ ( st , at ) ) converges to the optimal policy with standard assumptions ( Haarnoja et al. , 2017 ) . 3 COMPOSING POLICIES IN MAX-ENT REINFORCEMENT LEARNING . In this section we present two novel approaches for max-ent transfer learning . In section 4 we then outline a practical method for making use of these results . 3.1 MAX-ENT SUCCESSOR FEATURES AND GENERALIZED POLICY IMPROVEMENT . We introduce max-ent SF , which provide a practical method for computing the value of a maximum entropy policy under any convex combination of rewards . We then show the GPI theorem ( Barreto et al. , 2017 ) holds for maximum entropy policies . We define the action-dependent SF to include the entropy of the policy , excluding the current state , analogous to the max-entropy definition of Qπ in ( 4 ) : ψπ ( st , at ) ≡ φt + Eπ [ ∑∞ τ=i+1 γ τ−t ( φτ + α1 ·H [ π ( ·|s ) ] ) ] = φt + γEp ( st+1|st , at ) [ Υ ( st+1 ) ] ( 7 ) where 1 is a vector of ones of the same dimensionality as φ and we define the state-dependent successor features as the expected ψπ in analogy with V π ( s ) : Υπ ( s ) ≡ Ea∼π ( ·|s ) [ ψπ ( s , a ) ] + α1 ·H [ π ( ·|s ) ] . ( 8 ) The max-entropy action-value of π for any convex combination of rewards w is then given by Qπw ( s , a ) = ψ π ( s , a ) ·w . Max-ent SF allow us to estimate the action-value of previous policies on a new task . We show that , as in the deterministic case , there is a principled way to combine multiple policies using their action-values on task w. Theorem 3.1 ( Max-Ent Generalized Policy Improvement ) Let π1 , π2 , ... , πn be n policies with α-max-ent action-value functions Q1 , Q2 , ... , Qn and value functions V 1 , V 2 , ... , V n. Define π ( a|s ) ∝ exp ( 1 α maxiQ i ( s , a ) ) . Then , Qπ ( s , a ) ≥ max i Qi ( s , a ) for all s ∈ S and all a ∈ A , ( 9 ) V π ( s ) ≥ max i V i ( s ) for all s ∈ S , ( 10 ) where Qπ ( s , a ) and V π ( s ) are the α-max-ent action-value and value function respectively of π . Proof : See appendix A.1 . In our setup , we learn ψπi ( s , a ) , the SFs of policies πi for each task in T , we define the max-ent GPI policy for task w ∈ T ′ as πGPIw ( a|s ) ∝ exp ( 1α maxiQ πi w ( s , a ) ) = exp ( 1α maxiψ πi ( s , a ) ·w ) .

Summary: The authors introduce Divergence Correction (DC) for the problem of transfer learning by composing policies. There approach builds on GPI with a maximum entropy objective. They also prove that DC solves for the max-entropy optimal interpolation between two policies and derive a practical approximation for this algorithm. They provide experimental results in a gridworld problem and study their approximate algorithm in two continuous control problems.