Year: 2018

Abstract:
Deep reinforcement learning (RL) algorithms have made great strides in recent years. An important remaining challenge is the ability to quickly transfer existing skills to novel tasks, and to combine existing skills with newly acquired ones. In domains where tasks are solved by composing skills this capacity holds the promise of dramatically reducing the data requirements of deep RL algorithms, and hence increasing their applicability. Recent work has studied ways of composing behaviors represented in the form of action-value functions. We analyze these methods to highlight their strengths and weaknesses, and point out situations where each of them is susceptible to poor performance. To perform this analysis we extend generalized policy improvement to the max-entropy framework and introduce a method for the practical implementation of successor features in continuous action spaces. Then we propose a novel approach which, in principle, recovers the optimal policy during transfer. This method works by explicitly learning the (discounted, future) divergence between policies. We study this approach in the tabular case and propose a scalable variant that is applicable in multi-dimensional continuous action spaces. We compare our approach with existing ones on a range of non-trivial continuous control problems with compositional structure, and demonstrate qualitatively better performance despite not requiring simultaneous observation of all task rewards.

heading: 1 INTRODUCTION
text: Reinforcement learning algorithms coupled with powerful function approximators have recently achieved a series of successes (Mnih et al., 2015; Silver et al., 2016; Lillicrap et al., 2015; Kalashnikov et al., 2018). Unfortunately, while being extremely powerful, deep reinforcement learning (DRL) algorithms often require a large number of interactions with the environment to achieve good results, partially because they are often applied “from scratch” rather than in settings where they can leverage existing experience. This reduces their applicability in domains where generating experience is expensive, or learning from scratch is challenging.
The data efficiency of DRL algorithms is affected by various factors and significant research effort has been directed at achieving improvements (e.g. Popov et al., 2017). At the same time the development of basic locomotor behavior in humans can, in fact, require large amounts of experience and practice (Adolph et al., 2012), and it can take significant effort and training to master complex, high-speed skills (Haith & Krakauer, 2013). Once such skills have been acquired, however, humans rapidly put them to work in new contexts and to solve new tasks, suggesting transfer learning as an important mechanism.
Transfer learning has been explored extensively in multiple fields of the machine learning community (see e.g. Weiss et al., 2016, for a recent review). In RL and robotics the transfer of knowledge from one task to another has been studied from a variety of angles.
For the purpose of this paper we are interested in methods that are suitable for transfer in the context of high-dimensional motor control problems. We further focus on model-free approaches, which are evident in human motor control (Haith & Krakauer, 2013), and have recently been used by a variety of scalable deep RL methods (e.g. Lillicrap et al., 2015; Mnih et al., 2015; Schulman et al., 2017; Kalashnikov et al., 2018).
Transfer may be especially valuable in domains where a small set of skills can be composed, in different combinations, to solve a variety of tasks. Different notions of compositionality have been considered in the RL and robotics literature. For instance, ‘options’ are associated with discrete units of behavior that can be sequenced, thus emphasizing composition in time (Precup et al., 1998). In this paper we are concerned with a rather distinct notion of compositionality, namely how to combine and blend potentially concurrent behaviors. This form of composition is particularly relevant in high-dimensional continuous action spaces, where it is possible to achieve more than one task simultaneously (e.g. walking somewhere while juggling).
One approach to this challenge is via the composition of task rewards. Specifically, we are interested in the following question: If we have previously solved a set of tasks with similar transition dynamics but different reward functions, how can we leverage this knowledge to solve new tasks which can be expressed as a convex combination of those rewards functions?
This question has recently been studied in two independent lines of work: by Barreto et al. (2017; 2018) in the context of successor feature (SF) representations used for Generalized Policy Improvement (GPI) with deterministic policies, and by Haarnoja et al. (2018a); van Niekerk et al. (2018) in the context of maximum entropy policies. These approaches operate in distinct frameworks but both achieve skill composition by combining the Q-functions associated with previously learned skills.
We clarify the relationship between the two approaches and show that both can perform well in some situations but achieve poor results in others, often in complementary ways. We introduce a novel method of behavior composition that that can consistently achieve good performance.
Our contributions are as follows:
1. We introduce succcessor features (SF) in the context of maximum entropy and extend the GPI theorem to this case (max-ent GPI).
2. We provide an analysis of when GPI, and compositional “optimism” (Haarnoja et al., 2018a) of entropy-regularized policies transfer. We construct both tabular and continuous action tasks where both fail to transfer well.
3. We propose a correction term – which we call Divergence Correction (DC)– based on the Rényi divergence between policies which allows us, in principle, to recover the optimal policy for transfer for any convex combination of rewards.
4. We demonstrate a practical implementation of these methods in continuous action spaces using adaptive importance sampling and compare the approaches introduced here: max-ent GPI and DC with optimism(Haarnoja et al., 2018a) and Conditional Q functions (Schaul et al., 2015) in a variety of non-trivial continuous action transfer tasks.

heading: 2 BACKGROUND

heading: 2.1 MULTI-TASK RL
text: We consider Markov Decision Processes defined by the tupleM containing: a state space S, action spaceA, a start state distribution p(s1), a transition function p(st+1|st, at), a discount γ ∈ [0, 1) and a reward function r(st, at, st+1). The objective of RL is to find a policy π(a|s) : S → P(A) which maximises the discounted expected return from any state J(π) = Eπ,M [ ∑∞ τ=t γ
τ−trτ ] where the expected reward is dependent on the policy π and the MDPM. We formalize transfer as in Barreto et al. (2017); Haarnoja et al. (2018a), as the desire to perform well across all tasks in a setM ∈ T ′ after having learned policies for tasksM ∈ T , without additional experience. We assume that T and T ′ are related in two ways: all tasks share the same state transition function, and tasks in T ′ can be expressed as convex combinations of rewards associated with tasks in set T . So if we write the reward functions for tasks in T as the vectorφ = (r1, r2, . . . ), tasks in T ′ can be expressed as rw = φ ·w. We focus on combinations of two policies rb = bri + (1− b)rj but the methods can be extended to more than two tasks. We refer to a transfer method as optimal, if it achieves optimal returns on tasks in T ′, using only experience on tasks T .

heading: 2.2 SUCCESSOR FEATURES
text: Successor Features (SF) (Dayan, 1993) and Generalised Policy Improvement (GPI) (Barreto et al., 2017; 2018) provide a principled solution to transfer in the setting defined above. SF make the additional assumption that the reward feature φ is fully observable, that is, the agent has access to the rewards of all tasks in T but not T ′ during training on each individual task. The key observation of SF representations is that linearity of the reward rw with respect to the features φ implies the following decomposition of the value policy of π:
Qπw(st, at) = Eπ [ ∞∑ τ=t γτ−tφτ ·w|at ] = Eπ [ ∞∑ i=t γτ−tφτ |at ] ·w ≡ ψπ(st, at) ·w, (1)
where ψπ is the expected discounted sum of features φ induced by policy π. This decomposition allows us to compute the action-value for π on any task w by learning ψπ .
If we have a set of policies π1, π2, ..., πn indexed by i, SF and GPI provide a principled approach to transfer on task πw. Namely, we act according to the deterministic GPI policy πGPIw (st) ≡ argmaxat Q GPI w (st, at)) where
QGPIw (st, at) ≡ maxiQπiw (st, at) = maxiψπi(s, a) ·w (2) The GPI theorem guarantees the GPI policy has a return at least as good as any component policy, that is, V π GPI
w (s) ≥ maxi V πiw (s) ∀s ∈ S.

heading: 2.3 MAXIMUM ENTROPY RL
text: The maximum entropy (max-ent) RL objective augments the reward to favor entropic solutions J(π) = Eπ,M [ ∑∞ i=τ γ
τ−t(rτ + αH[π(·|sτ ))]] (3) where α is a parameter that determines the relative importance of the entropy term.
This objective has been considered in a number of works including Kappen (2005); Todorov (2009); Haarnoja et al. (2017; 2018a); Ziebart et al. (2008); Fox et al. (2015).
We define the action-value Qπ associated with eq. 3 as Qπ(st, at) ≡ rt + Eπ [∑∞ τ=t+1 γ τ−t(rτ + αH[π(·|sτ )]) ] (4)
(notice Qπ(st, at) does not include any entropy terms for the state st). Soft Q iteration
Q(st, at)← r(st, at, st+1) + γEp(st+1|st,at) [V (st+1)] (5) V (st)← Eπ [Q(st, at)] + αH[π(·|st)] = α log ∫ A exp( 1 α Q(st, at))da ≡ α logZ(st) (6)
where π(at|st) ∝ exp( 1αQ(st, at)) converges to the optimal policy with standard assumptions (Haarnoja et al., 2017).

heading: 3 COMPOSING POLICIES IN MAX-ENT REINFORCEMENT LEARNING
text: In this section we present two novel approaches for max-ent transfer learning. In section 4 we then outline a practical method for making use of these results.

heading: 3.1 MAX-ENT SUCCESSOR FEATURES AND GENERALIZED POLICY IMPROVEMENT
text: We introduce max-ent SF, which provide a practical method for computing the value of a maximum entropy policy under any convex combination of rewards. We then show the GPI theorem (Barreto et al., 2017) holds for maximum entropy policies.
We define the action-dependent SF to include the entropy of the policy, excluding the current state, analogous to the max-entropy definition of Qπ in (4):
ψπ(st, at) ≡ φt + Eπ [∑∞ τ=i+1 γ τ−t(φτ + α1 ·H[π(·|s)]) ] = φt + γEp(st+1|st,at) [Υ(st+1)]
(7)
where 1 is a vector of ones of the same dimensionality as φ and we define the state-dependent successor features as the expected ψπ in analogy with V π(s):
Υπ(s) ≡ Ea∼π(·|s) [ψπ(s, a)] + α1 ·H[π(·|s)]. (8) The max-entropy action-value of π for any convex combination of rewards w is then given by Qπw(s, a) = ψ
π(s, a) ·w. Max-ent SF allow us to estimate the action-value of previous policies on a new task. We show that, as in the deterministic case, there is a principled way to combine multiple policies using their action-values on task w.
Theorem 3.1 (Max-Ent Generalized Policy Improvement) Let π1, π2, ..., πn be n policies with α-max-ent action-value functions Q1, Q2, ..., Qn and value functions V 1, V 2, ..., V n. Define
π(a|s) ∝ exp ( 1 α maxiQ i(s, a) ) .
Then,
Qπ(s, a) ≥ max i Qi(s, a) for all s ∈ S and all a ∈ A, (9)
V π(s) ≥ max i V i(s) for all s ∈ S, (10)
where Qπ(s, a) and V π(s) are the α-max-ent action-value and value function respectively of π.
Proof: See appendix A.1. In our setup, we learn ψπi(s, a), the SFs of policies πi for each task in T , we define the max-ent GPI policy for task w ∈ T ′ as πGPIw (a|s) ∝ exp( 1α maxiQ πi w (s, a)) = exp( 1α maxiψ πi(s, a) ·w).

heading: 3.2 DIVERGENCE CORRECTION (DC)
text: Haarnoja et al. (2018a) introduced a simple approach to policy composition by estimating the actionvalue for the transfer task rb = bri + (1 − b)rj from the optimal action-values of the component tasks Qi and Qj
QOptb (s, a) ≡ bQi(s, a) + (1− b)Qj(s, a). (11)
When using Boltzmann policies defined byQ, the resulting policy, πOptb (a|s) ∝ exp( 1 αQ Opt b (s, a)), is the product distribution of the two component policies. We refer to πOptb as the compositionally “optimistic” (CO) policy, as it acts according to the optimistic assumption that the optimal returns of Qi and Qj will be, simultaneously, achievable1.
Both max-ent GPI we presented above, and CO can, in different ways, fail to transfer well in some situations (see fig. 1 for some examples in tabular case). Neither approach consistently performs optimally during transfer, even if all component terms are known exactly. We desire a solution for transfer that, in principle, can perform optimally.
Here we show, at the cost of learning a function conditional on the task weightings b, it is in principle possible to recover the optimal policy for the transfer tasks, without direct experience on those tasks, by correcting for the compositional optimism bias in QOptb . For simplicity, as in Haarnoja et al. (2018a), we restrict this to the case with only 2 tasks, but it can be extended to multiple tasks.
The correction term for CO uses a property noted, but not exploited in Haarnoja et al. (2018a). The bias inQOpt is related to the the discounted sum of Rényi divergences of the two component policies. Intuitively, if the two policies result in trajectories with low divergence between the policies in each state, the CO assumption that both policies can achieve good returns is approximately correct. When the divergences are large, the CO assumption is being overly optimistic and the correction term will be large.
Theorem 3.2 (DC Optimality) Let πi, πj be α max-ent optimal policies for tasks with rewards ri and rj with max-ent action-value functions Qi, Qj . Define C∞b (st, at) as the fixed point of
C (k+1) b (st, at) = −αγEp(st+1|st,at) [ log ∫ A πi(at+1|st+1) bπj(at+1|st+1)(1−b) exp(− 1αC (k) b (st+1, at+1))dat+1 ] 1Compositional optimism is not the same as optimism under uncertainty, often used in RL for exploration.
Given the conditions for Soft Q convergence, the max-ent optimal Q∗b(s, a) for rb = bri + (1− b)rj is
Q∗b(s, a) = bQ i(s, a) + (1− b)Qj(s, a)− C∞b (s, a) ∀s ∈ S, a ∈ A, b ∈ [0, 1].
Proof: See appendix A.2. We call this Divergence Correction (DC) as the quantity C∞b is related to the Rényi divergence between policies (see appendix A.2 for details). LearningC∞b does not require any additional information (in principle) than that required to learn policies πi and πj . Unlike with SF, it is not necessary to observe other task features while training the policies. On the other hand, unlike with GPI, which can be used to naturally combine any number of tasks with arbitrary weight vectors w, in order to apply DC one must estimate C∞b (s, a) for all values of b. so the complexity of learning C∞ increases significantly if more than 2 tasks are combined.
Supplementary Table 1 provides a comparison on the properties of the methods we consider here. We also compare with simply learning a conditional Q function Q(s, a|b) (CondQ) (e.g. Schaul et al., 2015; Andrychowicz et al., 2017). As with GPI, this requires observing the full set of task features φ, in order to compute rb for arbitrary b.
In this section we have introduced two new theoretical approaches to max-ent transfer composition: max-ent GPI and DC. We have shown how these are related to relevant prior methods. In the next section we address the question of how to practically learn and sample with these approaches in continuous action spaces.

heading: 4 ADAPTIVE IMPORTANCE SAMPLING FOR BOLTZMAN POLICIES ALGORITHM
text: The control of robotic systems with high-dimensional continuous action spaces is a promising use case for the ideas presented in this paper. Such control problems may allow for multiple solutions, and can exhibit exploitable compositional structure. Unfortunately, learning and sampling of general Boltzmann policies defined over continuous action spaces is challenging. While this can be mitigated by learning a parametric sampling distribution, during transfer we want to sample from the Boltzmann policy associated with a newly synthesized action-value function without having to learn such an approximation first. To address this issue we introduce Adaptive Importance Sampling for Boltzmann Policies (AISBP), a method which provides a practical solution to this challenge.
In the following we parametrise all functions with neural nets (denoting parameters by the subscript θ), including the soft action-value for reward i: QiθQ(s, a); the associated soft value function V i θV (s) and a proposal distribution qiθq (a|s), the role of which we explain below. We use an off-policy algorithm, so that experience generated by training on policy i can be used to improve policy j. This is especially important since our analysis requires the action-valueQi(s, a) to be known in all states. This is less likely to be the case for a on on-policy algorithm, that only updates Qi using trajectories generated by policy πi. During training experience generated by all tasks are stored in a replay buffer R, and mini-batches are sampled uniformly and used to update all function approximators. Soft Q iteration (see eq. 4) is used to learn Qi and V i. These updates are, in principle, straightforward using transitions sampled from the replay buffer.
Sampling from the Boltzmann policy defined by QiθQ , π i(a|s) ∝ exp 1αQ i θQ (s, a) is challenging as is estimating the partition function (the log of which is also the value, c.f. Eq. 6). One approach is to fit an expressible, tractable sampler, such as a stochastic neural network to approximate πi (e.g. Haarnoja et al., 2018a). This approach works well when learning a single policy. However, during transfer this may require learning a new sampler for each new value composition. AISBP instead uses importance sampling to sample π and estimate the partition function. The scalability of this approach is improved by using using a learned proposal distribution qθq (a|s), and by observing that modern architectures allow for efficient batch computation of a large number of importance samples. To facilitate transfer we restrict the parametric form of the proposals to mixtures of (truncated) Normal distributions. The well-known result that the product of Normal distributions can be computed in closed-form then allows us to construct effective compositional proposals during transfer.
More formally, for each policy in T we learn an action-value QiθQ(s, a), and value V i θV (s) network, and a proposal distribution qiθq (a|s) (we drop the task index i here when writing the losses for nota-
tional clarify, and write the losses for a single policy). The proposal distribution is a mixture of M truncated Normal distributions NT , truncated to the square a ∈ [−1, 1)n with diagonal covariances
qθq (a|s) = 1M ∑M m=1NT (a;µmθq (s), σ m θq (s),−1, 1) (12)
The proposal distribution is optimized by minimizing the forward KL divergence with the Boltzmann policy π(a|s) ∝ exp 1αQθQ(s, a). This KL is “zero avoiding” and over-estimates the support of π (Murphy, 2012) which is desirable for a proposal distribution (Gu et al., 2015),
L(θq) = ER [ Ea∼π(·|s)[log π(a|st)− log qθq (a|st)] ] (13)
where the expectation is over the replay buffer state density.
The inner expectation in the proposal loss itself requires sampling from π. We approximate his expectation by self-normalized importance sampling and use a target proposal distribution p(at|st) which is a mixture distribution consisting of the proposals for all policies along with a uniform distribution. For batchsize B and N proposal samples the estimator of the proposal loss is then
L(θq) ≈ − 1
B B∑ k=1 N∑ l=1 wkl log qθq (a|st); w′kl = 1 α (QθQ(sk, akl)) p(akl|sk) ; wkl = w′kl∑N m=1 w ′ km . (14)
The value function loss is defined as the L2 error on the Soft Q estimate of value
L(θV ) =ER
[ 1
2
( VθV (st)− α log ∫ A exp( 1 α QθQ(st, a))da )2] (15)
which is estimated using importance sampling to compute the integral.
L(θV ) ≈ 12B ∑B l=1 (VθV (sl)− α logZ) 2 ; Z = [ 1 N ∑N k=1 exp( 1αQθQ (sl,alk))
qθq (alk|sl)
] . (16)
This introduces bias due to the finite-sample approximation of the expectation inside the (concave) log. In practice we found this estimator sufficiently accurate, provided the proposal distribution was close to π. We also use importance sampling to sample from π while acting.
The action-value loss is just the L2 norm with the Soft Q target: L(θQ) =ER [ 1
2 (QθQ(st, at)− (r(st, at, st+1) + γVθ′V (st+1)))
2 ] . (17)
To improve stability we employ target networks for the value VθV ′ and proposal qθ′q networks (Mnih et al., 2015; Lillicrap et al., 2015) We also parameterize Q as an advantage QθQ(s, a) = VθV (s) + AθA(s, a) (Baird, 1994; Wang et al., 2015; Harmon et al., 1995) which is more stable when the advantage is small compared with the value. The full algorithm is give in Algorithm Box 1 and more details are provided in appendix C.

heading: 4.1 IMPORTANCE SAMPLED MAX-ENT GPI
text: The same importance sampling approach can also be used to estimate max-ent SF. Max-ent GPI requires us to learn the expected (maximum entropy) features ψi for each policy πi, in order to estimate its (entropic) value under a new convex combination task w. This requires that experience tuple in the replay contain the full feature vector φ, rather than just the reward for the policy which generated the experience ri. Given this information ψθψ and ΥθΥ can be learned with analogous updates to V and Q, which again requires importance sampling to estimate Υ.
As with VθV , we use a target network for Υθ′Υ and advantage parametrization. We found that, because these updates when using experience shared between tasks is far off-policy, it is necessary to have a longer target update period than for V . Full details are of the losses and samplers are in appendix C.
Algorithm 1 AISBP training algorithm Initialize proposal network θq , value network parameters θV and action-value network parameters θQ and replay R while training do . in parallel on each actor
Obtain parameters θ from learner Sample task i ∼ T Roll out episode using qiθq to importance sample πi(a|s) ∝ exp 1 αQ i θQ (s, a)
Add experience to replay R end while while training do . in parallel on the learner
Sample SARS tuple from R Improve L(θq), L(θV ), L(θQ) Improve additional losses for transfer L(θΥ), L(θψ), L(θC), L(θVb) L(θQb), if target update period then
Update target network parameters θV ′ ← θV , θq′ ← θq , θΥ′ ← θΥ, θV ′b ← θVb end if
end while

heading: 4.2 DIVERGENCE CORRECTION
text: All that is required for transfer using compositional optimism (eq. 11, Haarnoja et al. (2018a)) is the max-ent action values of each task, so no additional training is required beyond the base policies. In section 3.2 we have shown that if we can learn the fixed point of C∞b (s, a) we can correct this compositional optimism and recover the optimal action-value Q∗b(s, a).
We exploit the recursive relationship in C∞b (s, a) to fit a neural net CθC (s, a, b) with a TD(0) estimator. This requires learning a conditional estimator for any value of b, so as to support arbitrary task combinations. Fortunately, since C∞b depends only on the policies and transition function it is possible to learn an estimator C∞b for different values of b by sampling b during each update. As before, we use target networks and an advantage parametrization for CθC (s, a, b)
We learn C∞b as CθC (s, a, b), for each pair of policies πi, πj resulting in the loss L(θC) = Es∼R,b∼U(0,1)[ 12 (CθC (s, a, b) + αγEp(s′|s,a)[log ∫ A exp(b log πi(a ′|s′)+ (18)
(1− b)πj(a′|s′)− 1
α CθC′ (s
′, a′, b))da′])2].
As with other integrals of the action space, we approximate this loss using importance sampling to estimate the integral. Note that, unlike GPI and CondQ (next section), learning C∞b does not require observing φ while training.
We also considered a heuristic approach where we learned C only for b = 12 (this is typically approximately the largest divergence). This avoids the complexity of a conditional estimator and we estimate C∞b as Ĉ ∞ b (s, a) ∼ 4b(1 − b)C∞1/2(s, a). This heuristic, we denote DC-Cheap, can be motivated by considering Gaussian policies with similar variance (see appendix D) The max-ent GPI bound can be used to correct for over-estimates of the heuristic C∞b , Q DC−Cheap+GPI(s, a) = max(QOPT (s, a)− Ĉ∞b (s, a), QGPI(s, a)).

heading: 4.3 COND Q
text: As a baseline, we directly learn a conditionalQ function using a similar approach to DC of sampling b each update Q(s, a, b) (Schaul et al., 2015). This, like GPI but unlike DC, requires observing φ during training so the reward on task b can be estimated. We provide the full details in appendix C.

heading: 4.4 SAMPLING COMPOSITIONAL POLICIES
text: During transfer we would like to be able to sample from the Boltzmann policy defined by our estimate of the transfer action-valueQb (the estimate is computed using the methods we enumerated
above) without having to, offline, learn a new proposal or sampling distribution first (which is the approach employed by Haarnoja et al. (2018a)).
As outlined earlier, we chose the proposal distributions so that the product of proposals is tractable, meaning we can sample from qijb (a|s) ∝ (qiθq (a|s))
b(qjθ(a|s))(1−b). This is a good proposal distribution when the CO bias is low, since QOptb defines a Boltzmann policy which is the product of the base policies2 However, when C∞b (s, a) is large, meaning the CO bias is large, q
ij may not be a good proposal, as we show in the experiments. In this case none of the existing proposal distributions may be a good fit. Therefore we sample from a mixture distribution of all policies, all policy products and the uniform distribution.
pb(a|s) ≡ 14 (q i θq (a|s) + qjθq (a|s) + q ij b (a|s) + 1 VA ) (19)
where VA is the volume of the action space. Empirically, we find this is sufficient to result in good performance during transfer. The algorithm for transfer is given in supplementary algorithm 2.

heading: 5 EXPERIMENTS

heading: 5.1 DISCRETE, TABULAR ENVIRONMENT
text: We first consider some illustrative tabular cases of compositional transfer. These highlight situations in which GPI and CO transfer can perform poorly (Figure 1). As expected, we find that GPI performs well when the optimal transfer policy is close to one of the existing policies; CO performs well when both subtask policies are compatible. The task we refer to as “tricky” is illustrative of in which the optimal policy for the transfer task does not resemble either existing policy: In the grid world nonoverlapping rewards for each task are provided in one corner of the grid world, while lower value overlapping rewards are provided in the other corner (cf. Fig. 1). As a consequence both GPI and CO perform poorly while DC performs well in all cases.

heading: 5.2 CONTINUOUS ACTION SPACES
text: We next compare the different approaches in more challenging continuous control tasks. We train max-ent policies to solve individual tasks using the importance sampling approach from section 4 and then assess transfer on convex combinations of the rewards. All approaches use the same experience and proposal distribution.
Figure 2 examines the transfer policies in detail in a simple point-mass task and shows how the estimated C∞b corrects the CO Q Opt and dramatically changes the policy.
We then examine conceptually similar tasks in more difficult domains: a 5 DOF planar manipulator reaching task (figure 3), 3 DOF jumping ball and 8 DOF ant (figure 4). We see that DC recovers a qualitatively better policy in all cases. The performance of GPI depends noticeably on the choice of α. DC-Cheap, which is a simpler heuristic, performs almost as well as DC in the tasks we consider except for the point mass task. When bounded by GPI (DC-Cheap+GPI) it performs well for the point mass task as well, suggesting simple approximations of C∞b may be sufficient in some cases. 3
We focussed on “tricky” tasks as they are challenging form of transfer. In general, we would expect DC to perform well in most situations where OC performs well, since in this case the correction term C∞b that DC must learn is inconsequential (OC is equivalent to assuming C ∞ b = 0). Supplementary figure 5 demonstrates on a task with non-composible solutions (i.e. C∞b is large and potentially challenging to learn), DC continues to perform as well as GPI, slightly better than CondQ, and as expected, OC performs poorly.

heading: 6 DISCUSSION
text: We have presented two approaches to transfer learning via convex combinations of rewards in the maximum entropy framework: max-ent GPI and DC. We have shown that, under standard assump-
2πOptb (a|s) ∝ exp 1 α QOpt(s, a) = exp( 1 α (Q1(s, a) +Q2(s, a)) = π1(a|s)π2(a|s). 3 We provide videos of the more interesting tasks at https://tinyurl.com/yaplfwaq.
tions, the max-ent GPI policy performs at least as well as its component policies, and that DC recovers the optimal transfer policy. Todorov (2009) and (Saxe et al., 2017; van Niekerk et al., 2018) previously considered optimal composition of max-ent policies. However, these approaches
require stronger assumptions than max-ent SF or DC, namely that reward states are absorbing and that the joint reward is restricted to the softmax of the component rewards (soft OR). By contrast, DC does not restrict the class of MDPs and learns how compatible policies are, allowing approximate recovery of optimal transfer policies both when the component rewards are jointly achievable (AND), and when only one sub-goal can be achieved (OR).
We have compared our methods with conditional action-value functions (CondQ) (Schaul et al., 2015, e.g.) and optimistic policy combination (Haarnoja et al., 2018a). Further, we have presented AISBP, a practical algorithm for training DC and max-ent GPI models in continuous action spaces using adaptive importance sampling. We have compared these approaches, along with heuristic approximations of DC, and demonstrated that DC recovers an approximately optimal policy during transfer across a variety of high-dimensional control tasks. Empirically we have found CondQ may be harder to learn than DC, and it requires additional observation of φ during training.

heading: A PROOFS
text: A.1 MAX-ENT GENERALIZED POLICY IMPROVEMENT
Theorem 3.1 (Max-Ent Generalized Policy Improvement) Let π1, π2, ..., πn be n policies with α-max-ent action-value functions Q1, Q2, ..., Qn and value functions V 1, V 2, ..., V n. Define
π(a|s) ∝ exp ( 1 α maxiQ i(s, a) ) .
Then,
Qπ(s, a) ≥ max i Qi(s, a) for all s ∈ S and all a ∈ A, (9)
V π(s) ≥ max i V i(s) for all s ∈ S, (10)
where Qπ(s, a) and V π(s) are the α-max-ent action-value and value function respectively of π.
For brevity we denote Qmax ≡ maxiQi. Define the soft Bellman operator associated with policy π as
T πQ(s, a) ≡ r(s, a, s′) + γEp(s′|s,a) [ αH[π(·|s′)] + Ea′∼π(·|s′) [Q(s′, a′)] ] .
Haarnoja et al. (2018b) have pointed out that the soft Bellman operator T π corresponds to a conventional, “hard”, Bellman operator defined over the same MDP but with reward rπ(s, a, s′) = r(s, a, s′) + γαEp(s′|s,a) [H[π(·|s′)]]. Thus, as long as r(s, a, s′) and H[π(·|s′)] are bounded, T π is a contraction with Qπ as its fixed point. Appplying T π to Qmax(s, a) we have: T πQmax(s, a) = r(s, a, s′) + γEs′∼p(·|s,a),a′∼π(·|s′) [−α log π(a′|s′) +Qmax(s′, a′)]
= r(s, a, s′) + γEs′∼p(·|s,a),a′∼π(·|s′) [ −α log exp(α −1Qmax(s′, a′))
Zπ(s′) +Qmax(s′, a′) ] = r(s, a, s′) + γEs′∼p(·|s,a) [α logZπ(s′)] .
Similarly, if we apply T πi , the soft Bellman operator induced by policy πi, toQmax(s, a), we obtain: T πiQmax(s, a) = r(s, a, s′) + γEs′∼p(·|s,a),a′∼πi(·|s′) [−α log πi(a ′|s′) +Qmax(s′, a′)] .
We now note that the Kullback-Leibler divergence between πi and π can be written as
DKL(πi(·|s)‖π(·|s)) = Ea∼πi(·|s) [log πi(a|s)− log π(a|s)] = Ea∼πi(·|s) [ log πi(a|s)− 1
α Qmax(s, a) + logZπ(s)
] .
The quantity above, which is always nonnegative, will be useful in the subsequent derivations. Next we write
T πQmax(s, a)− T πiQmax(s, a) = γEs′∼p(·|s,a) [ α logZπ(s′)− Ea′∼πi(·|s′)[−α log πi(a ′|s′) +Qmax(s′, a′)] ]
= γEs′∼p(·|s,a) [ Ea′∼πi(·|s′)[α logZ π(s′) + α log πi(a ′|s′)−Qmax(s′, a′)] ] = γEs′∼p(·|s,a) [αDKL(πi(·|s′)‖π(·|s′))] ≥ 0. (20)
From (20) we have that
T πQmax(s, a) ≥ T πiQmax(s, a) ≥ T πiQi(s, a) = Qi(s, a) for all i. Using the contraction and monotonicity of the soft Bellman operator T π we have
Qπ(s, a) = lim k→∞ (T π)kQmax(s, a) ≥ Qi(s, a) for all i.
We have just showed (9). In order to show (10), we note that
V π(s) ≡ αH[π(·|s)] + Ea∼π [Qπ(s, a)] ≥ αH[π(·|s)] + Ea∼π [Qmax(s, a)] = α logZπ(s). (21)
Similarly, we have, for all i, V i(s) = Ea∼πi(·|s) [ Qi(s, a)− α log πi(a|s) ] ≤ Ea∼πi(·|s) [Q
max(s, a)− α log πi(a|s)] = α logZπ(s)− αDKL(πi(·|s)‖π(·|s)) ≤ α logZπ(s). (22)
The bound (10) follows from (21) and (22).
A.2 DC PROOF
Theorem 3.2 (DC Optimality) Let πi, πj be α max-ent optimal policies for tasks with rewards ri and rj with max-ent action-value functions Qi, Qj . Define C∞b (st, at) as the fixed point of
C (k+1) b (st, at) = −αγEp(st+1|st,at) [ log ∫ A πi(at+1|st+1) bπj(at+1|st+1)(1−b) exp(− 1αC (k) b (st+1, at+1))dat+1 ] Given the conditions for Soft Q convergence, the max-ent optimal Q∗b(s, a) for rb = bri + (1− b)rj is
Q∗b(s, a) = bQ i(s, a) + (1− b)Qj(s, a)− C∞b (s, a) ∀s ∈ S, a ∈ A, b ∈ [0, 1].
We follow a similar approach to Haarnoja et al. (2018a) but without making approximations and generalizing to all convex combinations.
First note that since πi and πj are optimal then πi(a|s) = exp( 1α (Q i(s, a)− V i(s))). For brevity we use s and s′ notation rather than writing the time index.
Define
Q (0) b (s, a) ≡ bQ i(s, a) + (1− b)Qj(s, a) (23) C(0)(s, a) ≡ 0 (24)
and consider soft Q-iteration on rb starting from Q (0) b . We prove, inductively, that at each iteration Q (k+1) b = bQ
i(s, a) + (1− b)Qj(s, a)− C(k+1)(s, a). This is true by definition for k = 0.
Q (k+1) b (s, a) = rb(s, a) + γαEp(s′|s,a)
[ log ∫ A exp 1 α Q (k) b (s ′, a′)da′ ]
(25)
= rb(s, a)+ (26) γαEp(s′|s,a) [ log ∫ A exp( 1 α (bQi(s′, a′) + (1− b)Qj(s′, a′)− C(k)(s′, a′)))da′ ] = rb(s, a)+ (27)
Ep(s′|s,a) [ bV i(s′) + (1− b)V j(s′) + α log ∫ A exp(b log πi(a ′|s′) + (1− b) log πj(a′|s′)− 1 α C(k)(s′, a′))da′ ] = bQi(s, a) + (1− b)Qj(s, a)+ (28)
αγEp(s′|s,a) [ log ∫ A exp(b log π1(a ′|s′) + (1− b) log π2(a′|s′)− 1 α C(k)(s′, a′))da′ ] = bQi(s, a) + (1− b)Qi(s, a)− C(k+1)b (s, a). (29)
Since soft Q-iteration converges to the α max-ent optimal soft Q then equation 31 holds at the limit.
One can get an intuition for C∞b (s, a) by noting that
C (1) b (s, a) = γαEp(s′|s,a) [(1− b)Db (π1(·|s)‖π2(·|s))] (30)
where Db is the Rényi divergence of order b. C∞b (s, a) can be seen as the discount sum of divergences, weighted by the unnormalized product distribution π1(a|s)bπ2(a|s)1−b.
A.3 N POLICIES
It is possible to extend Theorem 3.2 to the case with N policies in a straightforward way.
Theorem A.1 (Multi-policy DC Optimality) Let π1, π2, ..., πN be α max-ent optimal policies for tasks with rewards r1, r2, ..., rN with max-ent action-value functions Q1, Q2, ..., QN .
Define C∞w (st, at) as the fixed point of
C (k+1) w (st, at) = −αγEp(st+1|st,at) [ log ∫ A (∏N i=1 πi(at+1|st+1)wi ) exp(− 1αC (k) w (st+1, at+1))dat+1 ] Given the conditions for Soft Q convergence, the max-ent optimal Q∗w(s, a) for and convex combination of rewards rw = ∑N i=1 riwi is
Q∗w(s, a) = ∑N i=1 wiQ i(s, a)− C∞w (s, a)
∀s ∈ S, a ∈ A,w ∈ {w| N∑ i=1 wi = 1 and wi ≥ 0}
Note that wi refers to component i of the vector wi.
The proof is very similar to the two reward case above.
Define
Q(0)w ≡ N∑ i=1 wiQ i(s, a) (31)
C(0)w ≡ 0 (32)
and again consider soft Q-iteration on rw. We prove by induction that at each iteration
Q(k+1)w (s, a) = N∑ i=1 wiQ i(s, a)− C(k+1)w (s, a) (33)
Again, this is true by definition for k = 0. Now we consider a step of Soft Q iteration
Q(k+1)w = rw(s, a) + γαEp(s′|s,a) [ log ∫ A exp 1 α Q(k)w (s ′, a′)da′ ]
(34)
= rw(s, a) + γαEp(s′|s,a)
[ log ∫ A exp 1 α ( N∑ i=1 wiQ i(s′, a′)− C(k)w (s, a) ) da′ ] (35)
= rw(s, a) + γEp(s′|s,a) [ N∑ i=1 wiV i(s′) + α log ∫ A exp ( N∑ i=1 wi log πi(a ′|s′)− 1 α C(k)w (s ′, a′) ) da′ ] (36)
= N∑ i=1 wiQ i(s, a) + αγEp(s′|s,a)
[ log ∫ A exp( N∑ i=1 wi log πi(a ′|s′)− 1 α C(k)(s′, a′))da′ ] (37)
= N∑ i=1 wiQ i(s, a)− C(k+1)w (s, a) (38)
Since soft Q-iteration converges to the α max-ent optimal soft Q then Q∗w(s, a) =∑N i=1 wiQ
i(s, a)− C(k+1)w (s, a) for all s ∈ S, a ∈ A. Note that, in practise, estimating C∞w may be more challenging for larger N . For compositions of many policies, GPI may be more practical.

heading: B THEORETICAL PROPERTIES OF THE COMPOSITION METHODS

heading: C ALGORITHM DETAILS
text: C.1 TRANSFER ALGORITHM
Algorithm 2 AISBP transfer algorithm Load trained parameters θQ, θq , θsfQ, θC , θQb . Accept transfer task parameter b, transfer method ∈ CO, GPI, DC, CondQ. while testing do
Importance sample transfer policy πb(a|s) ∝ exp 1αQ method(s, a) with mixture proposal
pb(a|s)θq . end while
C.2 ALL LOSSES AND ESTIMATORS
We use neural networks to parametrize all quantities. For each policy we learn an action-value QθQ(s, a), value VθV (s) and proposal distribution qθq (a|s). We use target networks for the proposal distribution qθ′q (a|s) and value Vθ′V (s).
Here we enumerate all of the losses and their estimators. We use temporal difference (TD(0)) learning for all the RL losses, so all losses are valid off-policy. We use a replay buffer R and learn by sampling minibatches of SARS tuples of size B, we index over the batch dimension with l and use s′l to denote the state following sl, so the tuple consists of (sl, al, rl, s ′ l). For importance sampled estimators we sample N actions for each state sl and use alk to denote sample k for state l.
We learn a set of n policies, one for each task in T indexed by i. However, we write the losses for a single policy and drop i for notational simplicity.
C.2.1 PROPOSAL LOSS
The proposal loss minimizes the KL divergence between the Boltzmann distribution π(a|s) ∝ exp( 12Q(s, a)) and the proposal distribution.
L(θq) = ER [ Ea∼π(·|s)[log π(a|st)− log qθq (a|st)] ] (39)
As described in the text, this loss is estimated using importance sampling with a mixture distribution p(a|s) containing equally weighted components consisting of the target proposal distribution qθ′q (a|s) for all policies and the uniform distribution.
p(a|s) = 1 n+ 1
( 1
V A + n∑ i=1 qiθ′q (a|s)
) (40)
where V A is the volume of the action space (which is always bounded in our case).
The proposal loss is estimated using self-normalized importance sampling
L(θq) ≈ − 1
B B∑ k=1 N∑ l=1 wkl log qθq (a|st), (41)
w′kl = 1 α (QθQ(sk, akl))
p(akl|sk) ; wkl = wkl′∑N m=1 w ′ km . (42)
C.2.2 VALUE LOSS
The soft value loss is L(θV ) =ER [ 1
2 (VθV (st)− α log ∫ A exp( 1 α QθQ(st, a))da) 2 ] (43)
We estimate this using importance sampling with the proposal distribution qθq (a|s) which is trying to fit the policy π.
L(θV ) ≈ 1
2B B∑ l=1 (VθV (sl)− α logZ) 2 (44)
Z =
[ 1
N N∑ k=1 exp( 1αQθQ(sl, alk)) qθq (alk|sl)
] (45)
C.2.3 ACTION-VALUE LOSS
The TD(0) loss for QθQ is
L(θQ) =ER [ 1
2 (QθQ(st, at)− (r(st, at, st+1) + γVθ′V (st+1)))
2 ] (46)
This does not require importance sampling to estimate and can be straightforwardly estimated as
L(θQ) ≈ 1
2B B∑ l=1 (QθQ(sl, al)− (rl + γVθ′V (s ′)))2. (47)
The action-value is parametrized as an advantage function QθQ(s, a) = Vθ′V (s) +AθA(s, a).
C.2.4 STATE DEPENDENT SUCCESSOR FEATURES LOSS
To facilitate max-ent GPI we learn successor features for each policy, both state-action dependent features ψθψ (s, a) and state-dependent ΥθΥ(s). As with value, we use a target network for the state-dependent features Υθ′Υ(s)
L(θΥ) =ER [ 1
2 (ΥθΥ(st)− Eat∼π(at|st)[ψθψ (st, at) + α1(−QθQ(st, at) + α logZ(st))])
2 ] This loss is estimated using self-normalized importance sampling with proposal qθq
L(θΥ) ≈ 1
2B B∑ l=1 N∑ k=1 wlk [ (ψiθψ (sl, alk)−Q i θQ(sl, alk) + α logZ(sl)) 2 ] , (48)
wlk ∝ exp( 1αQ i(sl, alk))
qiθq (alk|sl) . (49)
We use the importance sampled estimate of Z from eq 47, rather than the value network which may be lagging the true partition function. As with the value estimate, we use self-normalized importance sampling to avoid the importance weights depending on α logZ(sl) (this introduces a bias, but in practise appears to work well).
C.2.5 STATE-ACTION DEPENDENT SUCCESSOR FEATURES LOSS
The state-action dependent successor feature loss is L(θψ) =ER [ 1
2 (ψθQ(st, at)− (φ(st, at, st+1) + γΥθ′Υ(st+1)))
2 ] . (50)
for which we use the following estimator
L(θψi) ≈ 1
2B B∑ l=1 (ψiθψ (sl, al)− (φl + γΥθ′Υ(s ′ l))) 2. (51)
ψθψ is parametrized as a “psi-vantage” ψθψ (s, a) = Υθ′Υ(s) +ψ A θA (s, a).
C.2.6 DC CORRECTION
We learn the divergence correction for each pair of policies πi(a|s), π(a|s). As described in the text, in order to learn CθC (s, a, b) for all b ∈ [0, 1], we sample b. We also use a target network Cθ′C (s, a, b). The loss is then
L(θC) = Es∼R,b∼U(0,1)[ 12 (CθC (s, a, b) + αγEp(s′|s,a)[log ∫ A exp(b log πi(a ′|s′)+ (52)
(1− b)πj(a′|s′)− 1
α CθC′ (s
′, a′, b))da′])2].
This loss is challenging to estimate, due to the dependence on two policies. We importance sample using a mixture of all proposal distributions uniform p(a|s) (equation 40). We denote the samples of b ∼ U(0, 1) for each batch entry bl. The importance sampled estimator is then
L(θC) ≈ 1
N B∑ l=1 ( CθC (sl, al, bl)− αγ log [ 1 N N∑ k=1 Ctargetθ′C (s′l, a ′ lk, bl) p(alk′ |sl) ])2 , (53)
Ctargetθ′C (s′l, a ′ lk, bl) ≡ exp(
1 α (blQ i θQ(s ′ l, a ′ lk) + (1− bl)Q j θQ (s′l, a ′ lk)− Cθ′C (s ′ l, a ′ lk, bl)). (54)
We parametrized CθC as an advantage function CθC (s, a, b) = C A θCA (s, a, b) + CBθCB (s, b) with an additional loss to constrain this parametrization
L(θB) = Ea∼q(·|s),s∼R [ 1
2 (CAθCA (s, a, b)) 2
] (55)
which can be straightforwardly estimated by sampling from q
L(θB) ≈ 1
2NB B∑ l=1 N∑ k=1 (CAθCA (sl, alk, bl)) 2 (56)
CondQ We also consider, as a control, learning the action-value function conditional on b directly (Schaul et al., 2015), in a similar way to the DC correction. We learn both a conditional value VθVb (s, b) and QθQb (s, a, b), again by sampling b uniformly each update.
L(θVb) = ER,b∼U(0,1) [ 1
2 (VθVb (s, b)− α log
∫ exp( 1
α QθQb (s, a, b)))
2 ] , (57)
LθQ = ER,b∼U(0,1) [ 1
2 (QθQb (s, a, b)− (rb + γVθVb (s
′, b)))2 ] , (58)
where computing rb for arbitrary b requires φ to have been observed.
We estimate Cond-Q with the same importance samples as C from p(a|s) and again sample b ∼ U(0, 1) for each entry in the batch. We use target networks for Vθ′V (s, b) and parametrize QθQ(s, a, b) = Vθ′V (s, b) +AθA(s, a, b).
The conditional value estimator is
L(θV ) ≈ 1
2B B∑ l=1 ( VθVb (sl, bl)− α log 1 N N∑ k=1 exp( 1αQθQb (sl, alk, bl) p(alk|sl) )2 (59)
and action-value estimator is
L(θQ) ≈ 1
2B B∑ l=1 ( QθQb (sl, al, bl)− (rb + γVθ′Vb (s ′ l, bl)) )2 (60)

heading: D JUSTIFICATION FOR THE DC-CHEAP HEURISTIC
text: We wish to estimate C∞b (s, a) (defined in Theorem A.1) while avoiding learning a conditional function of b. We make two (substantial) assumptions to arrive at this approximation.
Firstly, we assume policies πi(a|s), πj(a|s) are Gaussian πi(a|s) = exp ( − (a− µi(s)) 2
2σ(s)2
) (61)
and the variance σ(s) is the same for both policies given a state (it may vary across states).
Secondly, we assume C(k)b (s, a) = C (k) b (s) is independent of action. This is approximately correct when nearby states have similar Rényi divergences between policies.
We make use of a result by Gil et al. (2013) that states that the Rényi divergence of order b for two Gaussians of the same variance is
Db (N (µ1, σ)‖N (µ2, σ)) = 1
2
b(µ1 − µ2)2
σ2 . (62)
We first define Gb(s) ≡ (1− b)Db (πi(·|s)‖πj(·|s)) = − log ∫ πi(a|s)bπj(a|s)(1−b)da. (63)
From equation 61
Gb(s) = 4b(1− b)G 1 2 (s). (64)
Given these assumptions we show inductively that C(k)b (s, a) = 4b(1− b)C (k) 1/2(s, a) ∀k, b ∈ [0, 1].
Since C(0)b (s, a) = 0 ∀b ∈ [0, 1], a ∈ A, s ∈ S this is true for k = 0. We show it holds inductively
C (k+1) b (s, a) = −αγEp(s′|s,a)
[ log ∫ A πi(a ′|s′)bπj(a′|s′)(1−b) exp(− 1 α C (k) b (s ′, a′))da′ ] (65)
= γEp(s′|s,a) [ αGb(s ′) + C (k) b (s ′) ]
(66)
= 4(1− b)bC(k+1)1 2 (s, a). (67)
Obviously these assumptions are not justified. However, note that we estimate the true divergence for C∞1/2, i.e. without any assumptions of Gaussian policies and this heuristic is used to estimate C∞b from C ∞ 1/2. In practise, we find this heuristic works in many situations where the policies have similar variance, particulary when bounded by GPI.

heading: E ADDITIONAL EXPERIMENT

heading: F EXPERIMENT DETAILS
text: All control tasks were simulated using the MuJoCo physics simulator and constructed using the DM control suite (Tassa et al., 2018) which uses the MuJoCo physics simulator (Todorov et al., 2012).
The point mass was velocity controlled, all other tasks were torque controlled. The planar manipulator task was based off the planar manipulator in the DM control suite. The reward in all tasks was sparse as described in the main text.
During training for all tasks we start states from the randomly sampled positions and orientations. For the point mass, jumping ball and ant we evaluated transfer starting from the center (in the walker environments, the starting orientation was randomly sampled during transfer, the point mass does not have an orientation). For the planar manipulator transfer was tested from same random distribution as in training. For all tasks were learned infinite time horizon policies.
Transfer is made challenging by the need for good exploration. That was not the focus on this work. We aided exploration in several ways: during training we acted according to a higher-temperature policy αe = 2α. We also sampled actions uniformly in an -greedy fashion with = 0.1 and added Gaussian exploration noise during training. This was sufficient to explore the state space for most
tasks. For the planar manipulator and the jumping ball, we found it necessary to induce behavior tasks by learning tasks for reaching the blue target. This behavior policy was, of course, only used for experience and not during transfer.
Below we list the hyper-parameters and networks use for all experiment. The discount γ and α were the only sensitive parameters that we needed to vary between tasks to adjust for the differing magnitudes of returns and sensitivity of the action space between bodies. If α is too small then the policies often only find one solution and all transfer approaches behave similarly, while for large α the resulting policies are too stochastic and do not perform well.
The state vector was preprocessed by a linear projection of 3× its dimension and then a tanh nonlinearity. All action-state networks (Q, ψ, C) consisted of 3 hidden layers with elu non-linearities (Clevert et al., 2015), with both action and preprocessed state projected by linear layers to be of the same dimensionality and used for input the first layer. All value networks and proposal networks consisted of 2 layers with elu non-linearities. The number of neurons in each layer was varied between environments, but was kept the same in all networks and layers (we did not sweep over this parameter, but choose a reasonable number based on our prior on the complexity of the task).
Below we list the per task hyper-parameters

Review summaries:
The authors introduce Divergence Correction (DC) for the problem of transfer learning by composing policies. There approach builds on GPI with a maximum entropy objective. They also prove that DC solves for the max-entropy optimal interpolation between two policies and derive a practical approximation for this algorithm. They provide experimental results in a gridworld problem and study their approximate algorithm in two continuous control problems.

This paper proposes using Divergence Correction to compose max ent policies. Based on successor features, this method corrects the optimistic bias of Haarnoja 2018. The motivation for composing policies is sound. This paper addresses the problem statement where policies must accomplish different linear combinations of different reward functions. This method does not require observation the reward weights.