Title: COMBINING MODEL-BASED AND MODEL-FREE RL VIA MULTI-STEP CONTROL VARIATES

Abstract: Model-free deep reinforcement learning algorithms are able to successfully solve a wide range of continuous control tasks, but typically require many on-policy samples to achieve good performance. Model-based RL algorithms are sampleefficient on the other hand, while learning accurate global models of complex dynamic environments has turned out to be tricky in practice, which leads to the unsatisfactory performance of the learned policies. In this work, we combine the sample-efficiency of model-based algorithms and the accuracy of model-free algorithms. We leverage multi-step neural network based predictive models by embedding real trajectories into imaginary rollouts of the model, and use the imaginary cumulative rewards as control variates for model-free algorithms. In this way, we achieved the strengths of both sides and derived an estimator which is not only sample-efficient, but also unbiased and of very low variance. We present our evaluation on the MuJoCo and OpenAI Gym benchmarks.

1. INTRODUCTION:

Reinforcement learning algorithms are usually divided as two broad classes: model-based algorithms which try to learn forward dynamics models first (and derive a planning-based policy from it), and model-free algorithms which do not explicitly train any dynamics models but instead directly learn a policy. Model-free algorithms have been proven effective in learning sophisticated controllers in a wide range of applications, ranging from playing video games from raw pixels Mnih et al. (2013) ; Oh et al. (2016) to solving locomotion and robotic tasks Schulman et al. (2015) . Modelbased algorithms on the other hand are widely used in robot learning and continuous control tasks Deisenroth & Rasmussen (2011) .

Both approaches have their strengths as well as limitations. Although model-free RL algorithms can achieve very good performance, they usually require ten thousands of samples for each iteration (Schulman et al., 2015) , which is a very high sample intensity. A fundamental advantage of modelbased RL is that knowledge about the environment can be acquired in an unsupervised setting, even in trajectories where no rewards are available. While model-based algorithms are usually considered as more sample-efficient (Deisenroth et al., 2013) , currently the most successful modelbased algorithms rely on simple functional approximators Lioutikov et al. (2014) to learn local models over-fitting a few samples, usually one mini-batch. Thus ideas that rely on a global neural network model for planning often do not perform very well (Mishra et al., 2017; Gu et al., 2016b) due to the bias of the learned models and the limitations of the planning algorithms, e.g. MCTS (Chaslot, 2010) . As a result, although theoretically global models should be more sample efficient because models can be trained on off-policy data, they are seldom used in real problems.

In this paper, we attempt to combine the benefits of model-based and model-free reinforcement learning and reduce the drawbacks on both sides. Our algorithm can generally be viewed as using a multi-step forward model as a control variate (MSCV) for model-free RL algorithms. It works as follows: given a batch of on-policy trajectory samples, instead of directly performing on-policy gradient update based on the return, we first embed a portion of the trajectory into the dynamics model, yielding an imaginary trajectory which is usually very close to the real trajectory thus we call it a trajectory "embedding" to the dynamics model. From the imaginary trajectory, we can get low variance gradients via the reparameterization trick Kingma & Welling (2013) and backpropagationthrough-time (BPTT) Werbos (1990) . However, such gradients are in general biased. The funda-mental reason for this is that in real circumstances, the global dynamics is highly complicated while the model is usually biased. Thus afterwards, we use the return from the real trajectory to correct the model-based gradient. This way, we effectively combine the strength of both model-based and model-free RL while avoiding their shortcomings. We evaluated MSCV on OpenAI Gym Brockman et al. (2016) and MuJoCo Todorov et al. (2012) benchmarks.

The contributions of our work are the following: (1) We show how to train efficient multi-step value prediction models by utilizing techniques such as replay memory and importance sampling on offpolicy data. (2) We demonstrate that the model-based path derivative methods can be combined with model-free RL algorithms to produce low variance, unbiased, and sample efficient policy gradient estimates. (3) We show that techniques such as latent space trajectory embedding and dynamic unfolding can significantly boost the performance of the model based control variates.

2. RELATED WORKS:

Model-based and model-free RL are two broad classes of reinforcement learning algorithms with different weakness and strength. Thus no doubt there are many works trying to combine the strength of both sides and avoid their weakness. (Nagabandi et al., 2017) tries to train a neural network based global model and use model predictive control to initialize the policy for model free fine tuning. In this way, their algorithm can be significantly more sample-efficient than purely model free approaches. (Chebotar et al., 2017) combines local model based algorithms such as LQR to a model-free framework called path integral policy improvement. Local models such as LQR are sample efficient but they need to refit simple local models for each iteration, so they cannot work with high dimensional state spaces such as images.

QProp (Gu et al., 2016a) proposed to use a critic trained over the amortized deterministic policy using DDPG (Lillicrap et al., 2015) as a control variate. While this has brought good performance gain in terms of sample complexity, our new control variate provides several attractive properties over it. First, the Q function Q(s, a) fitted off-policy using DDPG is usually very different from the real on-policy expected return Q µ (s, a) (because the trajectory in the replay buffer are generated by different policies, they have different value estimates). As we've seen in our experiments, sometimes the gap between these two values is of orders of magnitude. However, in our algorithm, since their is no inconsistency of the objectives, our control variate is usually much more accurate than QProp. See e.g. Section 5. Second, Our control variate takes into consideration not only the current policy step, but also several future steps. This approach have brought two main gains to our algorithm: (1) our control variate can be fitted much closer to the actual return, resulting in less variance. (2) we can use backpropagation through time to optimize multiple steps in the imaginary trajectory jointly, this makes our algorithm be able to look further into the future. This is very useful when the reward cannot be observed very often.

Our algorithm is also closely related to stochastic value gradient (Heess et al., 2015) . In fact, if the control variate choose to unroll our multi-step model for one step, then the resulting algorithm is equivalent to use SVG(1) algorithm in (Heess et al., 2015) as a control variate for a model-free algorithm. However, our algorithm makes use of multi-step prediction and dynamic rollout which SVG did not take advantage of.

3. PRELIMINARIES:

Reinforcement learning aims at learning a policy that maximizes the sum of future rewards (Nagabandi et al., 2017) . Formally, in an MDP environment (Thie, 1983) (S, A, f, r), suppose that at time t the agent is in state s t ∈ S and it executes one action a t sampled according to its policy π θ (a t |s t ), then receives a reward r(s t , a t ) from the environment. Following this, the environment makes a transition to state s t+1 according to an unknown dynamics function f (s t , a t , η) where η is a random noise variable sampled from a fixed distribution, for example η ∼ N (0, 1). The trajectory {s i , a i , r i } N i=1 is the resulting sequence of state-action-reward records produced by the policy and the environment. The goal is to maximize the discounted total sum of reward +∞ t =t γ t -t r(s t , a t ), where γ is a discounting factor that prioritizes near-term rewards. In MSCV, to select the best actions, we train a stochastic policy a = µ(s, ) ∼ π θ (a|s), where ∼ N (0, 1) is another random noise variable.

In model-based reinforcement learning, a dynamics model is learned as an approximation of the real unknown dynamics f . In MSCV, we are learning a multi-step dynamics model f predicting the next state

s k+1 = f (s 1 , a 1 , η 1 • • • a k , η k ).

f is modeled as a RNN so that it can be unrolled for an arbitrary number of time steps k. We also need to fit a reward predictor r(s t , a t ) (Otherwise we can also assume the reward function r is available to us.). Besides these two state and reward predictive models, we also learn an on-policy value function V π (s) from off-policy data, which can be viewed as an approximation of the real expected future return V π (s) of state s.

4. MULTI-STEP MODELS AS CONTROL VARIATES:

Assume that we have a multi-step model as specified above. Given a real trajectory T = {s i , a i , r i , η i , i } N i=t collected using the current policy π θ , we define the empirical k-step return as Qπ

k (s t , a t ) = r t + • • • + γ k-1 r t+k-1 + γ k V π (s t+k ). Given the sub-trajectory T k = {s i , a i , r i , η i , i } t+k i=t , we can "embed" T k to an imaginary trajectory E(T ) = {s i , a i , r i , η i , i } t+k

i=t where E(T ) satisfies three conditions: (1) s t = s t , ( 2)

{s i+1 = f (s i , a i , η i )} t+k i=t (3) {a i = µ(s i , i )} t+k i=t .

One can easily verify f and µ θ 's effectiveness by seeing that when f = f , the imaginary trajectory E[T ] perfectly matches the real trajectory T .

In the following, for notational simplicity we assume the unknown dynamics f and the model f are deterministic. The stochastic cases are similar except that one need an extra inference network to infer the latent variables of a given generation. We can represent the k-step imaginary empirical return as:

Q(s t , µ θ (s t , t ), t+1 • • • t+k-1 ) = Q(s t , a t • • • a t+k-1 ) (1) = r(s t , a t ) + • • • + γ k-1 r(s t+k-1 , a t+k-1 ) + γ k V π (s t+k ),

where a = µ θ (s , ). Assume all { i } t+k i=k are sampled from fixed prior distribution P = N (0, 1), we have the following policy gradient formula:

∇J(θ) = E ρ π ,P [∇ θ log π(a t |s t ) Qπ k (s t , a t )] (2) = E ρ π ,P [∇ θ log π(a t |s t )[ Qπ k (s t , a t ) -Q(s t , a t , t+1 • • • t+k-1 )]] (3) + E ρ π ,P [∇ θ log π(a t |s t ) Q(s t , a t , t+1 • • • t+k-1 )] (4)

We assume in the above equation that the imaginary rollout to compute Q is an embedding of the real rollout to compute Q. The second term can be expressed via the reparameterization trick:

EQUATION

where the operator ∇ t θ stands for the partial derivative with respect to the θ of time step t only, namely we treat policy π θ occurring before and after time step t as fixed. This can be easily done by detaching the node from the computational graph with any modern deep learning framework exploiting automatic differentiation. Through this way, the variance of the second term G 2 can be reduced significantly since we cut off its dependency among former and latter timesteps. The algorithm we use for calculating the partial derivative of G 2 is backpropagation through time of RNN.

There are several interesting properties of the derived gradient estimator. First, imaging the extreme cases in which f , r are perfect, then G 1 is always zero. On the other hand, G 2 is reparameterized as a differentiable function from a fixed random noise, whose gradient is of low variance. In practice, one can replace k-step cumulative return Qk with the real return Q of the entire trajectory. In this case, the estimator will be unbiased at the expenses of a little more variance.

Dynamic rollout. There is a trade-off on the model rollout number k. If k is too small, the model bias is small but the training procedure is not able to take advantage of the multi-step trajectory optimization and the accurate reward predictions. Moreover, more burden would be put on the accuracy Value function. The value function V π can be conveniently fitted off-policy. Different from DDPG, the gradient estimator for V π is fitted using importance sampling, thus the objective is consistent. More precisely, if a transition (s, a, r, s) sampled from the replay buffer is generated by policy π θ0 (a|s), the current policy is π θ (a|s), we compute an importance weight w t = π θ (at|st) π θ 0 (at|st) , and we minimize the weighted bellman error: w t ( V (s, θ) -y) 2 , where y = r + V (s, θ).

Algorithm 2 Fitting Value Function V π Given experience replay memory M Given value function V (•, θ), outer loop time t θ = θ for m = 0 do Sample(s k , a k , r k , s k+1 ) from M (k < t). y m = r k + γ V (s k+1 ; θ ) w = p(a k |s k ) p(a k |s k ) . ∆ = ∇ ν new w 2 (y m -V (s k ; θ)) 2 .

Apply gradient-based update to θ using ∆. θ = σθ + (1 -σ)θ end for

6. EXPERIMENT:

We evaluate MSCV on the MuJoCo and OpenAI Gym benchmarks against TRPO. We perform experiments on Reacher, Hopper, Swimmer and Walker tasks. The batch size is 5000 for the first three tasks and 25000 for Walker. For all experiments we set the number of steps k to be 2. Although we tried other values of k, we found k = 2 is most effective. We set the discounting factor to be 0.995 and α = 10 for all experiments.

6.1. MULTI-STEP DYNAMICS MODEL:

In this section, we evaluate the dynamics model's capacity. We use the similar architecture with that in (Nagabandi et al., 2017) to predict the difference between the next states and current states.

Figure 3 shows our predicted rewards in Swimmer task. We randomly choose a starting state from a trajectory and roll out for 15 steps. Nevertheless, it is not enough to have accurate reward in our algorithm, because we would backpropogate through the trajectory as well. In Figure 2 , we compare the L2-norm of the actual states and imaginary states from the dynamic model. We also plot the norm of difference between two states and the cosine similarity. The results show that the dynamics model is able to predict the future states and rewards, which proves the fundamental assumptions in our algorithm. In this section, we present the comparison between MSCV and TRPO. For all tasks, both policy and value networks are two-layer neural networks with hidden size [64, 64] . We pre-train the dynamics model for each task using the technique described in section 5.

We summarize the results in Table 1 . We set a threshold for each task and recorded the number of iteration needed in order to reach the threshold. Table 1 shows that MSCV outperformed TRPO in every task. The most significant improvement is in Walker, where the setting is more complex than the other three. It shows that by combining model-free and model-based algorithms, we can learn to improve the policy more efficiently in the complex dynamics.

Figure 4 , 5, and 1 shows the improvement of policy after each iteration. It shows MSCV is able to converge faster and learn more efficiently than TRPO at each iteration. The improvement in Figure 1 is least significant, and we believe that it is because of the simple dynamic in this task that TRPO is already able to learn efficiently.

7. CONCLUSION:

In this paper we presented the algorithm MSCV which combines the advantages of model-free and model-based reinforcement learning algorithms while alleviates the shortcomings of both sides. The resulting algorithm utilizes BPTT and multi-step models as a control variate for model free algorithms.

Our algorithm can be viewed as a multi-step model-based generalization of QProp, in which, instead of training a Q function using unstable techniques such as DDPG, we use off-policy data to stably train a multi-step dynamics model, a reward predictor, and an on-policy value function.

Our experiments on MuJoCo and OpenAI Gym benchmarks not only prove the footstone assumptions of MSCV, but show that MSCV outperforms the baseline method significantly in various circumstances.

:



Figures and Tables:

Figure 1: Reacher

Figure 2: State Prediction Figure 3: Reward Prediction

Iteration Required for Every Task

Table content: <table><tr><td>Task</td><td colspan="3">Threshold MSCV TRPO</td></tr><tr><td>Swimmer</td><td>90</td><td>21</td><td>30</td></tr><tr><td>Reacher</td><td>-7</td><td>34</td><td>42</td></tr><tr><td>Walker</td><td>2500</td><td>167</td><td>279</td></tr><tr><td>Hopper</td><td>2000</td><td>35</td><td>57</td></tr></table>

Peer review (iclr2018):
 The main idea of the paper is to improve off-policy policy gradient estimates using control variates based on multi-step rollouts, and reduce the variance of those control variates using the reparameterization trick. This is laid out primarily in Equations 1-5, and seems like a nice idea, although I must admit I had some trouble following the maths in Equation 5. They include results showing that their method has better sample efficiency than TRPO (which their method also uses under the hood to update value function parameters). My main issue with this paper is that the empirical section is a bit weak, for instance only one run seems to be shown for both methods, there is no mention of hyper-parameter selection, and the measure used for generating Table 1 seems pretty arbitrary to me (how were those thresholds chosen?). In addition, one thing I would have liked to get out of this paper is a better understanding of how much each component helps. This could have been done via empirical work, for instance: - Explore the effect of the planning horizon, and implicitly compare to SVG(1), which as the authors point out is the same as their method with a horizon of 1. - Show the effect of the reparameterization trick on estimator variance. - Compare the bias and variance of TRPO estimates vs the proposed method.