S2VG: Soft Stochastic Value Gradient method
Authors: Xiaoyu Tan
Chao Qu
Junwu Xiong
James Zhang
Venue:  (2020)
Keywords: Model-based reinforcement learning
soft stochastic value gradient

Abstract: Model-based reinforcement learning (MBRL) has shown its advantages in sample-efficiency over model-free reinforcement learning (MFRL). Despite the impressive results it achieves, it still faces a trade-off between the ease of data generation and model bias. In this paper, we propose a simple and elegant model-based reinforcement learning algorithm called soft stochastic value gradient method (S2VG). S2VG combines the merits of the maximum-entropy reinforcement learning and MBRL, and exploits both real and imaginary data. In particular, we embed the model in the policy training and learn $Q$ and $V$ functions from the real (or imaginary) data set. Such embedding enables us to compute an analytic policy gradient through the back-propagation rather than the likelihood-ratio estimation, which can reduce the variance of the gradient estimation. We name our algorithm Soft Stochastic Value Gradient method to indicate its connection with the well-known stochastic value gradient method in \citep{heess2015Learning}.

Under review as a conference paper at ICLR 2020
S2VG: SOFT METHOD
STOCHASTIC
VALUE
GRADIENT
Anonymous authors Paper under double-blind review
ABSTRACT
Model-based reinforcement learning (MBRL) has shown its advantages in sample- efficiency over model-free reinforcement learning (MFRL). Despite the impres- sive results it achieves, it still faces a trade-off between the ease of data generation and model bias. In this paper, we propose a simple and elegant model- based reinforcement learning algorithm called soft stochastic value gradient method (S2VG). S2VG combines the merits of the maximum-entropy reinforcement learning and MBRL, and exploits both real and imaginary data. In particular, we embed the model in the policy training and learn Q and V functions from the real (or imaginary) data set. Such embedding enables policy gradient through the back-propagation rather than the likelihood-ratio esti- mation, which can reduce the variance of the gradient estimation. We name our algorithm Soft Stochastic Value Gradient method to indicate its connection with the well-known stochastic value gradient method in (Heess et 2015).
compute analytic
INTRODUCTION
Reinforcement learning be generally classified into two categories: model-free reinforcement learning (MFRL) and model-based reinforcement learning (MBRL). The last several years have witnessed the great success of e in playing video games, robotic control and motion animation (Mnih et al.) 2015 Lillicrap et al.) 2013 Schulman et al. 2017 Peng et al. 2018). How- ever, even for some simple tasks, hundreds of millions of samples are required for an agent to learn good control policy. In many industry scenarios, such as health care and financial services, the algorithm requiring tremendous interactions with the environment is not applicable or too expensive to deploy. To this end, several recent works have advocated the model-based approach, where the higher sample-efficiency is achieved by leveraging the learned dynamics and reward model (Buck- man et al. 2018 Feinberg et al., 2018). It generally augments the real data with the data from dynamics models, uses rollout to improve target for temporal difference learning, or directly incor- porates the model into the Bellman equation (Luo et al.] 2018) Heess et al. (2015). These works have demonstrated promising results on several benchmarks with a small number of interactions with the environment.
Despite its recent success, MBRL still faces a challenging problem, i.e., the model-bias, where the imperfect dynamics model would degrade the performance of the algorithm (Kurutach al. 2018). Unfortunately, such things always happen when the environment is sufficiently complex. There are few efforts to mitigate such issue by combining model-based and model-free approaches. Heess et al. (2015) compute the value gradient along real system trajectories instead of planned ones to avoid the compounded error. KKllweit & Boedecker (2017) mix the real data and imaginary data from the model and then train Q function. An ensemble of neural networks can be applied to model the environment dynamics, which effectively reduces the error of the model (Kurutach et al. 2018 Clavera et al. 2018 Chua et al. 2018).
We observe that most recent algorithms with promising results apply Dyna-style update (Sutton 1990 Kurutach et al., 2018 Luo et al. 2018). They collect real data using current policy to train the dynamics model. Then the policy is improved using state-of-the art model-free reinforcement learning algorithms with imagined data generated by the learned model. Our insight is that why not directly embed the model into the policy improvement? To this end, we derive a model-based reinforcement learning algorithm in the framework of the maximum entropy reinforcement learning

---

Under review as a conference paper at ICLR 2020
(iebarr et al. 2008). Dynamics model and reward model are trained with the real data set collected from the environment. Then we simply train Q and V function using the real data set with the update rule derived from the maximum entropy principle (several other advanced ways to include the imaginary data can also be applied, see details in section In the policy improvement step, the stochastic actor samples action with real state as the input, and then the state switches from to s' according to the learned dynamics model. We link the learned dynamics model, reward model, and policy compute an analytic policy gradient by the back poeck-ropagation Comparing with likelihood- ratio estimator usually used in MFRL method, such value gradient method would reduce the variance of the policy gradient (Heess et al.) 2015). The other merit of S2VG is its computational efficiency. Several state-of-the-arr MBRL algorithms generate hundreds of thousands imaginary data from the model and a few real samples. Then the huge imaginary data set feeds into MFRL algorithms, which On the contrary,
may be sample-efficient in terms of real samples but not computational-friende our algorithm embeds the model in the policy update. Thus we can implement it efficiently by computing policy gradient several times in each iteration (see our algorithm I and do not need to do calculation on the huge imaginary data set.
We name our algorithm soft stochastic value gradient to indicate its connection with SVG (Heess) jet al., 2015). Notice there are several differences between S2VG and SVG. Firstly, to alleviate the issue of the compounded error, SVG proposes relatively conservative algorithm where just real data is used to evaluate policy gradients. Thus imaginary data is wasted, even the data from the short rollouts from the model can be trusted some extent. our work, the policy is trained with the model and imaginary dataset m times in each iteration of the algorithm. Secondly, we derive our algorithm in the framework the maximum entropy reinforcement learning. The maximum entropy updates could improve the robustness under the model estimation error (Ziebart et al., 2010). In addition, it encourages the exploration, prevents the early convergence to the sub-optimal policies, and shows state-of-the-art performance in MFRL (Haamoja al., (0018). Thirdly, S2VG avoids the importance sampling in the off-policy setting by sampling the action from π and transition from fs, a), which further reduces the variance of the gradient estimation.
CONTRIBUTIONS
We derive an elegant, sample-efficient, and computationallfriende Dyna-style MBRL algorithm in the framework of the maximum entropy reinforcement learning with a principled way. Different from the traditional MBRL algorithm, we directly embed the model in the policy improvement, which could reduce the variance in the gradient estimation and avoid the computation on huge imaginary dataset. In addition, since the algorithm is off-policy, it is sample-efficient. At the same time, the maximum entropy principle encourages exploration and improves performance, which has been observed in MFRL (Haarnoja et al. (017)). We test our algorithm on several benchmark tasks in Mujoco simulation environment (Todorov et al. 2012) and demonstrate that our algorithm can achieve state-of-the-art performance
2 PRELIMINARIES
In this section, we first present some backgrounds on the Markov decision process. Then we in- troduce the knowledge the maximum entropy reinforcement learning (Ziebart et al. 2008 and stochastic value gradient (Heess et al. 2015) since parts of them are the building blocks of our algorithm.
2.1 MDP
Markov Decision Process (MDP) can be described by 5-tuple (S, space, AA is the action space. P = (P(s'|s, a)). are the transition probabilities. R (r(s,a))s,'ee) ESs. are the real-valued immediate rewards, and E (0, 1) is the discount factor. policy is used to select actions in the MDP. general, the policy stochastic and denoted by m, where 7 (a,]s,) is the conditional probability density at a associated with the policy. The state value evaluated on policy π could be represented by (=(() = E, D >r((Se, 0z)[s0 = s] on immediate reward return
S is the state
()),, 'ES with discount factor 7 E (0, 1) along the horizon
Code submitted anonymously http:://gttthbccomm///Gaaaaaannn
2

---

Under review as a conference paper at ICLR 2020
t. The state-action value evaluated on policy represent the expected return the specific action a: Q" a) = r(su, au) + E 2e 7 (s, at) ) so 8, Qu = a..
2.2 MAXIMUM ENTROPY REINFORCEMENT LEARNING
Maximum entropy reinforcement learning augments the reward with entropy term, such that the optimal policy aims to maximize the new reward function at each visited state:
max J(T)
Σ (e (((8, a.) + (H((T(118))))
(1)
where H((T(-1s)) is an entropy term scaled by a (Ziebart et al. 2008 equation can be obtained by the following soft-Q update (Fox et al. 2016).
The optimal policy in
a()),a)) -- r(st, az) +yllss 1- (V((+ and V(s.) < a log( / exp( Q az) dda)-
a
Above iterations define the soft Q operator, which is contraction. The optimal policy x² (als) can exp( 1 (se,ae))
be recovered by n² (ac(s=) = expí (s..a Idn."" where Q" is the fixed point of the soft-Q update.
We refer readers to the work (Ziebart al. 2008 Haarnoja et al. 2017) for more discussions. In
soft actor-critic (Haarnoja et al. (2188). the optimal policy π [a Sz is approximated by neural network no (s))))))). which solved by the following optimization problem
max E Ber- Bay (ar (aaaaaa))a(aas ax) a log (o(a,(s)) Toe (as((s)
2.3 STOCHASTIC VALUE GRADIENT
Stochastic value gradient method is model-based algorithm which is designed to avoid the com- pounded model errors by only using the real-world observation and gradient information from the model (Heess et al. 2013). The algorithm directly substitutes the dynamics model and reward model in the Bellman equation and calculates the gradient. To perform the backpropagation in the stochas- tic Bellman equation, re-parameterization trick is applied to evaluate the gradient on real-world data. The stochastic policy π (a|s;; 0) with parameter could be optimized by the policy gradient in the following way
(v(s) 00
@f(s. a) (r(ass) 7(
(V'(s') 0f a) (T((as))
Re Ey.cl
(2)
da
00
Os' da
00
where 1 and Ç are the policy and environment re- e--aramtterizationn noise which could be directly sampled from a prior distribution or inferred from generative model g(n, Cls, s'). The fss,a) and (s,, a) are dynamics model and reward model respectively.
SOFT STOCHASTIC VALUE GRADIENT METHOD
In this section, introduce our algorithm soft stochastic value gradient (S2VG) method. Our ob- jective is to design an off-po MBRL algorithm under the maximum entropy framework, which could improve the exploration, enhance the training robustness through entropy maximization. Par- ticularly, we optimize the following equation
((0) = Σ (p-())(((s),a) a.) + H(*((, sx))],
(3)
1=
where we omit the regularizer parameter a of the entropy term in the following discussion ease the exposition. År = p(s)) - f(r, a (ay)ss))))) f(s as) is the learned dynamics model, and A is the reward model. We then derive the update rule following the similar step in the probabilis- tic reinforcement learning (Levine. 2018), which generally includes policy evaluation and policy improvement steps.
3

---

Under review as a conference paper at ICLR 2020
Similar to the policy evaluation without the entropy term, we have the soft value function update ((st) = Eatar|s)) |s, (f at) - log π (ay [t)]ss) + yBoe V (s++1)]. For convenience, we define the Q
function in the following.
Q(st; ar) fst,; at) + (
(4)
The value function update can be reformulated
V(+e) le)) (a (Q(s,, - log (aelss))]
(5)
The optimal policy (policy improvement step) at each step is (aae st) J. exp Q aa aa
exp(Qis..a.)
The derivation is almost the same with (Levine) 2018 Haarnoja et al. 2018). expect that we use learned dynamics model and reward function here. Notice this optimal policy can be approximated by a parametric function (g(a)(s)) and obtained by solving (alss Esp-p(s,) (a, Ea (a)aa.lss(((sss.....e. MFRL can not leverage the model information. We leave the our derivation and discussion
[Q(st, ar) - log vT [p){)[()))]] However such way used in the
the policy improvement in section 3.3
3.1 MODEL LEARNING
The transition dynamics and rewards could be modeled by non-linear function approximations as two independent regression tasks which have the same imput but different output. Particularly, we train two independent deep neural networks with parameter w and op to represent the dynamics model f and reward model f respectively. To better represent the stochastic nature of the dynamic transitions and rewards, we implement re- ce-aarameterization trick on both f and with input noises Ew and Sp sampled from Gaussian distribution N (O, 1). Hence, networks would generate mean: Pun Hip, and variance: ow, dg separately and compute the result by n.d + owew and Mı + op respectively.
Above two models could optimized by sampling the data from the (real data) replay buffer D and minimizing the mean square error:
1 ((w] = - ED -') = Bo ,,, ((((s - r)
(6)
This supervised learning problem could be solved by off-the-shelf optimizers such as Adam opti- mizer (Kingma Ba 2014). Other techniques could also be leveraged to reduce the risk of over- fitting in the limited data set such adding dropout layers, performing early stopping, and training through ensemble models.
3.2 VALUE FUNCTION LEARNING
Equation 4 and equation 5 define model-based and model-free policy evaluation steps respectively. Equation would introduce the model error from the model dynamics into the value estimation and produce biased results. To avoid this model error, update Q with the real transition (s, a, r, s') from the real data replay buffer. Such update is non-biased but may suffer from high variance under the low-data regime. Therefore, we leverage the value expansion (Feinberg et al. 2018 balance the bias and variance by using both real-world data and imaginary rollout. If Q function and V function are parameterized by ó and e respectively. they could be updated by minimizing the new objective function with the value expansion on imaginary rollout:
1 #-1
H-1
J(Ø)
Σ
(Qo(5r.
FA t VV)(H)))
where only the initial tuple data, and later transitions are sampled from the imaginary rollout from the model. H here is the time step of value expansion using real data. 7 is the training tuple and To is the initial training tuple. Note that when = 1, it reduces to the case where just real data is used.
(So, āo fo, 81) is sampled from replay buffer D with real-world
The V function is learned by minimizing the following error
((b) = Ee,~ol (Vu(s)) - Eu.~x BarxIQulss a4) - log x (ag(s:))))
(8)
4

---

Under review as a conference paper at ICLR 2020
Notice the training of V function is only on the real data set D. In experimental section, do ablation study on H. Interestingly, we find that the case with H = 1 has the best result. One possible explanation is that we have already embedded model in the policy update. Hence, including the imaginary data in value function learning would mislead directions of policy gradients.
3.3 POLICY LEARNING
Then we consider the policy improvement step, i.e., to calculate the optimal policy each time step. One naive way is to optimize the following problem (axa( (eccop(s.)) (a.ee)( (8t,, P 0t) log [s))] as that in MFRL (Levine 2018 Haarnoja et al., 2018). However, such way cannot leverage the learned dynamics model and reward model. To incorporate the model information, notice that ((s)) Basnta,ls,)) (Q[ (st, az) - log π (au]s:)), thus the policy improvement step is equal
max s.ppp())) V(s.). <a,
In the following, we connect the dynamics model, reward model, and value function together by the soft Bellman equation. To begin with, we re-parameterize the dynamics model and policy. Particularly, we set a = k(s, ry; 6) and the dynamics model s' f(s, a, () for noise variables 7 ~ p(n) and Ç p(C), respectively. Now we can write the soft Bellman equation in the following
way.
(b) = [F,[F x(s,, m;; - log (als)) + + k((s, m O))
(10)
To optimize equation and leverage gradient information of the model, we sample s from the real data replay buffer D and take the gradient of V(s) w.r.t. 0
@v(s) E,-D 00
Or On
1 dm
(5' Of On
= D.nD Du 00
00 + +>l Os' Oa 00 ]]
(1I)
The equation demonstrates an interesting connection between our algorithm and SVG. Compared with the policy gradient step taken by SVG(1) algorithm (Heess al. 2015), equation [1 includes one extra term ((T /00) to maximize the entropy of policy. We drop importance sampling weights by sampling from the current policy. Also notice that the transition from (s, a) to s' is sampled from the learned dynamics model f. while the SVG(1) just utilizes the real data. Thus in the algorithm we can update policy several times in each iteration fully utilized the model rather than just use the real transition once.
3.4 S2VG ALGORITHM
We summarize our S2VG in Algorithm I At the beginning of each step. we train dynamics model f and reward model f by minimizing the L2 loss shown in equation Then the agent interacts with the environment and stores the data in the real data replay buffer D. Actor samples 8); from D and collect $k41 according to the dynamics model [(kk, ax). Such imaginary transition is stored in Dimg- Then we train Q, V and π according to the update rule in section 3 Similar to other value-based RL algorithms, our algorithm also utilizes two Q functions to further reduce the overestimation error by training them simultaneously with the same data but only selecting the minimum target in value updates (Fujimoto et al. 2018). We use the target function for V like that in deep Q-learning algorithm (Mnih et al.. (015)). and update it with an exponential moving average. We train policy using the gradient in equation 11 Remark that our s' is sampled from the dynamic model [(s, ()),a), while in SVG, it uses the true transition. Indeed there is bias and variance trade-off. True transition is unbiased but may have high variance due to the small scale of the data set. Here we claim that one-step rollout of the model is still accurate. Some remarks our implementation, we choose H = 1 in equation 7 generally, since we find S2VG with = has the best result in ablation study (see section 5.3)) In each iteration, we update policy several times, so that the algorithm can utilize the data generated from the model.
the algorithm are in the order. In
4 RELATED WORK
There are plethora of works on MBRL. They can be classified into several categories depending on the way utilize the model, to search the optimal policy or the function approximator of the
5

---

Under review as a conference paper at ICLR 2020
Algorithm 1 Soft Stochastic Value Gradient method
Replay buffer D, imaginary replay buffer Ding- policy value function Vi, target value function V3. Two Q functions with parameters Φ and Φ1, dynamic model f with parameter w, and reward model f with parameter yp for each iteration
Inputs:
1. Train the dynamics model and reward model
Calculate the gradients [w), ((p)) using equation 6 with D, update w and
2. Interact with environment
Sample Ge ~ π (ag [t), get reward rt, and observe the next state 8141
Append the tuple (s,, dzs Tt, 814-) into D
3. Update the actor, critics m (typically 3 to 5) times
Empty Domg
Sample (su, ao, ro. sy)
for each imaginary rollout step do
Sample 6g ~ T (as (sx), get reward = f(*k, ax), and sample 8k41 [(kk 0s)
Append the tuple (8k, Gk: Tk, 8k+1) ) into Dong
end for
Calculate the gradient ( using equation with & and Dimg
Calculate the gradient V V (v)) using equation with D
Calculate the gradient Vo (())s using equation with D.
Update o, v. and 0, update with Polyak averaging
end for
dynamics model. Iterative Linear Quadratic-Gaussial (iLQG) (Tassa et al.) 2012) assumes that the true dynamics are known to the agent. It approximates the dynamics with linear functions and the reward function with quadratic functions. Hence the problem can be transferred into the classic
LQR problem. In Guided Policy Search (Levine & Koltun 2013; Levine & Abbeel, [0014] Finn et al. 2016), the system dynamics are modeled with time-varying Gaussian-lined model. approximated the policy with a neural network by minimizing the KL divergence between iLQQ and т. A regularization term is augmented into the reward function to avoid the over-confidence on the policy optimization. Nonlinear function approximator
be leveraged to model more compli- cated dynamics. Deisenroth Rasmussen (201)) use Gaussian processes to model the dynamics of the environment. The policy gradient However, it may suffer from the curse of dimensionality which hinders its applicability in the real problem. Recently, more and more works incorporate the deep neural network into MBRL. Heess) et al. (2015) model the dynamics and reward with neural networks, and compute the gradient with the true data. Richards (2005): Nagabandi et (2018) optimize the action sequence maximize the expected planning reward along with the learned dynamics model and then the policy is fine- tuned with TRPO.
be computed analytically along the training trajectory.
al. (2018); Chua et al. (2018); Kurutach et al. (2018) use the current policy to gather the data from the interaction with the environment and then learn the dynamics model. In the next step. the policy is improved (trained by the model-free reinforcement learning algorithm) with large amount of imaginary data generated by the learned model. Ensemble learning can also be applied to further reduce the model error.
EXPERIMENTAL RESULTS
In this section, we would like answer two questions: (1) How does S2VG perform on some bench- mark reinforcement learning tasks comparing with other state-of of-the-art model-based and model- free reinforcement learning algorithms? (2) How many imaginary data we should use the value function update?
5.1
ENVIRONMENT
To answer these two questions, we do experiment in Mujoco simulation environment Tooorov et al, 2012): InvertedPendullum-V HalfCheetah--2 Reacher-v2, Hopper-v2, Swimmer-v2 Walker2d-v2. Each experiment is tested on five trailed using five different random seeds and initialized parameters. The details of the tasks and experiment implementations can be found in appendix

---

Under review as a conference paper at ICLR 2020
Averageratum
600
Ave
5BAG
SNC
13:30
130
20
(a) InvertedPendullmm
(b) HalsCheetah
(c) Reacher
172
Averago rutum
DOP0
0000
80
stops
Stops (1x3)
steps (L43)
(d) Hopper
(c) Swimmer
Walker2d
Figure 1: Performance of S2VG with H = 1 and other baselines in benchmark tasks. The x-axis the training step (epoch or step). Each experiment is tested on five trailed using five different random seeds and initialized parameters. For simple task, i..., InvertedPendullm, we limit the training steps at 40 epochs. For the other three complex tasks, the total training steps are 200K or 300K. The solid line is the mean of the average return. The shaded region represents the stan- werpoddennuuuu.HaaaCeeeeee
dard deviation. On baselines significantly. In the task Walker2d, SLBO is slightly better than S2VG. They both surpass other algorithms. On Reacher, S2VG and SAC perform best.
Hopper, Swimmer, S2VG outperforms the other
5.2 COMPARISON TO STATE-OF-TIIB-ART
We compare our algorithm with state-of-the-ar model-free and model-based reinforcement learn- ing algorithms in terms of sample complexity and performance. DDPG (Lillierap al., (015)) and Soft actor-critic (Haarnoja et al. 2018) are two model-free reinforcement learning algorithms on continuous action tasks. Soft actor-critic has shown its reliable performance and robustness on sev- eral benchmark tasks. Our algorithm also builds on the maximum entropy reinforcement learning framework and benefits from incorporating the model in the policy update. Two model-based rein- forcement learning baselines are SVG (Heess al. 2015) and SLBO (Luo et al. 2018). Comparing with SVG, our work avoids the importance sampling and utilizes the maximum entropy principle. Notice in SVG, the algorithm just computes the gradient in the real trajectory, while our S2VG up- dates policy using the imaginary data m times generated from the model. SLBO is a model-base algorithm with performance guarantees that applies TRPO (Schulman et al. 2015) the data generated from the rollout of the model.
Notice that our implementation, we do use any ensemble learning (Chua al. 2018 or uncertainty estimation (Malik et al.) (201)) on the model. These techniques are known to reduce the model biased. We do use distributed RL either accelerate the training. We believe that above-mentiooed skills are orthogonal to our work and could be integrated into the future work to further improve the performance. Also for the faimess, we just compare this pure version of S2VG with other baselines. We also notice that some recent works in MBRL modify the benchmarks to shorten the task horizons and simplify the model problem (Kurutach et al., 2018). the contrary, test algorithm in the full-length tasks. In all experiment, we implement S2VG with H and will do ablation study on /l the next section.
We present experimental results in Figure I In simple task, invertedPendulumm. S2VG achieves the asymptotic result just using 16 epochs. HalfCheetah, S2VG's performance is at around 8000 at 200k steps, while all the other baselines' performance is below 2500. In Reacher, S2VG and SAC has similar performance. Both of them are better than other algorithms. In Hopper, the final performance of S2VG is around 3300. The runner-up is SAC whose final performance is around

---

Under review as a conference paper at ICLR 2020
11-5
4ppc1
Steps (1e5)
opoch
(a) Pendulum
(b) HalfChectah
(c) Bias
Figure 2: S2VG with value expansion. We do the ablation study (a) Pendulum and (b) HalfChee- tah problem where the x-axis is the training step and the y-axis is the reward. (c) reflects the bias of the value function in the training procedure.
2000. In swimmer, the performance of S2VG best.In walker2d, SLBO is slighter better than S2VG. Both of them achieve the average return 2900 at 300k timesteps.
5.3 DESIGN EVALUATION
In this section, we make the ablation study to understand how much imaginary data we should include in the algorithm. Remark that our algorithm, the model is embedded Soft Bellman equation in the policy update step, which means we fully trust the model to compute the policy gradient. While in the value function update, we can either train Q and V using the true data or the data from imaginary rollout in equation 7 In section 5.2 apply relatively conservative way, i... train Q and V with true data set, i.c. test the algorithm with value expansion, particularly with horizon H 2 and H = 5. Our conclusion that including the imaginary data to train the value function in our algorithm would hurt the performance, especially in complex tasks.
equation 7 In the following experiment,
We demonstrate the performance of S2VG with value expansion in Figure We first test the al- gorithm on simple task pendulum from OpenAl gym (Brockman al. 2016 and show the result in Figure 2a S2VG with H converges to the optimal policy within several epochs. When we increase the value of H, the performance decreases. The agent with optimal policy at 50 epochs. Then we evaluate the performance of value expansion in complex task HalfCheetach from Mujoco environment (Todorov al. 2012). this task, value expansion with H 2 and H 5 does not work at all. The reason would be that the dynamics model of HalfCheetah introduces more significant model bias comparing to the simple task pendulum. Thus training both policy and value function in the imaginary data set may cause large error in policy gradient. In Figure 2c we plot the bias of the value function in the training procedure of Figure 2a We evaluate the value estimation by averaging the estimated value along 100 states sampled from the replay buffer. Then, we perform Monte Carlo sampling starting from each sampled states with 50 trials and average the discounted return the value estimation. We compared the value estimation results our proposed methods with or without utilizing value expansion. Comparing with the S2VG with the learning procedure.
just starts learn the
S2VG with Il 2 and Hl 5 introduce more value estimation bias in
6 CONCLUSION AND FUTURE WORKS
In this paper, we propose new model-based algorithm to directly incorporate models the policy improvement step. Comparing with the existing method, algorithm is both sample-efficient and computationallfrientt
We test our S2VG on several benchmark tasks and achieve state-of-the-ar art performance. We can integrate the existing techniques such ensemble learning and distributed reinforcement learning to further improve the learning speed of S2VG in the future work, since they orthogonal to core idea.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1606.00 117666000544, 2016.
Jacob Buckman, Danijar Hafner, George Tucker, Eugene Brevdo, and Honglak Lee. Sample- efficient reinforcement learning with stochastic ensemble value expansion. In Advances in Neural Information Processing Systems, pp. $224-8234, 2018.
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learn- ing handful of trials using probabilistic dynamics models. In Advances Neural Information Processing Systems, pp. 4754-4765, 2018.
Ignasi Clavera, Jonas Rothfuss, John Schulman, Yasuhiro Fujita, Tamim Asfour, and Pieter Abbeel. Model-based reinforcement learning via meta-policy optimization. arXiv preprint arXiv:1809.05214. 2018. 2018.
Mare Deisenroth and Carl E Rasmussen. Pilco: A model-based and data-efficient approach policy search. In Proceedings of the 28th International Conference on machine learning (ICML-II), pp- 465-472. 2011.
Feinberg, A Wan, Stoica, MI Jordan, JE Gonzalez, and Levine. Model-based value expan- sion for efficient model-free reinforcement learning. In Proceedings of the 35th International Conference on Machine Learning (ICML 2018), 2018.
Chelsea Finn, Sergey Levine, and Pieter Abbeel. Guided cost learning; Deep inverse optimal control via policy optimization. In International Conference on Machine Learning, pp. 49-58, 2016.
Roy Fox, Ari Pakman, and Naftali Tishby. Taming the noise in reinforcement learning via soft updates. UAI, 2016.
Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. arXiv preprint arXiv: 1802.09477 2018.
Tuomas Haarnoja. Haoran Tang. Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-based policies. Proceedings of the 34th International Conference on Machine Learning-Vool Volume 70, pp. 1352-1361. JMLR. org, 2017.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. arXiv preprint arXivv/801..00 2018.
Nicolas Heess, Gregory Wayne, David Silver, Timothy Lillicrap, Tom Erez, and Yuval Tassa. Learn ing continuous control policies by stochastic value gradients. In Advances in Neural Information Processing Systems, pp. 2944-24429222 2952, 2015.
Gabriel Kalweit and Joschka Boedecker. Uncertainty-drive imagination for continuous deep rein- forcement learning. In Conference Robot Learning, pp. 195-206, 2017.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Thanard Kurutach, Ignasi Clavera, Yan Duan, Aviv Tamar, and Pieter Abbeel. Model-ensemble trust-region policy optimization. arXiv preprint 18Xiv: 1059, 2018
Sergey Levine. Reinforcement learning and control as probabilistic inference: Tutorial and review. arXiv preprint (tte:::80000000 2018
Sergey Levine and Pieter Abbeel. Learning neural network policies with guided policy search under unknown dynamics. In Advances in Neural Information Processing Systems, pp. 1071-1079, 2014.
Sergey Levine and Vladlen Koltun. Guided policy search. In International Conference on Machine Learning, pp. 1-9, 2013.

---

Under review as a conference paper at ICLR 2020
Timothy P Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02990 2015.
Yuping Luo, Huazhe Xu, Yuanzhi Li, Yuandong Tian, Trevor Darrell, and Tengyu Ma. Algorith- mic framework for model-based deep reinforcement learning with theoretical guarantees. arXiv preprint wrXiv:1807.03856 2018.
Ali Malik, Volodymyr Kuleshov, Jiaming Song, Danny Nemer, Harlan Seymour, and Stefano Er- mon. Calibrated model-based deep reinforcement learning. 2019.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 5 518(7540):529, 2015.
Anusha Nagabandi, Gregory Kahn, Ronald S Fearing. and Sergey Levine, Neural network dynamics for model-based deep reinforcement learning with model-free finettnnig, In 2018 IEEE Interna- tional Conference on Robotics and Automation (ICRA), pp. 7559-7566. IEEE, 2018.
Xue Bin Peng, Pieter Abbeel, Sergey Levine, and Michiel van de Panne. Deepmimic: Example- guided deep reinforcement learning of physics-based character skills. ACM Transactions Graphics (TOG), 37(4):143, 2018.
Arthur George Richards. Robust constrained model predictive control. PhD thesis, Massachusetts Institute of Technology, 2005.
John Schulman, Sergey Levine, Picter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pp. 1889-1897, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06444 2017.
Richard Sutton. Integrated architectures for learning, planning, and reacting based on approxi- mating dynamic programming. In Machine Learning Proceedings 1990, pp. 216 116-224. Elsevier, 1990.
Yuval Tassa, Tom Erez, and Emanuel Todorov. Synthesis and stabilization of complex behaviors through online trajectory optimization. 2012 IEEF/RSJ International Conference on Intelligent Robots and Systems, pp. 9906-4913. IEEE, 2012.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physies engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033. IEEE, 2012.
Brian D Ziebart, Andrew Maas, J Andrew Bagnell, and Anind K Dey. Maximum entropy inverse reinforcement learning. 2008
Brian D Ziebart, J Andrew Bagnell, and Anind K Dey. Modeling interaction via the principle of maximum causal entropy. 2010.
10

---

Under review as a conference paper at ICLR 2020
A ENVIRONMENT OVERVIEW AND HYPERPARAMETER SETTING
In this section, we provide an overview of simulation environment Table setting for each environment is shown in Table 2
1 The hyperparameter
Environment Name
Observation Space Dimension
Action Space Dimension
Horizon
Pendulum
200
InvertedPendllum
1000
HalfCheetah
17
1000
Hopper
11
3
1000
Walker2D
17
6
1000
Table 1: The observation space dimension, action space dimension, and horizon for each simulation environment implemented in the experiment and ablation study.
Pendulum
InvertedPendulum
HalfCheetah
Hopper
Walker2D
Epoch
50
40
200
300
Policy Learning Rate
0.0003
Value Learning Rate
0.0003
0.001
0.001
0.0003
Model
0.0003
0.0001
Learning Rate
Alpha value
0.2
0.1
0.4
(in entropy term)
environment steps
1000
per epoch
Value and Policy
(256,256)
Network Architecture Model
(32,16)
(256,128)
(256,256)
Network Architecture
Train Actor-critic
5
5
3
Times (m)
The hyperparameter used training S2VG algorithm for each simulation environment. The number in policy, value, and model network architecture indicate the size of hidden units in each layer of MLP. The ReLu activation function implemented in all architecture.
Table 2:


Reviews:
[{"review_id": "HkehnBTnKH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "#rebuttal response\nThanks for the explanations for the difference between S2VG and SVG, new results on two more MuJoCo environments. However, I still think that the motivation of the paper is not clear: the authors may have a good algorithm. But the value of adding entropy on SVG or training the policy on imagination data is not well justified.  \n\n\n#review\nThis paper proposes a new model-based method that combines the maximum-entropy objective and stochastic value gradient methods. Based on this method, the authors present the Soft Stochastic Value Gradient method (S2VG). Experimental results show that S2VG beats SVG, SAC, DDPG, and SLBO on three MuJoCo tasks.\n\nFirst of all, the paper lacks a good motivation. SVG already directly embed the model into policy improvement. Then the novelty of this paper is adding an entropy regularizer to improve the robustness under the model estimation error. However, this point is not clarified in the paper. As shown in  Figure 1(a),(d), the performance of S2VG is not stable. Thus I do not find the value of adding a policy entropy regularizer on the SVG method.\n\nSecondly, the comparison is not fair. S2VG should be compared with STEVE ((Buckman et al., 2018), the state of the art model-based method. That is, the claim that S2VG beats state-of-the-art model-based methods is not appropriate.\n\nFinally, the performance improvement is somewhat weak as S2VG only beats other baselines on three MuJoCo tasks.\n\nQuestion:\n\n(1) Why is the recursive value gradient missing in Eq. (2)? \n(2) It is interesting that S2VG outperforms SVG significantly. Does the improvement come from better exploration?\n(3) In Figure 2(c), the bias of H=2 is larger than that of H=5. is there any explanation?", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "BJlMle0pFB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes an off-policy model-based reinforcement learning approach. The proposed paper combines several techniques together. The general framework fits a maximum entropy reinforcement learning, and the policy and q-function updates are formulated in a soft actor-critic manner. To leverage the learned dynamics, the proposed approach adopts value expansion to substitute the learned reward function and forward dynamics function in the Bellman equation for the value function. Also, the imagined roll-out could be used to update the q-function.\n\nI feel the novelty of this approach is a bit limited, considering that the novelty mainly comes from embedding a more powerful maximum entropy RL formulation upon the value expansion [Freinberg et al., 2018] and stochastic value gradient[Heess et al., 2015], rather than from the perspective of model-based modeling. Even though the experiment result is still very promising and it's good to see the method could outperform SLBO in 3 out of 4 cases.\n\n \nSome other comments:\n- The presentation in Sec 3.2 is not clear. Hard to follow without looking into the MVE paper. The definition of \\tau and H should be given at that place. Also, the authors claim Q is updated with real *transition*, but later part states only initial tuple is from real data.   \n- Up to how many roll-out step k could the model reliably work upon should be clearly stated.\n- For Walker2d, more training steps need to be shown.\n- Robustness of the algorithm: are the results in Figure 1 derived from different random seeds? If so, how many of them?\n- The experiment is only conducted in 4 control domains. The authors may consider to include more task domains, e.g., swimmer, and humanoid as well.\n- Are the steps in Figure 1 correspond to only real simulator steps or total steps including imagined ones?\n- The curves in Figure 2 are not shown with an uncertainty plot.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "B1g5oiHaFB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a slightly new model-based reinforcement learning algorithm, claiming that the new method is elegant and combines the merits of earlier methods. In particular, it claims that the new method is more sample efficient and computationally efficient than previous methods.The novelty of the new method has to do with how the model participates in “the policy improvement step”. The paper does not achieve a high level of quality in its presentation or empirical results, as i discuss further below, and as a result I do not consider it to make a scientific contribution, and I recommend rejection.\n\nThe presentation of the method is not clear; it uses an informal notation and leaves out many steps; in many cases the statements are not formally correct. I believe that many of these weaknesses are not introduced in this work but are present also in the works that it builds on. It is possible that someone intimately familiar with the previous works would be able to fully understand this method, but perhaps not, and certainly it does not stand alone. \n\nThe new method is presented in Section 3. Equation (3) defines an objective for the policy parameter. It uses things called r-hat, \\rho-hat, p, and f. It is not clear whether these things are functions, random variables, or distributions. What is the domain of f? Is r(s,a) a random variable? Is r(s1,a1) different from r(s2,a2) if s1=s2 and a1=a2? Do they have the same distribution? This is just scratching the surface. Many, many things are not well defined, and some critical things, like V, appear to be defined multiple times.\n\nThe new method is ultimately presented in Algorithm 1, but it is just a sketch, with essential things left out. What is an “iteration”? Where are the “epochs” discussed in the empirical results? Many steps ask us to calculate gradients with respect to objective functions given by equations in the text. Almost all of these are unclear. To literally calculate the gradient of the expected values would involve a full sweep over the replay buffer, which appears never to be emptied, and thus would grow without bound. It this really what is meant. \n\nThere are also many problems with the empirical results. To begin with, there is no discussion of how the many hyper-parameters were set. And the statistics are not present.  The results present means and standard deviations. The paper does not appear to say how many runs were done, but from the wide variation in the standard deviations, it would appear that there were not many. The very basics of a valid experiment are thus missing. We really should not conclude anything from such results, but if they are published, then many readers would conclude things. Our field should not go any further this way, in my opinion.\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors consider improvements to model-based reinforcement learning to improve sample efficiency and computational speed. They propose a method which they claim is simple and elegant and embeds the model in the policy learning step, this allows them to compute analytic gradients through the model which can have lower variance than likelihood ratio gradients. They evaluate their method on Mujoco with limited data.

All of the reviewers found the presentation confusing and below the bar for an acceptable submission. Although the authors tried to explain the algorithm better to the reviewers, they did not find the presentation sufficiently improved. I agree that the paper has substantial room for improvement around clarity. Reviewers also asked that experiments be run for more time steps. I agree that this would be an important addition as many model-based reinforcement learning approaches perform worse asymptotically model free approaches and it would be interesting to see how the proposed approach does. A reviewer pointed out that equation 2 is missing a term, and indeed I believe that is true. The authors response is not correct, they likely refer to an equation in SVG where the state is integrated out. Finally, the method does not compare to state-of-the-art model-based approaches, claiming that they use ensembles or uncertainty to improve performance. The authors would need to show that adding either of these to their approach attains similar performance to state-of-the-art approaches.

At this time, this paper is below the bar for acceptance.
Decision: Reject