Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper proposes an off-policy model-based reinforcement learning approach. The proposed paper combines several techniques together. The general framework fits a maximum entropy reinforcement learning, and the policy and q-function updates are formulated in a soft actor-critic manner. To leverage the learned dynamics, the proposed approach adopts value expansion to substitute the learned reward function and forward dynamics function in the Bellman equation for the value function. Also, the imagined roll-out could be used to update the q-function.

I feel the novelty of this approach is a bit limited, considering that the novelty mainly comes from embedding a more powerful maximum entropy RL formulation upon the value expansion [Freinberg et al., 2018] and stochastic value gradient[Heess et al., 2015], rather than from the perspective of model-based modeling. Even though the experiment result is still very promising and it's good to see the method could outperform SLBO in 3 out of 4 cases.

 
Some other comments:
- The presentation in Sec 3.2 is not clear. Hard to follow without looking into the MVE paper. The definition of \tau and H should be given at that place. Also, the authors claim Q is updated with real *transition*, but later part states only initial tuple is from real data.   
- Up to how many roll-out step k could the model reliably work upon should be clearly stated.
- For Walker2d, more training steps need to be shown.
- Robustness of the algorithm: are the results in Figure 1 derived from different random seeds? If so, how many of them?
- The experiment is only conducted in 4 control domains. The authors may consider to include more task domains, e.g., swimmer, and humanoid as well.
- Are the steps in Figure 1 correspond to only real simulator steps or total steps including imagined ones?
- The curves in Figure 2 are not shown with an uncertainty plot.
review_title: Official Blind Review #1

final_score: rating: 1: Reject
review_content: This paper presents a slightly new model-based reinforcement learning algorithm, claiming that the new method is elegant and combines the merits of earlier methods. In particular, it claims that the new method is more sample efficient and computationally efficient than previous methods.The novelty of the new method has to do with how the model participates in “the policy improvement step”. The paper does not achieve a high level of quality in its presentation or empirical results, as i discuss further below, and as a result I do not consider it to make a scientific contribution, and I recommend rejection.

The presentation of the method is not clear; it uses an informal notation and leaves out many steps; in many cases the statements are not formally correct. I believe that many of these weaknesses are not introduced in this work but are present also in the works that it builds on. It is possible that someone intimately familiar with the previous works would be able to fully understand this method, but perhaps not, and certainly it does not stand alone. 

The new method is presented in Section 3. Equation (3) defines an objective for the policy parameter. It uses things called r-hat, \rho-hat, p, and f. It is not clear whether these things are functions, random variables, or distributions. What is the domain of f? Is r(s,a) a random variable? Is r(s1,a1) different from r(s2,a2) if s1=s2 and a1=a2? Do they have the same distribution? This is just scratching the surface. Many, many things are not well defined, and some critical things, like V, appear to be defined multiple times.

The new method is ultimately presented in Algorithm 1, but it is just a sketch, with essential things left out. What is an “iteration”? Where are the “epochs” discussed in the empirical results? Many steps ask us to calculate gradients with respect to objective functions given by equations in the text. Almost all of these are unclear. To literally calculate the gradient of the expected values would involve a full sweep over the replay buffer, which appears never to be emptied, and thus would grow without bound. It this really what is meant. 

There are also many problems with the empirical results. To begin with, there is no discussion of how the many hyper-parameters were set. And the statistics are not present.  The results present means and standard deviations. The paper does not appear to say how many runs were done, but from the wide variation in the standard deviations, it would appear that there were not many. The very basics of a valid experiment are thus missing. We really should not conclude anything from such results, but if they are published, then many readers would conclude things. Our field should not go any further this way, in my opinion.

review_title: Official Blind Review #2

final_score: rating: 1: Reject
review_content: #rebuttal response
Thanks for the explanations for the difference between S2VG and SVG, new results on two more MuJoCo environments. However, I still think that the motivation of the paper is not clear: the authors may have a good algorithm. But the value of adding entropy on SVG or training the policy on imagination data is not well justified.  


#review
This paper proposes a new model-based method that combines the maximum-entropy objective and stochastic value gradient methods. Based on this method, the authors present the Soft Stochastic Value Gradient method (S2VG). Experimental results show that S2VG beats SVG, SAC, DDPG, and SLBO on three MuJoCo tasks.

First of all, the paper lacks a good motivation. SVG already directly embed the model into policy improvement. Then the novelty of this paper is adding an entropy regularizer to improve the robustness under the model estimation error. However, this point is not clarified in the paper. As shown in  Figure 1(a),(d), the performance of S2VG is not stable. Thus I do not find the value of adding a policy entropy regularizer on the SVG method.

Secondly, the comparison is not fair. S2VG should be compared with STEVE ((Buckman et al., 2018), the state of the art model-based method. That is, the claim that S2VG beats state-of-the-art model-based methods is not appropriate.

Finally, the performance improvement is somewhat weak as S2VG only beats other baselines on three MuJoCo tasks.

Question:

(1) Why is the recursive value gradient missing in Eq. (2)? 
(2) It is interesting that S2VG outperforms SVG significantly. Does the improvement come from better exploration?
(3) In Figure 2(c), the bias of H=2 is larger than that of H=5. is there any explanation?
review_title: Official Blind Review #3

Meta-review: The authors consider improvements to model-based reinforcement learning to improve sample efficiency and computational speed. They propose a method which they claim is simple and elegant and embeds the model in the policy learning step, this allows them to compute analytic gradients through the model which can have lower variance than likelihood ratio gradients. They evaluate their method on Mujoco with limited data.

All of the reviewers found the presentation confusing and below the bar for an acceptable submission. Although the authors tried to explain the algorithm better to the reviewers, they did not find the presentation sufficiently improved. I agree that the paper has substantial room for improvement around clarity. Reviewers also asked that experiments be run for more time steps. I agree that this would be an important addition as many model-based reinforcement learning approaches perform worse asymptotically model free approaches and it would be interesting to see how the proposed approach does. A reviewer pointed out that equation 2 is missing a term, and indeed I believe that is true. The authors response is not correct, they likely refer to an equation in SVG where the state is integrated out. Finally, the method does not compare to state-of-the-art model-based approaches, claiming that they use ensembles or uncertainty to improve performance. The authors would need to show that adding either of these to their approach attains similar performance to state-of-the-art approaches.

At this time, this paper is below the bar for acceptance.
Decision: Reject