Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: 

In this paper, the authors proposed the algorithm to introduce model-free reinforcement learning (RL) to model predictive control~(MPC), which is a representative algorithm in model-based RL, to overcome the finite-horizon issue in the existing MPC. The authors evaluated the algorithm on three environments and demonstrated the outperformance comparing to the model predictive path integral (MPPI) and soft Q-learning.

1, The major issue of this paper is its novelty. The proposed algorithm is a straightforward extension of MPPI [1], which adds a Q-function to the finite accumulated reward to predict the future rewards to infinite horizon. The eventual algorithm ends up like a straightforward combination of MPPI and DQN. The algorithm derivation in Sec. 4 follows almost exact as [1] without appropriate reference. 

2, The empirical comparison is quite weak. The algorithm is only tested on three environments and only one (Pendulum) from the MuJoCo benchmark. Without more comprehensive comparison  on other MuJoCo environments, the empirical experiment is not convincing. 


Minors:

1, The derivation from 15-16 is wrong. The first term should be negative.

There are many claims without justification. For example:
"Solving the above optimization can be prohibitively expensive and hard to accomplish online."

- This is not true. There are already plenty of work solving the entropy-regularized MDP online [2, 3, 4] and achieving good empirical performance.

"The re-optimization and entropy regularization helps in mitigating model-bias and
inaccuracies with optimization by avoiding overcommitment to the current estimate of the cost."
- There is no evidence that the entropy-regularization will reduce the mode-bias. Actually, as discussed in [4, 5] the entropy regularization will also incur some extra bias. 

[1] Grady Williams, Nolan Wagener, Brian Goldfain, Paul Drews, James M Rehg, Byron Boots, and Evangelos A Theodorou. Information theoretic mpc for model-based reinforcement learning. In 2017 IEEE International Conference on Robotics and Automation (ICRA), pp. 1714–1721. IEEE, 2017.
[2] Haarnoja, Tuomas, Aurick Zhou, Pieter Abbeel, and Sergey Levine. "Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor." arXiv preprint arXiv:1801.01290 (2018).
[3] Nachum, O., Norouzi, M., Xu, K. and Schuurmans, D., 2017. Bridging the gap between value and policy based reinforcement learning. In Advances in Neural Information Processing Systems (pp. 2775-2785).
[4] Dai, Bo, Albert Shaw, Lihong Li, Lin Xiao, Niao He, Zhen Liu, Jianshu Chen, and Le Song. "SBEED: Convergent reinforcement learning with nonlinear function approximation." arXiv preprint arXiv:1712.10285 (2017).
[5] Geist, Matthieu, Bruno Scherrer, and Olivier Pietquin. "A Theory of Regularized Markov Decision Processes." arXiv preprint arXiv:1901.11275 (2019).
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: This paper builds a connection between information theoretical MPC and entropy-regularized RL and also develops a novel Q learning algorithm (Model Predictive Q-Learning).  Experiments show that the proposed MBQ algorithm outperforms MPPI and soft Q learning in practice. 

The paper is well-written. The derivation is clean and easy to understand. Adding a terminal Q function, which makes horizon infinite, is a very natural idea. Most of the derivation is unrelated to the additional Q^pi^* (Eq 11-17) and Eq (18-20) are simply plugging in the formula of pi^*, which is derived at Eq (9), which is also quite natural if one knows the result for finite horizon horizon MPPI (i.e., without the terminal Q). For experiments, I'd like to see some results on more complex environments (e.g., continuous control tasks in OpenAI Gym) and more comparison with recent model-based RL work. 

Questions:

1. The experiment setup is that a uniform distribution of dynamics parameters (with biased mean and added noise) are used. Why not using a neural network? 
2. Eq (22): The estimation for eta is unbiased, However, 1/eta might be biased, so is the estimator for w(E) also biased? 

Minor Comments:
1. Eq (16): should the first term be negated? 
2. Page 6, last paragraph: performved -> performed.
3. Page 7, second paragraph: produces -> produce.
4. Algorithm 1, L6: N % N_update -> i % N_update. 
5. Appendix A.1, second paragraph: geneerating -> generating. 

review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: This paper studies how a known, imperfect dynamics can be used to accelerate policy search. The main contribution of this paper is a model-based control algorithm that uses n-step lookahead planning and estimates the value of the last state with the prediction from a learned, soft Q value. The n-step planning uses the imperfect dynamics model, which can be cheaply queried offline. A second contribution of the paper is an efficient, iterative procedure for optimizing the n-step actions w.r.t. the imperfect model. The proposed algorithm is evaluated on three continuous control tasks. The proposed algorithm outperforms the two baselines on one of the three tasks. Perhaps the most impressive aspect of the paper is that the experiments only require a few minutes of "real-world" interaction to solve the tasks.

I am leaning towards rejecting this paper. My main reservation is that the empirical results are not very compelling. In Figure 1, it seems like the proposed method (MPQ) only beats that MPPI baseline in BallInCupSparse. The paper seems remiss to not include comparisons to any recent MBRL algorithms (e.g., [Chua 2018, Kurutach 2018, Janner 2019]). Moreover, the tasks considered are all quite simple. Finally, for a paper claiming to "pave the way for deploying reinforcement learning algorithms on real-robots," it seems important to show results on real robots, or at least on a very close approximation thereof. Instead, since the experiments are done in simulation and the true model is known exactly, the paper constructs an approximate model that is a noisy version of the true model. 

I do think that this paper is tackling an important problem, and am excited to see work in this area. I would consider increasing my review if the paper were revised to include a comparison to a state-of-the-art MBRL method, if it included experiments on more complex task, and if the proposed method were shown to consistently outperform most baselines on most tasks.

Minor comments:
* The derivation of the update rule in Section 4.2 was unclear to me. In Equation 17, why can the RHS not be computed directly? In Equation 21, where did this iterative procedure come from?
* "Reinforcement Learning" -- No need to capitalize
* "sim-to-real approaches … suffer from optimization challenges such as local minima and hand designed distributions." -- Can you provide a citation for the "local minima" claim? The part about "hand designed distributions" seems to have a grammar error.
* "Model Predictive Control" -- No need to capitalize.
* "Computer Go" -- How is this different from the board game Go?
* "Entropy-regularized" -- Missing a period before this.
* Eq 1 -- The "\forall s_0" suggests that this equation depends on s_0. Can you clarify this dependency?
* "stochastic gradient descent" -- I believe that [Williams 1992] is a relevant citation here.

----------------------UPDATE AFTER AUTHOR RESPONSE------------------------
Thanks for the detailed response!  The new experiments seem strong, and help convince me that the method isn't restricted to simple tasks. I'm inclined to agree that it's not overly onerous to assume access to an imperfect model of the world. Because of this, I will increase my vote to "weak accept."

In the next version, I hope that the authors (1) include multiple random seeds for all experiments, and (2) student the degree to which model misspecification degrades performance (i.e., if you use the wrong model, how much worse do you do?).
review_title: Official Blind Review #4

Meta-review: The authors develop a novel connection between information theoretic MPC and entropy regularized RL. Using this connection, they develop Q learning algorithm that can work with biased models. They evaluate their proposed algorithm on several control tasks and demonstrate performance over the baseline methods.

Unfortunately, reviewers were not convinced that the technical contribution of this work was sufficient. They felt that this was a fairly straightforward extension of MPPI. Furthermore, I would have expected a comparison to POLO. As the authors note, their approach is more theoretically principled, so it would be nice to see them outperforming POLO as a validation of their framework.

Given the large number of high-quality submissions this year, I recommend rejection at this time.
Decision: Reject