Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: 
## Summary

The paper describes an algorithm to tackle model bias in the MPC. They address the question of optimal horizon length as well the model errors observed in MPC based systems. The paper is well motivated and written in clear concise manner. Experiments with cart-pole and robot models demonstrate the practical feasibility of the proposed method.

### Strong Points

1. Good description of the sources of errors in MPC based models
2. The Theorems are useful though I was not able to check their algebraic
   accuracy, the limiting constructs are intuitively correct. 
3. Choosing the horizon limit and tackling model errors are import challenges
   for MPC and the method proposed in this paper would be a good addition to the
   knowledge, hence recommendation for accept.

### To improve

1. I am not sure the MPPI is the SOTA baseline for this comparison, there are
   other MPC methods that achieve better results than MPPI.
2. Although citations from the machine learning community seem to be covered the
   standard MPC literature seem to be completely ignored. At the bare minimum,
   when speak of stability and fast horizon planning, no reference to tube based
   MPC [see Mayne 2011 ]
3. Major advantage of MPC is the ability to deal with constraints again see
   [Mayne et ref 2 below]. These are not recent developments but classic
   position papers that address lot of questions you pose and attempt to answer
   in the paper. 

4. The figures need a significant improvements. In their current form the plots
   are too thin to read them correctly. 



## Refs

1. Mayne, D.Q., Kerrigan, E.C., van Wyk, E.J. and Falugi, P. (2011), Tube‐based robust nonlinear model predictive control. Int. J. Robust Nonlinear Control, 21: 1341-1353. doi:10.1002/rnc.1758
2. D.Q. Mayne, J.B. Rawlings, C.V. Rao, P.O.M. Scokaert,
Constrained model predictive control: Stability and optimality,
Automatica,Volume 36, Issue 6,2000,Pages 789-814, ISSN 0005-1098,
review_title: Model based policy tuning akin to TD(\lambda) with model free base line.

final_score: rating: 5: Marginally below acceptance threshold
review_content: 
Blending MPC & Value Function Approximation for Efficient Reinforcement Learning
review:

summarization:

In this paper, the authors consider using a blending of Q value which is predicted directly from the neural network,
and a Q value which is predicted by unrolling the learnt dynamics.
The empirical results suggest improved performance, 
sample efficiency and good robustness in choosing different values of blending coefficient.

Pros:
1. The idea of blending the Q values is novel.
I also think the connection to GAE is quite natural and interesting,
where both algorithms consider trade off between bias and variance (in this paper’s case, bias in learned dynamics).

2. The supporting experiments consider some of the interesting questions.
For example in section 5.1, the question of how sensitive lambda is is addressed.

3. The direction of combining value estimation in model-predictive control is interesting and under-explored.
That being said, this paper can be inspiring and helpful towards future research.

Cons:

1. The experiment section lacks comparison among state-of-the-art algorithms.
While MPPI and PPO were generally considered state-of-the-art at the time when they are published (2017),
their performance is now outperformed heavily given the fast development in the research direction.
It would be great if some of the strong baselines between 2019-2020 are included (SAC, TD3, MBPO etc.).
Also given the similarity to MCTS algorithms, it would be more convincing to include one variant of it as a baseline.

2. Experiments on more environments are also appreciated. 

Questions:

I didn’t see study on the training time and testing time (how much time needed to generate one action during testing), but it seems to be referred to in the introduction?

review_title: Review for Blending MPC & Value Function Approximation for Efficient Reinforcement Learning

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary:

The paper provides and interesting analysis of and new method for model-predictive control in reinforcement learning. 
Specifically, it proposes a new framework MPQ(lambda) for joining model-based MPC with learned value estimates in RL.
The authors develop an formulation to find an optimal prediction horizon and and how this works in an online reinforcement learning framework.
The new approach is evaluated on 3 continuous control tasks and compared to some other baselines.

-----
Score reasoning:

This paper has interesting theoretical contributions to multiple areas of machine learning: model-based RL, MPC, and value estimation, but the somewhat limited experimental evaluation make the efficacy of the method more difficult to judge. Overall, the paper is well written and I enjoyed reading it. I now will address my conceptual comments followed by more minor suggestions.

-----
Rebuttal update: the authors have gone beyond the normal scope of a rebuttal phase to update their experiments and the motivation of the work, and for that reason I have improved my recommendation to be above the acceptance threshold.

-----
Experimental Validation Questions.
I am breaking this section of the review into it's own section because it is where the majority of my questions are.

E1) The authors bring up model-based RL algorithms, but do not baseline against other algorithms generally considered sample efficient. PPO is not always easy to use (authors mention it not converging, and not substantial parameter tuning), how about SAC? 
E1b) It would be very interesting to compare something similar to the PETS optimizer for MPC (cited in intro, but not really mentioned). These baseline changes could make the results much more believable.

E2) "All parameters were found using a coarse grid search" This makes the results suspect to me. Please clarify how coarse? Is the same search space used for all algorithms? Were defaults used for algorithms with previously published results? Do the results match?

E3) Was tuning of the reward functions done by hand in A.2.1, or are they referenced elsewhere? Are states like x-position and pole angle normalized in cartpole? This can have bigger effects in more complicated environments.

E4) "shaded regions represent standard deviation" for MPPI, is this over the same 30x3 evaluations? Very important to standard dev. is the number of samples.

E5) Cartpole swing up is a very similar task. It seems to be the only one where MPQ(lambda) substantially outperforms the baselines given (no error bars on MPQ too). How do the ablation studies of figure 2 reproduce on more challenging environments?

-----
Comments:
1) The authors refer to MPC as a "simpler, more practical alternative" to RL or a "more pragmatic approach" for "simple" policies. 
Some would argue that RL is a simpler approach because it does not require any model in the case of model-free RL. 
MPC also has many design decisions such as which optimizer to use or the planning horizon (multiple papers written on this topic). 
I would like the authors to explain this with more detail, or defend their stance.

2) The authors may consider including these two other papers that relate to model-based RL, model-bias, and MPC horizon https://arxiv.org/pdf/2009.09593.pdf, https://arxiv.org/pdf/2002.04523.pdf. 

3) How does model bias differ from model inaccuracy? 
In MBRL, model-bias often refers to the model being more accurate in some areas of the state-space than others, and how this impacts the downstream ranking of action choices. 
Do the authors consider this difference at all? How does model-accuracy drop when the bias terms are introduced in some basic metrics like mean-squared-error or negative-log-likelihood (metrics used in MBRL to quantify model-accuracy).

4) The position of the contribution in related works could be made stronger. I was unaware that MPQ was not the proposal of this paper until section 4. The difference between the two and why this matters should be in the introduction (unless the authors decide to add a dedicated related works section).
4b) how does entropy-regularized formulation impact the results? From my reading, that is an important part of the original MPQ paper, so I think it should be explained.

5) The conclusion to this paper is weak. It re-iterates what is done, but the authors should make a case how this impacts developments in robotics & control to better match up with the experiments and introduction. What should I take away from studying this paper? 

6) It would be interesting to see the authors propose how to combine the MPQ framework with other forms of MPC that don't have an implicit terminal cost included. This may be for future work, but I would be interested in a comment.

-----
Minor comments:
1) There are some typos that impede reading, but overall the paper is well written.
- intro, paragraph 2, "owing to its ability to" is weird
- section 2.2 "since it plans..." it is vague here
- some missing commas in first paragraph of section 3 "First..."
- Missing period at end of paragraph "Baselines", missing period Figure 3 end, double period before Conclusion
- Typo in PPO A.2.2 "The"

2) in 3.2, the authors show how to blend the model-based and model-free methods, but point to a reference that is not obviously connected to me and call the approach "common". 
I would suggest adding more references, or adjusting the claim.

3) Why was MPPI chosen as the MPC algorithm? It is a suitable choice, but could be added.

4) there is a lot of visuals in Figures 2 and 3. Maybe have fewer lines? The font could be enlarged and it is very confusing that the y-axis's are not all the same for similar data types.
4b) Figure 4b) has strange shading from the MPPI variance - it's not readable.

review_title: Interesting paper blending MPC with RL, limited by experimental evaluation.

final_score: rating: 6: Marginally above acceptance threshold
review_content: ##
**Summary:**
The paper proposes to combine MPC and model-free RL to overcome the possible modelling errors. Thereby the approach achieves the sample-efficiency of MPC and the control quality of model-free RL. The resulting MPQ(\lambda) algorithm uses MPPI to obtain the actions by optimizing the blended MPC objective. The Q-targets for fitting the q function also use the blended Q-estimate.


##
**Quality, Originality & Significance:**
The idea of combining MPC and model-free RL is straight forward and not novel (the paper also does not claim this). However, the exact instantiation is novel, very well motivated and feels natural. My biggest concerns are the experiments. The cartpole experiments show the improved performance compared to MPPI on the biased model and the impact of the lambda and model bias. However, the PPO baseline is missing for the cartpole, right? Furthermore, is PPO a fair comparison for the MPQ($\lambda$) algorithm to evaluate sample complexity? While PPO is a batched update, the MPQ($\lambda$) uses step based updates. Wouldn't a model-free step-based update algorithm such as DDPG, SAC etc. be a better baseline to evaluate the improved sample complexity? Regarding the high-dimensional tasks, the provided evaluations do not enable an evaluation whether the task is solved or not. Could you please provide videos of the final policies, otherwise the achieved reward is just a random number. Furthermore, the paper shows confidence bounds for the MPPI baselines but not for the MPQ($\lambda$) algorithm. Also the learning curves are cut before converging, could you train every instance until convergence? Furthermore, could you please include the asymptotic performance of your baseline in the plots. The definition of 'validation iteration' remains unclear. Given the current evaluations the stated claim of applicability to high-dim tasks cannot be made as the evaluations are not sufficient. The used modelling bias is also very limited as the paper only compares to biases of the model parameters but not against other sources of biases. Ultimately the increased performance can only be shown on the physical system.

##
**Clarity & Style:**
The paper is really well written and understandable! A few sections could be improved, e.g., text between Eq. 6 & Eq. 8. In this section it is a bit unclear what is expanded and how it is expanded. It would be beneficial to rethink the labeling of the Q-functions as they can be quite confusing. Maybe a table of the different subscript/superscript definitions would simplify the reading, as I had to search for the exact definitions frequently. Furthermore, there are minor styling issues:
* Inline equations are consuming too much space to mess up line spacing, e.g, Section 2.1 argmin, Theorem 3.1, norms,
* the min in Equation 13 needs two spaces and a subscript
* Experiment O4 ends with two dots
* White space around figure 2 can be optimized

##
**Conclusion:**
All in all the paper is nicely written with a clear and well motivated idea of combining MPC & model-free rl. Right now the main problem is the execution of the evaluations. The performance on the high-dim tasks is unclear and the baseline is missing for the cartpole. I would be happy to improve my score to
* **weak accept** if a step based model free RL algorithm is added to cartpole, claims regarding high-dim tasks are adapted and videos of the high-dimensional tasks are released.
* **accept** if the high-dimensional tasks are working properly with MPQ($\lambda$)
* **strong accept** if MPQ($\lambda$) shows this performance on a physical system


P.S. You might also try to get medium dimension tasks working such as hopper or cheetah. That might be a bit easier. 

##
**Post Discussion Comments:** So the author did a **filibuster** and **flooded the discussions** with bloated comments. In this manner it was close to impossible to keep track of anything. **There has to be character limit for responses otherwise this is not feasible**. I looked at the videos and your physics simulators looks **really catchy**. At one point in time, the pole of the cartpole is at 10-11 o'clock and the cart starts moving right (the pole has close to no velocity and hence only a very small angular momentum). In this setting it would be natural that the pole would fall down if this state is maintained for a longer period (which it is in the video). However the pole goes upwards into the balancing position. This is really weird. And don't get me started on the pen-orientation as the pen sometimes floats mid-air. For this setting the gravitational constant really does not seem right. I also wouldn't consider the task solved as this is more an really uncoordinated movements for three specific configurations. 

For the simulation studies some doubts remain, but the authors improved the paper. Therefore, I am going to increase my score to weak accept. Nevertheless, the experimental evaluation could be improved and the paper would really benefit from real experiments. 
review_title: Neat idea and clearly written paper but evaluations can be improved

Meta-review: The authors put a lot of effort in replying to questions and improving the paper (to a point that the reviewers felt overwhelmed).

Pros:
- An interesting way of dealing with model bias in MPC
- They successfully managed to address the most important concerns of the reviewers, with lots of additional experiments and insights
- R3's concerns have also been successfully addressed by the authors, the review & score were unfortunately not updated

Cons:
- The only remaining point is that the simulations seem to be everything but physically realistic (update at end of R1's review), which is probably a problem of the benchmarks and not the authors faults.
Decision: Accept (Poster)