Venue: CoRL 2021 Conference

Reviews:

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: The paper provides a new framework for sampling-based model predictive control applied to robot manipulation. Contrary to previous MPC formulations, the authors employ MPC directly in the joint space, which allows them to explicitly incorporate collision or joint limit constraints. Planning is formulated as an MDP with continuous state and actions. To efficiently find a policy over a fixed time horizon in realtime, the authors use a simplified robot model by using forward dynamics and to leverage low-discrepancy sampling to find valid control sequences (policies) over the desired time horizon. Using several manipulation-related cost function, the authors demonstrate that their framework can be used to quickly plan joint paths for a fixed-base manipulator robot to reach dynamic target locations while avoiding obstacles. They also show that their framework can be applied to dynamical balancing tasks, where a ball is balanced on a tray.
strengths_and_weaknesses: Strengths

-- The paper is well written and easily digestable. The authors did a fantastic job to motivate the problem, I really enjoyed reading the problem definition, which was easily understandable even though I am not an expert on reinforcement learning. The method section is self explanatory and I was able to follow their line of reasoning. While some details remained missing or had to be put in the supplementary material, I think the paper itself provides all the informations necessary to understand the ideas.

-- The experimental evaluation is great. While the applications are challenging and useful, the authors even made the extra step of providing real-world experiments on a real robot system. The balancing skills of the ball on the tray is impressive and it is a good showcase for their method. The videos nicely supplement and demonstrate the usefulness of their approach.

-- Reproducibility is often an issue with papers. The authors tackled this issue by providing code in the supplementary material, which would make it easy for other researchers to reproduce their results. 

Weaknesses

-- It is clear to me that the authors have a deep understanding of what design choices actually work, in order to make the algorithm actually applicable. However, it would be great if you could (maybe in a later publication) try to provide more explanations of why
your specific design choices work. 

-- So far the algorithm has been demonstrated on a fixed-base manipulator. It would be interesting to see if you can scale this approach to even more complex systems, for example mobile multi-robot or other robotic systems.  

-- While the results are great, I would have liked to see more reactive scenarios, i.e. where you add obstacles on-the-fly, push the robot away or remove obstacles. 

-- Similar to having more reactive scenarios, I would have liked to see more limitations of the approach, i.e. explicit scenarios where it fails. 
I guess one example could be where the time horizon is too small and the robot takes a wrong path. It would be nice for other researcher to see how far this idea can be pushed and at what points new methods/ideas are needed.

summary_of_recommendation: The paper provides a novel method to achieve reactive manipulation using sampling-based MPC. The experiments are convincing and I think the whole approach holds a lot of potential for a broader range of applications. While I would like to see more reactive and limiting experiments, I think the paper itself is well-written, technically sound and should be a strong accept.
issues: -- Please explain how you would make this method more rigorous in the future, i.e. what kind of guarantees might you be able to give if a practitioner wants to use this method.

-- Consider talking more about limitations of the approach and how we can overcome them in the future. For example, you never showed that the model-based approximation performs worse at higher speeds.
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: This paper proposes an integrated system for sampling based MPC for fast reactive manipulation control. Authors proposed different cost terms to achieve better manipulation behavior, also increase the sampling efficiency of sample-based MPC. The approach is validated both in simulation and on a real robot, and also authors plan to open source thier codebase.
strengths_and_weaknesses: Strengths:
- A clear description of the system description
- A formulation to enable parallel computation of MPC on GPU, which make this MPC formulation work in real-time and on the real hardware.
- A sample efficient approach to consider a longer horizon motion than OSC, which benefits behaviors like avoiding obstacles.

Weakeness:
- As this is a sampling-based approach, it would have been better if authors can provide a quantitative comparison by varying the number of samples needed and the length of horizon. That would give readers a better idea how the approach performs in terms of the motion quality and the computation time by varying different essential parameters.
- For the cost term in 3.2.3 and 3.2.4, both cost terms rely on the neural network based functions. How the uncertainty introduced from neural network computation affects the algorithm? The experiments show the extrinsic evaluation of the cost terms, but it would be good to understand more intrinsically about the cost terms. Will there be any catastrophic collision due to incorrect computation from neural networks? If not, can it be guaranteed for such safety-critical terms.



summary_of_recommendation: In spite of some detailed concerns, I think this paper could have a significant contribution to the community, especially as a basic component for robot manipulation control. 
issues: - Videos are too lengthy, could be shorter with more caption to describe what happens (or increase the playback speed)
- The last paragraph of section 3 says the sampling can also sample from a fixed set of predefined action trajectories, but it is not clear to me which experiments that explicitly demonstrate this point.
- Does the ball dynamics consider the collision between the edge of the tray and the ball? While the result is impressive, I found the statement of ball didn’t fall in all ten episodes not quite accurate. From the supplementary video, the ball did not fall off the tray clearly due to the edge of the tray tray to stop the ball, and it doesn’t seem that  the forward model takes such a contact model into account.
- In the supplementary, I can only see the computation time compared against OSC. Are there any performance comparisons? It would be good to show two video clips to demonstrate the difference between STORM and OSC formulation. And if you introduce potential field terms in nullspace projection, can OSC also perform a reasonable obstacle avoidance?
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: This paper proposes a MPC sampling-based controller that can be applied to real robotic manipulation tasks. The authors introduce several subcomponents to ensure that the tasks are solved using smooth and collision free movements. The system is tested both in simulation and on a robot, and the results look very promising.
strengths_and_weaknesses: Strengths:
- The approximate model used for evaluating action trajectories is straightforward and efficient enough for high-frequency robotic control.
- The idea of penalizing the trajectories that cannot stop at the end of horizon can be very effective for producing more energy-efficient motions.
- The paper is well-written and organized.

Weaknesses:
- The paper does not explain how the learned components for collision avoidance are trained.
- The paper claims that one advantage of their system is that it can incorporate learned components in the control loop. But to my understanding, that is the same for all MPC algorithms and I have seen a lot of works that use learned components to improve the performance/speed of their control systems.
summary_of_recommendation: Overall, I like this paper. The approach is interesting and the results look very promising. I'm interested in seeing how the learned components are helping. Maybe the authors can show this in a new ablation study.
issues: - In Section 3.2.3, why do the authors need a neural network for predicting the closest distance between the links of the robot? Can't this be computed using forward kinematics?
- In Figure 3, there seems to be an error in the caption where it says that initial and desired positions are shown in green and red, where it seems to be the other way around.
- It would be nice to see the effect of the learned components in the form of another ablation study.

Typos:
- Line 107: represent -> represents
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The presented work introduces a set of elements to improve and fasten the performance of Sampling-Based MPC methods for high-dof robots. The proposed elements are: a novel sampling approach to guarantee smoothness action trajectories and good action space cover, integrating pretrained cost function and parallelizing the computation of the joint dynamics.
strengths_and_weaknesses: Strenghts:
1. The presented elements allows to boost the computational time of Sampling-based MPC methods 100 times.
2. Sampling Strategy: The paper proposes a novel sampling proceduce to guarantee smooth action trajectories and better action space cover while optimizing. The method seems interesting and they show qualitative results comparing the results sampling from normal distributions and using their proposed methods (Halton + B-Spline).

Weaknesses:
1. The paper lacks any reference to previous works trying to solve the problem of improving the computational efficiency of Samplign based MPC methods [1].
2. Approximate Model: The paper proposes to parallelize the integration of the trajectories. Given the initial robot state, a set of control actions, and given the discretized dynamics are linear with respect to the input, they parallelize the computation of the joint position and velocity trajectories. I do have two coments with respect to the approximate model: (i) It would be more accurate if the acceleration term also influence in the position, i.e. x(t) = x(0) + v(0)t + 0.5 a(0)t^2. and you can still parallelize the computation and (ii) This parallelization only works with linear systems. In case you want to model nonlinear dynamics, for example, the robot interaction with a certain underactuated environmental element, you require to evovle the dynamics sequentially. Also, I don't see any major novelty of parallelizing the evolution of linear dynamics. 
3. Self Collision Avoidance: Computing self-collision might be computationally expensive. Thus, the paper proposes to use a pretrained neural network that predict the closest distance between the robot links. While it is a very interesting approach, this network has been previously presented in (in paper ref.[23])  and the paper focuses on applying it and adding minor modifications.
4. An important missing point in the paper is a proper set of metrics to evaluate the benefit of each integrated element in the overall framework. The paper could follow an ablation study to evaluate the benefit of the self-collision avoidance element or a quantitative metric to compare their sampling strategy w.r.t. gaussian policy. It would also interesting to provide the mathematical equation of the ball dynamics? Is it linear? Can you parallelize it in the optimization?

[1] Pinneri, Cristina, et al., "Sample-efficient Cross-Entropy Method for Real-time Planning", (2020)

summary_of_recommendation: The work is a set of individual elements to boost the computational time fro sample-based MPC methods. Evaluatiing each element individually, there is no a big contribution or novelty in any of them. The proposed approximate model  parallelization is standard and won't extend to nonlinear dynamics. Additionally, they should consider the acceleration effect in the joints position. The self-collision avoidance is just applying a previously presented work. 
issues: The authors might focus on an ablation studies of the proposed elements, in particular it would interetsing an ablation study for the sampling methods. It is possible to define some qunatitative metrics to evaluate the performance of the Halton+B-Spline w.r.t. the others. Does it have an impact also in higher dimensional problems? Does it improve the jerkyness of the robot?
review_title: null

Meta-review: 
The paper provides a new framework for sampling-based model predictive control applied to robot manipulation. To efficiently find a policy over a fixed time horizon in realtime, the authors use a simplified robot model by using forward dynamics and to leverage low-discrepancy sampling to find valid control sequences (policies) over the desired time horizon. Authors proposed different cost terms to achieve better manipulation behaviour increasing the sampling efficiency of sample-based MPC. The presented work includes a set of elements to improve the performance of Sampling-Based MPC methods for robots  with higher DOFs. The paper does not explain how the learned components for collision avoidance are trained. The authors need to clearly explain how their work is different from the prior works mentioned in the reviewers’ comments. 



**quality**: Good, but some references to the prior works are missing.

**Clarity**:  Good: Minor revision needed to improve organisation or clarity.

**Significance**:  The integration of different components and real-robot experimentations are interesting, but the theoretical significance is not clear in the current version of the paper. 

**Originality**:  The novelty and contribution of the paper are questioned by reviewers. Clarifying this may improve the paper.  

**Post reviewer discussion** the response to the reviewer's comments clarified many points. 

Decision: Accept (Oral)