Venue: CoRL 2021 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper presents an algorithm for learning a precondition for several transition models with different accuracies and speeds. The main intuition is that because these transition models must trade-off their speed with accuracy, if we can determine the model that would give us a small enough error, then we can choose the one that has the fastest speed. The precondition for model M is defined as the threshold testing on the distance between the true $s_{t+1}$ and the one predicted by M; if this deviation is small enough, the state and action parameter pair is in the precondition of M. The distance function is trained from planning experience using regression, and experiments demonstrate that by using this approach, we can reduce the planning time and achieve higher success rate than the single model approach that uses an analytical or high-accuracy (therefore slow) model alone.
strengths_and_weaknesses: I very much liked the intuition and approach that this paper takes. In manipulation planning, each skill needs to be simulated, and the model must trade-off accuracy with speed. This paper proposes to *learn* how we can make such a trade-off, which I found to be very interesting and novel. However, its weaknesses are the unclear articulation of assumptions and details.
summary_of_recommendation: 
While I liked the overall goal and the direction of the paper, I have found several places where this paper should improve to be published.

- It needs to precisely articulate when this approach can be used. It seems that this approach cannot be really used with trajectory optimization algorithms, because they must commit to it. Is it applicable, then, to all tree-search methods? Or is it applicable only to the planning algorithm shown in 4.3?  
- It is missing the details relevant for learning, especially the feature representation. It does mention using relational features, but looking at the experimental results, I do not fully grasp why relational features, which I presume to be first-order logic-based, are necessary. Also, is the representation tailored only for the given skills and tasks? Or can we apply it in the general case?
- It seems that the algorithm would heavily depend on the actual distance function used. For example, if your state involves angles, then Euclidean distance would not suffice. What exactly are your state representations in your tasks, and how do you suggest that the users define their distance functions?
- I found it to be extremely inefficient that it has to solve the same planning problem many times with different models to obtain the training data. Why can't it generalize from other problems? How different are problems in terms of state and goal representations?
- When the model is inaccurate, one approach you can use is model predictive control. How does this algorithm compare with MPC-like algorithms?


issues: Please see the summary of recommendation.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This work proposes a method for learning model deviations to define preconditions on the predictive models and thus inform the use of these models during task and motion planning. The problem is broken into two steps: training an "model deviation estimator" (MDE), which estimates model deviation for a given state, action, and model and then using the MDE to drive the TAMP search towards states which can be expanded using faster but less accurate models when possible. The paper evaluates the proposed framework in the context of 2 real-world manipulation tasks. 
strengths_and_weaknesses: The problem statement is well motivated and interesting. The paper is mostly clearly written (though there is a list of recommended fixed included below). The approach taken is interesting, i.e., by using MDE regression instead of classification, and by informing model preconditions rather than skill preconditions. A brief comparison to [1] would be worth adding, however. The experimental results look promising and motivate further exploration of the approach. 

There are some concerns about the experimental setup, however. The lack of an acceptable analytical model for the OpenDrawer task makes for an unsatisfying comparison. Furthermore, by showcasing skills which are so obviously contrasting in predictability (Pick vs LiftAndDrop) it is unclear whether the approach would offer as much time savings given a more diverse set of action primitives and models. In general, the current experimental setup has few skills and considers very few models. 

The idea of using a Multi-Heuristic A* in the context of model selection has been explored in some of the related work, e.g., reference 14 in the paper. Clearer statements about what is different in this work relative to the prior efforts that have followed a similar strategy would strengthen the contribution. 

Understandably, data collection for such an approach is likely a bottleneck and the authors acknowledge and use some basic data augmentation tool to improve the situation. Given the lack of sufficient experimentation, a discussion on how to more efficiently gather data for more complex setups for this approach would be helpful.

Proofreading:
Line 125: awkwards sentence doesn't make sense; should it be "... by computing distance ... from predicted state ... and actual state ... using d"?
Equation 1: unmatched parathesis
Section 4.2: Mention which regression models were used for training MDEs or reference section 5.
Line 174: Shouldn't it be highest value of j?
Line 254: Model acronyms used here before they are introduced in the next section.
Figure 4: What are the threshold units in the graph? I assume they are meters but it would be cleaner if written in cm given the text?
Figure 5: Caption too close to text.

[1] Kim, B., Wang, Z., Kaelbling, L. P., & Lozano-Pérez, T. (2019). Learning to guide task and motion planning using score-space representation. The International Journal of Robotics Research, 38(7), 793-812.

summary_of_recommendation: Overall the work proposes an interesting iteration over previous techniques for speeding up TAMP using estimation of model uncertainty but could be stronger in evaluation and better clarify the contribution to some of the related work. 
issues: A discussion on how to more efficiently gather data for more complex setups for this approach would be helpful. Better support why the current limited experimental setup (in terms of considered skills and models) is sufficient to argue the benefits of the framework.
review_title: null

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper presents a method for planning robot manipulation tasks using multiple forward models of different fidelities. The authors propose learning to predict the expected total error of a model's prediction for a given state and action. The authors then propose a planning method which simultaneously rejects actions which have high predicted error and biases the search towards using lower fidelity models. The approach is demonstrated on two experiments each with two models, showing improvements in planning speed over using only a high fidelity model, while remaining competitive in the success rate of the generated plans. 
strengths_and_weaknesses: Strengths
-The paper is well motivated, planning with multiple imperfect models is an area of great interest to robotics. 

-The planning method utilizing different models seems quite a sound idea. It is also an interesting idea that we might want to preferentially use different models for different situations. 

-Experiments demonstrate that the method is useful for speeding up planning without compromising performance on a real robot with a very small dataset. 

Weaknesses
-The main weakness of this paper is the clarity of writing and presentation. The paper would benefit from thorough revisions and editing for clarity in all sections. A few examples are described in the “Issues” below.

-The insight that model error underestimation may be better than overestimation is quite interesting and the proposed solution is straightforward. However, it is not clear the method which uses this assumption actually improves performance. Experiments are necessary to verify that underestimation is indeed better.

-The experimental results would benefit from an ablation. For instance, it is unclear why the second queue, which only expands using the simplest model, is necessary. The first queue already preferentially expands using the simplest model which satisfies the precondition, so does this not provide enough of a bias toward using the simple model?

-The author's planning framework could very easily incorporate using a binary classifier to predict if the action is in the precondition of the model. The authors claim their method of predicting total state deviation should be ‘more transferable across tasks’ (line 28), however this claim is unsubstantiated.

-I do not think the discussion and comparison of the different models (GP vs MLP etc.) used to learn the MDEs to be particularly useful or relevant to the main contributions of the paper. 

-The tasks used for the experiments are very simplistic and very simple models of pick, place, and drawer-slide actions can easily be used to accomplish them. In fact, such tasks have been done with kinematic motion planners for several decades. A task where the robot is manipulating an object that has interesting dynamics (which actually requires a high-fidelity simulator, at least sometimes) would show the benefit of this method much more clearly.

summary_of_recommendation: Overall the idea of using multiple models to efficiently find plans for manipulation tasks is interesting and the approach has some technical novelty. However, the tasks used in the experiments are very simple and can easily be done using traditional pick/place/open-drawer actions generated by existing planners with very simple models, thus the usefulness of this method over state-of-the-art is not clear. Also, the paper is very hard to read and the clarity needs to be significantly improved.
issues: -The caption of figure 1 describes a figure on the left which does not appear to exist

-Several points in the paper, the notation switches from M0 being the high fidelity model and MK being the low fidelity model  to the other way around. (For example figure 1 vs line 96 vs line 177). This makes it very difficult in section 4.3 to understand exactly which model is being used when, I am left having to guess the author's intentions. 

-I had a great deal of difficulty understanding section 4.3, particularly the partial expansion from open 1. Some sort of graphical explanation / a figure or algorithm might be very useful for getting the information across. 

-Videos could use some work. First caption of the planning explanation video is very difficult to parse for instance. 

-For the videos of the real robot experiments, the video for the first experiment doesn’t seem to complete, I am unsure what to make of it. The Video for the second experiment doesn’t include baselines. If time is an issue I would recommend changing the video speed for the experiments to be 4x real time (or something). 

-The fact that there is always a high fidelity model to fall back on means that in the worst case it can likely find a good plan albeit slowly. The results for the drawer experiment showing that the analytical model can be used even when it completely fails to account for the articulated object is very interesting. Could this framework be used if you instead had two low fidelity models with different limitations? This would be really interesting to see. 

-What happens when you use your framework using only 1 model? This should only expand nodes for which the action satisfies the pre-condition of the model, does this offer improvements over naively planning with the model?
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper presents an approach to reduce planning time by trading off between planning time and model fidelity to solve manipulation tasks. It’s assumed a collection of model-based predictors of varying fidelity are available (e.g. analytic model, learned model, physics-based simulator). Each predictor takes as input a state, action, and parameter vector (for a parametric skill) and predicts the next state that will result. The approach learns models (called model deviation estimators (MDEs)) that predict how much each predictor model will deviate from actual execution. The models are trained based on data collected from executing pre-defined skills in the environment (pick, lift-and-drop, open-drawer). A modified A-star algorithm is defined that incorporates the learned model deviation as a cost so that the search can favor using the least fidelity model it can get away with (i.e. it only needs to use the time-expensive high-fidelity simulator to model more complex tasks like opening a drawer and can use an analytic model for simpler motions pick-and-place of an object). Experiments are performed on a real robot for the tasks of picking and placing a rod into a box, and opening a drawer to drop the rod in the drawer.
strengths_and_weaknesses: Strengths:
+ The high-level idea of trading off between different models for planning and using simpler models when possible has high utility, enabling redirection of compute power to where it is actually needed (e.g. modeling complex object interactions).
+ The description of the problem setup and method were clear and easy to follow.

Weaknesses:
- The models predict deviations based on regression to point-based data, and predict point estimates. This means if a particular skill produces a wide distribution of outcomes given the same inputs, the deviation predictors will average over the outcomes and not predict a useful deviation. The skill of dropping an object can have large variance, depending on the height of the drop and the state of the environment below the object where it’s being dropped. This can be very sensitive to initial conditions, so the object might bounce drastically in different directions even though the same action and environment configuration are repeated several times. The deviation estimator models will only be effective when there is a tight distribution of resulting states on executing a skill.
- Leaving out the drawer from the analytic model entirely is a strawman comparison. The motivation is to necessitate using a higher fidelity model than the analytic one, but if the analytic model is in fact insufficient to generate a plan, that should happen anyways. Presumably the simulated drawer will be modeled with friction and other dynamics parameters that the analytic kinematic model would ignore, so it could still offer benefits over the analytic. But I think a kinematic model of the drawer should be included also in the analytic model, since the drawer (and its parts) are part of the state/goal space. If the analytic model gave a similar (or better) success rate for planning, it obviates the need for the higher fidelity models, and a more compelling experimental task should be used that necessitates high-fidelity simulation. In short, it’d be more compelling if Figure 4 wasn’t missing any bars.
- An important baseline is missing: randomly select at certain intervals whether to use the analytic model or the simulator model, and compare that performance to the MDE approach. It might be that a random mixing of analytic and simulator models is just as effective as the more complicated learned MDE approach.
- It would be more insightful to know at which points in each task the analytic or simulator models are invoked when running the MDE approach. This could demonstrate, for example, that point-to-point motions are easily modeled by an analytic planner while contact-rich manipulations require a more complicated simulator, which would coincide with our intuitions.


summary_of_recommendation: The concept of mixing models of different fidelity is a useful idea. However, the formulation of the deviation estimators seems susceptible to failure when skill outcomes have high variance due to environment conditions that are not controllable by the robot (e.g. object dispersion on dropping an object). Further experiments would be required to demonstrate this is not an issue. Further, a key experiment is missing, which is comparison to simpler mixing strategies of using an analytic model and a more complex simulation model, such as simply randomly selecting between the two. It may be that naively switching between analytic and simulation models is similarly effective, and additional experiments are necessary. More insightful analysis can be done regarding where in the task execution different kinds of models are invoked.

UPDATE: In view of the authors' additional experiments, I have updated my scores accordingly. The additional experiments with the analytical model for the drawer and the comparison to a random selection baseline address my main concerns with the experiments. There have been significant revisions in the writing of the paper as well that I feel better communicate the approach and contributions.
issues: Do the timing results in Table 2 include plan attempts that completely timed out at the 5 minute limit? If only one run resulted in the simulator plan timing out, it seems that could drastically skew the results to make both the simulator and MDE planners have higher planning times. For the MDE planner, it would be helpful to include counts of many times it used the analytic planner and how many times it used the simulation planner for each plan produced by the MDE planner.

Simply finding a plan in a certain amount of time is not always the only consideration, the optimality of the plan in terms of shortest path or time optimality may also be a factor. Is there a significant difference between the plans resulting from the analytic/MDE/simulator? To demonstrate this, it would require executing generating multiple plans from each and analyzing the path and execution of each.

The axes in Fig. 4 should be normalized to be the same (in range [0,1]), currently the right plot axis does not go to 1 which can result in a confused interpretation.

Why are the property values so different between Table 3 and 4 in appendix? It’s the same robot being modeled in both tasks, and the tasks are not drastically different (both rigid body manipulations), but the joint stiffness/damping values are very different and they use different physics backends.
review_title: null

Meta-review: This paper presents a method for speeding up planning by switching between models at different levels of accuracy.

Quality

(-) Reviewer MmFq and iJpM suggested additional experiments to better evaluate the performance of the proposed method

(+) Experiment with real robots

Clarity:

(-) Reviewers wM8w, SzMb and MmFq suggested many points that should be clarified

Originality:

(+) The proposed approach is novel

Significance:

(+) Several reviews agree that the topic is relevant and impactful

Additional comments:
- I am not sure I understand why do you need the \theta in the formulation of the forward dynamics (line 95)? in an MDP the distribution over the next state is uniquely determined by the current state and action.

---

After rebuttal:
The clarity of the paper was improved and additional experiments were included.
However, Reviewer MmFq is still not fully convinced by the rebuttal.
Please continue to address all remaining feedback.
Decision: Accept (Poster)