Venue: CoRL 2021 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The work proposes a TAMP framework that leverages functionals of SDF-based representations of objects. Learned functionals encode both dynamics constraints (based on one-step prediction) as well as kinematic success detectors, while analytic functionals encode problem constraints such as collision avoidance and contact establishment. The motivation of such a representation is twofold: (i) SDFs are a natural vision-based object representation, and (ii) models that reason using SDFs account for full continuous object geometry, as opposed to relying on pre-specified articulated points/surfaces. 

A mixed discrete-continuous optimization-based planner leverages the various functionals where the discrete optimization variable indexes “active” functionals and their relevant SDF operands, while the continuous variables correspond to the sequence of SE3 transformations on each SDF (object). Experiments are presented on two environments: (i) a static mug-hanging task, and (ii) a quasi-static pusher task. 
strengths_and_weaknesses: The motivation of using SDFs as the base object representation for dynamics models is compelling and the paper presents an encouraging attempt at a model-based planning framework utilizing such a representation. In the spirit of this novelty, I recommend that the paper is accepted. However, the authors may wish to address some of the following concerns:

1. Presumably, a potential advantage of using SDFs as the object representation is that one would be able to generalize to unseen objects, with similar *local* geometric features. The experiments involved training and evaluation over the same homotopy of objects, with variation only in size. Some experiments involving different object homotopies (shapes) would be extremely interesting.

2. The notion of a “dynamic” task is somewhat underwhelming - under the the author’s own classification, this is a quasi-static task. An example involving dynamic manipulation therefore would be more insightful to showcase the capabilities of SDF-based dynamics models.

3. The manipulation examples presented in the work constitute *passive* tasks - placing a mug on a handle, or pushing an object on a stable plane. Many of the challenging examples in manipulation involve at the very least, picking up/grasping and object and then performing a sequence of task-specific transformations. In such tasks, the *stability* of grasps and subsequent manipulations comes into play. Can the SDF representation handle such tasks or would one additionally require non-trivial controllability extensions? This is closely related to the previous point on dynamic tasks.

4. Computationally, there is a natural limitation to how SDFs are evaluated and processed through dynamics models - using pre-set grids. What then is the trade-off in computation speed vs accuracy of the learned model as a function of the grid resolution? Perhaps an encoder/decoder model could be jointly trained to reduce the dimensionality of the SDF representation used within the various functionals.
summary_of_recommendation: The motivation of using SDFs as the base object representation for dynamics models is compelling and the paper presents an encouraging attempt at a model-based planning framework utilizing such a representation. In the spirit of this novelty, I recommend that the paper is accepted. However, the main drawback concerns the imitations of the experiments. Two static/quasi-static manipulation examples are presented that are passively stable. Arguably, some of the more impact-generating examples would constitute dynamic manipulation tasks, involving at the very least, picking up objects. Potential for impact is high, given the elegance of SDF representations, however more compelling experiments are needed.
issues: From above, limitations (2) and (3) will likely involve a lot more work, and may not be suitable for inclusion within the revision period. However, limitations (1) and (4) (or at least (1) with some discussion on (4)) would be sufficient.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper modifies an existing formulation of task and motion planning (the Logic Geometric Programming framework) to incorporate constraints that are based on implicit surface representations of rigid-body objects. The paper formalizes what a dynamics model over Signed Distance Fields means, as well as other typical constraints present in TAMP problems, for example contact constraints, goal region constraints, as well as pairwise collision constraints. 

The motivating argument that the paper puts forth for TAMP with implicit surface representations is that traditional object geometries used in TAMP problems (eg. simple parametric shapes, or even meshes) are limited. In fact the paper claims that "it is unclear how these models can be grounded from sensor information." The paper mentions that the main advantages of SDFs are their ability to represent nonconvex objects, and that it is easier for them to incorporate sensor data, particularly depth data or pointclouds. The main underlying SDF representation assumed in the proposed network are neural networks trained from sampled points inside each object. 

I'd say the main contribution of the paper is an explicit formulation of TAMP with implicit surface constraints. There have been other related works that have followed similar approaches in the past using trajectory optimization, but not full TAMP.         
strengths_and_weaknesses: Strengths

S1. This is an interesting idea. Implicit surface representations are definitely well studied in graphics and animation, and the robotics community should pay more attention to them when it comes to motion planning.

S2. The paper is generally clearly written, with the exception of a few sections that I mention below. 

Weaknesses (* indicates major)

W1*. The reason presented as the motivation for studying SDFs over traditional parametric shapes used in TAMP is a bit misleading. It is unclear why robot perception would make learning SDFs any easier than parametric shapes or meshes. This is especially true when SDFs are represented with neural networks, which are generally much more difficult to update in an incremental/continual setting than shapes or mesh representations. Also, meshes make it easier to represent objects at different levels of fidelity according to the relevance of the object to the task at hand. I think the writing in the introduction should be updated to reflect more practical reasons why SDFs could be preferable.

W2. It is not clear how scalable this method is. The simulation experiments show very limited planning scenarios. How does this method scale when the number and volume of objects grows? 

W3*. The paper seems to be unaware of important prior work from the robotics community on this topic: https://motion.cs.illinois.edu/papers/WAFR2018-Hauser-SemiInfinite.pdf
https://motion.cs.illinois.edu/papers/ICRA2021_Zhang_SIPCC.pdf
These papers should not only be cited, but studied carefully for these types of problems. Other related papers include:
https://www.graphicon.ru/html/2003/Proceedings/Technical/paper495.pdf
https://mmacklin.com/sdfcontact.pdf
 
W4*. The TAMP problem formulation (2a-c) comes from LGP, but as far as I can understand, the paper resorts to gradient based optimization with multiple restarts, without making use of the discrete tree search that LGP makes use of for the symbolic state. This seems prone to local minima. Why was this choice made?

W5. The presentation of 2a-c is too notation-heavy. Please simplify it for readability.

W6*. SDFs are clearly promising for the dynamics prediction part of the problem. This is evident from the pushing experiments. However, they introduce an inner optimization problem when establishing contact between objects (eqn 10), so it is not clear why the added complexity and cost of including an optimization problem as a constraint is worth it, compared to other representations.        
summary_of_recommendation: I am recommending a weak reject. I think the problem is interesting and difficult, and more people in the robotics community should be looking at it. I think this paper needs a lot more work to provide a convincing solution to the problem, and the current experiments are too simple for the paper to have any impact, but it is enjoyable to read. It would be appreciated if more challenging planning environments could be included, possibly with more objects.  


POST-REBUTTAL UPDATE:
--------------------------------------

Thank you to the authors for addressing my concerns in such detail. I am upgrading my recommendation to weak accept. Best of luck!

issues: I would very much hope to see the weaknesses I listed as major addressed in the revised version. If I see convincing changes regarding the major issues, I am willing to update my recommendation to weak accept.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper aims to solve the TAMP problem for tasks that require kinematic and dynamic reasoning beyond the capabilities of analytical constraint modelling. Hence they propose to learn these models using SDF representations of different objects in the scene. The authors claim that this allows the model to learn various geometric and dynamic interactions implicitly which are necessary to solve the manipulation problems of interest.

strengths_and_weaknesses: Strengths

The authors have a good grasp on the current learning-based approaches and gaps in the literature which helps motivate their problem. 
The authors provide some interesting task constraint formulations that can be used for learning and solving two particular manipulation tasks with SDFs as inputs.
The method is described well and the formulation is sound.
The choice of SDF representations to capture models and formulate the optimisation problem is key in the framework. 

Weaknesses

While the method is sound and well-motivated, there is a lack of experimental evidence to justify the need for their method over any of the many existing ones for solving such problems.


summary_of_recommendation: The proposed formulation is interesting and well-motivated, however, due to weak experimental evaluation coupled with the incremental nature of the contribution it is difficult to justify publication.
issues: - Authors claim to propose a TAMP framework, however, it seems more suitable to say that they propose a learning-based approach to evaluating task constraints which facilitates task planning.
- While promising results have been achieved for the demonstrated tasks, there remains the technical challenge of how this method would scale to other scenarios. For example, for the proposed forward dynamics model, it seems non-trivial to have more than two interacting objects. Furthermore, the complexity of computing the interactions will increase non-linearly. Additionally, the training process seems quite lengthy and might be prohibitively slow for multiple objects, scenes and tasks. This could be further compounded depending on the size and resolution of the SDF.
- Dealing with the non-convexity of the problem via random restarts is unsatisfying and could be problematic depending on the complexity of the problem.
- Some grammatical errors. 
- Regarding the video, there is a lot of repeated experiments that seem similar and have no apparent added value. 
- Furthermore, it would be helpful to show the significance of your results by showing a comparison of your method alongside the failure cases of other methods.
review_title: null

Meta-review: This paper proposes the use of implicit SDF representations to enable planning with task constraints using learned dynamics functions on the object SDFs. The paper validates this approach with several planning tasks in simulation including bimanual robot experiments. 

The paper motivates the use of SDFs clearly for planning and shows the ability to learn accurate transition models using the SDF representation including learning the SDFs from segmented imagery. It further shows that SDFs perform better when compared to point cloud or occupancy-based binary representations.

The authors made substantial improvements to the paper based on the initial reviews and I find no lingering major issues with the work.
Decision: Accept (Poster)