Learning Models as Functionals of Signed-Distance Fields for Manipulation Planning
Authors: Danny Driess
Jung-Su Ha
Marc Toussaint
Russ Tedrake
Venue: CoRL2021 Poster (2021)
Keywords: Manipulation Planning
Signed Distance Fields
Model Learning
Implicit Models
Dynamic Model Learning
Implicit Object Representations

Abstract: This work proposes an optimization-based manipulation planning framework where the objectives are learned functionals of signed-distance fields that represent objects in the scene. Most manipulation planning approaches rely on analytical models and carefully chosen abstractions/state-spaces to be effective. A central question is how models can be obtained from data that are not primarily accurate in their predictions, but, more importantly, enable efficient reasoning within a planning framework, while at the same time being closely coupled to perception spaces. We show that representing objects as signed-distance fields not only enables to learn and represent a variety of models with higher accuracy compared to point-cloud and occupancy measure representations, but also that SDF-based models are suitable for optimization-based planning. To demonstrate the versatility of our approach, we learn both kinematic and dynamic models to solve tasks that involve hanging mugs on hooks and pushing objects on a table. We can unify these quite different tasks within one framework, since SDFs are the common object representation. 
Video: https://youtu.be/ga8Wlkss7co

Learning Models as Functionals of Signed-Distance Fields for Manipulation Planning
Danny Driess
!
Jung-Su Ha'
Mare Toussaint'
Russ Tedrake²
'TU Berlin, Germany
MMasachussetts Institute of Technology, USA
This work proposes an optimization-based manipulation planning framework where the objectives learned functionals of signed-distance fields that represent objects in the scene, Most manipulation planning approaches rely on analytical models and carefully chosen abstractioons/saat-spaaes to be effec- tive. A central question is how models can be obtained from data that are not primarily accurate their predictions. but, more importantly, enable efficient rea- soning within a planning framework, while at the same time being closely coupled to perception spaces. We show that representing objects as signed-distance fields not only enables learn and represent variety of models with higher accuracy compared to point-cloud and occupancy measure representations, but also that SDF-based models are suitable for optimization-baseee planning. To demonstrate the versatility of our approach, solve tasks that involve hanging mugs on hooks and pushing objects on a table. We can unify these quite different tasks within one framework, since SDFs are the common object representation. Video: https: attps://youtu. be/ ga8wlkss7co
Abstract:
learn both kinematic and dynamic models
Keywords: Manipulation Planning, Signed Distance Fields, Model Learning
Introduction
Manipulation plamning is challenging for multiple reasons. On the one hand, planning robot motions to solve a task can be formulated as decision problem over a high-dimensional, non-convex space, including discrete and continuous aspects. Especially long-horizon tasks that consist of multiple manipulation steps have the property that motions have to be coordinated globally with the future goal. This coupling of potentially all variables requires joint reasoning and makes the problem particularly challenging [1]. On the other hand, the problem solving capabilities of a planning framework is inherently dependent on its underlying models. The field Task and Motion Planning (TAMP) has made significant progress in solving challenging multi-step. long-horizon tasks [2], ranging from ones that involve mainly kinematic models [3, 4, 6. 7] dynamic tasks that require reasoning about forces, friction etc. based on more general dynamic equations [8, 9, 10. 11, 12, 13. 14]. However, most TAMP approaches rely on carefully chosen abstractions and analytically defined models in order to be successful and efficient. In particular, TAMP often makes simplifying assumptions on the possible geometries of objects it can deal with to define manipulation constraints in the first place. It is unclear how these models can be grounded from sensor information.
To overcome these issues, a natural idea is to replace the analytic models in TAMP frameworks with learned ones. Recent advances in deep learning have enabled learn predictive forward models even in high-dimensional observation spaces like images. The typical objective for learning a forward model is its predictive accuracy. However, having an accurate model does not necessarily imply that planning framework can utilize it efficiently. While having accurate forward prediction model might be sufficient for short-horizon tasks, especially for long-horizon tasks, learned models can exhibit too high combinatorics for sampling or non non-informative gradients for achieving future goals.
This paper aims to address these challenges by learning models that can be used effectively by planning framework while at the same time using a general object representation more closely TAMP framework where the
lated to semors spaces. To realize this, we present an optimization- objectives are learned functionals of signed-die -iinedddsstane fields (SDFs). The SDFs represent each object in the scene separately, while the functionals defined top of them induce constraints possible, physically plausible interactions between the objects within a trajectory optimization problem. The task planning aspect is realized by (discrete) decisions that determine which of those functionals are active at which phase of the planning horizon.
5th Conference Robot Learning (CoRL 2021), London, UK.

---

We argue that representing objects as SDFs has multiple advantages. First, an SDF can be seen as intermediate representation between raw perception like point-clouds images and full state information. While not the focus of this work, many methods have been developed to learn and obtain SDFs from, e.g., image observations of the scene. Further, SDFs can represent arbitrary, non- convex geometries, which is beneficial, since manipulation problems and physical phenomena often depend on the geometry of the interacting objects. Finally, we show that SDFs are particularly suited for learning and representing models that can later be used within a planning framework effectively. Since our models are functionals of the SDFs, the constraints can take the information about whole objects into account to reason about their geometry and therefore especially the interaction between objects. Compared to a representation that only describes the surface of an object like point clouds or occupancy measures, signed-distanee As we experimentally show, this not only leads to models that perform better in their prediction accuracy compared to models learned on top of point-cloud or occupancy object representations, but SDFs also enable the functionals/leamee models to have more useful gradients for planning.
field also provides information about the object at distance.
In the experiments, we demonstrate the versatility of our approach by tackling two completely dif- ferent tasks within one framework: On the one hand, a kinematic task where the goal to hang mugs of different shapes on hooks of different shapes. On the other hand, a pushing scenario where boxes and L-shaped objects should be pushed to different goal regions by pushers of different sizes. In the first case, the model predicts whether the static interaction between SDFs leads manipulation success, whereas in the latter case, the model predicts the forward dynamics in SDF space based on a history of SDF interactions of two objects. We show that our framework can be used to plan motions that involve multiple push phases. To summarize our main contributions, we propose
To learn a novel class of kinematic and dynamic models as functionals of SDFs,
* A manipulation planning framework that utilizes these learned functionals as constraints,
Comparison to other object representations showing the advantages of the SDFs.
2 Related Work
2.1 Signed Distance Fields as Object Representation
Representing objects or scenes as implicit surfaces [15, 16, 17] or SDFs [18, 19, 20, 22, 23] is an active research topic, due to aforementioned advantages like learning shape completion, non-convex shapes etc. Our focus is not to obtain SDFs from observations in the first place. Conversely, we are interested in what can be done with SDFs in the context of model learning and manipulation planning. There are some works that utilize SDFs within trajectory optimization [24, 25, 26], but without learning or integration into a TAMP framework. While some recent approaches [27, 28, 29] have suggested that grasping of diverse objects can be addressed using implicit functions, we present a manipulation framework that utilizes SDFs for learning and formulating more general models.
2.2 Perceptual Models
There is great interest in learning predictive models in perception spaces, especially applied to the problem of pushing. So-called visual foresight approaches [30, 31, 32] aim to predict the evolution of the scene in image space. Our SDF dynamics model is also elosely related to perception spaces, but, in comparison, is naturally differentiable. Xu et al. [33] use a voxelized SDF-based representa- tion of the whole scene to predict the motion of an object when an action is applied. Our approach more structured in the sense that we do not predict the scene flow for actions applied on a sin- gle object, but the dynamics of interacting of objects. In [34], the pushing dynamics in keypoints extracted from visual object observations is learned. However, their focus is to utilize the learned model to stabilize trajectory with MPC. We focus on planning complex pushing trajectory and not stabilization during execution. SE3 networks [35] learn a forward model that predicts a rigid transformation of an observed point cloud given actions. However, they need ground truth transfor- mations at training time (we only need SDF observations). Where most of these approaches differ from our approach that they assume the model to be function of the observation of a single ob- ject or the scene and an action as input. Therefore, these approaches are mostly limited to the same pusher geometry and make the assumption that actions can readily be applied to the object. Our model handles the interaction between objects of different shapes and can plan the contact estab- lishment phase as well. Transporter networks [36] or deep visual reasoning (7] predict manipulation sequences from image spaces. However, no dynamic models are considered in these approaches.

---

2.3 Manipulation Planning (with learned models)
In [37, 38), a manipulation framework based on point cloud observations and manipulation prim- itives is proposed. Our method plans the complete motions based on learned dynamic models. Sutanto et al. [39] is related to our formulation in the sense that they learn manifolds that are used as constraints sequential manipulation problems. However, there are no dynamic models depen- dencies on the geometry of the involved objects in the learned constraints. You et al. [40] address hanging task similar to our mug hanging experiment on a more diverse set of object categories. They use point-cloud-basee input representation to predict hanging pose. Therefore, they need special neural network for collision avoidance (similar to [41], while our SDF based representation can handle collisions directly. Further, we learn manifold of solutions instead of predicting a single hanging configuration. To summarize, what makes our approach unique is that propose to use SDFs as a common object representation that is closely connected to perception learn a variety of models that are able to take the interaction of objects into account and can be integrated in an optimization-base motion planning framework due to their differentiability.
3 Background on Signed-Distance Fields (SDFs)
Let Ω C RR3 be an object in the 3D Euclidean space. A function Φ R3 O Φ with <(x) = -d(x, 02) for x E 1 and ((x) = d(x,, 0Q) for x E R is called a signed-distance field of in R Here, d(x, 092) en || - x'112 and an the boundary of St. We assume & to be differentiable almost everywhere in R The way & is defined ensures that inside the object, attains negative values, on the boundary zeros, and outside positive ones. We denote with the set & the space of all functions é that are SDFs for some object.
Rigid Transformations of SDFs A central concept this work is to rigidly transform SDFs in space. This can be realized by transforming the input where the SDF is queried. To simplify the notation, we define rigid transformation, parameterized by g E R2 (translation quaternion),
T(W[)(() = o (R(A)" (- ------ -r()))
(1)
of an SDF o, where R(q)= E g3x3 is rotation matrix and r(g) E R the translation vector.
4 Manipulation Planning with Signed-Distance Functionals
The core idea of this work is to represent each object i in the scene as signed-distanee field d' in order learn predictive models as functionals H of these SDFs. Based on the learned functionals, we formulate a trajectory optimization problem where the decision variable is a trajectory of rigid transformations applied on the initial SDFs as they have been observed in the initial scene.
More specifically, through interaction with the environment, form H 4
aim to learn functionals of the x & + E that map multiple SDFs of multiple, possible different objects at possibly different consecutive times to a real number. These are trained in a way that a value of zero implies that the SDFs as input are compatible with what has been learned through interaction with the environment. Otherwise, they should attain positive value, hence functionals H discriminate correct from incorrect dynamics or desired from undesired manipulations.
The learned functionals then define constraints for the (hybrid) trajectory optimization problem
KT cl Sk(1))
min do ERTTIEERTTTT K KEN,
Σ
(2a)
TO Am1
s.t. VHEN(sa))) : H ((T()))o') (L))Z (useetlo)
(2b)
$1:K E S(S), 9o =
(2c)
The discrete variable 8 determines which functionals H from the set H(w()))) are active which of the K E N phases of the motion (k(t) [t/T]). This number of phases is part of the decision problem. The trajectory 97:K7 of rigid transformations
discretized in time into T E N steps per phase. If no is the number of objects in the scene S, then S. 7 leading continuous variables. Further, 8 selects through the set IH (Sx(e)) the time slice and object index tuples (t. i) that determine the SDFs Øi, which have been transformed through of at the times t of the trajectory on which the functional constraints H depends on. This problem formulation is
7 KTno
3

---

inspired by LGP [10], but the constraints are replaced by learned functionals of SDFs. The set S(S) contains all valid sequences S1:K of such discrete variables for the scene S. The goal of the manipulation problem is specified through S(S) by 8* selecting a desired goal functionna constraint that has to be fulfilled at the end ęKT of the trajectory. Solving (2) therefore involves a tree search over nodes $1:K such that the continuous optimization problem implied by the choice of $1:K at node of the tree is feasible. The role of q in the optimization problem is not absolute object poses, but rather rigid transformations applied to the SDFs ø' that represent the configurations of the objects as observed the scene initially. With the term c, we can include regularizing motion costs. As will be described in sec. 5.1, the forward dynamic model we learn for pushing implies a constraint on the evolution of one object based on the motion of another object. Therefore, we only add motion costs to those degrees of freedom that can be interpreted as being controlled, meaning the motion of the other object. From the perspective of (2), there is no explicit notion of controlled actions.
Deep Signed-Distance Functionals
This section presents two main types of models we propose. First, way of learning forward dy- namic models that predict the dynamics in SDF space based on the interaction between objects. Sec- ond, a kinematic success model that determines whether a static configuration of interacting SDFs leads to manipulation success. All functionals we consider are of the form H ; 4 x ... x ① + I. i.e. they only take the SDFs of interacting objects as input, there is no explicit notion of position, orientation, action etc. Therefore, the functionals can be used at arbitrary locations in space.
Bounding-Boo To define most of the following functionals and those in sec. 6, we utilize a set X' with the property Q C X' C Rd for all objects Q that are involved. This set should be large enough to cover the relevant workspace of the manipulation problem where the interaction between the objects should occur. A more detailed discussion about the role of X' can be found in sec. 5.3.
5.1 Forward Dynamic Models
Generally, forward model predicts future states/observations of a system given the current addi- tionally a history of states/observations. In the context of objects being represented solely as SDFs, propose learn forward model F + x ... x + -> & that predicts the SDF of an object of at time step f based on a history of SDF observations of the object 0 until time t 1 and the motion of another object oi until time t. This means F as
() F Fo
an SDF itself that can be queried in R Interactions between more than two objects are possible, but we focus on pair-interactions in the present work. If = 1, F is quasi-static model. Internally, F can be defined either directly predict the SDF of as in (3) or the flow
()(+) (-) + FRow (ei-ta-1,o-10 (.)
(4)
from دا_ to o!. In both cases, the functional H for planning is then naturally defined as
Hr Hr oi = J (d)(x) - -F
(x)) dx.
(5)
For perfect model F, this functional Hr attains a zero value if and only if the evolution of di-it and of is compatible with the underlying physical process in the space X. Therefore, the loss function to train F is also (5) for dataset D- { (ob.. oix); 1a 1=1 of such consecutive SDF motions of the two objects. Since F takes as input the complete SDFs of the objects and not just values like the distance between objects and their contact point locations, it can learn to reason not only about these quantities, but also the contact geometry, relative object movements, center of mass and inertial parameters (assuming an equal density of the objects), all of which are necessary quantities to represent the dynamies. This way, F inherently takes the geometry of the objects into account. Note that usually, forward models are understood in terms of a function that maps the current state (history) and an (abstract) action to the next state. For SDFs, this would mean model of the form of = Flot-1a-1> 11. In our case, however, there is no notion of an abstract action, instead, our formulation learns generic model of the interaction between two objects, where the motion of one object (Φ?) influences the other (o'). Therefore, while the transformation applied to d² can be interpreted as an action, the model has no action as input and hence can deal with different
geometries of o², which is not possible in case of an abstract action without ø² also being input.

---

5.2 Kinematic Success Models
Many tasks in manipulation planning can be specified in terms of static success models instead of full forward dynamics model. We call a model that predicts whether a configuration of potentially multiple SDFs at the same time slice leads to manipulation success a kinematic success model. Assume through interaction with the environment, dataset D = {(d²) of } of SDFs repre- senting [Z,] many objects has been obtained with y' indicating that the configuration of SDFs leads to manipulation success, y = 0 to failure. Then learning H is similar to classification prob- lem, where H ((z)) = 0 implies success prediction. This way, H can model a manifold of feasible configurations and not only a single solution. See sec. for details (loss function etc.).
5.3 Learning Functionals with Neural Networks
So far, we have not discussed how functionals of the form H + x + R can be learned or even queried in the first place with usual function approximators like neural networks, since, in general, the neural network would have to take functions as infinite dimensional objects as input. To approximate this, we choose in this work the straight-forward approach by evaluating ø E & on a discretized version of the set X, denoted by Xn. As discussed previously, the set X' should cover the relevant region of the workspace where the interaction between the objects takes place. We specifically do not assume X to be aligned or perfectly centered with the objects that are involved. This way, the dynamics model from sec. 5.1 can be realized by
of
== Fo (x.,,
with Fe being usual neural network architectures. During training, the integral in (5) is approximated over the same discretized Xr, for simplicity. Hence, the dataset to train F can contain the SDF obser- vations the grid points of Xr only. However, F9 still approximates an SDF which can be queried at arbitrary E R and does not only predict the values on the grid points. For general functionals H, Technically, Xr C Rdxhxw
the evaluation is analogous, i.e. H (o*) sEI) se Ha ((0 (AW)
a regular grid which allows us to encode (aK,) using 2D or 3D convolutions. In contrast to an occupancy grid, the evaluation of ((K)) contains more information about the object than whether
there is an object at the grid point or not. Note that the differentiabilty of H (XWI) tez)
with respect to q¹ is maintained, which is another advantage of representing such models as func- tionals of SDF functions evaluated sufficient to only have the SDF values evaluated on a gird, no other information like actions or velocity/pose estimations are needed.
grid instead of static values on grid. During training, it
6 Task Constraint Functionals
Here we present analytical functionals of SDFs that are useful to specify goals of a manipulation problem or other task aspects. These functionals are general as direct consequence of our object representations being SDFs. Therefore, there is advantage or need learn these given the SDFs.
6.1 Pair-Collision between Objects
Collision avoidance is inherent part of many task specifications. Given two SDFs Φi, 62 can measure whether they are collision via their overlap integral
Heaul (#1, x2) / (x) < 0] (02(x) < 0) dr.
The indicator bracket [-] means [P] = 1 if P is true, otherwise [P] = O. The integral in (7) integrates over the space where both SDFs are negative at the same time, which is only the case if the two objects overlap, hence are in collision. The gradients of (7) smoothed using the sigmoid function (2)
1+exp(-2) i.e. (x) < 0j (-a012(22) with parameter > 0.
6.2 Goal Region
If part of the task specification that an object Φ1 is fully contained inside the boundary of another object og. called the goal region, then similar integral as for the pair-collision can be utilized
) = / (x) < 0f (x) > 0] dx RJ / (-a01 (z)) a (z) dr.
(8)
Here, points outside of the goal region that are inside the object count towards the integral.
5

---

solution
success
total scenes
found
rate
solved
SDF opt. sampling
98.7%
88.5%
87.3%
SDF opt. only
51.3%
93.5%
47.3%
SDF sampling only iy
83.8%
82.3%
68.9%
occupancy V sampling Ka
100.0%
34.0%
34.0%
pointnet++, sampling 61
100.0%
82.0%
52.0%
Success rates of mug hanging experiment. Total scenes solved means the percentage of scenes in the evaluation dataset for which a solution was found that is not in collision and is stable when dropped. Only hest results for each object representation shown. Full results see Tab. 3 and sec. A.5.
Table 1:
Figure 1: Found solution configurations by the optimizer using the learned model.
6.3 Establishing Contact between Objects
Establishing and maintaining contact between objects is central for many manipulation tasks. One way to model that the distance between two objects Φ) and Φ should be zero is via the functional
Hpoc(01, Φ2) min I (p) + [=2(p)]-
PEA
7 Experiments
7.1 Mug-Hangings Kinematic Success Model
In this experiment, we want find rigid transformations applied on observed mugs of different shapes in a scene to hang them stably on hooks of different types. The functional /hang is therefore a kinematic success model that takes the SDFs of the mug and the hook as input. To generate data to learn Haung we randomly sample scenes of different mug and hook shapes (1600 scenes for training, 400 for testing and 150 for evaluation). See Fig. 15 for examples of mugs and hooks in the evaluation data. Then we sample for each scene in the training and test data the position and orientation of the mug uniformly in the bounding box X' until at least one successful configuratio where the mug does not fall on the ground when being dropped from the sampled configuration while at the same time not being in collision with the hook. We use Bullet [42] to simulate the dropping. In total, 20 configurations per scene are generated. Since sampling successful configuration is rare event, for the majority of the scenes, only one successful and 19 failure configurations are contained in the training and test data, making learning challenging. Another challenge of this task is that the model has to reason about both the hook and mug geometry jointly. Formulating an analytical model. e.g. on mesh-basee object representation, model this constraint non-trivial.
has been obtained
7.1.1 Performance with Optimization
Fig. shows solution configurations found by our model Hhang as an optimization objective. Inter- estingly, the solutions not always contain the intuitive solution, but also ones where other parts of the hook are being utilized (middle column in Fig. 1). The optimization problem (2) to solve this mug hanging problem has two objectives, the learned kinematic success functional Hhang and the pair-collision Hesn from sec. 6. While in principle Hhang also learns to avoid collisions, we found that the robustness in avoiding collisions increases when including Hoon- The learned functional Hhang is, in general, non-convex the rigid transformation q of the mug. Therefore, we observed that gradient- where Hhang predicts zero, in every instance [43]. To overcome this issue, tion procedure up to 20 times with a randomly sampled initial guess of the mug in X. Fig. 16 shows an example of a sampled initial configuration from which the optimizer is started (left), then the optimized configuration (middle) and finally, the configuration after simulation. Tab. shows the success rates on the evaluation scenes. As one can see, for the proposed approach where objects are represented as SDFs using optimization and sampling, in 98.7% of the evaluation scenes, solution is found where Hang predicts success and no collision is violated (first column). Out of these, 88.5% are stable configurations (checked by simulation) and the optimized configuration of the mug col- lision free with the hook, leading to 87.3% total solved scenes (last column). When the optimization is run only once (second row), then only in 51.3% of the cases it converges to a feasible solution.
optimization is not sufficient for the optimizer to find a feasible solution, i.e.
restart the optimiza-
7.1.2 Comparison to Sampling, Point-Cloud and Occupancy Measure Representations
This section shows that learning kinematic success model based on the proposed SDF object representation outperforms other representations (point-clouds and occupancy measures) and further highlights the advantage of the models learned with SDFs providing useful gradients by comparing it to sampling without optimization. For full results, refer to sec. A and Tab. 3. The sampling

---

Figure 2:
Different pushing scenarios in evaluation datasct. Yellow is the pusher, green the goal region and light blue the object. The two images from the right show out-of-distribution generalization shapes.
approach draws relative transformations of the mugs uniformly in X' until the evaluation with the learned Hany and the collision functional Heoll predicts a successful and collision free configuration with a threshold K. As one can see in Tab. and Tab. 3, our proposed approach has a significantly higher performance (87.3%) compared to the best threshold for pure sampling with SDF (68. 9%%) and the best of the other object representations (34% with occupancy measure, 52% with point-cloud).
7.2 Pushing Objects a Table: Dynamic Model
In this experiment, we consider the task of pushing boxes and L-shaped objects of different dimen- sions with a spherical pusher of different radii into a goal region dg on a table. Fig. 2 visualizes typical objects, pushers and goal regions. The goal region has to be large enough that all possible objects fit. We again use Bullet as a simulator to generate data to train a dynamics model of the from described in sec. 5.1 with
1, i.e. Hr is function of four SDFs 6/. Φ/_1 (object) and of 0;- (pusher). In total, 14975 different scenes (including shapes and initial configuration) sampled where random push actions biased roughly towards the object center are applied until the object leaves the table. Since the dynamics and interaction of the objects in this scenario can be described in the 2D plane, the 3D signed distance functions of the objects are evaluated in the 2D set Nr. E TR 1400 140 only. Therefore, the model F predicts the dynamics of o in this 2D projection.
7.2.1 Forward Prediction Error
contact phase
contact phase
Tab. 2 shows the one-step prediction error on the evalu- ation dataset for the flow model Fnow (4) and the direct SDF prediction F (3). The way we utilize the model within the trajectory optimization problem never asks for predictions more than one step into the future. We train one single dynamics model for both box and L-shaped objects and different pushers. The pre- diction error is the RMSE of predicting the correct SDF values in Nr. The last row shows the error if the model would simply predict the next state as the last state of the object. As one can see, Fnow achieves a lower error than F. This is due to F having to predict the complete SDF, while FROw only the flow. In phases of the motion where there is no contact between the object and the pusher, both models Frow and F have to learn that the object should not move (and F has to predict the complete SDF in this case as well), which is also non-trivial, but they accomplish this with low error.
Fnow F
3,4 + 1.6
1.4 = 1.8
5.8 + 1.7
5.2 1.6
0 =
10.8 + 3.4
Table 2: RMSE [mm] on evaluation dataset.
7.2.2 Comparison to other Object Representations (Point-Cloud and Occupancy Measure)
In sec. 1.6.1 and sec. A.6.2 we present and explain a comparison of the forward prediction error between models learned with object representations being SDFs, occupancy measures and point- clouds. As shown in Tab. 4 and Tab. 5, models learned with the SDF representation outperform models based on point-clouds and occupancy measures in their predictive performance. Further, in Tab. 5 and sec. A.4, we also show that one can learn image conditioned SDFs and dynamic models on top of the learned SDF simultaneously with no noticeable performance degradation.
7.2.3 Planning with the Learned Model and Execution Performance
Having learned the pushing dynamics prediction model, we now utilize it within (2) to solve the task of pushing the object into the goal region. There are four constraints. First, the dynamics model Hr and, second, the goal region Hy. While this seems be enough to specify the problem fully, we add two additional constraints, Hcoll and HPOc. The discrete variable s of (2) decides whether there are one or two push phases. Only in a push phase, HPo is active. Heall is always active. Similarly to the mug hanging experiment, local minima are core issue well. Therefore, we initialize the pusher position at phase 1 or on a set
different points around the object. These 4 points around the object are always the same in all scenarios, no matter of the size, shape or orientation of the object. Compared to other approaches where the action space has to be chosen much more carefully, we believe that this is a rather weak prior. The initialization also does not start from contact with the object or similar, because our problem contains the challenge of contact establishment and possible breakage to push from different side to achieve the goal. Therefore, due to this initialization, there
7

---

coverage of o inside goal dy
0.5
Fiow
Figure 4: Generalization experiments for pushing scenario. Left: interaction between three objects to solve the task. Middle: the goal is that the right robot touches the object. Right: the
boxes
L-shaped
Figure 3: Pushing performance on evaluation scenes in terms of the amount of Ø² that is inside of the goal region the end of the execution. A value of means that the object fully contained in the goal region.
robots have to collaborate to push the object into the goal region, which would not be possible with one arm alone.
are 20 different optimization problems we solve for each scene (4 for one pushing phase, 42 for two pushing phases). To evaluate the performance, we execute the planning result with the least constraint violation and cost open-loop in the simulator. Despite the fact that pushing unstable over nong-hoiizons, our proposed approach achieves a high performance. As shown in Fig. 3, which plots the amount of the object that is inside of the goal region at the end of the execution, using the learned FRow. the approach achieves 99.7% (median) coverage of the object inside the goal region on box pushing and 98.4% (median) on the L-shaped objects (50 evaluation scenes each). Please note that, although the goal region for small objects seems large, the optimizer usually moves the object until it just barely inside the goal region and not any further. Therefore, even very small deviations during the open-loop execution already lead to some parts sticking out. For the larger objects in the evaluation scenes, the goal region is barely large enough. With the direct F, the performance is bit worse, but still high (median 96.6% for boxes, 91.5% for L-shaped objects).
Sec. B demonstrates that our proposed SDF framework outperforms an approach where the opti- mization problem is formulated for objects represented as meshes and the analytic dynamic model from [13] for the pushing dynamics. Refer to Fig. 6 for quantitative results. Finally, in sec. A.6.3 and Fig. we investigate the importance of models learned on top of SDF representations providing useful gradients for planning in comparison to models with point-clouds and occupancy measures.
7.2.4 Ablation Study
Sec. D presents an ablation study regarding the importance of the additional objectives Heoll and Hy in the optimization problem. Results in Fig. 11 show that while it is possible to solve the scenes without them, the performance greatly increases they are part of the problem formulation.
7.2.5 Generalization to Outtof-Distributtion Shapes, Multiple Objects, and Robots
In see. C.I. we demonstrate that the framework and the learned model generalizes to shapes beyond the training distribution. See Fig. 2 and Fig. 7 for those shapes. Quantitative results presented in Tab. 6 and Fig. 8 indicate that the model achieves both high prediction accuracy and high perfor- mance when used for planning. Furthermore, as seen Tab. 6, a model learned with SDF' general- izes significantly better than with point-clouds, also relative to the results obtained on-distribution. We further show in sec. C.3 and sec. C.2 that the framework is capable of generalizing to scenes that contain obstacles (Fig. 10) and a scenario where three objects interact in order to solve the task (Fig. 9 and Fig. 4). Finally, in sec. E we demonstrate multiple scenarios where the learned pushing dynamics model is embedded into scene that contains robots. All these generalization experiments require no change in the methodology or to learn a new model, showing the generality and versatility of our proposed framework. For more details, refer to the respective sections in the appendix.
8 Conclusion
In this work, we have shown that the constraints of a trajectory optimization problem for solving manipulation problems can be formulated in terms of learned functionals of SDFs only. SDFs can serve as a common object representation across completely different tasks. The functionals can naturally model the interaction between objects of arbitrary shapes and can be learned directly from SDF observations, which is closely connected to perception. We have shown that learning models on top of SDFs outperform other object representations like point-clouds and occupancy measures both in terms of prediction accuracy and the ability to plan. The greatest challenge of our framework are local minima of the resulting trajectory optimization problem. While sampling strategies for initial guesses can mitigate this to some extend, it is an issue, which not unique to our approach, but many nonlinear trajectory optimization formulations. While we have considered rigid objects in this work only, we believe that the proposed approach can be extended to deformables as well.
8

---

Acknowledgments
Danny Driess thanks the International Max-Planck Research School for Intelligent Systems (IMPRS-IS) for the support. This research has been supported by the German Research Foundation (DFG) under Germany's s Excellence Strategy EXC 2002/1-390523135 "Science of Intelligence". The authors thank the anonymous reviewers for their comments.
References
[1] D. Driess, J.-S. Ha, R. Tedrake, and M. Toussaint. Learning geometric reasoning and control for long-horizon tasks from visual input. In Proc. of the IEEE Int. Conf. on Robotics and Automation (ICRA). 2021.
[2] C. Garrett, R. Chitnis, Holladay, B. Kim, T. Silver, P. Kaelbling, and Lozano-Pérer" Integrated task and motion planning. Annual review of control, robotics, and autonomous systems, 2021.
[3] L. P. Kaelbling and Lozano-Päre Pérez. Hierarchical planning in the now, In Proc. of the IEEE International Conference on Robotics and Automation (ICRA), 2011.
[4] S. Srivastava, E. Fang, L.. Riano, R. Chitnis, S. J. Russell, and P. Abbeel. Combined task and motion planning through an extensible planner-independent interface layer. In Proc. of the Int. Conf. on Robotics and Automation (ICRA), 2014.
[5] N. T. Dantam, Z. K. Kingston, S. Chaudhuri, and L. E. Kavraki. An incremental constraint- based framework for task and motion planning. International Journal on Robotics Research, 2018.
[6] V. N. Hartmann, O. S. Oguz, Driess, M. Toussaint, and A. Menges. Robust task and motion planning for long-horizon architectural construction planning. arXiv:2003.07754 2020.
[7] D. Driess, J.-S. Ha, and M. Toussaint. Deep visual reasoning: Learning to predict action sequences for task and motion planning from an initial scene image. arXiv:2006.05390 2020.
[8] I. Mordatch, E. Todorov, and Z. Popović. Discovery of complex behaviors through contact- invariant optimization. ACM Transactions on Graphics (TOG), (14):11-8, 2012.
[9] M. Posa, C. Cantu, and R. Tedrake. A direct method for trajectory optimization of rigid bodies through contact. The International Journal of Robotics Research, 33(1):69-81, 2014.
[10] M. Toussaint, K. R. Allen, K. A. Smith, and J. B. Tenenbaum. Differentiable physics and stable modes for tool-use and manipulation planning. In Robotics: Science and Systems, 2018.
[11] R. Hogan, E. Grau, and A. Rodriguez. Reactive planar manipulation with convex hybrid MPC. In Int. Conf. on Robotics and Automation (ICRA), 2018.
[12] N. Doshi, F. R. Hogan, and A. Rodriguez. Hybrid differential dynamic programming for planar manipulation primitive. In Int. Conf. on Robotics and Automation (ICRA), 2020.
[13] M. Toussaint, J.-S. Ha, and D. Driess. Describing physics for physical reasoning: Force-based sequential manipulation planning. IEEE Robotics and Automation Letters. 2020.
[14] D. Driess, J.-S. Ha, and M. Toussaint. Learning solve sequential physical reasoning prob- lems from a scene image. The International Journal of Robotics Research, to appear.
[15] Z. Chen and H. Zhang. Learning implicit fields for generative shape modeling, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019.
[16] Liu, S. Saito, W. Chen, and H. Li. Learning to infer implicit surfaces without 3d supervision. arXiv preprint arXiv: 100767,077700
[17] L. Mescheder, M. Oechsle, M. Niemeyer, S. Nowozin, and A. Geiger. Occupancy networks: Learning reconstruction in function space. In Proceedings IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2019.

---

[18] J. Park, P. Florence, J. Straub, R. Newcombe, and S. Lovegrove. Deepsdf: Learning continu- ous signed distance functions for shape representation. In The IEEE Conference on Computer Vision and Puttern Recognition (CVPR), June 2019.
[19] V. Sitzmann, E. R. Chan, R. Tucker, N. Snavely, and G. Wetzstein, Metasdf: Meta-learning signed distance functions. In arXiv, 2020.
[20] M. Atzmon and Y. Lipman. Sal: Sign agnostic learning of shapes from raw data, In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
[21] C. Jiang, A. Sud, A. Makadia. J. Huang, M. Nießner, T. Funkhouser. et al. Local implicit grid representations for 3d scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6001-601 2020.
[22] M. Macklin, K. Erleben, M. Müller, N. Chentanez, S. Jeschke, and Z. Corse. Local optimiza- tion for robust signed distance field collision. Proceedings of the ACM on Computer Graphics and Interactive Techniques, 2020.
[23] A. Fuhrmann and G. Sobottka. Distance fields for rapid collision detection in physically based modeling. 2003.
[24] K. Hauser. Semi-infinite programming for trajectory optimization with non-convex obstacles. The International Journal of Robotics Research, 2018.
[25] M. Zhang and K. Hauser. Semi- Semi-infinite programming with complementarity constraints for pose optimization with pervasive contact. In IEFE International Conference on Robotics and Automation, 2021.
[26] S. Pfrommer, M. Halm, and M. Posa. Contactnets: Learning of discontinuous contact dynamics with smooth, implicit representations. Conference on Robot Leurning, 2020.
[27] M. Breyer, J. Chung, Ott, S. Roland, and N. Juan. Volumetric grasping network: Real-time
6 dof grasp detection in clutter. In Conference on Robot Learning. 2020.
[28] 7.. Jiang, Y. 7hu, M. Svetlik, K. Fang, and Y. Zhu. Synergies between affordance and geometry: 6-dof grasp detection via implicit representations. arXiv preprint (exxv:2204.0154: 2021.
[29] M. Van der Merwe, Q. Lu, B. Sundaralingam, M. Matak, and T. Hermans. Learning continuous 3d reconstructions for geometrically aware grasping. In Ini. Conf. on Robotics and Automation (ICRA), 2020.
[30] Ebert, C. Finn, S. Dasari, A. Xie, A. Lee, and S. Levine. Visual foresight: Model-based deep reinforcement learning for vision-based robotic control. arXiv:1812.00568, 2018.
[31] R. Kandukuri, J. Achterhold, M. Moeller, and J. Stueckler. Learning to identify physical parameters from video using differentiable physics. In Pattern Recognition: 42nd DAGM German Conference, 2021.
[32] Z. Xu, J. Wu, A. Zeng, J. B. Tenenbaum, and S. Song. Densephysnet: object representations via multi-step dynamic interactions. arXiv preprint arXiv: 090.. 2019.
Learning dense physical
[33] Z. Xu, Z. He, J. Wu, and S. Song. Learning 3d dynamic scene representations for robot manipulation. In Conference on Robotic Learning (CoRL), 2020.
[34] L. Manuelli, Y. Li, Florence, and R. Tedrake. Keypoints into the future: Self-supervised correspondence in model-based reinforcement learning. In Conference on Robotic Learning (CoRL), 2020.
[35] A. Byravan and D. Fox. Se3-nets: Learning rigid body motion using deep neural networks. In IEEE International Conference on Robotics and Automation (ICRA), pages 173-180, 2017.
[36] A. Zeng, P. Florence, J. Tompson, S. Welker, J. Chien, M. Attarian, T. Armstrong. I. Krasin, D. Duong, V. Sindhwani, and Lee. Transporter networks: Rearranging the visual world for robotic manipulation. Conference on Robot Learning (CoRI.), 2020.
10

---

[37] A. Simeonov, Y. Du, B. Kim, F. R. Hogan, P. Agrawal, and A. Rodriguez. Learning to plan with pointeloud affordances for general-purpose dexterous manipulation. Conference on Robot Learning, 2020.
[38] S. Mukherjee, C. Paxton, A. Mousavian, A. Fishman, M. Likhachev, and D. Fox. Sim-to- real task planning and execution from perception via reactivity and recovery. arXiv preprint arXiv:2011.0864. 2020.
[39] G. Sutanto, I. M. R. Feránndez, P. Englert, R. K. Ramachandran, and G. S. Sukhatme. Learning equality constraints for motion planning on manifolds. arXiv preprint arXiv: 10200009.1822. 1182,, 2020.
[40] Y. You, L. Shao, T. Migimatsu, and J. Bohg. Omnihang: Learning to hang arbitrary objects us- ing contact point correspondences and neural collision estimation. In 2021 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2021.
[41] M. Danielczuk, A. Mousavian, C. Eppner, and D. Fox. Object rearrangement using learned implicit collision functions. wwXiv:20/1.107200 2020.
[42] E. Coumans and Y. Bai. Pybullet, python module for physics simulation for games, robotics and machine learning, 2016-2021.
[43] Ortiz-Haro, J.-S. Ha, Driess, and M. Toussaint. Structured deep generative models for sampling on constraint manifolds in sequential manipulation. In Proc. of the Annual Conf. on Robot Learning (CORL), 2021.
[44] X. Yan. Pointnet/pointnet+* pytorch. Pointnet2 pytorch, 2019.
https: /t gā thub. com/ potp////on/n/neee..tt.mm
[45] T. Pfaff, M. Fortunato, A. Sanchez-Gonzalez, and P. W. Battaglia. Learning mesh-based sim- ulation with graph networks. arXiv preprint arXiv:2010.03-409 2020.
[46] D. Driess, S. Schmitt, and M. Toussaint. Active inverse model learning with error and reachable set estimates, In Proc. of the IEEE Int. Conf. Intelligent Robots and Systems (IROS), 2019,


Reviews:
[{"review_id": "wq7hD15BfT-", "reviewer": "Reviewer_SZWq", "summary": "This paper modifies an existing formulation of task and motion planning (the Logic Geometric Programming framework) to incorporate constraints that are based on implicit surface representations of rigid-body objects. The paper formalizes what a dynamics model over Signed Distance Fields means, as well as other typical constraints present in TAMP problems, for example contact constraints, goal region constraints, as well as pairwise collision constraints. \n\nThe motivating argument that the paper puts forth for TAMP with implicit surface representations is that traditional object geometries used in TAMP problems (eg. simple parametric shapes, or even meshes) are limited. In fact the paper claims that \"it is unclear how these models can be grounded from sensor information.\" The paper mentions that the main advantages of SDFs are their ability to represent nonconvex objects, and that it is easier for them to incorporate sensor data, particularly depth data or pointclouds. The main underlying SDF representation assumed in the proposed network are neural networks trained from sampled points inside each object. \n\nI'd say the main contribution of the paper is an explicit formulation of TAMP with implicit surface constraints. There have been other related works that have followed similar approaches in the past using trajectory optimization, but not full TAMP.         ", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\nS1. This is an interesting idea. Implicit surface representations are definitely well studied in graphics and animation, and the robotics community should pay more attention to them when it comes to motion planning.\n\nS2. The paper is generally clearly written, with the exception of a few sections that I mention below. \n\nWeaknesses (* indicates major)\n\nW1*. The reason presented as the motivation for studying SDFs over traditional parametric shapes used in TAMP is a bit misleading. It is unclear why robot perception would make learning SDFs any easier than parametric shapes or meshes. This is especially true when SDFs are represented with neural networks, which are generally much more difficult to update in an incremental/continual setting than shapes or mesh representations. Also, meshes make it easier to represent objects at different levels of fidelity according to the relevance of the object to the task at hand. I think the writing in the introduction should be updated to reflect more practical reasons why SDFs could be preferable.\n\nW2. It is not clear how scalable this method is. The simulation experiments show very limited planning scenarios. How does this method scale when the number and volume of objects grows? \n\nW3*. The paper seems to be unaware of important prior work from the robotics community on this topic: https://motion.cs.illinois.edu/papers/WAFR2018-Hauser-SemiInfinite.pdf\nhttps://motion.cs.illinois.edu/papers/ICRA2021_Zhang_SIPCC.pdf\nThese papers should not only be cited, but studied carefully for these types of problems. Other related papers include:\nhttps://www.graphicon.ru/html/2003/Proceedings/Technical/paper495.pdf\nhttps://mmacklin.com/sdfcontact.pdf\n \nW4*. The TAMP problem formulation (2a-c) comes from LGP, but as far as I can understand, the paper resorts to gradient based optimization with multiple restarts, without making use of the discrete tree search that LGP makes use of for the symbolic state. This seems prone to local minima. Why was this choice made?\n\nW5. The presentation of 2a-c is too notation-heavy. Please simplify it for readability.\n\nW6*. SDFs are clearly promising for the dynamics prediction part of the problem. This is evident from the pushing experiments. However, they introduce an inner optimization problem when establishing contact between objects (eqn 10), so it is not clear why the added complexity and cost of including an optimization problem as a constraint is worth it, compared to other representations.        ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I am recommending a weak reject. I think the problem is interesting and difficult, and more people in the robotics community should be looking at it. I think this paper needs a lot more work to provide a convincing solution to the problem, and the current experiments are too simple for the paper to have any impact, but it is enjoyable to read. It would be appreciated if more challenging planning environments could be included, possibly with more objects.  \n\n\nPOST-REBUTTAL UPDATE:\n--------------------------------------\n\nThank you to the authors for addressing my concerns in such detail. I am upgrading my recommendation to weak accept. Best of luck!\n", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {"issues": "I would very much hope to see the weaknesses I listed as major addressed in the revised version. If I see convincing changes regarding the major issues, I am willing to update my recommendation to weak accept.  "}}, {"review_id": "fTNKEe6jPeS", "reviewer": "Reviewer_5kbt", "summary": "The work proposes a TAMP framework that leverages functionals of SDF-based representations of objects. Learned functionals encode both dynamics constraints (based on one-step prediction) as well as kinematic success detectors, while analytic functionals encode problem constraints such as collision avoidance and contact establishment. The motivation of such a representation is twofold: (i) SDFs are a natural vision-based object representation, and (ii) models that reason using SDFs account for full continuous object geometry, as opposed to relying on pre-specified articulated points/surfaces. \n\nA mixed discrete-continuous optimization-based planner leverages the various functionals where the discrete optimization variable indexes “active” functionals and their relevant SDF operands, while the continuous variables correspond to the sequence of SE3 transformations on each SDF (object). Experiments are presented on two environments: (i) a static mug-hanging task, and (ii) a quasi-static pusher task. ", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The motivation of using SDFs as the base object representation for dynamics models is compelling and the paper presents an encouraging attempt at a model-based planning framework utilizing such a representation. In the spirit of this novelty, I recommend that the paper is accepted. However, the authors may wish to address some of the following concerns:\n\n1. Presumably, a potential advantage of using SDFs as the object representation is that one would be able to generalize to unseen objects, with similar *local* geometric features. The experiments involved training and evaluation over the same homotopy of objects, with variation only in size. Some experiments involving different object homotopies (shapes) would be extremely interesting.\n\n2. The notion of a “dynamic” task is somewhat underwhelming - under the the author’s own classification, this is a quasi-static task. An example involving dynamic manipulation therefore would be more insightful to showcase the capabilities of SDF-based dynamics models.\n\n3. The manipulation examples presented in the work constitute *passive* tasks - placing a mug on a handle, or pushing an object on a stable plane. Many of the challenging examples in manipulation involve at the very least, picking up/grasping and object and then performing a sequence of task-specific transformations. In such tasks, the *stability* of grasps and subsequent manipulations comes into play. Can the SDF representation handle such tasks or would one additionally require non-trivial controllability extensions? This is closely related to the previous point on dynamic tasks.\n\n4. Computationally, there is a natural limitation to how SDFs are evaluated and processed through dynamics models - using pre-set grids. What then is the trade-off in computation speed vs accuracy of the learned model as a function of the grid resolution? Perhaps an encoder/decoder model could be jointly trained to reduce the dimensionality of the SDF representation used within the various functionals.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The motivation of using SDFs as the base object representation for dynamics models is compelling and the paper presents an encouraging attempt at a model-based planning framework utilizing such a representation. In the spirit of this novelty, I recommend that the paper is accepted. However, the main drawback concerns the imitations of the experiments. Two static/quasi-static manipulation examples are presented that are passively stable. Arguably, some of the more impact-generating examples would constitute dynamic manipulation tasks, involving at the very least, picking up objects. Potential for impact is high, given the elegance of SDF representations, however more compelling experiments are needed.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {"issues": "From above, limitations (2) and (3) will likely involve a lot more work, and may not be suitable for inclusion within the revision period. However, limitations (1) and (4) (or at least (1) with some discussion on (4)) would be sufficient."}}, {"review_id": "6hVvnAob2l", "reviewer": "Reviewer_NU21", "summary": "This paper aims to solve the TAMP problem for tasks that require kinematic and dynamic reasoning beyond the capabilities of analytical constraint modelling. Hence they propose to learn these models using SDF representations of different objects in the scene. The authors claim that this allows the model to learn various geometric and dynamic interactions implicitly which are necessary to solve the manipulation problems of interest.\n", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\nThe authors have a good grasp on the current learning-based approaches and gaps in the literature which helps motivate their problem. \nThe authors provide some interesting task constraint formulations that can be used for learning and solving two particular manipulation tasks with SDFs as inputs.\nThe method is described well and the formulation is sound.\nThe choice of SDF representations to capture models and formulate the optimisation problem is key in the framework. \n\nWeaknesses\n\nWhile the method is sound and well-motivated, there is a lack of experimental evidence to justify the need for their method over any of the many existing ones for solving such problems.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The proposed formulation is interesting and well-motivated, however, due to weak experimental evaluation coupled with the incremental nature of the contribution it is difficult to justify publication.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {"issues": "- Authors claim to propose a TAMP framework, however, it seems more suitable to say that they propose a learning-based approach to evaluating task constraints which facilitates task planning.\n- While promising results have been achieved for the demonstrated tasks, there remains the technical challenge of how this method would scale to other scenarios. For example, for the proposed forward dynamics model, it seems non-trivial to have more than two interacting objects. Furthermore, the complexity of computing the interactions will increase non-linearly. Additionally, the training process seems quite lengthy and might be prohibitively slow for multiple objects, scenes and tasks. This could be further compounded depending on the size and resolution of the SDF.\n- Dealing with the non-convexity of the problem via random restarts is unsatisfying and could be problematic depending on the complexity of the problem.\n- Some grammatical errors. \n- Regarding the video, there is a lot of repeated experiments that seem similar and have no apparent added value. \n- Furthermore, it would be helpful to show the significance of your results by showing a comparison of your method alongside the failure cases of other methods.\n"}}]

Author rebuttal:

Meta-review: This paper proposes the use of implicit SDF representations to enable planning with task constraints using learned dynamics functions on the object SDFs. The paper validates this approach with several planning tasks in simulation including bimanual robot experiments. 

The paper motivates the use of SDFs clearly for planning and shows the ability to learn accurate transition models using the SDF representation including learning the SDFs from segmented imagery. It further shows that SDFs perform better when compared to point cloud or occupancy-based binary representations.

The authors made substantial improvements to the paper based on the initial reviews and I find no lingering major issues with the work.
Decision: Accept (Poster)