Learning Neural Implicit Functions as Object Representations for Robotic Manipulation
Authors: Jung-Su Ha
Danny Driess
Marc Toussaint
Venue: ICLR 2022 Submitted (2022)
Keywords: Representation Learning
nerual implicit representation
robotic manipulation
task and motion planning

Abstract: Robotic manipulation planning is the problem of finding a sequence of robot configurations that involves interactions with objects in the scene, e.g., grasp, placement, tool-use, etc. To achieve such interactions, traditional approaches require hand-designed features and object representations, and it still remains an open question how to describe such interactions with arbitrary objects in a flexible and efficient way. Inspired by neural implicit representations in 3D modeling, e.g. NeRF, we propose a method to represent objects as neural implicit functions upon which we can define and jointly train interaction features. The proposed pixel-aligned representation is directly inferred from camera images with known camera geometry, naturally acting as a perception component in the whole manipulation pipeline, while at the same time enabling sequential robot manipulation planning.

Under review as a conference paper at ICLR 2022
LEARNING NEURAL IMPLICIT FUNCTIONS AS OBJECT REPRESENTATIONS FOR ROBOTIC MANIPULATION
Anonymous authors Paper under double-blind review
ABSTRACT
Robotic manipulation planning is the problem of finding a sequence of robot con- figurations that involves interactions with objects in the scene, e.g., grasp, place- ment, tool-use, etc. To achieve such interactions, traditional approaches require hand-designed features and object representations, and it still remains an open question how to describe such interactions with arbitrary objects in a flexible and efficient way. Inspired by recent advances in 3D modeling, e.g. NeRF, we pro- pose a method to represent objects as neural implicit functions upon which we can define and jointly train interaction constraint functions. The proposed pixel- aligned representation is directly inferred from camera images with known camera geometry, naturally acting as a perception component in the whole manipulation pipeline, while at the same time enabling sequential robot manipulation planning.
INTRODUCTION
Intelligent agents should be able to interact with objects in the environment, such as grasping and placing object, or more general tool-use, to achieve a certain goal. In roboties, such instances are formalized as manipulation planning, a type of a motion planning problem that solves not only for the robot's own movement but also for the objects' motions subject interaction constraints. Traditional approaches represent objects using meshes or combinations of shape primitives and de- scribe interactions as hand-crafted constraints in terms of that representation. The approach of using such traditional geometric representations has long-standing limitations in terms of their perception and generalizing to large varieties of objects and interaction modes: (i) The representations have to be inferred from raw sensory inputs like images point clouds raising the fundamental problem of perception and shape estimation. However, if the aim is manipulation skills, the hard problem of precise shape estimation might be unnecessary predict accurate interaction features" and an end-to-end object representation might be more appropriate than a standard perception pipeline. (ii) With increasing generality of object shapes and interaction, the complexity of representations grows and hand-engineering of the interaction features becomes inefficient.
What is a good representation of an object? Considering the representation will be used to predict interaction features, expect it to encode primarily task-specific information rather than only geo- metric. And some of the information should to be shared across different interaction modes. In other words, good representations should be task- so that the feature prediction can be simplified and, at the same time, be task agnostic enable synergies between the features. E.g., mug handles are called handles because we can handle the mug through them and also, once we learn the notion of handle, we can interact with the mug through them in many different ways. From the perception aspect, good representations should be easy to infer from raw sensory inputs and should be able to trade their accuracy off in favor of the feature prediction.
To this end, we propose novel data-driven approach to learning interaction features. The proposed feature prediction scheme is illustrated in Fig. I The whole pipeline is trained end-to-end directly with the task supervisions to simplify the interaction prediction. The object representation acts as a bottleneck and is shared across multiple feature predictions so that the task-agnostic representations can emerge. Particularly, the object representation is neural implicit function over the 3DD space (Park et al. 2019 Milden- hall et al. 2020) upon which equality constraint features are trained. The proposed neural implicit
to make the representation and perception task-specific and thus
We call an interaction constraint function interaction feature; when used as equality constraints, the interaction features, analogous energy potentials, return zero when feasible and non-zero otherwise.

---

Under review as a conference paper at ICLR 2022
obj images. cam poses, intrinsics
shared (Sec 3.1
task specific (Sec 3.2
PIFO
P1
y1
robot frame's 8 pose
feature
head
PN
PIFO
YNN
Figure 1: The interaction feature prediction scheme
function is pixel-aligned: The function takes images from multiple cameras as input (e.g, stereo) and, assuming known camera poses and intrinsics, the latent representation at a certain spatial loca- tion is directly related to pixels of the images. Once learned, the interaction features can be used by typical constrained optimal control framework to plan dexterous object-robot interaction. We adopt GegemeGe Programming (LGP) (Toussaint et al., 2018) as an optimization-base lation planning framework and show that this learned- based planning enables to compute trajectories that involve various types of interaction modes only from images. Due to the represen- tations' generalization, the learned features directly applicable to manipulation tasks involving unseen objects. To summarize, our main contributions are
manipu-
To represent objects as neural implicit functions upon which interaction features are trained,
An image-bane based manipulation planning framework with the learned features constraints, Comparison to non pixel-aligned, non implicit function, and geometric representations,
Demonstration in various manipulation scenarios ranging from simple pick-and- hang [videos] to longer-horizon manipulations [videos] and zero-shot imitations [videos]
RELATED WORK
2.1 NEURAL IMPLICIT REPRESENTATIONS IN 3D MODELING AND VIEW SYNTHESTS
Neural implicit representations have recently gained increasing attention in 3D modeling. The core idea is to encode an object or a scene in the weights of a neural network, where the network acts as a direct mapping from 3D spatial location an implicit representation of the model, such as occupancy measures (Mescheder et al. 2019, Songyou Peng 2020) or signed distance fields (Park) ct al. 2019 Gropp et al. 2020; Atzmon & Lipman 2020). In contrast to explicit representations like voxels, meshes or point clouds, the implicit representations don't require discretization of the 3D) space nor fixed shape topology but rather continuously represent the 3D geometry, thereby allowing for capturing complex shape geometry at high resolutions in memory efficient way.
There have been attempts to associate these 3D representations with 2D images using the principle of camera geometry. Exploiting the camera geometry in forward direction, i.c., 2D projection of 3D representations, yields a differentiable image rendering procedure and this idea can be used to get rid of 3D supervisions. For example, Sitzmann et al. (2019); Niemeyer et al. (2020): Yariv al. (2020); Mildenhall al. (202)) Henzler al. (2021); Reizenstein et al. (2027 showed that the representation networks can be trained without the 3D supervision by defining a loss function to be difference between the rendered images and the ground-truth. Another notable application of this idea is view synthesis. Based on the differentiable rendering. Park et al. (2020): Chen et al. 2220);; Yen-Chen et al. (202T) addressed unseen object pose estimation problems. where the goal to find object's pose relative the camera that produces rendered image closest to the ground truth. By conditioning 3D representations on 2D input images, one can expect the amortized encoder network to directly generalize to novel 3D geometries without requiring any test-time optimization. This can be done by introducing a bottleneck of a dinee--amensional global latent vector between the images and representations, but these global features often fail to capture fine aeee-graind details of the 3D models (Songyou Pengl 2020). To address this, the camera geometry the inverse direction to obtain pixel-aligned local representations, i.c., 3D reprojection of 2D image features. [Saito et al. (2019) and Xu et al. (2019) showed that the pixel-aligned methods can establish rich latent features because they can easily preserve
be exploited in
components the input images. Also. Yu et al. (2021) )) and Trevithick & Yang (2021) incorporated this idea within the view-synthesis framework and showed that their convolutional encoders have strong generalizations.
2

---

Under review as a conference paper at ICLR 2022
While the above work investigates neural implicit functions to model shapes appearances, train them to model physical interaction feasibility and thereby to provide a differentiable constraint model for robot manipulation planning.
2.2 OBJECT/SCENE REPRESENTATION FOR ROBOTIC MANIPULATIONS
Several works have proposed data-driven approaches to learning object representations and/or in- teraction features which are conditioned on raw sensory inputs, especially for grasping of diverse objects. One popular approach to train discriminative models for grasp assessments. For example, ten Pas et al. );: Mabler et al. (201)); Van der Merwe et al. (2020) trained a neural network that, for given candidate grasp poses, predicts their grasp qualities from point clouds. In addition. |Breyer ict al. (2020); Diang et al. (202I) proposed 3D convolutional networks that take as inputs a truncated SDF and candidate grasp poses and return the grasp affordances. Similarly, Zeng et al. (2020b a) addressed more general manipulation scenarios such as throwing or pick-and-place, where a con- volutional network outputs a task score image. On the other hand, neural networks also have been used generative models. For example, Mousavian al. (2019) and Murali et al.. (2020) adopted the approach of conditional variational autoencoders to model the feasible grasp pose distribution conditioned the point cloud. (Sundermeyer et al. (202I) proposed somewhat hybrid method, where the network densely generates grasp candidates by assigning grasp scores and orientations to the point cloud. You et al. (2021) addressed the object hanging tasks from point clouds where the framework first makes dense predictions of the candidate poses among which one is picked and refined. Compared to these works, our framework takes advantage of a trajectory optimization to jointly optimize an interaction pose sequence instead of relying on exhaustive search or heuristic sampling schemes, thus not suffering from the high dimensionality nor the combinatorial complex- ity of long-horizon planning problems.
Another important line of research is learning and utilizing keypoint object representations. Manuelli et al. (2019); Gao & Tedrake (202T); Qin et al. (2020); Turpin et al.] (202T) represented objects using set of 3D semantic keypoints and formulated manipulation problems in terms of such the keypoints. Similarly, Manuelli al. 2020) learned the object dynamics as function of keypoints upon which a model predictive controller is implemented. Despite their strong gener- alizations to unseen objects, the keypoint representations require semantics of the keypoints to be predefined. The representation part of our framework is closely related to dense object descriptions proposed by [Florence et al. (2018 2019). The idea is to train fully-convolutions neural networks that maps a raw input image to pixelwise object representations which directly generalize to un- seen objects. Our proposed framework can be seen as an extension of this pixelwise representation to dense representations over the 3D space which is learned by the task supervisions and can be seamlessly integrated into general sequential manipulation planning problems. Another recent re- lated work was proposed by Yuan et al. (202T), where the learned object-centric representations are used predict the symbolic predicates of the scene which turn enables symbolic-leve task plan- ning. In contrast, our framework predicts the task feasibility given a robot configuration and enables trajectory optimization of the lower-level continuous motions.
3 INTERACTION FEATURE PREDICTION VIA IMPLICIT REPRESENTATION
Given Nview images with their camera poses/intrinsics, ((I* TT,, K²), ..., NI Nres K KNire)); we define an interaction feature as a neural implicit function:
h = Crack (II) T TTKK,,,,., (I CENNi,,TNNne Now. Now
(I)
where E SE(3) is the pose of the robot frame interacting with the object. As shown in Fig- [ the feature prediction framework consists of two parts: the representation network, which we call a backbone, and the feature head networks. The backbone serves as an implicit functional representation of an object, which, conditioned representation vectors at queried 3D spatial locations. The interaction feature predictions are made through the feature heads, where each head is fed on a set of representation vectors obtained by querying the backbone at a set of key interaction points. While the multiple feature heads separately model different interactions, the backbone is shared across the tasks, making it learn more general object representations. The rest of this section will be devoted introduce each module in detail.
set of posed images, outputs d-dimensional
3

---

Under review as a conference paper at ICLR 2022
P
Figure 2: Image encoder and 3D reprojector of the backbone network.
3.1 PIXEL-ALIGNED IMPLICIT FUNCTIONAL OBJECT REPRESENTATION (PIFO)
Given Nyww posed images, define functional representation of an object mapping:
(p: {(I) T' K'),... (I N.te
K
Noie
}}}
(2)
where E IR3 and E Rd are a queried 3D position and a representation vector at that point, respectively. The representation network consists of three parts: image encoder, 3D reprojector, and feature aggregator. The first two modules compute representation vector for each image, as depicted in Fig. 2 and the last one combines them into one vector.
This module takes input image and computes feature image. In order to capture both local and global information in the image, we adopted the U-net architecture (Ron-) neberger et al.) 2015), especially with ResNet-34 (He et al. 2016) as its downward path and two residual 3 x 3 convolutions followed by up-convolution as the upward path, i.c.
Image Encoder:
F UI") Vn E {L....,
3D Reprojector: To endow the network with the multi-view consistency, all the 3D operations are performed in the view space. The 3D reprojector first converts queried point, p, to the image coordinate including depth, (p: T. K)
e 3 and then uses the projected point to extract the local image feature from the feature image. F, via bilinear interpolation. Finally, the extracted feature and the coordinate feature, which is necessary handle out-of-iman image query points, passed to a couple of fully connected layers to get a representation vector for a single image, i.e.,
y" = MMP(F" "", z" = <p; T" K"), Vn E Nview)
(4)
The feature aggregation from the multiple views should be permutation- invariant. While there many options, like summation, averaging, or sophisticated attention mechanisms, we simply take the averaging operation for it, i.e. y N.w Σn y"
Feature Aggregator:
3.2 FEATURE PREDICTION
A feature head network predicts the interaction constraint value of a robot frame's pose via the object representation. We first attach set of keypoints the robot frame and query the backbone those keypoint positions, i.e., Vi E {1, Nicepooit))
= + (p: [II" TT" K") E NLeww, p, R(g)p, + t(q),
(5)
where P. is 1tt keypoint's 's local coordinate, and R(q) and t(q) denote the rotation matrix and the translation vector, respectively. The feature head then takes input the resulting representation vectors and predicts a feature value through a couple fully connected layers, i.e..
h = MLP (y..... NN
(6)
Note that when the considered feature is an SDF, only single point needs to be queried, i.c., Nkeypoine so the whole architecture reduces the surface reconstruction in Saito al. (2019).
4

---

Under review as a conference paper at ICLR 2022
4 TRAINING
In this paper, we consider manipulation scenarios where a robot arm, Franka Emika Panda, or two have to pick and hang mugs on hooks. The environment contains mugs having different shapes and hooks on which mug be hung. To formulate this manipulation problem, three interaction features are considered: an SDF feature for collision avoidance and grasping/hanging features.
4.1 DATA GENERATION
We took 131 mesh models of mugs from ShapeNet (Chang et al., 2015) and convex convex-decomposed those meshes. The meshes are translated and randomly scaled that they can fit in bounding sphere with a radius of 10 ~ 15 cm at the origin. For each mug, we created datasets of the posed images and each interaction follows.
The posed image data consists of 100 images (128 128) with the corresponding camera poses and intrinsic matrices generated by an OpenGL renderer. Azimuths and elevations of the cameras sampled such that they can uniformly be distributed on the surface of a sphere, while their distances from the object center are randomly chosen. The azimuth, elevation and distance fully determine the camera's positions and making them upright and face the object center gives the camera's 's orientations. For the intrinsics, we used fov 2 (dcr)n(d/r)) where is the camera distance from the object center and r is the radius of the object's bounding sphere, so that the entire object appears in the image. Lighting also randomized.
Posed Images:
SDF: We sampled 12,500 3D positions and computed their signed distance values using the mesh-to- sdf library (Kleineberg) (021). Following the approach of DeepSDF (Park et al.) (219), we sampled more aggressively near the object surface to foster the learning of the object geometry.
We obtained 1,000 feasible grasping and hanging poses of the gripper and the hook, respectively. For grasping, used an antipodal sampling scheme, similarly to Eppner al. (202T), to create candidate gripper poses and checked their feasibility using Bullet (Coumans & Bai (2016-2021) 2021). For hanging, we randomly sampled collision telisioo--fee hook poses and checked if it's kinematically trapped by the mug in the directions perpendicular the hook's main axis.
Grasping & Hanging:
Fig. shows some rough looks of the generated data. the end, we have a dataset of:
D D= ( T1:1100 1-1100 p 1:12500 F111200
1:1000 1:1000) (1) 131 Igrasp Thang
fi=
which we divided into 78 training, 25 validation and 28 test sets.
4.2 DATA AUGMENTATION AND Loss FUNCTION
Data Augmentation:
While randomizing the azimuth, elevation and distance of the camera provides all possible appearances of the object, it still cannot account for the roll angles of the camera and off-centered images. To show the network all possible images that it can encounter when deployed later and mitigate the size-ambiguity issue, propose to use data augmentation technique based on Homography warping: In each iteration, for a randomly sampled set of images, we arti- ficially perturb the roll angle of each camera and the estimated object center position (at which the cameras are looking). Also, for is modified if the radius of the bounding sphere is 15 cm so that smaller objects can appear smaller in the transformed images. This results in new rotation matrices, R. and intrinsic matrices. K, of the cameras. We then compute the corresponding Homography transformation and warp the images accordingly (details in Appendix A.
u
: W(R, K)
U
+ WKRR RK
(8)
Random cutouts are also applied to address the object occlusion. Fig. depiits how this image augmentation works. For grasping and hanging, generate 6D random poses qusk P in each
iteration and, similarly to Atzmon & Lipman (ippmn)(202)), set the training target to be unsinged distances
in SE(3) to the set of the feasible poses: d(q: qusk [:Muu = min; I.... Noa llq q
"To encourage more precise prediction around the constraint manifolds, we used a weighted sum of feasible pose and random pose qusk tgtssiblle (1 t))rand ~ U(, 1) where the position of good is from the normal distribution and its quaternion is sampled uniformly.
5

---

Under review as a conference paper at ICLR 2022
Training: For the overall network training, we first choose minibatch of mugs from which subset of augmented images with their camera poses and intrinsics, (I2-Nore Tr1:Nown K
a subset of SDF data, (pl:Nste, FL:NNN and the grasping/hanging data, (g1:Naa dussk are sampled. The images are encoded only once per iteration and then the SDE, grasping, hanging features are queried at the sampled points and poses. The overall loss is given as Liotal Luar + Cggasp + Liung, where typical L1 loss is used for SDFs, i.c. Lutr Ngpr Exer
Nanr (dwr(p') S SDP,
and the sign-agnostic L1 loss in (Atzmon & Lipman 2020) for grasping and hanging, i.c., Lask Neyt Ee'r | | dinnk Vtask E {grasp, hang) The feature head and backbone are trained end-to-end. Specifically, shown in Fig. [1]] the grasp and hang interaction points are defined as (3 x 3 x 3) grid points around the gripper center and as 5 points along the hook's 's main axis, respectively.
used Neiews = 4, NSDF 300, N Ngrasp 100, Nhang 100. As
5 SEQUENTIAL MANIPULATION PLANNING WITH LEARNED FEATURES
raw images & masks,
multi-view processing (Sec 5.2)
obj images, cam poses, intrinsics
camera poses, intrinsics
P1
interaction
robot configuration
forward kinematics
robot frame's pose
feature
prediction
(Sec
3)
Figure 3: The proposed interaction feature prediction scheme for manipulation planning
In order to compute a full trajectory of the robot and the objects that it interacts with, the learned features can be integrated into any constraint-based trajectory optimization and manipulation plan- ning framework, using the features as differentiable interaction constraints. Logic- Geometrc Programming (LGP) (Toussaint et al., 2018) as an optimization-basel manipu- lation planning framework. In typical manipulation scenes, cameras are equipped such that their views cover a wide range of the environment. As shown in Fig. 3 we warp raw images of the entire scene to get object-centric images and corresponding camera extrinsics/intrinsics and compute the robot frame's pose via forward kinematics to feed them into the network. Section [5. presents how the learned constraint functions are integrated into sequential manipulation planning problems and Section 5.2 5.2 discusses the proposed warping procedure.
this work, we adopt
5.1
LOGIC-GEOMETRIC PROGRAMMING FOR MANIPULATION PLANNING
The core concept of manipulation planning is the rigid transformations of objects. For an object transformed by åg E SE()), we define rigid transformation of the interaction feature as:
[(g)) (-) Lusk (5 1.)
(9)
which is equivalent to transforming the representation as ((gg)l1)(( = (R(ōg)T (((5))))
By composing the forward kinematics with the feature
Husk (x, 5q) = (T(8q) o FK) (x),
(10)
we obtain an interaction feature as function of a robot joint configuration æ and object's rigid transformation that can be equipped into manipulation planning.
Now are ready formally formulate manipulation planning problems. For n-joint robot and m rigid objects, LGP is a hybrid optimization problem over the number of phases K N, sequence of discrete actions a and sequences of the robot joint configurations &1:KT, x E IR" and the object's rigid transformations åqI.KT, åq E SE(3) The trajectory is discretized into steps per phase. A discrete action aą describes which interaction should be fulfilled at the end of the phase k i.c., which mug to pick
which hook to hang grasped mug, and uniquely
For simplicity's 's sake, we abuse the notation of Equation as druak (9) when it doesn't lead confusion.

---

Under review as a conference paper at ICLR 2022
determines an interaction mode 5 suce(sk--1 ag), ie.. whether each mug is grasped or hung on a particular hook. Suppose that a discrete action sequence a and the corresponding modes $1.K with Sæ E Sgaal are proposed by logic tree search. We define the geometric path problem as 2nd order Markov optimization (Toussaint 2017):
KT Σ I ( s.t.Vk((I.....)) K) VHEH(s.m)) H ((2)-2a, oq 20).2 00, (11)
min RIRT SQUAT 1=1
where the initial joint states 1-1:0 and objects' transformations 6q-1:0 1:0 = O are given. f a path cost that penalizes squared accelerations of the robot joints. H[Sk, au ) is a set of path constraints the discrete state and action impose on the geometric level at each phase k(t)
Lt/TT; these constraints include physical consistency, collision avoidance, and the learned interaction constraints that ensure the success of the discrete action ag. Lastly, In a&) decides the time slice and object index that are subject to the constraint H. Appendix A. introduces the set of imposed constraints in detail. As all the cost and constraint terms are differentiable and their Jacobians/Hessians are sparse, we can solve this optimization problem efficiently using the Gewss-Newton optimization method.
5.2 MULTI- PREPROCESSING
Let MA E {0, 1| be the object masks available along with the raw images In, Vn = I,..., Neam- We first solve the following optimization to find position and radius of the minimal bounding sphere such that the warped images contain all the object pixels in the original images:
min PER
T,
s.t. (w,o)E((() (f) (M. ("" (1.... : II R. R), w)ll2 < 11, (12)
where R can be obtained from the sphere center p and the camera position t. and K is computed as for = 2 aresin( It pl 2/rr from which the warping W is defined as in 1 Equation g After solving the optimization above, we fix the camera orientations R. change the intrinsics as if the bounding sphere has radius of 15 em and finally warp the raw images accordingly. Fig. [2 shows the raw images from an example environment and the images warped by the multi-view processing.
EXPERIMENTS
6.1 PERFORMANCE OF LEARNED FEATURES
The key techniques of the proposed framework are threefold: the pixel-aligned local im- age features, the implicit function scheme. To examine the benefits from each component, three baselines are considered. (i) Global image features: The first baseline still represent an object as an implicit function but the image encoder outputs global image feature as shown in Fig. [3(b rather than having the pixel-aligned local feature extraction; we used the 344 architecture as the image encoder and fixed the other model specifications. (ii) Vector object representations: The second baseline represents
Baselines:
the 3D space representations and the task guided learning
object as a Lnite-dimensional vector instead of an implicit function, as shown in Fig. [3c] the representation network first computes the image features from the images using ResNet-34 and the camera features from the camera parameters using a couple of fully connected layers. Two features are then passed to another couple of fully connected layers to produce the object representation vector. The feature heads take as input the frame's pose well as the object representation vector. (iii) SDF represen- tations: The last baseline uses SDFs as object representations; the network architecture for the SDF feature remains the same, but the grasping and hanging heads take as inputs set of the keypoints' SDF values instead of the d-dimensional representation vectors. The SDF values are detached when passed to the grasp/hang heads so the backhone is only trained by the geometry (SDF) data.
Evaluation Metric: Regarding the shape reconstruction,
report the Volumetric IoU and the Chamfer distance. To measure these metries, we first randomly sampled 4 images from the dataset and reconstructed the meshes from the learned SDF feature using the marching cube algorithm (See Fig. I0 The volumetric IoU is the ratio between the intersection and the union of the reconstructed and ground -tuth meshes which is (approximately) computed on the 100 grid points around the objects. To compute the Chamfer distance, we sampled 10,000 surface points from each mesh and
NNote that og denotes rigid transformations applicd to object's implicit representation, not absolute poses.
7

---

Under review as a conference paper at ICLR 2022
S
Figure 4: SDF's predicted by PIFO and the global
Figure 5:
Sequential manipulation scenarios:
image feature model.
Single- three-mug hanging and handover.
ToU
Chamfer-LL (x10-3)
Grasp+c (%)
Hang+c (%)
PIFO
0.816 0/16/0.655 0.656
3.2676.990
88.1/82.5
94.0 78.9
Global Img. Feat.
0.697 0.581
7.42 / 9.49
82.7 75.7
91.2/78.2
Vector Obj. Repr.
0.036 0.014
38.6 / 39.7
0,5
0.0
SDF Obj. Repr.
0.845 0.667
4.90 / 6.83
67.9/ 64.3
3.7/4.3
PIFO (2 views)
0.760 0.577
6.14/ 8.84
82,97 77.1
88.2772.1
PIFO (8 views)
0.851 0.683
4.78 / 6.34
88.7 / 85.0
96.5 / 82.5
Table I: Individual Feature Evaluation with views (Training / Test).
averaged the forward and backward closest pair distances. To evaluate the learned task features, we solved the unconstrained optimization q" = arg min, I1 (q)ll², task E {grasp, hang} using the Gauss-Newton method, Starting from this solution, we then solved the second optimization problem by including the collision feature (details in Appendix A.3 q" arg ming (1)ll + Wealllobati ()l Because the local optimization method can be stuck at local optima, ran the algorithm from 10 random initial guesses in parallel and picked the best one. The optimized pose is then tested in simulation and the success rates are reported in Table []
Result: Table D shows that the SDF representation has the best shape reconstruction performance: PIFO is slightly worse, followed by the other two frameworks. On the other hand, the task perfor- mances of PIFO are significantly better than the others. The SDF representation is especially worse in the hanging task, which implies that SDFs along the line are not sufficient for the feature predic- tion and our task- fusk-guided representation simplifies the feature prediction. Fig. 4 depicts SDF values of an unseen mug with complex shape handle predicted by PIFO and the global image feature model; one can observe that the global image feature model reconstructed the handle shape as being more "typical" and the pixel-aligned representation was better able to capture fine-grained details. PIFO was also tested with the different numbers of input images and can be seen from the last two rows of the table that the more images we put in, the better performance the network shows. Tables 2 and 3 report all combinations of the metrics and the number of views.
6.2 SEQUENTIAL MANIPULATION PLANNING VIA LGP
As shown in Fig. 5 we considered three manipulation scenarios: Basic pick & hang, long- long-horizon three-mug hanging, and handover. In the first scenario, the environment contains one robot arm, one hook, one mug and cameras (as in Fig. a)), and the interaction modes are constrained by the discrete action sequence of [GRRSS, gripper, mug), (HANG, hook, mug)]. 10 mugs were picked from each of the training and test data sets and their initial poses are randomized When executed the optimized trajectory in the Bullet simulation, the success rates on the train and test mugs were 50 % and 40 %, respectively. If we allow the method to re-plan and execute when it failed, the success rates increased to 90% and 70%, respectively [playlisti] The three-mug scenario consists of 6 discrete phases with [GRRSSP, gripper, mugl), (HANG, M hook, mugl), (GRASP, gripper, mug2), (HANG, U_hook, mug2). (GRASP, gripper, mug3), (HANG, L.hook, mug3)]. The handover scenario has two arms at different heights and the target hook is placed very high, requiring two arms to coordinate handover motion; the corresponding discrete actions are [GRRSSP, R-gripper, mug), (GRASP, L_gripper, mug), (HANG, U_hook, ]. Fig. 5 shows the last configurations of the optimized plans; we refer the readers to Figs. [7 and the videos [playlist]] for clearer views.
One important attribute of our framework that, while most existing works train generative models that directly produce the interaction poses, ours models interactions as equality constraints which can jointly be optimized with other planning fea- tures. To see the benefits of such joint optimization, we considered the following inverse kinematics problem with generative model: For the basic pick & hang and handover scenarios, we optimized
Inverse Kinematics with Generative Models:
'effore solving the full trajectory optimization, first optimized each feature in Sec. 6.1 and added small regularization using the optimized poses guide the optimizer away from local optima.
8

---

Under review as a conference paper at ICLR 2022
0.125
0.100-
2
3.075
2.050
8.025
8.000
CP
IEP2
FCP
Feiep2
ICP
ICP2
FCP F-10P2
Figure 6: Position and Orientation Errors
Figure 7: (a) Reference (b-c) Imitation
each interaction pose separately as in Sec. 6.1 and checked if these individually optimized poses are kinematically feasible, i.e. whether or not the inverse kinematics problems have solution. Even though the mug's initial pose was given such that the first gasping is feasible, 53 out of 100 pairs of grasp and hang poses were infeasible for the pick & hang scenario and 86 out of 100 sets for the handover scenario, i.e., many of the sampled poses led to a collision or an infeasible robot configu- ration for hanging/handover. Some failure cases are depicted in Figs. 18 This will become even worse when the whole trajectory is optimized sequence length gets longer, not only should an exponentially larger number of planning problems be solved to find set of feasible poses, but also the found poses not guaranteed to be optimal. Again, the joint optimization with our constraint models doesn't 't raise such issues.
the mug's initial pose is given arbitrarily. As the
6.3
6D POSE ESTIMATION AND ZERO-SHOT IMITATION
Figs. 14d 15 visualize some principal components of the image features and of the representation vectors, respectively. It can be seen that each component represents a certain property of the objects, such as inside vs. outside, handle vs. other parts, or above vs. below. This enables the image- based pose estimation which we call feature-based closest point (FCP) matching, i.e., the problem of finding the relative pose of target mesh w.r.t a model mesh whose pose is given, without defining any canonical coordinate of the objects. To this end, we first queried the backbone at 103 and 5 grid points around the target and the model, respectively, (as shown in Fig. (0(dd) with their own images. For each model grid point, the corresponding target point is obtained such that their representations are closest. Finally, FCP computes the rigid transformation that minimizes the sum of their Fuclidean distances. We compared this to the conventional iterative closest point (ICP) algorithm on point clouds, i.c., the problem of finding the pose minimizing the Euclidean distance of two sets of point clouds obtained from depth cameras. Instead of using depth images, the point clouds can be obtained from the reconstructed meshed via the learned SDF features and we call ICP2. The point clouds' size was 1000. Fig. shows the position and orientation errors of the 6D pose estimation. FCP performs much better especially in the orientation because, as already widely known, ICP easily gets stuck at the local optima: ICP2 was similarly as worse as ICP. Specifically, significant improvement was observed in F+ICP2 where used the FCP results starting points of ICP2 (which is performed without depth images). Fig. 21 depicts some of the results.
Another observation from the PCAs is that the semantics of the representation are consistent across different objects, e.g. the handle parts of different mugs have similar representations, which im- plies a pose of one object can be transferred into another through it. We therefore considered an image-based zero-shot imitation scenario, where the environment contains one robot arm, one target mug (filled with small balls) and 4 cameras as shown in Fig. 7 We manually designed the pouring motion and captured the camera images of pre- and post-pouring postures of the mug, Ppre - (Tpre Tue. K pre) and post (Iposts Tposs poos), respectively. For new mug, we solved the sequential manipulation planning with [GRASS, gripper, mug), (POSEFCP, Ppre. mug), (POsEFCP, Ppest. mug)l, where (POSEFCP, :. imposes the aforementioned FCP constraint. That is, the tra- jectory optimizer tries to match each part of the new object to the corresponding part of the target while coordinating the global consistency of the full trajectory, thereby resulting in the imitation of the reference motions only from the posed images. Fig. shows the optimized post-pouring posture; Sshw show the reference and its imitations more clearly.
Figs. 22 23 and the videos [playlist3]
CONCLUSION
This work transferred the idea of neural implicit representations robotic manipulation applica- tions. The pixel-aligned nature enables the proposed representations to inferred from posed cam- era images and easily capture fine-grained
details of the objects. We demonstrated that the learned representations and interaction features allowed for formulating and solving general sequential ma- nipulation problems only from images within the LGP framework. The learned representations generalize to unseen objects, enabling manipulation planning and zero-shot imitations for them.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Matan Atzmon and Yaron Lipman. SAL: Sign agnostic learning of shapes from raw data. In Pro- ceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 2565- 2574, 2020.
Michel Breyer, Jen Jen Chung, Lionel Ott, Siegwart Roland, and Nieto Juan. Volumetric grasping network: Real-time dof grasp detection in elutter. In Conference on Robot Learning. 2020.
Angel X. Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, Jianxiong Xiao, Li Yi, and Fisher Yu. ShapeNet: An Information-Rich 3D Model Repository. Technical Report arXiv:1512.0001 [cs. Stanford University - Princeton University - Toyota Technological Institute at Chicago, 2015.
Xu Chen, Zijian Dong, Jie Song, Andreas Geiger, and Otmar Hilliges. Category level object pose estimation via neural 139-156. Springer, 2020.
In European Conference Computer Vision, pp-
Erwin Coumans and Yunfei Bai. Pybullet, python module for physics simulation for games, robotics and machine learning. Itttp://pybullet..omm 2016-2021.
Danny Driess, Jung-Su Su Ha, Mare Toussaint, and Russ Tedrake. Learning models as functionals of signed-distance fields for manipulation planning. In 5th Annual Conference Robot Learning, 2021. (RRL:tttts://ppeeeeeee
Clemens Eppner. Arsalan Mousavian, and Dieter Fox. ACRONYM: A large-scale grasp dataset based simulation. In 202/ IEEE Int. Conf. Robotics and Automation, ICRA, 2021.
Peter Florence, Lucas Manuelli, and Russ Tedrake. Dense object nets: Learning dense visual object descriptors by and for robotic manipulation. Conference on Robot Learning, 2018.
Peter Florence, Lucas Manuelli, and Russ Tedrake. Self-supervised correspondence visuomotor policy learning. IEEE Robotics and Automation Letters, 5(2): 2019.
Wei Gao and Russ Tedrake. kPAM 2.0: Feedback control for category-level robotic manipulation.
IEEE Robotics and Automation Letters, 6(2):2962-2266) 2021.
Amos Gropp, Lior Yariv, Niv Haim, Matan Atzmon, and Yaron Lipman. Implicit geometric regular- ization for learning shapes. In International Conference Machine Learning, pp. 3789-3799. PMLR, 2020.
Jung-Su Ha. Danny Driess, and Marc Toussaint. A probabilistic framework for constrained ma- nipulations and task and motion planning under uncertainty. In Proc. of the IEEE Int. Conf. on Robotics and Automation (ICRA), 2020. doi: 2.110090A
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition, 2016
Philipp Henzler, Jeremy Reizenstein, Patrick Labatut, Roman Shapovalov, Tobias Ritschel, Andrea Vedaldi, and David Novotny. Unsupervised learning of 3d object categories from videos in the wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4700 4709, 2021.
Zhenyu Jiang, Yifeng Zhu, Maxwell Svetlik, Kuan Fang, and Yuke Zhu. Synergies between affor- dance and geometry: 6-dof grasp detection via implicit representations. In Robotics: Science and Systems (RSS), 2021.
Marian Kleineberg, mesh-to-sdf Calculate signed distance fields for arbitrary meshes.
https:
//github.commmaaraaa
to sdf 2021.
Jeffrey Mahler, Jacky Liang, Sherdil Niyaz, Michael Laskey, Richard Doan, Xinyu Liu, Juan Apari- cio Ojea, and Ken Goldberg. Dex-net 2.0: Deep learning to plan robust grasps with synthetic point clouds and analytic grasp metrics. In Robotics: Science and Systems (RSS), 2017.
10

---

Under review as a conference paper at ICLR 2022
Lucas Manuelli, Wei Gao, Peter Florence, and Russ Tedrake. kPAM:
Keypoint affordances for category-level robotic manipulation. In International Symposium on Robotics Research (ISRR), 2019.
Lucas Manuelli, Yunzhu Li, Pete Florence, and Russ Tedrake. Keypoints into the future: Self- supervised correspondence model-based reinforcement learning. Conference on Robot Learn- ing. 2020.
Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Oc- cupancy networks: Learning 3d reconstruction in function space. In Proceedings the Conference Computer Vision and Pattern Recognition. pp. 4460-4470, 2019.
the IEEE/CVF
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. NeRF: Representing scenes as neural radiance fields for view synthesis. In European conference on computer vision, pp. 405-421. Springer, 2020.
Arsalan Mousavian, Clemens Eppner, and Dieter Fox. 6-dof GraspNet: Variational grasp generation for object manipulation, In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 2901-29100 2019.
Adithyavairavan Murali, Arsalan Mousavian, Clemens Eppner. Chris Paxton, and Dieter Fox. 6-dof grasping for target-driven object manipulation in clutter. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pp. 6232-6238. IEEE, 2020.
Michael Niemeyer, Lars Mescheder, Michael Oechsle, and Andreas Geiger. Differentiable volumet- ric rendering: Learning implicit 3d representations without 3d supervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3504-3515, 2020.
Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. DeepSDF: Learning continuous signed distance functions for shape representation. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 165-174, 2019.
Keunhong Park, Arsalan Mousavian, Yu Xiang, and Dieter Fox. LatentFusion: End-to-end differ- entiable reconstruction and rendering for unseen object pose estimation. Proceedings of the JEEE/CVF conference on computer vision and pattern recognition, pp. 107 10710-10719, 2020.
Zengyi Qin, Kuan Fang, Yuke Zhu, Li Fei-Fei, and Silvio Savarese. KETO: Learning keypoint representations for tool manipulation. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pp. 7278-7285. IEEE, 2020.
Jeremy Reizenstein, Roman Shapovalov. Philipp Henzler, Luca Sbordone, Patrick Labatut, and David Novotny. Common objects in 3d: Large-scale learning and evaluation of real-life 3d cat- egory reconstruction. In Proceedings of the 1EEE/CVF International Conference on Computer Vision, pp. 10901-10911, 2021.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedi- cal image segmentation. In International Conference Medical image computing and computer- assisted intervention, pp.
Springer. 2015.
Shunsuke Saito, Zeng Huang, Ryota Natsume, Shigeo Morishima, Angjoo Kanazawa, and Hao Li. PIFu: Pixel-aligned implicit function for high-resolution clothed human digitization. In International Conference on Computer Vision (ICCV), pp. 2304 2314, 2019.
Vincent Sitzmann, Michael Zollhoefer, and Gordon Wetzstein. Scene representation networks: Con- tinuous 3d-structure-aware neural scene representations. Advances in Neural Information Pro- cessing Systems, 32:1 1211111111.22. 2019.
Lars Mescheder Marc Pollefeys Andreas Geiger Songyou Peng, Michael Niemeyer. Convolutional occupancy networks. In European Conference on Computer Vision (ECCV), 2020.
Martin Sundermeyer. Arsalan Mousavian, Rudolph Triebel, and Dieter Fox. Crrsact-GraspNett Ef- ficient 6-dof grasp generation in cluttered scenes. In IEEE International Conference on Robotics and Automation (ICRA), 2021.
11

---

Under review as a conference paper at ICLR 2022
Andreas ten Pas, Mareus Gualtieri, Kate Saenko, and Robert Platt. Grasp pose detection in point clouds. The International Journal of Robotics Research, 36(13-14)14555-443, 2017.
Mare Toussaint, A tutorial on Newton methods for constrained trajectory optimization and relations to SLAM, Gaussian Process smoothing, optimal control, and probabilistic inference. In Jean-Paul Laumond (ed.), Geometric and Numerical Foundations of Movements. Springer, 2017.
Mare Toussaint, Kelsey Allen, Kevin A Smith, and Joshua B Tenenbaum. Differentiable physies and stable modes for tool-use and manipulation planning. In Robotics: Science and Systems, 2018.
Marc Toussaint, Jung-Su Ha, and Danny Driess. Describing physics for physical reasoning: Force- based sequential manipulation planning. IEEE Robotics and Automation Letters, 2020. doi: 200000
Alex Trevithick and Bo Yang. GRE: Learning general radiance field for scene representation and rendering. In International Conference on Computer Vision (ICCV), 2021.
Dylan Turpin, Liquan Wang, Stavros Tsogkas, Sven Dickinson, and Animesh Garg. GIFT: Gener- alizable interaction-aware functional tool affordances without labels. In Robotics: Science and Systems, 2021.
Mark Van der Merwe, Qingkai Lu, Balakumar Sundaralingam, Martin Matak, and Tucker Hermans. Learning continuous reconstructions for geometrically aware grasping. In 2020 IEEE Interna- tional Conference on Robotics and Automation (ICRA), pp. 11516- 11522. IEEE, 2020.
Qiangeng Xu, Weiyue Wang, Duygu Ceylan, Radomir Mech, and UIrich Neumann. DISN: Deep implicit surface network for high-quality single-view 3d reconstruction. Advances in Neural In- formation Processing Systems, 02:492-502, 2019.
Lior Yariv, Yoni Kasten, Dror Moran, Meirav Galun, Matan Atzmon, Basri Ronen, and Yaron Lip- man. Multiview neural surface reconstruction by disentangling geometry and appearance. Ad- vances in Neural Information Processing Systems, 33, 2020.
Lin Yen-Chen, Pete Florence, Jonathan T. Barron, Alberto Rodriguez, Phillip Isola, and Tsung- Yi Lin. iNeRF: Inverting neural radiance fields for pose estimation. In JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2021.
Yifan You, Lin Shao, Toki Migimatsu, and Jeannette Bohg. OmniHang: Learning to hang arbitrary objects using contact point correspondences and neural collision estimation. In IEEE Interna- tional Conference on Robotics and Automation (ICRA). IEEE, 2021.
Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa. PixeINeRF: Neural radiance fields from one or few images. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021.
Wentao Yuan, Chris Paxton, Karthik Desingh, and Dicter Fox. SORNet:
Spatial object-centric representations for sequential manipulation. In 5th Annual Conference on Robot Learning, 2021.
URL htt.ps:://ppereviieeeee net.eeewwweeeeeeeeeeeeeeeeeeeeee /iee/forre forum? 1d=
Andy Zeng, Pete Florence, Jonathan Tompson, Stefan Welker, Jonathan Chien, Maria Attarian, Travis Armstrong, Ivan Krasin, Dan Duong, Vikas Sindhwani, and Johnny Lee. Transporter networks: Rearranging the visual world for robotic manipulation. Conference on Robot Learning (CoRL), 2020a.
Andy Zeng, Shuran Song, Johnny Lee, Alberto Rodriguez, and Thomas Funkhouser. Tossingbot: Learning to throw arbitrary objects with residual physics. IEEE Transactions on Robotics, 36(4); 1307-1319, 2020b.
12

---

Under review as a conference paper at ICLR 2022
A APPENDIX
A.1 HOMOGRAPHY TRANSFORMATION
The idea of the Homography warping is that two images taken by cameras at the same position but with different orientations and intrinsics be transformed into each other. Suppose that we have a source image I with the camera position t, rotation matrix R and projection matrix K and that an object is inside bounding sphere at E RS with a radius E Rt. An image focusing the bounding sphere can be taken from a (synthetic) camera at the same position with the view direction as t p and the field of view angle as 2 aresin (|t pll2/r), from which we can compute the new camera rotation matrix R and the intrinsic K.
Given R and K, the new field warped by the corresponding Homography can be obtained as follows: First, a pixel in the source image, P1 (u, v, 1). is reprojected into ray in the 3D space: Pi = K- Next, the ray is viewed in the new camera coordinate: P2 = RP: Lastly, this ray is KP2. Putting all together, the Homography
projected back into pixel in the new camera: P22 warping is given as:
W(R, K)
w
K
R RK
(13)
where w is the parameter that makes the last element of the output homogeneous coordinate 1, which
n
t
results in the warped image I with its camera pose T
and intrinsic matrix K.
1
A.2 MANIPULATION CONSTRAINTS
In this work, consider two discrete actions, (GRASP, gripper, mug) and (HANG, hook, mug), for grasping and hanging, respectively. Each action imposes three constraints on the path as follows.
The action ak = (GRASP, gripper. mug) first imposes the learned grasping constraint at the end of its phase, H gapp (2t, )q:) 0, t KT, i.e.,
(T gmpp FK;) (xx) = 0,
(14)
where and j are indices of the mug and the gripper, respectively. It also imposes the zero-impact switching constraint at & kT, i.e.,
ú; 0,
(15)
where 6, is a joint velocity computed from -;-1 and æ, via finite difference. Lastly, introduces an equality constraint on the gripper's approaching direction for collision-free grasping; more precisely, the constraint imposed at / (KT - 2, kT - 1,TT] as:
0
a
Capproach
0
(16)
where dāū is the mug's s acceleration in the gripper's coordinate computed from 't(ôgi )t(og/_)) and (t(q)) via finite difference, and aapproach E RR+ is the predefined approach- ing acceleration magnitude. The gripper's z axis is depicted in Fig. [ blue arrow. Combined with the above zero-impact constraint, this constraint enforces the gripper to approach the mug in the gripper's -z axis direction and to stop moving at the end of the phase.
Similarly, the action the zero-impact and hanging approaching constraints
(HANG, hook, mug) consists of the learned hanging constraint,
(T (T(qq(r) [dhang] FK₃) (œET) 0,
(17)
T = 0,
(18)
0 0
à Gapproach
Vt E - -
(19)
where and j are indices of the mug and the hook, respectively, and the hook's z axis is the blue arrow in Fig. [T.bb (or outer product of the red and green arrow).
13

---

Under review as a conference paper at ICLR 2022
The discrete actions above affect the consecutive symbolic states. While 8 indicates mug is grasped by a gripper or hung on a hook at the phase k; we impose the following path constraint:
- (x) FK, (z Vt E {(A 1)T .KT}
(20)
where and j are indices of the mug and the gripper/hook, respectively. Effectively this introduces static joint between the two frames (Toussaint al. 2018) so the mug moves along with its parent frame (the gripper or hook). The collision constraints are also imposed along the trajectory, where the pair collisions with the mug are computed by the learned SDF feature. We introduce the collision feature in the following section.
We would like to emphasize that our manipulation planning framework is not limited by the con- straints we introduced above, but it can incorporate any existing other constraint models and meth- ods, e.g., (Toussaint et 2018 Ha al. 2020 Toussaint al. 2020 Driess al., 2021.
A.3 DEFINING MAIR-COLLISION CONSTRAINTS WITH SDFs
For manipulation planning problems written only by convex meshes, the distance or penetration of two objects, which we call pair-collision features, computed with either Gilbert-Johnson- Keerthii (GJK) for non-penetrating objects or Minkowski Portal Refinement (MPR) for penetrating objects. In this section, we introduce how to define pair-collision features when one or both objects are given SDFs.
SDF vs. Sphere: Let öq:- qj and ry be the rigid transformation of PIFO, the sphere's 's pose and radius, respectively. Then the pair-collision feature is simply given by:
dif (ōqq: (oSDE) (t(,))) r--
(21)
SDF vs. Capsule: Let og; 9j h2 and ry be the rigid transformation of PIFO, the capsule's pose, height and radius, respectively. The pair-collision feature is given by the solution of the following optimization:
0
=(()))
dij
min T(ōg) [OsDe] -h5/25:5h,p2
R(g,)
0
-r,
(22)
2
SDF vs. Mesh: Let åq. and q; be the rigid transformation of PIFO, the mesh's pose, respectively.
dij
min ER P2ER3 T(gg, Hesor (esor (p1)=0 (p2) =0
nTTPp (p2 P1),
(23)
where ny is the normal vector of ØsDr Pı and dy (p2) is the signed distance of P2 the mesh computed by GJK/MPR.
SDF vs. SDF: Let og, and ōg, be the rigid transformations of two PIFOs.
dij
min ER PIER. T T(6q: (P) T(84,) ) (p2))=
ni (p2 P1).
(24)
The optimizations in Equation 22 Equation 24 should be run multiple times from different initial guesses because the object shape represented as SDF can non-convex. In practice, we found approximating the meshes by number of spheres and computing the collision feature much more efficient because querying the network ÓSDF at multiple points can be done in parallel on GPUs.
A.4 NETWORK PARAMETERS
Image encoder has the U-net architecture (Ronneberger et al. 2015), especially with the headless ResNet-34 (He et al., 2016) its downward path and two residual 3 x 3 convolutions followed by up-convolution as the upward path. The number output channels is 64.
3D reprojector computes the coordinate feature as 32-dimensional vector using one linear+ReLU layer and concatenate it with the local image feature. They are passed to two hidden layers with the
14

---

Under review as a conference paper at ICLR 2022
width of (256, 128) followed by ReLUs. Therefore, the dimension of the representation vector is 128.
SDF head takes input one representation vector and computes the output through one hidden layer with the width of 128 followed by ReLUU
Grasp and hang heads take as input 27 and 5 representation vectors at their interaction points (depicted in Fig. 11 and predict the feature through two hidden layers with the widths of (256, 128) followed by ReLUs.
As shown in Figure 13 the network structures for comparison in Section 6.] was kept similar to the above possible. Image encoders of the global image feature and vector representation networks are the ResNet-34 returning o4-dimensional vector. The feature head structures remain the same, but, because the vector representation scheme doesn't 't represent objects as implicit functions, the in- put of their feature head is the frame's pose 7-dimensional vector (3D translation+ 4D quaternion). The grasp and hang heads of the SDF representation scheme take as input 27- and 5-dimensional vectors of their interaction points SDF values.
# of views
Method
SDF error (x10
Volumetric IoU
Chamfer-Ly (x10
PIFO
2.91 4466
0.760 / 0.577
6.14 8.84
2
Global Image Feature
3.58 / 4.96
0.642 / 0.515
8.50 / 10.8
Vector Representation
15.6 15.8
0.045 0.046
39.1 40.4
SDF Representation
2.11 3.48
0.786 / 0.622
5,78 / 8.13
PIFO
2.20 3.38
0.816 / 0.656
5,26 / 6.90
4
Global Image Feature
2.82 3.93
0.697 / 0.581
7.42 / 9.49
Vector Representation
1
0.036 0.014
38.6 39.7
SDF Representation
1,43 / 2.73
0.845 / 0.667
4.90 / 6.83
PIFO
1.68 / 2.72
0.851 / 0.683
4.78 / 6.34
8
Global Image Feature
3.51
0.728 / 0.607
6.75 8.80
Vector Representation
14.6 / 15.3
0.033 / 0.006 0.878 / 0.703
38.7 40.6
SDF Representation
1.07 / 2.07
4.51 / 6.06
Table 2: SDF Feature Evaluation (Training / Test). The SDF enrors were also measured at the same grid points loU.
15

---

Under review as a conference paper at ICLR 2022
of views
Method
Grasp (%)
Grasp+c (%)
Hang (%)
Hang+c (%)
PIFO
65.8 55.4
82.9 / /7.1
87.2 71.4
88.2 / 72.1
Global Image Feature
67.6 63.9
80,9 70.4
88.3 70.4
86.3 / 77.8
Vector Representation
13.2 / 12.9
0.8/ 0.4
25.6/211
0.0
SDF Representation
41.2/ 55.3
49.6/ 45.7
2.6/1.1
3.3
PIFO
69.0/ 63.9
88.1/88 82.5
88.7
94.0 78.9
4
Global Image Feature
62.3 61.8
82.7 775.7
90.3/75.7
91.2/78
Vector Representation
21.2 22.5
0.4
55.1/46.4
0.0
SDF Representation
49.1 46.1
67.9 64.3
3.3/299
3,7
PIFO
71.9 / 69.3
88.7/85.0
91.7 91.7/80.4
96.5 / 82.5
Global Image Feature
71.3 67.1
79.0
77..5
92.9 / 80.4
Vector Representation
29.0/23.9
0.7
/
0.0
SDF Representation
/1.4/52.1
/
4.6/6.1
5.7
Table 3: Task Feature Evaluation (Training Test).
(a) SDF
(b) Grasp
(c) Hang
(d) Camera
(e) Image
Figure 8: Data Generation
16

---

Under review as a conference paper at ICLR 2022
25
25
25
25
50
50
50
50
75
75
75
75
100
000
00
00
125
25
25
15
100
100
100
100
(a) Before augmentation
0-
25
25
25
25
50
50
50
50
75
75
75
75
100
000
000
00
25
25
100
50
100
(b) After augmentation
Figure 9: Image Data Augmentation
(a) Train Mugs
(b) Test Mugs
Figure 10: Reconstruction via marching cube, Red: ground truth, Blue: reconstructed
17

---

Under review as a conference paper at ICLR 2022
Figure 11:
Key interaction points on the gripper and hook. Before passed to PIFO, their global positions are computed from the gripper's or hook's 's pose q, i.e., R(g)p, + t(g), Vi e
{1, ..., 3-
18

---

Under review as a conference paper at ICLR 2022
(a) Scene. Four cameras' poses depicted as coordinate systems where the origin is the camera location. axis (blue) pointing the view direction, and 2 and yy (red and blue) the directions of (u, v) coordinate of images.
3:0
720
FDB
300
320
350
X44
1050
2/6
5M6
X000
1259
350
360
1000
1250
29
06
750
1600
150
403
400
300
160
1001
(b) Raw images and masks
300
189
165
183
165
100
100
300
300
(c) Warped images (via the multi-view processing)
Figure 12: Multi-view processing.
19

---

Under review as a conference paper at ICLR 2022
Mestkommk Car
lobal
(a) PIFO
alchal faat
Bas
(b) Global Image Feature
(c) Vector Object Represnetation
Figure 13: Baseline Networks used for comparison.
20

---

Under review as a conference paper at ICLR 2022
(b)
Figure 14: First 5 principal components from PCA on image features. The first component indicates the object non-object areas, the second component distinguishes the handle parts, and the third one spots the above vs. below of the mugs, etc. Note that the network is trained only via the task feature supervisions.
0.0
0.0
0.0
0.0
0.0
0.0
0.1
0.1
0.1
0.1
0.1
0.1
0.1
0.0
0.1
0.0
0.1
0.0
0.0
-0.1
0.0
0.0
0.1
0.0
0.1
0.0
0.1
0.1
0.1
0.1
0.1
0.1
0.0
0.0
-0.1
-0.1
-0.
-0.1
00
-0.1
0.1
-0.1
001
-0.
0.0
-.1
0.0
-0.
00.1
0.0
0.0
0.0
0.0
0.0
0.0
0.1
0.1
0.1
0.1
0.1
0.1
-0.1
0.1
00
-0.1
0.1
00.1
-0.1
0.0
0.0
0.0
0.0
0.0
0.0
0.1
0.1
0.1
0.1
0.1
0.1
(a) Train mugs
(b) Test mugs
Figure 15: The first principal component from PCA representation vectors of the 3D surface points. It distinguishes the handles of the mugs from the other parts and is consistent across different mugs.
21

---

Under review as a conference paper at ICLR 2022
L=S
(b) L8855
(c) t=10
(d) 15
c t=855
(f) t=20
(g) 1=25
(h) 1-28.
(i) t=30
Figure 16: The three-mug scenario. 60 steps of robot configurations and rigid transformations of three mugs are jointly optimized via the proposed manipulation framework. This optimization is 107 di7ledimmniional decision problem (one 7DOF arm for 60 steps and one 7DOF mug for 51, 31, 11 steps 1071, the mug's rigid transformations before grasped are not included in optimization) and solved within minute standard laptop.
22

---

Under review as a conference paper at ICLR 2022
(a) 1=0
(b) 1=
(c) t=10
(d) 1=13.5
(e) 1=15
Figure 17: The handover scenario. 30 steps of the two arms' configurations and rigid transformations of the mug are jointly optimize dvia the proposed manipulation framework. This optimization is 567-dimensional decision problem (two 7DOF arms for 30 steps and one 7DOF mug for 21 steps 567, the mug's rigid transformations at the first phase are not included in optimization) and is solved within minute on standard laptop.
23

---

Under review as a conference paper at ICLR 2022
(a) Sampled grasp pose
(b) Sampled hang pose
(c) result (infiseable)
(d) Sampled grasp pose
(e) Sampled hang pose
IK result (infiseable)
(g) Sampled grasp pose
(h) Sampled hang pose
(i) IK result (infiscable)
Sampled grasp pose
(k) Sampled hang pose
(l) TK result (infiseable)
Figure 18: IK with generative models Pick & Hang. Separately generated poses often can not be coordinated due to the kinematic infeasibility, i.e., the robot joint angle limits, or the collision constraints.
24

---

Under review as a conference paper at ICLR 2022
(a) Sampled graspl pose
(b) Sampled grasp2 pose
(c) result (infiseable)
(d) Sampled graspl pose
(e) Sampled grasp2 pose
IK result (infiseable)
gg) Sampled graspl pose
(h) Sampled grasp2 pose
(i) IK result (infiscable)
(i) Sampled g pose
(k) Sampled grasp2 pose
(l) TK result (infiseable)
Figure 19: IK with generative models Handover. Separately generated poses often can not be coordinated due to the kinematic infeasibility, i.e., the robot joint angle limits, or the collision con- straints.
25

---

Under review as a conference paper at ICLR 2022
(a) Model (right) and target (left) mugs
(b) Point clouds for ICP
(c) Point clouds for ICP2 obtained from meshes reconstructed via ΦSDF
(d) Grid points for FCP
Figure 20: 6D Pose Estimation. (b) Point clouds for ICP are obtained from depth cameras the same locations/orientationss as the RGB cameras. The size of point clouds 1000. (c) Point clouds for ICP are sampled from the surfaces of the meshes reconstructed via the learned ΦSDF- The size of the point clouds is 1000. (d) FCP uses 10a grid points for the target and 5 grid points (in smaller area) for the model, respectively.
26

---

Under review as a conference paper at ICLR 2022
(a) ICP
(b) ICP2
(c) FCP
(d) F+ICP2
(e) ICP
(I) ICP2
(g) FCP
(h) F+ICP2
ii ICP
(i) ICP2
(k) FCP
(I) F+ICP2
(m) ICP
(n) ICP2
(o) FCP
(p) F+ICP2
Figure 21: 6D Pose Estimation Results the estimated poses are applied to the green meshes. ICP easily gets stuck at local optima while FCP produces fairly accurate poses which help F+ICP2 escape the local optima; note that FCP does not iterate to get the results.
27

---

Under review as a conference paper at ICLR 2022
(a)
(b) t=10
(c) t=15
Figure 22: Zero-shot Imitation reference motion. Two sets of posed images are obtained at t
10, 15.
28

---

Under review as a conference paper at ICLR 2022
(a) 1=5
(b) t=10
(c) t=15
(d) t=5
(c) t=10
(f) t=15
gg) 1=
(h) t=10
ii) t=15
(i 1=
(k) 1=
(I) t=15
Figure 23: Zero-shot imitation optimized motions. The FCP constraints imposed at f 10. 15. The imitations are achieved only from images, without defining the canonical coordinate/pose the objects.
29


Reviews:
[{"review_id": "XIW6anmlqR9", "reviewer": "Reviewer_zYJc", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n+ The idea is interesting and a natural progression from recent progress in 3D vision/representation learning and is a welcome change of pace where a vision problem is contextualized in a robotic task and developed with the intent to be a part of the full system.\n+ The related work is well covered and the motivation is setup clearly i.e. we need vision backbones that work for manipulation tasks.\n\nWeakness/Comments\n- The interesting task-specific and task-agnostic dynamic is brought up but not really explored within the paper in the context of representation learning. Seems like a missed opportunity.\n- While the broad strokes are clear (Fig 1, 2, Eq 4, etc) it was a bit hard to parse how the various components came together, in particular when instantiated given a particular task. At inference time, does the planning problem just take one image as input? is the image ego-centric? what are other inputs and do any of them fall under assumptions (like a 3D environment representation). \n- Some more algorithmic details would be helpful. How to choose what set of points are used to condition PIFO and what if this choice changes between tasks? Could something like Mask-RCNN work for object segmentation vs Eq 5?\n- Some more experiment/result details would be helpful. Why 2 view task performance drops compared to (1 view?) PIFO? In the Zero-shot imitation what's 'unseen' in test setting, new object geometry, anything else?", "summary_of_the_paper": "This paper build on recent progress in implicit object representations and where the representations are trained(or fine-tuned) along with a feature head on down stream manipulation tasks. Experiments are conducted by using this trained representation within an LGP formulation solved with Gauss-Newton optimizer to find plans for manipulation problems.", "summary_of_the_review": "Interesting idea and application, results are promising, but the write-up could be improved to understand the approach clearly.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Tr_sHl7aOJp", "reviewer": "Reviewer_gsUt", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. The motivation from neural implicit functions is interesting.\n2. The videos show that the method works in the mug hanging task.\n\nWeaknesses:\n1. This paper is very difficult to follow.\n2. It is very difficult to understand the framework by looking at Figure 1. Specifically, there seems to have 3 blocks in Figure 1, but in Section 3.1, the authors stated that the proposed framework consists of two parts. This is confusing and makes readers hard to follow.\n3. In the introduction, what does \"key interaction points which are rigidly attached to the robot frame\" means? How can a point attach to a frame?\n4. The related work section is too long (more than 1.5 pages).\n5. The method is only trained on different mugs. It is unclear how well the method would perform if given data of other categories.\n6. Since this paper studies a robotics problem, it is not convincing without showing video results using real robots.\n7. In Figure 1, I do not understand how to compute forward kinematics for a given robot pose. Forward kinematics are typically computed when given joint angles for deriving poses. I believe this is a clear technical error.\n8. The authors mentioned in Section 4.1 that the posed image data consists of 100 images. The authors should include an ablation study that analyzes how many images are needed. Is 100 sufficient? \n9. How are ground-truth SDF values computed? The authors did not mention nor providing references in Section 4.1.\n10. Since the method is only trained on 131 mesh models (as mentioned in Section 4.1) which is a very small dataset, I wonder if there is a possibility that overfitting happens. ", "summary_of_the_paper": "This paper studies the problem of learning representations for robotic manipulation tasks. The authors developed a method that represents objects as neural implicit functions. To method is trained in a data-driven fashion that learns the training pipeline from camera images to interaction features end-to-end. At each time step in a motion planning task, the implicit function is queried at interaction points that are attached to the robot frame. With the pixel-aligned representations, the representations at a certain spatial location correspond to the associated pixels of the images. Experiments are conducted on a mug hanging task. Videos in simulation environments show the effectiveness of the proposed method.", "summary_of_the_review": "Although the problem studies by this submission is interesting, this paper is very difficult to follow. There are some details missing and some technical errors in the paper. There are also some unclear details and missing experiments mentioned in the above section.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "85UKbXEq_3", "reviewer": "Reviewer_23pA", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- Neural implicit functions are still quite new, and investigating their use for robotic manipulation is a valuable direction. In particular, I think the idea of incorporating NIFs with LGP is worth exploring.\n\nWeaknesses:\n- There are not enough quantitative results in the paper to convince the reader that this method is better than others. Also, hanging mugs only is a relatively narrow set of experiments (as the paper is titled \"... for robotic manipulation\").\n- There are no comparisons to other works. Since the introduction mentions that traditional methods (that use traditional representations such as meshes or point clouds) has limitations, a comparison should be made with such a method. Further comparisons should include the Omnihang paper (You et al.). Additionally, the ablations are not that meaningful. It is well known that dense features will give better results than a single global vector (or a single-vector representation of an object). \n- The exposition could be more clear. For example, PIFO predicts graspness scores and hang-ness scores, however this is not mentioned in the introduction. The introduction is rather vague. Additionally, it is not clear how exactly the NIF is used in Equation 4. This seems to be a major contribution of the paper, yet it lacks detail.\n\n\nQuestions/Comments:\n- In P1 of Section 1, a stated problem with traditional approaches is that \"the representations have to be inferred from raw sensory inputs like images or point clouds\", however this contradicts the fact that the proposed method infers the NIF from images. Additionally, the last line of P2 also directly contradicts this.\n- \"Interaction features\" should be defined somewhere, it is not clear what this means in the introduction.\n- The introduction is lacking citations. For example, NeRF (Mildenhall et al.) should be cited when discussing it.\n- In Section 3.5, there has been no discussion of data augmentation up to this point. This is confusing, please clarify this.\n- In Eq. 6, what is w? This is not explained in the main paper nor the appendix.\n- It is unclear how the graps/hang interaction points of the gripper/hook are used with the neural implicit function. How is the implicit function used to evaluate a grasp or a hang? Please clarify this.\n- In 5.1, the ablations (global latent, vector repr, SDF repr) have not yet been introduced. Please re-arrange sections for better flow of the text.\n- Some graphics of the baseline networks in the appendix would be helpful for the reader to better understand the baselines and what is being ablated.\n- The handover experiment for hanging is complex and interesting. It would be interesting to expand on those experiments and show that other baseline methods cannot solve such a complex task.\n- typos:\n    - P2 of 2.2: \"trowing\" -> \"throwing\"\n    - Eq. 5: I believe it should be Mn(u,v)=1\n    - P2 of 5.1: \"interaction\" -> \"intersection\"\n- Incorrect Citations\n    - Barron et al. (MIP-NeRF) is ICCV, not CVPR.\n    - Trevithick & Yang (GRF) is ICCV, not CVPR.\n    - Jiang et al. (Synergies Between...) is RSS, not ICRA. \n    - I would suggest double checking all citations.\n- Missing related works:\n    - Henzler et al. \"Unsupervised Learning of 3D Object Categories from Videos in the Wild\", CVPR 2021.\n    - Reizenstein et al. \"Common Objects in 3D: Large-Scale Learning and Evaluation of Real-life 3D Category Reconstruction\", ICCV 2021.\n    - Van der Merwe et al. \"Learning Continuous 3D Reconstructions for Geometrically Aware Grasping\", ICRA 2020.", "summary_of_the_paper": "This paper proposes a method that integrates neural implicit functions (NIFs) with planning methods for robot manipulation. A neural implicit function that represents geometry with SDFs, grasp scores, and hanging scores is learned. This is then integrated with a planner based on Logic-Geometric Planning (LGP). Experiments are run to test mug hanging in a few different scenarios, including hanging a single mug and hanging a mug with handover.\n", "summary_of_the_review": "While I like the direction the paper is investigating, I do not think the work is mature enough yet to warrant publication at ICLR. More experiments, comparisons, and clearer exposition would definitely make this paper a better candidate for publication.\n\n---- UPDATE -----\nRaised score from 3 to 5.\n\n---- UPDATE -----\nRaised score from 5 to 6 in light of comparison to hand-engineered methods.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "0YwImhjqqed", "reviewer": "Reviewer_seqp", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n1. The overall idea of combining pixel-wise image features, the implicit function over the 3D space, and task guided learning scheme makes sense to me. However, the separate idea of each component may not be very novel.\n\n2. The authors present some experiments (with videos) to illustrate their method.\n\nWeakness & confusions:\n1. The authors only compared their method with some ablated versions (i.e., global features, SDF representation) and didn't compare with other methods:\n    (a) methods also propose object representation for robot manipulation. Besides the works mentioned (but not compared) in related work, SORNet[1] is also a work needed to cite and compare.\n    (b) traditional baselines: pose estimation + heuristic landmarks + motion planning + (RL) + (grasp point proposal). There are lots of strong baselines for the pick and place style tasks. What's the advantage over them, and why not compare with them?\n\n2. It's not clear how the learned features are incorporated into the LGP framework. What are the detailed examples of h_path, g_path, h_switch, and g_switch (equation (4)) for the grasp and hang experiments?\n\n3. The current representation only cares about the relationship between the object and the pre-defined key points (e.g., gripper and fixed hook). It doesn't capture the relationship between the objects (e.g., whether object A is on top of object B, whether object A will collide with object B).\n\n4. It's not clear what will happen when the query points are out of images. What features shall we get?\n\n5. It's not clear about the runtime of the LGP framework. Is it efficient to optimize or time-consuming?\n\n6. The definition of \"learned features\" is not clear. Grasp feature and hang feature are pose distances between the current pose and the sampled success pose? Distance in SE3 or joint space? What about the collision feature?\n\n7. The definition of the \"feasible rate\" (in table 1) is not clear.\n\n8. For the \"zero-shot imitation\" experiment, where are the grid points placed? \"centers of the target and the actual bounding balls\" is not clear. Only the ball center is used? \n\n9. For POSEICP, what's the result of only using Euclidean distance without feature space.\n\n\n\nMinor comments:\n\n1. Please adjust the presentation near equation (1) to make it look neat.\n\n2. Page 5: \"Similarly to the aforementioned image data augmentation\", but data augmentation is not mentioned yet.\n\n3. The location of the cameras is not clear. Where are they placed?\n\n4. In table 1, why is PIFO (2 views) worse than PIFO?\n\n\n[1] Yuan, Wentao, et al. \"SORNet: Spatial Object-Centric Representations for Sequential Manipulation.\"  ", "summary_of_the_paper": "The method proposes an implicit-field-function-based representation, which can be directly inferred from the camera images and be used for robot manipulation. The proposed method infers the object representation by querying about the implicit features of some pre-defined key points. The architecture projects the 3D query point into the 2D images and then exploits the pixel-wise features. ", "summary_of_the_review": "The authors propose a new method for capturing object representation directly from images and used for object manipulation. The idea of each component is not very novel. More importantly, the experiments miss baseline algorithms. I am not very convinced of the practical contribution or benefit of the proposed method. Can it solve more problems or outperform the existing method? ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper initially received negative reviews; the authors did a good job during the response period: two reviewers have updated their scores to 6. The AC has carefully read the reviews, responses, and discussions, and agreed that the authors have also mostly addressed the concerns of reviewer gsUt as well. It is unprofessional for reviewer gsUt to not engage in discussions after multiple requests.

The AC however also agrees with reviewer seqp that the new changes are major, and submissions are supposed to be evaluated in their initial form. Further, neither of the positive reviewers would like to champion the paper. 

The final recommendation is to reject the paper. The authors are encouraged to further improve and flesh out the paper based on the reviews for the next venue.
Decision: Reject