Learning Neural Implicit Functions as Object Representations for Robotic Manipulation

1 INTRODUCTION . Intelligent agents should be able to interact with objects in the environment , such as grasping and placing an object , or more general tool-use , to achieve a certain goal . In robotics , such instances are formalized as manipulation planning , a type of a motion planning problem that solves not only for the robot ’ s own movement but also for the objects ’ motions subject to interaction constraints . Traditional approaches represent objects using meshes or combinations of shape primitives and describe interactions as hand-crafted constraints in terms of that representation . The approach of using such traditional geometric representations has long-standing limitations in terms of their perception and generalizing to large varieties of objects and interaction modes : ( i ) The representations have to be inferred from raw sensory inputs like images or point clouds – raising the fundamental problem of perception and shape estimation . However , if the aim is manipulation skills , the hard problem of precise shape estimation might be unnecessary to predict accurate interaction features1 , and an end-to-end object representation might be more appropriate than a standard perception pipeline . ( ii ) With increasing generality of object shapes and interaction , the complexity of representations grows and hand-engineering of the interaction features becomes inefficient . What is a good representation of an object ? Considering the representation will be used to predict interaction features , we expect it to encode primarily task-specific information rather than only geometric . And some of the information should to be shared across different interaction modes . In other words , good representations should be task-specific so that the feature prediction can be simplified and , at the same time , be task-agnostic to enable synergies between the features . E.g. , mug handles are called handles because we can handle the mug through them and also , once we learn the notion of a handle , we can interact with the mug through them in many different ways . From the perception aspect , good representations should be easy to infer from raw sensory inputs and should be able to trade their accuracy off in favor of the feature prediction . To this end , we propose a novel data-driven approach to learning interaction features . The proposed feature prediction scheme is illustrated in Fig . 1 . The whole pipeline is trained end-to-end directly with the task supervisions so as to make the representation and perception task-specific and thus to simplify the interaction prediction . The object representation acts as a bottleneck and is shared across multiple feature predictions so that the task-agnostic representations can emerge . Particularly , the object representation is a neural implicit function over the 3D space ( Park et al. , 2019 ; Mildenhall et al. , 2020 ) upon which equality constraint features are trained . The proposed neural implicit 1We call an interaction constraint function an interaction feature ; when used as equality constraints , the interaction features , analogous to energy potentials , return zero when feasible and non-zero otherwise . function is pixel-aligned : The function takes images from multiple cameras as input ( e.g . stereo ) and , assuming known camera poses and intrinsics , the latent representation at a certain spatial location is directly related to pixels of the images . Once learned , the interaction features can be used by a typical constrained optimal control framework to plan dexterous object-robot interaction . We adopt Logic-Geometric Programming ( LGP ) ( Toussaint et al. , 2018 ) as an optimization-based manipulation planning framework and show that this learned-feature based planning enables to compute trajectories that involve various types of interaction modes only from images . Due to the representations ’ generalization , the learned features are directly applicable to manipulation tasks involving unseen objects . To summarize , our main contributions are • To represent objects as neural implicit functions upon which interaction features are trained , • An image-based manipulation planning framework with the learned features as constraints , • Comparison to non pixel-aligned , non implicit function , and geometric representations , • Demonstration in various manipulation scenarios ranging from simple pick-and- hang [ videos ] to longer-horizon manipulations [ videos ] and zero-shot imitations [ videos ] . 2 RELATED WORK . 2.1 NEURAL IMPLICIT REPRESENTATIONS IN 3D MODELING AND VIEW SYNTHESIS . Neural implicit representations have recently gained increasing attention in 3D modeling . The core idea is to encode an object or a scene in the weights of a neural network , where the network acts as a direct mapping from 3D spatial location to an implicit representation of the model , such as occupancy measures ( Mescheder et al. , 2019 ; Songyou Peng , 2020 ) or signed distance fields ( Park et al. , 2019 ; Gropp et al. , 2020 ; Atzmon & Lipman , 2020 ) . In contrast to explicit representations like voxels , meshes or point clouds , the implicit representations don ’ t require discretization of the 3D space nor fixed shape topology but rather continuously represent the 3D geometry , thereby allowing for capturing complex shape geometry at high resolutions in a memory efficient way . There have been attempts to associate these 3D representations with 2D images using the principle of camera geometry . Exploiting the camera geometry in a forward direction , i.e. , 2D projection of 3D representations , yields a differentiable image rendering procedure and this idea can be used to get rid of 3D supervisions . For example , Sitzmann et al . ( 2019 ) ; Niemeyer et al . ( 2020 ) ; Yariv et al . ( 2020 ) ; Mildenhall et al . ( 2020 ) ; Henzler et al . ( 2021 ) ; Reizenstein et al . ( 2021 ) showed that the representation networks can be trained without the 3D supervision by defining a loss function to be difference between the rendered images and the ground-truth . Another notable application of this idea is view synthesis . Based on the differentiable rendering , Park et al . ( 2020 ) ; Chen et al . ( 2020 ) ; Yen-Chen et al . ( 2021 ) addressed unseen object pose estimation problems , where the goal is to find object ’ s pose relative to the camera that produces a rendered image closest to the ground truth . By conditioning 3D representations on 2D input images , one can expect the amortized encoder network to directly generalize to novel 3D geometries without requiring any test-time optimization . This can be done by introducing a bottleneck of a finite-dimensional global latent vector between the images and representations , but these global features often fail to capture fine-grained details of the 3D models ( Songyou Peng , 2020 ) . To address this , the camera geometry can be exploited in the inverse direction to obtain pixel-aligned local representations , i.e. , 3D reprojection of 2D image features . Saito et al . ( 2019 ) and Xu et al . ( 2019 ) showed that the pixel-aligned methods can establish rich latent features because they can easily preserve high-frequency components in the input images . Also , Yu et al . ( 2021 ) and Trevithick & Yang ( 2021 ) incorporated this idea within the view-synthesis framework and showed that their convolutional encoders have strong generalizations . While the above work investigates neural implicit functions to model shapes or appearances , we train them to model physical interaction feasibility and thereby to provide a differentiable constraint model for robot manipulation planning . 2.2 OBJECT/SCENE REPRESENTATIONS FOR ROBOTIC MANIPULATIONS . Several works have proposed data-driven approaches to learning object representations and/or interaction features which are conditioned on raw sensory inputs , especially for grasping of diverse objects . One popular approach is to train discriminative models for grasp assessments . For example , ten Pas et al . ( 2017 ) ; Mahler et al . ( 2017 ) ; Van der Merwe et al . ( 2020 ) trained a neural network that , for given candidate grasp poses , predicts their grasp qualities from point clouds . In addition , Breyer et al . ( 2020 ) ; Jiang et al . ( 2021 ) proposed 3D convolutional networks that take as inputs a truncated SDF and candidate grasp poses and return the grasp affordances . Similarly , Zeng et al . ( 2020b ; a ) addressed more general manipulation scenarios such as throwing or pick-and-place , where a convolutional network outputs a task score image . On the other hand , neural networks also have been used as generative models . For example , Mousavian et al . ( 2019 ) and Murali et al . ( 2020 ) adopted the approach of conditional variational autoencoders to model the feasible grasp pose distribution conditioned on the point cloud . Sundermeyer et al . ( 2021 ) proposed a somewhat hybrid method , where the network densely generates grasp candidates by assigning grasp scores and orientations to the point cloud . You et al . ( 2021 ) addressed the object hanging tasks from point clouds where the framework first makes dense predictions of the candidate poses among which one is picked and refined . Compared to these works , our framework takes advantage of a trajectory optimization to jointly optimize an interaction pose sequence instead of relying on exhaustive search or heuristic sampling schemes , thus not suffering from the high dimensionality nor the combinatorial complexity of long-horizon planning problems . Another important line of research is learning and utilizing keypoint object representations . Manuelli et al . ( 2019 ) ; Gao & Tedrake ( 2021 ) ; Qin et al . ( 2020 ) ; Turpin et al . ( 2021 ) represented objects using a set of 3D semantic keypoints and formulated manipulation problems in terms of such the keypoints . Similarly , Manuelli et al . ( 2020 ) learned the object dynamics as a function of keypoints upon which a model predictive controller is implemented . Despite their strong generalizations to unseen objects , the keypoint representations require semantics of the keypoints to be predefined . The representation part of our framework is closely related to dense object descriptions proposed by Florence et al . ( 2018 ; 2019 ) . The idea is to train fully-convolutional neural networks that maps a raw input image to pixelwise object representations which directly generalize to unseen objects . Our proposed framework can be seen as an extension of this pixelwise representation to dense representations over the 3D space which is learned by the task supervisions and can be seamlessly integrated into general sequential manipulation planning problems . Another recent related work was proposed by Yuan et al . ( 2021 ) , where the learned object-centric representations are used to predict the symbolic predicates of the scene which in turn enables symbolic-level task planning . In contrast , our framework predicts the task feasibility given a robot configuration and enables trajectory optimization of the lower-level continuous motions . 3 INTERACTION FEATURE PREDICTION VIA IMPLICIT REPRESENTATION . Given Nview images with their camera poses/intrinsics , { ( I1 , T 1 , K1 ) , ... , ( INview , TNview , KNview ) } , we define an interaction feature as a neural implicit function : h = φtask ( q ; { ( I1 , T 1 , K1 ) , ... , ( INview , TNview , KNview ) } ) , ( 1 ) where q ∈ SE ( 3 ) is the pose of the robot frame interacting with the object . As shown in Fig . 1 , the feature prediction framework consists of two parts : the representation network , which we call a backbone , and the feature head networks . The backbone serves as an implicit functional representation of an object , which , conditioned on a set of posed images , outputs d-dimensional representation vectors at queried 3D spatial locations . The interaction feature predictions are made through the feature heads , where each head is fed on a set of representation vectors obtained by querying the backbone at a set of key interaction points . While the multiple feature heads separately model different interactions , the backbone is shared across the tasks , making it learn more general object representations . The rest of this section will be devoted to introduce each module in detail .

Summary: The method proposes an implicit-field-function-based representation, which can be directly inferred from the camera images and be used for robot manipulation. The proposed method infers the object representation by querying about the implicit features of some pre-defined key points. The architecture projects the 3D query point into the 2D images and then exploits the pixel-wise features.