TASK-ORIENTED GRASPING OF OBJECTS

A computer-implemented method includes obtaining a collection of object models for a plurality of different types of objects belonging to a same object category, generating a canonical representation for objects belonging to the object category, performing a plurality of downstream tasks using a plurality of different robot grasps on instances of objects belonging to the category and evaluating each grasp according to success or failure of the downstream task; and generating one or more category-level grasping areas for the canonical representation for objects belonging to the object category including aggregating the evaluations of grasps according to the downstream task.

BACKGROUND

This specification relates to robotics, and more particularly to planning robotic movements.

Robotic manipulation refers to controlling the physical movements of robots in order to perform tasks. Robotic manipulation tasks require a robotic component, e.g., an end effector, to physically contact an object to effectuate some change in the object. For example, an industrial robot that builds cars can be programmed to first pick up a car part and then weld the car part onto the frame of the car.

Robotic manipulation often requires a suitable grasp equipped with semantic meaning aligned with the downstream task. An important application domain is industrial assembly, where the robot needs to perform constrained placement after grasping the objects. In such cases, a suitable grasp requires stability during object grasping and transporting, meanwhile avoiding obstructing the placement process. For instance, a grasp where the gripper fingers covering the thread portion of a screw can impede its placement through a hole, is not a task-oriented semantic grasp.

However, solving task-oriented semantic grasping is a challenging problem, as the grasp and task performances are co-defining and conditional on each other. In addition, task-oriented grasping involves high-level semantic information which is difficult to model analytically. Current techniques using deep learning are limited in at least two ways.

First, annotations on 3D models are significantly more challenging than 2D image alternatives, as the annotator needs to operate over multiple views of the 3D model to complete the labeling process. Second, because training supervision solely comes from pixel-wise segmentation or pre-defined keypoints, there lacks explicit capture of the cross-object instance information. Therefore, generalizing to novel objects remains a challenge, specifically with capturing the category-level priors related to task-oriented grasping through end-to-end training. While the environment constraints on grasps are significantly more complex, the solution space is much larger and introduces much more computational complexity.

SUMMARY

This specification describes techniques for learning category-level, task-relevant grasping for robotic manipulation tasks.

After training, the model can be directly applied to novel object instances with previously unseen dimensions and shape variations, saving the effort of acquiring 3D models or re-training for each individual instance. The system can generate a canonical representation shared across diverse instances within the object category. Once trained, this category-level, task-relevant grasping knowledge transfers across novel instances, and also effectively generalizes to real-world densely cluttered scenarios without the need for fine-tuning. In addition, the training process can be performed entirely in simulation.

This system can include a framework for learning category-level, task-relevant, grasping of objects and targeted placement; at least one virtual simulation for developing canonical object representations for multiple object categories; at least one 3D shape modeled with dense, point-wise task relevance; and at least one coordinate space for learning category-level object 6D poses and 3D scaling.

The framework can generalize to the real-world without any re-training by leveraging domain randomization, bi-directional alignment, and domain-invariant, hand-object contact heatmaps, representing suitable grasping areas, modeled in a category-level canonical space. This dense 3D representation eliminates the requirement of manually specifying keypoints, which is time intensive and prone to human error. This virtual simulation can be a Non-Uniform Normalized Object Coordinate Space (NUNOCS) representation for learning category-level object 6D poses and 3D scaling, which allows for non-uniform scaling across three dimensions. This virtual representation establishes reliable dense correspondence and enables fine-grained knowledge transfer across object instances with large shape variations. Given one or more instance models, all points in the canonical virtual simulation are normalized along each dimension to reside within a unit cube.

In addition to being used for synthetic training data generation, the instance models can also be used to create a category-level canonical template model, a grasping area heatmap, and a stable grasp codebook. To do so, each model can be converted to a given space, and the canonical template model is represented by the minimum sum of chamfer distances to all other models. The transformation from each model to this template can then be utilized for aggregating the stable grasp codebook and the task-relevant grasping area heatmap. The learning task can be formulated as a classification problem by discretizing space density into bins. Along with the predicted dense correspondence, the 9D object pose can be also recovered to provide an affine transformation from the predicted canonical space cloud to the observed object segment cloud, while ensuring the rotation component to be orthonormal.

During offline training, grasp poses can be uniformly sampled from the point cloud of each object instance, covering the feasible grasp space around the object. For each grasp, the grasp quality can be evaluated in simulation. To compute a continuous score as training labels, multiple neighboring grasp poses, for example 50, are randomly sampled in the proximity and executed to compute the empirical grasp success rate. Once the grasps are generated, they can then be exploited in two ways:

First, given the relative 9D transformation from the current instance to the canonical model, the grasp poses can be converted into the virtual simulation space and stored in a stable grasp codebook. During testing, given the estimated 9D object pose of the observed object's segment relative to the canonical space, grasp proposals can be generated by applying the same transformation to the grasps in the set. Compared with traditional online grasp sampling over the raw point cloud, this grasp knowledge transfer is also able to generate grasps from occluded object regions. The two strategies can be combined to form a robust hybrid mode for grasp proposal generation.

Second, the generated grasps can be utilized for training the grasping network, which can be a Grasping Q Net, which is built based on external software. In each dense clutter generated, the object segment in the 3D point cloud is transformed to the grasp's local frame given the object and grasp pose. The grasping network can take the point cloud as input and predict the grasp's quality, which can then be compared against the discretized grasp score, e.g., to compute cross entropy loss.

This system is designed to discover grasp affordance via self-interaction. In particular, the objective is to compute a probabilistic function that describes the relative merit of different grasping positions not only for a successful grasp but also the subsequent completion of a task. This probabilistic function can take the form of the equation below:

Where P(G) is the probability of a successful grasp and P(T,G) is the probability of a successful grasp followed by successful task completion.

To achieve these results, a dense 3D point-wise grasping area heatmap can be modeled. For each grasp in the codebook, a grasping process can first be simulated. The hand-object contact points can be identified by computing their signed distance with respect to the gripper mesh. If the grasp is stable, for example the object is lifted successfully against gravity, the count n(G) for all contacted points on the object can be increased by 1. Otherwise, the grasp can be skipped. For such stable grasps, a placement process can be simulated, for example placing the grasped object on a receptacle, to verify the task relevance. Collisions can be checked between the manipulator and the receptacle during this process. If the manipulator does not obstruct the placement, and if the object can steadily rest in the receptacle, the count of joint grasp and task success n(G,T) on the contact points can be increased by 1. After all grasps are verified, for each point on the object point cloud, its task relevance can be computed according to:

Eventually, for each of the training objects within the category, the hand-object contact heatmap P(T|G) is transformed to the canonical model. The task-relevant heatmaps over all training instances are aggregated and averaged to be the final canonical model's task-relevance heatmap. During testing, due to the partial view of the object's segment, the antipodal contact points are identified between the system and the transformed canonical model. For each grasp candidate, the score PG(T|G) is computed and combined with the predicted PG(G) from the grasping network to compute the grasp's task-relevance score PG(T,G). The highest task-relevance score indicates the best grasp for the object for the given task.

Particular embodiments of the subject matter described in this specification can be implemented so as to realize one or more of the following advantages. The techniques described in this specification allow a robotic control system to learn truly generalizable category-level grasping areas that apply to all objects belonging to the category.

The category-level grasping areas can be learned without human annotations from digital object models, e.g., CAD models, without requiring any real-world data collection in a workcell. Instead, massive amounts of training data can be collected by simply crawling publicly available sources, e.g., the Internet.

In addition, the grasping areas are task-specific, meaning that they are optimized for performing a particular downstream task. For example, while holding a nut by inserting an end effector in the hole of the nut might provide a secure grasp generally, a downstream task of attaching the nut to a bolt is sure to fail if the hole is obstructed. Using the techniques described in this specification will cause the system to automatically learn that all instances in the category of nuts should be grasped on the sides and not in the hole for fastener tasks that require attaching the nut to a corresponding bolt. And the system can automatically learn different grasps for different tasks. Thus, if the task is simply picking up a nut and placing it into a receptacle, the system might automatically learn that a grasp that uses the hole is best.

The category-level grasping areas generalize to many other types of objects in the same category. This means that the grasping tasks can be performed on new objects that have never been seen by the system.

In addition, grasping can be performed on a new object without any adjustments or adaptations. In other words, the downstream task can be performed on a new object in the category without adjusting the model, without collecting additional data, without collecting human annotations, and without retraining.

In addition, the techniques described below are far more robust than reinforcement based learning approaches. Furthermore, the techniques described below are particularly well-suited for contact-rich tasks in which parts of the downstream task are intended to make contact. Such contact-rich tasks are difficult to learn for reinforcement learning systems.

DETAILED DESCRIPTION

FIG.1illustrates an example system100. This system100can include major functional areas to include 3D model simulation training110, learnt grasping information120, and the workspace130which includes at least one object132which can be assigned to an object category. An object category is defined as a generic object form to which multiple objects may share similar characteristics. For example, an object category of ‘screw’ may cover all narrow cylindrical objects with a wide head structure above a more narrow thread. In this instance a screw of any size, material, length, thread size or shape, or other determined properties may fit in this object category. The same example could apply to a nut, washer, fastener, electronic connector, pipe, flange, panel, or any other object that can be said to have features that are similar to a holistic group of objects. 3D model simulation training110with the select object category is used to generate data that informs the grasping information120. Learnt grasping information120can include information from multiple sources, to include multiple canonical object category representations122, multiple grasping area heatmaps124, and a 6D grasp codebook126. A canonical object category representation122is a spatial representation of the object category to which new objects132are compared for the purpose of grasping. The data from these various methods is combined and ranked, with the ideal grasp of object132passed to the workspace130.

FIG.2illustrates an example framework200. The framework200can include different functional subsystems that include category level prior learning210, instance segmentation220, knowledge transfer230, and grasp candidates evaluation240. These subsystems can be implemented by a system of one or more computers in one or more locations.

The category level prior learning210functional subsystem can include different subprocesses to include robust grasp identification212, task-relevance self-discovery214, multiple CAD models215, multiple grasp codebooks216, task-relevant contact experience217, and multiple canonical models218. Given a collection of CAD models215for objects of the same category, the data is aggregated to generate a canonical model218for the category. This CAD data215is also used to inform robust grasp identification212and task-relevance self-discovery214. The multiple CAD models215can be supplied by the user or made available to the system from public information, to include the Internet. The CAD models215are further utilized in virtual simulation232to generate synthetic point cloud data for training in multiple point cloud222and 3D networks224. The virtual simulation232can be a Non-Uniform Normalized Object Coordinate Space (NUNOCS). The category-level grasp codebooks216, created from robust grasp identification212, and task-relevant contact experience217, created from task-relevance self-discovery214, are identified via self-interaction in virtual simulation232.

The instance segmentation220functional subsystem can include different subprocesses to include one or more point clouds222, one or more 3D networks224, center voting226, clustering227, and an object candidate queue228. The point cloud222is a spatial representation of the grasping area together with the objects, where many possible discrete grasping locations are assigned without regard to individual objects. The 3D network224is leveraged to predict point-wise centers of discrete objects in the point cloud222through center voting226. Clustering227is then used to separate this gross collection of points into collections that correspond to different objects. The object candidate queue228is then used to inform the virtual simulation232and sampled grasps248aof the original object environment.

The knowledge transfer230functional subsystem can include different subprocesses to include a virtual simulation232, a visual representation of said virtual simulation234, a 9D pose estimation transformation of said virtual simulation233, a dense correspondence method of transformation237, and transferred contact experience238. The virtual simulation232operates over an object's segmented point cloud222and predicts its representation234of the object to establish dense correspondence237with the canonical models218and compute its 9D pose236, which represents the degree of departure from the canonical model218. The associated precomputed category-level contact experience238for the object category and canonical model218is then transferred to the task-relevance score equation246.

The grasp candidates evaluation240functional subsystem can include different subprocesses to include sample grasps248aand transferred grasps248b, a grasping network247, a task-relevance score equation246, sorting of task-relevance scores244, and the determined best grasp242. Grasp proposals are generated by both directly sampled grasps248achosen through center voting226over the 3D network224, and transferred grasps248bfrom a grasp codebook216. Infeasible or in-collision grasps are rejected by a grasping network247. The grasping network247evaluates the stability of the accepted grasp proposals248aand248b. This information is combined with a task-relevance score computed from the grasp's contact region through a probabilistic task-relevant score equation246. The entire process can be repeated for multiple object segments. The task-relevant score equation246is given by the below:

Where P(G) is the probability of a successful grasp and P(T,G) is the probability of a successful grasp followed by successful task completion. The results of the task-relevance score equation246are sorted244and the best grasp242is determined and passed to the system100.

FIG.3illustrates task-specific heatmaps for example industrial objects. The industrial objects could be of any suitable size, shape, or material. These industrial objects can include nuts (FIG.3A), electronic connectors (FIG.3B), or screws (FIG.3C).

FIG.4is a flowchart of an example process for learning grasping areas for a canonical representation of objects belonging to a same object category. The example process can be performed by a system of one or more computers programmed in accordance with this specification, e.g., a distributed computer system that implements the framework illustrated and described with respect toFIG.2. The example process will be described as being performed by a system of one or more computers.

The system obtains a collection of object models for an object category (410). To begin the process of modeling grasping positions for object categories, CAD models of the objects are obtained. These models can either be provided by the user or loaded from publically available data, e.g., the Internet. There is no requirement for format, provided that there is enough fidelity to assign discrete points to the object.

For example, it is assumed that a collection of 3D models MC belonging to category C for training have been uploaded. This does not include any testing instance in the same category, i.e., MCtest∉MC. Offline, given a collection of models MC of the same category, synthetic data can be generated in simulation. Then, self-interaction in simulation provides hand-object contact experience, which is summarized in task-relevant grasping area heatmaps for grasping.

The system generates a canonical representation for the object category (420). In other words, the system can develop a canonical object representation that can be extended to include the object described in the provided CAD models. For example, the canonical NUNOCS representation allows the aggregation of category-level, task-relevant knowledge across instances. Online, the category-level knowledge is transferred from the canonical NUNOCS model to the segmented target object via dense correspondence and 9D pose estimation, guiding the grasp candidate generation and selection. Dense correspondence is established in 3D space to transfer knowledge from a trained model database MC to a novel instance MCtest. For example, given an instance model M, all the points can be normalized along each dimension, to reside within a unit cube:

The transformed points exist in the canonical NUNOCS C. In addition to being used for synthetic training data generation the models MC are also used to create a category-level canonical template model, to generate a grasping area heatmap and a stable grasp codebook. To do so, each model in MC is converted to the space C, and the canonical template model is represented by the one with the minimum sum of Chamfer distances to all other models in MC. The transformation from each model to this template is then utilized for aggregating the stable grasp codebook and the task-relevant grasping area heatmap.

For example, in the NUNOCS Net, the relation Φ: Po→PC is determined, where Po and PC are the observed object cloud and the canonical space cloud, respectively. Φ(.) is built with a PointNet-like architecture given it is light-weight and efficient. The learning task is formulated as a classification problem by discretizing pdC into a certain number of bins, for example 100. Softmax cross entropy loss is used as we found it more effective than regression by reducing the solution space. Along with the predicted dense correspondence the 9D object pose is also recovered, given below:

The 9D object pose is computed, for example, via RANSAC to provide an affine transformation from the predicted canonical space cloud PC to the observed object segment cloud Po, while ensuring the rotation component to be orthonormal.

The system performs downstream tasks using different robot grasps (430). The system can perform grasping trials using predetermined grasping locations. For example, during offline training, grasp poses can be uniformly sampled from a point cloud of each object instance, covering the feasible grasp space around the object. For each grasp G, the grasp quality can be evaluated in simulation. For example, to compute a continuous score:

50 neighboring grasp poses can be randomly sampled in the proximity of:

and executed to compute the empirical grasp success rate.

The system evaluates each grasp according to the performance of the downstream task (440). The probability of a successful grasp and of successful task completion can be captured for each position440. Once the grasps are generated, they are then exploited in two ways.

First, for example, given the relative 9D transformation from the current instance to the canonical model, the grasp poses are converted into the NUNOCS space and stored in a stable grasp codebook G. During test time, given the estimated 9D object pose of the observed object's segment relative to the canonical space C, grasp proposals can be generated by applying the same transformation to the grasps in G. Compared with traditional online grasp sampling over the raw point cloud, this grasp knowledge transfer is also able to generate grasps from occluded object regions. In practice, the two strategies can be combined to form a robust hybrid mode for grasp proposal generation.

Second, for example, the generated grasps are utilized for training the Grasping Q Net, which is built based on PointNet. Specifically, in each dense clutter generated, the object segment in the 3D point cloud is transformed to the grasp's local frame given the object and grasp pose. The Grasping Q Net takes the point cloud as input and predicts the grasp's quality P(G), which is then compared against the discretized grasp score sG to compute softmax cross entropy loss.

The total probability of successful task completion following a successful grasp is then calculated for each position and the results and ranked by probability450. For example, the objective is to compute P(T|G)=P(T,G)/P(G) automatically for all graspable regions on the object. To achieve this, a dense 3D point-wise grasping area heatmap is modeled. For each grasp in the codebook, a grasping process is first simulated. The hand-object contact points are identified by computing their signed distance with respect to the gripper mesh. If it is a stable grasp, for example the object is lifted successfully against gravity, the count n(G) for all contacted points on the object are increased by a fixed interval, for example, one. In this specification, a grasp being stable, or equivalently, grasping areas being stable, means that an object was successfully lifted using one or more grasping areas. Otherwise, the grasp is skipped. For these stable grasps, a placement process is simulated, for example placing the grasped object on a receptacle, to verify the task relevance. Collision is checked between the gripper and the receptacle during this process. If the gripper does not obstruct the placement and if the object can steadily rest in the receptacle, the count of joint grasp and task success n(G,T) on the contact points is increased by a fixed interval, for example, one. After all grasps are verified, for each point on the object point cloud, its task relevance can be computed as P(T|G)=n(G,T)/n(G).

To perform instance segmentation during training, the system can use the Sparse 3D U-Net due to its memory efficiency. The network takes as input the entire scene point cloud voxelized into sparse volumes and predicts per point offset with respect to predicted object centers. The training loss can be designed as the L2loss between the predicted and the ground-truth offsets. The network is trained independently, since joint end-to-end training with the following networks has been observed to cause instability during training. For example, during testing, the predicted offset is applied to the original points, leading the shifted point cloud to condensed point groups P+Poffset. Next, DBSCAN is employed to cluster the shifted points into instance segments. Additionally, the segmented point cloud is backprojected onto the depth image IDto form 2D segments. This approach provides an approximation of the per-object visibility by counting the number of pixels in each segment. Guided by this, the remaining modules of the framework prioritize the top layer of objects given their highest visibility in the pile during grasp candidate generation.

The system generates one or more category-level grasping areas for the canonical representation (450). In other words, the system aggregates the results of all the grasping trials to determine which areas on the canonical representation should be used for grasping for downstream tasks. To do so, the system can transform each of the grasping area heatmaps P(T|G) to the canonical model. The task-relevant grasping area heatmaps over all training instances can then be aggregated and averaged to be the final canonical model's task relevance grasping area heatmap. During testing, due to the partial view of the object's segment, the antipodal contact points pc are identified between the gripper mesh and the transformed canonical model. For each grasp candidate, the score is computed according to:

This score can then be combined with the predicted PG(G) from Grasping Q Net to compute the grasp's task-relevance score:

The highest success probability grasp can then be selected and used as the category-level grasping area.

After generating the task-specific, category-level grasping areas, the system can apply them to a newly seen instance of an object to perform the task. For example, the system can determine a correspondence between the new object and the canonical representation to generate task-specific, instance-specific grasping areas on the object. As mentioned previously, if the object is a nut and the downstream task is connector fastening, the task-specific, instance-specific grasping areas might be the sides of the nut. On the other hand, if the task is pick and place, the grasping area might be inside the hole of the nut.

The system can then use these task-specific, instance-specific grasping areas to cause a physical robot to perform the task by manipulating the physical object. In other words, the system can cause a manipulator or an end effector of the robot to make contact with the object at one or more of the specified task-specific, instance-specific grasping areas. And, as described above, the physical robot can automatically perform the downstream task without requiring any adaptation training, even when the new object was never observed during the training process.

Additional details of learning task-specific, category level grasping areas are described in Bowen Wen et al.,CaTGrasp: Learning Category-Level Task-Relevant Grasping in Clutter from Simulation, published in the proceedings of the IEEE International Conference on Robotics and Automation (ICRA) 2022, which is herein incorporated by reference. Additional techniques for generating suitable category-level representations are described in commonly owned U.S. Patent No. 63/304,533, entitled “Category-Level Manipulation from Visual Demonstration,” filed on Jan. 28, 2022, which is herein incorporated by reference.