Patent Description:
Grasping of objects is a fundamental problem in robotics as it enables numerous applications. Robotic manipulators are already an integral part in modern workplaces where they are often used for repetitive tasks. While human-robot collaboration can even help in medical applications, it is often restricted to cases with clearly defined a priori motion patterns. When the interaction is more intricate and active manipulation is required, a priori definition becomes non-trivial such as in less structured domestic environments where service robots typically operate. Current approaches for robotic grasping lack generalization capabilities as they either concentrate on estimating the object pose (position and orientation in space), or learn grasping points which require detailed prior information of the object or a large set of annotations. Just like human hands, robotic grippers and arms have natural limits in their range of motion and have limited degrees-of-freedom, which restrict their possible grasping poses.

From the prior art, model-based grasping methods have been proposed. These methods typically comprise determining the 6D pose of objects to be grasped (i.e. the 3D orientation and the 3D translation) (<NPL>; <NPL>). Most traditional methods rely on local image features such as the SIFT method, or template matching. With the rise of consumer depth cameras, there has been a growing interest in 6D pose estimation from RGB-D data (Red Green Blue Depth). Also, with the help of the additional depth map, template matching has been used while other proposed to use handcrafted 3D descriptors, or to learn the pose task. With the advent of deep learning, 6D pose estimation received another boost in attention as consistently faster and more accurate approaches have been introduced. In essence, there are three different lines of work for estimating the 6D pose. The first is grounded on 3D descriptors (<NPL>. These descriptors can be computed for example via metric learning or by means of autoencoding. Other methods directly infer the 6D pose. While some methods regress the output pose, <NPL>") turn it into a classification problem. A few methods also solve for the pose by means of pose refinement. The last and most prominent branch of works establishes 2D-3D correspondences and optimizes for pose using a variant of the PnP/RANSAC paradigm. While the accuracy of estimating the 6D pose keeps steadily increasing, it also comes with a heavy burden in annotating data for 3D CAD model and 6D pose, which scales poorly to multiple objects, often rendering it impractical for real applications.

From the prior art, model-free methods are also known. Model-free approaches are generally very attractive compared to model-based approaches due to their ability to generalize to previously unseen geometry. In essence, model-free grasping can be divided into discriminative approaches, and generative approaches. Discriminative methods sample grasping instructions which are then scored by a neural network, whereas generative models directly output grasping configurations. Different training data modalities are used to train discriminative approaches. Dex-Net (<NPL>; <NPL>) collects a large amount of samples from a simulator, which is then used to train their proposed Grasp Quality Convolutional Neural Network (CNN). The authors further extend Dex-Net to support suction grippers and dual-arm robots. While the previous methods all require depth data, some methods (<NPL>; <NPL>) propose to only use RGB inputs to score grasp candidates. To train on real data, collecting over <NUM> real grasps over the course of two months has been proposed. This leads to training a CNN which scores the grasp success probability given the corresponding motor command. In another document (<NPL>) the authors use deep learning to train a robot policy which is capable of fast evaluation of millions of grasp candidates. Mousavian et al. ) leverage a variational auto-encoder to map from partial point cloud observations to a set of possible grasps for the object of interest. In contrast to generative approaches, discriminative models are usually computationally expensive as each of the proposals needs to be evaluated before grasping can be attempted.

Concerning generative models, Jiang et al. ) teaches to infer oriented rectangles in the image plane, representing plausible gripper positioning. Redmon et al. ) simultaneously predict object class and, similar to Jiang et al. , oriented rectangles depicting the grasping instructions. They further extend their method to return a set of multiple possible grasps as most objects can be usually grasped at several locations. Lenz et al. ) propose a cascade network, where the first network produces candidate grasps, which are subsequently scored by the second network. In some documents (<NPL>; "<NPL>. ), the authors leverage a small CNN to generate antipodal grasps. They predict the grasping quality, angle and gripper width for each individual pixel. The approach called S4G (<NPL>) uses a Single-Shot Grasp Proposal Network grounded on Point-Net++ (<NPL>) to efficiently predict amodal grasp proposals.

Generative methods are thereby usually tailored towards the robot or gripper used during training.

Another approach lies in reinforcement learning to teach grasping to a robot. For example, it has been proposed to utilize human demonstrations to learn robotic grasping (<NPL>. However, such methods require additional hardware and a manual recording of a vast amount of training grasps. Acquiring a large grasping data collection can be labor intensive and difficult to optimize for a particular environment.

The known techniques are still not satisfactory.

The present disclosure overcomes one or more deficiencies of the prior art by proposing a method for controlling a grasping robot comprising:.

Thus, the above method comprises a learning phase in which a grasping sample will be obtained, which will be used to determine how to grasp the object by the robot.

In this approach, it is not required to perform a full reconstruction of the object as was performed in the methods of the prior art in the training phase. Thus, the present method is more efficient and requires less input data.

The training output is a pair of a virtual hand model (having a position known) and the point cloud of the object (the two elements of the pair have been aligned with each other).

The virtual hand model can also be a point cloud but comprises information such as the respective location of each finger or of at least the thumb and the index (points clouds can be matched with one another). By way of example, the virtual hand model can be the MANO hand model (<NPL>). The virtual hand model may also be a skeleton representation of a hand which indicates where the finger joints are located. Determining a positioned hand model comprises determining where these finger joints (for example) are located when the hand grasps the object as shown in the sequence.

Determining a grasping control can be performed because a correspondence table can be prepared to match grasping robot controls with the virtual hand model, as it indicates how to grasp an object (i.e. where fingers should be for grasping an object). From the matching step, it is possible to deduce how the hand should be located for the real object present in the scene (having a known pose), and from this it is possible to deduce how the robot can grasp the object.

According to a particular embodiment, step c further comprises a step in which the shape of the 3D volume of the object is completed using a 3D-convolutional neural network.

The person skilled in the art will be able to train this 3D-convolutional neural network to perform the task of completing the shape of the object. By completing the shape, what is meant is that portions which are not visible in the sequence are predicted by the 3D-convolutional neural network.

Using a 3D-convolutional neural network is advantageous as it can receive a 3D volume as input.

According to a particular embodiment, the 3D-convolutional neural network is trained using a dataset of synthetic 3D volumes of the object.

According to a particular embodiment, step c comprises successively processing each RGB-D image including:.

The aligning step can be performed, for example, using the iterative Closest Point (ICP) algorithm.

According to a particular embodiment, step b is performed using a convolutional neural network trained with a dataset of synthetic RGB-D or RGB images of hands holding objects.

According to a particular embodiment, step i comprises using a encoder to obtain a descriptor of the scene and a descriptor of the grasping sample, and determining if there is a match between the two descriptors.

This will facilitate the matching step. By way of example the encoder can be PPFoldNet (<NPL>).

The use of this encoder allows working with partial reconstructions of the object from the scene.

If there are multiple training outputs, the best match is selected.

According to a particular embodiment, the steps of the learning phase are performed for a plurality of sequences of RGB-D images each showing a hand holding an object, the objects of at least two sequences being different.

The invention also proposes a device for controlling a grasping robot comprising a memory including the grasping sample obtained in the training phase of the method as defined above, and a processor configured to perform the steps of the grasping phase of the method as defined above.

The invention also proposes a system comprising the device as defined above, an RGB-D camera, and a grasping robot.

In one particular embodiment, the steps of the method are determined by computer program instructions.

Consequently, the invention is also directed to a computer program for executing the steps of a method as described above when this program is executed by a computer.

The invention is also directed to a computer-readable information medium containing instructions of a computer program as described above.

The information medium can be any entity or device capable of storing the program. For example, the medium can include storage means such as a ROM, for example a CD ROM or a microelectronic circuit ROM, or magnetic storage means, for example a diskette (floppy disk) or a hard disk.

Alternatively, the information medium can be an integrated circuit in which the program is incorporated, the circuit being adapted to execute the method in question or to be used in its execution.

Further features and advantages of the present disclosure will become apparent from the following description of certain embodiments thereof, given by way of illustration only, not limitation, with reference to the accompanying drawings in which:.

We will now describe a method for controlling a grasping robot, which leverages information from sequences of RGB-D images in which a human hand grasps an object and shows how should a hand interact with an object, to determine controls to be applied to the grasping robot.

In particular, it should be noted that the sequences of RGB-D images can be short (for example within <NUM>-<NUM> frames), which is particularly advantageous.

On <FIG>, a system <NUM> including a grasping robot <NUM> is shown. The grasping robot <NUM> is equipped with an RGB-D camera <NUM> which observes a scene in which an object OBJ is present. The grasping arm of the grasping robot can operate (i.e. grasp objects) within this scene.

The present method comprises a learning phase, designated as demonstration in <FIG>, in which sequences of RGB-D images are processed to obtain grasping samples, as will be explained hereinafter, so that in a grasping phase (observation on the figure), it is possible to control the robot by deducing the commands to the used from the grasping samples.

In the present description, a grasping command is successful when the grasping robot grasps and lifts the object to be grasped for several seconds, for example.

The decomposition of the method into the learning and the grasping phase has been observed to allow the method to scale to several different grasping robots, given an adequate mapping from human hand (or the virtual hand model) to grasping robot. This leads to teaching the grasping robot through a human demonstration, which allows reaching a higher level of automation, especially in less structured environments (i.e. in contrast to environments such as factories, where the object instance is precisely know priorly as well as location of conveyors, lighting etc; which are all well understood and controlled). For example, this can be performed in different home environments where there are other objects and where lighting is influenced by daytime.

We will now describe in greater detail the learning phase and the grasping phase.

On <FIG>, the steps of the method are schematically represented.

As input, the learning phase uses a sequence SEQ of RGB-D frames showing a (human) holding an object (the same object for a given sequence) under different orientations. While a single image could be used, better results are obtained according to the invention with sequences of a plurality of images showing different orientation of the object in different images. The hand and the object will be later on reconstructed as a rigid body (movement between the two should be minimal).

By way of example, it is possible to use a synthetic sequence from the dataset known to the person skilled in the art as ObMan (<NPL>) and real sequences acquired by an RGB-D camera. On each RGB-D image of the sequence SEQ, a segmentation (step S01) is performed to identify the pixels that belong to the hand and the pixels that belong to the object so as to form a hand segmentation mask and an object segmentation mask for each image. This step S01 can be implemented by means of a convolutional neural network such as the R-CNN (<NPL>). Training this convolutional neural network can be performed using the labels from the above mentioned dataset, and a binary cross entropy loss can be used to adapt the weights of the convolutional neural network for each class (i.e. mug, pen, etc.) and instance of object (i.e. a type of mug, etc.) so as to enhance the segmentation quality by preventing inter-class competition.

In a subsequent step S02, a 3D volume of the hand and a 3D volume of the object for sequence SEQ are formed, using the depth information of the RGB-D images and the corresponding segmentation masks (hand mask/object mask). Performing this step can comprise successively processing each RGB-D image including:.

The above steps perform a continuous tracking of the hand and the object together (initially as a single 3D volume), which stabilizes the process and makes the camera pose estimation more robust. This results from the fact that household object can have a rather simple geometry (too symmetrical), and using the hand will introduce a more complex structure to be tracked.

It should be noted that the above steps can be performed using the pipeline known to the person skilled in the art as Kinectfusion, where the TSDF volumes of both hand and object are continuously fused, and the point cloud are aligned through ICP with the extracted point cloud from the fused TSDF volume.

It has been observed that the above steps also filter out the rest of the human body (arm, etc.) to better focus on the hand and the object.

Forming the 3D volume of the object can include an optional step of completing the shape of the 3D volume of the object using a 3D-convolutional neural network.

After obtaining an initial 3D volume, the object 3D volume may not be complete due to self-occlusion and partial visibility. As a detailed and complete mesh may have a positive influence on the subsequent tasks, a shape completion step is performed. This step could be performed using different approaches. For example, it is possible to use a 3D convolutional neural network to directly correct the 3D volume of the object, for example expressed as a TSDF volume. It has been observed that despite the cubic complexity scaling of TSDF, they still provide a suitable solution where both the object and the hand have a limited spatial extent. By way of example, the 3D convolutional neural network can be a 3D version of the convolutional neural network known as UNet (<NPL>), with skip-connections for improved accuracy.

The 3D convolutional neural network can be fed with the TSDF volume of the object obtained from the depth information of the images and the segmentation masks. By way of example, the TSDF volumes can have a 64x64x64 resolution. The 3D volume can be encoded into a <NUM>-dimensional feature descriptor which is additionally concatenated with RGB features to provide complementary image information in addition to the geometry. The decoder consists of <NUM> upconvolutions with a kernel size of <NUM> × <NUM> × <NUM> followed by a max-pooling layer to eventually reach the input resolution (the encoder and the decoder form the 3D convolutional neural network).

The object mesh prediction is modeled as a classification task where each voxel in the output volume represents its prediction score, denoting if a voxel is occupied. Since only a small portion of voxels in the volume is occupied, the training data is highly imbalanced. Thus, a class average loss is applied and the focal loss (<NPL>) as objective function to re-balance the loss contributions is additionally used according to the following function: <MAT>.

Wherein Pos and Neg respectively represent the occupied and empty voxels and y = <NUM> (as proposed in document "Focal loss for dense object detection").

The 3D convolutional neural network can be trained for the task of shape completion by first rendering multiple images from different views of various hand-object interactions. The associated object TSDF are then extracted by implementing steps S01 and S02 and inputted to the 3D convolutional neural network so as to train this network, while the object meshes obtained when the synthetic data was generated is used as ground truth.

Step S03 can then be carried out in which a virtual hand model is positioned in space using the sequence of RGB-D images, and positioned similarly to the hand in the RGB-D sequence.

A convolutional neural network can be used to implement this step.

By way of example, the given virtual hand model can be the MANO hand model (<NPL>) which maps parameters for hand pose and shape to a mesh. The MANO hand model is developed from multiple scanned real hands featuring realistic deformations. As prediction under occlusion is difficult, the method of Y. HASSON et al. (<NPL>) is used to jointly train the CNN used for hand pose and object mesh estimation, using auxiliary contact and collision losses to encourage natural predictions without collision, which mutually improves both tasks. It can be observed that the method of Hasson uses a 2D CNN for extracting features from RGB images and proves that jointly estimating the hand and object mesh together improves the network performance. Then, it is possible to leverage the hand branch of the architecture of document "Learning joint reconstruction of hands and manipulated objects" for a complete mesh reconstruction. As the hand is part of a joint reconstruction, the pose is already close to the actual grasping location.

Nevertheless, to further improve grasping, the Iterative Closest Point algorithm is further used ICP to tightly align the hand mesh with the point cloud extracted (step S04) from the TSDF volume of the partial hand reconstruction (step S05 on the figure). This allows calculating the transformation of the hand mesh in robot view coordinates ready to calculate grasping instructions. It should be noted that for this step, a point cloud has been extracted from the TSDF volume as the virtual hand model is also a point cloud. In the present description, the expression "3D volume" can designate a TSDF volume.

This amounts to aligning the virtual hand model of step S03 with point cloud form of the hand obtained in step S04 so as to obtain a training output which includes the 3D volume of the object and the positioned virtual hand model.

Acquiring a plurality of different grasping samples can be performed during the learning phase.

Subsequently, the grasping phase can be performed. The first step of this phase is step S06 in which an RGB-D image of a scene in which an object of the same type of the object of the sequence SEQ is present; the grasping robot will be able to operate within the scene, for example to grasp the object.

In step S06, a point cloud of the object and its pose are determined, for example (for example using a convolutional neural network trained to identify this object and by forming a point cloud from the RGB-D image (the pose is known from the depth information)).

A matching step (S07) is then performed in which the point cloud of the object determined from the image of the scene is matched with the point cloud of the object of the training output to deduce the position of a virtual hand model grasping the object in the scene similarly to the virtual hand model of the training output.

This step may be performed by means of an encoder such as PPFFoldNet (<NPL>) which encodes the information of the local geometry in a high dimensional feature descriptor. Thereby, we calculate feature descriptors for both grasping sample and the acquired RGB-D image. We then align both point clouds through robust feature matching using RANSAC. It should be noted that if there are multiple training outputs, the best matching is selected.

The virtual hand model, known from the training output, may indicate the location of the index and of the thumb, and from the position of these fingers, it is possible to determine the grasping instructions to be applied by a grasping robot, using the pose of the object in the scene. By way of example, it is possible to prepare a correspondence table between the positions of the thumb and of the index and the grasping instructions to be used. Consequently, it is easy to adapt the method to a new grasping robot. By way of example, the method can be implemented using the grasping robot manufactured by Toyota under the name Human Support Robot with objects arranged one meter away from the robot. This robot is described in document "<NPL>).

The present method can be implemented by a processor of the grasping robot or by a processor of a device controlling the robot such as a computer.

To test the method, different objects have been tested. Figure <NUM> shows a drill in four frames of a sequence, and the corresponding grasping sample obtained from the corresponding sequence.

Also, for example, the sequences can have a length of less than one minute (approximatively <NUM>-<NUM> frames) and may observe the human hand-object interaction at a distance of about <NUM> meter.

The present method uses a demonstration, by a human, of how to grasp an object, and leverages information from a sequence of RGB-D images of this interaction to obtain a grasping sample. This sample is subsequently used to teach the grasping robot how to grasp an object.

Throughout the description, including the claims, the term "comprising a" should be understood as being synonymous with "comprising at least one" unless otherwise stated. In addition, any range set forth in the description, including the claims should be understood as including its end value(s) unless otherwise stated. Specific values for described elements should be understood to be within accepted manufacturing or industry tolerances known to one of skill in the art, and any use of the terms "substantially" and/or "approximately" and/or "generally" should be understood to mean falling within such accepted tolerances.

Claim 1:
A method for controlling a grasping robot comprising:
a learning phase including:
a. obtaining a sequence (SEQ) of RGB-D images showing a human hand holding an object under different orientations,
b. performing (S01), on each RGB-D image of the sequence, a segmentation to identify the pixels that belong to the hand and the pixels that belong to the object so as to form a hand segmentation mask and an object segmentation mask for each image,
c. forming (S02), using the depth information of the RGB-D images and the corresponding segmentation masks, a 3D volume of the hand and a 3D volume of the object,
d. extracting (S04) a point cloud of the hand from the 3D volume of the hand and a point cloud of the object from the 3D volume of the object,
e. positioning (S03) a virtual hand model in space using the sequence, positioned similarly to the hand in this sequence,
f. aligning the virtual hand model obtained in step d with the point cloud of the hand obtained in step c so as to obtain a training output comprising the point cloud of the object and the positioned virtual hand model; and
characterized in that the method comprises a grasping phase including:
g. acquiring an RGB-D image of a scene in which an object of the same class as the object of the sequence of RGB-D images is present, the grasping robot being able to operate within the scene,
h. determining a point cloud of the object and its pose from the image acquired in step g
i. matching the point cloud of the object determined from the image acquired in step g with the point cloud of the object of the training output to deduce the position of a virtual hand model grasping the object in the scene similarly to the virtual hand model of the training output,
j. determining a grasping control to be used to grasp, by the grasping robot, the object, by determining a control corresponding to the virtual hand model having a position deduced in step i.