Patent Document ID: 9687983
Application ID: 15151926
Patent Status: 1

Claim One:
1. A method implemented by one or more processors, comprising: providing an image for display on a client computing device, the image capturing at least a portion of an object in an environment and being generated based on image data generated by one or more cameras that are associated with a robot and that are viewing the environment; receiving selection data in response to providing the image, the selection data indicating one or more selected pixels of the image, the selected pixels being a subgroup of pixels of the image and being selected by a user via a user interface input device of the client computing device; selecting, based on the one or more selected pixels indicated by the selection data, at least one particular three-dimensional (3D) point from a group of 3D points, the group of 3D points being generated based on the image data generated by the cameras or determined based on additional sensor data generated by a 3D scanner that is associated with the robot and that is viewing the environment; fitting a local plane based on the particular 3D point and based on one or more additional 3D points, of the group of 3D points, that are in a neighborhood of the particular 3D point; identifying a grasp approach vector based on a normal of the local plane; determining a grasp pose that conforms to the grasp approach vector and that avoids collision with the object and with other objects in the environment; and providing one or more control commands that cause a grasping end effector of the robot to traverse to the grasp pose and attempt a grasp of the object subsequent to traversing to the grasp pose.