Patent Description:
Robots are in widespread use throughout industry, performing an ever-increasing variety of tasks. Although many robots are task-specific, having been designed to execute a particular set of actions in a single manufacturing environment, more versatile robots capable of many tasks have become common. Programming a general-purpose robot to perform a particular set of actions is often a tedious process. The user must somehow communicate to the robot the intent of the task (e.g., pick up this object at this location and place it at a goal location) and the task constraints (e.g., don't hit anything in the workspace). Such task-specific programming can be painstaking and applies only to a specific configuration of the robot and workspace. If the workspace changes, the robot typically must be reprogrammed. <CIT> discloses a system that enables a user to interact with a virtual control panel using a user controlled pointing object. <CIT> discloses systems and methods to create an immersive virtual environment using a virtual reality system that receives parameters corresponding to a real-world robot. <CIT> discloses vision-guided robots and methods of training them. <CIT> discloses a method and a system for use in connection with off-line programming of an industrial robot. <CIT> discloses a placement determination system. <NPL>) discloses an experimental setup comprising two phases. A user pose detection phase has the purpose of finding a human pose in an Operating Space. An adaptive projection phase consists of projecting a user-interface on a suitable surface.

The invention relates to a method of training a robot according to claim <NUM>, and, a robot controller according to claim <NUM>.

In order to cut time and skill needed to train a robot, embodiments of the present invention permit the user touch or click on a display projected in the actual workspace in order to define task goals and constraints. A planning procedure fills in the gaps and makes this information something that the robot can execute. This "teach-by-touch" approach frees the user from having to program the robot either offline or by actually handling the robot itself, which requires the robot to provide a training mode that allows for such handling.

As used herein, the terms "approximately" and "substantially" mean ±<NUM>%, and in some embodiments, ±<NUM>%. Reference throughout this specification to "one example," "an example," "one embodiment," or "an embodiment" means that a particular feature, structure, or characteristic described in connection with the example is included in at least one example of the present technology. Thus, the occurrences of the phrases "in one example," "in an example," "one embodiment," or "an embodiment" in various places throughout this specification are not necessarily all referring to the same example. Furthermore, the particular features, structures, routines, steps, or characteristics may be combined in any suitable manner in one or more examples of the technology. The headings provided herein are for convenience only and are not intended to limit or interpret the scope or meaning of the claimed technology.

Also, the drawings are not necessarily to scale, with an emphasis instead generally being placed upon illustrating the principles of the invention. In the following description, various embodiments of the present invention are described with reference to the following drawings, in which:.

Refer first to <FIG>, which illustrates a representative system <NUM> in accordance with embodiments of the present invention. In the illustrated embodiment, a robot <NUM> to be programmed includes a control block <NUM>, which for convenience is illustrated separately from the robot. The control block <NUM> includes a perception module <NUM>, a planning module <NUM>, and an interaction module <NUM>. In typical implementations, these modules are implemented within the robot <NUM>, e.g., as part of its internal controller. This is not necessary, however, and in fact the modules <NUM>, <NUM>, <NUM> can be realized in a separate device. For example, the modules may be implemented on a server in wireless and/or wired contact with numerous robots <NUM> and separately servicing each of the robots.

A conventional camera <NUM>, preferably an RGB-D camera that combines red-green-blue color information with per-pixel depth information (that is, the camera <NUM> assigns to each recorded pixel a color and a depth coordinate relative to the camera), is located within or adjacent to the workspace (or, in some embodiments, is part of the robot <NUM>). Visible to the camera <NUM> - i.e., within its field of view <NUM> - is a fiducial <NUM> on the robot and another fiducial <NUM> on a wand or other pointing device <NUM> used within the workspace, i.e., the camera <NUM> and its field of view <NUM> are sufficient to encompass both the robot fiducial <NUM> and the wand fiducial <NUM> when the robot <NUM> is trained as discussed herein. The fiducials <NUM>, <NUM> may be, for example, a 2D barcode or other camera-visible indicium, e.g., an APRILTAG fiducial.

It should be stressed that the use of fiducials is not essential; any suitable means of establishing the relative pose of the robot <NUM> relative to the camera <NUM> and the pose of the wand <NUM> relative to the camera <NUM> can be employed. Furthermore, alternatives to the wand <NUM> are also possible, e.g., 3D hand recognition can be employed to allow the user to signal with his hand, e.g., by pointing. Suitable machine-vision algorithms facilitating hand and gesture recognition are well-known in the art and include, for example, pattern matching against a template. See, e.g., <NPL>. Another approach, illustrated in <FIG>, utilizes voice recognition as an alternative modality to receive commands. A conventional speech-recognition module <NUM> converts speech detected by a microphone <NUM> into natural-language text; for example, the speech-recognition module <NUM> may utilize readily available APIs to GOOGLE or AMAZON speech-to-text applications or other voice algorithms. The speech-recognition module <NUM> matches keywords in the resulting text stream to text in symbolic fluents and parameters for those fluents; see, e.g., <NPL>. The fluents and their parameters may reside in a world representation <NUM>, which is described in greater detail below. Hence, as used herein, the term "gesture" includes physical and/or verbal manifestations of intent, including movement of a hand or wand in the former case and uttering a command in the latter case. (For convenience of presentation, the ensuing discussion will mostly assume use of the wand <NUM>.

The camera <NUM> is paired with a projector <NUM> whose function and operation are described below. The perception module <NUM> receives visual information from the camera <NUM> and continuously or periodically extracts the 3D position of the fiducials <NUM>, <NUM>. This function may be performed completely by (or under the direction of) the perception module <NUM> or may be shared with the camera <NUM>, which may perform image preprocessing or object identification. The planning module <NUM> sends commands to the robot <NUM> and receives mechanical state information (typically joint angles and velocity of the robot arm(s), gripper position, etc.) from the robot <NUM>. The interaction module <NUM> operates the projector <NUM> to facilitate interaction with the user as described below. It should be understood that the term "projection," as used herein, refers to any modality for creating a visible image, real or virtual, on the workspace and from which a user may make selections.

<FIG> illustrates the integration of the control block <NUM> within a representative control system <NUM> of the robot <NUM>. The control system <NUM> includes a central processing unit (CPU) <NUM> (e.g., a quad-core Intel processor), system memory <NUM>, and one or more non-volatile mass storage devices (such as one or more hard disks and/or optical storage units) <NUM>. The system <NUM> further includes a bidirectional system bus <NUM> over which the CPU <NUM>, memory <NUM>, and storage device <NUM> communicate with each other as well as with internal or external input/output (I/O) devices such as an LCD display <NUM> and control peripherals <NUM>, which may include buttons or other control devices on the robot <NUM>. The control system <NUM> may also include a communication transceiver <NUM> and one or more I/O ports <NUM>. The transceiver <NUM> and I/O ports <NUM> may provide a network interface. The term "network" is herein used broadly to connote wired or wireless networks of computers or telecommunications devices (such as wired or wireless telephones, tablets, etc.). For example, a computer network may be a local area network (LAN) or a wide area network (WAN). When used in a LAN networking environment, computers may be connected to the LAN through a network interface or adapter. When used in a WAN networking environment, computers typically include a modem or other communication mechanism. Modems may be internal or external, and may be connected to the system bus via the user-input interface, or other appropriate mechanism. Networked computers may be connected over the Internet, an Intranet, Extranet, Ethernet, or any other system that provides communications. Some suitable communications protocols include TCP/IP, UDP, or OSI, for example. For wireless communications, communications protocols may include IEEE <NUM>. 11x ("Wi-Fi"), Bluetooth, ZigBee, IrDa, near-field communication (NFC), or other suitable protocol. Furthermore, components of the system may communicate through a combination of wired or wireless paths, and communication may involve both computer and telecommunications networks. The I/O ports <NUM> also provide control and acutation commands to the various motors and joints of the robot <NUM>.

The CPU <NUM> is typically a microprocessor, but in various embodiments may be a microcontroller, peripheral integrated circuit element, a CSIC (customer-specific integrated circuit), an ASIC (application-specific integrated circuit), a logic circuit, a digital signal processor, a programmable logic device such as an FPGA (field-programmable gate array), PLD (programmable logic device), PLA (programmable logic array), RFID processor, smart chip, or any other device or arrangement of devices that is capable of implementing the steps of the processes of the invention.

The system memory <NUM> contains a series of frame buffers <NUM>, i.e., partitions that store, in digital form, images obtained by the camera <NUM>. System memory <NUM> contains instructions, conceptually illustrated as a group of modules, that control the operation of the CPU <NUM> and its interaction with the other hardware components. These include the control block <NUM>, conventional robot control routines <NUM> (which include suitable drivers to operate the robot <NUM>), and an operating system <NUM> (e.g., WINDOWS or LINUX) directs the execution of low-level, basic system functions such as memory allocation, file management and operation of the mass storage device <NUM>. The control block <NUM> (in particular, the perception module <NUM>) may analyze the images in the frame buffers <NUM> to identify the fiducials <NUM>, <NUM> and interpret user gestures. Any suitable programming language may be used to implement without undue experimentation the functions of the control block <NUM> as described herein. Illustratively, the programming language used may include assembly language, Ada, APL, Basic, C, C++, C*, COBOL, dBase, Forth, FORTRAN, Java, Modula-<NUM>, Pascal, Prolog, Python, REXX, and/or JavaScript for example. Further, it is not necessary that a single type of instruction or programming language be utilized in conjunction with the operation of the system and method of the invention. Rather, any number of different programming languages may be utilized as is necessary or desirable.

During an initialization phase, the position of the projector <NUM> relative to the camera <NUM> is established in the coordinate system of the 3D workspace using any suitable calibration technique, e.g., manual measurement and entry into the memory <NUM>. In particular, a suitable approach to calibration takes in images of checkerboard patterns on a posterboard and a checkerboard pattern projected by the projector. This outputs the relative pose and camera model parameters between the projector <NUM> and the camera <NUM>. Next, the position of the robot <NUM> relative to the camera <NUM> is established in the 3D workspace coordinate system. This may be accomplished by locating the fiducial <NUM> on the camera image of the robot and measuring its size; the size of the fiducial, combined with its known location on the robot, are sufficient to establish the distance of the robot from the camera and thereby calibrate the camera's depth perception in the room coordinate system.

In an alternative embodiment, which does not require fiducials, a 3D computer-aided design (CAD) model of the gripper is computationally fitted to 3D voxels of the gripper in the scene. From one or more views, the relative position of the gripper and the robot may be regressed to the camera <NUM>. In particular, provided with a robot arm and gripper with known kinematics and a CAD model of the gripper, as well as a camera <NUM> with 3D capability, registration may be accomplished using an initial guess of a camera-to-robot coordinate transformation, e.g., using a visible indicium on the robot, a manually entered location, or the last-used registration data In particular, the following algorithm may be employed:.

The ICP algorithm is described, for example, in <NPL> Camera-object registration using human-guided ICP is described in Marion et al. , "LabelFusion: A Pipeline for Generating Ground Truth Labels for Real RGBD Data of Cluttered Scenes," Computer Vision and Pattern Recognition, available at https://arxiv. org/abs/<NUM>.

The functions performed by the perception, planning, and interaction modules <NUM>, <NUM>, <NUM> are illustrated in greater detail in <FIG>. In essence, these modules cooperate to enable a user to train the robot <NUM> through workspace interaction rather than direct programming; the interface via which the user interacts with the robot <NUM> is projected into the workspace, enabling the user to train the robot while walk through the workspace and, if necessary, maintaining a safe distance from the robot <NUM>. The perception module <NUM> receives a 3D point cloud (rather than a complete depth map) of the workspace surrounding the robot from the camera <NUM>; the point cloud may consist of or comprise RGB and depth data. The perception module <NUM> computes a 3D segmentation of the cloud, e.g., using Euclidean clustering with point-normal thresholding. Object (e.g., workpiece) surfaces and work surfaces may be clustered as separate entities, and each cluster may be represented as a collection of points in 3D space. With additional reference to <FIG> and <FIG>, these representations collectively form a "world representation" <NUM> of the workspace that is stored in the main memory <NUM> and/or storage device <NUM>. The world representation may include an occupancy grid, which marks pixels or voxels of the workspace as empty or occupied, and facilitates correlation of a gesture with selection items of a projected menu. Detecting user gestures indicative of selections or other commands may involve estimating the pose of the wand <NUM>. The perception module <NUM> sends detected gestures (e.g., clicks) to the planning module <NUM>, and may notify the planning module of updates to the world representation <NUM>. (In general, the world representation <NUM> is a data structure accessible to the entire control block <NUM>.

The interaction block <NUM> serves as the interface between user and robot, allowing the user both to define the task and get feedback from the robot <NUM> as to what the robot understands about the workspace and how it will act. As noted, an important innovation is use of the actual workspace itself to help the user tell the robot what it should do. In particular, as directed by the interaction module <NUM>, the projector <NUM> may project a menu and/or graphical interface to the user within the workspace; for example, the menu may appear on the wall or span multiple surfaces in the workspace. The user makes selections from the menu using the wand <NUM>, and the planning module <NUM>, "knowing" both the location of the wand fiducial <NUM> and the projected menu options, recognizes selection of a menu option as the user points at it with the wand <NUM>. In this way, the user can "explain" a task to the robot <NUM> as s/he would to another human As detailed below, the planning module <NUM> interprets the commands and creates a workflow program that the robot can run.

In operation, with additional reference to <FIG>, the user initially moves the wand <NUM> within the workspace and indicates where the menu <NUM> is desired. The perception module <NUM> geometrically computes the minimum distance from the wand fiducial <NUM> to any point cluster Using the surface cluster closest to the fiducial <NUM>, the perception module <NUM> fits a planar model to the surface cluster points. To create a menu, the interaction module <NUM> computes a set of 3D points that defines the menu with the appropriate number of selection buttons; for example, the lower left corner of a rectangular menu may be defined as the point in the plane closest to the wand fiducial <NUM>. Menu points are defined in the workspace and used to direct projection of the menu. Using projection mapping, the interaction module <NUM> projects a 3D-defined color menu onto a 2D color image, and that image is displayed by the projector <NUM> where the user has indicated. Using conventional rendering techniques, the interaction module <NUM> may distort the image so that perspective is maintained in the projected image relative to the features of the workspace.

Based on the 3D segmentation, the projected menu buttons have known 3D workspace coordinates, and these are used to detect button "clicks" - i.e., gestures made by the user using the wand <NUM>. A click may be detected using a metric based on the 3D marker position and the 3D position of a button defined by a set of 3D points. For example, a 3D box may be defined around a rectangular button and if the fiducial <NUM> enters the box, the button is deemed "clicked " A button click may be registered in the workspace by, for example, changing the projected color of the button to indicate that it has been selected. The menu buttons allow users to initiate task-definition modes that turn user clicks into task planning goals and constraints. For example, clicking an "Object" button allows the user to select (using marker clicks in the workspace) an object in the workspace to be be manipulated by the robot. Clicking the "Obstacle" button as indicated in <FIG> allows the user to indicate an obstacle.

More generally, user clicks and other detected gestures can define a task. Clicks that define explicit task goals can include manipulation goals such as object pick goals and object place goals; end-effector placement goals such as drill, insert, screw, and snap goals; object-relative goals such as place object A relative to object B, place cap on bottle; and volume/area goals can allow the user to define a volume of space for an object or end-effector goal, or to pick/place from. User clicks can also define explicit task constraints. Such constraints can involve the workpiece(s) (e.g., objects to be manipulated and secondary objects such as jigs, guide rails, receiving components like connectors), obstacles (e.g., objects to avoid collision with), and "keep-out" zones that define points or regions in free space that the robot and its appendages should not enter. The perception module <NUM> may process the camera images to define implicit task constraints based on, e.g., perceived obstacles based on pose estimation from the point-cloud data. The perception module <NUM> may also define implicit task goals, e.g., a range of points on a perceived object that are accessible to a robot drill.

The planning module <NUM> bridges the gap between task definition and task execution, computing a world representation from explicit and implicit task definitions. In particular, the planning module <NUM> utilizes task-planning and motion-planning methodologies to create the robot workflow program from the tasks and constraints provided by the user and/or the perception module <NUM>. Obstacles and free-space may be represented by a discrete 3D occupancy grid map, and the planning block may compile poses of all objects and workpieces within the workspace. The planning module <NUM> may compute a task plan using a hierarchical task network (HTN), which takes as input task goals and constraints and computes a sequence of high-level actions until an executable action (primitive) is reached. For example, move, pick, and place actions correspond to primitives, while filling a box with objects represents a high-level goal. The task-planning methodology may be hybrid, i.e., extend over temporal, spatial, and resource reasoning, in addition to sequencing and goal achievement. Suitable algorithms are described, for example, in <NPL>) and <NPL>).

The task plan is translated into motion plans for each primitive (e.g., target joint angles achieved using a situationally appropriate joint velocity) by a conventional robot motion planner using, for example, the well-known rapidly-exploring random tree (RRT) algorithm and trajectory waypoints, e.g., taking as input an occupancy grid and a goal and computing robot trajectory commands and gripper commands to implement the goal.

The interaction module <NUM> may employ workspace projection to give the user feedback on task definition and execution. By converting 3D points/poses into a 2D image and projecting the image onto workspace surfaces, various aspects of task definition and task execution may be presented to the user. For example, the planning module <NUM> may report, via projected images, that a user goal is not feasible, or that an execution plan could not be found or will result in a collision. Alternatively or in addition, as illustrated in <FIG>, the interaction module <NUM> may use text-to-speech capability to provide verbal feedback via a text-to-speech module <NUM> and a speaker <NUM>. Once again readily available APIs to GOOGLE or AMAZON text-to-speech applications may be utilized by the text-to-speech module <NUM>.

A representative sequence of interaction steps with an embodiment of the invention is shown in <FIG> In a first step <NUM>, the user touches a surface (usually, but not necessarily, flat), and the interaction module <NUM> responsively causes projection of a selection menu on the pointed-to surface (step <NUM>). Typically, the location of the projected menu is spatially distinct from where task-related activity takes place, i.e., the initial or goal position of an object to be manipulated. The projected menu contains a selection item labeled "object" - for example, the menu may have buttons for defining "object," "goal," "obstacles," and "execute" _ and in step <NUM>, the user touches (or otherwise gestures to indicate) the "object" selection item with the wand <NUM>. The user then touches the actual work object with the wand <NUM> (step <NUM>) to identify the work object as the subject of a robot task. In step <NUM>, the user touches the "obstacles" button with the wand <NUM>, and to define the obstacle, thereupon touches four points to indicate a region between the initial and goal position (step <NUM>). The planning module <NUM> computationally "extrudes" the obstacle vertically as a 3D object from the indicated four points.

In step <NUM>, the user selects the "goal" button with the wand <NUM> and thereupon touches, again with the wand <NUM>, a location to be designated as the goal location (step <NUM>). The planning module <NUM> may define the goal location not as a point but as a small area, e.g., a circle on the touched surface with an area larger than the designed work object. When the user selects "execute" from the projected menu using the wand <NUM> (step <NUM>), the planning module <NUM> computes a trajectory (using, e.g., the RRT algorithm) that will allow the robot to bring the designated object from the initial position to the goal position while avoiding the defined obstacle, and causes the robot <NUM> to execute a grasp, a move to goal while avoiding the obstacle, and a release of the object at the designated goal location.

Claim 1:
A method of training a robot (<NUM>) situated in a workspace, the method comprising the steps of:
(a) representing the robot (<NUM>) in a 3D coordinate space encompassing at least a portion of the workspace by establishing a relative position of the robot (<NUM>) relative to a camera (<NUM>), wherein the camera (<NUM>) is located within or adjacent to the workspace or is part of the robot (<NUM>);
(b) detecting, using data from the camera (<NUM>), a user gesture within the workspace, the gesture indicating a location, and based on the detected gesture, projecting a real image comprising an interface onto a surface within the workspace at the indicated location, the location being spatially distinct from where the robot is to perform a control function;
(c) detecting gestural user selections of an element of the projected interface, the element of the projected interface corresponding to an item selection function;
(d) detecting, using data from the camera (<NUM>), a user gesture comprising touching a physical item within the workspace upon which the robot control function is to operate;
(e) detecting, using data from the camera (<NUM>), gestural user selection of an element of the projected interface corresponding to a goal location function, and a user gesture comprising touching a goal location within the workspace;
(f) detecting, using data from the camera (<NUM>), gestural user selections of an element of the projected interface, the element of the projected interface corresponding to the robot control function;
(g) computing a trajectory that will allow the robot to move the physical item to the goal location; and
(h) causing the robot to execute the control function on the physical item, the robot control function comprising moving the physical item to the goal location according to the calculated trajectory.