Patent ID: 12226920

DETAILED DESCRIPTION

FIG.1illustrates an example environment in which implementations disclosed herein can be implemented. The example environment includes a robot110, a computing device120, a computing device130, a user input system140, a training system150, and a robot system160. One or more of these components ofFIG.1can be communicatively coupled over one or more networks195, such as local area networks (LANs), wide area networks (WANs), and/or any other communication network. Each of the computing devices120,130can include various input devices and output devices. For example, the computing device120takes the form of a virtual or augmented reality headset that can be utilized to render various graphical user interfaces described herein. Further, the computing device120may utilize controller121as an input device, or simply track eye and/or hand movements of a user of the computing device120via various sensors of the computing device120to control the robot110. As another example, the computing device130can include a display and various input devices, such as a keyboard and mouse. Although particular components are depicted inFIG.1it should be understood that is for the sake of example and is not meant to be limiting. Moreover, although various components ofFIG.1are depicted as separate inFIG.1, it should be understood that is for the sake of example and is not meant to be limiting. For example, one or more of the computing devices120,130can implement one or more aspects of the user input system140, the training system150, and/or the robot system. Also, for example, one or more aspects of the user input system140, the training system150, and/or the robot system can be implemented by a remote system (e.g., server(s)) in communication with one or more of the computing devices120,130over one or more of the networks195.

A user can utilize the computing devices120,130, the user input system140, the training system, and the robot system160to train a robotic control policy for controlling the robot110in performance of various robotic tasks. The robotic control policy can correspond to one or more machine learning (ML) ML model(s) stored in ML model(s) database164A and a system that utilizes output, generated using the ML model(s), in controlling a robot, such as the robot system160and/or various engines thereof. As described herein, the techniques described herein relate to training and refining robotic control policies using imitation learning techniques. In particular, the robotic control policy can be initially trained based on demonstration data that is stored in demonstration data database152A and that is based on human demonstrations of various robotic tasks. Further, and subsequent to the initial training, the robotic control policy can be refined based on human interventions that are received during performance of various robotic tasks by the robot110. Moreover, and subsequent to the refining, the robotic control policy can be deployed for use in controlling the robot110during future robotic tasks.

The robot110illustrated inFIG.1is a particular real-world mobile robot. However, additional and/or alternative robots can be utilized with techniques disclosed herein, such as additional robots that vary in one or more respects from robot110illustrated inFIG.1. For example, a stationary robot arm, a mobile telepresence robot, a mobile forklift robot, an unmanned aerial vehicle (“UAV”), and/or a humanoid robot can be utilized instead of or in addition to robot110, in techniques described herein. Further, the robot110may include one or more engines implemented by processor(s) of the robot and/or by one or more processor(s) that are remote from, but in communication with, the robot110.

The robot110includes one or more visions component111that can generate instances of vision data (e.g., images, point clouds) related to shape, color, depth, and/or other features of object(s) that are in the line of sight of the vision component111. The instances of the vision data generated by one or more of the vision components can form some or all of state data (e.g., environmental state data and/or robot state data). The robot110can also include position sensor(s), torque sensor(s), and/or other sensor(s) that can generate data and such data, or data derived therefrom, can form some or all of state data (if any). Additionally, or alternatively, one or more vision components190that can generate the instances of the vision data may be located external from the robot110.

One or more of the vision components111,190may be, for example, a monocular camera, a stereographic camera (active or passive), and/or a light detection and ranging (LIDAR) component. A LIDAR component can generate vision data that is a 3D point cloud with each of the points of the 3D point cloud defining a position of a point of a surface in 3D space. A monocular camera may include a single sensor (e.g., a charge-coupled device (CCD)), and generate, based on physical properties sensed by the sensor, images that each includes a plurality of data points defining color values and/or grayscale values. For instance, the monocular camera may generate images that include red, blue, and/or green channels. A stereographic camera may include two or more sensors, each at a different vantage point, and can optionally include a projector (e.g., infrared projector). In some of those implementations, the stereographic camera generates, based on characteristics sensed by the two sensors (e.g., based on captured projection from the projector), images that each includes a plurality of data points defining depth values and color values and/or grayscale values. For example, the stereographic camera may generate images that include a depth channel and red, blue, and/or green channels.

The robot110also includes a base113with wheels117A,117B provided on opposed sides thereof for locomotion of the robot110. The base113may include, for example, one or more motors for driving the wheels117A,117B of the robot110to achieve a desired direction, velocity, and/or acceleration of movement for the robot110.

The robot110also includes one or more processors that, for example: provide control commands to actuators and/or other operational components thereof (e.g., robotic control policy engine166as described herein). The control commands provided to actuator(s) and/or other operational component(s) can form part of the action data (if any) that is included in the episode data162.

The robot110also includes robot arm114with end effector115that takes the form of a gripper with two opposing “fingers” or “digits.” Additional and/or alternative end effectors can be utilized, or even no end effector. For example, alternative grasping end effectors can be utilized that utilize alternate finger/digit arrangements, that utilize suction cup(s) (e.g., in lieu of fingers/digits), that utilize magnet(s) (e.g., in lieu of fingers/digits), etc. Also, for example, a non-grasping end effector can be utilized such as an end effector that includes a drill, an impacting tool, etc.

As noted above, a robotic control policy can be initially trained based on human demonstrations of various robotic tasks. As the human demonstrations are performed, demonstration data can be generated via the user input system140, and can be stored in demonstration data database152A. The demonstration data can include, for example, instances of vision data generated by one or more of the vision components112,190during performance of a given human demonstration of a given robotic task, state data of the robot110and/or the environment corresponding to the instances of the vision data captured during the given human demonstration of the given robotic task, corresponding sets of values for controlling respective components of the robot110corresponding to the instances of the vision data captured during the given human demonstration. For example, user input engine142can detect user input to control the robot110, and intervention engine144can generate the corresponding sets of values for controlling the respective components of the robot110. The corresponding sets of values utilized in controlling a respective component of the robot110can be, for example, a vector that describes a translational displacement and/or rotation (e.g., a sine-cosine encoding of the change in orientation about an axis of the respective component) of the respective component, lower-level control command(s) (e.g., individual torque commands that control corresponding actuator(s) of the robot110, individual joint angles of component(s) of the robot, etc.), binary values for component(s) of the robot (e.g., indicative of whether a robot gripper should be opened or closed), other values for component(s) of the robot110(e.g., indicative of an extent to which the robot gripper115should be opened or closed), velocities and/or accelerations of component(s) of the robot110(e.g., robot arm movement, robot base movement, etc.), and/or other values that can be utilized to control the robot110.

In some implementations, a human (or user) can utilize one or more of the computing devices120,130(or input devices thereof) to control the robot110to perform the human demonstrations of the robotic tasks. For example, the user can utilize the controller121associated with the computing device120to control the robot110, an input device associated with the computing device130to control the robot, or any other input device of any computing device in communication with the robot110, and the demonstration data can be generated based on the instances of the vision data captured by one or more of the vision components112,190, and based on the user control the robot110. In additional or alternative implementations, the user can physically manipulate the robot110or one or more components thereof (e.g., the base113, the robot arm114, the end effector115, and/or other components). For example, the user can physically manipulate the robot arm114, and the demonstration data can be generated based on the instances of the vision data captured by one or more of the vision components112,190, and based on the physical manipulation of the robot110. The user can repeat this process to generate demonstration data for performance of various robotic tasks.

In some implementations, the human demonstrations can be performed in a real-world environment using the robot110. For example, in the environment depicted inFIG.1, the user can control the robot110to perform a motion task by causing the robot110to traverse towards a table191, and perform a grasping task by causing the robot110to pick up a cup192. In additional or alternative implementations, the human demonstrations can be performed in a simulated environment using a simulated instance of the robot110via a robotic simulator164. For example, in implementations where the human demonstrations are performed in the simulated environment using a simulated instance of the robot110, sim configuration engine162can access object model(s) database162A to obtain a simulated instance of the table191and a simulated instance of the cup192. Further, the user can control the simulated instance of the robot110to perform a simulated motion task by causing simulated instance of the robot110to traverse towards the simulated instance of the table191, and perform a simulated grasping task by causing the simulated instance of the robot110to pick up the simulated instance of the cup192.

In some implementations, the robotic simulator164can be implemented by one or more computer systems, and can be utilized to simulate various environments that include corresponding environmental objects (e.g., using the sim configuration engine162), to simulate an instance the robot110operating in the simulated environment depicted inFIG.1and/or other environments, to simulate responses of the robot in response to virtual implementation of various simulated robotic actions in furtherance of various robotic tasks, and to simulate interactions between the robot and the environmental objects in response to the simulated robotic actions. Various simulators can be utilized, such as physics engines that simulate collision detection, soft and rigid body dynamics, etc. Accordingly, the human demonstrations and/or performance of various robotic tasks described herein can include those that are performed by the robot110, that are performed by another real-world robot, and/or that are performed by a simulated instance of the robot110and/or other robots via the robotic simulator164.

Training instance engine152can utilize the demonstration data stored in the demonstration data database152A to generate a plurality of training instances for bootstrapping a robotic control policy. Notably, the plurality of training instances can be generated based on single human demonstration or multiple human demonstrations (e.g., as described with respect toFIG.4). Each of the plurality of training instances include training instance input and corresponding training instance output. The training instance input can include, for example, an instance of vision data, state data when the instance of the vision data was captured (e.g., environmental state data and/or robot state data of the robot110), force value(s) of various components of the robot110when the instance of the vision data was captured, and/or other sensor data generated when the instance of the vision data was captured. The training instance output can include, for example, ground truth corresponding sets of values for respective components of the robot that are associated with the instance of the vision data (and optionally subsequent instances of the vision data that are subsequent to the instance of the vision data). For example, the training instance input can include an instance of vision data capturing a robot arm performing an object manipulation task (e.g., based on human control of the robot arm), and the training instance output can include ground truth corresponding sets of values for a robot arm and a robot end effector that correspond to values utilized by the robot in performance of the object manipulation task.

In some implementations, the training instance output can include the corresponding sets of values for the respective components of the robot110utilized to implement an action based on the instance of the vision data, and can also include additional corresponding sets of values associated with next actions that follow the action based on the instance of the vision data. In other words, the training instance output can include corresponding sets of values associated with a sequence of actions that are performed by the robot during the human demonstration. By including not only the corresponding sets of values for the action corresponding to the instance of the vision data included in the training input, but also the corresponding sets of values for one or more next actions that follow the action, the robotic control policy can be trained to generate the sequence of actions. As described herein, at inference (e.g., in an autonomous mode described herein), the robotic control policy may only generate the next action (and not one or more next actions that follow the action). However, by training the robotic control policy to generate the one or more next actions, the action that is generated for implementation by the robot110can be generated in view of the robotic task as a whole, rather than seemingly discrete actions. For example, assume the robotic task is a grasping task and the robot gripper115needs to move to the left in the environment towards an object to be grasped. In this example, the action generated using the robotic control policy can include moving the robot gripper115to the left even if it may not immediately impact performance of the grasping task. For instance, the robot gripper115may move forward and still accomplish the task. However, generating an action that causes the robot gripper115to move towards the left, rather than forward or right, is the best action for the robot110to implement in furtherance of the robotic task. Moreover, this may mitigate jerky movements of the robot110and result in smoother performance of the robotic task in terms of kinematic motion of respective components of the robot110. In these and other manners, the robotic control policy can be bootstrapped based on the demonstration data generated based on the human demonstrations.

In some implementations, one or more of the plurality of training instances may be labeled. The labels can indicate whether a given training instance, of the plurality of training instances, is a positive training instance or a negative training instance. For example, training instances that include the user correctly performing a given robotic task can be labeled as positive training instances, whereas training instances that include the user incorrectly performing the given robotic task can be labeled as negative training instances. For instance, if the demonstration data corresponds to the user correctly grasping an object, then the training instance can be labeled as a positive training instance. However, if the demonstration data corresponds to the user incorrectly grasping an object, bumping into an object, etc., then the training instance can be labeled as a negative training instance. By using labeled training instances in these and other manners, a robotic control policy can be trained in a more efficient manner, and also results in a trained robotic control policy having greater precision and/or recall. For example, a robotic control policy trained based on labeled training instances can predict, in various implementations, when the robot will fail in performance of a robotic task.

Training engine154can utilize the plurality of training instances generated by the training instance engine152to bootstrap a robotic control policy. As used herein, a robotic control policy refers to machine learning (ML) model(s) (e.g., stored in ML model(s) database) and a system that utilizes output, generated using the ML model(s), in controlling the robot110. The training engine152can train a given robotic control policy to generate a sequence of actions based on processing an instance of vision data (and optionally other data described herein) for a given training instance input. The sequence of actions can include a next action to be performed by the robot110in furtherance of the robotic task and a sequence of predicted actions that are predicted to follow the next action. In some implementations, training engine152can further utilize the state data for the given training instance input in generating the sequence of actions, but it should be understood that the sequence of actions described herein may be generated exclusively based on the instances of the vision data. Moreover, although techniques are described herein with respect to generating a sequence of actions, it should be understood that is for the sake of example and is not meant to be limiting. For instance, the techniques described herein may be utilized to generate the next action without generating any sequence of predicted actions that are predicted to follow the next action.

The ML model(s) representing the robotic control policy can include one or more input layers, a plurality of intermediate layers, and a plurality of disparate control heads. For example, and referring briefly toFIG.2A, the training engine154can cause an instance of vision data201A (and optionally a corresponding instance of state data201B) can be provided as input to one or more input layers211, and one or more intermediate layers212can process the instance of the vision data201A to generate an intermediate representation of the instance of the vision data202(e.g., an embedding, an encoding, etc.). In some implementations, one or more of the input layers211can perform one or more data processing functions on the instance of the vision data201A and/or the instance of the state data201B (e.g., concatenation and/or other functions) prior to providing the instance of the vision data201A and/or the instance of the state data201B to one or more of the intermediate layers212.

Further, the intermediate representation of the instance of the vision data202can be provided as input to a plurality of disparate control heads213. In some implementations, each of the plurality of disparate control heads213can be utilized to generate corresponding sets of values for controlling respective components of the robot110. For example, a first control head2131can be used to generate a corresponding first set(s) of values2031that reflect a translational displacement (e.g., two-dimensional or three-dimensional) the robot gripper114for one or more of the actions included in the sequence of actions; a second control head2132can be used to generate corresponding second set(s) of values2032that reflect an orientation of the robot gripper115for one or more of the actions included in the sequence of actions; a third control head2133can be used to generate a corresponding third set(s) of values2033that reflect an actuation state robot gripper115(or an extent of the actuation state of the robot end effector) for one or more of the actions included in the sequence of actions; a fourth control head2034can be used to generate a corresponding first set(s) of values2031that reflect the robot base113velocity and/or acceleration (or for the individual wheels117A,117B) for one or more of the actions included in the sequence of actions; and so on for an Nth control head213Nthat is used to generate corresponding Nth set(s) of values203N, where N is a positive integer. Although particular control heads associated with particular components of the robot110are described above, it should be understood that is for the sake of example and is not meant to be limiting, and that the plurality of control heads213utilized by the robotic control policy may be a function of the components of the robot110performing the robotic task. For instance, if the robot does not include the robot base113(e.g., a stationary robot arm), then a given control head associated with the robot base113may not be utilized. Also, for instance, if the robot includes robot legs, rather than the robot base113, a control head associated with one or more robot legs can be utilized in lieu of a control head for the robot base113. Moreover, although particular control heads are described above with respect to controlling disparate components of the robot, it should be understood that is for the sake of example and is not meant to be limiting. For instance, a first control head can be associated with controlling a first portion of a first component of the robot (e.g., the second control head2132described above with respect to the orientation of the robot gripper115), and a disparate second control head can be associated with controlling a second portion of the first component of the robot (e.g., the third control head2133described above with respect to the actuation state of the robot gripper115).

In various implementations, the intermediate representation of the instance of the vision data202can be provided as input to an additional control head (also referred to as a failure head) to generate a corresponding set of values associated with performance of the robotic task itself, rather than utilizing the corresponding set of values in controlling the respective components of the robot110as described above. The corresponding set of values associated with performance of the robotic task can indicate, for example, whether the robot110will continue performance of the robotic task, whether the robot110will fail in performance of the robotic task, and/or whether the robot110has completed performance of the robotic task. In other words, the robotic control policy can utilize this corresponding set of values to determine whether to prompt the user to intervene in performance of the robotic task. The corresponding set of values associated with performance of the robotic task can be, for example, a vector of values that may be mutually exclusive binary values indicative of whether the robot110will continue performance of the robotic task, whether the robot110will fail in performance of the robotic task, and/or whether the robot110has completed the robotic task, probabilities associated with whether the robot110will continue performance of the robotic task, whether the robot110will fail in performance of the robotic task, and/or whether the robot110has completed performance of the robotic task, and/or other values corresponding to performance of the robotic task. For example, the corresponding set of values generated using this failure head can include a vector of values for [fail, continue, complete] where each value of the vector is a binary value that is indicative of performance of the robotic task. As another example, the corresponding set of values generated using this failure head can include a vector of values for [fail/continue, complete] where the “fail/continue” value is a probability associated with whether the robot110will fail, and the “complete” value is a binary value associated with whether the robot110has completed performance of the robotic task.

In some implementations, update engine156can generate one or more one or more losses, and utilize one or more of the losses to update the control heads213. In some of those implementations, one or more (e.g., all) of the losses can be generated utilizing a loss function that is different from one or more (e.g., all) of the loss functions utilized in generating the other losses. As one particular example, a first loss utilized to update a first control head of the robotic control policy can be generated based on comparing a first corresponding set of values to a corresponding alternative first set of values for the next action and/or for one or more of the plurality of additional predicted actions, a second loss utilized to update a second control head of the robotic control policy can be generated based on comparing a second corresponding set of values to a corresponding alternative second set of values for the next action and/or for one or more of the plurality of additional predicted action, and so on for each of the control heads utilized in controlling the various components of the robot. In this example, a first loss function can be utilized to generate the first loss, and a distinct second loss function can be utilized to generate the second loss. Further, the respective losses can be utilized to update the respective control heads.

For example, and referring briefly toFIG.2B, the update engine156can generate: one or more first losses1561based on comparing the corresponding first set of value(s)2031to respective ground truth corresponding first set(s) of values2041for the given training instance; one or more second losses based on comparing the corresponding second set of value(s)2032to respective corresponding second set(s) of values2042for the given training instance; one or more third losses based on comparing the corresponding third set of value(s)2033to respective ground truth corresponding third set(s) of values2043for the given training instance; one or more fourth losses based on comparing the corresponding fourth set of value(s)2033to respective ground truth corresponding fourth set(s) of values2044for the given training instance; and so on for one or more Nth losses156N. The update engine156can subsequently utilize the respective losses to update the respective control heads213. Notably, various loss functions can be utilized in generating the one or more losses for each of the control heads. For example, one or more of the first losses can be generated using a mean squared error loss function, one or more of the second losses can be generated using a mean squared error loss function and/or a quaternion norm loss function, one or more of the third losses can be generated using a log loss function, and so on. The training engine154can repeat this process for a plurality of additional training instances. Although only the one or more first losses1561and the one or more Nth losses156Nare depicted inFIG.2Bas being utilized to update their respective control heads, it should be understood that is for the sake of clarity and is not meant to be limiting.

In some implementations, and referring back toFIG.1, evaluation engine158can evaluate the robotic control policy and determine whether to transition from a bootstrapping mode (e.g., training the robotic control policy based on the plurality of training instances as described above) to a semi-autonomous mode to further refine the robotic control policy. The evaluation engine158can determine whether to transition from the bootstrapping mode to the semi-autonomous mode when one or more conditions are satisfied. The one or more conditions can include, for example, occurrence of training based on at least a threshold quantity of training instances, a threshold duration of training based on the training instances, and/or other conditions described herein.

In the semi-autonomous mode, RCP engine166can utilize the trained robotic control policy to control the robot110in performance of various robotic tasks. In some implementations, the robot110may initiate performance of a robotic task in response to receiving user input to initiate performance of the robotic task. For example, the user can provide user input at one of the computing devices120,130or the robot110itself to initiate performance of a robotic task. In some implementations, the RCP engine166can utilize the robotic control policy to control the robot in performance of the robotic task by processing instances of vision data to generate one or more actions to be performed, and the robot110can autonomously perform one or more of the actions. In some versions of those implementations, the user may be prompted to intervene in performance of the robotic task when the RCP engine166determines that the robot110has failed in performance of the robotic task and/or is predicted to fail in performance of the robotic task.

For example, assume the robot110is performing a robotic task of opening a latched door that requires the robot110to traverse a path to the latched door using the robot base113, manipulate a door handle via the robot gripper115coupled to the robot arm114, and traverse another path to push or pull the door to an open position. In this example, the RCP engine166can process, using the robotic control policy, an instance of vision data generated by one or more of the vision components112to generate a sequence of actions for a first iteration. The sequence of actions can include an initial action and a plurality of predicted actions that are predicted to follow the initial action. Further, each of these actions can be associated with corresponding sets of values for controlling respective components of the robot110. Notably, a representation of the sequence of actions can be provided for presentation to a human via a graphical user interface of one of the computing devices120,130(e.g., described with respect toFIGS.3B and3C). Assuming that the RCP engine166determines that the robot110will not fail in performance of the robotic task based on processing of the instance of the vision data using the failure head (and optionally along with the sequence of actions), the robot can utilize the corresponding sets of values to perform the initial action in furtherance of the robotic task of opening the latched door. Further, the RCP engine166can process, using the robotic control policy, an additional instance of the vision data generated by one or more of the vision components112to generate an additional sequence of actions for a second iteration. The additional sequence of actions can include a next action and a plurality of additional predicted actions that are predicted to follow the next action. An additional representation of the additional sequence of actions can be provided for presentation to the human via the graphical user interface of one of the computing devices120,130(e.g., described with respect toFIGS.3B and3C). The RCP engine166can continue until performance of the robotic task is completed.

However, in implementations where the RCP engine166determines, using the robotic control policy, that the robot will fail in performance of the robotic task based on processing of the instance of the vision data or the user proactively intervenes in performance of the robotic task, the user can be prompted to intervene in performance of the robotic task of opening the latched door. The prompt can be generated using the intervention engine144. In this example, the user can provide user input via an input device of one of the computing devices120,130(e.g., controller121) to control the robot110responsive to receiving the prompt. In some implementations, and based on the user input, the intervention engine144can generate alternative sets of corresponding values for controlling respective components of the robot110. The update engine156can utilize these alternative sets of corresponding values for controlling various components of the robot110can be utilized in generating one or more losses for updating the robotic control policy (e.g., in the same or similar manner described with respect toFIG.2B). For example, assume the user was prompted to intervene in performance of the robotic task. In this example, the corresponding sets of values generated across the disparate control heads can be compared to the alternative corresponding sets of values generated by the intervention engine144based on the user input. The comparison(s) can be used to generate one or more losses, and the robotic control policy (or the ML model(s) can be updated based on one or more of the losses. In some implementations, a corresponding loss can be determined for each of multiple control heads and the corresponding loss utilized to update the corresponding control head. In implementations where the human intervenes in performance of the robotic task, such as when the user believes the robot will continue traversing the path through the door without stopping to open the door, one or more losses can be generated in the same or similar manner and utilized to update the control heads.

In some implementations, the evaluation engine158can evaluate the refined robotic control policy and determine whether to transition from the semi-autonomous mode to an autonomous mode to validate the robotic control policy. The evaluation engine158can determine whether to transition from the semi-autonomous mode to the autonomous when one or more conditions are satisfied. The one or more conditions can include, for example, convergence of the robotic control policy (e.g., zero loss(es) or within a threshold range of zero loss(es)), determination that the robotic control policy performs better (e.g., with respect to precision and/or recall) than the instance of the robotic control policy currently being utilized (if any), occurrence of refining based on at least a threshold quantity of human interventions, and/or a threshold duration of refining in the semi-autonomous mode.

In the autonomous mode, the RCP engine166can also utilize the trained robotic control policy to control the robot110in performance of various robotic tasks. However, and in contrast with the semi-autonomous mode, the user may not be prompted to intervene in performance of robotic tasks. Rather, the robot110is able to mitigate failure and/or recover from failure in performance of various robotic tasks by virtue of the robotic control policy being trained and refined using the techniques described herein. For example, if the RCP engine166determines that a next action to be performed will result in the robot110colliding with an obstacle, the RCP engine166can avoid performing the next action, and implement a different next action to be performed by the robot110at a subsequent iteration.

By using the techniques described herein, various technical advantages can be achieved. As one non-limiting example, by prompting a user to intervene with performance of a robotic task when it is determined the robot110will fail in performance of the robotic task, the robotic control policy is trained to learn how to perform the robotic task and how to recover from failure in performance of the robotic task, rather than simply being trained to learn how to perform the robotic task. As a result, the user may be prompted to intervene less frequently as the robotic control policy is trained when compared to other known techniques, thereby conserving computational resources, network resources, and/or robot resources. As another non-limiting example, by enabling the human to intervene based on visual representations of actions the robot may perform, collisions with various obstacles in an environment of the robot110may be avoided. As a result, any potential damage to these obstacles and the robot can be mitigated or avoided completely. As another non-limiting example, by using the architecture described herein, each of the control heads that are dedicated to respective components (or portions of the respective components) of the robot110can be refined based on one or more losses that are specific to those control heads. As a result, the robotic control policy can be trained in a quicker and more efficient manner and based on fewer human demonstrations by the user.

Turning now toFIGS.3A-3C, various non-limiting examples of a graphical user interface300of a platform utilized in training and refining robotic control policies are depicted. A user can interact with the platform to train and refine robotic control policies. The platform can be implemented, at least in part, by a computing device (e.g., one or more of the computing devices120,130ofFIG.1, server(s), and/or other computing devices), and can include, for example, a graphical user interface300, input devices, and/or output devices. For example, the platform can enable the user to create or select a robotic control policy, and initially bootstrap the robotic control policy based on human demonstrations stored in one or more databases (e.g., the demonstrations data database152A ofFIG.1) and/or generated as the user interacts with robot(s) in a human demonstration mode352B1(e.g., described with respect toFIG.3A). Further, the platform can enable the user to transition control of the robot(s) from the human demonstration mode352B1to a semi-autonomous mode352B2to further refine the robotic control policy (e.g., described with respect toFIG.3B). Moreover, the platform can enable the user to transition control of the robot(s) from the semi-autonomous mode352B2to an autonomous mode352B3to further evaluate the robotic control policy (e.g., described with respect toFIG.3C). In some implementations, the user can transition between these various modes352B by selecting corresponding graphical elements provided for display at the graphical user interface300of the platform. In additional or alternative implementations, the platform can automatically transition between these modes352B based on evaluating performance of the robotic control policy.

In some implementations, and referring specifically toFIG.3A, the human demonstration mode352B1can be utilized to bootstrap a robotic control policy based on human demonstrations. For example, a human demonstrator can use teleoperation to control a robot, kinesthetically manipulation the robot, and/or utilize other techniques to demonstrate how the robot should perform the task. Training instances320can be generated based on the human demonstrations, and each of the training instances can include training instance input322A and training instance output323A. The training instance input321A can include, for example, an instance of vision data and/or state data when the instance of the vision data was captured (e.g., environmental state data and/or robot state data) as indicated by various training instance inputs321A1,321A2,321A3, and321A4depicted inFIG.3A(and optionally other data as described herein, such as force value(s) for various components of the robot when the instance of the vision data was captured). The training instance output322A can include, for example, corresponding sets of values for respective components of the robot that are associated with the human demonstration as indicated by various training instance outputs322A1,322A2,322A3, and322A4depicted inFIG.3A. Notably, the corresponding sets of values included in the training instance output322A may include corresponding sets of values for multiple actions performed by the robot subsequent to the instance of the vision data being captured. In some implementations, each of the training instances320can be associated with a particular robotic task323A as indicated by various robotic tasks323A1,323A2,323A3, and323A4depicted inFIG.3A. For example, the training instance input321A1can include an instance of vision data capturing a robot arm performing an object manipulation task (e.g., a grasping task as indicated by323A1), and the training instance output321A1can include at least corresponding sets of values for a robot arm and a robot end effector (e.g., a robot gripper) that correspond to values utilized by the robot in performance of the object manipulation task.

The platform can instruct the user to perform various tasks to guide the user in training the robotic control policy in the human demonstration mode352B1. For example, the platform can provide an indication that the user should perform human demonstrations of various robotic tasks at352B1A. Further, the platform can include a log of human demonstrations performed at352B1B. In this example, the platform indicates that 58 human demonstrations of object manipulation tasks have been performed, 47 human demonstrations of motion tasks have been performed, 15 human demonstrations of combined tasks have been performed, and 12 human demonstrations of other tasks have been performed. The training instances320shown inFIG.3Acan be generated based on these human demonstrations.

In some implementations, the training instances320can be sorted based on one or more criteria. The one or more criteria can include can include, for example, a type of robotic task associated with a given training instance (e.g., grasping task, motion task, etc.), a type of training instance output (e.g., corresponding sets of values associated with specific robot components), a type of training instance (e.g., positive or negative), and/or other criteria. For example, in response to a selection of a sort training instances graphical element382, the platform can cause the training instances320to be sorted based on one or more of the criteria. The sort training instances graphical element382, when selected, can optionally cause a dropdown menu with various criteria for sorting the training instances to be visually rendered at the graphical user interface300.

In some implementations, one or more of the training instances320may include one or more errors. The one or more errors can include, for example, mislabeled training instances (e.g., positive training instance or negative training instance), insufficient vision data and/or state for the training instance input321A, insufficient corresponding sets of values for the training instance output322A, and/or other errors. In these implementations, the user can view the training instances320that include errors, and can cause one or more actions to be performed to current one or more of the errors. For example, in response to a selection of a training instance errors graphical element383, the platform can cause the training instances320that include one or more errors to be presented to the user. The user can then relabel one or more of these training instances, discard one or more of these training instances, and/or perform any other action to address one or more of the errors.

In some implementations, the platform can cause the robotic control policy to be automatically trained based on the training instance320(e.g., as described with respect toFIG.4). In additional or alternative implementations, the platform can cause the robotic control policy to be trained in response to user input detected at the platform. For example, in response to a selection of a training graphical element384, the platform can cause the robotic control policy to be trained based on the training instances320. In these and the manners, the robotic control policy can be bootstrapped based on the human demonstrations.

In some implementations, and referring specifically toFIG.3B, the semi-autonomous mode352B2can be utilized to refine the robotic control policy based on human interventions received during performance of a semi-autonomous robotic task330subsequent to the bootstrapping of the robotic control policy. While in the semi-autonomous mode352B2, the robotic control policy can be evaluated based on a quantity of human interventions received during performance of the robotic task. In other words, the robot can be evaluated based on how well it performs without the human interventions.

For example, assume the user provides input of “go pick up the cup” as indicated by352B2A. In this example, the robot can utilize the trained robotic control policy to generate a sequence of actions to traverse a path towards the table191, and pick up the cup192. In some implementations, the graphical user interface380can additionally or alternatively be utilized to visually render representations of the sequences of actions as the robot performs robotic tasks as indicated by331,332,333,334,335, and336. In some versions of those implementations, the representation of the sequence of actions visually rendered via the graphical user interface380includes a sequence of corresponding waypoints overlaying an environment330A of the robot captured in the instance of the vision data (e.g., as depicted inFIG.3B). Each of the corresponding waypoints can be associated with one or more components of the robot in response to a given action, included in the sequence of actions, being implemented by the robot. In the example ofFIG.3B, the sequence of waypoints may correspond to a robotic gripper utilized in performance of the grasping task. Although the sequence of actions depicted inFIG.3Bare waypoints for the robot gripper, it should be understood that is for the sake of example and is not meant to be limiting. In additional or alternative implementations, the representation of the sequence of actions visually rendered via the graphical user interface380includes a sequence of corresponding states of the robot overlaying the environment330A of the robot captured in the instance of the vision data. Each of the corresponding states of the robots may correspond to a given state of the robot in response to a given action, included in the sequence of actions, being performed by the robot. In various implementations, the representation of each of the actions can be selectable such that, when selected, the corresponding sets of values for the various components of the robot can be visually rendered for presentation to the user. For example, assume the user selects waypoint331. In response to the selection of the waypoint331by the user, one or more values for the robot can be presented for presentation to the user as indicated by331A.

In some implementations, the robotic control policy may determine that the robot has failed in performance of the robotic task and/or is predicted to fail to in performance of the robotic task (e.g., using the failure head of the robotic control policy described with respect toFIGS.1,2A, and2B). In those implementations, the platform can generate a prompt337, and the prompt can be provided for presentation to the user via the graphical user interface380. The prompt337can request that the user intervene in performance of the robotic task. For example, the user can utilize an input device (e.g., the controller121ofFIG.1) to take control of the robot and correct one or more actions in performance of the robotic task. One or more losses can be generated based on the user intervening in performance of the robotic task (e.g., as described with respect toFIGS.1,2A, and2B), and one or more of the losses can be subsequently utilized to refine the robotic control policy. In some of those implementations, the user can dismiss the prompt337as indicated by337A. In implementations where the prompt337is dismissed, the failure head can be subsequently updated to mitigate occurrences of prompting the user when similar instances of vision data are processed using the robotic control policy and/or when similar actions are generated using the robotic control policy.

In additional or alternative implementations, the user may proactively intervene in performance of the robotic task based on the representation of the sequence of actions visually rendered for presentation to the user. For example, waypoints335and336indicate that the robot gripper is not predicted to grasp the cup192based on the actions. Rather, it appears that the robot will either bump into the table191and reach over the cup192to perform the grasping task, or go to a far side of the table191to perform the grasping task. As a result, the user may proactively utilize an input device (e.g., the controller121ofFIG.1) to take control of the robot and correct one or more actions in performance of the robotic task. One or more losses can be generated based on the user intervening in performance of the robotic task (e.g., as described with respect toFIGS.1,2A, and2B), and one or more of the losses can be subsequently utilized to refine the robotic control policy.

Subsequent to any human intervention in performance of the robotic task, the robotic control policy can take control of the robot from the human. However, if during performance of the robotic task, the robotic control policy determines that the robot has failed again or is predicted to fail again, then the platform can generate an additional prompt, and the additional prompt for presentation to the user via the graphical user interface380. This process can be repeated until the robot completes performance of the robotic task.

In various implementations, one or more metrics related to performance of the robotic task in the semi-autonomous mode352B2can be provided for presentation to the user. For example, one or more of the metrics can be associated with performance of the current robotic task in the semi-autonomous mode352B2as indicated by352B2B, performance of historical robotic tasks performed in the semi-autonomous mode352B2as indicated by352B2C, and/or other metrics. In various implementations, the platform can make a recommendation as to particular tasks that should be performed in the semi-autonomous mode352B2as indicated by352B2D.

In some implementations, and subsequent to performance of the semi-autonomous robotic task330, the robotic control policy can be automatically updated based on one or more losses generated based on the interventions. In some versions of those implementations, a threshold quantity of semi-autonomous robotic tasks may be performance prior to causing the robotic control policy to be refined. In additional or alternative implementations, the robotic control policy can be refined in response to receiving user input. For example, in response to a selection of a refine policy graphical element385, the robotic control policy can be refined based on one or more of the losses generated during performance of the semi-autonomous robotic task330.

In some implementations, and referring specifically toFIG.3C, the autonomous mode352B3can be utilized to evaluate performance of the robotic control policy subsequent to refining the robotic control policy in the semi-autonomous mode352B2. In the autonomous mode352B3, the user may not be prompted to intervene in performance of an autonomous robotic task340. A robot can utilize the robotic control policy to perform various robotic tasks autonomously in an environment340A, and the robotic control policy can be evaluated based on whether or not the robot completes the robotic task and/or how efficient the robot performs the robotic tasks. For example, assume the user provides input of “go pick up the cup” as indicated by352B3A. In this example, the robot can utilize the refined robotic control policy to generate a sequence an action in furtherance of traversing a path towards the table191, and picking up the cup192. Notably, in various implementations at inference, the robotic control policy may only generate a next action, rather than the sequence of actions described above with respect to the semi-autonomous mode352B2. However, waypoints341,342,343,344, and345are depicted inFIG.3Cfor the sake of example, and illustrate that performance of the robotic task has improved based on the robotic control policy being refined in the semi-autonomous mode352B2(e.g., as indicated by fewer waypoints and waypoint345indicating the robot gripper will correctly grasp the cup192).

Similar to the semi-autonomous mode352B2, in various implementations, one or more metrics related to performance of the robotic task in the autonomous mode352B3can be provided for presentation to the user. For example, one or more of the metrics can be associated with performance of the current robotic task in the autonomous mode352B3as indicated by352B3B and/or other metrics. In various implementations, the platform can make a recommendation based on performance of the robotic control policy in performing the autonomous robotic task340. For example, as indicated by352B3C, the platform can recommend that the robotic control policy should be further refined in the semi-autonomous mode352B2

AlthoughFIGS.3A-3Care described with respect to particular robotic tasks, it should be understood that is for the sake of example and is not meant to be limiting. For example, other robotic tasks can be performed to further refine the robotic policy, such as opening doors, sorting through various objects, opening a push-pull door, opening a latched door, navigation and obstacle avoidance tasks, and/or any other task that can be performed by the robot I a real or simulated environment.

Turning now toFIG.4, a flowchart illustrating an example method400of generating training instances based on human demonstrations and bootstrapping a robotic control policy based on the human demonstrations is depicted. In other words, the method400describes a bootstrapping mode for training a robotic control policy. For convenience, the operations of the method400are described with reference to a system that performs the operations. This system may include one or more processors, such as processor(s) of user input system140and/or training system150. Moreover, while operations of method400are shown in a particular order, this is not meant to be limiting. One or more operations may be reordered, omitted or added.

At block452, the system obtains demonstration data for a robot that is generated based on a human demonstration of a robotic task. The demonstration data can be generated based on human demonstrations performed in a simulated environment using a simulated robot and/or in a real-world environment using an instance of a simulated robot.

At block454, the system generates a plurality of training instances based on the demonstration data. Each of the training instances can include training instance input and training instance output. The training instance input can include, for example and as indicated at sub-block454A, an instance of vision data and/or state data associated with the instance of the vision data. The training instance output can include, for example and as indicated at sub-block454B, one or more corresponding ground truth sets of values for respective components of the robot when the instance of the vision data was captured. Generating the plurality of training instances is described herein (e.g., with respect to the training instance engine152ofFIG.1).

At block456, the system determines whether to generate a plurality of additional training instances. The system can determine whether to generate a plurality of additional training instances based on, for example, a quantity of training instances that are available to train a robotic control policy, a quantity of training instances associated with particular robotic tasks, and/or other criteria. If, at an iteration of block456, the system determines to generate a plurality of additional training instances, the system returns to block452to obtain additional demonstration data for the robot and generates a plurality of additional training instances based on the additional demonstration data at an additional iteration of454. If, at an iteration of block456, the system determines not to generate a plurality of additional training instances, the system proceeds to block458.

At block458, the system trains a robotic control policy based on the plurality of training instances. Training the robotic control policy is described in detail herein (e.g., with respect to the training engine154ofFIG.1).

At block460, the system determines whether one or more conditions are satisfied. The one or more conditions can include, for example, occurrence of training based on at least a threshold quantity of training instances, a threshold duration of training based on the training instances, and/or other conditions described herein. If, at an iteration of block460, the system determines the one or more conditions are not satisfied, the system returns to block452and repeats the operations of block452-458. If, at an iteration of block460, the system determines one or more of the conditions are satisfied, the system proceeds to block462.

At block462, the system causes the robot to utilize the robot control policy in a semi-autonomous mode to refine the robotic control policy. The semi-autonomous mode is described in detail herein (e.g., with respect toFIGS.1and3B).

Turning now toFIG.5, a flowchart illustrating an example method500of refining a robotic control policy based on a human intervening in performance of a robotic task is depicted. In other words, the method500describes a semi-autonomous mode for refining a trained robotic control policy. For convenience, the operations of the method500are described with reference to a system that performs the operations. This system may include one or more processors, such as processor(s) of user input system140, training system150, and/or robot system160ofFIG.1. Moreover, while operations of method500are shown in a particular order, this is not meant to be limiting. One or more operations may be reordered, omitted or added.

At block552, the system causes a robot to initiate performance of a robotic task. The system can cause the robot to initiate performance of the robotic task in response to receiving user input to initiate performance of the robotic task at one or more computing devices.

At block554, the system receives, from one or more vision components, an instance of vision data capturing an environment of the robot. The vision components can be a component of the robot and/or external to the robot.

At block556, the system processes, using a robotic control policy, the instance of the vision data to generate a sequence of action that includes an initial action and a plurality of predicted actions that are predicted to follow the initial action. In some implementations, and as indicated at sub-block556A, the system processes, using an intermediate portion of a robotic control policy, the instance of the vision data to generate an intermediate representation of the instance of the vision data. Further, and as indicated at sub-block556B, the system processes, using a plurality of disparate control heads of the robotic control policy, the intermediate representation of the instance of the vision data to generate corresponding sets of values for respective components of the robot. In some implementations, the system may only generate the sequence of actions in a particular mode, such as in the semi-autonomous mode described herein (e.g., with respect toFIGS.1,3B, and3C). In additional or alternative implementations, the system may only generate the initial action.

At block558, the system causes a representation of the sequence of actions to be presented to a user. In some implementations, the system may only cause the representation of the sequence of actions to be presented to the user in a particular mode, such as in the bootstrapping mode, the semi-autonomous mode, and/or the autonomous mode described herein (e.g., with respect toFIGS.1and3A-3C).

At block560, the system determines whether the robot has failed, or is predicted to fail, in performance of the robotic task. The system can determine whether the robot has failed, or is predicted to fail, in performance of the robotic task based on output generated using the robotic control policy (e.g., using the failure head described with respect toFIGS.1,2A, and2B).

If, at an iteration of block560, the system determines that the robot has failed, or is predicted to fail, in performance of the robotic task, the system proceeds to block562. At block562, the system causes the robot to implement the initial action. For example, the system can utilize the corresponding sets of values for the respective components of the robot to actuate the respective components of the robot to perform the initial action. The system may then return to block554to receive, from one or more of the vision components, an additional instance of the vision data capturing the environment of the robot and repeat the remaining operations of the method500until performance of the robotic task is complete.

If, at an iteration of block560, the system determines that the robot has not failed, and is not predicted to fail, in performance of the robotic task, the system proceeds to block564. At block564, the system causes a prompt that requests the human intervene in performance of the robotic task to be presented to the user. The prompt can be visually and/or audibly rendered for presentation to the user via the robot or a computing device.

At block566, the system receives user input that intervenes in performance of the robotic task. The user input can control the robot and recover from the failure in performance of the robotic task and/or prevent the failure in performance of the robotic task. In some implementations, the user can provide the input via one or more computing devices in communication with the robot responsive to the prompt. In additional or alternative implementations, the user may proactively intervene in performance of the robotic task without any prompt being rendered for presentation to the user.

At block568, the system generates one or more losses based on the user input and/or based on implementing the initial action. In some implementations, such as when the user input is received, the corresponding sets of values for the respective components of the robot generated using the robotic control policy at block556, one or more of the losses can be generated based on comparing the corresponding sets of values for the respective components of the robot to corresponding alternative sets of values for the respective components of the robot. In additional or alternative implementations, such when the robot implements the initial action at block562, the system can compare the corresponding sets of values for the respective components of the robot to corresponding demonstration sets of values from one or more human demonstrations of the same robotic task (if any). Generating one or more of the losses in this manner is described in detail herein (e.g., with respect toFIGS.1and2).

At block570, the system causes the robotic control policy to be updated based on one or more of the losses. In some implementations, one or more of the losses generated at block568are associated with specific control heads, and the corresponding specific control heads can be updated based on the associated one or more losses. The system may then return to block554to receive, from one or more of the vision components, an additional instance of the vision data capturing the environment of the robot and repeat the remaining operations of the method500until performance of the robotic task is complete. In some implementations, the system can cause the robot to initiate performance of an additional robotic task. In some versions of those implementations, the system can continue performing additional robotic task until one or more conditions are satisfied. The one or more conditions can include, for example, convergence of the robotic control policy (e.g., zero loss(es) or within a threshold range of zero loss(es)), determination that the robotic control policy performs better (e.g., with respect to precision and/or recall) than the instance of the robotic control policy currently being utilized (if any), occurrence of refining based on at least a threshold quantity of human interventions, and/or a threshold duration of refining in the semi-autonomous mode.

Turning now toFIG.6, a flowchart illustrating an example method600of utilizing a robotic control policy in autonomously controlling a robot is depicted. In other words, the method600describes an autonomous mode for utilizing a refined robotic control policy. For convenience, the operations of the method600are described with reference to a system that performs the operations. This system may include one or more processors, such as processor(s) of robot system160ofFIG.1. Moreover, while operations of method600are shown in a particular order, this is not meant to be limiting. One or more operations may be reordered, omitted or added.

At block652, the system causes a robot to initiate performance of a robotic task. At block654, the system receives, from one or more vision components, an instance of vision data capturing an environment of the robot. At block656, the system processes, using a robotic control policy, the instance of the vision data to generate a sequence of action that includes an initial action and a plurality of predicted actions that are predicted to follow the initial action. The system can perform the operations of block652-656in the same or similar manner described with respect to blocks552-556, respectively, ofFIG.5. At block658, the system causes the robot to implement the initial action. The system can implement the initial action in the same or similar manner described with respect to block562ofFIG.5.

At block660, the system determines whether the robot has completed performance of the robotic task. The system can determine whether the robot has completed performance of the robotic task based on output generated using the robotic control policy (e.g., using the failure head described with respect toFIGS.1,2A, and2B). If, at an iteration of block660, the system determines the robot has not completed performance of the robotic task, the system returns to block654. The system may then return to block754to receive, from one or more of the vision components, an additional instance of the vision data capturing the environment of the robot and repeat the remaining operations of blocks654-660until performance of the robotic task is complete. If, at an iteration of block660, the system determines the robot has completed performance of the robotic task, the system proceeds to block662.

At block662, the system determines whether one or more conditions are satisfied. The one or more conditions can include, for example, determination that the robotic control policy performs better (e.g., with respect to precision and/or recall) than the instance of the robotic control policy currently being utilized (if any), achievement of a threshold measure of autonomy of the robot without failing in performance of the robotic task, and/or other conditions described herein. If, at an iteration of block662, the system determines the one or more conditions are not satisfied, the system proceeds to block664. At block664, the system causes the robot to utilize the robotic control policy in a semi-autonomous mode to refine the robotic control policy. In other words, the system can test the robotic control policy in the autonomous mode, but determine the robotic control policy needs to be further refined and return the semi-autonomous mode (e.g., described with respect toFIG.5). If, at an iteration of block662, the system determines one or more of the conditions are satisfied, the system proceeds to block666.

At block666, the system causes the robot to utilize the robotic control policy in performance of future robotic tasks. In some implementations, the robotic control policy may be trained, refined, and utilized for specific robotic tasks (or specific aspects of a given robotic task). For example, the system may train and refine a first robotic control policy for grasping tasks, a second robotic control policy for motion tasks, etc. In this example, the first and second robotic control policies can be utilized in performance of their respective robotic tasks or both utilized in performance of a combined task (e.g., opening a latched door). In other implementations, a single robotic control policy can be trained, refined, and utilized for multiple robotic tasks (or multiple aspects of a given robotic task). For example, the system may train and refine a robotic control policy for grasping tasks, motion tasks, etc. In this example.

Turning now toFIG.7, an example architecture of a robot720is schematically depicted. The robot720includes a robot control system760, one or more operational components740a-740n, and one or more sensors742a-742m. The sensors742a-742mmay include, for example, vision sensors, light sensors, pressure sensors, pressure wave sensors (e.g., microphones), proximity sensors, accelerometers, gyroscopes, thermometers, barometers, and so forth. While sensors742a-742mare depicted as being integral with robot720, this is not meant to be limiting. In some implementations, sensors742a-742mmay be located external to the robot720, e.g., as standalone units.

Operational components740a-740nmay include, for example, one or more end effectors and/or one or more servo motors or other actuators to effectuate movement of one or more components of the robot. For example, the robot720may have multiple degrees of freedom and each of the actuators may control actuation of the robot720within one or more of the degrees of freedom responsive to the control commands. As used herein, the term actuator encompasses a mechanical or electrical device that creates motion (e.g., a motor), in addition to any driver(s) that may be associated with the actuator and that translate received control commands into one or more signals for driving the actuator. Accordingly, providing a control command to an actuator may comprise providing the control command to a driver that translates the control command into appropriate signals for driving an electrical or mechanical device to create desired motion.

The robot control system760may be implemented in one or more processors, such as a CPU, GPU, and/or other controller(s) of the robot720. In some implementations, the robot720may comprise a “brain box” that may include all or aspects of the control system760. For example, the brain box may provide real time bursts of data to the operational components740a-740n, with each of the real time bursts comprising a set of one or more control commands that dictate, inter alio, the parameters of motion (if any) for each of one or more of the operational components740a-740n. In some implementations, the robot control system760can be used to implement actions described herein, whether they be actions selected based on an engineered policy according to techniques disclosed herein, or actions selected using an RL policy model that is trained based at least in part on RL compatible data generated according to techniques disclosed herein.

Although control system760is illustrated inFIG.7as an integral part of the robot720, in some implementations, all or aspects of the control system760may be implemented in a component that is separate from, but in communication with, the robot720. For example, all or aspects of control system760may be implemented on one or more computing devices that are in wired and/or wireless communication with the robot720, such as computing device810.

Turning now toFIG.8, a block diagram of an example computing device810that may optionally be utilized to perform one or more aspects of techniques described herein is depicted. Computing device810typically includes at least one processor814which communicates with a number of peripheral devices via bus subsystem812. These peripheral devices may include a storage subsystem824, including, for example, a memory subsystem825and a file storage subsystem826, user interface output devices820, user interface input devices822, and a network interface subsystem816. The input and output devices allow user interaction with computing device810. Network interface subsystem816provides an interface to outside networks and is coupled to corresponding interface devices in other computing devices.

User interface input devices822may include a keyboard, pointing devices such as a mouse, trackball, touchpad, or graphics tablet, a scanner, a touchscreen incorporated into the display, audio input devices such as voice recognition systems, microphones, and/or other types of input devices. In general, use of the term “input device” is intended to include all possible types of devices and ways to input information into computing device810or onto a communication network.

User interface output devices820may include a display subsystem, a printer, a fax machine, or non-visual displays such as audio output devices. The display subsystem may include a cathode ray tube (CRT), a flat-panel device such as a liquid crystal display (LCD), a projection device, or some other mechanism for creating a visible image. The display subsystem may also provide non-visual display such as via audio output devices. In general, use of the term “output device” is intended to include all possible types of devices and ways to output information from computing device810to the user or to another machine or computing device.

Storage subsystem824stores programming and data constructs that provide the functionality of some or all of the modules described herein. For example, the storage subsystem824may include the logic to perform selected aspects of the method ofFIG.4, the method ofFIG.5, and/or the method ofFIG.6.

These software modules are generally executed by processor814alone or in combination with other processors. Memory825used in the storage subsystem824can include a number of memories including a main random-access memory (RAM)830for storage of instructions and data during program execution and a read only memory (ROM)832in which fixed instructions are stored. A file storage subsystem826can provide persistent storage for program and data files, and may include a hard disk drive, a floppy disk drive along with associated removable media, a CD-ROM drive, an optical drive, or removable media cartridges. The modules implementing the functionality of certain implementations may be stored by file storage subsystem826in the storage subsystem824, or in other machines accessible by the processor(s)814.

Bus subsystem812provides a mechanism for letting the various components and subsystems of computing device810communicate with each other as intended. Although bus subsystem812is shown schematically as a single bus, alternative implementations of the bus subsystem may use multiple busses.

Computing device810can be of varying types including a workstation, server, computing cluster, blade server, server farm, or any other data processing system or computing device. Due to the ever-changing nature of computers and networks, the description of computing device810depicted inFIG.8is intended only as a specific example for purposes of illustrating some implementations. Many other configurations of computing device810are possible having more or fewer components than the computing device depicted inFIG.8.

In some implementations, a method implemented by one or more processors is provided and includes receiving, from one or more vision components of a robot, an instance of vision data capturing an environment of the robot, the instance of the vision data being captured during performance of a robotic task by the robot; processing, using a robotic control policy, the instance of the vision data to generate a sequence of actions to be performed by the robot during the robotic task, the sequence of actions including an initial action to be performed by the robot in furtherance of the robotic task and a plurality of predicted actions that are predicted to follow the initial action; determining, based on processing the instance of the vision data using the robotic control policy, whether the robot will fail in performance of the robotic task; and in response to determining that the robot will fail in performance of the robotic task: causing a prompt to be rendered via an interface of a computing device or the robot, the prompt requesting a user of the computing device intervene in performance of the robotic task; receiving, from a user of the computing device, and based on the prompt, user input that intervenes with performance of the robotic task, the user input being received via an input device of the computing device or an additional computing device; and causing the robotic control policy to be updated based on the user input.

These and other implementations of the technology disclosed herein can include one or more of the following features.

In some implementations, the method can further include in response to determining that the robot will not fail in performance of the robotic action, causing the robot to perform the initial action. The method can further include, until the robot completes performance of the robotic task: receiving, from one or more of the vision components of the robot, an additional instance of vision data capturing the environment of the robot, the additional instance of the vision data being captured during performance of the robotic task by the robot; processing, using the robotic control policy, the additional instance of the vision data to generate an additional sequence of actions to be performed by the robot during the robotic task, the additional sequence of actions including a next action to be performed by the robot in furtherance of the robotic task and an additional plurality of predicted actions that are predicted to follow the next action; and determining, based on processing the additional instance of the vision data using the robotic control policy, whether the robot will fail in performance of the robotic task.

In some versions of those implementations, each action included in the sequence of actions can include a corresponding first set of values for a first component of the robot, and each action included in the sequence of actions can also include a corresponding second set of values for a second component of the robot.

In some further versions of those implementations causing the robot to perform the initial action can include causing the robot to utilize the corresponding first set of values to actuate the first component of the robot, and causing the robot to utilize the corresponding second set of values to actuate the second component of the robot. In even further versions of those implementations, the first component of the robot can be one of: a robot arm, a robot end effector, a robot base, or a robot head. In yet further versions of those implementations, the second component of the robot can be another one of: a robot arm, a robot end effector, a robot base, or a robot head.

In some additional or alternative further versions of those implementations, causing the robotic control policy to be updated based on the user input can be subsequent to determining that the robot has completed performance of the robotic task.

In some implementations, processing the instance of the vision data to generate the sequence of actions using the robotic control policy can include processing, using an intermediate portion of a robotic control policy, the instance of the vision data to generate an intermediate representation of the instance of the vision data; processing, using a first control head of the robotic control policy, the intermediate representation of the instance of the vision data to generate, for each action included the sequence of actions, a corresponding first set of values for a first component of the robot; and processing, using a second control head of the robotic control policy, the intermediate representation of the instance of the vision data to generate, for each action included the sequence of actions, a corresponding second set of values for a second component of the robot. In some versions of those implementations, the method can further include, in response to receiving the user input that intervenes with performance of the robotic task: generating, based on the user input, and for one or more actions included in the sequence of actions, a corresponding alternative first set of values, for the first component of the robot, and a corresponding alternative second set of values, for the second component of the robot, that the robot should utilize in performance of the robotic task; generating, based on comparing the corresponding first set of values to the corresponding alternative first set of values, a first loss; and generating, based on comparing the corresponding second set of values to the corresponding alternative second set of values, a second loss. Causing the robotic control policy to be updated can be based on the first loss and the second loss. In some further versions of those implementations, the first loss can be generated using a first loss function, and the second loss can be generated using a distinct second loss function.

In some implementations, processing the instance of the vision data to generate a sequence of actions to be performed by the robot during the robotic task can include processing, using an intermediate portion of the robotic control policy, the instance of the vision data to generate an intermediate representation of the instance of the vision data. The sequence of actions can be generated based on the intermediate representation of the instance of the vision data.

In some versions of those implementations, determining whether the robot will fail in performance of the robotic task can include processing, using a control head of the robotic control policy, the intermediate representation of the instance of the vision data to generate, for one or more actions included in the sequence of actions, one or more corresponding sets of values associated with performance of the robotic task; and determining that the robot will fail in performance of the robotic task based on the corresponding set of values.

In some further versions of those implementations, determining that the robot will fail in performance of the robotic task can be based on one or more of the corresponding set of values associated with the initial action. In additional or alternative versions of those further implementations determining that the robot will fail in performance of the robotic task can be based on one or more of the corresponding set of values associated with one or more of the plurality of predicted actions that follow the initial action. In additional or alternative versions of those further implementations, the corresponding set of values associated with performance of the robotic task can include a corresponding value associated with one or more of: whether the robot will fail in performance of the robotic task, whether the robot will continue in performance of the robotic task, or whether the robot has completed performance of the robotic task.

In some implementations, the robot can be a simulated robot, the environment of the robot can be a simulated environment of the simulated robot, and the robotic task can be performed by the simulated robot in the simulated environment.

In some implementations, the robot can be a real robot, the environment of the robot can be a real-world environment of the real robot, and the robotic task can be performed by the real robot in the real-world environment.

In some implementations, a method implemented by one or more processors is provided and includes receiving, from one or more vision components of a robot, an instance of vision data capturing an environment of the robot, the image data being captured during performance of a robotic task by the robot; processing, using a robotic control policy, the instance of the vision data to generate a sequence of actions to be performed by the robot during the robotic task, the sequence of actions including an initial action to be performed by the robot in furtherance of the robotic task and a plurality of predicted actions that follow the initial action; causing, during performance of the robotic task, a representation of the sequence of actions to be visually rendered via a graphical user interface of a computing device; receiving, from a user of the computing device, and based on the representation of the sequence of actions, user input that intervenes with performance of the robotic task, the user input being received via the computing device or an additional computing device; and causing the robotic control policy to be updated based on the user input.

These and other implementations of the technology disclosed herein can include one or more of the following features.

In some implementations, the representation of the sequence of actions visually rendered via the graphical user interface of the computing device can include a sequence of corresponding waypoints overlaying the environment of the robot captured in the instance of the vision data, each of the corresponding waypoints being associated with one or more components of the robot in response to a given action, included in the sequence of actions, being performed by the robot.

In some implementations, the representation of the sequence of actions visually rendered via the graphical user interface of the computing device can include a sequence of corresponding states of the robots overlaying the environment of the robot captured in the instance of the vision data, each of the corresponding states of the robots corresponding to a given state of the robot in response to a given action, included in the sequence of actions, being performed by the robot.

In some implementations, processing the instance of the vision data to generate the sequence of action using the robotic control policy can include processing, using an intermediate portion of a robotic control policy, the instance of the vision data to generate an intermediate representation of the instance of the vision data; processing, using a first control head of the robotic control policy, the intermediate representation of the instance of the vision data to generate, for each action included the sequence of actions, a corresponding first set of values for a first component of the robot; and processing, using a second control head of the robotic control policy, the intermediate representation of the instance of the vision data to generate, for each action included the sequence of actions, a corresponding second set of values for a second component of the robot.

In some versions of those implementations, the representation of the sequence of actions visually rendered via the graphical user interface of the computing device can include a corresponding representation of each action included in the sequence of actions. In some further versions of those implementations, the corresponding representation of each action included in the sequence of actions can be selectable and, when selected, causes the one or more of the corresponding first set of values for the first component or the corresponding second set of values for the second component to be visually rendered via the graphical user interface of the computing device.

In some versions of those implementations, the method can further include, in response to receiving the user input that intervenes with performance of the robotic task: generating, based on the user input, and for one or more actions included in the sequence of actions, a corresponding alternative first set of values, for the first component of the robot, and a corresponding alternative second set of values, for the second component of the robot, that the robot should utilize in performance of the robotic task; generating, based on comparing the corresponding first set of values to the corresponding alternative first set of values, a first loss; and generating, based on comparing the corresponding second set of values to the corresponding alternative second set of values, a second loss. Causing the robotic control policy to be updated can be based on the first loss and the second loss.

In some implementations, the method can further include receiving, from the user of the computing device, and subsequent to performance of the robotic task, additional user input associated with data generated during performance of the robotic task. Causing the robotic control policy to be updated can be further based on the additional user input. In some versions of those implementations, the additional user input can relabel data generated during performance of the robotic task, and the data generated during performance of the robotic task can be generated using the robotic control policy or can be generated based on the user input.

In some implementations, a method implemented by one or more processors is provided and includes receiving, from one or more vision components of a robot, an instance of vision data capturing an environment of the robot, the instance of the vision data being captured during performance of a robotic task by the robot; processing, using an intermediate portion of a robotic control policy, the instance of the vision data to generate an intermediate representation of the instance of the vision data; processing, using a first control head of the robotic control policy, the intermediate representation of the instance of the vision data to generate, for an action to be performed by the robot in furtherance of the robotic task, a corresponding first set of values for a first portion of control of a component of the robot; processing, using a second control head of the robotic control policy, the intermediate representation of the instance of the vision data to generate, for the action, a corresponding second set of values for a second portion of control of the component of the robot; receiving, from a user of a computing device, user input that intervenes with performance of the robotic task, the user input being received via the computing device or an additional computing device; and causing the robotic control policy to be updated based on the user input.

These and other implementations of the technology disclosed herein can include one or more of the following features.

In some implementations, causing the robotic control policy to be updated based on the user input can include generating, based on the user input, and for the action, a corresponding alternative first set of values and a corresponding alternative second set of values that the robot should utilize in performance of the robotic task; generating, based on comparing the corresponding first set of values to the corresponding alternative first set of values and using a first loss function, a first loss; and generating, based on comparing the corresponding second set of values to the corresponding alternative second set of values and using a second loss function, a second loss. Causing the robotic control policy to be updated can be based on the first loss and the second loss.

In some versions of those implementations, causing the robotic control policy to be updated based on the first loss and the second loss can include updating the first control head of the robotic control policy based on the first loss; and updating the second control head of the robotic control policy based on the second loss.

In some versions of those implementations, the method can further include processing, using a third control head of the robotic control policy, the intermediate representation of the instance of the vision data to generate, for the action, a corresponding third set of values for an additional component of the robot. In some versions of those implementations, causing the robotic control policy to be updated based on the user input further can include generating, based on the user input, and for the action, a corresponding alternative third set of values; and generating, based on comparing the corresponding third set of values to the corresponding alternative third set of values, a third loss. Causing the robotic control policy to be updated can be further based on the third loss. In some further versions of those implementations, causing the robotic control policy to be updated based on the third loss can include updating the third control head of the robotic control policy based on the third loss.

In some implementations, the first component of the robot can be one of: a robot arm, a robot end effector, a robot base, or a robot head. In some versions of those implementations, the first portion of control can be a positional change for the first component and the second portion of control can be an orientation change for the first component.

In some implementations, a method implemented by one or more processors is provided and includes receiving, from one or more vision components of a robot, an instance of vision data capturing an environment of the robot, the instance of the vision data being captured during performance of a robotic task by the robot; processing, using an intermediate portion of a robotic control policy, the instance of the vision data to generate an intermediate representation of the instance of the vision data; processing, using a first control head of the robotic control policy, the intermediate representation of the instance of the vision data to generate, for an action to be performed by the robot in furtherance of the robotic task, a corresponding first set of values for a first portion of control of a component of the robot; processing, using a second control head of the robotic control policy, the intermediate representation of the instance of the vision data to generate, for the action, a corresponding second set of values for a second portion of control of the component of the robot; and causing the robot to perform the action. Causing the robot to perform the action can include causing the robot to utilize the corresponding first set of values and the corresponding second set of values in controlling the component of the robot.

These and other implementations of the technology disclosed herein can include one or more of the following features.

In some implementations, the method can further include processing, using a third control head of the robotic control policy, the intermediate representation of the instance of the vision data to generate, for the action, a corresponding third set of values for at least a portion of control of an additional component of the robot. Causing the robot to perform the action further can include causing the robot to utilize the corresponding third set of values to control the third component of the robot.

Various implementations can include a non-transitory computer readable storage medium storing instructions executable by one or more processor(s) (e.g., a central processing unit(s) (CPU(s)), graphics processing unit(s) (GPU(s)), and/or tensor processing unit(s) (TPU(s))) to perform a method such as one or more of the methods described herein. Yet other implementations can include a system of one or more computers and/or one or more robots that include one or more processors operable to execute stored instructions to perform a method such as one or more of the methods described herein.