Patent Description:
<CIT> discloses a multi-layer learning based control system and method for an autonomous vehicle or mobile robot. A mission planning layer, behavior planning layer and motion planning layer each having one or more neural networks are used to develop an optimal route for the autonomous vehicle or mobile robot, provide a series of functional tasks associated with at least one or more of the neural networks to follow the planned optimal route, and develop commands to implement the functional tasks.

The matter for protection is set out in the appended claims. Implementations disclosed herein utilize a trained action sequence prediction model in determining a predicted sequence of actions for a robotic task based on an instance of vision data captured by a robot. In many implementations, the sequence of predicted actions can be conditioned on the environment of the robot where the robot can encounter object(s) in the environment in different initial positions when completing the task. For example, a robot can encounter an open door when completing a task for the first time, and can encounter a closed door when completing the same task for a second time.

Performing a robotic task requires a robot to respond to changes in the environment. As an illustrative example, an object manipulation task can include the goal of having a cup containing a ball inside a closed cabinet. When performing the task, the robot can encounter different environment states such as different locations of the cup, different locations of the ball, and/or different states of the cabinet door (i.e., the cabinet door being open or the cabinet door being closed). Potential actions required to complete the task can include: opening the cabinet door, closing the cabinet door, moving the cup, moving the ball, moving the ball into the cup, and moving the ball with the cup.

When performing the task for a first time, the robot can encounter a closed cabinet with the ball and the cup sitting outside of the cabinet. In many implementations, the action sequence prediction model can be used in determining a predicted sequence of actions of (<NUM>) moving the ball into the cup, (<NUM>) opening a cabinet door, (<NUM>) moving the ball with the cup into the cabinet, and (<NUM>) closing the cabinet door. In the illustrated example, some combinations of actions have an inherent sequential order, such as opening the cabinet door before moving the ball with the cup into the cabinet, moving the ball with the cup into the cabinet before closing the cabinet door, etc. Additionally or alternatively, some combinations of actions can be performed in any order. For example, there is no inherent order between placing the ball into the cup and opening the cabinet door. In other words, the robot can complete the task by placing the ball into the cup before or after opening the cabinet door.

As a further example, when performing the task for a second time, the robot can encounter the ball inside a closed cabinet. The action sequence prediction model be used in determining a predicted sequence of actions of (<NUM>) opening the cabinet door, (<NUM>) moving the cup, (<NUM>) moving the ball, (<NUM>) moving the ball to the cup, (<NUM>), moving the ball with the cup into the cabinet, and (<NUM>) closing the cabinet door. As described above, some sequence(s) of actions have an inherent order while other sequence(s) of actions do not have an inherent order. For example, the cabinet door should be opened before the cup and the ball can be moved. Similarly, the ball with the cup should be moved into the cabinet before the cabinet door should be closed. However, there is no inherent order in moving the cup and moving the ball outside of the cabinet and these actions can be performed in either order to complete the task.

The environment-conditioned action sequence prediction model can include a convolutional neural network model ("CNN") portion as well as a sequence-to-sequence model portion. In a variety of implementations, an instance of vision data captured using sensor(s) of the robot, such as an image captured using a camera of the robot, can be processed using the CNN portion to determine an embedding corresponding to the instance of vision data. The embedding can be processed using the sequence-to-sequence model portion (such as an encoder-decoder model, a transformer model, etc.) to determine the predicted sequence of actions to complete the robotic task.

The predicted sequence of actions can change depending on the instance of vision data captured by the robot. For example, as described above, a robot can encounter a closed cabinet with the ball and the cup sitting outside of the cabinet. The action sequence prediction model can be used to determine a predicted sequence of actions of: (<NUM>) moving the ball into the cup, (<NUM>) opening a cabinet door, (<NUM>) moving the ball with the cup into the cabinet, and (<NUM>) closing the cabinet door. Additionally or alternatively, the robot can encounter the cup and the ball inside a closed cabinet, where the action sequence prediction model can be used in determining a predicted sequence of actions of (<NUM>) opening the cabinet door, (<NUM>) moving the cup, (<NUM>) moving the ball, (<NUM>) moving the ball to the cup, (<NUM>), moving the ball with the cup into the cabinet, and (<NUM>) closing the cabinet door. In many implementations, the sequence of actions determined using the action sequence prediction model can contain different actions depending on actions necessary to complete the robotic task. For example, when the robot encounters a closed cabinet with the ball and the cup sitting outside of the cabinet, the corresponding predicted sequence of actions does not include moving the ball outside of the cabinet and/or moving the cup outside of the cabinet because the actions are unneeded. Similarly, when the robot encounters the cup and the ball inside a closed cabinet, the corresponding predicted sequence of actions will include the actions of moving the ball and moving the cup.

Each action in the predicted sequence can be used to determine a corresponding action network, where output generated by processing additional robotic sensor data using the corresponding action network can be used in controlling the robot. In many implementations, the action network corresponding to each action can be a distinct network, such as a distinct policy network. In some such implementations, each policy network can be trained using reinforcement learning based on kinesthetic demonstrations such as demonstrations performed by teleoperation of the robot.

The above description is provided only as an overview of some implementations disclosed herein. These and other implementations of the technology are disclosed in additional detail below.

Complex object manipulation tasks may span over long sequences of operations. Task planning over long-time horizons can be challenging, and its complexity can grow exponentially with an increasing number of subtasks. Implementations disclosed herein are directed towards a deep learning network that learns dependencies and transitions across subtasks solely from a set of demonstration videos. Each subtask can be represented as an action symbol (e.g. move cup), and these symbols can be learned and predicted directly from image observations. Learning from demonstrations may make the learning tractable as it provides the network with information about the most frequent transitions and relevant dependency between subtasks (instead of exploring all possible combination), while learning from visual observations may cause the network to continuously monitor the task progress and thus to interactively adapt to changes in the environment. Some implementations may be evaluated using two long horizon tasks: (<NUM>) block stacking of puzzle pieces being executed by humans, and (<NUM>) a robot manipulation task involving pick and place of objects and sliding a cabinet door on a <NUM>-DoF robot arm. For some implementations, it can be shown that complex plans can be carried out when executing the robotic task and the robot can interactively adapt to changes in the environment and recover from failure cases.

Some implementations described herein may utilize a two layer representation of complex tasks, where a set of abstract actions or sub-tasks can be used as an intermediate representation of a set of abstract actions or sub-tasks. Each action can be represented by a symbol that describes what needs to happen to complete a sub-task in an abstract manner (e.g. move cup). This discretization allows us to reason about the structure of tasks without being faced with the intricacies of real environments and the related physics (e.g. object pose).

Each symbol can be used to select an individual policy that describes how an object or the agent itself needs to be manipulated toward the higher-level goal. In some implementations, when executing an action, the complexity imposed by the real scene, such as finding an object or identifying its pose to grasp it, can be considered. The goal is to execute complex and long-horizon tasks by learning the sequential dependencies between task-relevant actions. To learn sequences of sub-tasks while respecting changes in the scene, some implementations employ a sequence-to-sequence model, commonly used for natural language processing, to translate sequences of image embeddings to action symbols.

The capabilities of sequence prediction may be tested by evaluating some implementations on two environments. First, a robot arm can be used to manipulate objects in an office environment, where the goal is to find objects in a cabinet, perform operations on the objects, and to move them back into the cabinet. For example, the task can be to find a cup, to put a ball in the cup, and to move both objects together back to the cabinet. Different sequences of sub-tasks can lead to a successful completion of the task. For example, while the robot has to first open the cabinet door, it can then either move the cup or the ball outside the cabinet, to eventually put the ball in the cup and both objects back into the cabinet. Second, a robot can perform a stacking task that requires to move blocks from an initially random configuration to three stacks of blocks.

Unlike existing work that mostly focuses on learning action symbols implicitly - e.g. as latent variables - implementations described herein can represent actions explicitly, which in turn may provide more semantics of a task. Additionally or alternatively, the action symbols can be learned directly from sequences of images. This may facilitate inferring the correct order of actions necessary to complete a task, while also allowing a response to changes in the environment. Each individual action is then executed with an individual policy.

In some implementations, the sequential structure of tasks can be learned by factorizing them into task-relevant actions. This is motivated by the observation that many tasks are as well combinatorial as they are continuous. They are combinatorial in that an agent has to select among a discrete set of objects to perform a task. For example, a stacking task requires to arrange a number of objects. However, an agent has to also operate in a physical environment that requires interacting with objects in continuous ways.

Optimizing for both of the aforementioned factors to perform long-term planning may be challenging due to the uncertainty imposed by the actual scene. Therefore, to perform long-term planning, long-horizon tasks can be factorized into a discrete set of actions. These actions represent what needs to happen to complete a sub-task, but at a very high-level of abstraction and without any notion of how an agent has to perform the action. For example, an action might just be 'move cup'. Second, once a task is structured into task-relevant actions expert policies can be obtained from learned demonstrations to perform individual actions.

A set of action symbols can be utilized as an abstract representation of sub-tasks. These symbols represent basic actions, such as 'open door', 'move cup', 'put ball', etc.,. In some implementations, the set of action symbols may be manually defined for different tasks. Additionally or alternatively, the set of action symbols may be automatically defined for different tasks (e.g., a trained machine learning model can be utilized to generate the set of action symbols). Sequences of symbols can provide an abstraction of the task that can be learned to be predicted and then executed on a robot. The set of symbols is denoted as K.

Action symbols can be used in two ways: first, a single frame action classifier can be trained, that allows the generation of embeddings of images. Second, an encoder-decoder sequence-to-sequence model can be trained to translate sequences of image embeddings to sequences of action symbols. Together, both models can be used to predict the next action based on the current state of the scene as well as according to which sub-tasks were already completed.

In some implementations, to obtain a representation of the scene as well as of ongoing actions, a convolutional neural network can be trained as an action recognition model. For example, the CNN can include a ResNet50 backbone with one extra dense layer (<NUM> dimensions) to extract image features and another dense layer followed by a Softmax to fine tune the network on action symbols as labels. This model may be trained as a single image action predictor on images of sequences, where each image is labeled with an action symbol. Action recognition based on a single frame may be a challenging problem, as an action shown in a single image can be ambiguous; e.g. reaching toward a cup looks the same as moving away from it. However, our goal is not to use the resulting classification of this model, but instead to use the resulting embedding as input to our sequence-to-sequence model. The sequence-to-sequence model can then translate the produced embeddings to action symbols. Furthermore, as the sequence-to-sequence model maintains an internal state it can resolve ambiguities introduced by wrongly predicted action symbols of the action classifier.

In some implementations, sequence models can be used to predict future action symbols given a history of image embeddings. Given a sequence of image embeddings (E<NUM>,. , ET) up to current time t, the next k action symbols can be predicted (at+<NUM>,. , at+k):
<MAT>.

In some implementations, the above formulation may be cast as a 'translation' of image embeddings to action symbol sequence. Therefore, a sequence-to-sequence model (i.e., a neural translation formulation) may be utilized where the embedding sequence may be mapped to an action sequence. In some implementations, the sequence-to-sequence model may consist of an encoder and decoder LSTM. The encoder can consume the input image as a sequence of embeddings and can encode it into a single vector, which may be subsequently decoded into an action symbol sequence by a second LSTM.

In some implementations, learning the sequential structure of tasks based on image embeddings and action symbols may facilitate the performance of tasks in varying combinations of sub-tasks and depending on a given scene configuration. For example, the stacking task shown requires stacking colored blocks in a specific configuration. Two blocks (red, yellow) need to be in place before other blocks (pink, green) can be stacked on top of them. Given this task description, the task can be performed in different orders. For example, the blue block can be put up independently of the other blocks, while the green and pink blocks depend on the red and yellow blocks.

In some implementations, action symbols can be modeled as motion primitives. A motion primitive is a parameterized policy to perform an atomic action, such as grasping, placing, etc. Primitives can be used as building blocks that can be composed, for example by a state machine, to enable more advanced robot behavior. For example, the task of putting an object into a cabinet may have motion primitives of: grasping, opening/closing cabinet, and placing. The state machine may be used for sequencing the primitives based on the world state. Initially it may trigger the cabinet opening primitive. Upon its success, it may switch to the grasping primitive and may condition it on the particular object that needs to be grasped. Then it may proceed with the placing primitive, followed by the closing cabinet primitive. In case of a failure, the state machine may switch the primitive to recover from the error. Note that the use of state-machine may implicitly require access to a success detection module in order to properly transit from one primitive to another.

In some implementations, the state machine may be used together with motion primitives. The symbol prediction network may replace the state machine and success detection module. Each of the action symbols may correspond to a motion primitive, hence we have separate primitives to grasp a cup, grasp a ball, move a cup, move a ball, slide the door, and so on. Note that without loss of generality, some implementations may utilize different grasping/moving primitives for each object to simplify the run-time execution. Alternatively, in some other implementations, all grasping primitives could be unified to one grasping policy for multiple objects, e.g. cup and ball.

In some implementations, each of the motion primitive can be modeled as a dynamical systems policy (DSP), which can be trained from a few demonstrations. Given a target pose, i.e. the object pose, DSP drives the robot arm from its initial pose to the target pose while exhibiting a similar behavior as the demonstrations. In some implementations, each of the primitives can be trained based on five demonstrations captured through Kinesthetic demonstrations. The input to each DSP primitive may be the current object and arm end-effector pose, and the output may be the next end-effector pose. In some implementations, the robot is equipped with a perception system that performs object detection and classification and can provide the Cartesian pose of each object with respect to the robot frame, which may be passed to DSP primitives. DSP representation may allow for quickly modeling each primitive with a couple of demonstrations, however, at this may be at the cost of depending on a perception system. Additional and/or alternative method(s) can be used in place of DSP such as using an end-to-end deep network policy to represent each primitive to avoid this dependency.

In some implementations, once the sequential model determines the next action, the corresponding primitive may be called with the poses of relevant objects and the robot may start executing the motion. Note that in some implementations there are two loops: <NUM>) the DSP control loop which, for example, runs at <NUM> and is in charge of moving the arm to the target location, and <NUM>) the symbolic switching loop which, for example, runs at <NUM> and determines the next primitive that needs to be executed solely based on the stream of images.

In some implementations, the action classifier may be trained on single pairs of images and action symbols and these pairs can be randomly (or pseudo randomly) selected from all sequences of the training data. Furthermore, the action classification model may be trained separately for each dataset until it converges. In some implementations, the action classification model converges within <NUM> epochs.

In some implementations, the sequence-to-sequence network may be trained on sequences of image embeddings and action symbols. Instead of training on the full sequences, the network can be trained on sub-sequences of a specified sequence length (SL). For example, the sequence to sequence network may be trained using a sub-sequence of lengths <NUM>, <NUM>, <NUM>, and/or additional or alternative lengths. The sub-sequences can be generated as 'sliding windows' over an entire sequence. In some implementations, the sequence to sequence model can be trained so as to translate sequences of image embeddings to predict a sequence of action symbols. However, the sequence of predicted action symbols may be offset by k, where k represents the number of steps we want to predict in the future. For example, the number of steps can be set to k = <NUM>, which means that an action one step ahead in the future can be predicted. In some implementations, the number of steps can be set to k = N number of steps which means that an action N steps ahead in the future can be predicted (e.g.,k = <NUM> where an action one step ahead in the future can be predicted; k = <NUM> where an action two steps ahead in the future can be predicted; k = <NUM> where an action three steps ahead in the future can be predicted, etc.).

In some implementations, the encoder may take the input frame embeddings and may generate a state embedding vector from its last recurrent layer, which encodes the information of all input elements. The decoder can then take this state embedding and convert it back to action symbol sequences. In some implementations, both networks can be trained individually for each task. The sequence-to-sequence model can be trained with a latent dimension of <NUM> and usually converges after <NUM> epochs. In some implementations, the hyperparameters of either model may not be specifically fine-tuned. Additionally or alternatively, in some implementations, both networks can be trained simultaneously for each task.

Turning to the figures, <FIG> illustrates an example of generating a set of predicted actions using an environment-conditioned action sequence prediction model, and using the set of predicted actions in controlling a robot. In the illustrated example, an instance of sensor data <NUM> can be processed using an environment-conditioned action sequence prediction model <NUM> to generate a set of predicted actions <NUM> and a corresponding order to perform the actions in the set. The instance of sensor data <NUM> can be captured using one or more sensors of a robot. For example, the instance of sensor data <NUM> can be captured using a variety of sensor(s) of the robot such as vision sensor(s), light sensor(s), pressure sensor(s), pressure wave sensor(s) (e.g., microphones), proximity sensor(s), accelerometer(s), gyroscope(s), thermometer(s), barometer(s), and so forth. In a variety of implementations, the instance of sensor data <NUM> can include an instance of vision data captured using a camera of the robot. Environment-conditioned action sequence prediction model <NUM> can include a convolutional neural network model portion <NUM>, an encoder portion <NUM>, a decoder portion <NUM>, and/or additional or alternative machine learning model portion(s) (not depicted). In many implementations, sensor data <NUM> can be processed using convolutional neural network model portion <NUM> to generate an embedding corresponding to the instance of sensor data. Additionally or alternatively, encoder <NUM> and decoder <NUM> can be used in processing the embedding to generate the set of predicted actions <NUM>.

The set of predicted actions <NUM> can include action <NUM>, action <NUM><NUM>,. , action N <NUM>. In many implementations, the set of actions has a corresponding particular order such as performing action <NUM>, performing action <NUM><NUM>,. , performing action N <NUM>. Each predicted action can have a corresponding action network in the action networks <NUM>. For example, action network <NUM> can correspond to action <NUM>, action network <NUM><NUM> can correspond to action <NUM><NUM>,. , action network N <NUM> can correspond to action N <NUM>. In a variety of implementations, a selected action network can be used in processing one or more instances of additional sensor data <NUM> for use in robotic control <NUM> to cause the robot to perform action <NUM>. Once action <NUM> is complete, action network <NUM><NUM> can be used in processing one or more additional instances of additional sensor data <NUM> for use in robotic control <NUM> to cause the robot to perform action <NUM>. Additionally or alternatively, once action N-<NUM> is complete, action network N <NUM> can be used in processing one or more additional instances of additional sensor data <NUM> for use in robotic control <NUM> to cause the robot to perform action N.

<FIG> illustrates an example object manipulation robotic task. Example <NUM> illustrates an object manipulation task with a goal of positioning blocks in desired locations. Block <NUM> is a circle placed in desired position <NUM>. Block <NUM> is a triangle placed in desired position <NUM> stacked on top of block <NUM>. Block <NUM> is a square placed in desired position <NUM>. The placement of blocks <NUM> and <NUM> have an inherent sequential order, where block <NUM> must be placed in position <NUM> prior to the placement of block <NUM> in position <NUM> on top of block <NUM>. Additionally or alternatively, the placement of block <NUM> does not have an inherent sequential order as block <NUM> can be placed before block <NUM>, between the placement of block <NUM> and block <NUM>, and/or after the placement of block <NUM>. In other words, a variety of sequences of actions can be utilized for block placement in the illustrated example including: (<NUM>) place block <NUM>, place block <NUM>, place block <NUM>; (<NUM>) place block <NUM>, place block <NUM>, place block <NUM>, and (<NUM>) place block <NUM>, place block <NUM>, place block <NUM>.

<FIG> illustrates an instance of vision data <NUM> capturing the initial poses of objects <NUM>, <NUM>, and <NUM>. Object <NUM> is initially at position <NUM>, object <NUM> is initially at position <NUM>, and object <NUM> is initially at position <NUM>. An environment-conditioned action sequence prediction model in accordance with many implementations can be utilized to determine a set of predicted actions as well as a corresponding order to place object <NUM> at position <NUM>, object <NUM> at position <NUM>, and object <NUM> at position <NUM>. For example, the environment-conditioned action sequence prediction model can determine the set of predicted actions with a corresponding particular order of (<NUM>) place block <NUM>, place block <NUM>, place block <NUM>; (<NUM>) place block <NUM>, place block <NUM>, place block <NUM>, and/or (<NUM>) place block <NUM>, place block <NUM>, place block <NUM>. In the illustrated example, the desired goal positions <NUM>, <NUM>, and <NUM> are depicted by dash lines.

<FIG> illustrates another instance of vision data <NUM> capturing the additional initial poses of objects <NUM>, <NUM>, and <NUM>. Object <NUM> is initially in position <NUM>, object <NUM> is initially in position <NUM>, and object <NUM> is initially in position <NUM>. The initial positioning of object <NUM> is such that it is already positioned in the desired goal position <NUM>. In other words, the robot does not need to move object <NUM> to complete the task. The environment-conditioned action sequence prediction model can be utilized to determine the set of predicted actions with a corresponding particular order of (<NUM>) place block <NUM>, place block <NUM>; and/or (<NUM>) place block <NUM>, place block <NUM>.

<FIG> illustrates an example environment in which implementations described herein may be implemented. <FIG> includes an example robot <NUM>, an action sequence prediction system <NUM>, a prediction engine <NUM>, an action network engine <NUM>, a prediction model training engine <NUM>, an action network training engine <NUM>, and/or additional or alternative engine(s) (not depicted). Also included is prediction model <NUM>, prediction training examples <NUM>, action models <NUM>, and action training examples <NUM>.

Robot <NUM> is a "robot arm" having multiple degrees of freedom to enable traversal of grasping end effectors along any of a plurality of potential paths to position the grasping end effector in desired locations. Robot <NUM> further controls two opposed "claws" of the grasping end effector to actuate the claws between at least an open position and a closed position (and/or optionally a plurality of "partially closed" positions). Furthermore, robot <NUM> can include a variety of sensors that can generate images related to shape, color, depth, and/or other features of object(s) that are in the line of sight of the sensors. The vision sensors may be, for example, monographic cameras, stereographic cameras, and/or 3D laser scanners. A 3D laser scanner may be, for example, a time-of-flight 3D laser scanner or a triangulation based 3D laser scanner and may include a position sensitive detector (PDS) or other optical position sensor. Robot <NUM> can include additional and/or alternative sensors.

Although a particular robot <NUM> is described with respect to <FIG>, additional and/or alternative robots may be utilized, including additional robot arms that are similar to robot <NUM>, robots having other robot arm forms, robots having a humanoid form, robots having an animal form, robots that move via one or more wheels (e.g., self-balancing robots), submersible vehicle robots, an unmanned aerial vehicle ("UAV"), and so forth. Additional and/or alternative end effects may be utilized, such as alternative impactive grasping end effectors (e.g., those with grasping "plates", those with more or fewer "digits"/"claws"), ingressive grasping end effectors, astrictive grasping end effectors, contigutive grasping end effectors, or non-grasping end effectors.

Action sequence prediction system <NUM> can be utilized by robot <NUM> to generate a set of predicted actions and a corresponding particular order. In many implementations, action sequence prediction system <NUM> can include prediction engine <NUM> as well as action network engine <NUM>. In many implementations, prediction engine <NUM> can process an instance of sensor data such as an instance of vision data using environment-conditioned action sequence prediction model <NUM> to generate a set of predicted actions as well as a corresponding particular order to perform the predicted actions. Prediction model training engine <NUM> can train prediction model <NUM> using prediction training examples <NUM>. Training an environment-conditioned action sequence prediction model such as prediction model <NUM> in accordance with a variety of implementations is described with respect to process <NUM> of <FIG> and/or process <NUM> of <FIG>.

Action network engine <NUM> can process the set of predicted actions determined using prediction engine <NUM> to determine one or more corresponding action models of actions models <NUM>. In many implementations, each predicted action has a distinct corresponding action model. Additionally or alternatively, action network engine <NUM> can process additional instance(s) of sensor data to perform each action in the set of predicted actions by processing the additional instance(s) of sensor data using the corresponding action model until the action is complete and the set of actions are complete. Action network training engine <NUM> can train action models <NUM> based on action training examples. In a variety of implementations, action training examples <NUM> can include a kinesthetic demonstration of an operator performing each action using the robot. In a variety of implementations, each action model can be a policy network which is trained by action network training engine <NUM> using reinforcement learning. Training one or more action networks such as action models <NUM> in accordance with many implementations is described herein with respect to process <NUM> of <FIG>.

<FIG> is a flowchart illustrating an example process <NUM> of controlling a robot based on a set of predicted actions according to implementations disclosed herein. For convenience, the operations of process <NUM> are described with reference to a system that performs the operations. This system may include various components of various computer systems, such as one or more components depicted in <FIG> and/or <FIG>. Moreover, while operations of process <NUM> are shown in a particular order, this is not meant to be limiting. One or more operations by be reordered, omitted, and/or added.

At block <NUM>, the system processes an instance of sensor data (e.g., sensor data <NUM> of <FIG>) using an environment-conditioned action sequence prediction model to determine a set of predicted actions and a corresponding particular order to perform the actions for a robotic task. The instance of sensor data can include an instance of vision data captured using one or more vision sensors of the robot. For example, the instance of vision data can include an image captured using a camera of the robot. However, this is merely illustrative and additional and/or alternative instances of sensor data may be utilized in accordance with some implementations described herein.

In many implementations, the environment-conditioned action sequence prediction model can include a convolutional neural network portion, a sequence-to-sequence network portion, and/or additional or alternative portion(s). In some such implementations, the instance of vision data can be processed using the convolutional neural network model portion to determine an embedding corresponding to the instance of vision data. The embedding can be processed using the sequence-to-sequence model portion (e.g., an encoder-decoder model portion) to determine the set of predicted actions and the corresponding order to perform the predicted actions for the robotic task. Environment-conditioned action sequence prediction model(s) in accordance with many implementations can be utilized with a variety of robotic tasks including object manipulation tasks, navigation tasks, and/or additional robotic tasks. Process <NUM> of <FIG> is described with respect to convolutional neural network portion and a sequence-to-sequence network portion. However, this is merely illustrative and additional and/or alternative network(s) may be utilized. For example, any of a variety of networks may be utilized including recurrent neural network portion(s), transformer model portion(s), and/or additional or alternative neural network portion(s).

In many implementations, the set of predicted actions is conditioned on the environment of the robot (e.g., conditioned on the instance of sensor data). For example, in the object manipulation object described with respect to <FIG>, the environment-conditioned action sequence prediction model can be used in determining different sets of predicted actions based on different initial instances of sensor data.

At block <NUM>, the system selects, in the particular order, an unperformed action in the set of predicted actions.

At block <NUM>, the system selects an action network corresponding to the selected action. In many implementations, a distinct action prediction network corresponds with each predicted action. For example, the action prediction networks can each be policy networks trained using dynamical system policies. In some implementations, the action network corresponding to the selected action can be selected using action network engine <NUM> from action models <NUM> as described in <FIG>.

At block <NUM>, the system processes an additional instance of sensor data using the selected action network to generate output. For example, the additional instance of sensor data can include a current pose of each object with respect to the robot.

At block <NUM>, the system controls a robot based on the generated output. For example, when the robot is performing a navigation task the generated output can indicate a trajectory for the robot. As another example, the generated output can indicate motor command(s) to move an end effector to grasp an object.

At block <NUM>, the system determines whether the robot has finished performing the selected action. If no, the system proceeds back to block <NUM> to process an additional instance of sensor data using the selected action network to generate additional output and proceeds to block <NUM> to control the robot based on the additional generated output. If the robot has finished performing the selected action, the system proceeds to block <NUM>.

At block <NUM>, the system determines whether there are any unperformed actions in the set of predicted actions. If yes, the system proceeds to block <NUM> and selects, in the particular order, the next unperformed action in the set of predicted actions before proceeding to blocks <NUM>, <NUM>, and <NUM> using the next unperformed action. If no, the process ends.

<FIG> is a flowchart illustrating an example process <NUM> of training a convolutional neural network model portion of an environment-conditioned action sequence prediction model according to implementations disclosed herein. For convenience, the operations of process <NUM> are described with reference to a system that performs the operations. This system may include various components of various computer systems, such as one or more components depicted in <FIG> and/or <FIG>. Moreover, while operations of process <NUM> are shown in a particular order, this is not meant to be limiting. One or more operations by be reordered, omitted, and/or added.

At block <NUM>, the system selects a training sequence of robotic task vision data where each action in the training sequence has a corresponding label. For example, a training sequence can capture a robot performing the object manipulation task described above with respect to <FIG> where corresponding actions in the training sequence have labels of: move block <NUM>, move block <NUM>, move block <NUM>. In many implementations, the training sequence of robotic task vision data can be selected from prediction training examples <NUM> of <FIG>.

At block <NUM>, the system selects an instance of vision data and a corresponding label of an action in the training sequence selected at block <NUM>. For example, the system can select an instance of vision data in the training sequence capturing the robot moving block <NUM>, and can select a corresponding action label of "moving block <NUM>".

At block <NUM>, the system processes the selected instance of vision data using a convolutional neural network model portion of an environment-conditioned action sequence prediction model to generate predicted output. For example, the system can process the selected instance of vision data using CNN <NUM> of action sequence prediction model <NUM> as illustrated in <FIG>.

At block <NUM>, the system determines a difference between the predicted output and the corresponding action label.

At block <NUM>, the system updates one or more portions of the convolutional neural network model portion based on the difference determined at block <NUM>. For example, the system can update one or more weights of the convolutional neural network model using backpropagation.

At block <NUM>, the system determines whether to process any additional actions in the training sequence. If yes, the system proceeds back to block <NUM> and selects an additional instance of vision data and an additional corresponding action label before proceeding to blocks <NUM>, <NUM>, and <NUM> using the additional instance of vision data and the additional corresponding action label. If no, the system proceeds to block <NUM>.

At block <NUM>, the system determines whether to process any additional training sequences of the robotic task. If yes, the system proceeds back to block <NUM> and selects an additional training sequence of robotic vision data before proceeding to blocks <NUM>, <NUM>, <NUM>, <NUM>, and <NUM> using the additional training sequence. If not, the system ends.

<FIG> is a flowchart illustrating an example process <NUM> of training a sequence-to-sequence model portion of an environment-conditioned action sequence prediction model according to implementations disclosed herein. For convenience, the operations of process <NUM> are described with reference to a system that performs the operations. This system may include various components of various computer systems, such as one or more components depicted in <FIG> and/or <FIG>. Moreover, while operations of process <NUM> are shown in a particular order, this is not meant to be limiting. One or more operations by be reordered, omitted, and/or added.

At block <NUM>, the system selects a training sequence of robotic task vision data, where each action in the sequence has a corresponding label. In many implementations, the training sequence can be selected from prediction training examples <NUM> of <FIG>.

At block <NUM>, the system processes the selected training sequence using a convolutional neural network model portion of the environment-conditioned action sequence prediction model to generate an output sequence of embeddings. In some implementations, the convolutional neural network model portion of the environment-conditioned action sequence prediction model is trained in accordance with process <NUM> of <FIG>.

At block <NUM>, the system selects a generated embedding of an action and a corresponding label from the training sequence for the action.

At block <NUM>, the system processes the selected embedding using a sequence-to-sequence model portion of the environment-conditioned action sequence prediction model to generate prediction action output.

At block <NUM>, the system determines a difference between the predicted action output and the corresponding action label.

At block <NUM>, the system updates one or more portions of the sequence-to-sequence model based on the difference determined at block <NUM> (e.g., update through backpropagation).

At block <NUM>, the system determines whether to process any additional embeddings of the action generated using the convolutional neural network. If yes, the system proceeds back to block <NUM> and selects an additional embedding of the action before proceeding to blocks <NUM>, <NUM>, and <NUM> using the additional action. If no, the system proceeds to block <NUM>.

At block <NUM>, the system determines whether to select any additional training sequences of robotic task vision data. If yes, the system proceeds back to block <NUM> and selects an additional training sequence before proceeding to blocks <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, and <NUM> using the additional training sequence. If not, the system ends.

<FIG> is a flowchart illustrating an example process <NUM> of training one or more action networks according to implementations disclosed herein. For convenience, the operations of process <NUM> are described with reference to a system that performs the operations. This system may include various components of various computer systems, such as one or more components depicted in <FIG> and/or <FIG>. Moreover, while operations of process <NUM> are shown in a particular order, this is not meant to be limiting. One or more operations by be reordered, omitted, and/or added.

At block <NUM>, the system selects an action of a robotic task.

At block <NUM>, the system selects a sequence of action training data corresponding to the action. In many implementations, the action training data is selected from action training examples <NUM> depicted in <FIG>.

At block <NUM>, the system generates updated policy parameters for an action model corresponding to the action using the selected sequence of action training data.

At block <NUM>, the system updates the action model using the updated policy parameters.

At block <NUM>, the system determines whether to process an additional training sequence for the robotic action. If yes, the system proceeds back to block <NUM> and selects an additional sequence of training data corresponding to the action before proceeding to blocks <NUM>, and <NUM> using the additional sequence of training data. If no, the system proceeds to block <NUM>.

At block <NUM>, the system determines whether to train an additional action model corresponding to an additional action. If yes, the system proceeds back to block <NUM> and selects an additional action of the robotic task before proceeding to blocks <NUM>, <NUM>, <NUM>, and <NUM> using the additional action. If no, the system ends.

<FIG> schematically depicts an example architecture of a robot <NUM>. The robot <NUM> includes a robot control system <NUM>, one or more operational components 825a-825n, and one or more sensors 842a-<NUM>. The sensors 842a-<NUM> may include, for example, vision sensors, light sensors, pressure sensors, pressure wave sensors (e.g., microphones), proximity sensors, accelerometers, gyroscopes, thermometers, barometers, and so forth. While sensors 842a-m are depicted as being integral with robot <NUM>, this is not meant to be limiting. In some implementations, sensors 842a-m may be located external to robot <NUM>, e.g., as standalone units.

Operational components 840a-840n may include, for example, one or more end effectors and/or one or more servo motors or other actuators to effectuate movement of one or more components of the robot. For example, the robot <NUM> may have multiple degrees of freedom and each of the actuators may control actuation of the robot <NUM> within one or more of the degrees of freedom responsive to the control commands. As used herein, the term actuator encompasses a mechanical or electrical device that creates motion (e.g., a motor), in addition to any driver(s) that may be associated with the actuator and that translate received control commands into one or more signals for driving the actuator. Accordingly, providing a control command to an actuator may comprise providing the control command to a driver that translates the control command into appropriate signals for driving an electrical or mechanical device to create desired motion.

The robot control system <NUM> may be implemented in one or more processors, such as a CPU, GPU, and/or other controller(s) of the robot <NUM>. In some implementations, the robot <NUM> may comprise a "brain box" that may include all or aspects of the control system <NUM>. For example, the brain box may provide real time bursts of data to the operational components 840a-n, with each of the real time bursts comprising a set of one or more control commands that dictate, inter alia, the parameters of motion (if any) for each of one or more of the operational components 840a-n. In some implementations, the robot control system <NUM> may perform one or more aspects of processes <NUM>, <NUM>, <NUM>, and/or <NUM> described herein. As described herein, in some implementations all or aspects of the control commands generated by control system <NUM> can position limb(s) of robot <NUM> for robotic locomotion tasks. Although control system <NUM> is illustrated in <FIG> as an integral part of robot <NUM>, in some implementations, all or aspects of the control system <NUM> may be implemented in a component that is separate from, but in communication with robot <NUM>. For example, all or aspects of control system <NUM> may be implemented on one or more computing devices that are in wired and/or wireless communication with the robot <NUM>, such as computing device <NUM>.

For example, in some implementations computing device <NUM> may be utilized to provide desired locomotion by robot <NUM> and/or other robots.

Storage subsystem <NUM> stores programming and data constructs that provide the functionality of some or all of the modules described herein. For example, the storage subsystem <NUM> may include the logic to perform selected aspects of the process of <FIG>, <FIG>, <FIG>, and/or <NUM>.

Claim 1:
A method implemented by one or more processors of a robot, the method comprising:
processing (<NUM>) an instance of sensor data using an environment-conditioned action sequence prediction model, wherein the sensor data includes an instance of vision data captured by a vision component of the robot, wherein the vision data comprises an image of at least one object in the environment of the robot, and wherein the environment-conditioned action sequence prediction model is a trained machine learning model;
determining, based on output generated based on the processing using the environment-conditioned action sequence prediction model, a set of predicted actions for an object manipulation robotic task associated with the at least one object, and a particular order for performing the predicted actions of the set;
controlling the robot to perform the predicted actions of the set in the particular order, wherein controlling the robot to perform each of the predicted actions of the set in the particular order comprises:
for each of the predicted actions, and in the particular order:
selecting (<NUM>) a corresponding action model that corresponds to the predicted action;
until determining (<NUM>) that the predicted action is complete:
processing (<NUM>) corresponding additional instances of sensor data, of the robot, using the corresponding action model, and
controlling (<NUM>) the robot based on action output, generated based on the processing using the corresponding action model.