Patent Description:
A robot is generally defined as a reprogrammable and multifunctional manipulator designed to move material, parts, tools, or specialized devices through variable programmed motions for a performance of tasks. Robots may be manipulators that are physically anchored (e.g., industrial robotic arms), mobile robots that move throughout an environment (e.g., using legs, wheels, or traction based mechanisms), or some combination of a manipulator and a mobile robot. Robots are utilized in a variety of industries including, for example, manufacturing, transportation, hazardous environments, exploration, and healthcare. As such, the ability to program robots in a quick and an efficient manner for various movement routines provides additional benefits to such industries.

<CIT> describes a motion editing system which includes a motion editor to edit motions of an upper body and whole body of a robot and a foot trajectory editor to create a gait pattern and lower-body motion to stabilize the entire robot.

One aspect of the disclosure provides a method for generating a joint command. The method includes receiving, at data processing hardware, a maneuver script including a plurality of maneuvers for a legged robot to perform. Here, each maneuver is associated with a cost. The method further includes identifying, by the data processing hardware, that two or more maneuvers of the plurality of maneuvers of the maneuver script occur at the same time instance. The method also includes determining, by the data processing hardware, a combined maneuver for the legged robot to perform at the time instance based on the two or more maneuvers and the costs associated with the two or more maneuvers. The method additionally includes generating, by the data processing hardware, a joint command to control motion of the legged robot at the time instance where the joint command commands a set of joints of the legged robot. Here, the set of joints correspond to the combined maneuver.

Implementations of the disclosure may include one or more of the following optional features. In some implementations, the legged robot corresponds to a quadruped robot. In some examples, one of the plurality of maneuvers include a hint where the hint corresponds to a body movement for a body of the legged robot. In these examples, the method may further include determining, by the data processing hardware, whether the hint is compatible with another maneuver of the plurality of maneuvers occurring at a same instance of time as the hint and modifying, by the data processing hardware, the other maneuver to incorporate the body movement of the hint when the hint is compatible with the other maneuver. In some configurations, each maneuver is configured to use an active controller for the legged robot or to modify an active controller for the legged robot, wherein a maneuver that modifies an active controller for the legged robot defines a hint. Optionally, the cost associated with each maneuver includes a user-defined cost indicating an importance for the legged robot to perform the maneuver.

In some examples, at least one of the plurality of maneuvers includes a footstep maneuver where the footstep maneuver includes a location and a time for a touchdown of a swing leg of the legged robot. In some implementations, at least one of the plurality of maneuvers includes an arm maneuver where the arm maneuver includes a pose of a manipulator of the legged robot. In some configurations, the maneuver script includes a dance script and each maneuver includes a dance move. Here, the method may further include synchronizing, by the data processing hardware, each dance move of the dance script with a beat of a song. Additionally or alternatively, the method also may include determining, by the data processing hardware, that an exit state of a first maneuver of the plurality of maneuvers of the maneuver script complies with an entry state of a subsequent maneuver.

Another aspect of the disclosure provides a robot capable of generating a joint command. The robot includes a body, two or more legs coupled to the body, and a computing system in communication with the body and the two or more legs. The computing system includes data processing hardware and memory hardware in communication with the data processing hardware. The memory hardware stores instructions that when executed on the data processing hardware cause the data processing hardware to perform operations. The operations include receiving a maneuver script including a plurality of maneuvers for the robot to perform. Here, each maneuver is associated with a cost. The operations further include identifying that two or more maneuvers of the plurality of maneuvers of the maneuver script occur at the same time instance. The operations also include determining a combined maneuver for the robot to perform at the time instance based on the two or more maneuvers and the costs associated with the two or more maneuvers. The operations additionally include generating a joint command to control motion of the robot at the time instance where the joint command commands a set of joints of the robot. Here, the set of joints correspond to the combined maneuver.

This aspect may include one or more of the following optional features. In some implementations, the legged robot corresponds to a quadruped robot. In some examples, one of the plurality of maneuvers include a hint where the hint corresponds to a body movement for the body of the robot. In these examples, the operations may further include determining whether the hint is compatible with another maneuver of the plurality of maneuvers occurring at a same instance of time as the hint and modifying the other maneuver to incorporate the body movement of the hint when the hint is compatible with the other maneuver. In some configurations, each maneuver is configured to use an active controller for the robot or to modify an active controller for the robot, wherein a maneuver that modifies an active controller for the robot defines a hint. Optionally, the cost associated with each maneuver includes a user-defined cost indicating an importance for the robot to perform the maneuver.

In some examples, at least one of the plurality of maneuvers includes a footstep maneuver where the footstep maneuver includes a location and a time for a touchdown of a swing leg of the robot. In some implementations, at least one of the plurality of maneuvers includes an arm maneuver where the arm maneuver includes a pose of a manipulator of the robot. In some configurations, the maneuver script includes a dance script and each maneuver includes a dance move. Here, the operations may further include synchronizing each dance move of the dance script with a beat of a song. Additionally or alternatively, the operations also may include determining that an exit state of a first maneuver of the plurality of maneuvers of the maneuver script complies with an entry state of a subsequent maneuver.

Often an entity that owns or controls a robot wants to preprogram movement for the robot. For example, the entity wants the robot to perform a repeatable movement routine. Unfortunately, preprogrammed movements may require hardcoding to generate the desired movements. Here, hardcoding refers to someone, such as a programmer, generating or editing a set of instructions written in a computer programming language. Hardcoding is generally time consuming and may lead to bottlenecks such that only a limited number of people (e.g., programmers/coders) may be able to code movements for the robot. Moreover, modifications to coded movements or the debugging of coded movements may lead to lengthy feedback loops to implement a movement routine. Stated differently, robots often lack a means to rapidly author robot behavior. Additionally, a manufacturer of a robot may prevent the ability of hardcoding new robot movements and/or prevent the ability to edit code previously hardcoded for existing robot movements.

Another potential issue with preprogrammed movement is that it may or may not account for the environment about the robot and/or the physics of the robot itself. In other words, someone coding the preprogrammed movements may not realize that there may be other constraints on the robot during the preprogrammed movements. Without accounting for these other constraints, a preprogrammed routine may fail or damage the robot or the environment about the robot during execution. For instance, the coder for the preprogrammed movement routine may not realize that the robot will be carrying an additional twenty-five pound load during the routine or that the routine may occur in a dynamic environment with static or dynamic obstacles. This clearly puts the onus on the entity or the coder to account for all or a majority of conditions that the robot will experience during a movement routine in order to achieve a successful movement routine.

To overcome some of these issues, referring to <FIG>, a user <NUM> designs a movement routine at a user interface (UI) <NUM> for a robot <NUM>. The robot <NUM> includes a maneuver system <NUM> that communicates with the UI <NUM>. Here, the UI <NUM> enables the user <NUM> of the robot <NUM> to generate a movement routine in the form of a maneuver script <NUM>. The maneuver script <NUM> generally refers to a plurality of maneuvers <NUM>, 210a-n (e.g., that potentially overlap in time) that the user <NUM> designs at the UI <NUM> for the robot <NUM> to perform in a robot environment <NUM>. Here, the UI <NUM> is capable of building movements for the robot <NUM> without requiring any new hardcoding. In other words, the user <NUM> uses the UI <NUM> to obtain existing maneuvers <NUM> for the robot <NUM> that may be layered or combined together to generate the movement routine for the robot <NUM> in an intuitive and nearly real-time manner. Moreover, some maneuvers <NUM> of the UI <NUM> are parameterized such that the user <NUM> may customize aspects of these maneuvers <NUM> at the UI <NUM> to fit his/her desired movement routine. As will become apparent, while maneuvers <NUM> obtained and selected by the user <NUM> for the movement routine may themselves be preprogrammed or hardcoded into the robot (e.g., by the manufacturer of the robot <NUM>), the combination and sequence of maneuvers <NUM> in the maneuver script <NUM> enable the robot <NUM> to perform/execute the movement routine without requiring any hardcoding of the movement routine by the user <NUM>.

In some examples, the user <NUM> runs the UI <NUM> on a user device <NUM>. Here, the user device <NUM> executes the UI <NUM> using computing resources <NUM>, <NUM> such as data processing hardware <NUM> and memory hardware <NUM> in communication with the data processing hardware <NUM>. Some examples for the user device <NUM> include a mobile device (e.g., a mobile phone, a tablet, a laptop, etc.), a personal computer (PC), a workstation, a terminal, or any other computing device with computing capabilities to execute the UI <NUM>. The UI <NUM> may be web-based (e.g., a web-based application launched from a web browser), local to the user device <NUM> (e.g., installed on and/or configured to execute on the computing resources <NUM>, <NUM> of the user device <NUM>), or some combination of both. In some implementations, the user device <NUM> uses the UI <NUM> to communicate the maneuver script <NUM> to the robot <NUM> using a wired or a wireless connection. For instance, the UI <NUM> communicates the maneuver script <NUM> to the robot <NUM> over a network (e.g., the network <NUM> shown in <FIG>).

With continued reference to <FIG>, the robot <NUM> includes a body <NUM> with locomotion based structures such as legs 120a-d coupled to the body <NUM> that enable the robot <NUM> to move about the environment <NUM>. In some examples, each leg <NUM> is an articulable structure such that one or more joints J permit members <NUM> of the leg <NUM> to move. For instance, each leg <NUM> includes a hip joint JH coupling an upper member <NUM>, <NUM>U of the leg <NUM> to the body <NUM> and a knee joint JK coupling the upper member <NUM>U of the leg <NUM> to a lower member <NUM>L of the leg <NUM>. Although <FIG> depicts a quadruped robot with four legs 120a-d, the robot <NUM> may include any number of legs or locomotive based structures (e.g., a biped or humanoid robot with two legs) that provide a means to traverse the terrain within the environment <NUM>.

In order to traverse the terrain, each leg <NUM> has a distal end <NUM> that contacts a surface of the terrain (i.e., a traction surface). In other words, the distal end <NUM> of the leg <NUM> is the end of the leg <NUM> used by the robot <NUM> to pivot, plant, or generally provide traction during movement of the robot <NUM>. For example, the distal end <NUM> of a leg <NUM> corresponds to a foot of the robot <NUM>. In some examples, though not shown, the distal end <NUM> of the leg <NUM> includes an ankle joint JA such that the distal end <NUM> is articulable with respect to the lower member <NUM>L of the leg <NUM>.

In some examples, the robot <NUM> includes an arm <NUM> that functions as a robotic manipulator. The arm <NUM> may be configured to move about multiple degrees of freedom in order to engage elements of the environment <NUM> (e.g., objects within the environment <NUM>) or to perform gestures (e.g., aesthetic gestures). In some examples, the arm <NUM> includes one or more members <NUM> where the members <NUM> are coupled by joints J such that the arm <NUM> may pivot or rotate about the joint(s) J. For instance, with more than one member <NUM>, the arm <NUM> may be configured to extend or to retract. To illustrate an example, <FIG> depicts the arm <NUM> with three members <NUM> corresponding to a lower member <NUM>, an upper member 128u, and a hand member <NUM>H. Here, the lower member <NUM>L may rotate or pivot about a first arm joint JA1 located adjacent to the body <NUM> (e.g., where the arm <NUM> connects to the body <NUM> of the robot <NUM>). The lower member <NUM>L is coupled to the upper member 128u at a second arm joint JA2 and the upper member 128u is coupled to the hand member <NUM>H at a third arm joint JA3. In some examples, the hand member <NUM>H includes additional members to enable different types of grasping. These additional members may range from a simple two member claw-like hand member <NUM>H to a more complicated hand member <NUM>H that simulates the digits of a human hand. In some implementations, the arm <NUM> connects to the robot <NUM> at a socket on the body <NUM> of the robot <NUM>. In some configurations, the socket is configured as a connector such that the arm <NUM> may attach or detach from the robot <NUM> depending on whether the arm <NUM> is needed for operation.

The robot <NUM> has a vertical gravitational axis (e.g., shown as a Z-direction axis AZ) along a direction of gravity, and a center of mass CM, which is a position that corresponds to an average position of all parts of the robot <NUM> where the parts are weighted according to their masses (i.e., a point where the weighted relative position of the distributed mass of the robot <NUM> sums to zero). The robot <NUM> further has a pose P based on the CM relative to the vertical gravitational axis Az (i.e., the fixed reference frame with respect to gravity) to define a particular attitude or stance assumed by the robot <NUM>. The attitude of the robot <NUM> can be defined by an orientation or an angular position of the robot <NUM> in space. Movement by the legs <NUM> relative to the body <NUM> alters the pose P of the robot <NUM> (i.e., the combination of the position of the CM of the robot and the attitude or orientation of the robot <NUM>). Here, a height generally refers to a distance along the z-direction. The sagittal plane of the robot <NUM> corresponds to the Y-Z plane extending in directions of a y-direction axis AY and the z-direction axis Az. In other words, the sagittal plane bisects the robot <NUM> into a left and a right side. Generally perpendicular to the sagittal plane, a ground plane (also referred to as a transverse plane) spans the X-Y plane by extending in directions of the x-direction axis AX and the y-direction axis AY. The ground plane refers to a ground surface <NUM> where distal ends <NUM> of the legs <NUM> of the robot <NUM> may generate traction to help the robot <NUM> move about the environment <NUM>. Another anatomical plane of the robot <NUM> is the frontal plane that extends across the body <NUM> of the robot <NUM> (e.g., from a left side of the robot <NUM> with a first leg 120a to a right side of the robot <NUM> with a second leg 120b). The frontal plane spans the X-Z plane by extending in directions of the x-direction axis AX and the z-direction axis Az.

When a legged-robot moves about the environment <NUM>, each leg <NUM> of the robot <NUM> may either be in contact with the ground surface <NUM> or not in contact with the ground surface <NUM>. When a leg <NUM> is in contact with the ground surface <NUM>, the leg <NUM> is referred to as a stance leg <NUM>ST. When a leg <NUM> is not in contact with the ground surface <NUM>, the leg <NUM> is referred to as a swing leg <NUM>SW. A leg <NUM> transitions from a stance leg <NUM>ST to a swing leg <NUM>SW when the leg <NUM> lifts-off (LO) from the ground surface <NUM>. Conversely, a swing leg <NUM>SW may also transition to a stance leg <NUM>ST when the swing leg touches down (TD) against the ground surface <NUM> after not being in contact with the ground surface <NUM>. Here, while a leg <NUM> is functioning as a swing leg 120sw, another leg <NUM> of the robot <NUM> may be functioning as a stance leg <NUM>ST (e.g., to maintain balance for the robot <NUM>).

In order to maneuver about the environment <NUM>, the robot <NUM> includes a sensor system <NUM> with one or more sensors <NUM>, 132a-n (e.g., shown as a first sensor <NUM>, 132a and a second sensor <NUM>, 132b). The sensors <NUM> may include vision/image sensors, inertial sensors (e.g., an inertial measurement unit (IMU)), force sensors, and/or kinematic sensors. Some examples of vision/image sensors <NUM> include a camera such as a stereo camera, a scanning light-detection and ranging (LIDAR) sensor, or a scanning laser-detection and ranging (LADAR) sensor. In some examples, the sensor <NUM> has a corresponding field(s) of view Fv defining a sensing range or region corresponding to the sensor <NUM>. For instance, <FIG> depicts a field of a view Fv for the robot <NUM>. Each sensor <NUM> may be pivotable and/or rotatable such that the sensor <NUM> may, for example, change the field of view Fv about one or more axis (e.g., an x-axis, a y-axis, or a z-axis in relation to a ground plane).

When surveying a field of view Fv with a sensor <NUM>, the sensor system <NUM> generates sensor data <NUM> (also referred to as image data) corresponding to the field of view Fv. In some examples, the sensor data <NUM> is image data that corresponds to a three-dimensional volumetric point cloud generated by a three-dimensional volumetric image sensor <NUM>. Additionally or alternatively, when the robot <NUM> is maneuvering about the environment <NUM>, the sensor system <NUM> gathers pose data for the robot <NUM> that includes inertial measurement data (e.g., measured by an IMU). In some examples, the pose data includes kinematic data and/or orientation data about the robot <NUM>, for instance, kinematic data and/or orientation data about joints J or other portions of a leg <NUM> of the robot <NUM>. With the sensor data <NUM>, various systems of the robot <NUM> may use the sensor data <NUM> to define a current state of the robot <NUM> (e.g., of the kinematics of the robot <NUM>) and/or a current state of the environment <NUM> about the robot <NUM>.

In some implementations, the sensor system <NUM> includes sensor(s) <NUM> coupled to a joint J. In some examples, these sensors <NUM> couple to a motor M that operates a joint J of the robot <NUM> (e.g., sensors <NUM>, 132a-b). Here, these sensors <NUM> generate joint dynamics in the form of joint-based sensor data <NUM>. Joint dynamics collected as joint-based sensor data <NUM> may include joint angles (e.g., an upper member <NUM>U relative to a lower member <NUM>L), joint speed (e.g., joint angular velocity or joint angular acceleration), and/or forces experienced at a joint J (also referred to as joint forces). Here, joint-based sensor data <NUM> generated by one or more sensors <NUM> may be raw sensor data, data that is further processed to form different types of joint dynamics <NUM>JD, or some combination of both. For instance, a sensor <NUM> measures joint position (or a position of member(s) <NUM> coupled at a joint J) and systems of the robot <NUM> perform further processing to derive velocity and/or acceleration from the positional data. In other examples, a sensor <NUM> is configured to measure velocity and/or acceleration directly.

As the sensor system <NUM> gathers sensor data <NUM>, a computing system <NUM> is configured to store, to process, and/or to communicate the sensor data <NUM> to various systems of the robot <NUM> (e.g., the control system <NUM> and/or the maneuver system <NUM>). In order to perform computing tasks related to the sensor data <NUM>, the computing system <NUM> of the robot <NUM> includes data processing hardware <NUM> and memory hardware <NUM>. The data processing hardware <NUM> is configured to execute instructions stored in the memory hardware <NUM> to perform computing tasks related to activities (e.g., movement and/or movement based activities) for the robot <NUM>. Generally speaking, the computing system <NUM> refers to one or more locations of data processing hardware <NUM> and/or memory hardware <NUM>.

In some examples, the computing system <NUM> is a local system located on the robot <NUM>. When located on the robot <NUM>, the computing system <NUM> may be centralized (i.e., in a single location/area on the robot <NUM>, for example, the body <NUM> of the robot <NUM>), decentralized (i.e., located at various locations about the robot <NUM>), or a hybrid combination of both (e.g., where a majority of centralized hardware and a minority of decentralized hardware). To illustrate some differences, a decentralized computing system <NUM> may allow processing to occur at an activity location (e.g., at motor that moves a joint of a leg <NUM>) while a centralized computing system <NUM> may allow for a central processing hub that communicates to systems located at various positions on the robot <NUM> (e.g., communicate to the motor that moves the joint of the leg <NUM>).

Additionally or alternatively, the computing system <NUM> includes computing resources that are located remotely from the robot <NUM>. For instance, the computing system <NUM> communicates via a network <NUM> with a remote system <NUM> (e.g., a remote server or a cloud-based environment). Much like the computing system <NUM>, the remote system <NUM> includes remote computing resources such as remote data processing hardware <NUM> and remote memory hardware <NUM>. Here, sensor data <NUM> or other processed data (e.g., data processing locally by the computing system <NUM>) may be stored in the remote system <NUM> and may be accessible to the computing system <NUM>. In some examples, the computing system <NUM> is configured to utilize the remote resources <NUM>, <NUM> as extensions of the computing resources <NUM>, <NUM> such that resources of the computing system <NUM> may reside on resources of the remote system <NUM>. In some configurations, the remote system <NUM> and/or the computing system <NUM> hosts the UI <NUM> such that the user <NUM> accesses the UI <NUM> via the network <NUM> using a web browser application.

In some implementations, as shown in <FIG>, the robot <NUM> includes a control system <NUM>. The control system <NUM> may be configured to communicate with systems of the robot <NUM> such as the at least one sensor system <NUM> and/or the maneuver system <NUM>. The control system <NUM> may perform operations and other functions using hardware <NUM>. The control system <NUM> includes at least one controller <NUM> that is configured to control the robot <NUM>. For example, the controller <NUM> controls movement of the robot <NUM> to traverse about the environment <NUM> based on input or feedback from the systems of the robot <NUM> (e.g., the sensor system <NUM>, the control system <NUM>, and/or the maneuver system <NUM>). In some implementations, the controller <NUM> controls movement between poses and/or behaviors of the robot <NUM>.

In some examples, a controller <NUM> controls the robot <NUM> by controlling movement about one or more joints J of the robot <NUM>. In some configurations, a controller <NUM> is software with programming logic that controls at least one joint J or a motor M which operates, or is coupled to, a joint J. For instance, the controller <NUM> controls an amount of force that is applied to a joint J (e.g., torque at a joint J). As programmable controllers <NUM>, the number of joints J that a controller <NUM> controls is scalable and/or customizable for a particular control purpose. A controller <NUM> may control a single joint J (e.g., control a torque at a single joint J) or control multiple joints J of the robot <NUM>. When controlling multiple joints J, the controller <NUM> may apply the same or different torques to each joint J controlled by the controller <NUM>. By controlling multiple joints J, a controller <NUM> may coordinate movement for a larger structure of the robot <NUM> (e.g., the body <NUM>, one or more legs <NUM>, the arm <NUM>). In other words, for some maneuvers <NUM>, a controller <NUM> may be configured to control movement of multiple parts of the robot <NUM> such as, for example, two legs 120a-b, four legs 120a-d, or two legs 120a-b combined with the arm <NUM>. For instance, <FIG> depicts an example relationship between motors M of the robot <NUM> and the joints J that these motors M control for a given structure of the robot <NUM>. Here, <FIG> illustrates four controllers <NUM>, 172a-d that are setup to control portions of the robot <NUM>. A first controller <NUM>, 172a controls motors M for knee joints JK and hip joints JH related to a first leg <NUM>, 120a and a second leg <NUM>, 120b. A second controller 172b is shown configured to control motors M for a knee joint JK and a hip joint JH related to a single leg, such as a third leg <NUM>, 120c. A third controller 172c is shown configured to control motors M related to the third leg 120c, a fourth leg <NUM>, 120d, and one of the motors M associated with an arm joint JA1. A fourth controller 172d is shown configured to control motors M related to only the arm <NUM> of the robot <NUM>. For practical comparison, the third controller 172c may allow the robot <NUM> to lean to side of the robot's body <NUM> over the third leg 120c and the fourth leg 120d while also grasping something with the arm <NUM>. In contrast to the third controller 172c, the robot <NUM> may perform a lean and grasp function with three controllers <NUM> where two controllers <NUM> individually control the third and fourth leg 120c-d while a fourth controller 172d controls the arm <NUM>. There may be some advantages or reasons to coordinate multiple controllers <NUM> (e.g., for fine motor control) or to use a single controller <NUM> (e.g., for simplicity).

Referring to <FIG>, the UI <NUM> is configured to allow the user <NUM> to construct a movement routine in the form of a maneuver script <NUM>. In some examples, the maneuver script <NUM> is a computer-readable format that the user <NUM> (e.g., via the UI <NUM>) communicates to the robot <NUM> to enable the robot <NUM> to perform the movement routine. In some implementations, the maneuver script <NUM> is a sequence of movements over a set period of time t that the user <NUM> requests the robot <NUM> to perform. To construct the maneuver script <NUM>, the UI <NUM> provides a plurality of maneuvers 210a-n for the user <NUM> to select. Here, the maneuvers <NUM> refer to hardcoded movements for the robot <NUM> coordinated by controllers <NUM> of the control system <NUM>. A maneuver <NUM> may correspond to a controller <NUM> of the robot <NUM> (e.g., an active controller programmed for the control system <NUM>), a modification to a controller <NUM> of the robot <NUM>, one or more input parameters <NUM> for a controller <NUM> of the robot <NUM>, or any combination thereof. In some examples, the maneuver <NUM> defines a set of joints J necessary to perform the underlying movement(s) associated with the maneuver <NUM> (e.g., by designating a controller <NUM> that controls a specific joint J or joints J). Some different types of maneuvers <NUM> include the following: footstep maneuvers <NUM>, 210f (or leg maneuvers) that move the feet <NUM> of the robot <NUM>; arm maneuvers <NUM>, <NUM>a that move the arm <NUM> of the robot <NUM>; and/or body maneuvers <NUM>, <NUM>b that move the body <NUM> of the robot <NUM>. Moreover, since a maneuver <NUM> may correspond to a controller <NUM>, the movement of the maneuver <NUM> may correspond to any number of joints J, even joints J from different portions of the robot <NUM> (e.g., from two legs <NUM> on different sides of the robot's body <NUM>). In other words, a maneuver <NUM> may be configured to be a hybrid of different types of maneuvers. For example, a maneuver <NUM> that combines a footstep maneuver <NUM>f and an arm maneuver <NUM>a. In some configurations, a maneuver <NUM> is parameterized such that the maneuver <NUM> has been hardcoded with parameters <NUM> that may be modified to generate a set amount of customizable variance in the movement. For instance, a footstep maneuver <NUM>f includes parameters <NUM> such as designating which foot <NUM> to step with, a coordinate/position for the footstep (e.g., a LO/TD position), a swing height for the designated stepping foot <NUM>, and/or a step velocity for the designated stepping foot <NUM>. In the case of a body maneuver <NUM>b, the parameters <NUM> may include a sway area, a location to sway around, a direction to sway, a sway speed, and/or a sway style.

In some examples, in order to sequence one or more maneuvers <NUM> to form a maneuver script <NUM>, the UI <NUM> divides both the maneuvers <NUM> and the maneuver script <NUM> into segments of time called slices tS. In other words, a slice tS of time from a maneuver <NUM> is compatible with a slice tS of time of the maneuver script <NUM> such that a maneuver <NUM> may occupy one or more slices tS of the maneuver script <NUM>. By subdividing a maneuver script <NUM> into units, such as slices tS, the maneuver script <NUM> may be scalable to different lengths of time. Stated differently, the UI <NUM> divides a maneuver script <NUM> into a number of discrete maneuvers <NUM> over time where each maneuver <NUM> corresponds to a fixed number of slices tS. For example, a script builder <NUM> for building the maneuver script <NUM> in <FIG> in a timeline <NUM> shows the maneuver script <NUM> having four maneuvers <NUM>, 210a-d and lasting a total of forty slices tS. Here, a first maneuver 210a occupies twelve slices tS, a second maneuver 210b and a third maneuver 210c occupy eight slices tS each, and a fourth maneuver 210d occupies twelve slices tS. When the user <NUM> wants to perform this forty-slice maneuver script <NUM> in forty seconds, the robot <NUM> allocates twelve seconds to each of the first maneuver 210a and the fourth maneuver 210d and eight seconds to each of the second maneuver 210c and the third maneuver 210d. In contrast, when the user <NUM> wants to perform this forty-slice maneuver <NUM> in twice the time of eighty seconds, the robot <NUM> scales the slices of the maneuvers <NUM> such that the robot <NUM> allocates twenty-four seconds to each of the first maneuver 210a and the fourth maneuver 210d and sixteen seconds to each of the second maneuver 210c and the third maneuver 210d. Additionally or alternatively, since each maneuver <NUM> also occurs for a set period of time tM with a start time and an end time, a maneuver script <NUM> may include unoccupied slices before, between, or after maneuvers <NUM> as part of the movement routine.

In some implementations, the UI <NUM> includes a list <NUM> of maneuvers <NUM> from which the user <NUM> may select to build a maneuver script <NUM>. When the user <NUM> selects from the list <NUM> (e.g., the selection shown as a bolded outline around a maneuver <NUM> in <FIG>), the selection of the maneuver <NUM> may display parameters <NUM> specific to the maneuver <NUM> that the user <NUM> may customize (e.g., shown in a simulator <NUM>). In some examples, the user <NUM> adds the maneuver <NUM> to the maneuver script <NUM> by selection or by selecting and dragging the maneuver <NUM> into the script builder <NUM>. Some examples of a selection include double-clicking or selecting the maneuver <NUM> and clicking a GUI element, such as "Add," (e.g., as shown in <FIG>). Here, the script builder <NUM> may be another panel within the same window with the maneuver list <NUM> or a separate window.

When a maneuver <NUM> is added to the script builder <NUM> to construct the maneuver script <NUM>, the maneuver <NUM> may have a default size (e.g., number of slices) based on the hardcoding of the maneuver <NUM>. For instance, a single maneuver <NUM> that corresponds to a simple movement may, by default, occupy less slices ts than a more complicated movement (e.g., a handstand). In either case, the default number of slices tS for the maneuver <NUM> may correspond to a minimum amount of time that the robot <NUM> needs to successfully perform the maneuver <NUM>. In some implementations, with the maneuver <NUM> added to the script builder <NUM>, the maneuver <NUM> may then be further modified. For example, the user <NUM> resizes the maneuver <NUM> to extend the maneuver <NUM> over a number of slices ts greater than the default number of slices ts for a maneuver <NUM>. In some examples, the script builder <NUM> is a panel or window that includes a timeline <NUM> divided into segments (e.g., slices ts). Here, the user <NUM> imports maneuver(s) <NUM> into the timeline <NUM> (e.g., by selection or a drag and drop process) at a slice position within the timeline <NUM> that the user <NUM> designates.

In some examples, such as <FIG>, the timeline <NUM> includes more than one layer <NUM>. Here, a layer <NUM> refers to a space in the timeline <NUM> that a maneuver <NUM> may occupy when it impacts particular joints J allocated to the layer <NUM>. In other words, a layer <NUM> defines the allocation of joints J for the robot <NUM>. In some configurations, each joint J of the robot <NUM> is allocated to a particular layer <NUM> of the timeline <NUM>. For instance, a first layer <NUM>, 224a corresponds to leg joints such as a knee joint JK and a hip joint JH. In some examples, one or more layers <NUM> do not have any joints J. An example of a layer <NUM> without any joints J may be when the layer <NUM> corresponds to a specific type of maneuver <NUM>, such as a body maneuver <NUM>b. In other words, there may be certain maneuvers <NUM> that passively control joints J. Stated differently, when a maneuver <NUM> is a modification to a controller <NUM> or an input to a controller <NUM>, the maneuver <NUM> may impact a joint J that is already designated or allocated to a particular layer <NUM>. For example, practically speaking, movement of the body <NUM> occurs by making a change to at least one joint J of at least one leg <NUM> of the robot <NUM>. Yet, when the joints J of the leg <NUM> are already allocated to a layer <NUM> (e.g., for footstep maneuvers 210f or leg maneuvers), a movement of the body <NUM> may cause joint control interference between two layers <NUM>. In some examples, to prevent issues caused by joint interference, the UI <NUM> may communicate a body maneuver <NUM>b as a suggested modification to a controller <NUM> of another layer <NUM> or a parameter of movement for a particular controller <NUM>. In other words, a body maneuver <NUM>b is a secondary consideration that may impact joint control when it would not interfere with, for example, a leg maneuver. Here, a maneuver <NUM> that may modify control for a joint J is referred to as a hint <NUM>H. When a hint <NUM>H occurs at a particular time instance, the hint <NUM>H is considered by each other active controller <NUM> (e.g., each time-current controller <NUM>). When a controller <NUM> interprets the hint <NUM>H, the controller <NUM> may determine that the hint <NUM>H corresponds to joints J different from joints J that the controller <NUM> controls and ignore modifying its controller behavior for the hint <NUM>H. When the hint <NUM>H corresponds to the same joints J, the controller <NUM> may incorporate the movement or movement parameters <NUM> of the hint <NUM>H into its own behavior.

In some examples, such as <FIG>, the script builder <NUM> of the UI <NUM> allows a user <NUM> to have a maneuver <NUM> on each layer <NUM> (e.g., shown as three layers <NUM>, 224a-c) at each instance in time. When joints J are allocated to layers <NUM> to prevent interference, multiple maneuvers <NUM> may be layered at the same time instance on the timeline <NUM>. In other words, with layers <NUM>, when a maneuver <NUM> controls only certain parts of the robot <NUM>, multiple maneuvers <NUM> may be layered on top of each other to engage a wider range of dynamics for the robot <NUM>. For instance, the designation of a hint <NUM>H allows multiple maneuvers <NUM> to be layered at the same time instance on the timeline <NUM> because one layer <NUM> will be a primary consideration for control (e.g., a leg maneuver) while another layer <NUM>, such as a layer <NUM>, 224b for body maneuvers <NUM>b will be a secondary consideration for control. By having layers <NUM>, maneuvers <NUM> may be combined and/or sequenced in various combinations to give the user <NUM> many more options than simply the number of maneuvers <NUM> available on the list <NUM>. For example, <FIG> shows a first maneuver <NUM>a, 210a on a first layer 224a (e.g., an arm maneuver layer), a second maneuver <NUM>b, 210b on a second layer 224b (e.g., a body maneuver layer), and a third maneuver <NUM>f, 210c on a third layer 224c (e.g., a leg maneuver layer).

Although the UI <NUM> allows a layered construction of maneuvers <NUM>, the UI <NUM> generally does not determine the feasibility of the combination of maneuvers <NUM>. Rather, as long as the layering of maneuvers <NUM> does not violate rules at the UI <NUM> (e.g., at the script builder <NUM>), the UI <NUM> communicates the maneuver script <NUM> with the combination of the layers <NUM> to the robot <NUM> (e.g., to the maneuver system <NUM> of the robot <NUM>). In some examples, a maneuver <NUM> is dedicated to a particular layer <NUM>. More particularly, the UI <NUM> would not allow the user <NUM> to place a body maneuver <NUM>b on a layer <NUM> that does not control joints J related to the body maneuver <NUM>b. For example, as illustrated by <FIG>, the user <NUM> cannot place a sway body maneuver <NUM>b that impacts joints J of one or more legs <NUM> on a layer <NUM> that only controls arm joints JA. In some implementations, when the user <NUM> adds a maneuver <NUM> to the timeline <NUM> of the script builder <NUM>, the maneuver <NUM> auto-populates to the layer(s) <NUM> associated with joints J that the maneuver <NUM> impacts (e.g., needs to control for the underlying movement of the maneuver <NUM>). Here, a user <NUM> controls where on the timeline <NUM> the maneuver <NUM> is positioned, but not the layer <NUM> itself.

In some configurations, a maneuver <NUM> is configured to operate joints J on more than one layer <NUM> of the timeline <NUM>. In these configurations, the script builder <NUM> includes the maneuver <NUM> on multiple layers <NUM> occupying slices ts of each layer <NUM> corresponding to the default number of slices ts for the maneuver <NUM>. For instance, <FIG> shows a second maneuver 210b on the timeline <NUM> of the script builder <NUM> occupying twelve slices ts of a second layer <NUM>, 224b and a twelve slices ts of a third layer <NUM>, 224c at the same time. Here, the maneuver <NUM> is shown as a single block such that UI <NUM> prevents the user <NUM> from manipulating the same maneuver <NUM> differently in different layers <NUM>, which would likely cause an error when the robot <NUM> attempts to perform the maneuver <NUM>. Referring to <FIG>, since the timeline <NUM> includes three layer 224a-c, another maneuver <NUM> may be combined with the second maneuver 210b by occupying the first layer 224a at some portion of the same time instance.

In some configurations, the UI <NUM> includes rules regarding a starting state (i.e., an entry state <NUM>) and ending state (i.e., an exit state 218e) for maneuvers <NUM>. The UI <NUM> incorporates these rules to prevent a likely error when the robot <NUM> attempts the maneuver script <NUM>. For example, as illustrated in <FIG>, when a second maneuver <NUM>, 210b ends with an ending state 218e of the robot <NUM> on its knees, the following maneuver <NUM> (e.g., shown as a third maneuver 210c) cannot assume that the robot <NUM> is immediately capable of making a maneuver <NUM> predicated on a starting state <NUM>S that the robot <NUM> is standing (e.g., certain types of footstep maneuvers <NUM>f).

To overcome these potential issues between neighboring maneuvers <NUM>, each maneuver <NUM> may include an entry state <NUM> and/or an exit state 218e. With each maneuver <NUM> including an entry state <NUM> and/or exit state 218e, the UI <NUM> (e.g., the script builder <NUM>) may compare the transitioning states between two neighboring maneuvers <NUM> (e.g., an exit state 218e of a first maneuver 210a and an entry state <NUM> of an adjacent second maneuver 210b) to determine whether these transitioning states are compatible. When these transitioning states are not compatible, the UI <NUM> communicates to the user <NUM> that the sequence of maneuvers <NUM> is incompatible. In some examples, the UI <NUM> recognizes that a particular exit state 218e may be compatible with more than one entry state <NUM>.

Alternatively, the UI <NUM> may make this determination when the user <NUM> is attempting to place or to select an adjacent maneuver <NUM>. Here, when the adjacent maneuver <NUM> is not compatible, the UI <NUM> does not allow the user <NUM> to make the placement or the selection of the maneuver <NUM>. In the example depicted by <FIG>, the selected maneuver 210c that is incompatible is greyed out and shown with an "X" to communicate to the user <NUM> that the maneuver <NUM> is not compatible at that location in the timeline <NUM>. In some implementations, the UI <NUM> displays a message at the UI <NUM> that communicates that maneuver <NUM> is not compatible for the location where the user <NUM> wants to place the maneuver <NUM> on the timeline <NUM>. In some examples, the list <NUM> is dynamic such that the list <NUM> displays maneuvers <NUM> that are compatible with a previous selection. For example, the user <NUM> may select (i.e., click-on) a maneuver <NUM> placed on the timeline <NUM> and the maneuver list <NUM> displays compatible maneuvers <NUM>. In some examples, the UI <NUM> includes a GUI element (e.g., from a menu or toolbar in the UI <NUM>) that that the user <NUM> selects to display compatible maneuvers <NUM> based on entry state <NUM>S and/or exit state <NUM>e. Additionally or alternatively, the UI <NUM> may provide assistance to the user <NUM> when the user <NUM> selects or tries to place a maneuver <NUM> at incompatible position on the timeline <NUM>. For instance, the UI <NUM> may suggest that the user <NUM> adds a maneuver <NUM> as a transition between the maneuvers <NUM> that are incompatible when adjacent to each other in time on the timeline <NUM>. Here, the UI <NUM> suggests a maneuver <NUM> that matches the exit state <NUM>e of the first maneuver 210a with the entry state <NUM>S of the second maneuver 210b.

In some examples, such as <FIG>, the script builder <NUM> is scalable for more than one robot <NUM>, 100a-b. For instance, the script builder <NUM> includes a plurality of timelines <NUM>, 222a-b where each timeline <NUM> has layers <NUM> particular to a robot <NUM>. In <FIG>, a first timeline <NUM>, 222a is for a first robot 100a and a second timeline <NUM>, 222b is for a second robot 100b. By coordinating timelines <NUM> for multiple robots <NUM>, the UI <NUM> may be used to generate coordinated movement routines between robots <NUM>. For instance, the first robot 100a performs a first phase p<NUM> of movement(s), pauses while the second robot 100b then performs a second phase p<NUM> of movement(s), and then the first robot 100a together with the second robot 100b performs a final third phase of movement(s). In some implementations, the UI <NUM> may be compatible with different types of robots <NUM> such that types or models of robots <NUM> may be loaded with a robot profile specific profile. Here, the robot specific profile may load a timeline <NUM> with a number of layers <NUM> specific to the robot <NUM> of the robot specific profile. To illustrate, <FIG> depicts the first timeline 222a with three layers <NUM>, 224a-c and the second timeline 222b with only two layers <NUM>, 224a-b. The second timeline <NUM> has less layers <NUM> because the corresponding robot 100b does not include an arm <NUM> and therefore the layer <NUM> corresponding to joints J of the arm <NUM> is unnecessary. Although <FIG> shows the timelines 222a-b displayed in the same window, each timeline <NUM> may have its own window (e.g., that the user <NUM> may toggle between).

Referring back to <FIG>, in some examples, the UI <NUM> also includes a simulator <NUM>. The simulator <NUM> may be a panel within a main window of the UI <NUM> or its own window of the UI <NUM>. In some implementations, the simulator <NUM> is configured to display a model <NUM> or an avatar for the robot <NUM> as a visual simulation for a maneuver <NUM>. For example, the user <NUM> selects one or more maneuvers <NUM> and the simulator <NUM> displays a sample rendition of selected one or more maneuvers <NUM>. In some configurations, such as <FIG>, the simulator <NUM> includes GUI elements <NUM> that may input or adjust parameters <NUM> associated with the maneuver <NUM>. In these configurations, the simulator <NUM> may display the maneuver <NUM> applying the parameters <NUM> input by the user <NUM>. For instance, the sliders shown in <FIG> as GUI elements <NUM> for parameters <NUM> may adjust a tilt or pitch of the body <NUM> of the robot <NUM> during a maneuver <NUM> (e.g., a body maneuver <NUM>b). Although <FIG> illustrates a single set of parameters <NUM> for the selection of maneuver 210n, the simulator <NUM> may be configured to display different GUI elements <NUM> that correspond to the parameters <NUM> of the selected maneuver <NUM>. For example, the slider GUI elements <NUM> may change to simple input fields when the user <NUM> selects the second maneuver 210b.

In some configurations, the UI <NUM> is configured to communicate portions of the maneuver script <NUM> to the robot <NUM>. By communicating portions of the maneuver script <NUM>, the user <NUM> may troubleshoot or debug sub-portions of the movement routine. For instance, when a user <NUM> is frustrated that the maneuver script <NUM> is failing to execute successfully at the robot <NUM>, the user <NUM> uses the UI <NUM> to select a portion of the maneuver script <NUM> to communicate to the robot <NUM>. In some implementations, the UI <NUM> requests the robot <NUM> to perform the maneuver script <NUM> or portion of the maneuver script <NUM> in a loop. Here, the user <NUM> may update the maneuver script <NUM> at the UI <NUM> while the robot <NUM> loops a performance of a maneuver script <NUM> or portion thereof. In some configurations, the user <NUM> designates in the timeline <NUM> where to loop the maneuver script <NUM>. For instance, the user <NUM> inserts a loop widget <NUM> (<FIG> and <FIG>) (e.g., shown as an adjustable vertical bar) into the timeline <NUM> to designate a position in the maneuver script <NUM> where the movement routine should repeat (e.g., restart). To illustrate, <FIG> shows the loop widget <NUM> before the third phase p<NUM> of the maneuver scripts 202a-b such that the user <NUM> may troubleshoot the maneuvers <NUM>, 210i-q of the first phase p<NUM> and the second phase p<NUM>.

Additionally, the UI <NUM> may be configured to save and/or to load maneuver scripts <NUM> that have been built by the user <NUM> or are in the process of being built. For instance, the UI <NUM> communicates with the computing resources <NUM>, <NUM> of the user device <NUM> or the computing resources <NUM> of the remote system <NUM> to store or to load a particular maneuver script <NUM>. In some examples, the UI <NUM> allows the user <NUM> to share a maneuver script <NUM> with other users such that maneuver scripts <NUM> may be built in collaboration. In some implementations, the UI <NUM> includes storage containers (e.g., file directories) that include one or more maneuver scripts <NUM> or a pointer to a location of one or more maneuver scripts <NUM>. Additionally or alternatively, the computing system <NUM> of the robot <NUM> may store maneuver scripts <NUM> (e.g., temporarily or for some extended period of time).

In some implementations, a maneuver script <NUM> corresponds to a dance routine where each maneuver <NUM> is a dance move that the robot <NUM> may perform. Here, the time slices ts of the timeline <NUM> may be used to coordinate with a tempo (i.e., rate) of a particular song. For instance, a song is identified (e.g., by the UI <NUM>) to be a particular number of beats per minute (BPM) and each slice ts corresponds to a division of a single beat b. Since the maneuvers <NUM> are scalable, the tempo of a song may be changed (e.g., sped up or slowed down) or the actual song may be changed and the performance of the maneuver script <NUM> may be adapted to the changed tempo.

In some implementations, the script builder <NUM> includes a setting (e.g., a GUI element <NUM>) that sets the timeline <NUM> to a particular tempo (e.g., BPM) to correspond to a particular song. This setting, for example, may allow the user <NUM> to directly map a beat b of a song to a maneuver <NUM> or portions of a maneuver <NUM>. For instance, <FIG> depicts that a footstep maneuver <NUM>f indicates a touchdown (TD) and/or a liftoff (LO) position in a block representation of the footstep maneuver <NUM>f on the timeline <NUM>. With this TD/LO indication, the user <NUM> may align (or misalign on the upbeat) the TD or LO directly with a beat b of a song. For example, in <FIG>, the timeline <NUM> displays the beat b of a song by a darker bar between slices ts.

In some configurations, the UI <NUM> (e.g., the script builder <NUM>) includes features to promote synchronization of a maneuver script <NUM> with audio from a song. In some examples, such as <FIG>, the UI <NUM> includes a slider bar <NUM> for the timeline <NUM>. The slider bar <NUM> may be a time-synchronization tool where the user <NUM> may set the slider bar <NUM> at a custom position in the timeline <NUM>. The position of the slider bar <NUM> may be used to indicate a start position where the robot <NUM> should start performing the movement routine (e.g., the dance) of the maneuver script <NUM>. For example, <FIG> depicts the slider bar <NUM> before the first maneuver 220a.

In some implementations, such as <FIG>, the UI <NUM> sends the robot <NUM> the maneuver script <NUM> as a configuration file along with a current system time <NUM> of the UI <NUM> (e.g., a time stamp according to a clock of the user device <NUM>) and a start time <NUM> at which the robot <NUM> should start the movement routine (e.g., as indicated by the position of the slider bar <NUM>). Here, the robot <NUM>, upon receiving the maneuver script <NUM> with the system time <NUM> of the UI <NUM> and the start time <NUM> for the movement routine, will wait the amount of time specified by the UI <NUM> and then begin the movement routine (e.g., start dancing). In some examples, when the robot <NUM> receives the maneuver script <NUM>, the robot <NUM> presumes that the communication of the maneuver script <NUM> between the UI <NUM> and the robot <NUM> has taken some amount of communication time and synchronizes its own clock to account for this amount of communication time. For instance, the robot <NUM> synchronizes its own clock to be the clock time stamp <NUM> received from the UI <NUM> plus an additional amount for the communication time. Based on this time synchronization, the robot <NUM> begins the movement routine (e.g., the dance) at some future time (e.g., at a time in the future equal to a difference between system time <NUM> of the UI <NUM> and the designated start time <NUM>).

Additionally or alternatively, the robot <NUM> provides feedback <NUM> to the UI <NUM> as to a time location where the robot <NUM> perceives it is in the movement routine (e.g., the dance). For instance, the robot <NUM> provides its own clock time <NUM>, 208a (<FIG>) (e.g., in the form of a time stamp) as feedback <NUM>. Here, the robot's time should closely resemble the system time <NUM> of the UI <NUM> because of previous synchronization when the robot <NUM> received the maneuver script <NUM>. In some examples, when there is variance between the system time <NUM> of the UI <NUM> and the clock time 208a of the robot <NUM>, the UI <NUM> may resend its system time <NUM> to the robot <NUM> to resynchronize. In some implementations, when resynchronization occurs, the robot <NUM> may be configured to speed up or to slow down maneuvers <NUM> of the maneuver script <NUM>. For instance, the robot <NUM> identifies a candidate maneuver <NUM> that the robot <NUM> may perform more quickly or more slowly to adjust the maneuver routine <NUM> with the resynchronization. In yet other examples, instead of feedback <NUM> from the robot <NUM>, the UI <NUM> periodically communicates its current time clock <NUM> (e.g., at some designated frequency) to the robot <NUM> such that the robot <NUM> checks for synchronization between the UI <NUM> and the robot <NUM>. In these examples, the robot <NUM> performs resynchronization when the periodic communication from the UI <NUM> indicates a misalignment of time. In some configurations, the robot <NUM> resynchronizes when the difference between the system time <NUM> of the UI <NUM> and the time clock 208a of the robot <NUM> satisfies a time shift threshold (e.g., exceeds a value set as the time shift threshold). This approach may prevent the robot <NUM> from frequently resynchronizing when small time shifts exist between the UI <NUM> and the robot <NUM>.

Referring to <FIG>, the maneuver system <NUM> includes an interface <NUM>, such as an application programming interface (API), a sequencer <NUM>, and a dynamics planner <NUM>. From an overall perspective, the goal of the maneuver system <NUM> is to receive a motion request (e.g., the maneuver script <NUM> with maneuver(s) <NUM>) as an input and convert that input to a solution that generates motion for the robot <NUM>. Here, even if the input is impossible (e.g., from a physics or robot dynamics perspective), the maneuver system <NUM> is configured to return a solution that is a closest possible solution to generate motion for the robot <NUM> based on the inputs. In other words, the maneuver system <NUM> is configured to determine the compliance or actual feasibility of the robot <NUM> to perform the maneuvers <NUM> of the maneuver script <NUM>. This approach is particularly useful when a combination of maneuvers <NUM> occur in a maneuver script <NUM> at a given time instance (i.e., the user <NUM> has layered maneuvers <NUM>).

In some examples, the interface <NUM> functions as a means of communication between the UI <NUM> and the robot <NUM>. In other words, the interface <NUM> is a way for the entire maneuver script <NUM> to be communicated from the UI <NUM> to the robot <NUM>. In some implementations, the sequencer <NUM> interprets the maneuver script <NUM> from the interface <NUM> to understand the time sequence of when the robot <NUM> should attempt to execute a particular maneuver <NUM> of the maneuver script <NUM>. In some examples, the sequencer <NUM> operates as a schedule for the maneuvers <NUM> of the maneuver script <NUM>. In some configurations, the sequencer <NUM> operates by identifying the clock time 208a for the robot <NUM> and identifying the maneuver(s) <NUM> that should be running at the identified time. For instance, based on the time clock 208a of the robot <NUM>, the sequencer <NUM> identifies a particular slice ts of the maneuver script <NUM> and determines whether the robot <NUM> should be performing or attempting to perform a maneuver <NUM> at that time slice ts. When the sequencer <NUM> identifies that one or more maneuvers <NUM> are associated with the identified time slice ts, the sequencer <NUM> may communicate these maneuver(s) <NUM> to the dynamic planner <NUM>. For instance, <FIG> depicts the sequencer <NUM> with a clock symbol that identifies the current time 208a for the robot <NUM> and a time slice ts of the maneuver script <NUM> that corresponds to the current time 208a. Here, there are three maneuvers <NUM>-j that occur at that time slice ts (e.g., shown with a dotted box around the three maneuvers <NUM>-j).

The maneuver system <NUM> is configured to generate commands <NUM> to control motion of the robot <NUM> to move or to attempt to move based on the maneuver(s) <NUM> of the maneuver script <NUM>. The maneuver system <NUM> may use the dynamic planner <NUM> to generate all of or a portion of the commands <NUM>. In some examples, the dynamic planner <NUM> is configured to generate commands <NUM> for joints J of the robot <NUM> in order to control the robot's motion (also referred to as joint commands <NUM>). Here, the joint commands <NUM> may correspond to joint positions, joint forces (e.g., joint torques), joint angles, angular velocities related to joints J, etc. Controllers <NUM> use the dynamic planner <NUM> as a utility when the controllers <NUM> are activated by a maneuver <NUM> at a particular instance of time to determine the joint commands <NUM> (e.g., joint torques) to achieve or to attempt to achieve the movement of the maneuver(s) <NUM>. In examples where the robot <NUM> is a legged robot (e.g., a quadruped), the dynamic planner <NUM> is often used to coordinate leg maneuvers <NUM> since leg movement is a common type of control for a legged robot <NUM> (e.g., often layered together). With a legged robot <NUM>, the control system <NUM> may control stance leg(s) <NUM>ST with force control (e.g., joint torque) and swing leg(s) <NUM>SW by position (e.g., with joint angles and angular velocities).

In order to generate joint commands <NUM>, the dynamic planner <NUM> includes a solver <NUM>. Here, the solver <NUM> refers to an optimization model that incorporates various constraints <NUM> for the robot <NUM> and/or the surroundings of the robot <NUM> (e.g., structural constraints, physical-world constraints, user-defined constraints, etc.). In order to coordinate movement, the solver <NUM> of the dynamic planner <NUM> may receive or be preprogrammed to satisfy these constraints <NUM> such that the robot <NUM> can continue functionality (e.g., maintain its balance) while performing requested movements (e.g., requested through the maneuver script <NUM>). Some more specific examples of constraints <NUM> include range of motion constraints for the robot <NUM>, constraints based on structural dimensions of the robot <NUM> (e.g., leg lengths), estimates of friction on the robot <NUM> (e.g., at the feet <NUM> of the robot <NUM>), etc..

To generate joint commands <NUM>, the dynamic planner <NUM> receives information, such as the parameters <NUM>, regarding one or more maneuvers <NUM> from the maneuver script <NUM>. In some examples, the dynamic planner <NUM> identifies a cost <NUM> associated with a maneuver <NUM> (e.g., with each maneuver <NUM> received at the dynamic planner <NUM>). Here, the cost <NUM> refers to a value which designates the importance for the movement or a particular portion of the movement of a maneuver <NUM>. For instance, the cost <NUM> corresponds to an importance of a target event pose PT for the maneuver <NUM>. The target event pose PT generally refers to a desired body position or orientation for a given movement. In some implementations, each maneuver <NUM> received by the dynamic planner <NUM> includes a cost <NUM>. Here, as shown in <FIG>, the user <NUM> may define the cost <NUM> as a parameter <NUM> when the building the maneuver script <NUM>. In other words, the user <NUM> defines how critical a particular movement of a maneuver <NUM> is by assigning a value to the target event pose PT for the maneuver <NUM>.

By having costs <NUM> associated with maneuvers <NUM> that are input into the solver <NUM>, the solver <NUM> is configured to optimize multiple costs <NUM> to understand how to best to perform more than one maneuver <NUM> at a particular instance of time in the maneuver script <NUM>. In other words, when a first maneuver 210a and a second maneuver 210b occur at the same time instance, each maneuver 210a-b includes a cost <NUM>, 336a-b that informs the solver <NUM> what is important (e.g., to the user <NUM>) about each maneuver 210a-b. For instance, the first maneuver 210a is a footstep maneuver <NUM>f where the user <NUM> really wants to achieve a particular touchdown position for the foot <NUM> during the footstep (e.g., during swing), but does not care (or value) the movement speed during the footstep or the orientation of the foot <NUM> during the footstep maneuver <NUM>f. Here, the user <NUM> may set the cost <NUM> of the touchdown position for the footstep maneuver <NUM>f to reflect such high importance (e.g., a large cost value). In this example, when the user <NUM> layers an additional maneuver <NUM>, such as the body maneuver <NUM>b of sway, as a second maneuver 210b, the solver <NUM> may decide to reduce the amount of sway for this second maneuver 210b so that it does not compromise the accuracy of the touchdown position for the first maneuver 210a. Here, the amount of sway may have been designed as a parameter <NUM> at the UI <NUM>. In this scenario, the second maneuver 210b may be configured with a cost 336b that reflects a lower importance for the amount of sway. In this example, when each maneuver 210a-b has a high cost <NUM> that poses a potential conflict, the solver <NUM> may try to optimize the movement result to resolve the conflict or compromise between the conflicting commands.

In some configurations, maneuvers <NUM> that are hints <NUM>H are resolved prior to the dynamic planner <NUM>. In other words, the maneuver system <NUM> and/or the control system <NUM> determines whether the hint <NUM>H is compatible with other maneuvers <NUM> that occur at a same time instance. In these configurations, when a hint <NUM>H is compatible with maneuver(s) <NUM> occurring at the same time as the hint <NUM>H, the maneuver(s) <NUM> are modified to incorporate the movements of the hint <NUM>H or the parameters <NUM> of the hint <NUM>H. Based on this process, the dynamic planner <NUM> may receive a maneuver <NUM> already modified by a hint <NUM>H at the solver <NUM>.

In some examples, a controller <NUM> of the robot <NUM> executes a maneuver <NUM> without utilizing the dynamic planner <NUM>. In other words, the solver <NUM> that generates the joint commands <NUM> is not necessary for particular maneuvers <NUM>. For example, particular maneuvers <NUM> do not pose a joint control conflict. In other words, when an arm maneuver <NUM>a, which only affects the joints J of the arm <NUM>, and a leg maneuver <NUM>f, which does not impact the joints J of the arm <NUM>, occur at a same time instance, the controller <NUM> operating the arm maneuver <NUM>a may proceed in its own joint commands <NUM> for the arm <NUM> without utilizing the dynamic planner <NUM>. In some examples, maneuvers <NUM> bypass the dynamic planner <NUM> when optimization is not necessary (e.g., when only one maneuver <NUM> occurs without the potential of joint interference during execution of the movement(s) for that maneuver <NUM>). In some implementations, the maneuver system <NUM> is configured to not allow maneuvers <NUM> to bypass the dynamic planner <NUM>, even for basic maneuvers <NUM>. In this approach, using the dynamic planner <NUM> even for basic maneuvers <NUM> will help ensure the robot <NUM> obeys real-world constraints (e.g., constraints <NUM>) during movement for the maneuver <NUM>. This approach also enables the robot <NUM> to maintain balance and motion fluidity during movement.

While not all maneuvers <NUM> necessarily utilize the dynamic planner <NUM>, in some implementations, bypassing the dynamic planner <NUM> is not predicated on the controller(s) <NUM> using disjoint sets of joints J. Instead, in some examples, the maneuver system <NUM> may use tracks (or "layers") to avoid situations of multiple controllers <NUM> attempting to command the same joints J. In some instances, many, but not all, controllers <NUM> are implemented using the dynamic planner <NUM>. The dynamic planner <NUM> incorporates hints <NUM>H to avoid resolving multiple controllers <NUM> that want to command the same joints J.

In some implementations, a maneuver <NUM> is either a controller <NUM> or a hint <NUM>H. Optionally, the maneuver <NUM> can be both a controller <NUM> and a hint <NUM>H. Controllers <NUM> directly control joints J of the robot <NUM>. In the examples shown, hints <NUM>H are received by the controllers <NUM> as requests to modify the behavior of the respective controllers <NUM> in a specific way. Dance moves (e.g., maneuvers <NUM>) are assigned to one or more tracks (layers <NUM>). Which track(s) a dance move goes on is an inherent property of the type of that individual dance move, not something the user <NUM> selects. For example, a "Step" move may always goes on a legs track and a "Sway" move may always goes on a body track. Tracks define whether their dance moves are hints <NUM>H or controllers <NUM>. In some instances, all leg and arm moves are controllers <NUM> and all body moves are hints <NUM>H. If a dance move uses multiple tracks, it is a controller <NUM> if any of them are controller tracks. The dynamic planner <NUM> is a utility used by many of the leg-track controllers. It knows how to incorporate hints <NUM>H to modify its behavior.

In some implementations, to generate joint commands <NUM> for joints J of the robot <NUM>, the maneuver system <NUM> accounts for a current state <NUM> of the robot <NUM>. Here, the current state <NUM> of the robot <NUM> may refer to anything that is measured related to the robot <NUM> or the surroundings of the robot <NUM>. For instance, the current state <NUM> includes sensor data <NUM> from the sensor system <NUM> about the time when the robot <NUM> is attempting to perform a maneuver <NUM> (e.g., the time identified by the sequencer <NUM>). Some examples of the current state <NUM> include the position of the robot <NUM>, the velocity of the robot <NUM>, and/or the forces that the robot <NUM> is experiencing. When the maneuver system <NUM> accounts for the current state <NUM>, the maneuver system <NUM> (e.g., the dynamic planner <NUM>) may update constraints <NUM> based on the current state <NUM> to enable accurate optimization for the solver <NUM>.

<FIG> is an example arrangement of operations for a method <NUM> that generates a command to control motion of the robot <NUM>. At operation <NUM>, the method <NUM> receives a maneuver script <NUM> including a plurality of maneuvers <NUM> for a legged robot <NUM> to perform. Here, each maneuver <NUM> is associated with a cost <NUM>. At operation <NUM>, the method <NUM> identifies that two or more maneuvers <NUM> of the plurality of maneuvers <NUM> of the maneuver script <NUM> occur at the same time instance. At operation <NUM>, the method <NUM> determines a combined maneuver for the legged robot <NUM> to perform at the time instance based on the two or more maneuvers <NUM> and the costs <NUM> associated with the two or more maneuvers <NUM>. At operation <NUM>, the method <NUM> generates a joint command <NUM> to control motion of the legged robot <NUM> at the time instance, the joint command <NUM> commanding a set of joints J of the legged robot <NUM>. Here, the set of joints J correspond to the combined maneuver.

In some examples, the plurality of maneuvers <NUM> include a hint that corresponds to a body movement for a body <NUM> of the legged robot <NUM>. Here, the method <NUM> determines whether the hint is compatible with another maneuver <NUM> of the plurality of maneuvers <NUM> occurring at a same instance of time as the hint and modifies the other maneuver <NUM> to incorporate the body movement of the hint when the hint is compatible with the other maneuver <NUM>. In some implementations, receiving the maneuver script <NUM> includes receiving the maneuver script <NUM> from a user device <NUM> in communication with the data processing hardware. Here, the maneuver script <NUM> is defined by a user <NUM> of the user device <NUM> at a user interface <NUM> executing on the user device <NUM>. In some configurations, the method <NUM> synchronizes each dance move of the dance script with a beat of a song when the maneuver script <NUM> includes a dance script and each maneuver <NUM> includes a dance move.

<FIG> is schematic view of an example computing device <NUM> that may be used to implement the systems (e.g., the UI <NUM> and/or the maneuver system <NUM>) and methods (e.g., the method <NUM>) described in this document.

The computing device <NUM> includes a processor <NUM> (e.g., data processing hardware), memory <NUM> (e.g., memory hardware), a storage device <NUM>, a high-speed interface/controller <NUM> connecting to the memory <NUM> and high-speed expansion ports <NUM>, and a low speed interface/controller <NUM> connecting to a low speed bus <NUM> and a storage device <NUM>. Also, multiple computing devices <NUM> may be connected, with each device providing portions of the necessary operations (e.g., as a server bank, a group of blade servers, or a multiprocessor system).

For example, it may be implemented as a standard server 500a or multiple times in a group of such servers 500a, as a laptop computer 500b, as part of a rack server system 500c, or as part of the robot <NUM>.

Claim 1:
A method (<NUM>) comprising:
receiving, at data processing hardware, a maneuver script (<NUM>) comprising a plurality of maneuvers (<NUM>) for a legged robot (<NUM>) to perform, each maneuver (<NUM>) associated with a cost (<NUM>);
identifying, by the data processing hardware, that a first maneuver (<NUM>) and a second maneuver (<NUM>) of the plurality of maneuvers (<NUM>) of the maneuver script (<NUM>) occur at a same time instance;
determining, by the data processing hardware, a conflict between the first maneuver (<NUM>) and the second maneuver (<NUM>) in which both the first maneuver (<NUM>) and the second maneuver (<NUM>) command a first joint (J);determining, by the data processing hardware, a combined maneuver for the legged robot (<NUM>) to perform at the time instance based on the first maneuver (<NUM>) and the second maneuver (<NUM>) and the costs (<NUM>) associated with the first maneuver (<NUM>) and the second maneuver (<NUM>); and
generating, by the data processing hardware, a joint command (<NUM>) to control motion of the legged robot (<NUM>) at the time instance, the joint command (<NUM>) commanding a set of joints (J) including the first joint (J) of the legged robot (<NUM>), the set of joints (J) corresponding to the combined maneuver, the joint command (<NUM>) controlling motion of the first joint (J) to implement at least a portion of the combined maneuver.