POTENTIOMETERS AS POSITION SENSOR IN DEXTEROUS ROBOTICS FINGERS

Provided are mechanisms for spatially decoupling an actuator from a sensor measurement point.

BACKGROUND

The present disclosure relates generally to robotics and, more specifically, to sensing position of joints.

2. Description of the Related Art

Dynamic mechanical systems are often controlled with computational processes. Examples include robots, industrial processes, life support systems, and medical devices. Generally, such a process takes input from sensors indicative of state of the dynamic mechanical system and its environment and determines outputs that serve to control various types of actuators within the dynamic mechanical system, thereby changing the state of the system and potentially its environment. In recent years, control of dynamic mechanical systems has been improved using machine learning, and potential applications for dynamic mechanical systems, like robots, are numerous.

SUMMARY

Some applications include, in robotic systems that operate under tight volumetric constraints at a point of articulation, a compact force transmission means and a compact sensing means. Examples of a compact force transmission means may include, but are not limited to, a tendon, like a cable, and compact sensing means may include, but are not limited to, a position sensor.

An example embodiment of a tendon may couple a member having a point of articulation, such as at a joint, to an actuator at a point of actuation that drives the tendon (e.g., pulls on the tendon). The actuator (e.g., point of actuation) may be disparately located from the member and the point of articulation.

An example embodiment of a sensor may be positioned at or coupled to a point of articulation, such as at or coupled to a joint from which a member articulates. Example embodiments of a sensor may generate a feedback signal indicative of movement or position of the member coupled to the joint. Some embodiments of a sensor may be housed within the joint and detect rotation of the member about the joint (e.g., with a single degree of freedom). Some embodiments of a sensor may be housed within the joint and detect rotation of the member about the joint (e.g., with multiple degrees of freedom).

Some embodiments implement a process to control an actuator disparately located from a point of articulation. While actuation may be effectively physically separated from the point of articulation, such as by a tendon or other linkage, a machine learning model, like a control model, may rely on precise knowledge of position parameters corresponding to the point of articulation. In some embodiments, an encoder obtains feedback data from a sensor coupled to the point of articulation, from which the encoder may determine a state vector including information indicative of position of a joint or member corresponding to the point of articulation. A control model may output one or more values by which to adjust the actuator based on the state vector and compare a resulting state based on updated feedback data relative to a desired state to determine an amount of position change caused by the one or more output values.

Some aspects include a system, including: one or more processors; one or more inertial measurement units; and memory storing instructions that when executed by the processors cause the processors to effectuate operations of the above-mentioned applications.

DETAILED DESCRIPTION

Many dynamic mechanical systems are subject to tight volumetric constraints, for example, at the point of actuation. As robots become more feature rich and capable of more complex tasks, these improvements are often achieved by virtue of increased complexity and number of joints included in a robot. Increases in numbers of components and features for performing more varied tasks, even in a relatively large robot, are expected to create even further crowded in scarce space on robots and other dynamic mechanical systems. Further, positioning heavy actuators on distal portions of a kinematic chain can increase stress, add to inertia to moving parts, and make it harder to dampen undesirable oscillations.

Robotic controls often rely on feedback indicative of robot state (e.g., joint positions) to control actuators, and in many cases, that feedback comes from the actuator itself, e.g., a step count of a stepper motor or a position reading from a potentiometer integrated into a servo motor. Stepper motors, servo motors, and other actuators that are capable of providing precise and consistent feedback data are often too large or costly to incorporate within one or more joints, members, or other components at a point of articulation due to tight volumetric constraints. Even where stepper motor or other motor or actuator could be produced to satisfy tight volumetric constraints, this is typically achieved at the expense of other performance metrics, such as maximum torque or durability, not to mention cost of bespoke designs.

However, when the actuator is shifted up the kinematic chain (e.g., all the way to a base) to mitigate some of the issues noted above, this can make the feedback from the actuator indicative of joint position less accurate. Error compounds down the kinematic chain, making feedback from the actuator a poor proxy for direct measurement of joint position at the joint itself (Again, none of which is to suggest that these or any other approaches are disclaimed.) For example, while a stepper motor or other actuation component traditionally used in such applications may be driven with a high degree of precision based on similarly precise feedback data from a stepper motor (e.g., which typically justifies their utilization despite their expense relative to other components), moving the actuation point disparate from the articulation point may drastically increase error in feedback data, often both initially and overtime, e.g., due to wear, losses, tolerance, or other issues. Lack of precision and changes in feedback data can cause difficulties in determining (e.g., during training) associated parameters of control models or cause associated parameters of trained control models to be or become suboptimal (e.g., causing errors).

Some embodiments mitigate these and other issues with shifting actuators up the kinematic chain by co-locating rotary potentiometers (or other position sensors) at the joint axes and integrating their output into the control loop for the driving an actuator (e.g., in some cases, integrated potentiometers in servos or step counters in steppers may be omitted, disregarded, or supplemented with the on-joint measurements). Indeed, some embodiments may use a motor without integrated position sensing (e.g., non-servo, non-stepper motor) with a control circuit taking the potentiometer position as its control signal. By spatially decoupling the encoder from the driving motor, some embodiments effectively created a physically distributed servo device, which is expected to be particularly well suited for (but not limited to, which is not to suggest other descriptions are limiting) cable driven control systems. In some cases, such “distributed servos” are expected to be less expensive and more precise than systems exclusively using integrated servos (e.g., with feedback sensing, motor control, and the motor in a single housing dedicated to the servo). (Again, none of which is to suggest that these or any other approaches are disclaimed.)

As discussed in more detail below in connection withFIGS.2-4, a robot system (e.g., the robot system302(FIG.3), the robot system219(FIG.2A), etc. may be trained using machine learning (e.g., reinforcement learning) to perform tasks. Performing a task in the real world presents a challenge to reinforcement learning because of the large state space (e.g., the large number of actions that a robot can perform, the many positions or locations a robot can be in may be too numerous, etc.). To reduce the state space (e.g., which may make it easier to train a robot system), two portions of a robot hand may be joined such that joint motion is coupled together (e.g., one portion of a finger moves when a second portion of a finger moves). A physical mechanism may be used to mechanically couple distal finger joint motion together. For example, an s-bar may be used to join two joints together. Joining two joints using a mechanism (e.g., an s-bar) may allow the finger to still curl around an object (e.g., which may allow the robot to grasp objects), while removing the need for independent actuation (e.g., due to volumetric constrains or other factors, such as cost or complexity). This may reduce the cost and complexity for machine learning training (e.g., as described in connection withFIGS.2-4). In some embodiments, alternative implementations such as a rubberized, or flexible, linkage may be used, for example, to allow for selective compliance at one or more joints (e.g., the outermost joints of the robot hand). A rubberized, or other flexible linkage may allow the robot to create a more robust grasp.

FIG.1Ashows a cross-sectional view of an example robot hand100(e.g., the pinky of the robot hand100). An S-bar102may be used to mechanically couple a distal phalange109with an intermediate phalange108. The s-bar may be attached to the distal phalange109at location101and may be attached to the intermediate phalange108at location103. A thumb107and a wrist140are shown for reference. The s-bar102is attached to the pinky of the robot hand100. The s-bar102may be made out of a rubberized material (or other flexible material), metal, plastic, fiberglass, or a variety of other materials.

FIG.1Bshows an additional view of the robot hand100. The s-bar102may be attached to a finger of the hand at location101and location102.

FIG.1Cshows an additional view of an index finger141of the robot hand100. The s-bar105may be attached to the index finger141of the hand at location104and location106.

FIG.1Dshows an additional view of the robot hand100with the palm142of the robot facing up. S-bar mechanisms110-113may be used to mechanically couple corresponding intermediate and distal phalanges on fingers of the hand.

FIG.1Eshows an additional view of an s-bar mechanism114which may be used to mechanically couple an intermediate and distal phalange of a thumb of the robot hand100.

FIG.1Fshows a zoomed-in view of the hand100. An s-bar110may be used to mechanically couple finger joint motion of the hand100.

FIG.1Gshows a zoomed-out view of the robot hand100. The joint motion of one or more fingers may be mechanically coupled using an s-bar mechanism (e.g., such as the s-bar120).

FIG.1Hshows an angled view of the pinky of the robot hand100. An s-bar121may be used to mechanically couple joint motion in the pinky. The s-bar121may be attached to the pinky of the robot hand100at a location that is behind a potentiometer122.

FIG.1Ishows a top-down view of the robot hand100. A portion of s-bars130-133can be seen via the top-down view shown.

In connection with or separate from the above aspects pertaining to an s-bar, an actuator may be coupled to a member or a joint (or joints) like that described above, or another joint, to actuate one or more members. Due to volumetric space constraints, the actuator (e.g., like a motor) may be located disparately from a point of articulation (e.g., a joint) and coupling may be provided via a linkage, like a tendon, such as a cable. Other example linkages may include one or more rigid bars or one or more gears. As a result, the point of actuation (e.g., location of the actuator) may be disparately located from the point of articulation (e.g., location of the joint that is actuated). In order to address issues like those noted above, among others, example embodiments disclosed herein provide a sensor located at the point of articulation to provide feedback data corresponding to the actuator. In other words, embodiments spatially decouple an actuator from a sensor measurement point corresponding to the actuator (e.g., for encoding and processing within a control loop).

FIG.1A, as mentioned above, show a robot hand of a robotic system. The robot hand may have human-like proportions, and thus may be representative of an application in which one or more components of a robotic system operate under tight volumetric constraints at the point of actuation (e.g., joints of a biomimetic humanoid robot hand). In many cases, it is infeasible to include servos or motors directly at the final joints (e.g., either within the joints or members) of a kinematic chain. Example embodiments may locate servos or motors (e.g., actuators) spatially separate from the points of actuation, such as via means of compact force transmission, such as a cable-driven tendon.

For example, a tendon may be coupled to member108(e.g., like a component of a finger) to cause the member108to rotate via joint170A relative to another member (e.g.,167, corresponding to a hand/palm), such as to grasp an object. The tendon may be coupled to the member108at a point along its length, or at the joint170A. Example embodiments may include a plurality of joints, e.g.,170A,170B,170C to which members in a chain of members are coupled. Tasks assigned to a robot may require actuation of one or more members in a chain.

While actuation can be effectively physically separated from the joint in question, such as via one or more tendons, machine learning algorithms by which actions of a robot to perform a task are controlled, may require precise knowledge of physical joint position (e.g., to determine information about the members coupled by the joint). Traditionally, by employing a motor at the joint or member coupled to the joint, an in-servo encoder of the motor at the driven joint may provide precise feedback data. Relocation of the motor to a spatially distanced location from the driven joint, as explained above, may diminish the precision (or accuracy) of feedback data.

Some example embodiments may implement a sensor, like a position sensor, within or coupled to the joint by which amount of rotation and thus position of a joint or member coupled to the joint may be determined. One example position sensor may be a rotary potentiometer disposed at a joint axis. Other examples of position sensors may include stretch potentiometers, capacitive-based position sensors, or optical position sensors.

In some example embodiments, a position sensor, may output a signal or reading indicative of a given position or orientation or by which a given position or orientation may be determined. For example, a sensor may output signals (e.g., a voltage indicative of position) that correspond to joint or member position measurements, and those measurements may be provided as feedback data into the control loop for driving an actuator spatially distanced from the articulation point.

Some embodiments may implement an actuator, e.g., a motor, without a servo, or that is otherwise less precise that those previously employed (e.g., to reduce cost) because the output of the sensor at the joint may be obtained as a measurement of position from which a control signal for a control circuit of a motor. In other words, the sensor positioned at the point of articulation may provide an encoder with signals by which control signals for driving a motor may be determined. By removing the need for high-precision, pre-made servos of a motor with an additional layer of control logic based on measurement signals at the joint being implemented above the motor/drive circuitry, system cost may be reduced while the physically separate sensor device may maintain high-precision measurements at the point of articulation for system control.

FIG.1B, as mentioned above, shows an additional view of the robot hand of a robotic system. Also shown is an example joint170. The example joint170may be subject to relatively tight volumetric space constraints and driven via a tendon coupled to a disparately located actuator. The actuator (not shown) may drive (e.g., pull) on a tendon coupled to member108(or component of the joint170coupled to member108). Driving a tendon may thus cause the member108to move, such as by rotation181around the joint.

Some example embodiments of a joint170may include a housing171for a sensor. Thus, for example, a sensor, like a position sensor, may determine a position of a member108as it rotates181in relation to the joint170(or another member, e.g.,167). Some example embodiments of a housing171of a joint170for a sensor may include a shape corresponding to that of a body of the sensor or an index point177by which the orientation of a sensor may be fixed within the housing171. Some embodiments of a housing171may include one or more channels179by which sensor leads (e.g., like wires) may be guided out from the joint170. In some example embodiments, a member108may be coupled to or include a shaft interface175by which it is coupled to and rotates within the joint170. The shaft may be supported within the joint170by one or more bushings. In some alternative embodiments, the member108may be coupled to the bushings and the shaft may be coupled to another member169to which member108rotates in relation.

FIG.1Jshows an example view of a finger of a robot system and joints upon which example techniques for determining position of space constrained joints may be implemented in accordance with some example embodiments. As shown, a finger (or other appendage) of a robot may have a number of joints171A-171C having respective members that may be driven to rotate181A-181C around their respective joint axis. Control of the finger (or other appendage) may rely on accurate position information corresponding to the joints171A-171Cs for various tasks, such as grabbing or otherwise manipulating an object. Example joints171A-171C may be subject to tight space constraints that are prohibitive to the inclusion of actuators at respective points of joint articulation.

FIG.1Kshows an example view of a joint and position sensor by which example techniques for determining position of space constrained joints may be implemented in accordance with some example embodiments.

Member108may rotate181in relation to joint170or another member167. Member108may be coupled to a shaft interface175which rotates with the member108. The shaft interface175may include splines, or a cut face, to which a sensor component may be coupled. In other embodiments, the shaft interface175may be a component of the sensor and coupled to the member108, such as via one or more splines or a cut face. In either example, the member108and the shaft interface175may rotate relative to a sensor housing171.

A body191of the sensor may be disposed with the sensor housing171. In some examples, the sensor housing171is shaped or includes an index177to retain the body191of the sensor in position when the shaft interface175rotates.

Some example embodiments of a sensor may include an arm193coupled to the shaft interface175. The arm193may be conductive (e.g., efficiently convey an electrical current) or include a conductive portion194at an interface194that engages a track192of resistive material (e.g., resists an electrical current relative to the conductive material). The track192of resistive material may be a carbon-based or other semi-resistive material.

Considering a track192of resistive material having a resistance R between a first lead196A and a second lead196B (e.g., like a V+ voltage and a V ground, respectively), interface194may intersect with track192at a given position based on a position (e.g., rotation) of member108to provide an output voltage tap (e.g., measurement) based on input voltage and the RA and RB values (e.g., where RA+RB=R of the track192) resulting from the position of the interaction.

Interface194of the arm may be coupled to a third lead195, which may be an output, such as an output indicative of a position of the interface194along the track. For example, a Vout of the sensor may be measured at lead195based on a Vin of the voltage across leads196A and196B and the position of the conductive interface194along the track102. E.g.:

where the resistance value Rb changes based on position of the conductive interface194because of rotation of the member108and shaft interface175. Rb may change linearly in accordance with a ratio of resistance to rotation (although logarithmic or other scaling could be utilized). Thus, different positions (e.g., rotation) of the shaft interface175may be related to each other based on their respective Vout values.

FIG.1Lshows an example view of a position sensor for determining position of space constrained joints in accordance with some example embodiments.

Other example sensor types may be utilized to output a position measurement.FIG.1Killustrates an example member154B and interface shaft154B, which may rotate relative to a sensor152coupled to a joint151. Rotation of the interface shaft154B may cause a corresponding rotation of a dial153. Sensor152may read, e.g., optically, magnetically, capacitively or via a conductive interface, a value indicative of a position of the dial153and thus the shaft interface154A and corresponding member154B based on their rotation182relative to the joint151.

In some example embodiments, the dial153may include a code (or codes) or pattern that may be read by a sensor154to determine a position of the dial. For example, the dial153may include a pattern of lines corresponding to copper tracks etched in a PCB strip. The sensor154may also include a pattern of lines corresponding to copper tracks etched in a PCB. The sensor154may be positioned proximate to the dial153and the patterns may form a variable capacitor. As the dial153moves relative to the sensor154, the sensor154may detect changes in capacitance to determine a measurement indicative of the position of the dial153relative to the sensor154.

In another example embodiments, the dial153may include a pattern of lines or dots or other markings that may be read optically. For example, the sensor154may be an optical sensor and track movement of the dial153or read a pattern to determine a position of the dial153relative to the sensor154.

FIG.1Mshows an example view of a position sensor for determining position of space constrained joints in accordance with some example embodiments.

In some examples, one or more sensors152may be employed to track movement of a member154B within a joint151with multiple degrees of freedom. Rather than a shaft/bushing type interface, an example member154B may include a ball154A interface with a joint151and rotate with multiple degrees of freedom within the single joint. In some examples, the ball154A may be engraved with a pattern (e.g., on its surface) by which one or more sensor152may optically, capacitively, or magnetically track its position with multiple degrees of freedom. In some examples, such as for optical sensors152, a position and orientation of one or more points of a pattern on the ball detected by one or more sensors may be read to determine position and orientation of the member154B. For example, a pattern of three or more points, like a constellation, may be analyzed to determine position and orientation information.

FIG.2Ashows an example computing system for training robots to perform tasks. The system200may include a robot216. The robot216may include any component of the robot system302discussed below in connection withFIG.3. The robot216may include a hand such as the robot hand100or fingers discussed above in connection withFIGS.1A-1M. In some example embodiments, S-bars discussed herein (e.g., with reference toFIGS.1A-1I) may be used to reduce state space or increase the efficiency of training one or more machine learning models discussed in connection withFIGS.2-3. An encoder which determines vectors corresponding to robot state within a state space may take input from sensors (e.g., as discussed with reference toFIGS.1A,1B and1J-1Mand elsewhere herein) that are disposed at points of articulation that are physically distanced from the actuators that drive the articulated components. The robot216may be an anthropomorphic robot (e.g., with legs, arms, hands, or other parts), like those described in the application incorporated by reference. The robot may be an articulated robot (e.g., an arm having two, six, or ten degrees of freedom, etc.), a cartesian robot (e.g., rectilinear or gantry robots, robots having three prismatic joints, etc.), Selective Compliance Assembly Robot Arm (SCARA) robots (e.g., with a donut shaped work envelope, with two parallel joints that provide compliance in one selected plane, with rotary shafts positioned vertically, with an end effector attached to an arm, etc.), delta robots (e.g., parallel link robots with parallel joint linkages connected with a common base, having direct control of each joint over the end effector, which may be used for pick-and-place or product transfer applications, etc.), polar robots (e.g., with a twisting joint connecting the arm with the base and a combination of two rotary joints and one linear joint connecting the links, having a centrally pivoting shaft and an extendable rotating arm, spherical robots, etc.), cylindrical robots (e.g., with at least one rotary joint at the base and at least one prismatic joint connecting the links, with a pivoting shaft and extendable arm that moves vertically and by sliding, with a cylindrical configuration that offers vertical and horizontal linear movement along with rotary movement about the vertical axis, etc.), self-driving car, a kitchen appliance, construction equipment, or a variety of other types of robots. The robot216may include one or more cameras, joints, servomotors, stepper motors, pneumatic actuators, or any other component discussed in U.S. patent application Ser. No. 16/918,999, filed 1 Jul. 2020, titled “Artificial Intelligence-Actuated Robot,” which is incorporated by reference in its entirety. The robot216may communicate with the agent215, and the agent215may be configured to send actions determined via the policy222. The policy222may take as input the state (e.g., a vector representation generated by the encoder model203) and return an action to perform.

The robot216may send sensor data to the encoder model203, e.g., via the agent215. The encoder model203may take as input the sensor data from the robot216. The encoder model203may use the sensor data to generate a vector representation (e.g., a space embedding) indicating the state of the robot. The encoder model203may be trained via the encoder trainer204. The encoder model may use the sensor data to generate a space embedding (e.g., a vector representation) indicating the state of the robot or the environment around the robot periodically (e.g., 30 times per second, 10 times per second, every two seconds, etc.). A space embedding may indicate a current position or state of the robot (e.g., the state of the robot after performing an action to turn a door handle. A space embedding may reduce the dimensionality of data received from sensors. For example, if the robot has multiple color 1080p cameras, touch sensors, motor sensors, or a variety of other sensors, then input to an encoder model for a given state of the robot (e.g., output from the sensors for a given time slice) may be tens of millions of dimensions. The encoder model may reduce the sensor data to a space embedding in an embedding space (e.g., a space between 10 and 2000 dimensions in some embodiments). Distance between a first space embedding and a second space embedding may preserve the relative dissimilarity between the state of a robot associated with the first space embedding and the state of a robot (which may be the same or a different robot) associated with the second space embedding.

The anomaly detection model209may receive vector representations from the encoder model203and determine whether each vector representation is anomalous or not. Although only one encoder model203is shown inFIG.2A, there may be multiple encoder models. A first encoder model may send space embeddings to the anomaly detection model209and a second encoder model may send space embeddings to other components of the system200.

The dynamics model212may be trained by the dynamics trainer213to predict a next state given a current state and action that will be performed in the current state. The dynamics model may be trained by the dynamics trainer213based on data from expert demonstrations (e.g., performed by the teleoperator).

The actor-critic model206may be a reinforcement learning model. The actor-critic model206may be trained by the actor-critic trainer207. The actor-critic model206may be used to determine actions for the robot216to perform. For example, the actor-critic model206may be used to adjust the policy by changing what actions are performed given an input state.

The actor-critic model206and the encoder model203may be configured to train based on outputs generated by each model206and model203. For example, the system200may adjust a first weight of the encoder model203based on an action determined by a reinforcement learning model (e.g., the actor-critic model206). Additionally or alternatively, the system200may adjust a second weight of the reinforcement learning model (e.g., the actor-critic model206) based on the state (e.g., a space embedding) generated via the encoder model203.

The reward model223may take as input a state of the robot216(e.g., the state may be generated by the encoder model203) and output a reward. The robot216may receive a reward for completing a task or for making progress towards completing the task. The output from the reward model223may be used by the actor-critic trainer207and actor-critic model206to improve ability of the model206to determine actions that will lead to the completion of a task assigned to the robot216. The reward trainer224may train the reward model223using data received via the teleoperation system219or via sampling data stored in the experience buffers226. The teleoperation system219may be the teleoperation system304discussed below in connection withFIG.3. In some embodiments, the system200may adjust a weight or bias of the reinforcement learning model (e.g., the actor-critic model206), such as a deep reinforcement learning model, in response to determining that a space embedding (e.g., generated by the encoder model203) corresponds to an anomaly. Adjusting a weight of the reinforcement model may reduce a likelihood of the robot of performing an action that leads to an anomalous state.

The experience buffers226may store data corresponding to actions taken by the robot216(e.g., actions, observations, and states resulting from the actions). The data may be used to determine rewards and train the reward model223. Additionally or alternatively, the data stored by the experience buffers226may be used by the actor-critic trainer to train the actor-critic model206to determine actions for the robot216to perform. The teleoperation system219may be used by the teleoperator220to control the robot216. The teleoperation system219may be used to record demonstrations of the robot performing the task. The demonstrations may be used to train the robot216and may include sequences of observations generated via the robot216(e.g., cameras, touch sensors, sensors in servomechanisms, or other parts of the robot216).

One or more machine learning models discussed herein may be implemented (e.g., in part), for example, as described in connection with the machine learning model242ofFIG.2B. With respect toFIG.2B, machine learning model242may take inputs244and provide outputs246. In one use case, outputs246may be fed back to machine learning model242as input to train machine learning model242(e.g., alone or in conjunction with user indications of the accuracy of outputs246, labels associated with the inputs, or with other reference feedback and/or performance metric information). In another use case, machine learning model242may update its configurations (e.g., weights, biases, or other parameters) based on its assessment of its prediction (e.g., outputs246) and reference feedback information (e.g., user indication of accuracy, reference labels, or other information). In another example use case, where machine learning model242is a neural network and connection weights may be adjusted to reconcile differences between the neural network's prediction and the reference feedback. In a further use case, one or more neurons (or nodes) of the neural network may require that their respective errors are sent backward through the neural network to them to facilitate the update process (e.g., backpropagation of error). Updates to the connection weights may, for example, be reflective of the magnitude of error propagated backward after a forward pass has been completed. In this way, for example, the machine learning model242may be trained to generate results (e.g., response time predictions, sentiment identifiers, urgency levels, etc.) with better recall, accuracy, and/or precision.

In some embodiments, the machine learning model242may include an artificial neural network. In such embodiments, machine learning model242may include an input layer and one or more hidden layers. Each neural unit of the machine learning model may be connected with one or more other neural units of the machine learning model242. Such connections can be enforcing or inhibitory in their effect on the activation state of connected neural units. Each individual neural unit may have a summation function which combines the values of one or more of its inputs together. Each connection (or the neural unit itself) may have a threshold function that a signal must surpass before it propagates to other neural units. The machine learning model242may be self-learning or trained, rather than explicitly programmed, and may perform significantly better in certain areas of problem solving, as compared to computer programs that do not use machine learning. During training, an output layer of the machine learning model242may correspond to a classification, and an input known to correspond to that classification may be input into an input layer of machine learning model during training. During testing, an input without a known classification may be input into the input layer, and a determined classification may be output. For example, the classification may be an indication of whether an action is predicted to be completed by a corresponding deadline or not. The machine learning model242trained by the ML subsystem314may include one or more embedding layers at which information or data (e.g., any data or information discussed above in connection withFIGS.1-3) is converted into one or more vector representations. The one or more vector representations of the message may be pooled at one or more subsequent layers to convert the one or more vector representations into a single vector representation.

The machine learning model242may be structured as a factorization machine model. The machine learning model242may be a non-linear model and/or supervised learning model that can perform classification and/or regression. For example, the machine learning model242may be a general-purpose supervised learning algorithm that the system uses for both classification and regression tasks. Alternatively, the machine learning model242may include a Bayesian model configured to perform variational inference, for example, to predict whether an action will be completed by the deadline. The machine learning model242may be implemented as a decision tree and/or as an ensemble model (e.g., using random forest, bagging, adaptive booster, gradient boost, XGBoost, etc.).

FIG.3shows an example computing system300for using machine learning to train robots (e.g., the robot system302, the robot216, etc.) to perform tasks. The computing system300may include a robot system302, a teleoperation system304, or a server306. The robot system302may include a communication subsystem312, a machine learning (ML) subsystem314, and sensors316.

At least some of the sensors316may have an architecture like that of example sensors described herein, may provide position information corresponding to joints of the robot, and may be spatially decoupled from the actuators that control movement of the joints.

The ML subsystem314may include a plurality of machine learning models. For example, the ML subsystem314may pipeline an encoder and a reinforcement learning model that are collectively trained with end-to-end learning, the encoder being operative to transform relatively high-dimensional outputs of a robot's sensor suite into lower-dimensional vector representations of each time slice in an embedding space, and the reinforcement learning model being configured to update setpoints for robot actuators based on those vectors. Some embodiments of the ML subsystem314may include an encoder model, a dynamic model, an actor-critic model, a reward model, an anomaly detection model, or a variety of other machine learning models (e.g., any model described in connection withFIG.2A-2B, or ensembles thereof). One or more portions of the ML subsystem314may be implemented on the robot system302, the server306, or the teleoperation system304. Although shown as distinct objects inFIG.3, functionality described below in connection with the robot system302, the server306, or the teleoperation system304may be performed by any one of the robot system302, the server306, or the teleoperation system304. The robot system302, the server306, or the teleoperation system304may communicate with each other via the network350.

FIG.4is a physical architecture block diagram that shows an example of a computing device (or data processing system) by which some aspects of the above techniques may be implemented. Various portions of systems and methods described herein, may include or be executed on one or more computer systems similar to computing system1000. Further, processes and modules described herein may be executed by one or more processing systems similar to that of computing system1000.

Computing system1000may include one or more processors (e.g., processors1010a-1010n) coupled to system memory1020, an input/output I/O device interface1030, and a network interface1040via an input/output (I/O) interface1050. A processor may include a single processor or a plurality of processors (e.g., distributed processors). A processor may be any suitable processor capable of executing or otherwise performing instructions. A processor may include a central processing unit (CPU) that carries out program instructions to perform the arithmetical, logical, and input/output operations of computing system1000. A processor may execute code (e.g., processor firmware, a protocol stack, a database management system, an operating system, or a combination thereof) that creates an execution environment for program instructions. A processor may include a programmable processor. A processor may include general or special purpose microprocessors. A processor may receive instructions and data from a memory (e.g., system memory1020). Computing system1000may be a uni-processor system including one processor (e.g., processor1010a), or a multi-processor system including any number of suitable processors (e.g.,1010a-1010n). Multiple processors may be employed to provide for parallel or sequential execution of one or more portions of the techniques described herein. Processes, such as logic flows, described herein may be performed by one or more programmable processors executing one or more computer programs to perform functions by operating on input data and generating corresponding output. Processes described herein may be performed by, and apparatus can also be implemented as, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). Computing system1000may include a plurality of computing devices (e.g., distributed computer systems) to implement various processing functions.

I/O device interface1030may provide an interface for connection of one or more I/O devices1060to computer system1000. I/O devices may include devices that receive input (e.g., from a user) or output information (e.g., to a user). I/O devices1060may include, for example, graphical user interface presented on displays (e.g., a cathode ray tube (CRT) or liquid crystal display (LCD) monitor), pointing devices (e.g., a computer mouse or trackball), keyboards, keypads, touchpads, scanning devices, voice recognition devices, gesture recognition devices, printers, audio speakers, microphones, cameras, or the like. I/O devices1060may be connected to computer system1000through a wired or wireless connection. I/O devices1060may be connected to computer system1000from a remote location. I/O devices1060located on remote computer system, for example, may be connected to computer system1000via a network and network interface1040.

Network interface1040may include a network adapter that provides for connection of computer system1000to a network. Network interface may1040may facilitate data exchange between computer system1000and other devices connected to the network. Network interface1040may support wired or wireless communication. The network may include an electronic communication network, such as the Internet, a local area network (LAN), a wide area network (WAN), a cellular communications network, or the like.

I/O interface1050may be configured to coordinate I/O traffic between processors1010a-1010n, system memory1020, network interface1040, I/O devices1060, and/or other peripheral devices. I/O interface1050may perform protocol, timing, or other data transformations to convert data signals from one component (e.g., system memory1020) into a format suitable for use by another component (e.g., processors1010a-1010n). I/O interface1050may include support for devices attached through various types of peripheral buses, such as a variant of the Peripheral Component Interconnect (PCI) bus standard or the Universal Serial Bus (USB) standard.

Embodiments of the techniques described herein may be implemented using a single instance of computer system1000or multiple computer systems1000configured to host different portions or instances of embodiments. Multiple computer systems1000may provide for parallel or sequential processing/execution of one or more portions of the techniques described herein.

In this patent, to the extent any U.S. patents, U.S. patent applications, or other materials (e.g., articles) have been incorporated by reference, the text of such materials is only incorporated by reference to the extent that no conflict exists between such material and the statements and drawings set forth herein. In the event of such conflict, the text of the present document governs, and terms in this document should not be given a narrower reading in virtue of the way in which those terms are used in other materials incorporated by reference.