Patent ID: 12246450

DETAILED DESCRIPTION OF EXAMPLE EMBODIMENTS

In model-based reinforcement learning, an approximate model {tilde over (p)} of transition probabilities or deterministic transitions of a real system p are learned. That is, the learned model takes as input the current state and control action, and returns either the next state or a distribution over the next state. This model can be consequently used to optimize a policy using existing policy optimization methods in conjunction with {tilde over (p)}. The proposed on-policy corrections take a fixed or learned approximation to {tilde over (p)}, and extend it with a state and/or time-dependent term to decrease the model predictions on-policy, that is under actions selected by the current control policy π.

Shown inFIG.1is a flow-chart diagram of an embodiment of the method for learning a policy for controlling a robot.

The method starts with Initializing (S1) policy (πθ) and transition dynamics model ({tilde over (p)}), which predicts a next state (st+1) of the environment and/or the agent in case the agent carries out an action (a) depending on both a current state (st) and said action (a).

Thereupon follows a loop until a termination condition is fulfilled:

The first step of the loop is recording (S2) at least one episode of interactions of the agent with its environment following policy (πθ). Subsequently, the recorded episodes are added (S3) to a set of training data (Denv)

After that, the step optimizing (S4) is carried out. Here, the transition dynamics model ({tilde over (p)}) is optimized based on the training data (Denv) such that the transition dynamics model ({tilde over (p)}) predicts the next states of the environment depending on the states and actions contained in the training data (Denv).

The global part g of the model {tilde over (p)} can be learned using any existing method, including approximate inference, methods that minimize prediction errors, and method to optimize long-term prediction errors. The correction term dtis optimized with respect to eq. 5 above.

After finishing step S4, step S5follows. This step comprises optimizing policy (πθ) parameters (θ) based on the training data (Denv) and the transition dynamics model ({tilde over (p)}) by optimizing a reward over at least one episode by following the policy (πθ). Given the transition dynamics model ({tilde over (p)}) of step S4, any policy-optimization method can be used. Examples include soft actor-critic, stochastic value gradients, proximal policy optimization and maximum a-posteriori policy optimization, among many others.

If step S5has been finished and the termination condition is not fulfilled, the loop can be repeated.

If the loop has terminated, the resulting optimized policy can be used to compute a control signal for controlling a physical system, e.g., a computer-controlled machine, a robot, a vehicle, a domestic appliance, a power tool, a manufacturing machine, or an access control system. It does so by learning a policy for controlling the physical system and then operating the physical system accordingly. Generally speaking, a policy obtained as described above interacts with any kind of system. As such the range of application is very broad. In the following some applications are exemplarily described.

Shown inFIG.2is one embodiment of an actuator10in its environment20. Actuator10interacts with a control system40. Actuator10and its environment20will be jointly called actuator system. At preferably evenly spaced distances, a sensor30senses a condition of the actuator system. The sensor30may comprise several sensors. An output signal S of sensor30(or, in case the sensor30comprises a plurality of sensors, an output signal S for each of the sensors) which encodes the sensed condition is transmitted to the control system40. Possible sensors include but are not limited to: gyroscopes, accelerometers, force sensors, cameras, radar, lidar, angle encoders, etc. Note that oftentimes sensors do not directly measure the state of the system but rather observe a consequence of the state, e.g., a camera detects an image instead of directly the relative position of a car to other traffic participants. However, it is possible to filter the state from high-dimensional observations like images or lidar measurements.

Thereby, control system40receives a stream of sensor signals S. It then computes a series of actuator control commands A depending on the stream of sensor signals S, which are then transmitted to actuator10.

Control system40receives the stream of sensor signals S of sensor30in an optional receiving unit50. Receiving unit50transforms the sensor signals S into states s. Alternatively, in case of no receiving unit50, each sensor signal S may directly be taken as an input signal s.

Input signal s is then passed on to policy60, which may, for example, be given by an artificial neural network.

Policy60is parametrized by parameters ϕ, which are stored in and provided by parameter storage St1.

Policy60determines output signals y from input signals s. The output signal y may be an action a. Output signals y are transmitted to an optional conversion unit80, which converts the output signals y into the control commands A. Actuator control commands A are then transmitted to actuator10for controlling actuator10accordingly. Alternatively, output signals y may directly be taken as control commands A.

Actuator10receives actuator control commands A, is controlled accordingly and carries out an action corresponding to actuator control commands A. Actuator10may comprise a control logic which transforms actuator control command A into a further control command, which is then used to control actuator10.

In further embodiments, control system40may comprise sensor30. In even further embodiments, control system40alternatively or additionally may comprise actuator10.

In one embodiment policy60may be designed signal for controlling a physical system, e.g., a computer-controlled machine, a robot, a vehicle, a domestic appliance, a power tool, a manufacturing machine, or an access control system. It does so by learning a policy for controlling the physical system and then operating the physical system accordingly.

In still further embodiments, it may be envisioned that control system40controls a display10ainstead of an actuator10.

Furthermore, control system40may comprise a processor45(or a plurality of processors) and at least one machine-readable storage medium46on which instructions are stored which, if carried out, cause control system40to carry out a method according to one aspect of the invention.

FIG.3shows an embodiment in which control system40is used to control an at least partially autonomous robot, e.g. an at least partially autonomous vehicle100.

Sensor30may comprise one or more video sensors and/or one or more radar sensors and/or one or more ultrasonic sensors and/or one or more LiDAR sensors and or one or more position sensors (like, e.g., GPS). Some or all of these sensors are preferably but not necessarily integrated in vehicle100.

Alternatively or additionally sensor30may comprise an information system for determining a state of the actuator system. One example for such an information system is a weather information system which determines a present or future state of the weather in environment20.

For example, using input signal s, the policy60may for example control the at least partially autonomous robot to achieve a predefined goal state. Output signal y controls the at least partially autonomous robot.

Actuator10, which is preferably integrated in vehicle100, may be given by a brake, a propulsion system, an engine, a drivetrain, or a steering of vehicle100. Preferably, actuator control commands A may be determined such that actuator (or actuators)10is/are controlled such that vehicle100avoids collisions with objects in the environment of the at least partially autonomous robot.

Preferably, the at least partially autonomous robot is an autonomous car. A possible description of the car's state can include its position, velocity, relative distance to other traffic participants, the friction coefficient of the road surface (can vary for different environments e.g. rain, snow, dry, etc.). Sensors that can measure this state include gyroscopes, angle encoders at the wheels, camera/lidar/radar, etc. The reward signal for this type of learning would characterize on how well a pre-computed trajectory, a.k.a. reference trajectory, is followed by the car. The reference trajectory can be determined by an optimal planner. Actions for this systems can be a steering angle, brakes and/or gas. Preferably, the break pressure or the steering angle is outputted by the policy, in particular such that a minimal braking distance is achieved or to carry out an evasion maneuver, as a (sub-) optimal planner would do it.

It is noted that for this embodiment, the policy can be learned for controlling dynamics and/or stability of the at least partially autonomous robot. For example if the robot is in a safety critical situation, the policy can control the robot to maneuver it out of said critical situation, e.g. by carrying out an emergency break. The policy can then output a value characterizing a negative acceleration, wherein the actor is then controlled depending on said value, e.g. breaks with a force related to the negative acceleration.

In further embodiments, the at least partially autonomous robot may be given by another mobile robot (not shown), which may, for example, move by flying, swimming, diving or stepping. The mobile robot may, inter alia, be an at least partially autonomous lawn mower, or an at least partially autonomous cleaning robot.

In a further embodiment, the at least partially autonomous robot may be given by a gardening robot (not shown), which uses sensor30, preferably an optical sensor, to determine a state of plants in the environment20. Actuator10may be a nozzle for spraying chemicals. An actuator control command A may be determined to cause actuator10to spray the plants with a suitable quantity of suitable chemicals.

In even further embodiments, the at least partially autonomous robot may be given by a domestic appliance (not shown), like, e.g., a washing machine, a stove, an oven, a microwave, or a dishwasher. Sensor30, e.g., an optical sensor, may detect a state of an object which is to undergo processing by the household appliance. For example, in the case of the domestic appliance being a washing machine, sensor30may detect a state of the laundry inside the washing machine. Actuator control signal A may then be determined depending on a detected material of the laundry.

Shown inFIG.4is an embodiment in which control system40is used to control a manufacturing machine11, e.g. a punch cutter, a cutter or a gun drill) of a manufacturing system200, e.g. as part of a production line. The control system40controls an actuator10which in turn control the manufacturing machine11.

Sensor30may be given by an optical sensor which captures properties of e.g. a manufactured product12. Policy60may determine depending on a state of the manufactured product12an action to manipulate the product12. Actuator10which controls manufacturing machine11may then be controlled depending on the determined state of the manufactured product12for a subsequent manufacturing step of manufactured product12. Or, it may be envisioned that actuator10is controlled during manufacturing of a subsequent manufactured product12depending on the determined state of the manufactured product12.

A preferred embodiment for manufacturing relates to autonomously (dis-)assemble certain objects by robotics. State can be determined depending on sensors. Preferably, for assembling objects the state characterizes the robotic manipulator itself and the objects that should be manipulated. For the robotic manipulator, the state can consist of its joint angles and angular velocities as well as the position and orientation of its end-effector. This information can be measured by angle encoders in the joints as well as gyroscopes that measure the angular rates of the robot joints. From the kinematic equations, it is possible to deduct the end-effectors position and orientation. Instead, it is also possible to utilize camera images or lidar scans to infer the relative position and orientation to the robotic manipulator. The reward signal for a robotic task could be for example split into different stages of the assembly process. For example when inserting a peg into a hole during the assembly, a suitable reward signal would be to encode the peg's position and orientation relative to the hole. Typically, robotic systems are actuated via electrical motors at each joint. Depending on the implementation, the actions of the learning algorithms could therefore be either the required torques or directly the voltage/current applied to the motors.

Shown inFIG.5is an embodiment in which control system40is used for controlling an automated personal assistant250. Sensor30may be an optic sensor, e.g. for receiving video images of a gestures of user249. Alternatively, sensor30may also be an audio sensor e.g. for receiving a voice command of user249.

Control system40then determines actuator control commands A for controlling the automated personal assistant250. The actuator control commands A are determined in accordance with sensor signal S of sensor30. Sensor signal S is transmitted to the control system40. For example, policy60may be configured to, e.g., determine an action depending on the state characterizing a gesture recognition, which can be determined by an algorithm to identify a gesture made by user249. Control system40may then determine an actuator control command A for transmission to the automated personal assistant250. It then transmits said actuator control command A to the automated personal assistant250.

For example, actuator control command A may be determined in accordance with the identified user gesture recognized by classifier60. It may then comprise information that causes the automated personal assistant250to retrieve information from a database and output this retrieved information in a form suitable for reception by user249.

In further embodiments, it may be envisioned that instead of the automated personal assistant250, control system40controls a domestic appliance (not shown) controlled in accordance with the identified user gesture. The domestic appliance may be a washing machine, a stove, an oven, a microwave or a dishwasher.

Shown inFIG.6is an embodiment in which control system controls an access control system300. Access control system may be designed to physically control access. It may, for example, comprise a door401. Sensor30is configured to detect a scene that is relevant for deciding whether access is to be granted or not. It may for example be an optical sensor for providing image or video data, for detecting a person's face.

Shown inFIG.7is an embodiment in which control system40controls a surveillance system400. This embodiment is largely identical to the embodiment shown inFIG.5. Therefore, only the differing aspects will be described in detail. Sensor30is configured to detect a scene that is under surveillance. Control system does not necessarily control an actuator10, but a display10a. For example, the machine learning system60may determine a classification of a scene, e.g. whether the scene detected by optical sensor30is suspicious. Actuator control signal A which is transmitted to display10amay then e.g. be configured to cause display10ato adjust the displayed content dependent on the determined classification, e.g. to highlight an object that is deemed suspicious by machine learning system60.

Shown inFIG.8is an embodiment of a control system40for controlling an imaging system500, for example an MRI apparatus, x-ray imaging apparatus or ultrasonic imaging apparatus. Sensor30may, for example, be an imaging sensor. Policy60may then determine based on its input state the an action characterizing a trajectory to take a recoding of the imaging system500.

The term “computer” covers any device for the processing of pre-defined calculation instructions. These calculation instructions can be in the form of software, or in the form of hardware, or also in a mixed form of software and hardware.

It is further understood that the procedures cannot only be completely implemented in software as described. They can also be implemented in hardware, or in a mixed form of software and hardware.