Patent ID: 12202519

DETAILED DESCRIPTION

As autonomous driving moves forward to real world application, more and more attention is paid to the planning part of the system, and more complex driving policies need to be employed to allow negotiation, adaptation to vastness of road scenarios and cooperation with other road users. Allowing autonomous vehicle to negotiate with other road users and act in an intelligent, complex manner in various road scenarios, may be provided using data driven approaches and artificial intelligence techniques, like Reinforcement Learning (RL), machine learning techniques (wherein Reinforcement Learning is a subfield for intelligent agent creation).

However, like all machine-learning techniques, Reinforcement Learning agents always learn only to some extent, raising concerns regarding safety, which is priority in safety-critical systems like Autonomous Driving (AD). One way of dealing with that issue may be to punish the agent for causing an accident or dangerous situations during the training phase. Although to assure safety on sensible level, differences in magnitudes between safety-related objectives and effectiveness, efficiency, and comfort-related objectives may yield an agent caring only about safety aspects.

Another approach is to embed a RL-based block in deterministic safety envelope, which may validate the RL-based block outputs and may suppress illegal and dangerous actions. This however may bring a lot of confusion to the RL-agent itself, because it may make it a lot harder to understand effects of its own action which is filtered by safety envelope. This may also slow down the training process. Furthermore, effects of own actions from the agent perspective are not clear anymore, as they are filtered out by safety envelope afterwards.

According to various embodiments, safety formalization framework (which may be referred to as Safety Framework) may be utilized (or combined) with a customized neural network architecture, allowing to predefine available actions which are legal and safe and then assuring that only those action may be picked by RL agent.

The safety framework may introduce a formalization of safety for autonomous driving application and may define a set of constrains determining safe way of driving, for example based on an AD system.

According to various embodiments, a behavioral and/or trajectory planning system may be provided which is responsible for defining a driving policy of an autonomous driving vehicle. The policy may include a control interface by which the behavior of the autonomous vehicle (for example car) may be changed.

According to various embodiments, the Safety Framework and a general common-sense rules may be used to define possible, legal, and safe action which may be marked as “available” to the RL-based planning block. Then, the neural network may produce a probability distribution only over available actions, for example using mechanisms as described herein for various embodiments, for example using a dot product of an output of a fully connected layer of a neural network and a mask which is determined based on a subset of potential maneuvers).

In an example, the RL-based planning block may be responsible for picking high-level semantic actions in form of a maneuver to execute. This block may be followed by one or more other blocks responsible for execution of such a maneuver.

In this example, a set of possible maneuvers may be defined as a list of three actions:Follow Lane: the vehicle may be requested to follow its current lane while keeping safe distance to cars in front;Lane Change Left: the vehicle may be requested to change lane to one on the left side; andLane Change Right: the vehicle may be requested to change lane to one on the right side.

Then, a mechanism of marking which maneuvers are currently available for execution may be provided. This mechanism may be based on one or more of the following considerations:Machine state based: while performing a lane change maneuver to one side, in the next time instance, it may only be allowed to pick a lane change maneuver to the same side again or to abort the maneuver by switching to follow lane, while not allowing to jump straight to a lane change maneuver to other side (because this may lead to a zig-zag route).Road based: if the lane on a given side is unavailable (for example because it does not exist or because there is a solid line between the lanes), lane change to this side may be marked as unavailable.Object based: if other vehicles are present in the target lane, and according to the safety framework evaluation, lane change to this lane may not be performed in a safe manner, the lane change maneuver to this side may be marked as unavailable.

After deciding which maneuvers are available, this information may be used to define a probability distribution over only available actions.

FIG.1shows an illustration100of a methodology based on a neural network architecture for maneuver determination according to various embodiments. The methodology illustrated inFIG.1may provide an attention mechanism (which may be combined with a safety framework) which may be used to select a maneuver from available maneuvers list.

For example, a neural network may analyze a perception of a scene to generate a maneuver to execute. After prior processing, the agent at some stage of processing may have a basic understanding of what to do. The control head which may be responsible for maneuver selection is illustrated inFIG.1. At input to this part, a general hidden state may represent the current desire of policy, followed by a fully connected (FC) layer102, responsible for parsing the general desire to the desired maneuver strategy. Then, an available maneuvers mask may be used to pick related embeddings106(for example set of weights) based on the available maneuvers104. Then a dot product108, followed by a softmax activation function110may result in a probability distribution over only available actions. As a final step, by sampling from the resulting probability distribution or a picking action highest ranked maneuver may be determined, as illustrated by block112, to obtain the maneuver114to be performed.

Following a similar approach, other control interfaces, after discretization, may be considered. Those interfaces may be used as a separate control schemes or in parallel to each other, creating general planning system. Some among others are:Lane Bias: the RL-based planning block may decide on a desired lane bias in a lane, while understanding what kind of biasing is safe (for example due to assurance of minimal distance to other road users based on Safety Framework).Distance to car in front: the RL-agent may decide on a distance to a car in front to keep, while the safety framework defines what is a minimal safe distance in the current circumstances and marks only distances greater than minimal one as available.Velocity set point: the RL-agent may be responsible for picking desired speed, while the safety framework along with traffic sign recognition defines a maximum speed allowed and marks only speeds below defined threshold as available.

The processing flow within the neural network itself may stay unchanged, but the rules used for determining which lane bias, distance or velocity are available may be changed.

To determine what lane-biases are available, some small obstacles on a road may be analyzed, based on a maneuver (wherein for example lane biases in opposite direction to lane change may be forbidden) or based on other vehicles which may partially occupy our lane.

For velocity (in both absolute and additive way of providing it) determination may be based on actual speed limit, leading vehicle distance and speed and velocities of nearby car(s) while performing other maneuvers.

As described above, according to various embodiments, a safe reinforcement learning-based driving policy by deterministic action space predefinition may be provided.

FIG.2shows a flow diagram200illustrating a method for determining a maneuver to be executed by an autonomous vehicle according to various embodiments. At202, a plurality of potential maneuvers may be determined. At204, a subset of the potential maneuvers may be determined based on a pre-determined condition. At206, a mask may be determined based on the subset. At206, an output of a fully connected layer of a neural network may be determined. At208, a dot product of the output of the fully connected layer and the mask may be determined. At210, the maneuver to be executed may be determined based on the dot product.

According to various embodiments, the pre-determined condition may include or may be at least one of lanes availability, legality of lane changes, objects appearance, or time constrains.

According to various embodiments, the mask may include a value of “1” for safe actions and a value of “0” for non-safe actions.

According to various embodiments, the dot product may be followed by a softmax activation function.

According to various embodiments, the softmax activation function may be followed by sampling.

According to various embodiments, the softmax activation function may be followed by an argmax function.

According to various embodiments, the neural network may be used in a reinforcement learning method.

According to various embodiments, the subset of the potential maneuvers may include or may be possible, safe, and legal maneuvers.

According to various embodiments, it may be determined whether a potential maneuvers is possible, safe, and legal based on at least one of available maneuvers, available distances, current speed, possible lane biases, and/or possible velocities.

According to various embodiments, the potential maneuvers may include or may be all maneuvers which the vehicle is able to perform.

According to various embodiments, at least one of a lane bias, a distance to another vehicle or a velocity set point may be determined for the potential maneuvers.

Each of the steps202,204,206,208,210and the further steps described above may be performed by computer hardware components.

FIG.3shows a maneuver determination system300according to various embodiments. The maneuver determination system300may include a potential maneuvers determination circuit302, a subset determination circuit304, a mask determination circuit306, an output determination circuit308, a dot product circuit310, and a maneuver determination circuit312. The maneuver determination system300may be a system for determining a maneuver to be executed by an autonomous vehicle.

The potential maneuvers determination circuit302may be configured to determine a plurality of potential maneuvers.

The subset determination circuit304may be configured to determine a subset of the potential maneuvers based on a pre-determined condition.

The mask determination circuit306may be configured to determine a mask based on the subset.

The output determination circuit308may be configured to determine an output of a fully connected layer of a neural network.

The dot product circuit310may be configured to determine a dot product of the output of the fully connected layer and the mask.

The maneuver determination circuit312may be configured to determine the maneuver to be executed based on the dot product.

The potential maneuvers determination circuit302, the subset determination circuit304, the mask determination circuit306, the output determination circuit308, the dot product circuit310, and the maneuver determination circuit312may be coupled with each other, e.g. via an electrical connection314, such as e.g. a cable or a computer bus or via any other suitable electrical connection to exchange electrical signals.

A “circuit” may be understood as any kind of a logic implementing entity, which may be special purpose circuitry or a processor executing a program stored in a memory, firmware, or any combination thereof.

FIG.4shows a computer system400with a plurality of computer hardware components configured to carry out steps of a computer implemented method for determining a maneuver to be executed by an autonomous vehicle according to various embodiments. The computer system400may include a processor402, a memory404, and a non-transitory data storage406.

The processor402may carry out instructions provided in the memory404. The non-transitory data storage406may store a computer program, including the instructions that may be transferred to the memory404and then executed by the processor402.

The processor402, the memory404, and the non-transitory data storage406may be coupled with each other, e.g. via an electrical connection408, such as e.g. a cable or a computer bus or via any other suitable electrical connection to exchange electrical signals.

The terms “coupling” or “connection” are intended to include a direct “coupling” (for example via a physical link) or direct “connection” as well as an indirect “coupling” or indirect “connection” (for example via a logical link), respectively.

It will be understood that what has been described for one of the methods above may analogously hold true for the maneuver determination system300and/or for the computer system400.