Patent Description:
Vehicles are becoming equipped with increasingly advanced cruise control and pilot assist functions leading to semi-autonomous vehicles and ultimately to fully autonomous vehicles. Deploying an autonomous drive assist system or an autonomous drive system safely requires that the system can recognize the scenarios in which it can operate autonomously in a reliable manner. Accordingly, it is desirable to recognize any scenario which lies outside of an intended operation region. If a scenario outside of the intended operation region is recognized, the system may alert the driver that the driving mode of vehicle must be switched to manual mode.

However, due to the complexity of the operating environment for vehicles it is impossible to foresee and account for all unknown scenarios for an autonomous drive application. Hence, the systems cannot rely on specifying each operation region where the system is not allowed to operate.

Accordingly, there is a need for a reliable way of classifying the present scenario such that an appropriate driving mode for semi-autonomous or autonomous vehicle can be selected.

<CIT> discloses a neural network to be applied to recognize known and unknown environments of a vehicle by evaluating, using the neural network, differences between an original scene and its reconstruction image. An operation mode of the vehicle is selected based on the environmental information.

<NPL>, discloses an autonomous driving agent using an auto-encoder to learn the environment of a vehicle by evaluating a difference image which is calculated by background subtraction between two consecutive frames.

<NPL>, discloses an improved reconstruction-based auto-encoder to better detect out-of-distribution samples.

<NPL>, discloses a system that trains an inpainter to reconstruct road patches with frequent road patterns.

The disclosed subject matter generally relates to a method for selecting an operation mode among a plurality of operation modes for an at least partly self-driving vehicle.

The proposed method provides for selecting a driving mode based on, in contrast to prior art methods, concluding that a present scenario captured in a visual representation is part of an unknown scenario, i.e. that it is not recognized. If the captured visual representation is determined to be part of a known scenario, so-called inlier data representing a normal region of input data that can be recognized, the driving mode may be selected accordingly. However, if the visual representation is determined to be part of the unknown scenario, so-called outlier data that is at least partly un-recognized, another driving mode may be selected accordingly.

The proposed method provides the advantage that the unknown scenarios do not have to be recognized as such, it is sufficient to conclude that they do not belong to the known region of scenarios or data, i.e. that they are not part of the inlier data.

The above advantages are obtained by the features according to method claim <NUM>. The reconstructed visual representation being produced based on a machine learning algorithm applied to the visual representation. The difference values thus represent the deviations, in a pixel-by-pixel manner, between the visual representation and the reconstructed visual representation. A large deviation generally indicates outlier data.

However, to accurately determine if the visual representation should be considered outlier data or inlier data, a difference metric is constructed based on all the difference values that exceed a threshold value. In this way it is possible to classify the visual representation as being either part of a known scenario, i.e. inlier data, or part of an unknown scenario, based on how well the algorithm is able to reconstruct the visual representation. If the algorithm is not able to reconstruct the visual representation with sufficient accuracy this will be reflected in the difference metric which then indicates that the visual representation is not part of inlier data and should be classified as outlier data. Thereby, with the herein proposed method, it is not necessary to foresee and account for all unknown scenarios.

In embodiments, the difference metric is calculated based on a sum of the difference values that are above a threshold value. Thereby proving for a relatively straight forward way of constructing the difference metric.

The inventors also propose a control unit according to claim <NUM>, and a corresponding computer program product according to claim <NUM>.

The skilled person realize that different features of the present disclosure may be combined to create embodiments other than those described in the following, without departing from the scope of the present disclosure.

These and other aspects of the disclosed subject matter will now be described in more detail, with reference to the appended drawings showing example embodiments of the present disclosure, wherein:.

In the present detailed description, various embodiments of the present disclosure are described. However, embodiments of the present disclosure may be embodied in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided for thoroughness and completeness, and to fully convey the scope of the disclosure to the skilled person. In some instances, well known structures and devices are shown in block diagram form in order to avoid obscuring the novelty of the exemplary embodiments presented herein.

<FIG> is a box diagram illustrating a control unit <NUM> for selecting an operation mode among a plurality of operation modes for an at least partly self-driving vehicle based on evaluating a visual representation of the present scene of the vehicle captured by a scene capturing device <NUM>. The control unit <NUM> is configured to calculate a set of difference values between the pixel values of the visual representation and pixel values of a reconstructed visual representation, the reconstructed visual representation being produced based on a machine learning algorithm applied to the visual representation. The control unit selects the operation mode based on a difference metric being calculated based on the difference values that exceed a threshold value.

According to embodiments, the control unit <NUM> may be configured to control the at least partly self-driving vehicle to switch to the selected operation mode. The control unit may for example control a drive system <NUM> responsible for controlling the operation such as propulsion and steering of the vehicle to switch to the selected operation mode.

According to embodiments, the control unit <NUM> may be configured to control an alert device <NUM> in the at least partly self-driving vehicle to provide an alert to the driver before switching to the selected operation mode. The alert device may include one or more of an audio alert device, a haptic device, a visual alert device, etc..

<FIG> is a flow-chart of method steps according to embodiments of the present disclosure. In step S102, acquiring a visual representation of the present scene of the at least partly self-driving vehicle, wherein the visual representation is captured by a scene capturing device. In step S104 calculating a set of difference values between pixel values of the visual representation and pixel values of a reconstructed visual representation, the reconstructed visual representation being produced based on a machine learning algorithm applied to the visual representation. In step S106, selecting the operation mode based on a difference metric being constructed based on the difference values that exceed a threshold value.

The proposed method is based on the realization that by comparing the visual representation that is captured by a scene capturing device, with a reconstructed visual representation being a reconstruction of the visual representation based on a machine learning algorithm applied to the visual representation, it is possible to classify the visual representation as one of inliers, i.e. a known range of data, or as an outlier, i.e. unknown range of data. The machine learning algorithm have been trained to reconstruct visual representations related to only one of the operation modes. More precisely, the machine learning algorithm may have been trained on inlier data, that is, data in which one of the operation modes is safe to use. Once the comparison indicates a deviation of some predetermined degree, the control unit may identify the present visual representation to belong to the outlier range of data, i.e. to include data from the unknown range. This may trigger the control unit to select a different operation mode. The selectable operation modes may include at least a manual driving mode and an at least partly self-driving mode.

The machine learning algorithm is a deep neural network. The deep neural network uses an auto-encoder architecture, which is learned to reconstruct the visual representations to the perception layer of the network. The network is trained on the inlier data only and thus learns to reconstruct the corresponding inlier visual representations. When the algorithm is subjected to novel data, i.e. outlier data it is not able to reconstruct the visual representations to the same degree of accuracy. Hence the difference between the acquired visual representation and the reconstructed visual representation can be taken as a metric of how well a scenario fits into the inlier region. In this way the algorithm doesn't need to learn all unknown scenarios, it is enough that it can determine whether a scenario is in the inlier region.

Embodiments of the present disclosure provide for selecting whether a self-driving function of an at least partly self-driving vehicle should be kept on, or if the driver should take control of the vehicle. If outliers are confirmed, then the driver may be prompted to take control of the vehicle.

The difference metric is a measure of to what extent the visual representation is an outlier. The difference metric may directly or indirectly be a measure of how accurately the machine learning algorithm is able to reconstruct the visual representation. A poor reconstruction indicates that the visual representation is an outlier and an accurate reconstruction indicates that the visual representation is an inlier.

The threshold value is selected based on studying the distribution of the difference values for inlier data and known outlier data used for verification. The threshold value should be selected such that pixels in outlier regions can be identified. This will be further described with reference to <FIG>.

<FIG> conceptually exemplifies embodiments of the present disclosure. A scene capturing device may capture a visual representation <NUM>. In the visual representation <NUM>, there is shown a road surface <NUM> of a highway on which the at least partly self-driving vehicle from which the visual representation <NUM> was taken is driving, and a truck <NUM> is driving in front of and to the side of the at least partly self-driving vehicle. The visual representation <NUM> used as input to a machine learning algorithm <NUM>. The machine learning algorithm, which is a deep learning network, reconstructs the visual representation and produces a reconstructed visual representation <NUM> including a reconstructed truck <NUM>' and reconstructed road surface <NUM>'.

The machine learning algorithm have been taught on inlier data represented by e.g. empty highway scenarios. Therefore, the road surface <NUM>' is accurately reconstructed and the truck <NUM>' is not accurately reconstructed.

A set of difference values is calculated based on a difference between pixel values of the visual representation <NUM> and pixel values of the reconstructed visual representation <NUM>. In <FIG>, the difference values are shown, for illustrative purposes, as a difference visual representation <NUM>, i.e. an image <NUM>, including pixels of difference values. High difference values are represented in the difference visual representation <NUM> by pixels of high intensity. For example, the road surface <NUM>" is in a region of low difference values which represents data well reconstructed by the machine learning algorithm. However, the truck <NUM>" constitutes high difference values thereby indicating that the visual representation <NUM> belongs to the outlier region not well reconstructed by the machine learning algorithm <NUM>. Thus, the visual representation <NUM> may be determined to be an outlier whereby a manual driving mode may be selected, and the driver is prompted to take over control of the vehicle. Preferably, an alert is provided to the driver indicative of a change from one operation mode to another one of the operation modes.

Accordingly, in one embodiment, calculating the set of difference values includes a pixel by pixel subtraction of the pixel values of one of the visual representation and the reconstructed visual representation from the other one of the visual representation and the reconstructed visual representation. As described below, calculating the set of difference values may include to square the pixel by pixel subtraction.

The selection of the operation mode is based on whether a difference metric exceeds a threshold value. The difference metric is calculated based on a sum of the difference values that exceed the threshold value. Accordingly, by summing the difference values, e.g. the pixel values of the difference visual representation <NUM>, it is concluded whether the driver should take control of the vehicle or not, or if another operation mode should be selected.

Mathematically, the preferred difference metric, L, may be calculated by:
<MAT>
where xij is the pixel value of one of the the visual representation and the reconstructed visual representation, x̂ij is the pixel value of the other one of the the visual representation and the reconstructed visual representation, x<NUM> is the threshold value, and θ is the Heaviside step function such that only difference values in the difference visual representation with values that exceed the threshold value x<NUM>, contributes to the difference metric L. The difference metric L is the sum of the individual pixel-by-pixel subtractions squared.

The threshold value, x<NUM>, may be selected based on studying the distribution of the difference values for inlier data and outlier data. <FIG> illustrates two histograms, a histogram <NUM> representing the distribution of difference values for inlier visual representations, and a histogram <NUM> representing the distribution of difference values for outlier visual representations. As can be seen, the histogram <NUM> includes a higher number of large difference values compared to the histogram <NUM>, i.e. the "tail" of the histogram <NUM> is longer than the "tail" of the histogram <NUM>. Accordingly, the threshold value <NUM> may be selected to be at a difference value so that as much as possible of the tail of the histogram <NUM> fall above the threshold without including much of the inlier histogram <NUM>. This type of evaluation may be performed for multiple inlier and outlier data sets in order to tune the threshold value such that an acceptable level of false negative and false positives of outlier classifications is obtained. Note that the number axis in the histograms <NUM> and <NUM> have been cut in order to better visualize the tails.

The operation mode is selected by comparing the difference metric to a metric threshold. The metric threshold is empirically determined based on applying the algorithm to test data. Turning to <FIG>, which illustrates two histograms, a first histogram <NUM> represents difference metrics for visual representation data known to be inlier visual representations, and a second histogram <NUM> represents difference metrics for visual representation data known to be outlier visual representations. The P is a normalized frequency/occurrence of the difference metrics, commonly used for histograms. The histograms <NUM>, <NUM> represent multiple visual representations for obtaining statistical accuracy. The metric threshold <NUM> should be selected somewhere between the two histograms such that the outlier visual representations can be detected. For example, the metric threshold <NUM> may be selected such that the difference metrics for the inlier visual representations, i.e. the histogram <NUM> are below the metric threshold <NUM>.

<FIG> is a flow-chart of method steps according to example embodiments of the present disclosure. In addition to steps S102 and S104 also shown in <FIG>, there is a step S205 including detecting unknown objects in the visual representation based on the positions of the pixels associated with the difference values that exceed the threshold value. In step S206, selecting the operation mode based on a difference metric being constructed based on the difference values that exceed the threshold value, and based on the location of the detected unknown object. Turning to <FIG> again, by analyzing the location of the high pixel values in a difference visual representation <NUM> representing the truck <NUM>", in other words the pixels in the visual representation associated with the difference values that exceed the threshold value, it is possible to determine the position of the object <NUM> captured by the associated pixels. In this way, it may for example be determined that an outlier detected in an opposite lane can be disregarded and a self-driving operation mode can be maintained.

The control unit may have access to a memory storage device having thereon stored computer program code for executing the machine learning algorithm.

In addition, there is provided a computer program product comprising a computer readable medium having stored thereon computer program means for selecting an operation mode for an at least partly self-driving vehicle based on evaluating a visual representation of the present scene of the vehicle captured by a scene capturing device, wherein the computer program product comprises: code for calculating a set of difference values between the pixel values of the visual representation and pixel values of a reconstructed visual representation, the reconstructed visual representation being produced based on a machine learning algorithm applied to the visual representation; and code for selecting the operation mode based on a difference metric being constructed based on the difference values that exceed a threshold value.

The computer program product includes code for reconstructing the visual representation based on the machine learning algorithm for producing the reconstructed visual representation.

A system <NUM> according to embodiments of the present disclosure comprises a scene capturing device <NUM> configured to acquire visual representations of the vicinity of the at least partly self-driving vehicle; and a control unit <NUM> as described above. <FIG> illustrates an at least partly self-driving vehicle <NUM> comprising such a system <NUM>. The scene capturing device <NUM> is here shown as a forward-looking image capturing device of the at least partly self-driving vehicle. However, the scene capturing device may equally well be a <NUM> camera to get a better understanding of the traffic scenario. Here the forward-looking image capturing device <NUM> captures a visual representation including the truck <NUM> which may be considered to contribute to outlier data, as conceptually described with reference to <FIG>.

"Acquiring" a visual representation may equally well be to receive or to obtain a visual representation.

An acquired visual representation is an image captured by an image capturing device. The acquired visual representation is considered an input visual representation to the algorithm. Further, a reconstructed visual representation is a reconstructed image, and a difference visual representation is a difference image.

The communication between the control unit and other devices, systems, or components may be hardwired or may use other known electrical connection techniques, or wireless networks, known in the art such as via CAN-buses, Bluetooth, Wifi, Ethernet, <NUM>, <NUM>, <NUM>, etc..

A control unit may include a microprocessor, microcontroller, programmable digital signal processor or another programmable device, as well as be embedded into the vehicle/power train control logic/hardware. The control unit may also, or instead, include an application-specific integrated circuit, a programmable gate array or programmable array logic, a programmable logic device, or a digital signal processor. Where the control unit includes a programmable device such as the microprocessor, microcontroller or programmable digital signal processor mentioned above, the processor may further include computer executable code that controls operation of the programmable device. The control unit may comprise modules in either hardware or software, or partially in hardware or software and communicate using known transmission buses such as CAN-bus and/or wireless communication capabilities.

A control unit of the present disclosure is generally known as an ECU, electronic control unit.

The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measured cannot be used to advantage.

Computer-readable media include computer-readable storage media, which corresponds to a tangible medium such as data storage media, or communication media including any medium that facilitates transfer of a computer program from one place to another, e.g., according to a communication protocol.

Disk and disc, as used herein, includes compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), and Blu-ray disc, where disks usually reproduce data magnetically, while discs reproduce data optically with lasers.

Instructions may be executed by one or more processors, such as one or more digital signal processors (DSPs), general purpose microprocessors, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), complex programmable logic devices (CPLDs), or other equivalent integrated or discrete logic circuitry. In addition, in some aspects, the functionality described herein may be provided within dedicated hardware and/or software modules.

The techniques of this disclosure may be implemented in a wide variety of devices or apparatuses, including an integrated circuit (IC) or a set of ICs (e.g., a chip set).

Claim 1:
A method for selecting an operation mode among a plurality of operation modes for an at least partly self-driving vehicle (<NUM>), the method comprising:
acquiring (S102) an image (<NUM>) of the present scene of the at least partly self-driving vehicle, wherein the image is captured by a scene capturing device (<NUM>);
calculating (S104) a set of difference values between the pixel values of the acquired image and pixel values of a reconstructed image, the reconstructed image being produced by a deep neural network using an autoencoder architecture applied to the acquired image, the deep neural network being trained on inlier data to reconstruct images of empty highway scenarios;
selecting (S106; S205) the operation mode based on comparing a difference metric to a metric threshold empirically determined to distinguish inlier data representing empty highway scenarios from outlier data representing unknown scenarios, the difference metric being constructed based on a sum of difference values that exceed a threshold value, wherein the selectable operation modes include manual driving mode and an at least partly self-driving mode, the at least partly self-driving mode is selected in case of the difference metric being below the metric threshold; and
detecting (S205) positions of unknown objects in the acquired image based on the positions of the pixels in the acquired image associated with the difference values that exceed the threshold value, wherein the operation mode is selected also based on the location of the detected unknown object.