Patent ID: 12190606

DETAILED DESCRIPTION OF EXAMPLE EMBODIMENTS

FIG.1is a schematic flow chart of an exemplary embodiment of method100. Using this method, primarily a figure of merit4ais determined for a relevance evaluation function4. This in turn makes it possible to evaluate a plausibility6of the output of ANN1.

In step110, images2as well as assignments3of these images2to classes3a-3cof a predetermined classification are provided. In step120, combinations made up of one image2and one assignment3each are processed with the aid of a relevance evaluation function4to obtain a relevance evaluation2aof image2. This relevance evaluation2aindicates which parts of image2have contributed, to what extent, to assignment3.

In step130, a further classifier5is trained to determine from one image2and a relevance evaluation2aascertained for this image, a reconstruction3′ of the assignment3to which this relevance evaluation2arelates. For example, during such a training, parameters such as weights, for instance, of classifier5may be optimized so that for images2used during the training, reconstruction3′ corresponds as well as possible to assignment3actually determined by ANN1. Based on the agreement between reconstructions3′ and actual assignments3which can be attained by this training, in step140, a figure of merit4ais ascertained for relevance evaluation function4.

As an example, according to block131, the quantity of available combinations of images2, assignments3and relevance evaluations2amay be split into a first subset of training data for further classifier5and a second subset of test data for further classifier5. According to block132, further classifier5may then be trained on the basis of the training data. The agreement between reconstructions3′ and actual assignments3may then be determined on the basis of the test data (block133). As explained previously, further classifier5is then unable to “fake” a learning success by overfitting with respect to the training data. In principle, however, all combinations of images2, assignments3and relevance evaluations2amay also be utilized as training data, and it is then possible, for example, to measure how good the agreement is on average between reconstructions3′ and actual assignments3.

Within box140, various exemplary embodiments are indicated for how with the aid of figure of merit4a, a relevance evaluation function4particularly suitable for the specific application may also be discovered at the same time.

According to block141, figure of merit4amay be determined for a selection of several candidate relevance evaluation functions4*. According to block142, one candidate relevance evaluation function4* having the best value of the figure of merit may then be selected as relevance evaluation function4.

According to block143, figure of merit4amay additionally be determined forthe identical mapping of image2, and/oran area filled with random values, and/oran area filled with one constant value, and/ora semantic segmentation2bof image2, and/oran edge detection2cfrom image2as location-resolved comparison relevance evaluation2a*.

According to block144, a relevance evaluation function4or candidate relevance evaluation function4*, whose figure of merit4ais poorer than figure of merit4adetermined for comparison relevance evaluation2a*, may then be rejected as implausible.

According to block145, a parameterized ansatz4′ having free parameters may be set up for relevance evaluation function4. According to block146, the parameters of this ansatz4′ may then be optimized with the objective that figure of merit4aof relevance evaluation function4will assume an extremum.

Relevance evaluation function4, and/or figure of merit4aof this relevance evaluation function, and/or a relevance evaluation2aascertained with this relevance evaluation function4may be used in step150to evaluate a plausibility6of the output of ANN1. However, this is optional. For instance, method100may also be utilized merely with the goal of discovering an optimal relevance evaluation function4.

FIG.2is a schematic flow chart of an exemplary embodiment of method200. This method200is based on the assumption that a suitable relevance evaluation function4is already implemented.

Analogous to step110of method100, in step210of method200, at least one image2, for which ANN1has determined an assignment3to one or more classes3a-3cof a predetermined classification, as well as assignment3determined by ANN1are provided.

Analogous to step120of method100, in step220of method200, a location-resolved relevance evaluation2aof image2is determined for the combination of image2and assignment3by utilizing relevance evaluation function4. This relevance evaluation2aagain indicates which parts of image2have contributed, to what extent, to assignment3.

In step230, a correlation7is determined between relevance evaluation2aon one hand and a semantic segmentation2bof image2, and/or an edge detection2cfrom image2, on the other hand. In step240, this correlation7is judged to be a measure for plausibility6of the output of ANN1.

This plausibility6, or perhaps correlation7directly, may then be compared in step250to a threshold value7a, and a system acting in at least partially automated fashion may be controlled to the effect that disadvantageous consequences of an incorrect assignment are reduced.

Within box250, various examples are indicated as to how this control may be implemented especially for vehicles.

According to block251, at least one additional physical sensor may be activated for observing the vehicle environment.

According to block252, the driving speed of a vehicle driving in at least partially automated fashion may be reduced. On the turnpike, for example, by way of precaution, the vehicle may be steered into the slower traffic in the right traffic lane.

According to block253, a driver assistance system, and or a system for the at least partially automated guidance of the vehicle may be completely or partially deactivated.

According to block254, a vehicle driving in at least partially automated fashion may be brought to a stop on a preplanned emergency-stop trajectory. Such an emergency-stop trajectory is kept available as standard in any system for the at least partially automated driving, in the event of a system failure.

FIG.3shows an example for how an attack by an “adversarial example”, thus, a manipulated image2, on an ANN1used as classifier is able to be detected using method200. In this example, image2shows a stop sign21, which with malicious intent, was provided with a sticker22. This sticker22is consciously formed with the intent that ANN1will classify stop sign21not as a stop sign, but rather as a 70 Km/h sign, for instance.

Should it succeed in “deceiving” ANN1in such a way, this implies that the area having sticker22has a particularly strong influence on assignment3determined by ANN1. This means that this area has an especially high weight in location-resolved relevance evaluation2acompared to the rest of image2. This is illustrated inFIG.3, in that only the area having sticker22is marked and provided with reference numeral2a.

On the other hand, edge detection2cfrom image2particularly highlights the features of stop sign21, while sticker22is only faintly discernible, if at all. Stickers22which are applied with malicious intent on signs, are intended precisely to stand out as little as possible visually, so that no one discovers and removes them.

Thus, exactly the features of stop sign21which especially stand out in edge detection2c, virtually do not come into play at all in location-resolved relevance evaluation2a. In the same way, sticker22, which is so important for relevance evaluation2a, virtually does not come into play at all in edge detection2c. Correlation7between relevance evaluation2aand edge detection2cis thus poor, which is able to be detected using method200.