IDENTIFYING CHARACTERISTICS OF A SCENE, WITH HIGH LEVEL OF SAFETY INTEGRITY, BY COMPARING SENSOR DATA WITH AN EXPECTATION

A method for evaluating spatially resolved actual sensor data acquired using at least one sensor. The method includes: ascertaining a location and an orientation of the sensor at the time of acquiring the sensor data; retrieving a spatially resolved expectation from a spatially resolved map on the basis of the location and the orientation of the sensor; checking to what extent the actual sensor data are consistent with the expectation; at least with respect to the locations for which the actual sensor data are consistent with the expectation, determining that the scene observed by the sensor has a characteristic stored in the map in conjunction with the expectation.

FIELD

The present invention relates to identifying characteristics of a scene observed by means of at least one sensor, for example for the at least partially autonomous control of a vehicle or robot.

BACKGROUND INFORMATION

Driving assistance systems and systems for the at least partially automated guidance of vehicles or robots sense the environment of the vehicle or robot by means of one or more sensors and ascertain therefrom a plan for the future behavior of the vehicle or robot. Neural networks are frequently used to ascertain such a plan, or an environmental representation as a pre-product for the plan. German Patent Application No. DE 10 2018 008 685 A1 describes a method for training a neural network for determining a path prediction for a vehicle.

SUMMARY

The present invention provides a method for evaluating spatially resolved actual sensor data acquired by means of at least one sensor. In particular, this sensor may, for example, be a mobile sensor carried by a person, a robot or any land vehicle, watercraft or aircraft. However, the sensor may also, for example, be a stationary sensor that monitors a busy intersection, for example. The term “spatially resolved actual sensor data” is understood to mean any sensor data that are assigned to specific locations during their capture. For example, radar data and lidar data can be present as point clouds of measured values that are assigned to points in three-dimensional space from which the respectively used scanning radiation was reflected. Images assign intensity values to the pixels of the sensor used to sense them, the pixels usually being arranged in a regular grid. Depending on the perspective from which the image was acquired, and the optics used, each pixel in turn corresponds to a location in space from which the respective incident light comes.

According to an example embodiment of the present invention, a location and an orientation of the sensor are ascertained at the time the sensor data are acquired. This process is also called “registration” or “localization” of the sensor in the physical world. For this purpose, any system can be used individually or in combination. For example, navigation systems on the basis of radio signals emitted by satellites or terrestrial stations can be used. Alternatively or in combination, an inertial navigation system may be used, for example. If the sensor is a stationary sensor, its registration or localization is particularly simple because its location and its orientation are known in advance.

According to an example embodiment of the present invention, on the basis of the location and the orientation of the sensor, a spatially resolved expectation for the actual sensor data is retrieved from a spatially resolved map. This expectation is stored in the spatially resolved map in association with at least one characteristic of the observed scene. If the scene observed by means of at least one sensor actually has this characteristic stored in the map, it is expected that the actual sensor data are consistent with the expectation retrieved from the spatially resolved map.

Conversely, this means that, if the actual sensor data are consistent with the expectation, the characteristic stored in the map is also present. Checking whether this characteristic exists is precisely the aim of the method.

It is therefore checked to what extent the actual sensor data are consistent with the expectation. At least with respect to the locations for which this is the case, it is determined that the scene observed by means of the at least one sensor has the characteristic stored in the map in conjunction with the expectation.

The expectation stored in the map represents a “fingerprint” of the scene observed by means of the sensor, so to speak. This “fingerprint” can comprise any characteristics that can be evaluated from sensor data, such as a geometry, texturing, or a multispectral response. Basic physical characteristics, such as magnetic resonance, may, for example, also come into consideration as characteristics that can be evaluated.

The sensor data may, for example, in particular be sensor data acquired by a sensor on a vehicle or robot. The vehicle may be any land vehicle, watercraft or aircraft. An important application of the method in the context of controlling vehicles and robots is to check which locations can be freely accessed by the vehicle or robot. For this purpose, the characteristic stored in the map can include a statement as to the extent to which locations to which the expectation relates can be freely accessed by the vehicle or robot. In the same way, the method can also be used with mobile sensors that are carried by a person, for example in order to signal the accessible areas to a blind person.

This can be illustrated by a simple example in which camera images are used as sensor data. Camera images are then included in the map, and it is in each case labeled in these images which areas are freely accessible by the vehicle and which are not. If, during the trip of the vehicle, certain areas are recognized at the correct locations on the basis of the camera images currently acquired by the vehicle, there is a guarantee that the statement stored in the map for these areas as to whether they are freely accessible by the vehicle or not corresponds to the current state of these areas. If, for example, there is an obstacle at a certain point in the camera images acquired during the trip and if this obstacle deviates with respect to at least one characteristic, such as its visual appearance or its geometry, from the same characteristic of the image stored in the map, this object can be safely identified.

In this way, an open classification task as to whether an area is freely accessible (or has any other characteristic important for trip planning) can thus be reduced to a comparison with known information stored in the map. It has been found that the identification as to which areas are freely accessible can be carried out with considerably better safety integrity as a result. If a spatial area at a certain location produces the same or at least similar sensor data as those stored in the map as an expectation, it necessarily follows that the area has the same occupancy with objects in comparison to its state when the map was created (i.e., for example, it is free of objects and thus accessible) and that there are also no further objects between the sensor and this area. Any unplanned object in the area in question or in the line of sight between the sensor and the area destroys the agreement between actual sensor data and expectation and thus results in the area no longer being identified as freely accessible.

In comparison to an open classification task, such as those solved by means of neural networks, for example, this has the advantage that a not freely accessible area is identified as not freely accessible even if the reason for its lack of accessibility is extremely unusual and therefore is not included in the training data used to train the neural network. Events that are too unlikely to be included in training data are also referred to as “long tails.”

For example, very unusual objects, such as furniture, large electrical appliances, skis or bicycles, are sometimes lost on highways because the load is inadequately secured. The appearance of such objects on the roadway on which the ego vehicle drives is a highly dangerous situation and fortunately occurs only rarely. However, this also means that when collecting real data for training neural networks during test drives, such examples are highly unlikely to occur. Deliberately recreating such situations in public streets is not practical.

The same applies to the dreaded “blow-ups” that suddenly occur during heat waves and in which the concrete road of the highway bulges or breaks open. Such situations cannot be trained either, since the moment a “blow-up” appears in the camera image, an accident can hardly be prevented.

In addition, identifying freely accessible areas by recognizing them in the map is also not susceptible to deliberate manipulation. By maliciously introducing interference patterns, many image classifiers based on neural networks can be caused to output an incorrect classification. For example, a stop sign can be manipulated by attaching a seemingly inconspicuous sticker so that it is classified as a “70 km/h” sign. Experiments have already shown that by attaching a film with an inconspicuous semi-transparent dot pattern to a camera lens, the ability of the downstream image classifier to identify pedestrians can be completely eliminated. The pedestrians were classified as freely accessible space.

An attempt at such manipulation is either completely ignored within the framework of the method presented here or, in the worst case, leads to an area that is actually freely accessible is not identified as freely accessible. Any unexpected problem will therefore result in the area in question being avoided instead of being driven through (“fail-safe”).

Furthermore, the expectation can already be used to determine an upper limit for the areas that can be identified as freely accessible. The recognition of an area in the expectation can only trigger an assessment of this area as freely accessible if the area was marked as freely accessible in the context of the expectation. Areas that have not been marked as freely accessible here, such as concrete barriers or trees at the edge of the road, can never be identified as freely accessible.

The significantly improved safety integrity when ascertaining characteristics of the scene observed by means of the at least one sensor has the result that the control of a vehicle or robot according to the characteristics thus ascertained is more likely to be appropriate to the particular situation. It is therefore advantageous to ascertain a control signal for the vehicle or robot by using the determination as to the locations for which the scene observed by the sensor has the characteristic stored in the map in conjunction with the expectation. The vehicle or robot is controlled using this control signal so that the driving dynamics of the vehicle or robot are influenced according to the control signal.

The comparison of the actual sensor data with the expectation is not limited to a 1:1 recognition. Instead, a tolerance or, for example, an abstraction into certain features can be provided in any form for this recognition. For example, even two camera images of one and the same scene that are acquired immediately one after the other are generally not completely identical.

In a further, particularly advantageous embodiment of the present invention, the actual sensor data and the expectation are converted into a common spatial reference system and/or into a common workspace. The actual sensor data are compared with the expectation in this reference system or workspace. In this way, sensor data and expectations that were acquired by means of different modalities can also be compared with one another. For example, the expectation can includea spatially resolved three-dimensional geometry, and/ortexturing, and/ora reflectance amplitude, and/ora multispectral response, and/ora magnetic resonance
of the scene observed by the at least one sensor. Such a geometry can, for example, be ascertained on the basis of image acquisitions. Such a geometry can be easily labeled with regard to freely accessible areas or other characteristics of the scene. It can then be checked, for example, to what extent radar data or lidar data are consistent with this geometry.

In addition to radar sensors and lidar sensors, stereoscopically arranged cameras or multi-camera systems, for example, in particular also come into consideration for capturing the sensor data. Such camera arrangements also provide depth information, which can be checked against the geometry of the expectation. Moving monocular cameras are also possible for generating depth information. Furthermore, multispectral cameras, time-of-flight (ToF) sensors, cameras with pixels responding on the basis of events, ultrasonic sensors or even magnetic sensors may, for example, also be used.

For example, the check as to whether sensor data acquired by means of a stereoscopic camera arrangement are consistent with the three-dimensional geometry of the expectation may comprise a check of the so-called stereo hypothesis. This check is based on the geometry of the scene coupling the images provided by both cameras of the stereoscopic camera arrangement, to one another. If one of the images and the geometry of the expectation are present, the other image is thus at least largely determined.

Therefore, an image provided by the first camera of the camera arrangement can be transformed, on the basis of the geometry of the expectation, into an expectation for the image provided by the second camera of the camera arrangement. It can then be checked to what extent this expectation is consistent with an image actually provided by the second camera of the camera arrangement. In particular, the transformation may, for example, comprise distorting the image provided by the first camera on the basis of the geometry so that it fits with the perspective of the second camera.

For comparing the image provided by the second camera with the expectation, features can, for example, in particular be extracted, respectively, from the image provided by the second camera on the one hand, and from the expectation for this image on the other hand. These features can then be compared with one another. This abstraction into features can smooth out insignificant differences for the comparison, for example with regard to colors or lighting.

For example, for each feature to be compared, a binary decision can in particular be made as to whether a feature from the image is consistent with the corresponding feature from the expectation for this image. From the number of features that are consistent with one another, a degree of agreement between the image and the expectation can then be ascertained. For example, a Hamming distance can thus be ascertained between the examined combinations of features, which is the larger, the more features of the image on the one hand and of the expectation on the other hand are inconsistent with one another.

In a further advantageous embodiment of the present invention, it is additionally checked to what extent a predetermined test image, which does not show the scene observed by the sensor, is consistent with the expectation for the image provided by the second camera of the camera arrangement. This degree of agreement is then used as the noise level for the ascertained agreement between the image provided by the second camera of the camera arrangement and the expectation for this image. In this way, a signal-to-noise ratio can be ascertained for the agreement of the image provided by the second camera, with the expectation for this image. This signal-to-noise ratio is more meaningful than the agreement alone. For example, the meaningfulness of the image may be reduced because large parts thereof have become saturated due to overexposure or underexposure.

The method can also be generalized in that actual sensor data are acquired by means a plurality of sensor modalities and the results are subsequently merged. In a further advantageous embodiment of the present invention, for actual sensor data acquired by means of a plurality of different sensors, it is therefore checked, respectively and separately, for which locations these actual sensor data are consistent, respectively, with the expectation retrieved from the map. Only for the locations for which the actual sensor data of all sensors are consistent, respectively, with the expectation, it is then determined overall that the actual sensor data overall are consistent with the expectation. For example, an area is only assessed to be freely accessible to a vehicle or robot if it has been identified as freely accessible on the basis of the sensor data, provided by a plurality of sensors of different sensor modalities (such as lidar and stereoscopic camera), independently of one another.

For example, it can first be checked to what extent current lidar data are consistent with a geometry of the scene that is stored in the map. In parallel, it can be checked, for example, to what extent images provided by a stereoscopic camera arrangement are consistent with this geometry. For this purpose, the geometry can, for example, be transformed into the reference system of the first camera of the camera arrangement. The stereo hypothesis can then be checked, as described above, by transforming the image provided by the first camera of the camera arrangement, on the basis of the geometry into an expectation for the image provided by the second camera of the camera arrangement, and by comparing this expectation with the image actually provided by the second camera. Only for locations for which both the lidar data and the images provided by the stereoscopic camera arrangement are consistent, respectively, with the geometry stored as an expectation in the map, it can then be determined that these locations have the desired characteristic (for example, the free accessibility of these locations) according to the map.

In a further advantageous embodiment of the present invention, actual sensor data, the agreement of which with the expectation is checked, are checked for plausibility against actual sensor data acquired by means of a further sensor. Agreement with the expectation is then determined or maintained only with respect to the locations for which this plausibility check is positive.

Only one comparison instead of two comparisons between a sensor modality and the expectation thus takes place. Furthermore, the two sensor modalities are compared with one another. For example, an area can only be declared to be freely accessible if, on the one hand, it turns out to be freely accessible on the basis of the comparison of lidar data to the geometry stored as an expectation, and if, on the other hand, the lidar data in this area are consistent with images provided by a stereoscopic camera arrangement.

In a further advantageous embodiment of the present invention, an ascertained location and/or an ascertained orientation is optimized with the aim of maximizing the agreement of the actual sensor data with the expectation. As explained above, comparing actual sensor data with the expectation depends on retrieving the expectation for the correct location and the correct orientation of the sensor from the map. Only then can the expectation be correctly recognized on the basis of the current actual sensor data. However, every method for determining the location and the orientation has limited accuracy. If the agreement between the sensor data and the expectation can, for example, be significantly improved by an additional shift of the ascertained location of the sensor and/or by an additional tilting or rotation of the ascertained orientation of this sensor, this indicates that the previously ascertained location or the previously ascertained orientation of the sensor was not entirely correct. Alternatively or in combination, any other technique can be used to convert the actual sensor data and the map into a common reference system.

According to an example embodiment of the present invention, the method can in particular be wholly or partially computer-implemented. For this reason, the present invention also relates to a computer program comprising machine-readable instructions which, when executed on one or more computers, cause said computer(s) to carry out the method of the present invention described above. In this sense, control devices for vehicles and embedded systems for technical devices, which are also capable of executing machine-readable instructions, are also to be regarded as computers.

The present invention also relates to a machine-readable data carrier and/or to a download product comprising the computer program of the present invention. A download product is a digital product that can be transmitted via a data network, i.e., can be downloaded by a user of the data network, and can, for example, be offered for immediate download in an online shop.

Furthermore, a computer can be equipped with the computer program, with the machine-readable data carrier, or with the download product.

Further measures improving the present invention are explained in more detail below, together with the description of the preferred exemplary embodiments of the present invention, with reference to figures.

DETAILED DESCRIPTION OF EXAMPLE EMBODIMENTS

FIG.1is a schematic flow diagram of an exemplary embodiment of the method100for evaluating spatially resolved actual sensor data2with regard to the locations at which the scene observed by a sensor1has an interesting characteristic5.

In step105, sensor data2acquired by a sensor1on a vehicle50or robot60may be selected.

In step106, sensor data2acquired by means of a radar sensor, a lidar sensor, and/or a stereoscopic camera arrangement may be selected.

In step110, a location1aand an orientation1bof the sensor1at the time of acquiring the sensor data2are ascertained.

In step120, a spatially resolved expectation4is retrieved from a spatially resolved map3on the basis of the location1aand the orientation1bof the sensor1. Furthermore, an interesting characteristic5is also stored with spatial resolution in the map3. The characteristic5is coupled with the expectation4in that, assuming that the scene observed by the sensor1has the characteristic5from a certain location, the actual sensor data2should be consistent with the expectation4.

In step130, it is checked to what extent the actual sensor data2are consistent with the expectation4.

In step140, at least with respect to the locations for which the actual sensor data2are consistent with the expectation4, it is determined that the scene observed by the sensor1has the characteristic5stored in the map3in conjunction with the expectation4. This is illustrated inFIG.2using an example.

In step150, a control signal150afor the vehicle50or robot60is ascertained by using the determination as to the locations for which the scene observed by the sensor1has the characteristic5stored in the map3in conjunction with the expectation4.

In step160, the vehicle50or robot60is controlled using this control signal150aso that the driving dynamics of the vehicle50or robot60are influenced according to the control signal150a.

According to block111, an ascertained location1aand/or an ascertained orientation1bcan be optimized with the aim of maximizing the agreement of the actual sensor data2with the expectation4.

According to block121, the characteristic5stored in the map3can, for example, in particular include a statement as to the extent to which locations to which the expectation4relates can be freely accessed by the vehicle50or robot60.

According to block122, the expectation4can, for example, in particular include a spatially resolved three-dimensional geometry of the scene observed by the sensor1.

According to block131, the actual sensor data2and the expectation4can be converted into a common spatial reference system and/or into a common workspace. According to block132, the actual sensor data2can then be compared with the expectation4in this reference system or workspace.

According to block133, for checking as to whether sensor data2acquired by means of a stereoscopic camera arrangement are consistent with the three-dimensional geometry of the expectation4, an image provided by the first camera of this camera arrangement can be transformed, on the basis of the geometry of the expectation4, into an expectation for the image provided by the second camera of the camera arrangement.

According to block134, it can then be checked to what extent this expectation is consistent with an image actually provided by the second camera of the camera arrangement.

This check in turn may include, according to block134a, extracting features, respectively, from the image provided by the second camera on the one hand, and from the expectation for this image on the other hand and, according to block134b, comparing these features with one another.

This comparison in turn may include, according to block134c, making a binary decision for each feature as to whether a feature from the image is consistent with the corresponding feature from the expectation for this image and, according to block134d, ascertaining a degree of agreement between the image and the expectation from the number of features that are consistent with one another.

According to block135, it can additionally be checked to what extent a predetermined test image, which does not show the scene observed by the sensor1, is consistent with the expectation for the image provided by the second camera of the camera arrangement. According to block136, this degree of agreement can then be used as the noise level for the ascertained agreement between the image provided by the second camera of the camera arrangement and the expectation for this image.

According to block137, for actual sensor data2acquired by means of a plurality of different sensors1, it can be checked, respectively and separately, for which locations these actual sensor data1are consistent, respectively, with the expectation4retrieved from the map3. According to block138, only for the locations for which the actual sensor data2of all sensors1are consistent, respectively, with the expectation, it can then be determined overall that the actual sensor data2overall are consistent with the expectation4.

According to block141, actual sensor data2, the agreement of which with the expectation4is checked, are checked for plausibility against actual sensor data2acquired by means of a further sensor1. According to block142, agreement with the expectation4can then be determined or maintained only with respect to the locations for which this plausibility check is positive.

FIG.2shows an exemplary application of the method100to a street scene.

In this example, the sensor1is carried by a vehicle not shown. From a perspective determined by the location1aand the orientation1bof the sensor1, the sensor1captures actual sensor data2within its detection range1c. In the example shown inFIG.2, the scene includes a road10with a preceding vehicle12and a tree11at the edge of the road.

The spatially resolved map3likewise includes the road10and the tree11, but the preceding vehicle12is missing. The interesting characteristic5, namely, that the area is freely accessible, is stored for this area of the road10.

The view and/or geometry of the scene that is/are contained in the map3is/are compared as expectation4with the actual sensor data2. In the process, it is largely determined for the region of the road10that the actual sensor data2are consistent with the expectation4and, in conjunction with the expectation4, the characteristic5that the region is a freely accessible region is at the same time stored in the map3. Accordingly, this region is deemed to be freely accessible.

The only exception is the area with the preceding vehicle12. Since this vehicle is missing in the map3, the actual sensor data2deviate from the expectation4. Accordingly, the area with the preceding vehicle12is not deemed to be freely accessible.