Patent ID: 12243321

DETAILED DESCRIPTION

FIG.1schematically depicts a host vehicle11which includes a camera13and a sensor14for monitoring the environment of the vehicle11. The camera13and the sensor14belong to a system12for determining a semantic free space in the environment of the vehicle11according to the disclosure. The system12further includes a free space module15and a classification module16which are connected to each other and which are provided with data from the camera13and from the sensor14.

The camera13is a monocular camera providing a two-dimensional visual image as shown e.g. as camera image17inFIG.2. The sensor14(seeFIG.1) is generally a sensor providing three-dimensional distance data for objects in the environment of the vehicle11. In the present example, the sensor14is a LIDAR sensor. However, as an alternative, a radar sensor may also be used for the sensor14, or any suitable sensor providing distance or depth information.

FIG.2depicts the visual image17which is provided by the camera13(seeFIG.1). The visual image17is a two-dimensional representation of the environment in front of the host vehicle11. The visual image17includes a road18in front of the vehicle11including three lanes19and a boundary20of the road18which is represented by guide rails. In addition, the visual image17includes further objects or obstacles in front of the vehicle11, i.e. other passenger cars21and trucks23driving in front of the vehicle11or in an opposite lane. The boundaries20of the road18and the other vehicles, i.e. the passenger cars21and the trucks23, limit the drivable area which is available for the host vehicle11.

FIG.3depicts a representation of a free space25which is derived from the visual image17as shown inFIG.2. In detail, a first convolutional neural network is applied to the two-dimensional visual image17as shown inFIG.2which is available in the form of pixel data provided by the camera13(seeFIG.1). Via the convolutional neural network, a limitation29(seeFIG.3) is determined for the free space25. Therefore, the upper part ofFIG.3denoted by27represents a non-drivable area which is currently not available for the navigation of the host vehicle11. The limitation29of the free space25is determined by the free space module15(seeFIG.1) which includes the first neural network.

FIG.4additionally depicts a contour30which extends continuously along the limitation29between the free space25and the non-drivable area27as also shown inFIG.3. The contour30is determined by applying a border following algorithm, for which examples are known in the art.

FIG.5shows the visual image17fromFIG.2together with a projection of three-dimensional data points33which are provided by the sensor14. That is, the three-dimensional data33captured by the sensor14are transformed to the plane of the visual image17via a projecting transformation in order to provide a representation of the data points33as shown inFIG.5. In detail, the data points33are provided by the LIDAR system and represent the respective shortest distance of the next obstacle or object with respect to the host vehicle11.

As may be recognized inFIG.5, a part of the distance data points33are assigned to a respective passenger car21, whereas another part of the distance points33is assigned to the boundaries20of the road18.

FIG.6depicts the assignment of the three-dimensional distance data33to the representation of the free space25as shown inFIGS.3and4. In detail, the distance data33provided by the sensor14are related to respective points of the continuous contour30of the free space25as shown inFIG.4.

For the assignment of the projected distance data33to the points of the limitation29or contour30, a fixed number (e.g. one to five) of closest points from the projected distance data points33is determined for each point of the contour30. The closest points from the distance data33with respect to the points of the contour30are those points which have the shortest distance within the representation ofFIG.6with respect to a certain point of the contour30. For determining the closest points33or nearest neighbor points33with respect of the points of the contour30, efficient methods like k-d trees may be used which are known in the art.

In order to determine a distance or “depth” with respect to the host vehicle11for each point of the contour30, an average over the measured distances of the closest distance data points33is estimated for each point of the contour30. This average may be a weighted average wherein each weight depends on the respective distance to the point of the contour30under consideration.

The estimated distance or depth of the points of the contour30is used for transforming the contour30to a bird's-eye view coordinate system35as shown inFIG.7. For this transformation, the inverse transform of the projecting transform is used which is applied for projecting the three-dimensional distance data33provided by the sensor14to the visual image17, as shown inFIG.5.

As shown inFIG.7, the bird's-eye view coordinate system35comprises an x-axis38and a y-axis39which are located in a plane parallel to a tangent to the lane in which the host vehicle11is currently driving. In other words, inFIG.7one is looking from above at the environment in front of the host vehicle11. In the bird's-eye view coordinate system35ofFIG.7, the free space25is shown again which is limited by the limitation29or transformed contour31which has been transformed from the representation according to the visual image17(seeFIGS.3and4) to the bird's-eye view ofFIG.7using the distance information provided by the three-dimensional distance data33from the sensor14. A smoothing by applying a moving average is used for the x- and y-coordinates of the contour31in the bird's-eye view coordinate system35in order to achieve a smooth representation of the transformed contour31of the free space25.

In the area close to the host vehicle11, the free space25is limited according to an angle37representing the instrumental field of view of the camera13. In addition, the projection of the three-dimensional distance data33from the sensor14is shown within the bird's-eye view coordinate system35. Since the respective distance of the points belonging to the contour31is determined based on the three-dimensional distance data33from the sensor14, the projection of the distance data33is positioned at the contour31in the bird's-eye view coordinate system35.

The free space25as shown inFIG.7represents the drivable area in front of the host vehicle11. However, it is not yet known for the points of the contour31as shown inFIG.7by which type of object the free space or drivable area25is restricted. For actions regarding the navigation of the host vehicle11it is desirable, however, to have additional information regarding the objects restricting the free space25, i.e. whether the free space25is limited by another vehicle like the cars21as shown inFIG.2, or by a pedestrian or by debris located on one of the lanes19(seeFIG.2), for example. Depending on the type of the object limiting the free space25, different decisions or actions may be taken for the navigation of the host vehicle11. In other words, additional “semantic” information is desired regarding the objects in the environment of the host vehicle11.

In order to provide such semantic information, objects are identified within the visual image17provided by the camera13, as shown inFIG.8. For each identified object, a two-dimensional bounding box41is determined together with a classification43of the object. In detail, a respective minimum and maximum value for an x- and a y-coordinate is determined within the visual image17in order to determine the frame of respective bounding box41which encloses the respective object within the visual image17. In addition, the classification43is determined for each bounding box41, i.e. whether the respective bounding box encloses a car or a truck, as shown inFIG.8. That is, the items “car” and “truck” describe the respective detection class or classification43for each of the identified objects. Moreover, a detection certainty or class probability45is determined for each bounding box41.

In order to determine the position of each bounding box41, i.e. its coordinate within the visual image17, the classification43and the class probability45, a second convolutional neural network is applied to the pixel data of the visual image17. The determination of the bounding boxes41together with the classification43and the class probability45based on the visual image17using a convolutional neural network is also referred to as single shot multi-box detection (SSD) since no segmentation of the visual image17is previously performed. The second convolutional neural network is included in the classification module16(seeFIG.1). The module also performs all further method steps as described below.

In order to relate the bounding boxes41as well as their classification43and class probability45to the free space25as shown inFIG.7, a center is determined for each bounding box41, and for each center of the respective bounding box41the closest points33are determined from the projection of the three-dimensional distance data provided by the sensor14. The points33representing the projection of the three-dimensional distance data33are shown together with the bounding boxes41inFIG.9.

The projection of the dimensional distance data points33to the visual image17is the same as shown inFIG.5. Therefore, no additional calculation is required regarding the distance data points33before assigning these points to the respective centers of the bounding boxes41. For each center of the bounding boxes41, a fixed number (e.g. one to five) of closest data points33is determined, in the same manner as for the points of the contour30as shown inFIGS.4and6. That is, the fixed number of data points33is determined for each center of the bounding boxes41which have the shortest distance to the respective center of the bounding boxes41. Again, an average or a weighted average for the distances of the closest data points33is calculated for each center of the bounding boxes41in order to determine a depth or distance of each center of the bounding boxes41with respect to the host vehicle11.

However, only those centers of the bounding boxes41are selected for the further procedure, i.e. for a transform to the bird's-eye view coordinate system35, for which the distance to the closest distance data point33within the visual image17is less than a predetermined distance. By this means, only those centers of the bounding boxes41are selected which are “reasonably” close to at least one of the distance data points33.

For example, the bounding boxes41of the passenger cars21as shown inFIG.9each comprise distance points33which are quite close to the center of the respective bounding boxes41. In contrast, the bounding boxes41of the trucks23do not include distance data points33which are close to the center of the respective bounding box41. Therefore, the bounding boxes41of the passenger cars as shown inFIG.9are selected for the further procedure only and transformed to the bird's-eye view coordinate system35as shown inFIG.10.

FIG.10depicts the free space25in front of the vehicle11in the bird's-eye view coordinate system35, which has already been shown inFIG.6. The free space25is limited by the transformed contour31on which the free dimensional distance data33are also shown.

In addition,FIG.10depicts transformed bounding boxes51which are assigned to the contour31. The bounding boxes41as shown inFIGS.8and9are transformed from the visual image17to the bird's-eye view coordinate system35via the same transform as for the contour30(seeFIGS.4and6), i.e. the inverse transform of the projecting transform which is used for projecting the three-dimensional distance data33to the visual image17.

In detail, the respective center of the bounding boxes41(seeFIGS.8and9) is transformed from the visual image17to a respective transformed center53of the bounding box41in the bird's-eye view coordinate system35. In addition, the semantics of the bounding boxes41are also known from applying the second neural network to the visual image17. That is, the type or class of the respective object is known for the representation as shown inFIG.10for each bounding box41. For the present example, it is known that the three bounding boxes51each represent a car21inFIG.10.

Finally, the contour31of the free space25in front of the vehicle11is equally divided by a predetermined azimuth angle with respect to the vehicle11, and each segment55of the contour31is classified by assigning the respective segment55to the respective classification of the center53of the bounding box51(seeFIG.10) if this segment comprises the center53of the bounding box51. Classified segments55are shown inFIG.11which includes the same representation of the free space25asFIG.10. That is, the segments55of the contour31are classified as in “passenger car”21which means that the free space25in front of the vehicle11is limited by a passenger car21(seeFIG.2) at the classified segments55.

In summary, the method according to the disclosure determines the limits or the contour31of the free space25in front of the vehicle in bird's-eye view via the first neural network, and in addition, the semantic of the segments of the contour31is determined via the second neural network such that it will be known which part of the free space25is limited by which kind of object. In case that no center of a bounding box can be assigned to a specific segment of the contour31, a default classification may be assumed for these segments, e.g. a classification as boundary20of the road18(seeFIG.2) or a limitation by the instrumental field of view of the camera13.

In addition, a certainty score is estimated for each classified segment55of the contour31based on the class probability45which is determined for each bounding box41via the second neural network. The certainty score and the semantic free space represented by the segments of the contour31(seeFIG.11) can be used for the navigation of the host vehicle11, e.g. in autonomous driving.