Patent ID: 12243323

FIG.1shows a respective preferred exemplary embodiment of an inventive system100for preparing dynamic spatial scenario data as well as an inventive system200for training an artificial neural network1. The system100for data preparation comprises a determination module2and a generation module3, wherein the determination module2is configured to ascertain from sensor data S a time curve of an angular sector which, from the perspective of an ego object, is covered by another object, and wherein the generation module3is configured to generate a display of the ascertained time curve. The system200for training the artificial neural network1comprises the generation module3and an interface4, wherein the interface4is configured to feed the generated display to the artificial neural network1together with information on a dynamic spatial scenario.

The sensor data S is for example generated by environmental sensors of an ego vehicle when detecting a traffic scenario and characterize for instance the number of nearby vehicles, the relative arrangement, in particular the relative distances of the other vehicles to the ego vehicle, the speed of the ego vehicle, and/or the like. The determination module2can preferably ascertain from this sensor data S the width of at least one angular sector obscured by the other vehicles in the field of view of the ego vehicle and in which the position this angular sector is arranged, for example relative to the direction of travel of the ego vehicle.

Since the environmental sensors detect the surroundings of the ego vehicle preferably continuously but yet at least with high frequency, the time curve of the angular sector from the perspective of the ego vehicle, in particular a change in its width and/or its position, can be ascertained from the resulting sensor data S. The generation module3can use the time curve to generate a graphical display which abstractly depicts the traffic scenario. In other words, the generation module3is configured to encode the traffic scenario-related information contained in the sensor data S in the display, particularly in compressed form.

Preferably, the sensor data S is sensor data which is classified with respect to dynamic spatial scenarios; i.e. the sensor data S is for example assigned to one of a plurality of classes of traffic scenarios such as for instance passing maneuvers or lane changing maneuvers. This classification can for example be performed manually, for instance by viewing an image data stream. The display generated on the basis of the ascertained time curve of the angular sector can thus be transmitted from the interface4to the artificial neural network1along with information about the dynamic spatial scenario.

The artificial neural network1is preferably configured to recognize at least one respective pattern in all of the displays assigned to the same traffic scenario class. A template characterizing a known traffic scenario can be defined on the basis of such a recognized pattern. The templates defined in this way can be stored in a database5for further use, for instance for evaluating sensor data generated during a vehicle's regular operation.

FIG.2shows a preferred embodiment of a system300for analyzing sensor data S suited to characterizing a dynamic spatial scenario with respect to an ego object and at least one other object. The system300comprises a generation module3configured to generate, on the basis of the sensor data S, a display of a time curve of an angular sector which, from the perspective of the ego object, is covered by another object. The system300further comprises a comparison module6configured to compare the generated display to at least one predefined template of a known dynamic spatial scenario. To that end, the comparison module6preferably has access to a database5in which at least one predefined template is stored.

The result of the comparison is preferentially output by the comparison module6and can for instance be used to control a driver assistance system with which an ego vehicle is equipped. If at least one predefined measure of correspondence is for example ascertained between the generated display and the at least one predefined template, for instance by analyzing a measure of similarity generated during the comparison, it can be concluded that the known scenario is present and the comparison module6can output for instance a scenario class as an output signal. Alternatively or additionally, the beginning and/or end of an identified driving maneuver can also be output, in particular signaled.

FIG.3shows an example of ascertaining an angular sector Φ in a dynamic spatial scenario10in which an ego object11moves along a direction of movement, here along the x-axis of a coordinate system, and is surrounded by other objects12. In the present example, the spatial scenario10is a traffic scenario in which an ego vehicle11ais moving in the direction of travel in the center lane and is surrounded by other vehicles12a,12bin adjacent lanes.

FIG.3Ashows the traffic scenario from a bird's eye view, wherein the depicted display summarizes a chronological sequence of spatial scenes. The other vehicles12a,12bare seen in different positions relative to the ego vehicle11aat different times, these being indicated by different fillings of the rectangles representing the vehicles12a,12b. A denser filling thereby corresponds to a position earlier in time. As is readily discernible in this display, a first vehicle12achanges lanes from an outer lane to the lane used by the ego vehicle11a and thereby merges in front of the ego vehicle11a. A second vehicle12b, which is initially at approximately the same height as the ego vehicle11a in relation to the direction of movement, falls back over time. Because the positional change of the second vehicle12bis relatively small, the positions occupied by the second vehicle12bat the different points in time overlap in this display.

In order to generate an alternative display of the traffic scenario, an angular sector can be ascertained for each of the two vehicles12a,12bwhich indicates the area in the field of view of the ego vehicle11a covered by the respective vehicle12a,12b.FIG.3Bshows an example of this for the first vehicle12a. The contour13of the first vehicle12awhich results relative to the perspective of the ego vehicle11a is indicated as a solid line and spans angular sector Φ. A position φ of the angular sector Φ, and thus also of the first vehicle12a, from the perspective of the ego vehicle11acan thereby be indicated relative to a predetermined direction, for example the direction of movement. If the position of the vehicle12achanges relative to the ego vehicle11a, both the width of the angular sector Φ as well as its position φ can change.

This is shown inFIG.3C. The bar chart shown therein, in which the inverse distance d between the ego vehicle11a and the other vehicles12a,12b, indicated inFIG.3Aas a black arrow, is plotted against the position φ of the angular sector Φ, constitutes an abstract display of the traffic scenarios depictedFIG.3A. The right three bars correspond to the angular sector Φ covered by the first vehicle12awhile the left three bars correspond to the angular sector Φ covered by the second vehicle12b. As also inFIG.3A, the different points in time are indicated by the corresponding filling of the bars.

Thus, the angular sector Φ covered by the first vehicle12ashifts toward a 0° position when merging in front of the ego vehicle11a, whereby the 0° position of the angular sector Φ corresponds to a position in front of ego vehicle11a. As the distance d between the first vehicle12aand the ego vehicle11a thereby decreases as shown inFIG.3A, the width of the angular sector Φ covered by the first vehicle12aincreases. In addition to the angular sector Φ width, the distance d between the first vehicle12aand the ego vehicle11a is also encoded in the height of the bars, which increases with the passage of time.

On the other hand, the angular sector Φ covered by the second vehicle12bshifts away from the 0° position as the second vehicle12bfalls behind the ego vehicle11a. Since the distance between the second vehicle12band the ego vehicle11a thereby increases, the height of the bars as well as their width also decrease.

FIG.4shows a first example illustrating the correlation between a preferential embodiment of an inventive display20and a corresponding dynamic spatial scenario10.FIG.4Ato that end shows the display20of time curves of angular sectors Φa, Φb, Φc covered by other objects from the perspective of an ego object, for instance ego vehicle11a. The time t is plotted against the position φ of the angular sectors Φa, Φb, Φc in the display20, wherein the 0° position corresponds to a position in front of the ego object. It is noted that the display20may be considered a representation20, for purposes of the present disclosure.

The time curves of angular sectors Φa, Φb, Φc are depicted as lines, their width corresponding to the distance of the respective object from the ego object. A first object, for instance a first vehicle12a, is thus initially located at some distance in front of the ego object at t=0. In contrast, a second object, e.g. a second vehicle12b, is initially closer to the ego object, although in a position to the side of the ego object. At approximately t=40, the second object changes position and edges between the ego object and the first object. From this point, the time curve of the angular sector Φb covered by the second object overlaps the course of the angular sector Φa covered by the first object.

At approximately t=70, a further angular sector Φc is covered by a third object, for instance a third vehicle12c. The third object is to the side of the ego object, and on that side of the ego object opposite from the side which accommodated the second object at t=0. The position φ of the angular sector Φc covered by the third object then shifts toward the 0° position. This time curve can for example be induced by a movement of the third object parallel to the ego object's direction of movement, whereby the distance of the third object to the ego object increases.

Such time curves of the angular sectors Φa, Φb, Φc can for example be characteristic of the spatial dynamic scenario10shown inFIG.4B, here a traffic scenario. The chronological development is indicated inFIG.4Bby the dotted trajectories of the vehicles on the roadway.

The ego vehicle11ais initially located at position (x=0, y=0) and the first vehicle12ais located approximately at position (x=60, y=0). Accordingly, the first vehicle12ais driving in a center lane at an approximate distance of Δx=60 ahead of the ego vehicle11a. The second vehicle12bis initially located approximately at position (x=45, y=−4), whereby it is accordingly traveling between the ego vehicle11aand the first vehicle12ain a neighboring lane; i.e. laterally offset relative to the ego vehicle11a.

Over the further time curve, the second vehicle12bmerges into the center lane between the ego vehicle11a and the first vehicle12a. The merging movement of the second vehicle12bbegins at approximately position (x=100, y=−4) and ends at approximately position (x=150, y=0). After the other vehicle12bmerges, the ego vehicle11a, the first vehicle12aand the second vehicle12bcontinue traveling together in the center lane. Because the second vehicle12bnow obscures the view of the first vehicle12afrom the perspective of the ego vehicle11a, only one line is visible in the 0° position in the correspondingFIG.4Adisplay20.

The third vehicle12cis initially behind the ego vehicle11asuch that it is not initially detected, for instance by environmental sensors of the ego vehicle11a. The third vehicle12cis however moving at a higher speed than the ego vehicle11asuch that in the further time curve it overtakes the ego vehicle11a in a further neighboring lane at y=4. The third vehicle12conly becomes visible to the ego vehicle11aupon passing it, such that the time curve of the angular sector Φc covered by the third vehicle12conly starts from this point in time in the correspondingFIG.4Adisplay20.

The display20resulting from the time curves of the angular sectors inFIG.4Aexhibit a pattern characteristic of the described spatial dynamic scenario10. When many such displays are generated on the basis of sensor data repeatedly collected while detecting such a scenario, an artificial neural network can learn this pattern or be trained to recognize this pattern respectively. From the sensor data thereby generated, the time curve of angular sectors can then be depicted and analyzed by the trained artificial neural network, in particular compared to the learned pattern, essentially in real time during the regular operation of a vehicle. Whether or respectively at what point in time a known traffic scenario exists is thereby preferably ascertained.

FIG.4Cshows the result of such a comparison, for instance between the display20shown inFIG.4Aand a corresponding template, wherein an output signal a is plotted against the time t. At approximately time t=30, the output signal a jumps to value 6, whereby the presence of the known traffic scenario is signaled, in this case the merging of a second vehicle12b. The output signal a drops back down to the value of zero at approximately time t=85, thereby signaling the end of the merging maneuver.

In addition to the start and the duration of the driving maneuver, the traffic scenario classification can also be displayed by way of the output signal a value. In another, not shown, example, the comparison of the generated display to different templates could for instance show the highest correspondence to a template associated with a veering maneuver and the output signal a correspondingly assume a different value.

FIG.5shows a second example illustrating the correlation between a preferential exemplary embodiment of an inventive display20and a corresponding dynamic spatial scenario10. Here,FIG.5Ashows a scene from the traffic situation corresponding to dynamic spatial scenario10, in this case the merging of an ego vehicle11ainto a lane in which another first vehicle12ais already driving, indicated by the arrow. In addition to the first vehicle12a, a further second vehicle12bis driving in another lane. It is noted that the display20may be considered a representation20, for purposes of the present disclosure.

The chronological development of the dynamic spatial scenario10is shown inFIG.5Bby the dotted trajectories of the vehicles11a,12a,12b. The ego vehicle11ais initially located approximately at position (x=0, y=2.5) and the first vehicle12ais driving in a center lane ahead of the ego vehicle11aapproximately at position (x=35, y=0); i.e. at an approximate distance of Δx=35. The second vehicle12bis initially located approximately at position (x=25, y=−2.5).

Over the further time curve, the ego vehicle11abegins to merge into the center lane at approximately position (x=60, y=2) and ends the driving maneuver at approximately position (x=100, y=0). The other two vehicles12a,12bcontinue driving straight ahead at a slightly higher speed such that the ego vehicle11aslowly drops behind.

FIG.5Cshows the time curves of angular sectors Φa, Φb in a display20, each covered by the first vehicle12aor second vehicle12brespectively from the perspective of the ego vehicle11a. Here, the time t is plotted against a position φ of angular sectors Φa, Φb. A 0° position plotted inFIG.5ccorresponds to a position in front of the ego vehicle11a.

As described above, the angular sectors Φa, Φb shift toward the 0° position over the time curve since the first and second vehicles12a,12bare moving away from the ego vehicle11adue to their higher speed. At the time the ego vehicle11amerges, here at approximately the time of t=30, the time curves of the angular sectors Φa, Φb additionally curve in the direction of the 0° position. The angular sector Φa covered by the first vehicle12asubsequently runs along the 0° position since, as shownFIGS.5A and5B, the first vehicle12ais traveling in front of the ego vehicle11a in the same lane.

Based on the resulting pattern, it is initially only possible to conclude the relative movement of the ego vehicle11a with respect to the first and second vehicle12a,12b; i.e. it is not initially clear from the shape of the time curve of the angular sectors Φa, Φb whether the ego vehicle11ais switching to the center lane or whether the first and second vehicles12a,12bare respectively changing lanes, in particular substantially at the same time, whereby the first vehicle12aswitches to the lane of the ego vehicle11a and the second vehicle12bto the lane previously used by the first vehicle12a.

In order to be able to distinguish between these two cases, in addition to the angular sectors Φa, Φb, the display20shows a value which is characteristic of the in particular transversal distance between the ego vehicle11a and the respective other vehicle12a,12band/or of the in particular transversal speed of the ego vehicle11a at time t. The transversal distance or transversal speed thereby relates to a transverse component of the distance or respectively speed; i.e. the y components in the display shown inFIG.5B.

The display20shows the characteristic value of the distance and/or the speed as a coloration of the angular sector time curves indicated by shading. For example, a dark shading thereby corresponds to a high transversal speed of the ego vehicle11a. Thus, able to be seen from the display20inFIG.5Cis that the ego vehicle begins a transverse movement; i.e. perpendicular to the path of the lane, at approximately t=20, whereby the ego vehicle11areaches the highest transversal speed at approximately t=30 and the transverse movement ends at approximately t=40.

Since the ego vehicle11adoes not change its (transversal) speed when one of the other vehicles12a,12bchanges lanes, it can be concluded in the present case that the ego vehicle11a has switched to the lane used by the first vehicle12a.

FIG.6shows a preferential exemplary embodiment of an inventive method V1for data preparation and an inventive method V2for training an artificial neural network respectively.

In method step S1, sensor data is generated, for instance by sensory detection of an ego object's surroundings, and classified, i.e. assigned to different dynamic spatial scenarios. The classification can for example be performed manually, for instance by evaluating an image data stream. Alternatively, the sensor data can also be classified automatically, particularly in a case of sensor data generated by a simulator during the simulation of various dynamic spatial scenarios.

In a further method step S2, a time curve of an angular sector covered by another object from the perspective of the ego object is ascertained on the basis of the sensor data. For example, the contour of the other object, in particular the cross-sectional area, can be ascertained and its width within or proportion of the field of view of the ego object determined. Furthermore, a geometric center of gravity of the contour or the cross-sectional area respectively can be ascertained and its position in the field of view of the ego object depicted, in particular relative to the direction of movement of the ego object.

In a further method step S3, an in particular graphical display which depicts the time curve is preferably generated from the time curve of the angular sector. The time curve of the angular sector can form for example a pattern in such a display, for instance a figure. Preferably, the width of the angular sector, in particular its proportion of the field of view of the ego object, and its position in the ego object's field of view, in particular relative to the ego object's direction of movement, is depicted for each point in time at which sensor data is generated or at least one angular sector covered by another object is ascertained.

Preferentially, a speed of the ego object, in particular a transversal speed, and/or a distance, in particular a transversal distance, of the ego object to the other object is also taken into account in method step S3when generating the display. In particular, a value can be ascertained and stored or the display colored accordingly, for instance based on a function in which parameters which preferably characterize the dynamic spatial scenario such as e.g. the speed and/or the distance are entered as input variables. Preferably, the generated display thus provides information regarding the width and position of the angular sector covered by the other object in the field of view of the ego object and the speed of the ego object and/or the distance to the other object.

In a further method step S4, the generated display is fed to an artificial neural network, for instance via an interface, which is thereby in particular trained to recognize patterns in the generated display. To that end, the artificial neural network is preferably additionally fed information on the dynamic spatial scenarios according to which the sensor data was classified in method step S1so that the artificial neural network can in each case correlate the generated display, or the recognized pattern respectively, to one of the dynamic spatial scenarios.

FIG.7shows a preferential exemplary embodiment of an inventive method V3for analyzing sensor data suited to characterizing a dynamic spatial scenario with respect to an ego object and at least one other object. The sensor data is preferably sensor data generated by environmental sensors of an ego vehicle.

In method step S3, a display of a time curve of an angular sector is generated from the sensor data. The angular sector thereby corresponds to the area in the ego object's field of view which is covered by the other object. Such a display can for example be an image in which the time curve of the angular sector forms a pattern, e.g. a figure.

Where applicable, the time curve of the angular sector can thereby also be ascertained in a separate preceding method step (not shown) on the basis of the sensor data.

In a further method step S5, the generated display is compared to at least one predefined template of a known dynamic spatial scenario. This enables the ascertaining of which dynamic spatial scenario the ego object is currently located in and, if necessary, the appropriate control of a driver assistance system.

In a further method step S6, the generated display can optionally be stored as a further predefined template, e.g. in a database, when no or at least inadequate correspondence of the generated display to the at least one predefined template associated with a known dynamic spatial scenario can be ascertained. A catalog of predefined templates suitable for identifying dynamic spatial scenarios can in this way be generated, in particular essentially in real time.

LIST OF REFERENCE NUMERALS

1artificial neural network2generation module3determination module4interface5database6comparison module10spatial dynamic scenario11ego object11aego vehicle12different vehicle12afirst vehicle12bsecond vehicle12cthird vehicle13contour20display or representation100data preparation system200artificial neural network training system300sensor data analyzing systemΦ angular sectorφ position of angular sectorS sensor datay direction of movementd distancea output signalV1data preparation methodV2artificial neural network training methodV3sensor data analyzing methodS1-S6method steps