Patent Description:
Road vehicles have been equipped to emit optical light signals to the surrounding of the vehicle. There have been some applications like messages to other vehicles, messages to passengers about the possibility to cross the street in front of the vehicle and messages to cyclists about the imminent opening of a door. The emission of these light signals can be easily performed on surfaces which are dark, flat, and horizontal. However, most surfaces in the surroundings of a vehicle do not meet these conditions. <CIT>, <CIT>, <CIT>, <CIT> and <CIT> each disclose an image projection apparatus for a vehicle that projects an image onto a road surface or other projection surface.

It is an object of the present invention to improve the emission of optical light signals from road vehicles.

This object is achieved by the subject matter of the independent claims. Advantageous modifications are explained in the dependent claims.

A first aspect of the present invention is a method of modifying an optical light signal
which is to be projected from a vehicle onto a surface in the periphery of the vehicle, the method comprising:.

An "optical light signal" in the sense of this invention is an image which is created by a projector or a beamer. Typically, such a light signal is created by digitally controlling a number of light sources or by directing the beam of a single light source into different directions e. by a MEMS device.

The "surface" onto which the optical light signal is to be projected is shown herein as the ground. However, the term may also include a wall or any other object which extends upward (against the direction of gravity) from the ground. This further includes the body of another vehicle. Such a surface may also consist partially in an area of the ground and in other parts of the surface(s) of one or more walls or other objects in the light path.

Light signals are often projected onto the pavement of the road in front of the vehicle or onto a sidewalk on the side of the vehicle. These surfaces are relatively flat and rather horizontal. Tolerances in inclination of a few degrees do not have a big effect on the intensity of the light signal. As the curvature of such surfaces is normally also limited, distortion does not affect the light signal very much. However, under circumstances which are not so close to ideal, the perception of the light signal on a surface with a significant amount of inclination and/or curvature will show distortion and or strong variations in intensity, depending on the geometry of the surface. As mentioned above, it may become necessary to project the image partially onto the ground and other parts of the image onto one or more walls or other obstructions. The solution according to the first aspect results in projected light signals which are correctly perceived by an observer, irrespective of surface geometry, because the modification adapts the projected light signal to the respective surface.

In some embodiments, the method further comprises:.

An "observer" in the sense of this invention is often a human observer, be it an occupant of the vehicle like the person driving the vehicle or a person in the periphery of the vehicle like a passenger, a person riding a bicycle, a motorcycle or a scooter. The term "observer" may further comprise an animal. Another "observer" may be camera and electronics of another road vehicle, which may or may not be an autonomous vehicle.

The position of the observer relative to the projected light signal is also very important for the light signal to be perceived correctly. Therefore, it is very helpful if the position of the observer is also taken into account to modify the projected image.

In some embodiments, the modification of the light signal is based on optical properties of the surface.

"Optical properties" of the surface in the sense of this invention are color, brightness (i. intensity), reflectivity, surface structure.

It is helpful if the optical light signal is modified so that it is easier to perceive by the observer. For example, on a light-colored surface, the light signal should be displayed in a darker color. On a very dark or black surface the optical light signal should be projected in white-color.

In some embodiments, the modification of the light signal is based on data representing environmental conditions.

"Environmental conditions" in the sense of this invention are humidity which can be detected by the rain sensor of the vehicle and the temperature of the surrounding of the vehicle. Other sensory data which is detected from the environment of the vehicle can also be used. An example is the sound of a siren detected with a microphone.

It is helpful to determine environmental conditions like rain or freezing temperatures and use data on these environmental conditions in order to modify the signal accordingly. A black surface of the road can exhibit mirror like reflection when it is wet with rain or covered by a few millimeters of water. It is helpful if this mirror like property of the surface is taken into account for the modification of the optical light signal.

In some embodiments, the determination of the position of the observer comprises determining a position of a headbox or an eye position of the observer.

Cameras and lidar or radar sensors can be used in order to determine the position of a headbox or the eye position of the observer. To determine the eye position, the method of eye tracking is often used. Using data on the position of the headbox or the eye position of the observer will allow the calculation of the modification to become more precise.

In some embodiments, the modification of the light signal is based on the presence of an obstruction.

An "obstruction" in the sense of the invention is an object which is preventing the observer from perceiving the optical light signal correctly if it is blocking the line of sight. A wall for example may obstruct the view of the observer or result in a distorted image when the image is supposed to be projected on the surface of the ground on which the vehicle is standing.

It is helpful to take obstructions like walls into account and modify the optical light signal accordingly so that it can be better perceived by the observer without being blocked or distorted.

A second aspect of the present invention is a computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out the method according to the first aspect or any of its embodiments.

A third aspect of the present invention is a computer readable storage medium comprising instructions which, when executed by a computer, cause the computer to carry out the method according to the first aspect or any of its embodiments.

A fourth aspect of the present invention is a signaling device of a vehicle, as defined in claim <NUM>.

A "signaling device" in the sense of this invention is a device which is suitable to output the light signal representing the image to the surroundings of the vehicle. A typical example of a signaling device comprises a control computer and a projector which is mounted to the vehicle and which is beaming information onto a surface in the periphery of the vehicle. The headlights of the vehicle can also be used as a projector. The method of the first aspect can be beneficially realized with a device according to the fourth aspect.

As mentioned, the signaling device is configured to modify the optical light signal according to data representing at least three-dimensional spatial coordinates of the surface.

Such a signaling device is used to avoid distortions of the optical light signal and/or to avoid missing parts of the image of the optical light signal.

In some embodiments, the signaling device is configured to modify the optical light signal according to data representing a position of a headbox or an eye position of the observer.

Such a signaling device is able to display the image of the optical light signal correctly independent of the position of the observer.

A fifth aspect of the present invention is a road vehicle comprising a signaling device according to the fourth aspect.

The method according to the first aspect is preferably executed with one of the devices according to the fourth aspect, especially one of the aspect embodiments. The devices according to the fourth are preferably configured to execute the method according to the first aspect, especially one of its embodiments.

Further advantages, features and applications can be found in the detailed description as follows.

Throughout the Figures, the same reference numerals are used for the same elements of the invention or for elements corresponding to each other.

<FIG> is a block diagram showing the controller <NUM> of the vehicle <NUM>. This controller <NUM> is connected to sensors like the camera system <NUM>, which may contain a VCSEL or other illumination system, the radar or lidar sensors <NUM> for measuring the distance of objects and an outdoor microphone <NUM>. Further, the controller <NUM> is connected to sensors like rain sensor (not shown), temperature sensor (not shown), humidity sensor (not shown), in-cabin sensors and other sensors. An environment representation device <NUM> creates data of the objects in the environment of the vehicle. This environment representation device <NUM> contains an image classifier <NUM> to determine objects like walls, pavement of the road, road signs, trees and other objects.

Further, the image classifier <NUM> is equipped to determine the presence of humans, animals and other vehicles like cars, motorcycles, scooters and bicycles. For animals and humans, the environment representation device <NUM> also contains a device <NUM> for the determination of a head box and/or for eye tracking. This device <NUM> for determining a position of an observer is configured to detect the position in space of a head box of a human or an animal and/or the position in space of an eye or the eyes of a human or an animal relative to the vehicle with the controller <NUM>. The environment representation device <NUM> also contains a surface distance measurement device <NUM> and an optical properties determination device <NUM>.

The observer position determination device <NUM>, the surface measurement device <NUM> and the optical properties determination device <NUM> will build their respective data based on sensor signals from the camera system <NUM> and the radar or lidar sensors <NUM>. These sensor signals are processed by the image classifier <NUM>. A decision logic device <NUM> receives data from the image classifier <NUM> in the environment representation device <NUM>. These received data are a representation of the periphery of the vehicle. The data received by the decision logic device <NUM> will contain data representing positions of animated beings like humans or animals in the surroundings of the vehicle. The data received by the decision logic device <NUM> will further contain data representing positions of other vehicles in the surroundings of the vehicle. The data received by the decision logic device <NUM> will also contain data representing a three-dimensional map of the surroundings of the vehicle. These data will represent the relative distances of objects in the surrounding of the vehicle like the pavement, trees, walls and sidewalks The term "distance" as used herein comprises a horizontal distance from the vehicle to the object, an angular distance, for example an angle with respect to the driving direction of the vehicle and/or a distance in height relative to the vehicle. These map data are obtained from the surface measurement device <NUM>. It should be understood that the data received by the decision logic device <NUM> from the environment representation device <NUM> are dynamic data. This means on one hand that the vehicle may be moving. Thereby the relative distances of objects in the surrounding of the vehicle will change. Furthermore, the relative distances between objects in the surrounding of the vehicle may also change because humans, animals or vehicles may be moving with respect to pavement, trees, walls and sidewalks.

The decision logic device <NUM> is configured to determine an image which should be projected onto a surface in the surroundings of the vehicle. This image may be a line showing the recommended distance for a bicycle, motorcycle or other vehicle to observe when passing the vehicle. This line may be enhanced by characters indicating the recommended distance like "<NUM>" or "5ft", depending on the location. The image may further be a projection of a light beam to serve as a deterrent for an animal approaching the vehicle. The image may also be a projection outlining the rescue lane or an emergency corridor in the case of a traffic jam. The image may also be an indication of a suggested walking direction for a person exiting the vehicle. Another example of the image can be a simple warning sign.

The decision logic device <NUM> will, after determination of the image, pass data representing the image to the signaling device <NUM>. The decision logic device <NUM> will also indicate to the signaling device <NUM> onto which surface in the surroundings of the vehicle the image should be projected. The signaling device <NUM> will further receive data from the environment representation device <NUM>. These data will especially include data representing the distance of the surface from the vehicle. These distance data will be received by the signaling device <NUM> via the environment representation device <NUM> from the surface measurement device <NUM>. In the same manner, the signaling device <NUM> will receive data on the optical properties of the surface from the optical properties determination device <NUM>. Normally the signaling device <NUM> will simply adapt the light signal representing the image according to the relative angle between light beam and the projection surface so that the image can be perceived correctly by an observer.

Based on the data from the distance measurement device <NUM> and the optical properties determination device <NUM>, the signaling device <NUM> calculates the light signal to be emitted by the transducer <NUM>. Further the signaling device <NUM> determines if the light signal emitted by the transducer <NUM> is to be modified. Such a modification becomes necessary if the surface onto which the optical light signal is to be projected is not horizontal. Other modifications may become necessary to compensate for the optical properties of the surface onto which the optical light signal is to be projected. This will be explained in detail below.

<FIG> is a first schematic front view of a vehicle emitting a warning signal. It shows the vehicle <NUM> with a transducer <NUM> on the right-hand mirror and with headlights <NUM> and <NUM>. The mounting of the transducer <NUM> to the right-hand mirror is only shown as an example. There are numerous other possibilities where the transducer <NUM> can be mounted to the vehicle <NUM>. Some of these possibilities are the B column, the sill underneath the doors or the bumper. It is to be understood that it is also possible to mount several transducers <NUM>. The headlights <NUM> and <NUM> may also be used as transducers.

A warning signal is to be projected onto the ground next to the right-hand side of the vehicle <NUM>. As an example, an image of a warning signal <NUM> is shown in <FIG>. The ground on which the vehicle is standing is denoted with the reference numeral <NUM>. To the right of the vehicle <NUM>, the ground <NUM> is inclined downward. In this example, the ground surface <NUM> is horizontal. The horizontal surface extends to the right of the vehicle where points on the ground <NUM> and <NUM> are indicated. At point <NUM>, the inclined surface <NUM> extends further to the right side of the vehicle through points <NUM> and <NUM>.

<FIG> is a x-y graph of the intensities of the situation of <FIG>. The surface distance measurement device <NUM> in the environmental representation device <NUM> is used to determine geometrical data on the surfaces of inclined ground <NUM> and horizontal ground <NUM> on which the vehicle is standing. The surface distance measurement device <NUM> determines the surface for the projection as horizontal between <NUM> and <NUM> and as inclined between <NUM> and <NUM>. These positions are shown on the horizontal x-axis of the graph in <FIG>. For easier reference, dashed lines parallel to the y-axis of the graph indicate these positions as well. The y-axis indicates the intensity of the light of the image as it is projected on surfaces <NUM> and <NUM>.

Without any modification of the signal, the signaling device <NUM> calculates the light signal for a horizontal surface which extends from point <NUM> to point <NUM> and further to a point above point <NUM> in <FIG>. This horizontal surface is indicated by a dotted line in <FIG>. The intensity of the projected image thus results as indicated by line <NUM> for points <NUM> and <NUM> on surface <NUM>. Because of the inclination of the surface <NUM> between points <NUM> and <NUM>, the intensity of the projected image decreases with increasing distance from point <NUM>, reaching its lowest value at point <NUM>. This is indicated by line <NUM>. In other words, depending on the inclination of surface <NUM>, the projected image on surface <NUM> will fade away more and more towards point <NUM>. Therefore, the signaling device <NUM> uses the data from the surface distance measurement device <NUM> in the environmental representation device <NUM> to increase the intensity of the projected image parts gradually between points <NUM> with the highest value for the intensity at point <NUM>. This results in an intensity as indicated by lines <NUM> and <NUM> which is identical for all points between points <NUM> and <NUM>. This is an example of determining data representing the surface onto which the light signal is to be projected, wherein these data are three-dimensional spatial coordinates of inclined surface <NUM> and horizontal ground surface <NUM> on which the vehicle is standing.

<FIG> is a second schematic front view of a vehicle emitting a warning signal. It shows the vehicle <NUM>, and an observer <NUM>. The observer <NUM> is shown as a symbol of a human eye. The vehicle <NUM> is shown with a transducer <NUM> on the right-hand mirror and with headlights <NUM> and <NUM>.

A warning signal is to be projected onto the ground next to the right-hand side of the vehicle <NUM>. This warning signal is to be perceived by the observer <NUM>. The position of the observer <NUM> is determined from image classification device <NUM> and head box determination and/or eye tracking device <NUM> based on data from the camera system <NUM> and radar/lidar sensor <NUM> (see <FIG>). This is an example of the part of the method of determining a position of an observer of the projected light signal in three-dimensional space relative to the vehicle. The decision logic <NUM> will decide which image should be displayed based on information from the camera system and other sensors. Such an image to be displayed can be a warning that an opening of the right-hand door of the vehicle <NUM> is imminent. This is an example of the part of the method of determining a first image which is to be projected onto the surface by means of the light signal.

As an example, an image of a warning signal <NUM> is shown in <FIG>. The ground on which the vehicle is standing is denoted with the reference <NUM>. There is a curb <NUM> to a sidewalk <NUM>. The observer <NUM> is standing on the sidewalk <NUM>. Because of this curb <NUM>, the observer <NUM> is not able to perceive any light signal projected into the area <NUM>, which is depicted as shaded. The surface distance measurement device <NUM> in the environmental representation device <NUM> is used to determine geometrical data on the surfaces of sidewalk <NUM>, curb <NUM> and ground <NUM> on which the vehicle is standing. This is another example of determining data representing the surface onto which the light signal is to be projected, wherein these data are three-dimensional spatial coordinates of the surfaces of sidewalk <NUM>, curb <NUM> and ground <NUM> on which the vehicle is standing.

If the warning signal <NUM> is to be projected between points <NUM> and <NUM>, the transducer <NUM> has to project a first part of the light signal to the area between the points <NUM> and <NUM> and a second part of the light signal to the area between the points <NUM> and <NUM>.

It should be noted that this modification of the light signal is strongly dependent on the position of the observer <NUM>. Should the observer <NUM> be located inside the vehicle <NUM> (not shown in <FIG>), the projection of the warning signal would also include part of the image of the warning signal to be projected onto curb <NUM>. In other words, the shaded area <NUM> is not visible to an observer <NUM> as shown in <FIG>. Therefore, an image for this observer <NUM> cannot be projected into area <NUM>. In contrast, for an observer inside vehicle <NUM>, the projection of the light signal must absolutely include the area <NUM> in order to avoid gaps in the projected image.

Data from the camera system <NUM> can also be used by the optical properties determination device <NUM> to determine the optical properties of the surfaces of sidewalk <NUM>, curb <NUM> and/or ground <NUM> on which the vehicle <NUM> is standing. These optical properties are color, reflectivity and reflection properties like diffuse or mirror-like. These optical data are also influenced by environmental conditions. For example, the reflection property becomes more mirror-like for a wet surface and more diffuse when the surface of the ground is covered by frost in cold temperatures during winter. Therefore, it is also helpful to use data from the temperature sensor and the rain sensor in order to determine environmental properties in the periphery of vehicle <NUM> using the environmental representation device <NUM>.

<FIG> shows the image of the warning signal as it is being projected by transducer <NUM>. As described above, the first part <NUM> of the light signal is projected between the lines <NUM> and <NUM>. The line <NUM> is going through the point <NUM> and is vertical to the plane of <FIG>. The line <NUM> is going through the point <NUM> and is vertical to the plane of <FIG>. In this way, the observer <NUM> is enabled to perceive the left-hand part <NUM> of the image <NUM> on the surface of the sidewalk <NUM>. The second part <NUM> of the light signal is projected between the lines <NUM> and <NUM>. The line <NUM> is going through the point <NUM> and is vertical to the plane of <FIG>. The line <NUM> is going through the point <NUM> and is vertical to the plane of <FIG>. In this way, the observer <NUM> is enabled to perceive the right-hand part <NUM> of the image <NUM> on the surface of ground <NUM> on which the vehicle is standing. <FIG> shows an example of the modification of the light signal so that the image <NUM> which is projected onto the surfaces <NUM> and <NUM> can be visually correctly perceived as the first image by the observer <NUM>.

It is further helpful to modify the light signal in color and brightness based on the data on optical properties of the surfaces of sidewalk <NUM> and ground <NUM> on which the vehicle is standing and based on environmental properties in the periphery of the vehicle <NUM>.

<FIG> is a top view of a road vehicle emitting an optical light signal. In the same way as described above in connection with <FIG>, the decision logic device <NUM> of the vehicle <NUM> has arrived at the conclusion that an animal is approaching the vehicle <NUM>. Based on data from image classification device <NUM>, the decision logic device <NUM> decides that the animal approaching the vehicle is a herbivore. Further the decision logic device <NUM> sends an instruction to the signaling device <NUM> to display the image <NUM> of the jaws of the carnivore. In this case the modification consists in displaying the image <NUM> in a color which serves as a deterrent for the herbivore. For deer, this color can be blue.

<FIG> is a second top view of a road vehicle emitting an optical light signal. In the same way as described above in connection with <FIG>, the decision logic device <NUM> of the vehicle <NUM> has arrived at the conclusion that another vehicle is approaching the vehicle <NUM>. The decision logic device <NUM> sends an instruction to the signaling device <NUM> so that it causes the transducer <NUM> to display an image <NUM> to display the safety distance by a line and the safety distance in characters, in this example "<NUM>". Both the position of the line and the characters indicating the safety distance can be modified according to local regulations and the distance measuring units in the respective locality of the vehicle <NUM>.

Similar to <FIG> it is also possible that the road vehicle emits a different optical signal so that an emergency corridor is displayed on the surface of the road. The decision logic device <NUM> has derived from the sound of a siren of an emergency vehicle that such a vehicle is approaching. The sound of the siren is detected by the microphone <NUM> and will have been processed by the environmental detection device <NUM>.

<FIG> is a third top view of a road vehicle emitting an optical light signal. When the right-hand door of the vehicle <NUM> is about to be opened, the decision logic device <NUM> may display the image <NUM> of an arrow. This arrow serves to indicate the suggested walking direction of the passenger who is about to exit the vehicle.

<FIG> is a fourth top view of a road vehicle emitting an optical light signal. The situation is very similar to the situation depicted in <FIG>. However, there is a wall <NUM> which is obstructing the surface area which is needed for projecting the arrow <NUM>. Therefore, the arrow <NUM> is not displayed. Instead, the image is modified so that a smaller arrow <NUM> is projected by the transducer <NUM>.

Claim 1:
Method (<NUM>) of modifying an optical light signal which is to be projected from a vehicle (<NUM>) onto a surface (<NUM>, <NUM>) in the periphery of the vehicle (<NUM>), the method comprising:
determining (<NUM>) a first image (<NUM>) which is to be projected onto the surface by means of the light signal;
determining (<NUM>), using an environment representation device (<NUM>) which comprises a surface distance measurement device (<NUM>) and an optical properties determination device (<NUM>), data representing the surface (<NUM>, <NUM>) onto which the light signal is to be projected;
modifying (<NUM>) the light signal so that the image which is projected onto the surface (<NUM>, <NUM>) can be visually correctly perceived as the first image by an observer (<NUM>) based on the data representing the surface (<NUM>, <NUM>);
wherein the data representing the surface (<NUM>, <NUM>) comprise at least three-dimensional spatial coordinates of the surface (<NUM>, <NUM>).