Patent Description:
Systems for automatic headlight control typically base their decisions on explicit detection of individual vehicles or light sources. This makes it difficult to handle situations properly where vehicles appear in curves or over crests, or when headlights of oncoming vehicles are occluded by guard rails, barriers or other objects.

<CIT> discloses a light aura detection approach to assist detection of vehicles appearing in curves or over crests. This approach, however, puts rather strict requirements on the appearance of such an aura and does not address the issue of other structures or objects occluding the light sources. Additionally, this approach must be complemented by a light source detector or vehicle detector. Furthermore, to control the headlights additional logic is needed for interpreting the detections and to decide how the road ahead should be illuminated.

<CIT> discloses an illumination device including at least one light source configured to perform illumination with a plurality of illumination patterns, a detection unit for detecting state information on the state of an illumination target that is to be illuminated by the light source, an arithmetic unit configured to calculate, using a neural network, illumination pattern information for generating an illumination pattern appropriate for the illumination target from the state information, and an illumination control unit configured to control the light source in order to perform illumination with an illumination pattern based on the illumination pattern information.

The problem underlying the present invention is to provide a holistic approach for a headlight control system adapted to handle different light conditions including all kinds of situations where structures and objects occlude light sources.

The invention solves this problem with the features of the independent claims. According to the invention, a machine learning model is implemented in the data processing device which is trained to estimate and output an output signal representing a desired illumination of the vehicle environment from one or more images received as input from the imaging apparatus.

A human is able to interpret a multitude of light cues, such as halos, stray light or reflections, and thereby, for example, determine likely locations of other vehicles. To mimic this approach, the invention provides a machine learning model which has been trained to holistically consider an entire image, or a plurality of images, to directly estimate a headlight control signal, or more generally an output representing a desired illumination of the vehicle environment. Generally, a desired illumination of the vehicle environment by one or more headlights includes the angular distribution and/or intensity distribution of all light sources present in the headlight of a motor vehicle, including an illumination state of a high beam light source.

In a preferred embodiment of the invention, the machine learning model is a convolutional neural network. The machine learning model is trained to output a desired illumination profile, which is defined as a curve delimiting a desired area of illumination in an image, given an input image or several input images. In this case, the output signal output by the machine learning model advantageously comprises a desired illumination profile as defined above, expressed in angles relative to the optical axis of a headlight.

Ground truth data is generated by manual annotation, i.e., by a human annotator. For distances significantly longer than the separation between imaging device (camera) and headlight, the desired illumination profile is closely approximated by a curve delimiting the desired area of illumination in the image, which can easily be identified by a human annotator. Therefore, the machine learning model can advantageously be trained to output a curve delimiting a desired area of illumination in an image, rather than, for example, a desired illumination profile.

Alternative output representations of the desired illumination, to be output by the machine learning model, may be utilized. For example, the machine learning model may be trained to output a desired distance profile, which is preferably defined as the illumination distance per horizontal angular section. In this case, the output signal output by the machine learning model advantageously comprises a desired distance profile as defined above. The above described "alternative output" is not part of the claimed invention.

In another embodiment, which is not part of the claimed invention, the machine learning model may be trained to output a desired per-pixel intensity map of the headlight. In other words, the output signal output by the machine learning model advantageously comprises a desired per-pixel intensity map. This output representation can be particularly useful in combination with matrix headlights, wherein each headlight comprises a matrix of light sources, like LEDs, the intensity of which can be controlled individually.

To make use of temporal information in the input, the machine learning model could take several sequential images as input, and/or exhibit internal feedback, e.g. as in a recurrent neural network.

Preferably, the output signal is transmitted to a headlight controller generating a headlight adjustment signal, wherein preferably the headlight adjustment signal is fed back to the machine learning model. In other words, for improved performance, the machine learning model may be provided with the current status of the headlights, which may be called an external feedback.

In a preferred embodiment, one might want to lower the intensity or illumination angle of the headlights actively, to enable detection of, for example, a halo from an oncoming vehicle that is occluded from view to the host vehicle by the terrain. The machine learning model could therefore be trained to identify situations where it is desirable to actively lower the intensity or illumination angle of the headlights, and to output an appropriate control or output signal for achieving this. The period of time for which the light intensity is lowered is preferably shorter than can be perceived by the driver.

If desirable, for example for reasons related to robustness or memory footprint, the machine learning model could preferably be adapted to process individually different parts of an input image in order to provide local contributions to the desired illumination, for example to the desired illumination profile, in said output signal from different parts of the field of view of the imaging apparatus.

The invention also refers to a method of training a machine learning model for a headlight control system as described above. The machine learning model is trained in a supervised manner using a ground truth training data set specifying the desired output for each input sample, namely one image or a plurality of images, in the training data set.

Preferably, the ground truth training data set comprises one or more images, which are preferably captured by an existing imaging apparatus.

In the embodiment of the invention, ground truth data is generated by manual annotation of said one or more images. According to the invention, the ground truth training data set comprises annotations in the form of a curve delimiting the desired area of illumination.

Ground truth training data may be generated in several alternative or complementary, which do not form part of the present invention. For example, the ground truth training data set may be generated using object detections from an existing object detection system, and/or an object tracker to track detected objects over consecutive images. Since the object detections are used offline in this case, the objects may preferably be tracked in a non-causal manner to a point outside or beyond the initial detection range.

Alternatively, the ground truth training data set may be generated using semi-automatic annotations using output from an existing headlight control system or object detection system.

In still another alternative embodiment, the ground truth training data set is generated using a recording of manual high beam control signals.

In the following the invention shall be illustrated on the basis of preferred embodiments with reference to the accompanying drawings, wherein:.

The headlight control system <NUM> is to be mounted in or to a motor vehicle and comprises an imaging apparatus <NUM> for capturing images of a region surrounding the motor vehicle, for example a region in front of the motor vehicle. The imaging apparatus <NUM> may be mounted for example behind the vehicle windscreen or windshield, in a vehicle headlight, or in the radiator grille. Preferably the imaging apparatus <NUM> comprises one or more optical imaging devices <NUM>, in particular cameras, preferably operating in the visible wavelength range, in the infrared wavelength range, or in both visible and infrared wavelength range, where infrared covers near IR with wavelengths below <NUM> microns and/or far IR with wavelengths beyond <NUM> microns. In some embodiments the imaging apparatus <NUM> comprises a plurality of imaging devices <NUM> in particular forming a stereo imaging apparatus <NUM>. In other embodiments only one imaging device <NUM> forming a mono imaging apparatus <NUM> can be used.

The imaging apparatus <NUM> is coupled to an on-board data processing device <NUM> adapted to process the image data received from the imaging apparatus <NUM>. The data processing device <NUM> is preferably a digital device which is programmed or programmable and preferably comprises a microprocessor, a microcontroller, a digital signal processor (DSP), and/or a microprocessor part in a System-On-Chip (SoC) device, and preferably has access to, or comprises, a digital data memory <NUM>. The data processing device <NUM> may comprise a dedicated hardware device, like a Field Programmable Gate Array (FPGA), an Application Specific Integrated Circuit (ASIC), a Graphics Processing Unit (GPU) or an FPGA and/or ASIC and/or GPU part in a System-On-Chip (SoC) device, for performing certain functions, for example controlling the capture of images by the imaging apparatus <NUM>, receiving the electrical signal containing the image information from the imaging apparatus <NUM>, rectifying or warping pairs of left/right images into alignment and/or creating disparity or depth images. The data processing device <NUM>, or part of its functions, can be realized by a System-On-Chip (SoC) device comprising, for example, FPGA, DSP, ARM, GPU and/or microprocessor functionality. The data processing device <NUM> and the memory device <NUM> are preferably realised in an on-board electronic control unit (ECU) and may be connected to the imaging apparatus <NUM> via a separate cable or a vehicle data bus. In another embodiment the ECU and one or more of the imaging devices <NUM> can be integrated into a single unit, where a one box solution including the ECU and all imaging devices <NUM> can be preferred. All steps from imaging, image processing to possible activation or control of a safety device <NUM> are performed automatically and continuously during driving in real time.

The invention is applicable to autonomous driving, where the ego vehicle is an autonomous vehicle adapted to drive partly or fully autonomously or automatically, and driving actions of the driver are partially and/or completely replaced or executed by the ego vehicle.

The headlight control system <NUM> comprises one or more, for example two headlights <NUM> with at least one light source <NUM>. Preferably, each headlight <NUM> is dynamically adjustable, i.e. the light profile of at least one light source <NUM> including the angular distribution and/or the intensity of the emitted light can be changed over time by an adjustment device <NUM> and controlled by a headlight controller <NUM>. The headlight controller <NUM> can be part of the processing device <NUM> or a separate processing device and part of the same ECU with the processing device <NUM>, or a different ECU. The imaging apparatus <NUM> is preferably directed in approximately the same direction as the headlights <NUM>, such that the field of view of the imaging apparatus <NUM> and the illumination region of the headlights <NUM> at least partially overlap.

The adjustment device <NUM> may be adapted to adjust the corresponding light source <NUM> in such a manner that the light beam or light cone <NUM> emitted by the headlight <NUM> is moved in a lateral direction and/or in a vertical direction or any other direction, as indicated by the arrows at the side of the light cones <NUM>. The adjustment device <NUM> can be adapted to turn the complete headlight <NUM>, to block or shield different parts of the light beam <NUM>, to move one or more optical elements within the headlight <NUM>, to change optical properties of one or more optical elements within the headlight <NUM>, or any other suitable mechanism. The adjustable headlight <NUM> may be an advanced lighting system, in particular based on LEDs, which can shape the light beam <NUM> around the oncoming vehicle without dazzling the oncoming driver.

The adjustment device <NUM> may be adapted to perform high beam control, i.e. to turn on and off the high beam comprised in the headlight <NUM> automatically as controlled by a high beam controller <NUM>. The high beam controller <NUM> is preferably part of the headlight controller <NUM>, but may also be a separate part.

As will be described in the following in more detail with respect to <FIG>, the headlight controller <NUM> and/or the high beam controller <NUM>, and thus the adjustment devices <NUM> for the headlights <NUM>, are controlled during driving by the data processing device <NUM> on the basis of results obtained from data processing of the images received from the imaging apparatus <NUM>. This is also called dynamic headlight control. Therefore, the data processing device <NUM> together with the headlight controller <NUM> and/or the high beam controller <NUM> forms a dynamic headlight control device or controller.

According to the invention, a machine learning model <NUM>, for example a convolutional neural network, is implemented in the processing device <NUM>. The machine learning model <NUM> has been trained in a training phase prior to implementing it in the processing device <NUM>, to directly estimate and output a headlight control signal and/or the desired illumination profile <NUM>, from one or more entire images <NUM> received from the imaging apparatus <NUM> and input into the machine learning model <NUM>. The training process will be described in more detail later with respect to <FIG>.

During driving of the host vehicle, an image <NUM> or a plurality of images <NUM> captured by the imaging apparatus <NUM> are input to the machine learning model <NUM>. The machine learning model <NUM> is capable of outputting an output signal <NUM> comprising a representation of a desired illumination of the vehicle environment by the headlights <NUM>. The representation is for example a curve <NUM> delimiting a desired area of illumination in an image <NUM>. As can be seen in <FIG>, the curve <NUM> is determined in a manner that the road <NUM>, in particular the ego lane <NUM> and a region <NUM> at the road edge <NUM> of the ego lane <NUM>, where pedestrians, bicyclists and/or large animals may be expected, is well illuminated, however, without dazzling the driver of an oncoming vehicle <NUM>. The curve <NUM> usually is an excellent approximation to the illumination profile, i.e. an upper vertical illumination angle per horizontal angular section. Usually, the region below the curve <NUM> can be well illuminated with high intensity, whereas the region above the curve <NUM> shall not be illuminated, or only with low intensity not dazzling the driver of an oncoming vehicle <NUM>.

The output signal <NUM> is forwarded to a headlight controller <NUM> which in turn sends a headlight control signal <NUM> to the headlights <NUM>, in particular the headlight adjusting section <NUM> thereof. The headlight control signal <NUM> adjusts the headlights <NUM> in a manner that the region below the curve <NUM> is well illuminated and the region above the curve <NUM> is not, or only sparsely, illuminated. This includes possible automatic dim-out or switching off of the high beam in the headlights <NUM>.

The training of the machine learning model <NUM> is described in the following with respect to <FIG>. A large set of training images <NUM> is generated, for example with an imaging apparatus <NUM>, which may preferably be similar or even identical to the imaging apparatus <NUM> of the vehicle where the machine learning model <NUM> is implemented. The set of training images <NUM> preferably covers an extensive range of light conditions which may occur in motor vehicle traffic.

Claim 1:
A headlight control system (<NUM>) for a motor vehicle, comprising:
a controllable headlight (<NUM>) adapted to generate variable illumination of the vehicle environment;
an imaging apparatus (<NUM>) adapted to capture images (<NUM>) from a region in front of the motor vehicle; and
a data processing device (<NUM>) adapted to perform image processing of images (<NUM>) captured by said imaging apparatus (<NUM>) and to vary the light characteristics of said controllable headlight (<NUM>) depending on said image processing, wherein a machine learning model (<NUM>) is implemented in said data processing device (<NUM>) which is trained to estimate and output an output signal (<NUM>) representing a desired illumination of the vehicle environment from one or more images (<NUM>) received as input from the imaging apparatus (<NUM>), said output signal (<NUM>) comprises a desired illumination profile;
characterized in that the machine learning model is trained using a ground truth data set comprising one or more images and manual annotations on each of the one or more images in the form of a curve delimiting the desired area of illumination, and the output signal (<NUM>) representing said desired illumination profile is defined as a curve (<NUM>) delimiting a desired area of illumination in an image (<NUM>) wherein the region below the curve (<NUM>) is well illuminated and the region above the curve (<NUM>) is not, or only sparsely, illuminated.