Patent Description:
Nowadays, cabin sensing cameras used for tracking and monitoring cabins of vehicles (like cars, trucks etc.) typically use imagers that are able to record color images according to the red/green/blue (RGB) color space as well as infrared (IR) images. This may be achieved by using an RGB-IR color filter array (CFA). Hence, individual pixels may have different filters to capture different parts of the light spectrum. Native IR pixels are primarily sensitive to IR light, having a wavelength of around <NUM> nanometers (nm). Visible light pixels are primarily sensitive to the colors red, green, and blue. However, all visible light pixels are in addition sensitive to the IR light spectrum. Hence, the raw sensor information always contains both RGB and IR pixels and it depends on the further processing whether the image results in a color image or an IR image.

Typically, one has to decide in which mode an RGB-IR camera should operate at a time, or whether it should alternate between different modes. For example, such a camera could be configured to acquire only RGB or only IR images. In addition, the camera could be configured to acquire RGB and IR images in an alternating way, for example according to a sequence RGB - IR - RGB - IR - RGB - IR. In such an alternating mode, the RGB images are single exposure frames with limited dynamic range. Specifically in cabin sensing applications one has to find a quality tradeoff between parts of an image that show the cabin of the vehicle and parts that show the surrounding of the vehicle captured through the vehicle windows. If an RGB-IR camera is configured to provide the parts of an image that show a cabin and/or the passengers of a vehicle in good quality, the parts of the image that show the surrounding of the vehicle (e.g., visible through the windows of a vehicle) are often-times saturated due to much higher intensity of light, which is specifically true for daylight scenes. To provide an acceptable image quality, HDR images could be used, which could be generated by stacking two or more RGB images with different exposure times of the alternating sequence shown above. However, since an IR image lies between two RGB images, the captured scene might have changed between the two RGB images so that a stacking would lead to an unsatisfying image quality (e.g., due to motion blurring, ghosting effects, etc.). Since the IR images are typically required for computer vision task, one cannot simply replace them with further RGB images with different exposure times.

Hence, using methods known in the art to obtain both IR images and HDR images would require a capture sequence of three images: IR - RGB with long exposure time - RGB with short exposure time (then RGB image with long exposure time could be merged with the RGB image with short exposure time to obtain an HDR image). However, since three frames are required for this process, the overall frame rate, specifically for the provision of IR images, would be decreased, preventing them for tracking fast eye movements of a vehicle's driver.

<CIT> discloses an image capturing device comprising an image sensor making it possible to obtain both an infrared image and an image in the visible range by means of an array of elementary optical filters comprising first optical filters, which are at least partially transmissive in the infrared range, and second optical filters which are at least partially transmissive in the visible range. The image capturing device comprises a calculator which is programmed to control the image sensor to perform two shots, one being exposed according to an ambient brightness in the infrared range for obtaining the first image and the other being exposed as a function of an ambicin brightness in the visible range for obtaining the second image.

<CIT> discloses a method for processing RGB-Infrared (RGB-IR) sensor data that includes receiving a raw RGB-IR image, determining whether to process the raw RGB-IR image in day mode or night mode, generating, when day mode is determined, an infrared (IR) subtracted raw Bayer image from the raw RGB-IR image and processing the IR subtracted raw Bayer image in an image signal processor (ISP), and generating, when night mode is determined, an IR image from the raw RGB-IR image.

Therefore, there is a need to provide a method and apparatus for generating an HDR image from captured images by at the same time providing IR images at a high frame rate.

Aspects of the present disclosure are set out in the accompanying independent and dependent claims.

In an aspect, the present invention concerns a computer-implemented method according to claim <NUM> for generating a high dynamic range (HDR) image. The method comprises the steps of obtaining a short exposure color-IR image captured with an activated infrared (IR) illuminator and a long exposure color-IR image captured with a deactivated IR illuminator, wherein the short exposure color-IR image is captured with a shorter exposure time than the long exposure color-IR image. The method further comprises determining an IR light impact of the of the short exposure color-IR image and of the long exposure color-IR image, mitigating the IR light impact in the short exposure color-IR image to obtain a short exposure IR-suppressed color image, mitigating the IR light impact in the long exposure color-IR image to obtain a long exposure IR-suppressed color image, and merging the short exposure IR-suppressed color image with the long exposure IR-suppressed color image to obtain a merged HDR image. Generating an HDR image as described above benefits from the fact that the short exposure color-IR image, preferably stored in a RAW format, captured with an active IR illumination (e.g., provided by a near infrared (NIR) illuminator or an IR illuminator connected and controlled by the camera used for capturing the images) contains little color information in regions where the active IR illumination is strong. In regions that are very bright in the long exposure color-IR image, preferably stored in a RAW format, captured with an inactive IR illumination (e.g., areas of the image showing the surrounding of a vehicle through one or more windows of the vehicle during daytime with strong sunlight), the corresponding regions in the short exposure color-IR image captured with an active IR illumination have less impact compared to the amount of visible light and ambient IR light in the sun spectrum. Hence, during merging of the two images, the regions in the short exposure color-IR image captured with an active IR illumination that correspond to the bright regions in the long exposure color-IR image captured with an inactive IR illumination may be weighted higher, resulting in a merged HDR image with improved quality compared to a single exposure RGB image. Since the merged HDR image is derived from color-IR images, it is also possible to derive an IR-image from the color-IR images (without or with reduced impact of visible light), which may then be used for computer vision tasks, whereas the merged HDR image may be used for human vision tasks (using the merged HDR image for human vision tasks may require a further processing of the merged HDR image). Generating and HDR image and an IR image in the above described way allows to provide both image type with a higher frame rate compared to prior art methods, which typically rely on an image sequence comprising one IR image and two color images.

In a further aspect, the step of mitigating the IR light impact in the short exposure color-IR image further comprises transforming the short exposure color-IR image into an IR image, and subtracting the IR image from the short exposure color-IR image.

Mitigating the IR light impact in the short exposure color-IR image in the above-described way allows using the IR image for two tasks. As a first task, the IR image may be used for the mitigation of the impact of IR light in the short exposure color-IR image resulting in the short exposure IR-suppressed color image. Second, the IR image may be reused in computer vision tasks (e.g., object detection and/or classification), which typically operate on IR images. Hence, the above-described aspect provides an efficient way for realizing the mitigation since the intermediate result (the IR image) may be further used for other tasks. Furthermore, the short exposure IR-suppressed color image comprises only (or almost only) visible light information, rendering it suitable for merging it with another long exposure IR-suppressed color image to produce a merged HDR image.

In yet a further aspect, the step of mitigating the IR light impact in the short exposure color-IR image further comprises rearranging a color-IR channel configuration of the short exposure color-IR image to match an output of a common color camera.

Rearranging the color-IR channel configuration may replace the original RGB-IR color filter array (CFA) pattern with a CFA pattern of a standard color camera (e.g., RGGB). This may simplify the further processing of the image since a plurality of know (hardware) accelerators for supporting/accelerating image processing are only able to operate on CFA patterns of standard color cameras.

In a further aspect, the steps of subtracting and rearranging are executed in parallel.

Performing the aforementioned steps in parallel may reduce the computational time required for performing the steps, resulting in an improved efficiency.

In a further aspect, the step of mitigating the IR light impact in the long exposure color-IR image further comprises subtracting IR information from the long exposure color-IR image, and preferably during the step of subtracting IR information from the long exposure color-IR image, rearranging a color-IR channel configuration of the long exposure color-IR image to match an output of a common color camera.

Rearranging the color-IR channel configuration may replace the original RGB-IR CFA pattern with a CFA pattern of a standard color camera (e.g., RGGB). This may simplify the further processing of the image since a plurality of know (hardware) accelerators for supporting/accelerating imaging processing are only able to operate on CFA patterns of standard color cameras. Performing both steps in parallel may reduce the computational time required for performing the steps, resulting in an improved efficiency.

In yet a further aspect, the step of merging further comprises merging the short exposure IR-suppressed color image with the long exposure IR-suppressed color image based on a blending factor. The method may further comprise determining the blending factor based on the short exposure IR-suppressed color image and the long exposure IR-suppressed color image; or the short exposure color-IR image and the long exposure color-IR image; or the short exposure IR-suppressed color image, the long exposure IR-suppressed color image, the short exposure color-IR image and the long exposure color-IR image.

Merging based on a blending factor allows to control how strong which parts of each of the two merged images affect to merged HDR image to be generated and thus provides a control mechanism improving the image quality of the merged HDR image, specifically when the image includes overly bright and overly dark regions. Providing different ways for calculating the blending factor may improve flexibility of the method.

In yet a further aspect, the step of merging further comprises determining a brightness of the short exposure IR-suppressed color image and the long exposure IR-suppressed color image, aligning the brightness of the short exposure IR-suppressed color image and the long exposure IR-suppressed color image, performing a white balance adjustment of the short exposure IR-suppressed color image and the long exposure IR-suppressed color image based on the aligned brightness.

Aligning the brightness of the two images in a processing step allows using a standard white balance adjustment algorithm. Performing a white balance adjustment may improve the quality of the HDR image, since, for example, color distortion resulting from a highly saturated pixel may be avoided.

In yet a further aspect, the step of merging further comprises merging the short exposure IR-suppressed color image with the long exposure IR-suppressed color image to obtain the merged HDR image, wherein the short exposure IR-suppressed color image is a predecessor or a successor in time of the long exposure IR-suppressed color image.

Merging the two images in this way may allow outputting an IR image (e.g., usable in computer vision tasks) with the half frame rate supported by the used camera, whereas the generated HDR images may be generated at full frame rate without requiring the imager of the camera to have the capability of embedded HDR merging. In addition, it allows generating the HDR images only at half of the supported frame rate, resulting in a generation process that requires less computational power.

In yet a further aspect, the step of merging comprises merging the short exposure IR-suppressed color image with the long exposure IR-suppressed color image and with a further long exposure IR-suppressed color image to obtain the merged HDR image, wherein the long exposure IR-suppressed color image is a predecessor in time of the short exposure IR-suppressed color image, and wherein the further long exposure IR-suppressed color image is a successor in time of the short exposure IR-suppressed color image.

According to this aspect, the HDR image is generated based on one short exposure IR-suppressed color image and on two long exposure IR-suppressed color images, resulting in a HDR image with a higher dynamic range compared to an HDR image that has been generated based on two images.

In yet a further aspect, the method further comprises obtaining the HDR image by performing at least one of: performing a color reconstruction of the merged HDR image and/or applying an image enhancement algorithm to the merged HDR image, preferably comprising one or more of: tone mapping based on a tone map, lens shading correction, black level correction, image warping/distortion correction, and/or denoising.

By performing a color reconstruction (e.g., demosaicing), the HDR image rmay be usable for human vision tasks. Applying one or more image enhancement algorithms may further improve image quality.

In yet a further aspect, the method further comprises using the IR image in a computer vision task, preferably comprising one or more of the following: sensing the state of a cabin of a vehicle, determining whether a driver of the vehicle is subject to microsleeping, determining the number of passengers of the vehicle and/or determining a gaze direction of the driver. In addition, the method may comprise using the HDR image in a human vision task, preferably comprising one or more of the following: displaying the HDR image on a display and/or streaming the HDR image to an output unit, preferably a video call unit.

Having both the IR image and a high quality HDR image available, preferably generated at half or full frame rate of the used camera, allows using both image types in different tasks. The IR images are particularly well suited for computer vision tasks, whereas the HDR images are particularly well suited for human vision tasks. Since both image types are provided according to a high frame rate, specifically computer vision tasks operating on IR images, like gaze direction detection, benefit from the disclosed method.

In a further aspect, the present invention relates to a computer-implemented method for generating a high dynamic range (HDR) image for use in an infotainment system. The method comprises generating the HDR image according to the method as disclosed above, and using the HDR image in a human vision task, preferably comprising displaying the HDR image on a display, and/or streaming the HDR image to an output unit, preferably a video call unit.

A further aspect of the present invention relates to an apparatus according to claim <NUM> comprising means configured for performing the method according to claim <NUM>.

A further aspect of the present invention relates to a computer program according to claim <NUM> comprising instructions, which when executed by a computer, cause the computer to perform the method according to claim <NUM>.

A further aspect of the present invention relates to a vehicle comprising the aforementioned apparatus.

Various aspects of the present invention are described in more detail in the following by reference to the accompanying figures without the present invention being limited to the embodiments of these figures.

<FIG> schematically shows an RGB-IR camera <NUM> connected to a NIR illuminator <NUM>. The NIR illuminator <NUM> may also be arranged in the camera <NUM>. The camera <NUM> may be configured to acquire images, wherein the NIR illuminator <NUM> may be turned on and off in an alternating way. <FIG> depicts a situation in which the NIR illuminator <NUM> is inactive or deactivated, whereas <FIG> depicts a situation in which the NIR illuminator <NUM> is active or activated, as indicated by the IR light rays <NUM>. An image acquired while the NIR illuminator <NUM> is active may be a short exposure color-IR image, whereas an image acquired while the NIR illuminator <NUM> is inactive may be long exposure color-IR image. The short exposure color-IR image may be acquired with an exposure time that is shorter than the exposure time used for acquiring the long exposure color-IR image, as expressed by the terms "long exposure" and "short exposure".

Both the short exposure color-IR image and the long exposure color-IR image are preferably acquired in RAW format (e.g., as RGB-IR CFA images). Both image types are affected by visible and IR light (both types of light are included in ambient light, e.g. sun light). Since the short exposure color-IR image is captured with an active NIR illuminator <NUM>, the range of light captured by the camera <NUM> includes more infrared light during capturing this image type compared to acquiring a long exposure color-IR image with an inactive NIR illuminator <NUM>.

To produce an HDR image and an IR image, the IR part of the short exposure color-IR image may be extracted (at least up to a certain degree of accuracy) and stored as in IR image, which may then be further used, e.g., in computer vision tasks. The IR image may then be subtracted from the short exposure color-IR image. The resulting image may be stored as a first processed image, having a reduced (or even eliminated) IR light impact. Also from the long exposure color-IR image, the IR light impact may be removed (which is less string compared to the short exposure color-IR image since the long exposure color-IR image has been acquired with the NIR illuminator <NUM> deactivated) at least to some degree, resulting in a second processed image. Both the first and second processed image may then be merged, and optionally further processed, to obtain an HDR image. The generation of the HDR image and the merging operation will be further explained in detail with respect to <FIG> and <FIG>.

<FIG> depicts an algorithm <NUM> for generating an HDR image <NUM>. As an input for algorithm <NUM>, two images <NUM> and <NUM> are used that have been captured by an RGB-IR camera <NUM> having a corresponding imager, wherein the visible light pixels of the imager are sensitive to both visible and IR light. Image <NUM> is a short exposure color-IR image that has been captured with an active IR illumination. The IR illumination may be provided by a NIR illuminator <NUM>, which is active during capturing of image <NUM>. The NIR illuminator <NUM> may be connected to and controlled by the RGB-IR camera <NUM>. Preferably, image <NUM> is an RGB-IR CFA image in RAW format captured with a short exposure time. Image <NUM> is a long exposure color-IR image captured with an inactive IR illumination, which means that no NIR illuminator <NUM> is active during capturing. Preferably, image <NUM> is an RGB-IR CFA image in RAW format captured with a long exposure time. In other words, image <NUM> is captured with a longer exposure time than image <NUM>. Image <NUM> maybe captured before or after image <NUM>. However, the time between the two capturing processes is preferably short such that the captured scene is essentially the same in both images <NUM> and <NUM>.

Capturing the short exposure color-IR image <NUM> and the long exposure color-IR image <NUM> could be done with independent automatic exposure times. Alternatively, the short exposure color-IR image <NUM> cold be captured with an automatic exposure time, whereas the long exposure color-IR image may be captured with an exposure time relative to the automatic exposure time of the short exposure color-IR image. Asa third option, the long exposure color-IR image may be captured with an automatic exposure time, whereas the short exposure color-IR image is captured with an exposure time relative to the automatic exposure time of the long exposure color-IR image.

Providing different ways of exposure control may improve flexibility and adaptability of the disclosed method to different scenery conditions, e.g., with respect to varying light conditions in different regions of the captured images. Moreover, it may be controlled which parts of the image (dark or light regions) are to be provided in high quality.

On raw signal level, both images <NUM> and <NUM> have the same format and CFA pattern. However, due to the difference in ambient illumination (with and without active IR illumination) and exposure time, the images capture different parts of the dynamic range of a scene. Image <NUM>, captured with active IR illumination, still contains some color information in the color pixels.

In step <NUM>, a first preprocessing algorithm is executed. The first preprocessing algorithm processes image <NUM> to obtain an IR image <NUM>, preferably in full resolution, suitable for computer vision tasks. Hence, the first preprocessing algorithm according to step <NUM> is configured to only output the IR part of the light spectrum captured in image <NUM> and thus removes the impact of visible light from image <NUM>. An example implementation of the first preprocessing algorithm may upscale the IR channel of image <NUM> to full resolution and use the visible light pixels to steer the amount of averaging to keep the edges in the image <NUM> sharp (e.g., by means of bilateral filtering).

Both image <NUM> and IR image <NUM> are further processed by a second preprocessing algorithm <NUM> to obtain a short exposure IR-suppressed color image <NUM>. Preferably, image <NUM> is a full resolution RGGB color image in RAW format that only comprises the visible part of the light spectrum, which is achieved by removing the impact of invisible IR light. The second preprocessing algorithm maybe implemented in two different ways.

In a first instance, the second preprocessing algorithm <NUM> is a combination of subtracting the IR image <NUM> or its content from image <NUM> and of applying a remosaicing algorithm that rearranges the native RGB-IR channel configuration from the imager's CFA to a pattern that matches the output of a common RGB camera <NUM>. Such pattern may be beneficial for further processing using a hardware accelerated demosaicing function that could not work with IR pixels in the CFA (the remosaicing may be skipped if an RGB-IR pattern is directly supported by the demosaicing step <NUM>). In a second instance, the second preprocessing algorithm <NUM> may subtract IR image <NUM> or its content from image <NUM> and may perform the remosaicing after the subtraction (the remosaicing step may be skipped if an RGB-IR pattern is directly supported by the demosaicing step <NUM>). Both instances of the second preprocessing algorithm <NUM> may comprise correction routines for handling saturated pixels.

Image <NUM> is processed by a third processing algorithm according to step <NUM> to obtain a long exposure IR-suppressed color image <NUM>. The third processing algorithm <NUM> removes the contribution of IR light from the visible light pixels and applies a remosaicing, as described with respect to the second preprocessing algorithm according to step <NUM>. Again, the remosaicing step may be skipped according to the conditions as outlined with respect to the second preprocessing algorithm <NUM>. The remosaicing step may be configured differently compared to step <NUM> to take into account the absence of active IR illumination. To perform the preprocessing algorithm <NUM>, methods known in the art may be used. For the further processing of images <NUM> and <NUM>, however, it has to be ensured that both images <NUM> and <NUM> are based on the same RAW color image pattern.

In step <NUM>, images <NUM> and <NUM> are merged to obtain a merged HDR image <NUM>, preferably in RGGB RAW format. The step of merging will be described in more detail with respect to <FIG>. The merging algorithm <NUM> takes into account the fact that image <NUM> contains little color information in regions where active IR illumination is strong, while in regions that are very bright in image <NUM> (e.g., for a vehicle cabin scene if looking outside the windows at sunlight), image <NUM> is weighted much higher in merging step <NUM> since the active IR part is much less impactful compared to the amount of IR light in the sun spectrum. The weighting may computed as a function of the ratio of visible light and IR light in the local neighborhood of a given pixel position.

Optionally, as indicated by the dashed arrows in <FIG> that connect images <NUM> and <NUM> with step <NUM>, images <NUM> and <NUM> may also be passed to merging algorithm <NUM>, which may further improve the image quality of the resulting HDR image <NUM> and/or may reduce the required computation time.

In step <NUM>, demosaicing is applied to the merged HDR image <NUM> to reconstruct a full color image from the incomplete color samples output from the imager (image sensor) of the used camera <NUM> overlaid with a CFA. In addition, optionally, one or more image enhancement algorithms may be applied in step <NUM>, comprising one or more of tone mapping based on a tone map, lens shading correction, black level correction, image warping/distortion correction and/or denoising. Both steps are preferably executed by an image signal processor (ISP).

Using algorithm <NUM> enables providing an IR image <NUM>, which may be a full resolution monochrome IR image, usable for computer vision tasks by at the same time providing an HDR image <NUM>, usable for human vision tasks.

<FIG> depicts an algorithm <NUM> for merging images <NUM> and <NUM> that may be applied in step <NUM> of algorithm <NUM>. In step <NUM>, a blending factor calculation is performed based either on images <NUM> and <NUM> or based on images <NUM> and <NUM>. Optionally, all four images <NUM>, <NUM>, <NUM> and <NUM> may be used for the blending factor calculation. In step <NUM>, a blending factor <NUM> may be calculated based on the luma component of image <NUM> and the amount of remaining signal in image <NUM> after IR light component mitigation/suppression. In addition or alternatively, images <NUM> and image <NUM> may be used to conduct this step. Using only images <NUM> and <NUM> may improve the performance of the blending factor calculation.

Using images <NUM> and <NUM> (in addition or alternatively) for the blending factor calculation may result in an improved blending factor computation, specifically in use cases where the images reflect the inside of a vehicle cabin. In such a case, the short exposure color-IR image <NUM> may favor outside pixels (e.g., pixels referring to regions visible through a vehicle window), while it the long exposure color-IR image <NUM> may favor the inside of the cabin. This allows detecting outside pixels (high ambient light pixels) based on the amount of IR light in native IR pixels. This may result in higher values for pixels that are illuminated from the active illumination (IR or NIR illumination; typically close to camera). These pixels are likely inside the cabin accordingly.

Benefits in image quality may comprise fewer ghosting effects (double edges in case of mixing long and short exposure images <NUM> and <NUM> from different capture times). If all pixels inside a window region are coming from one frame, they are all typically captured from the same time step and thus do not show the double edges.

In step <NUM>, an exposure compensation is performed based on both images <NUM> and <NUM>. In this step, the brightness of the images <NUM> and <NUM> is adjusted to be at the same level. This may help to have a seamless output, resulting in two brightness-adjusted images <NUM> and <NUM> based on images <NUM> and <NUM>, respectively.

In step <NUM>, a white balance preprocessing is performed based on the brightness-adjusted images <NUM> and <NUM>. Since the brightness of the two images <NUM> and <NUM> has already been aligned in step <NUM>, a standard white balancing algorithm may be used in step <NUM>. As an output, step <NUM> produces two white-balanced pre-adjusted images <NUM> and <NUM> based on images <NUM> and <NUM>, respectively. The reason for performing a white balance preprocessing is that the color of the pixels with high luma values may be distorted, because of the saturation in one channel. Typically, the green color channel gets saturated at first (other channels can also saturate, but are typically less sensitive). Without adjusting the green channel bright pixels will become pinkish after white balance correction. Hence, the corresponding pixels are eliminated in step <NUM>.

In step <NUM>, the two white-balanced pre-adjusted images <NUM> and <NUM> are merged according to the blending factor <NUM> to generate the merged HDR image <NUM>, preferably in terms of an RGGB image, which may then be further processed according to steps <NUM> and <NUM> as depicted in <FIG> according to algorithm <NUM> to generate the HDR image <NUM>.

Example merging sequences that may be applied in step <NUM> of algorithm <NUM> are depicted in <FIG>.

<FIG> depicts a first example merging sequence <NUM>. A short exposure IR-suppressed color image <NUM>, which corresponds to short exposure IR-suppressed color image <NUM> according to algorithm <NUM>, is merged with a long exposure IR-suppressed color image <NUM>, which corresponds to long exposure IR-suppressed color image <NUM> according to algorithm <NUM>, to obtain and HDR image <NUM>, which corresponds to HDR image <NUM> according to algorithm <NUM>. In this example, image <NUM> is derived from a short exposure color-IR image <NUM> (see <FIG>), whereas image <NUM> is derived from a long exposure color-IR image <NUM> (see <FIG>), wherein short exposure color-IR image <NUM> has been captured before the long exposure color-IR image <NUM>, as indicated by the time axis according to the black arrow. Preferably, both images <NUM> and <NUM> were taken at short time intervals, so that the scenery shown in both images has not changed or has changed only slightly.

After having generated HDR image <NUM>, a further HDR image <NUM> may be obtained by merging image <NUM> with image <NUM>. Image <NUM> is a short exposure IR-suppressed color image, which corresponds to short exposure IR-suppressed color image <NUM> according to algorithm <NUM>, whereas image <NUM>, as outlined above, is a long exposure IR-suppressed color image, which corresponds to long exposure IR-suppressed color image <NUM> according to algorithm <NUM>. Image <NUM> is derived from a short exposure color-IR image <NUM> (see <FIG>), whereas image <NUM> is derived from a long exposure color-IR image <NUM> (see <FIG>), wherein short exposure color-IR image <NUM> has been captured after the long exposure color-IR image <NUM>, as indicated by the time axis according to the black arrow. Preferably, both images <NUM> and <NUM> were taken at short time intervals, so that the scenery shown in both images has not changed or has changed only slightly. In a similar way, HDR images <NUM> and <NUM> are generated by merging images <NUM>, <NUM> and <NUM>.

Using this merging approach allows outputting HDR images <NUM> at full frame rate. Assuming that the used camera <NUM> supports a frame rate of <NUM> frames per second, <NUM> HDR images <NUM> could be generated per second. At the same time, <NUM> IR images <NUM> may be produce per second, since IR images can only be derived from short exposure color-IR images <NUM>. This is because only short exposure color-IR images <NUM> have been captured with active IR illumination.

<FIG> depicts a second example merging sequence <NUM>. A short exposure IR-suppressed color image <NUM>, which corresponds to short exposure IR-suppressed color image <NUM> according to algorithm <NUM>, is merged with a long exposure IR-suppressed color image <NUM>, which corresponds to long exposure IR-suppressed color image <NUM> according to algorithm <NUM>, to obtain and HDR image <NUM>, which corresponds to HDR image <NUM> according to algorithm <NUM>. In this example, image <NUM> is derived from a short exposure color-IR image <NUM> (see <FIG>), whereas image <NUM> is derived from a long exposure color-IR image <NUM> (see <FIG>), wherein short exposure color-IR image <NUM> has been captured after the long exposure color-IR image <NUM>, as indicated by the time axis according to the black arrow. Preferably, both images <NUM> and <NUM> were taken at short time intervals, so that the scenery shown in both images has not changed or has changed only slightly.

After having generated HDR image <NUM>, a further HDR image <NUM> may be obtained by merging image <NUM> with image <NUM>. Image <NUM> is a short exposure IR-suppressed color image, which corresponds to short exposure IR-suppressed color image <NUM> according to algorithm <NUM>, whereas image <NUM> is a long exposure IR-suppressed color image, which corresponds to long exposure IR-suppressed color image <NUM> according to algorithm <NUM>. Image <NUM> is derived from a short exposure color-IR image <NUM> (see <FIG>), whereas image <NUM> is derived from a long exposure color-IR image <NUM> (see <FIG>), wherein short exposure color-IR image <NUM> has been captured after the long exposure color-IR image <NUM>, as indicated by the time axis according to the black arrow. Preferably, both images <NUM> and <NUM> were taken at short time intervals, so that the scenery shown in both images has not changed or has changed only slightly.

Using this merging approach allows outputting HDR images <NUM> at half frame rate. Assuming that the used camera <NUM> supports a frame rate of <NUM> frames per second, <NUM> HDR images <NUM> could be generated per second. Identical to the first merging sequence <NUM>, <NUM> IR images <NUM> may be produced per second. Compared to the first merging sequence <NUM>, merging sequence <NUM> comprises a lower number of merging operations, resulting in less computational power required for performing the merging sequence <NUM> compared to merging sequence <NUM>.

<FIG> depicts a third example merging sequence <NUM>. A short exposure IR-suppressed color image <NUM>, which corresponds to short exposure IR-suppressed color image <NUM> according to algorithm <NUM>, is merged with a long exposure IR-suppressed color image <NUM>, which corresponds to long exposure IR-suppressed color image <NUM> according to algorithm <NUM> and with a further long exposure IR-suppressed color image <NUM>, to obtain and HDR image <NUM>, which corresponds to HDR image <NUM> according to algorithm <NUM>. In this example, image <NUM> is derived from a short exposure color-IR image <NUM> (see <FIG>), whereas images <NUM> and <NUM> are derived from first and second long exposure color-IR images <NUM> (see <FIG>), wherein short exposure color-IR image <NUM> has been captured after the first long exposure color-IR image <NUM> and before the second long exposure color-IR image <NUM>, as indicated by the time axis according to the black arrow. Preferably, all three images <NUM> and <NUM> were taken at short time intervals, so that the scenery shown in the images has not changed or has changed only slightly.

Using this merging approach based on three images allows capturing first and second long exposure color-IR images <NUM> with different exposure times, which may result in a higher dynamic range in HDR image <NUM>.

<FIG> depicts a short exposure color-IR image <NUM> showing the cabin of a car. Image <NUM> shows a right front side window <NUM>, a right rear side window, <NUM>, a rear window <NUM>, a left rear side window <NUM>, a left front side window <NUM>, a driver seat <NUM> and a passenger seat <NUM>. In addition, image <NUM> shows two color test patterns <NUM> and <NUM>.

<FIG> depicts a long exposure color-IR image <NUM> showing the same cabin of a car as depicted in <FIG>. Hence, image <NUM> shows the right front side window <NUM>, the right rear side window, <NUM>, the rear window <NUM>, the left rear side window <NUM>, the left front side window <NUM>, the driver seat <NUM> and the passenger seat <NUM>. In addition, image <NUM> shows the two color test patterns <NUM> and <NUM>.

Since image <NUM> according to <FIG> has been captured with active IR illumination, the outside scenery with a large light intensity (due to sunlight) surrounding the car is shown with more details compared to <FIG>, as can be seen in windows <NUM>, <NUM> and <NUM> of <FIG>, whereas the outside scenery with less sunlight impact is shown with less details, as can be seen in windows <NUM> and <NUM> of <FIG>. In contrast, since image <NUM> has been captured with an inactive IR illumination and with a higher exposure time than image <NUM>, the outside scenery with less sunlight impact is shown with more details, as can be seen in windows <NUM> and <NUM> of <FIG>, whereas the scenery with a larger sunlight impact is shown with less details, as can be seen in windows <NUM>, <NUM> and <NUM> (the corresponding pixels are saturated).

In addition, the image <NUM> according to <FIG>, the two color test patterns <NUM> and <NUM> are shown with more color details compared to image <NUM> according to <FIG>, due to the longer exposure time.

<FIG> depicts a blending factor <NUM>, which may be determined in step <NUM> of algorithm <NUM> and used for merging images <NUM> and <NUM> in step <NUM> of algorithm <NUM>. The brighter a pixel is in the blending factor <NUM>, the higher the corresponding pixel in the short exposure IR-suppressed color image <NUM> is weighted during merging. Likewise, the darker a pixel is in the blending factor <NUM>, the higher the corresponding pixel in the long exposure IR-suppressed color image <NUM> is weighted during merging.

<FIG> depicts a merged HDR image <NUM> which may result from merging algorithm <NUM> and wherein a demosaicing according to step <NUM> has already been applied.

<FIG> depicts an HDR image <NUM> that has been processed by image enhancement algorithms <NUM>. In this example, specifically a tone mapping has been conducted, where in the range of tones has been remapped as well as a color tuning and/or a color calibration. As can be seen from <FIG>, HDR image <NUM> shows details in all side windows <NUM>, <NUM>, <NUM>, and <NUM>. The same is true for rear window <NUM>. In addition, also details of the seats <NUM> and <NUM> are visible. As can be seen from the color test patterns <NUM> and <NUM>, HDR image <NUM> also comprises more color details compared to images <NUM> and <NUM>.

This makes HDR image <NUM> particularly suitable for being displayed on display for human vision tasks as well as for being used in video calls. For example, a video call unit may receive the HDR images as input. In addition, an HDR image could be used as a selfie that could be posted in a social media network.

By contrast, IR image <NUM> may be used in computer vision tasks that typically operate on infrared images. Example tasks are sensing the state of a cabin of a vehicle, determining whether a driver of the vehicle is subject to microsleeping (if microsleeping is detected, a visual, audiovisual or audio-based alert could be issued), determining the number of passengers of the vehicle and determining a gaze direction of the driver. A further example comprises reducing the risk of accidents by detecting drowsiness of the driver and alerting the driver before it comes to microsleeping.

The disclosed method can be configured in different variants of exposure settings for short and long exposure images. It does not require a fixed ratio of exposures, nor does the merge algorithm <NUM> require the knowledge of the exposure time of the input images. However, if the exposure times are known, this information can be used for brightness adjustment.

Variant A: Short exposure color-IR image <NUM> captured with an active IR illumination is captured with an independent auto exposure setting, e.g., to optimize the image <NUM> for driver state sensing tasks with good visibility of the driver's faces. Long exposure color-IR image <NUM> captured with an inactive IR illumination is captured with an auto exposure setting, e.g., configured to obtain good color images of the cabin inside, accepting that windows can saturate.

Variant B: Short exposure color-IR image <NUM> captured with an active IR illumination is captured with an independent auto exposure setting, e.g., to optimize the image <NUM> for driver state sensing tasks with good visibility of the driver's faces. Long exposure color-IR image <NUM> captured with an inactive IR illumination is captured with an exposure setting relative to image <NUM>, e.g., using a configurable fixed ratio.

Variant C: Long exposure color-IR image <NUM> captured with an inactive IR illumination may be captured with an auto exposure setting, e.g., configured to obtain good color images of the cabin inside, accepting that windows can saturate. Short exposure color-IR image <NUM> captured with an active IR illumination may be captured with an exposure setting relative the auto exposure setting of image <NUM>, e.g., using a configurable fixed ratio.

Variant D: Image statistics from merged images are considered for auto an exposure control algorithm, wherein long exposure color-IR image <NUM> captured with an active IR illumination and short exposure color-IR image <NUM> captured with an inactive IR illumination are derived from the statistics.

The disclosed method may not only be used in cameras <NUM> for sensing cabins of vehicles, but also in surveillance cameras <NUM> or other camera types.

For example, in some embodiments, the present invention may be realized as a computer-implemented method, a computer-readable memory medium, or a computer system.

In some embodiments, a computing device may be configured to include a processor (or a set of processors) and a memory medium, where the memory medium stores program instructions, where the processor is configured to read and execute the program instructions from the memory medium, where the program instructions are executable to implement any of the various method embodiments described herein (or, any combination of the method embodiments described herein, or, any subset of any of the method embodiments described herein, or, any combination of such subsets). The device may be realized in any of various forms.

Claim 1:
A computer-implemented method for generating a high dynamic range, HDR, image (<NUM>), the method comprising:
obtaining a short exposure color-infrared, color-IR, image (<NUM>) captured with an activated infrared, IR, illuminator (<NUM>) and a long exposure color-IR image (<NUM>) captured with a deactivated IR illuminator (<NUM>), wherein the short exposure color-IR image (<NUM>) is captured with a shorter exposure time than the long exposure color-IR image (<NUM>);
determining an IR light impact of the short exposure color-IR image (<NUM>) and of the long exposure color-IR image (<NUM>);
mitigating the IR light impact in the short exposure color-IR image (<NUM>) to obtain a short exposure IR-suppressed color image (<NUM>);
mitigating the IR light impact in the long exposure color-IR image (<NUM>) to obtain a long exposure IR-suppressed color image (<NUM>); and
merging the short exposure IR-suppressed color image (<NUM>) with the long exposure IR-suppressed color image (<NUM>) to obtain a merged HDR image (<NUM>).