Patent Description:
With the increased availability of devices such as terminal devices, i. smartphones, tablets or the like, photography with these terminals becomes more and more popular increasing the demands on image quality and appearance. Therefore, the acquired raw image undergoes several correction steps usually implemented in an image signal processor (ISP). One of these correction steps is white-balancing or color correction considering the chromaticity of the illumination in the scene and restoring the real color of the object under this illumination.

Automatic white-balancing (AWB) is a camera control algorithm that estimates the chromaticity of the illumination in the scene and has high impact on the colors and perceptual quality of the final image. Therein, current techniques have a color cast problem when image was captured under mixed illuminations. In mixed illumination cases in which there is e.g. a window or other image areas that corresponds to different illumination than the main illumination in the scene, white-balancing and color correction might lead to insufficient results. Conventional ISP and AWB support only one white point per image. This means that in mixed illumination conditions there needs to be a trade-off between multiple illuminations, and varying color casts are left in the image that will be further amplified by color space conversions and other color processing that increases color saturation. This is also the solution for all current smartphone ISPs. The single white point white balancing (also called computational color constancy) method has been extensively researched. However, for example, image regions of screens and signs are self-illuminated, and often use higher/other Correlated Color Temperatures (CCT) light sources than typical indoor illumination. Consequently, bluish color cast results on the screens and signs when a single white point is used that suits the ambient indoor illumination.

In the prior art a learning-based method exist for correction of mixed illumination using a convolutional neural network. A disadvantage of these methods is that the training data creation is an extremely labor-intensive process, which can make it unfeasible for productization in devices that must operate well anywhere in the world under uncontrolled illumination conditions.

<CIT> describes a method including: obtaining an image of a scene; generating an illumination map for the obtained image; dividing the values in the illumination map to determine a number of estimated illuminant regions, wherein each region corresponds to at least one estimated illuminant present in the captured scene; estimating a white point for each region; and applying white balancing operations, based on the estimated white points for each region.

Further prior art is known from <NPL>; and<NPL>.

Thus, it is an object of the present invention to provide a solution for the problems of the prior art.

The problem is solved by a method for acquiring an image according to claim <NUM>, a device according to claim <NUM>, a computer program according to claim <NUM> and an image signal processor according to claim <NUM>.

In a first aspect the present invention relates to a method for image processing, implemented in a terminal, including the steps of:.

In a first step an initial image is acquired. Therefore, the terminal might comprise an image sensor to directly acquire the initial image. Alternatively, the terminal might be connected via any conventional method to another device wherein the other device provides the initial image. Thus, the initial image might be stored on the other device and thus transferred via the connection to the terminal. Additionally or alternatively, the additional device may comprise an image sensor to directly acquire the initial image.

In a subsequent step at least one region of interest (ROI) is determined in the initial image. Therein, the ROI may relate to an area of changed illumination of the initial image being subject to another color of illumination or an illumination having another Correlated Color Temperature (CCT). More specific, the ROI may relate to an area of the initial image being subject to another Spectral Power Distribution (SPD). Alternatively, the ROI is determined by image recognition identifying for example objects which are usually self-illuminate such as signs, windows, displays and screens or the like. However, some of these objects which are detected could have similar CCTs or SPD as the ambient illumination.

For the at least one ROI at least one first correction parameter is determined. Further, at least one second correction parameter is determined for the initial image outside the ROI. Thus, by the first correction parameter the influence of the changed illumination inside the ROI can be corrected, wherein by the at least one second correction parameter the usual color correction or white-balancing (WB) of the initial image may be performed. Therein, the first correction parameter can be understood as spatial or local correction parameter, wherein the second correction parameter can be understood as global correction parameter.

Thus, the first correction parameter is applied to the ROI, wherein the second correction parameter is applied to the initial image outside the ROI in order to generate the final image.

Of course, before the steps of the present invention and after the steps of the present invention other steps for image correction and processing can be carried out and the initial image does not need to be the raw image for example acquired directly from an image sensor. Also, the final image might undergo further correction adjustment steps or processing steps in an image pipeline and does not need to be the image presented to the user.

By the present invention spatial white-balancing is implemented, wherein first only those areas are identified which need to be corrected and have likely an illumination deviating from the illumination of the remaining image. Thereby the first correction parameter can be determined with higher accuracy for the ROI with less computational efforts since determining of the first correction parameter is limited to the ROI. Also, the second correction parameter can be determined with higher accuracy since the image limited to the outside of the ROI is subject to the same or homogeneous illumination and thus a common second correction parameter is applicable to this area and provides sufficient results without or at least reduces color casts.

Preferably, the at least one ROI is determined by a first neural network (NN). Thus, by training of the first NN reliable detection of the ROI in the initial image can be performed. Therein, the first NN can be software implemented or hardware implemented, i.e. part of an image signal processor (ISP) or provided by a dedicated and separate processor.

Preferably, the at least one ROI is determined by object recognition in the initial image. Therein for example typical objects having a deviating illumination could be identified, such as screens or windows. The NN might be trained to detect objects which likely have a different SPD. Therein, it is noted that trained NNs usually are able to detect shapes of objects faster than their chromatic information. In particular, if the object is identified having a self-illumination, this information about the object can be used when determining the at least one first correction parameter. It is known that for example screens, signs and displays have another limited range for the possible correlated color temperature (CCT), wherein when determining the first correction parameter the range could be limited to the potential range of the CCT.

Preferably, the at least one ROI is determined by detecting an illumination transition region between the ROI and the area outside the ROI. Therein, in particular for detecting of the illumination transition region, the ROI might be segmented and the illumination might be analyzed in these segments to detect the transition region of the illumination. If an illumination transition region is detected for a specific ROI, the first correction parameter for this ROI can be determined with high accuracy wherein computational efforts are reduced due to the limited area of the ROI in the initial image. More reliable prediction of the first correction parameter is thus feasible.

Preferably, before determining the at least one ROI, the initial image is downsampled and the at least one ROI is determined in the downsampled image. Thereby computational efforts for determining the ROI are further reduced and the speed for processing the image can be increased.

Preferably, before determining the at least one ROI, the method includes: detecting mixed illumination in the initial image and if no mixed illumination is detected, determining one common correction parameter for the complete initial image.

Thus, if it is detected that the initial image has a common and generic illumination, conventional determining of the correction parameter is applied for the complete initial image. Thus, upon of the initial test for a mixed illumination in the initial image, it can be quickly determined whether identification of one or more ROIs is necessary to handle these areas individually.

Preferably, detection of the mixed illumination is performed by a second neural network. Therein, the second NN can be software implemented or hardware implemented, i.e. part of the image signal processor (ISP) or provided by a dedicated and separate processor.

Preferably, several correctional parameters are determined for the at least one ROI and the area outside. Thus, by the several correction parameters a 2D correction map might be determined to be applied for correction of the initial image. Therein, the 2D correction map might include the one or more first correction parameters for the at least one ROI as local correction parameters and the one or more second correction parameter as global correction parameters for the area outside the at least one ROI together. Therein, the number of the first correction parameter is independent from the number of the second correction parameter. Thus, one common second correction parameter as global correction parameter can be used while for the at least one ROI a plurality of local first correction parameters are used. Alternatively, one common first correction parameter can be used for the at least one ROI, while for the area outside the at least one ROI a plurality of second correction parameters are used which might be generated by interpolation. Alternatively, a plurality of local first correction parameter can be used for the at least one ROI, while for the area outside the at least one ROI a plurality of local second correction parameters are used.

Preferably, the 2D correction map is upsampled before applying to the initial image. Therein usual upsampling algorithms such as interpolation, bicubic interpolation, nearest neighbour interpolation, bilinear interpolation, bicubic spline interpolation, generalized bicubic interpolation, bilateral upscaling, or the like can be used for upsampling. Upsampling is in particular necessary if the initial image is downsampled for detecting the ROI, wherein the first correction parameter is determined for the downsampled initial image.

Preferably, a plurality of ROIs are determined in the initial image wherein for each ROI at least one first correction parameter is determined and subsequently applied. Consequently, a plurality of first correction parameters is determined wherein at least one second correction parameter is determined for the area of the initial image outside each of the ROI. Alternatively, for at least two ROIs a common correction parameter is determined. This case is applicable for example in the case that areas of changed illumination relate to the same illumination source such as a row of windows to the outside. Another reason might be that when the ROI is small, it might be difficult to get an accurate WP from the limited area of the ROI. Then, a common correction parameter might be used.

Preferably, to at least one ROI a combination of the first correction parameter and the second correction parameter is applied. Thus, by applying a combination of the first correction parameter and the second correction parameter to the at least one ROI, too sharp transitions in the final image can be avoided and a more natural appearance of the final image can be achieved. Therein, combination can be performed by averaging WP or using weighted averaging, wherein the weights are for example determined by the second NN. If it is a gradual illumination transition, a gradually changed weight might be used. Instead, an abrupt weight might be applied to the border of ROI.

According to the present invention, the first correction parameter is one or more of a white point (WP) used for white-balancing of the ROI, or a color correction value used for color correction of the ROI. Alternatively or additionally, the second correction parameter is one or more of a white point (WP) used for white-balancing of the area outside the ROI, or a color correction value used for color correction of the area outside the ROI. Therein the white point either of the ROI or the area outside the ROI may be determined by conventional white-balancing algorithms limited to the specific area of the initial image, i. the at least one ROI or the plurality of the ROIs individually, or the area outside the ROIs, respectively.

In a further aspect of the present invention a terminal is provided comprising an image sensor and a processor, wherein the processor is configured to perform the steps of the method described before. In particular, the terminal might be a smartphone, tablet, a camera or the like.

In a further aspect, a software is provided including instructions when executed by a computing device performing the steps of the method described before.

In a further aspect an image signal processor (ISP) is provided configured to perform the steps of the method described before. Therein, the ISP might be implemented in a terminal.

In the following the present invention is described in more detail with reference to the accompanying figures.

Referring to <FIG> showing a flow diagram according to the present invention. In step S01, an initial image is acquired. Therein, the initial image might be acquired by an image sensor or the terminal implementing the steps of the invention. Alternatively, the initial image might be transmitted to the terminal implementing the steps of the invention from another device. The initial image may be raw image data or a pre-processed image.

In step S02, at least one region of interest (ROI) is determined in the initial image. Therein, the ROI may relate to an area of changed illumination of the initial image being subject to another color of illumination or an illumination having another Correlated Color Temperature (CCT). More specific, the ROI may relate to an area of the initial image being subject to another Spectral Power Distribution (SPD). Alternatively, the ROI is determined by image recognition identifying for example objects which are usually self-illuminate such as signs, windows, displays and screens or the like. However, some of these objects which are detected could have similar CCTs or SPD as the ambient illumination. Thus, in the prior art, ROI would not be detected and a common correction parameter would be determined globally for complete image which would result in insufficient images and color casts.

However, in the present invention, in step S03, at least one first correction parameter for the at least one ROI and at least one second correction parameter for the initial image outside the ROI is determined. Therein, the first correction parameter can be understood as spatial or local correction parameter of the ROI and the second correction parameter can be understood as global correction parameter. Thus, the illumination change of the ROI is individually considered and allows for tailored color correction of the ROI independent from the color correction of the area outside the ROI, i.e. relating to the main illumination of the initial image. Some of sign, screens and windows which are detected by object recognition could have similar CCTs or SPD as ambient illumination. In this case, the correction parameters may not change much.

In step S04, the first correction parameter is applied to the ROI and the second correction parameter is applied to the initial image outside the ROI to generate the final image.

Thereby the first correction parameter can be determined with higher accuracy for the ROI with less computational efforts since determining of the first correction parameter is limited to the ROI. Also, the second correction parameter can be determined with higher accuracy since the image limited to the outside of the ROI is subject to the same or homogeneous illumination and thus a common second correction parameter is applicable to this area and provides sufficient results without or at least reduces color casts.

Referring to <FIG>, the present invention is exemplified for the spatial automatic white-balancing (AWB). However, the present invention can also be used for color correction in general. The AWB starts at block <NUM>. Therein, AWB might be a process in the image pipeline, preferably implemented in an image signal processor of a terminal, such as a smartphone, tablet or camera. In a subsequent block <NUM> it is detected whether a mixed illumination is present in the initial image. If no mixed illumination can be detected in the initial image, the conventional AWB algorithm is used in block <NUM>, wherein for the example of AWB only one white point (WP) is applied to the complete initial image.

Detection of the mixed illumination may be performed by a neural network that has been trained. Therein, the mixed illumination detection can be implemented as a separate NN or as a part of an AI based single WP AWB algorithm (e.g. re-use the CNN feature layers and have a separate fully connected NN layers for mixed illumination classification).

If in block <NUM> a mixed illumination is detected, i.e. that at least two different illuminations are present in the initial image, spatial AWB analysis is started. In block <NUM> first one or more ROIs are detected indicating those areas in the initial image which have a deviating illumination or SPD than the remaining initial image or can be identified as objects likely to be self-illuminant such as displays, screens, signs and windows. Since in block <NUM> there is a detection whether a changed illumination is present in the initial image, at least one ROI is present in the initial image and detected in block <NUM>. In block <NUM> it is determined that the ROI has a sharp transition region. This can be done by segmentation of the ROI and object recognition to detect the transition regions and thereby evaluating a sharp transition. If the ROI has a sharp transition, then only one WP might be determined for the ROI. Contrary in block <NUM>, if the ROI has a gradual transition between the ROI and the remaining initial image, one or more WP are determined for the ROI, wherein the gradual transition can be provided for interpolation between different WP inside the ROI and outside the ROI. Blocks <NUM> and <NUM> are repeated for each ROI detected in the initial image. In block <NUM> a full 2D gain map is determined on the basis of the WP of the at least one ROI or plurality of white points for the one or more ROIs, and the remaining image. In block <NUM> the 2D gain map is applied to the initial image in order to white-balance the initial image and create the final image.

The present invention provides a solid solution for solving the white balance of images under mixed illuminations. It might be used prior knowledge for ROI detection, and segmentation methods for better boundary definition. And it proposed to differently deal with different illumination transitions.

The main steps of the technique are as follows:.

Referring to <FIG> showing the steps of the method according to the present invention. <FIG> shows the initial image <NUM>. The initial image <NUM> contains two screens <NUM> having a different illumination, i. are self-illuminant. These two screens <NUM> are detected as ROIs for example by segmentation as shown in <FIG>. For the individual ROIs WPs are determined as shown in <FIG> ("screen white points). Therein, in the example of <FIG> one common white point for both screens <NUM> is determined. Further, <FIG> shows the ground truth white point of the area outside the ROIs provided by the main light source in the image at the ceiling. For this area outside the ROIs a single WP is determined ("Single Estimate white points") which accurately fits to the ground truth of the initial image. Further, in <FIG> the upper and lower boundaries for usual WP are indicated by the lines. These boundaries might limit the search for possible WPs. <FIG> shows the ROIs for which the white points are determined and to which the receptive determined white points are applied. Thus, <FIG> represents a 2D WP gain map to be applied to the initial image for white-balancing. <FIG> shows the final image wherein to the ROIs respective white-balancing is applied on the basis of the determined white points wherein the area outside the ROIs is white-balanced on the basis of the remaining illumination of the initial image, i. the main light source on the ceiling of the initial image. Thus, a more natural appearance of the image is achieved and the image quality is improved.

In order to detect the area or the ROIs, training data can be generated referring to <FIG> shows an illumination mask including gradual transition between the areas of different illumination and a sharp transition of the illumination. The illumination mask might be applied to an initial image as shown in <FIG> having a uniform illumination (no mixed illumination) in order to create artificially training image with mixed illumination. The resulting image, as shown in <FIG> thus has a mixed illumination by superposition of the illumination mask of <FIG> and the original image (not shown). The <FIG> can then be used for training of a neural network in order to determine the ROIs. In particular, due to the illumination map of <FIG>, position of the ROI and the transition of the ROI in the resulting training image are known. Thereby training sets can be easily generated to train the neural network for detecting the ROIs in the image with high accuracy. Therein the present invention does not require accurate annotation of spatial white points or reliable object recognition to be able to detect the ROIs.

Referring to <FIG> wherein sharp illumination transition is used in screen and sign areas. Gradual illumination transition is also present by the entrance of the corridor by gradually superimposing of the artificial light within the corridor and the light outside. Both, sharp and gradual illumination transition are used in general mixed illumination regions. <FIG> shows the initial image comprising several ROI identified and for the purpose of illustration also annotated. <FIG> shows the ROIs for which the white points are determined and to which the receptive determined white points are applied. Thus, <FIG> represents a 2D WP gain map to be applied to the initial image for white-balancing. <FIG> shows the final image, wherein the appearance of the signs and screens as well as the transition to the outside of the corridor have a more natural appearance and color casts of these areas is avoided.

Claim 1:
Method for image processing implemented in a terminal including:
acquiring an initial image;
determining at least one region of interest, ROI, in the initial image;
determining at least one first correction parameter, for the at least one ROI and at least one second correction parameter for the initial image outside the ROI, wherein the correction parameter is one or more of a white point and a color correction, when the at least one ROI has a sharp transition region, one white point is determined for the at least one ROI; and when the at least one ROI has a gradual transition between the at least one ROI and the initial image outside the ROI, one or more white points are determined for the at least one ROI;
applying the first correction parameter to the ROI and the second correction parameter to the initial image outside the ROI to generate the final image.