Method and apparatus for automatic brightness adjustment of image signal processor

An automatic brightness adjusting method and apparatus for image signal processor (ISP) is provided. The image processing apparatus may include a histogram generating unit, a cumulative distribution function calculator, and a histogram equalization (HE) unit. The histogram generating unit may generate a histogram of brightness values of pixels in an input image. The cumulative distribution function calculator may generate a cumulative distribution function and an inverse cumulative distribution function, based on the generated histogram. The HE unit may generate a conversion function based on the cumulative distribution function and the inverse cumulative distribution function, and may apply HE to the input image based on the conversion function so as to generate an output image.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims the benefit of Korean Patent Application No. 10-2010-0121428, filed on Dec. 1, 2010, in the Korean Intellectual Property Office, the disclosure of which is incorporated herein by reference.

BACKGROUND

1. Field of the Invention

The present invention relates to an automatic brightness adjusting method and apparatus for an image signal processor (ISP), and more particularly, to an automatic brightness adjusting method and apparatus that may adjust a brightness of an output image to an appropriate brightness so as to improve a contrast.

2. Description of the Related Art

An image signal processor (ISP) may be a processor that pre-processes an image signal input from an image sensor so that the image signal may be used in an application.

The ISP may perform image compensation, for example, color interpolation, color correction, gamma correction, color space conversion, and the like, and subsequently, may perform image compensation, for example, automatic white balancing (AWB) and automatic exposure (AE), and the like.

The AE may indicate a process that calculates a brightness of an image so as to automatically set the image to have an appropriate brightness or may indicate a block performing the process. That is, the AE may be a digital control technology that maintains the image to have an optimal brightness, irrespective of an illuminance of a light source or may be a device to which the digital control technology is applied.

The AE may sense an amount of projected light so as to control an iris, a shutter speed, and an analog gain.

Generally, the AE may calculate an average brightness of an image and may adjust a gain so that the image has the same brightness as a predetermined target brightness. A brightness of a screen is adjusted based on the AE and thus, a naturalness of the image may be enhanced.

The brightness adjustment by the AE may be insufficient to extract characteristics of an object requested by a vision system. Unlike a general camera, an image processing technology may need to be applied to the vision camera.

When a brightness of an image is intensively distributed in a predetermined range, the AE may be insufficient to clearly distinguish an outline or characteristics of an object. In this example, there is a desire for a process that broadens original intervals of brightness that are intensively distributed so as to improve a contrast of the image.

A histogram equalization (HE) may indicate a scheme that generates a conversion function based on a histogram of an input image, and may be used to improve a contrast.

The HE may generate an output image in which pixels having low brightness values and pixels having high brightness values are evenly distributed, irrespective of a histogram of brightness values of pixels of the input image. The HE may adjust an average brightness of an image to be constant. Therefore, the HE may be used instead of an AE function of the ISP.

In terms of implementation of hardware, the HE may be implemented by a simple operation and thus, an additional memory space for storing a histogram may be requested. Since the HE is simply implemented, the HE may process a relatively large image in a relatively quick period of time.

Generally, the HE may generate an output image that is equalized to be brighter, based on a histogram cumulative distribution function of the input image.

Therefore, when a brightness of the input image is intensively distributed in a relatively low brightness band or a relatively high brightness band, the output image based on the HE may have an improved contrast but may be distorted in an unnatural manner.

Examples of an algorithm that supplements the drawback of the HE may include a brightness preserving bi-histogram equalization (BBHE) algorithm, a dualistic sub-image histogram equalization (DSIHE) algorithm, a recursive mean-separate histogram (RMSHE) algorithm, and the like. The algorithms may be complex and may require a great amount of calculation. Therefore, when the algorithms are implemented in hardware, performance of an entire ISP chip may deteriorate.

SUMMARY

An aspect of the present invention provides an automatic brightness adjusting apparatus and method based on a histogram equalization (HE) scheme.

Another aspect of the present invention also provides an automatic brightness adjusting apparatus and method based on a a cumulative distribution function and an inverse cumulative distribution function.

According to an aspect of the present invention, there is provided a method of automatically adjusting a brightness of an image, the method including generating a histogram of brightness values of pixels of an input image, generating a conversion function based on a cumulative distribution function and an inverse cumulative distribution function of the histogram, and generating an output image by applying, based on the conversion function, histogram equalization (HE) to the input image.

The generating of the histogram may include sampling the pixels of the input image, and generating the histogram based on the sampled pixels.

The generating of the conversion function may include generating the cumulative distribution function and the inverse cumulative distribution function, based on the histogram, and generating the conversion function based on the cumulative distribution function and the inverse cumulative distribution function.

The inverse cumulative distribution function may be generated based on a number of pixels used for generating the histogram, the cumulative distribution function, and the histogram.

The conversion function may be generated based on a first HE function that is based on the cumulative distribution function and a second HE function that is based on the inverse cumulative distribution function.

The conversion function may be configured as a look-up table.

According to another aspect of the present invention, there is provided an image processing apparatus, the apparatus including a histogram generating unit to generate a histogram of brightness values of pixels of an input image, a cumulative distribution function calculator to generate a cumulative distribution function and an inverse cumulative distribution function, and a histogram equalization (HE) unit to generate a conversion function based on the cumulative distribution function and the inverse cumulative distribution function, and to generate an output image by applying, based on the conversion function, HE to the input image.

The histogram generating unit may include a dual-port static random access memory (SRAM), and the HE unit may include a single-port SRAM.

A brightness value of a pixel of the input image may be input as an address of the dual-port SRAM, and a data value of the address of the dual-port SRAM may be increased by 1 due to the input.

The HE unit may apply, based on the conversion function, HE to the input image by utilizing a look-up table.

The cumulative distribution function calculator may apply a shift operation to a value indicating a number of pixels of the input image so as to calculate an approximate value of a number of pixels used for generating the histogram.

The histogram generating unit may perform sampling of the pixels of the input image, and may generate the histogram based on the sample pixels.

EFFECT

Embodiments of the present invention may include an automatic brightness adjusting method and apparatus based on a histogram equalization (HE) scheme.

Embodiments of the present invention may include an automatic brightness adjusting method and apparatus based on a cumulative distribution function and an inverse cumulative distribution function.

DETAILED DESCRIPTION

FIG. 1illustrates a comparison of an automatic exposure (AE) scheme and a histogram equalization (HE) scheme according to an embodiment of the present invention.

Hereinafter, a scheme of calculating an AE gain will be described.

The scheme of calculating the AE gain applied to an output frame is given by Equation 1.

In Equation 1, Target_Y denotes a brightness value set for the output frame. Frame_AVE denotes an average brightness value of an input image. Frame_Gain denotes a gain value with respect to a frame.

A scheme of calculating a brightness value of a pixel of the output frame is given by Equation 2.
Pixelout=Frame_Gain×pixelin[Equation 2]

In Equation 2, Pixelindenotes a brightness value of a predetermined pixel of the input image. Pixeloutdenotes a brightness value of a pixel corresponding to the predetermined pixel of the output image.

In terms of implementation of hardware, a division operation may be required to calculate the AE gain based on Equation 1 since the average brightness value of the input image may not be constant. Also, a multiplication operation may be required to apply the calculated AE gain to the input image based on Equation 2. Conversely, the division operation and the multiplication operation may be inefficient for hardware for a high definition moving picture.

A vision system may appropriately adjust a brightness of an image and may adjust the image so that objects in the image may be distinguished.

The AE scheme based on Equation 1 and Equation 2 may adjust only a brightness of an image. Accordingly, the AE scheme may adjust a image which have a strong exposure or weak exposure, but may not provide an effect of improving a contrast.

Hereinafter, the HE scheme will be described.

A cumulative distribution function of the HE scheme may be calculated based on Equation 3.

In Equation 3, Hist (k) denotes distribution of brightness values of pixels in an image. For example, Hist (k) may denote a number of pixels having a brightness value of k among the pixels in the image. That is, Hist may be a histogram of brightness values of pixels of an input image.

Here, CDF (n) may denote a cumulative distribution function of a histogram (Hist(k)) of the input image. CDF (n) may denote a value obtained by adding all values of histograms with respect to values less than or equal to n.

An HE function with respect to a predetermined brightness value of n, that is, HE(n), may be calculated according to Equation 4.

In Equation 4, Pixeltotaldenotes a number of pixels used for generating a histogram. For example, Pixeltotalmay denote a total number of pixels in an image. Bmaxdenotes a maximum value of brightness values of pixels. HE(n) denotes a brightness value converted from the brightness value of n.

To apply the HE scheme to an image, a look-up table including HE (n) calculated based on Equation 4 may be used. In this example, n may be an integer value greater than or equal to 0 and less than or equal to Bmax. The look-up table may be a table that stores output brightness values corresponding to input brightness values. That is, an index of the look-up table may correspond to an input brightness value, and a value of an entry in the look-up table may correspond to an output brightness value.

When the look-up table is used, an output brightness value corresponding to a predetermined input brightness value may be calculated without an additional operation, as shown in Equation 5.
Pixelout=HE(Pixelin )[Equation 5]

Subsequently, a performance of the AE scheme and a performance of the HE scheme may be compared.

Referring toFIG. 1, images110,120, and130on the left side have a relatively even histogram distribution. The histogram distribution may be histogram distribution of brightness values of pixels in an image. In images115,125, and135on the right side, brightness values of pixels in the images are intensively distributed on upper portions. Images110and115at the first row are original images.

The images120and125at the second row are AE images. A target brightness of AE may be a half the maximum value of a brightness value of a pixel.

The images130and135at the third row are HE images.

According to an analysis with respect to the images110,120, and130, the original image110and the AE image120has little difference. When the HE image130is compared to the original image110, the HE image130may maintain an average brightness value and have an improved contrast.

According to an analysis with respect to the images115,125, and135, the AE image125may decrease a brightness of an entire image based on a target brightness. Accordingly, the AE image125is distorted. The HE image135has a dramatically improved contrast. Conversely, a difference in a brightness between the original image115and the HE image135increases and thus, the HE image is unnatural.

According to a comparison between the AE scheme and the HE scheme in terms of implementation of hardware, the AE scheme may require a division operation to calculate a gain value as given in Equation 1. Also, a multiplication operation may be required to apply the calculated gain value to an input image, as given in Equation 2. Therefore, as a size of an image increases, a load due to the division operation and the multiplication operation may be applied to the AE scheme.

The HE scheme may generate a look-up table according to Equation 4. In general, a number of pixels in an image and a maximum value of brightness values of the pixels in the image are constant. Accordingly, the HE scheme may be embodied without using a division operation and a multiplication operation.

When the look-up table is used, the HE scheme may be quickly applied without an additional operation.

FIG. 2illustrates an HE scheme according to an embodiment of the present invention.

A first image215may be an original image. A first histogram210may be a histogram of the first image215.

A second image225may be an AE image. A second histogram220may be a histogram of the second image225.

A third image235may be an HE image. A third histogram230may be a histogram of the third image235.

As described in the foregoing, the HE scheme may improve a contrast of an image using only a simple operation.

When brightness values of pixels are intensively distributed in a predetermined area, an image generated based on the HE scheme may be an unnatural image such as the images130and135in the third row ofFIG. 1, since a result value of a cumulative distribution function with respect to a predetermined value of n, that is, CDF (n), includes a histogram value Hist (n) of n.

Specifically, the predetermined area in which the brightness values are intensively distributed is a dark portion, the corresponding area may be converted to a significantly bright area based on the HE scheme.

Here, brightness values of pixels in the original image215are distributed in a predetermined area. In the original image215, a darkest portion is double the brightest portion.

The original image215may have an average brightness. Accordingly, a brightness and a contrast of the AE image225is modestly changed.

The HE image235may have an improved contrast when compared to the original image215and the AE image225. In this example, histogram distribution moves toward a high brightness band. Therefore, the HE image235is unnatural.

When a histogram of an input image is intensively distributed in a relatively low brightness band or a relatively high brightness band, the HE scheme may equalize the input image to be brighter. Accordingly, an output image distorted by the HE scheme may be generated.

FIG. 3illustrates a pixel sampling process for generating a histogram according to an embodiment of the present invention.

When an original image is an image in a level of full high-definition (HD), the original image may include two million pixels, that is, 1920×1080 pixels.

To calculate a histogram with respect to an image of 1920×1080 pixels, a memory of about 16.59 kilobytes (Kb) may be required.

An input image may be generated by sampling the original image, and memory consumption and an amount of operation may be reduced and thus, a hardware resource may be used effectively.

Generally, a high definition image and a low definition image obtained by decreasing the high definition image to a predetermined size has little difference in terms of information associated an image.

Referring toFIG. 3, a total average brightness of an image of 1920×1080 and a total average brightness of an image of 480×270, sampled to be one-sixteenth the size of the image of 1920×1080, have little difference. Therefore, an error between the both images may be negligible.

When sampling is performed, Pixeltotaldenotes a number of sampled pixels.FIG. 4illustrates an automatic brightness adjusting method using a cumulative distribution function and inverse cumulative distribution function according to an embodiment of the present invention.

In operation S410, a histogram with respect to brightness values of pixels in an input image may be generated.

In this example, operation S410may include sampling of the pixels in the input image and generating of the histogram based on the sampled pixels.

In operation S420, a cumulative distribution function and an inverse cumulative distribution function may be generated based on the generated histogram.

The drawback of the HE scheme, described with reference toFIG. 2in the foregoing, is caused since only the cumulative distribution function is used. However, an improved HE scheme that uses the cumulative distribution function and the inverse cumulative distribution function may supplement the drawback of the conventional HE scheme.

The inverse cumulative distribution function may be calculated according to Equation 6.
CDFinv(n)=Pixeltotal−CDF(n)+Hist(n)   [Equation 6]

In Equation 6, CDFinvdenotes the inverse cumulative distribution function. CDFinvmay be generated based on Pixeltotalindicating a number of pixels used for generating a histogram, CDF indicating the cumulative distribution function, and Hist indicating a histogram.

In operation S430, a conversion function may be generated based on the cumulative distribution function and the inverse cumulative distribution function.

The conversion function may be generated based on an HE function (HE)?) based on the cumulative distribution function and an inverse HE function (HEinv) based on the inverse cumulative distribution function.

HEinvmay be calculated according to Equation 7.

In Equation 7, Proposed_HE indicating an improved conversion function may be a weighted-sum of HE and HEinv. When the same weighted-sum is applied to HE and to HEinv, Proposed_HE may be calculated according to Equation 8.

Equation 9 may be formulated based on Equation 6 and Equation 8.

In operation S440, an output image may be generated by applying the HE scheme to the input image based on the conversion function.

In terms of implementation of hardware, implementation based on Equation 9 may be performed by adding “−Hist(n)/2” to the implementation based on Equation 4.

The conversion function may be configured as a look-up table.

FIG. 5illustrates a simulation result of an automatic brightness adjusting method using a cumulative distribution function and an inverse cumulative distribution function according to an embodiment of the present invention.

Proposed_He image515may be generated by applying a scheme described with reference toFIG. 3to the first image215ofFIG. 2. A histogram510may be a histogram of the Proposed_He image515.

The Proposed_He image515may have an improved contrast when compared to the original image215or the HE image235. The histogram510may not be intensively distributed in a high brightness band or a low brightness band.

FIG. 6illustrates a configuration of an image processing apparatus600that performs automatic brightness adjustment.

The image processing apparatus600may include an image signal processor (ISP) for a moving picture camera having a high definition. The image processing apparatus600may be a component included in the ISP and may supplement an automatic adjusting function of the ISP.

The image processing apparatus600may include a histogram generating unit610, a cumulative distribution function calculator620, and an HE unit630.

The histogram generating unit610may generate a histogram of brightness values of pixels of an input image.

The histogram generating unit610may perform sampling of the pixels in the input image, and may generate the histogram based on the sampled pixels.

The cumulative distribution function calculator620may generate a cumulative distribution function and an inverse cumulative distribution function based on the generated histogram.

The HE unit630may generate a conversion function based on the cumulative distribution function and the inverse cumulative distribution function, may apply an HE scheme to the input image based on the conversion function, and may generate an output image.

The histogram HE unit630may include a look-up table. The HE unit630may embody the conversion function based on the look-up table, and may apply the HE scheme to the input image based on the conversion function, through the look-up table.

The histogram generating unit610, the cumulative distribution function calculator620, and the HE unit630may include components as shown inFIG. 6.

For example, the histogram generating unit610may include a dual-port static random access memory (SRAM), and the HE unit630may include a single-port SRAM.

Here, operational principles of the histogram generating unit610, the cumulative distribution function calculator620, and the HE unit630will be described with reference toFIG. 6.

For example, the cumulative distribution function calculator620may apply a shift operation to a value indicating a number of pixels included in the input image so as to calculate a number of pixels used for generating the histogram or an approximate value of the number of pixels used for generating the histogram. The number of pixels included in the input image may be divided by 2n through the shift operation. Here, n denotes a number of bits corresponding to movement by the shift operation. In this example, the number of pixels obtained as a result of the shift operation and the number of pixels that are actually used for generating the histogram may not be exactly the same. In this example, a predetermined amount of error may be allowed between the two values.

To reduce the error, Hist (k) may be obtained by adding a predetermined value to a number of pixels having a brightness value of k among pixels included in an image. That is, by checking brightness values of the pixels in the image, Hist (k) may be initialized to a predetermined value different from 0, before Hist (k) is calculated.

For example, the histogram generating unit610may use a brightness value of a pixel in the input image as a memory access address of an SRAM. The histogram generating unit610may read a data value using the memory access address, and may increase the read data value by 1 so as to write the increased data value.

For example, when all pixel data in one frame is input to the dual-port SRAM of the histogram generating unit610, the histogram of the input image may be completed. An operation based on Equation 9 may be performed while vertical synchronization of an image is performed by a control block and thus, a result value of the operation may be stored in the single-port SRAM of the HE unit630.

Embodiments of the present invention described with reference toFIGS. 1 through 5will be applicable to the present embodiment and thus, detailed descriptions thereof will be omitted for conciseness.

FIGS. 7 through 9illustrate simulation results according to an embodiment of the present invention.

Referring toFIGS. 7 through 9, first images and first histograms, that is, a first image and a first histogram710, a first image and a first histogram810, and a first image and a first histogram910, are original images and histograms associated with the original images, respectively.

The original images are generally dark and have narrow histogram distributions.

Referring toFIGS. 7 through 9, second images and second histograms, that is, a second image and a second histogram720, a second image and a second histogram820, and a second image and a second histogram920, are AE images and histograms associated with the AE images, respectively.

The original images that are generally dark are forcibly converted by an AE scheme to have an average brightness. In this example, areas having a relatively high brightness in the original images may be saturated in AE images, and a blur effect may occur. Also, the brightness intensively distributed in the original images may not be completely equalized in the AE images. Accordingly, a contrast of the AE images may be low.

Referring toFIGS. 7 through 9, third images and third histograms, that is, a third image and a third histogram730, a third image and a third histogram830, and a third image and a third histogram930, are HE images and histograms associated with the HE images, respectively.

Generally, HE images have a good quality when compared to the AE images. Referring to the third histograms, brightness distributions of darkest areas moves toward a high brightness band, and brightness distributions of brightest areas are intensively distributed in a maximum value of brightness values.

Referring toFIGS. 7 through 9, fourth images and fourth histograms, that is, a fourth image and a fourth histogram740, a fourth image and a fourth histogram840, and a fourth image and a fourth histogram940, are improved HE images and histograms associated with the improved HE images, respectively.

In the improved HE images, dark areas have a relatively lower brightness value when compared to the HE images. Accordingly, the improved HE images may have an improved contrast.