Method and apparatus for adaptive and self-calibrated sensor green channel gain balancing

A method and apparatus for adaptive green channel odd-even mismatch removal to effectuate the disappearance of artifacts caused by the odd-even mismatch in a demosaic processed image. In one example, a calibrated GR channel gain for red rows and a calibrated GB channel gain for blue rows are determined and are a function of valid pixels only in each respective region. After the calibration, in a correction process, the green pixels in red rows of a region are multiplied by the calibrated GR channel gain, and the green pixels in blue rows are multiplied by the calibrated GB channel gain.

BACKGROUND OF THE INVENTION

I. Field of the Invention

The present invention relates generally to image correction methods and, more specifically, to a process for adaptively removing green channel odd-even mismatch.

As the sensor pixel count increases, the area of each pixel photodiode shrinks. The signal readout circuit has to take care of reading and transferring the weaker signal levels. For sensors with a RGB bayer pattern, the Green channel on the odd and even rows normally are read out via a different circuit. More specifically, the metal wire layout of the photo diode, electronic leakage, light incident angle and the signal output circuit, causes the green channel of a bayer pattern sensor to exhibit an unbalanced response. This imbalance contains both global and local variation. Although the circuit layout is identical, the imperfect manufacturing process can cause the read-out and amplifier circuit to be mismatched. Also, the non-uniformity of the color filter array and lens coating and mounting, etc., can also cause the green channel to exhibit odd-even mismatch. Therefore, the overall green channel odd-even mismatch is location dependent and non-uniform. The green channel odd-even mismatch makes the image processing task difficult because the green channel odd-even mismatch translates into cross hatched patterns of artifact as shown inFIG. 1.

InFIG. 1, a flat field image10was created with demosaic operation. This flat field image is supposed to be flat because the lens was covered with a diffusion lens. There should not be any texture on the image after it is processed. However, as is seen inFIG. 1, cross hatched patterns are prevalent across the entire image10. Further investigation reveals that this artifact is caused by the green channel odd-even mismatch.

The demosaic algorithm normally depends greatly on the green channel signal to determine the edge because 50% of the bayer pixels are green. An exemplary bayer pixel arrangement is shown inFIG. 10B. However, if there is a green channel odd-even mismatch, such mismatch is treated as an edge and the demosaic module tries to preserve such edge in either vertical or horizontal directions. The end result is the cross hatched patterns shown inFIG. 1after demosaic processing. This artifact is most obvious when the image is zoomed in around 300%.

One failed solution proposed a global green channel gain balance. If the channel read-out and amplifier circuit were the only factors for green odd-even mismatch, then applying a global green channel gain balance may solve the problem. However, for a Sony™ 3 MP sensor, the use of a global green channel gain balance did not work. Further analysis reveals that the odd-even mismatch is not uniform across the entire image.

Dividing the 3 MP sensor image into regions with 32×32 pixels per region, the flat field image is performed with a region-based channel balance calibration. The required Gr gain and Gb gain to balance the green channel is shown in theFIGS. 2A and 2B. As can be easily seen fromFIGS. 2A and 2B, the green channel balance is very non-uniform across the entire image. As a result, applying global green channel gains can not solve the problem or eliminate the cross hatched pattern of artifact shown inFIG. 1.

Another possible solution employs an adaptive bayer filter. The adaptive bayer filter can be applied only on green pixels to smooth out the odd-even mismatch. The issue is, for the Sony sensor under study, some regions show a green channel odd-even mismatch of 13%. If such a large mismatch is intended to be smoothed out, the true edges in the images may suffer too. As a result, the images will be blurred.

Furthermore, the computation cost of the adaptive bayer filter is relatively high in terms of software/firmware. The computations would also add a considerable amount of delay time to the snap shot image processing.FIG. 3illustrates the resultant image20after applying an adaptive bayer filter to the flat field image ofFIG. 1. The resultant image20has gone through the entire processing pipeline. A moderate amount of smoothing is applied in the adaptive bayer filter. While, in the resultant image20some cross hatched pattern artifact is smoothed out, some still remains.

If a much larger amount of smoothing is applied in the adaptive bayer filter, the cross hatched patterns can be completely removed but at the cost of blurred texture in the images.

If a straightforward smoothing is performed on the raw images on the bayer domain, the edges and textures will suffer. If each pair of green pixels (Gr and Gb) is forced to be equal, the high frequency edges suffer.

SUMMARY OF THE INVENTION

It is an object of the present invention to provide a method for adaptive green channel odd-even mismatch removal to effectuate the disappearance of artifacts created by such mismatch.

It is also an object of the present invention to provide an adaptive green channel odd-even mismatch removal module to effectuate the disappearance of artifacts created by such mismatch.

It is also an object of the present invention to provide program instructions executable by a processor to adaptively remove green channel odd-even mismatch to effectuate the disappearance of artifacts created by such mismatch.

It is a further object of the present invention to provide for adaptive green channel odd-even mismatch removal that is easily implemented in a manner that minimizes computation complexity and does not reduce image processing speed.

It is a further object of the present invention to provide for adaptive green channel odd-even mismatch removal in a manner that adaptively calibrates to correct the odd-even mismatch region-by-region to compensate for image content variances as well as indoor and outdoor image variances.

It is a further object of the present invention to provide for adaptive green channel odd-even mismatch removal in a manner that adaptively compensates for spatial variant green channel odd-even mismatch.

It is a further object of the present invention to provide for adaptive green channel odd-even mismatch removal in a manner which uses an adaptive approach to solve the green channel odd-even mismatch with great preservation of the edges including high frequency edges and edges in either the vertical direction or horizontal direction.

In view of the above objects, the objects of the present invention are carried out by a method for adaptive green channel odd-even mismatch removal comprising the steps of: dividing a raw image from a sensor into a plurality of regions; and, for each region, adaptively removing green channel odd-even mismatch in the raw image to effectuate the disappearance of artifact in a demosaic processed image.

The objects of the present invention are carried out by a method which adaptively removes the green channel odd-even mismatch by calibrating region-by-region of the raw image a green (GR) channel gain for red rows and a green (GB) channel gain for blue rows. After the calibrating step, then applying, region-by-region, the GR channel gain to green pixels in the red rows and the GB channel gain to the green pixels in the blue rows calibrated for each respective region to remove the green channel odd-even mismatch.

The objects of the present invention are carried out by a method to adaptively remove the green channel odd-even mismatch by, for each region in the raw image, by generating a weighted center green pixel value based on a first weighting factor for a center green pixel; summing weighted green pixel values based on a second weighting factor for surrounding green pixels in a first tier layer with respect to the center green pixel of the region to form a first tier layer sum; summing weighted green pixel values based on a third weighting factor for surrounding green pixels in a second tier layer with respect to the center green pixel of the region to form a second tier layer sum; summing the weighted center green pixel value, the first tier layer sum and the second layer sum to form a weighted green pixel sum total. After the weighted green pixel sum total is created, the weighted green pixel sum total is normalized. The normalized weighted green pixel sum total replaces the center green pixel value of the region to remove the green channel odd-even mismatch.

The objects of the present invention are carried out by a method which removes the green channel odd-even mismatch from a raw bayer image.

The objects of the present invention are carried out by a method which removes the green channel odd-even mismatch before demosaic processing by removing edge pixels region-by-region of an image when calibrating the gains.

The objects of the present invention are carried out by a method for adaptive green channel odd-even mismatch removal that when calibrating, filters out bad pixels and edge pixels in each region to form a set of valid pixel pairs.

The objects of the present invention are carried out by a method for adaptive green channel odd-even mismatch removal that when calibrating, counts a number of the valid pixel pairs in the region, computes an average number of the valid green pixels for the red rows, and computes an average number of the valid green pixels for the blue rows.

The objects of the present invention are carried out by a method for adaptive green channel odd-even mismatch removal that when calibrating, filters the GR channel gain and the GB channel gain with a GR channel gain and a GB channel gain of a previous image to reduce noise variance. The applied GR channel gain and the applied GB channel gain are the filtered GR channel gain and the filtered GB channel gain, respectively.

The objects of the present invention are carried out by a method for adaptive green channel odd-even mismatch removal that includes multiplying the green pixels in red rows in each region with the GR channel gain; and multiplying the green pixels in blue rows with the GB channel gain to correct the odd-even mismatch and effectuate the disappearance of the artifact after demosaic processing.

The objects of the present invention are carried out by program code executed by a processing device comprising instructions operable upon execution to calibrate region-by-region in an image a GR channel gain and a GB channel gain. The instruction are also operable to apply, region-by-region, the GR channel gain and the GB channel gain calibrated for each respective region to adaptively remove green channel odd-even mismatch from the image.

The objects of the present invention are carried out by an adaptive green channel odd-even mismatch removal module comprising: means for calibrating region-by-region in an image a GR channel gain and a GB channel gain. The module also includes means for applying, region-by-region, the GR channel gain to green pixels in the red rows and the GB channel gain to the green pixels in the blue rows calibrated for each respective region for removing the green channel odd-even mismatch.

The objects of the present invention are carried out by an adaptive green channel odd-even mismatch removal module the comprises a means for generating a weighted center green pixel value based on a first weighting factor for a center green pixel. The module further comprises a means for summing weighted green pixel values based on a second weighting factor for surrounding green pixels in a first tier layer with respect to the center green pixel of the region to form a first tier layer sum, and a means for summing weighted green pixel values based on a third weighting factor for surrounding green pixels in a second tier layer with respect to the center green pixel of the region to form a second tier layer sum. The module also includes a means for summing the weighted center green pixel value, the first tier layer sum and the second layer sum to form a weighted green pixel sum total, a means for normalizing the weighted green pixel sum total, and a means for replacing a pixel value of the center green pixel with the normalized weighted green pixel sum total to remove the green channel odd-even mismatch.

The objects of the present invention are carried out by program code executed by a processing device comprising instructions operable upon execution to generate a weighted center green pixel value based on a first weighting factor for a center green pixel. The program code is further operable to sum weighted green pixel values based on a second weighting factor for surrounding green pixels in a first tier layer with respect to the center green pixel of the region to form a first tier layer sum, and sum weighted green pixel values based on a third weighting factor for surrounding green pixels in a second tier layer with respect to the center green pixel of the region to form a second tier layer sum. The program code is further operable to sum the weighted center green pixel value, the first tier layer sum and the second layer sum to form a weighted green pixel sum total, normalize the weighted green pixel sum total, and replace a pixel value of the center green pixel with the normalized weighted green pixel sum total to remove the green channel odd-even mismatch.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

While this invention is susceptible of embodiments in many different forms, this specification and the accompanying drawings disclose only some forms as examples of the use of the invention. The invention is not intended to be limited to the embodiments so described, and the scope of the invention will be pointed out in the appended claims.

The preferred embodiments of the green channel odd-even mismatch removal methods according to the present invention are described below with a specific application to a snap shot image. However, it will be appreciated by those of ordinary skill in the art that the present invention is also well adapted for other types of images requiring green channel correction. Referring now to the drawings in detail, wherein like numerals are used to indicate like elements throughout, there is shown inFIGS. 6A-6Band7, a self-calibration process and correction process, generally designated at100and120, according to the present invention.

However, to permit understanding of the invention, the odd-even mismatch refers to the green pixels on red rows with red and green pixels and to the green pixels on blue rows with blue and green pixels that are mismatched. Due to the multiple reasons mentioned previously, the green pixel response is different even though the scene is a smooth flat field image. The mismatch is normally characterized as the ratio of Gr/Gb. Where Gr means the green pixels on the red rows and Gb means the green pixels on the blue rows. Ideally, this ratio should be 1.0.

As shown inFIGS. 2A and 2B, the green channel odd-even mismatch is very non-uniform across the entire image. The mismatch pattern can not be modeled in a simple way. It is clear that the green channel mismatch varies from sensor to sensor. Moreover, different modules of the same sensor model can deviate. By comparing a green channel mismatch of an indoor image inFIG. 4with a green channel mismatch of an outdoor image inFIG. 5captured by the same sensor, it can be readily seen that the green channel mismatch depends on the image contents as well.

In the first exemplary embodiment, the green channel odd-even mismatch removal method includes an adaptive region-by-region green channel gain self-calibration process100described in relation toFIGS. 6A-6Band a correction process120described in relation toFIG. 7. In general, the green channel odd-even mismatch removal method compensates for the green channel mismatch with an adaptive region-by-region green channel gain self-calibration process100, and then applying the green channel gains region-by-region to every final snap shot image output from the sensor module210(FIG. 11) of the snap shot imaging device200in the correction process120.

Referring now toFIGS. 6A-6B,8,9and10A-10B, the adaptive region-by-region green channel gain self-calibration process100, begins with step S102wherein an entire image150output from the sensor module210(FIG. 11), such as a raw image, is divided into X by Y regions (FIG. 10A) with M×M pixels (FIG. 10B) where M is a multiple of 2. In the exemplary embodiment, the image150is divided into 4×3 (X=4, Y=3) regions where each region is divided into 8×8 pixels. For this example, there are 12 regions in the image150. The hatched region labeled R1is divided into M×M pixels as shown inFIG. 10B. The image150is a raw bayer image and has not been subjected to demosaic processing230. Thus,FIG. 10Billustrates a raw bayer representation of the region R1.

For illustrative purposes only, the first row inFIG. 10Bis a blue row with alternating green and blue pixels. The green pixels in the blue row are denoted as GB. The second row immediately following the first row is a red row having alternating green and red pixels. The green pixels on the red row are denoted as GR. In the exemplary embodiment, the first column inFIG. 10Bincludes alternating green and red pixels.

Returning again to the flowchart ofFIG. 6A, step S102follows step S103where the region is set to 1. Step S103is followed by step S104where for each region, the ratio of adjacent green pixels GR and GB on red rows and blue rows is computed. In the exemplary embodiment, two GB, GR pixel pairs of every region is selected. Step S104is followed by step S106where the bad pixels and edge pixels are filtered out.

Bad pixels can be detected based on neighboring pixels of the same color. For example, if the comparison of the current pixel and a neighboring pixel of the same color exceeds some threshold, then the current pixel may be determined to be bad. On the other hand, edge pixel detection may employ a window of A×A size and 2-D convolution. The output of the 2-D convolution is compared with a threshold. If the output is greater than the threshold, the output is an edge. Otherwise, the output is not an edge. There are numerous bad pixel detection and edge pixel detection algorithms. Hence, the above description of bad pixel detection and edge pixel detection are for illustrative purposes only.

Step S106is followed by step S108where the average of GB and GR pixel values, denoted as Gr_avg and Gb_avg, for the non-bad pixels within the region are computed. Step S108is followed by step S10inFIG. 6B. The Gr_avg and Gb_avg are calculated below based on equations Eq. (1) and Eq. (2), respectively, set forth below.

Referring now toFIG. 8, the process for calculating the Gr_avg and Gb_avg pixel values is described. Also an exemplary code for calculating Gr_avg and Gb_avg is provided in the Appendix following this section. The process of step S108begins with step S140where the number of valid pixel pairs (#VP) in the region under consideration is counted or determined. The valid pairs are non-bad pixel pairs remaining after the filtering step S106. Step S140is followed by step S142where i is set to 0. Step S142is followed by step S144, a determination step, which determines whether i is less than the number of valid pairs (#VP) of the region. If the determination is “YES”, a sum, denoted as Gr_sum, is calculated for the GR pixel values for the non-bad green pixels GR in the red rows at step S146. Step S146is followed by step S148where a sum, denoted as Gb_sum is calculated for the GB pixel values for the non-bad green pixels GB in blue rows. Step S148is followed by step S150where i is incremented by 1.

Step S150returns to step S144. Steps S144, S146,148and150are a loop and are repeated until i is less than the number of valid pairs. Thus, at step S146, the sum is incremented by the green pixel value for each corresponding non-bad GR pixel in the region. At step S148the sum is incremented by the green pixel value for each corresponding non-bad GB pixel. Once all of the non-bad GR and GB pixels are separately summed, Step S144is followed by step S152where Gr_avg (the average pixel value for non-bad green pixels in red rows of a region) is calculated based on equation Eq. (1) defined as:
Gr_avg=Gr_sum/Number of Valid Pairs per Region.  Eq. (1)
Step S152is followed by step S154where the Gb_avg (the average pixel value for non-bad green pixels in blue rows of a region) is calculated based on equation Eq. (2) defined as:
Gb_avg=Gb_sum/Number of Valid Pairs per Region.  Eq. (2)

Referring now toFIG. 6BandFIG. 9, at step S110the average gain of the 2 channel green pixel values Gr_gain and Gb_gain is calculated. This means the weaker green channel is applied a digital gain >1 while the stronger green channel is applied a digital gain <1.0. Note the goal of this process is to balance the green pixels from the 2 channels, not among different color channels, therefore applying a gain <1.0 will not cause color shift. Therefore, the channel gain of each (GB, GR) pair could be derived in the following equations Eq. (3) at step S160, Eq. (4) at step S162and Eq. (5) at step S164defined as:
avg=(Gr_avg+Gb_avg)/2;  Eq. (3)
Gr_gain=avg/GR_avg;  Eq. (4)
Gb_gain=avg/GB_avg;  Eq. (5)
where avg is the average value calculated from the average for the valid (non-bad) green pixels GR in the red rows calculated in equation Eq. (1) and the valid (non-bad) green pixels GR in the blue rows calculated in equation Eq. (2) for the non-bad or valid pixel pairs within the region.

Step S110produces the channel gains of Gr_gain and Gb_gain which are passed to step S112. At step S112, the Gr_gain and Gb_gain of the current image150could be lowpass filtered with the previous image's channel gains (Gr_gain and Gb_gain) to reduce the noise variance. The filtered Gr_gain and Gb_gain of the current image is denoted as Gr_gain′ and Gb_gain′.

The box representing step S112has two outputs denoted as Gr_gain′ and Gb_gain′ which would be stored for use in calculations in the correction process. Step S112is followed by step S114where the region is incremented.

The process inFIGS. 6A-6Bis repeated to calculate the Gr_gain and Gb_gain or if filtered Gr_gain′ and Gb_gain′ for each region. Accordingly, Step S114is followed by a determination step S116to determine if there are any more regions. If “YES,” step S116returns to step S104ofFIG. 6Ato self-calibrate the next region. Otherwise, if there are no more regions, the self-calibration process100to calibrate the two channel gain ends.

Referring now toFIG. 7, the correction process120using the Gr_gain′ and Gb_gain′ for each region will now be described. The process120begins with step S122where region is set to 1. Step S122is followed by step S124where the pixel value for each green pixel GB in blue rows is multiplied with Gb_gain′. Step S124is followed by step S126where the pixel value for each green pixel GR in red rows is multiplied with Gr_gain′. Step S126is followed by step S128where the region is incremented. Step S128is followed by step S130where a determination is made whether there are any more regions. If “NO,” the process120ends. Otherwise, if “YES,” step S130returns to step S124where the correction is applied to the next region.

With a region size of 32×32 pixels, the self-calibration and correction processes100and120were performed with a test image, and the demosaic output of the test image no longer shows any cross hatched patterns. Since the region size of 32×32 is small enough, therefore, the corrected image does not show any perceivable region boundary artifact. However, if the region size is too large, such as 256×256, the blockiness artifact may become perceivable.FIG. 12shows the corrected image10′ after demosaic processing. Compared toFIG. 1, the image10′ inFIG. 12is a great improvement.

Referring now toFIG. 11, the snap shot imaging device200includes a lens202and a sensor module210having an image processing unit212and a color filtering unit214. The color filtering unit214is a bayer color filter array which produces a raw bayer image. This raw bayer image is corrected by the green channel odd-even mismatch removal module220. The sensor values for all three primary colors red, green and blue at a single pixel location, are interpolated from adjacent pixels. This process of interpolation is carried out by the demosaic processing unit230. There are a number of demosaicing methods such as pixel replication, bilinear interpolation and median interpolation. The output of the adaptive green channel odd-even mismatch removal module220provides a corrected raw bayer image to the demosaic processing unit230.

The green channel odd-even mismatch removal method performed by the green channel odd-even mismatch removal module220can be implemented using firmware, software, and hardware. For a firmware implementation a digital signal process (DSP)222reads one region at a time, the ARM (Advanced RISC Machine)226supplies the Gr_gain′ and Gb_gain′ to the DSP222. The DSP222performs the multiplication on the Green pixels. The processing is in place, i.e., the input and output pixels share the same buffer228. In other words, the image pixels can be directly replaced with a new value without having to allocate another buffer for processing. The program instructions224when executed are operable to perform the adaptive region-by-region green channel gain self-calibration process100and the correction process120.

While the DSP222and the ARM226are shown as part of the green channel odd-even mismatch removal module220, the snap shot imaging device200may already include the DSP222and the ARM226to carry out the functions in the image processing unit212, the color filtering unit214and the demosaic processing unit230. Thus, the processing devices to execute the program instructions224may already be present.

On the other hand, for the software implementation the program instructions written in a programming language, such as without limitation, C code, runs on the ARM226to divide the raw image, such as a raw bayer image, into regions and performs multiplication on the green pixels using the Gr_gain′ and Gb gain′ for that region. The ARM226is generally pre-existing and can be used to execute the program instructions224. Thus, the ARM226performs both the self-calibration and correction processes100and120. With the software implementation, the processing is also in place so that the image pixels can be directly replaced with a new value without having to allocate another buffer for processing.

For the hardware implementation, the self-calibration and correction processes100and120can be implemented in hardware as long as the size of the look-up table is not an issue.

The green channel odd-even mismatch creates a whole new problem for the video front end (VFE) processing of an image processor. Due to the nature of the non-uniform mismatch distribution, the global channel gain did not solve the problem. The region-by-region calibration and correction processes100and120provide an efficient and fast method to solve the problem related to the non-uniform mismatch distribution.

FIG. 13Ashows a typical RGB bayer pattern with green pixel indexing as will be described in more detail later. An alternative adaptive green channel odd-even mismatch removal method300for adaptively balancing the green channels to remove the odd-even mismatch is described in relation to the flowcharts ofFIGS. 14A-14Eand images ofFIGS. 13A-13B. In this embodiment, the program instructions224(FIG. 11) would be modified to include instructions operable to perform the adaptive green channel odd-even mismatch removal method300described herein.

The adaptive green channel odd-even mismatch removal method300begins with step S302where a raw image, such as a raw bayer image, as best seen inFIG. 13A, is obtained.FIG. 13Ahas green pixels indexed to permit understanding of the method300. Step S302is followed by step S304where regions of N×N pixels are created from the image. In the exemplary embodiment, N is odd and is equal to 5. Step S304is followed by step S306where a center green pixel (CGP) is selected. In this embodiment, the CGP is denoted as G22inFIG. 13A. Step S306is followed by step S308where a first weighting factor is assigned to the CGP G22. In the exemplary embodiment, the first weighting factor is 8. Step S308is followed by step S310where a second weighting factor is assigned for all green pixels in the region N×N at a distance of 1 pixel from CGP. In the exemplary embodiment, there are four (4) nearby green pixels that are in the opposite green channel and the distance to the CGP is 1. These nearby pixels with a distance of 1 pixel will hereby be referred to as “GP1” and together define a first tier layer. The GP1s of the first tier layer include the green pixels indexed as G11, G13, G31and G33. In the exemplary embodiment, the second weighting factor is four (4).

Step S310is followed by step S312where the green pixels with a distance of two (2) pixels from the CGP G22are assigned a third weighting factor. These nearby pixels with a distance of 2 pixels will hereby be referred to as “GP2” and together define a second tier layer. In the exemplary embodiment, there are 8 GP2s in the second tier layer indexed as G00, G02, G04, G20, G24, G40, G42and G44and each gets a weighting factor of one (1). Therefore, the overall weighting factor is 32, so normalization can be easily done by downshift of 5 bits or division by 25wherein the pixel values maybe represented by 8, 10 or 12 bits using a binary representation. Normalization will be described later.

Step S312is followed by step S314where F-max, F_min are set and calculated. F_max is the upper bound threshold of the ratio of max Green mismatch. F_min is the lower bound threshold of the ratio of max Green mismatch. Step314is followed by Step316where an Offset is calculated wherein the offset is the intensity threshold of performing smoothing.

One important factor of the green channel mismatch is due to the cross talk of the surrounding red pixels. That is, the Gr/Gb channel variance depends on the red channel value. Therefore, the Offset is adaptive to the surrounding red pixels to remove the spatial variant green channel odd-even mismatch accurately. In the exemplary embodiment the surrounding red pixels are index and denoted as R10, R12, R14, R30, R32, and R34(FIG. 13B). In the exemplary embodiment, there are six (6) surrounding red pixels. The Offset parameter is defined by equation Eq. (6) as:
Offset=k*mean(R10,R12,R14,R30,R32,R34)  Eq. (6)
where k is a parameter that adjusts the magnitude of the correction for cross talk; and R10, R12, R14, R30, R32, R34denote the pixel value for the corresponding indexed red pixel.

In addition, the Offset is capped by a constant denoted as Offset Cap to avoid an overly large offset threshold. Therefore, step S316is followed by step S317wherein if Offset is greater than the Offset Cap, the Offset is set to Offset Cap or other constant at step S318. Step S318is followed by step S319. However, if the Offset is not greater than the Offset Cap then Step S317is followed by step S319.

At step S319, for the CGP G22, the variables P_max, P_min and G_sum are computed by equations Eq. (7), Eq. (8) and Eq. (9a) defined as:
P_max=max(F_max*G22,G22+offset);  Eq. (7)
P_min=min(F_min*G22,G22−offset);and  Eq. (8)
G_sum=G22<<3  Eq. (9a)
where G22denotes the green pixel value for the center pixel G22; P_max is a maximum value for the green pixel; and P_min is a minimum value of a green pixel. Furthermore, the symbol “<<” denotes an upshift of 3 bits. In other words, G_sum is equal to the pixel value of the green center pixel G22multiplied by its weighing factor 8 (23). Thus, equation Eq. (9a) can also be written as equation Eq. 9(b) defined as:
G_sum=pixel value ofG22*weighting factor forG22.  Eq. 9(b)

As can be readily seen, G-sum in Eq. (9a) or Eq. (9b) generates a weighted center green pixel value based on a first weighting factor for the center green pixel (CGP) G22.

Step S319is followed by step S320where the first green pixel at distance1from the center green pixel (CGP) G22is obtained in the first tier layer. Step S320is followed by step S322where a determination is made whether the pixel value for the green pixel GP1is greater than or equal to P_min and is less than or equal to P_max (see step S322). In other words, step S322determines whether the green pixel under evaluation is within range. If the determination at step S322is “YES,” step S322is followed by step S324where the value G_sum is increased by the green pixel value of the first green pixel GP1(such as indexed pixel G11) upshifted by 2 bits to produce a weighted pixel value for the first tier layer. More specifically, the G_sum is increased by the equations Eq. (10a) or Eq. (10b):
G_sum+=GP1<<2;or  Eq. (10a)
G_sum=G_sum+(GP1*weighting factor ofGP1);  Eq. (10b)(weighting factor for the first tier layer,=4)
wherein GP1is the pixel value for the indexed green pixel in the first tier layer surrounding the center green pixel. In the exemplary embodiment, the GP1s in the first tier layer include G11, G13, G31and G33. As a result, G_sum of Eq. (10a) or Eq. (10b) is increased by the pixel value of each GP1(G11, G13, G31and G33) multiplied by the first tier layer weighing factor (4) or (22) if the green pixel value GP1under evaluation is within range defined by P_min and P_max.

On the other hand, if the determination at step S322is “NO,” then the pixel value for the green pixel GP1is not greater than or equal to P_min and/or not less than or equal to P_max. In other words, the green pixel value GP1under evaluation is out-of-range. Thus, step S322is followed by step S326where the value G_sum is increased by the pixel value of the center green pixel value denoted as G22upshifted by 2. More specifically, the G_sum is increased by equations Eq. (11 a) or Eq. (11 b):
G_sum+=G22<<2; or  Eq. (11a)
G_sum=G_sum+(G22*weighting factor ofGP1)  Eq. (11b)(weighting factor for the first tier layer,=4)
wherein G22denotes the pixel value for indexed center green pixel G22. The same operation and weighting are given to G13, G31and G33of the first tier layer. The G_sum equation at Eq. (11a) or Eq. (11b) is used if the green pixel value GP1under evaluation is out of the range defined by P_min and P_max. As can be readily seen in Eq. (11 a) or Eq. (11 b) the out of range green pixels of the tier layer are replaced with the pixel value of the center green pixel G22.

Steps S324and S326are followed by steps S328to determine if there is another GP1. If so, step S328returns to step S320so that steps S322, S324, S326are reevaluated based on the pixel values for the next GP1.

At the end of the loop defined by steps S320, S322, S324, S326and S328, the G_sum of Eqs. (10a), (10b), (11a) and/or 11(b) has added together the weighted green pixel values of the first tier layer which in general forms a first tier layer sum. In the proposed program code provided, the G_sum equations also add the first tier layer sum to the previously calculated G_sum for the weighted center green pixel value.

When there are no more GP1s in the first tier layer, step S328is followed by step S330where the first green pixel at distance2from the center green pixel (CGP) G22is obtained in the second tier layer. The step S330is followed by step S332where a determination is made whether the pixel value for the green pixel GP2is greater than or equal to P_min and is less than or equal to P_max (see step S332) or in range. If the determination at step S332is “YES,” step S332is followed by step S334where the value G_sum is increased by the green pixel value of the first green pixel GP2(such as indexed pixel GOO). More specifically, the G_sum is increased by the equations Eq. (12a) or Eq. (12b):
G_sum+=GP2; or  Eq. (12a)
G_sum=G_sum+GP2*weighting factor ofGP2  Eq. (12b)(weighting factor for the second tier layer,=1)
wherein GP2is the pixel value for the indexed green pixel in the second tier layer. In the exemplary embodiment, the GP2s in the second tier layer include G00, G02, G04, G20, G24, G42, and G44. As a result, G_sum is increased by the pixel value of the GP2since the weighting factor is 1 if the green pixel value GP2under evaluation is within the range defined by P_min and P_max.

On the other hand, if the determination at step S332is that the pixel value for the green pixel GP2is not greater than or equal to P_min and/or not less than or equal to P_max or out-of-range, then step S332is followed by step S336where the value G_sum is increased by the pixel value of the center green pixel value denoted as G22. More specifically, the G_sum is increased by equations Eq. (13a) or Eq. (13b):
G_sum+=G22; or  Eq. (13a)
G_sum=G_sum+(G22*weight factor ofGP2)  Eq. (13b)(weighting factor for the second tier layer,=1)
wherein G22denotes the pixel value for indexed center green pixel G22. The same operation and weighting are given to the GP2s denoted as G00, G02, G04, G20, G24, G42, and G44in the second tier layer. The G_sum equation at Eq. (13a) or Eq. (13b) is used if the green pixel value GP2under evaluation is out-of-range defined by P_min and P_max. Thus, the green pixel value of the out of range green pixels in the second tier layer are replaced with the pixel value of the center green pixel G22.

As can be readily seen, Eqs. (12a), (12b), (13a) and/or 13(b) sum the weighted pixel values of the second tier layer.

Steps S334and step S336are followed by steps S338to determine if there is another GP2. If so, step S338returns to step S330wherein steps S332, S334, S336are reevaluated based on the pixel values for the next GP2. At the end of the loop defined by steps S330, S332, S334, S336and S338, G_sum has added together the weighted green pixel values of the second tier layer which forms a second tier layer sum. In the proposed program code provided, G_sum has also added together the second tier layer sum, the first tier layer sum and the weighted center green pixel value to form a weighted green pixel sum total.

Referring now toFIG. 14E, after the green pixels in the N×N (N=5) region are processed, the G_sum (weighted green pixel sum total) is normalized at step S340. G_sum (weighted green pixel sum total is normalized by downshifting the G_sum by 5 bits (25=total weighting factor 32 for a binary representation). The total weighting factor equals the sum total of the first weighting factor, the second weighting factor multiplied by the number of green pixels in the first tier layer and the third weighting factor multiplied by the number of green pixels in the second tier layer. In the exemplary embodiment, the first weighting factor is 8. The second weighting factor multiplied by the number of green pixels in the first tier layer equals 16. The third weighting factor multiplied by the number of green pixels in the second tier layer equals 8.

At step S342, the pixel value of the center green pixel G22is replaced with the normalized G_sum calculated in step S340. More specifically, the new pixel value of the center green pixel (G22) is defined by the equation Eq. (14)
New G22=G_sum>>5;  Eq. (14)
where G22denotes the pixel value for the center green pixel G22and the symbol “>>” denotes downshifting; and G_sum in Eq. (14) is the weighted green pixel sum total. Downshifting by 5 bits is the same as dividing by 25or 32.

With the adaptive green channel odd-even mismatch removal method300, for the green pixels that are close to the center green pixel G22, they are used to perform lowpass filtering. If the green pixels are beyond the range of the defined closeness, they are skipped (replaced with the pixel value of the center green pixel). In the exemplary embodiment, the defined closeness is the pixels at the distance of one (1) pixel or at the distance of two (2) pixels. Thus, the normalization factor is a constant. Therefore, division can be replaced with a simple downshift.

Step342is followed by step S344where the method300is repeated for the next region of the image until there are no more regions. At step S344, if the determination is “NO,” the method300ends since there are no more regions. On the other hand, if the determination is “YES,” step S344loop back to step S304inFIG. 14Ato repeat the process for the next region of the whole frame image.

Alternately, step S344can be moved to a location before the normalization step S340which would allow all center pixels to be normalized at the same time.

Note that the values P_max and P_min utilize both ratio and Offset parameters. For small signals, the ratio can not produce a useful range. With the help of the Offset, it can provide a meaningful range of [P_min, P_max]. The side benefit is the reduction of noise as well. For large signals, the ratio dominates and matches the calibrated worst Gr/Gb ratio mismatch which is estimated from the bright grey signal during the calibration process.

With the adaptive green channel odd-even mismatch removal method300, only the worst mismatch of the sensor (such as sensor module210) has to be known as a prior knowledge. At the run time, there is no parameter to change or tune. The worst mismatch is known during the sensor calibration procedure.

TheFIGS. 15A,15B,16A,16B,17A,17B,18A,18B,19A,19B,20A and20B show comparisons of images with and without the adaptive green channel odd-even mismatch removal method300, with demosaic processing. It can be easily seen that with the adaptive green channel odd-even mismatch removal method300(hereinafter referred to as an “adaptive green channel balancing method”), the high frequency components (sharp lines and curves) are preserved very well and the cross hatched pattern is removed. No new artifact is introduced into the image with the adaptive green channel balancing method. The images ofFIGS. 15A,15B,16A,16B,17A,17B,18A,18B,19A,19B,20A and20B are processed with F_max=1.13, F_min=0.87 and Offset limited to 5.FIGS. 15A-15Billustrate a flat field image without and with the adaptive green channel balancing method (zoomed 300% and with demosaic);FIGS. 16A-16Billustrate a resolution chart image (center circles) without and with the adaptive green channel balancing method (zoomed 300% and with demosaic processing);FIGS. 17A-17Billustrate a resolution chart image (vertical lines) without and with the adaptive green channel balancing method (zoomed 300% and with demosaic processing);FIGS. 18A-18Billustrate a resolution chart image (horizontal lines) without and with the adaptive green channel balancing method (zoomed 300% and with demosaic processing);FIGS. 19A-19Billustrate a MacBeth chart image without and with the adaptive channel balancing algorithm (zoomed 300% and with demosaic processing); andFIGS. 20A-20Billustrate a MacBeth chart image without and with the adaptive channel balancing (with demosaic processing).

As can be readily seen fromFIGS. 15B,16B,17B,18B,19B and20B, the adaptive green channel balancing method300preserves the edges in the image very well while the odd-even mismatch artifact (cross hatched pattern) can be removed. The adaptive green channel balancing method300does not need to have run-time tuning or supervision. Only the off-line calibration of the worst odd-even mismatch is required to determine the parameters (F_max and F_min) used in the adaptive green channel balancing method300. The calibration procedure will provide the green channel mismatch gain and the gain variance. Based on that, the parameters (F_max and F_min) can be derived.

The adaptive green channel balancing method300is suitable to be implemented in hardware, firmware or software.

APPENDIX

The Gr_avg and Gb_avg values for the non-bad green pixels within the region are computed according to the following procedure:

The channel gain of each (GB, GR) pair could be derived using the pseudo code:

The following code may be used for the alternative adaptive green channel odd-even mismatch removal method300.

The following exemplary code may be used for the alternative adaptive green channel odd-even mismatch removal method300for summing the parameter G_sum based on for green pixels at a distance of 1 from the green center pixel in the first tier layer.

The following exemplary code may be used for the alternative adaptive green channel odd-even mismatch removal method300for summing the parameter G_sum based on for green pixels at a distance of 2 from the green center pixel in the second tier layer.