Abstract:
A method and apparatus for correcting for vignetting include associating each pixel in the two-dimensional array with a pair of polar coordinates referenced to a preselected origin pixel and partitioning the two-dimensional array of image pixels into a plurality of sectors. For each sector, the method includes computing an average R value, an average G value and an average B value; converting the average R value, the average G value and the average B value for each sector to logarithm space; comparing color gradients along a radial sector line to a gradient threshold; selecting gradients that do not exceed the threshold; using the selected gradients, estimating parameters of a model of a lens which produced the image; and, using the parameters, updating the model of the lens and correcting the image.

Description:
BACKGROUND 
       [0001]    1. Technical Field 
         [0002]    This disclosure relates to optical imaging systems and, more particularly, to a method and apparatus for correcting for color and/or intensity vignetting in an image and in the imaging system which generated the image. 
         [0003]    2. Discussion of the Related Art 
         [0004]    Digital imaging systems typically include an optical system for gathering and projecting or imaging light onto a receiver, detector or sensor. The optical system typically includes one or more imaging lenses for forming the image on the sensor. The sensor is commonly a two-dimensional array of optical detectors or sensors, such as complementary metal-oxide-semiconductor (CMOS) image sensors. 
         [0005]    Vignetting is a common phenomenon which occurs in optical systems. It refers to the color and intensity variation with respect to the lens&#39; chief ray angle. As the chief ray angle increases, the quantum efficiency of the sensors or detectors decreases, resulting in a decrease in the color and intensity as the distance from the center of the image increases. 
         [0006]    Many approaches to correcting for the vignetting artifact have been introduced. Most correction methods require specific scenes for calibration, which usually must be uniformly lit, and the calibrated model is applied to the real image or video. In practice, the vignetting model actually changes with light spectrum and focal length, which is difficult to calibrate across different light sources. This phenomenon is more pronounced in CMOS image sensors with small pixel size. Other approaches work on a single image individually and require image segmentation and a large buffer, which is too complex to embed in an image sensor. 
       SUMMARY 
       [0007]    According to a first aspect, a method for correcting for vignetting in an image is provided. The image includes a two-dimensional array of image pixels, each image pixel comprising an R value, a G value and a B value representative of red, green and blue intensities, respectively. According to the method, each pixel in the two-dimensional array is associated with a pair of polar coordinates referenced to a preselected origin pixel. The two-dimensional array of image pixels is partitioned into a plurality of sectors, each sector comprising a plurality of pixels of the two-dimensional array of image pixels, the plurality of sectors comprising a plurality of groups of sectors, each group of sectors extending radially along an associated sector line through the preselected origin pixel. For each group of sectors: (i) for each sector in the group of sectors, an average R value, an average G value and an average B value are computed, the average R value being an average of the R values of the pixels in the sector, the average G value being an average of the G values of the pixels in the sector, the average G value being an average of the G values of the pixels in the sector; (ii) the average R value, the average G value and the average B value for each sector are converted to logarithm space to generate an R vector, a G vector and a B vector; (iii) a median filter is applied to the R vector, the G vector and the B vector to identify color gradients along the sector line; (iv) the color gradients are compared to a gradient threshold; (v) gradients that do not exceed the threshold are selected; (vi) parameters of a model of a lens which produced the image are estimated using the selected gradients; and (vi) the model of the lens is updated and the image is corrected using the parameters. 
         [0008]    According to another aspect, an image sensor with correction for vignetting comprises: a two-dimensional array of image pixels, each image pixel comprising an R value, a G value and a B value representative of red, green and blue intensities, respectively; and a processing circuit coupled to the two-dimensional array of pixels. The processing circuit is adapted to: (i) associate each pixel in the two-dimensional array with a pair of polar coordinates referenced to a preselected origin pixel, (ii) partition the two-dimensional array of image pixels into a plurality of sectors, each sector comprising a plurality of pixels of the two-dimensional array of image pixels, the plurality of sectors comprising a plurality of groups of sectors, each group of sectors extending radially along an associated sector line through the preselected origin pixel, and (iii) for each group of sectors, for each sector in the group of sectors: compute an average R value, an average G value and an average B value, the average R value being an average of the R values of the pixels in the sector, the average G value being an average of the G values of the pixels in the sector, the average G value being an average of the G values of the pixels in the sector: convert the average R value, the average G value and the average B value for each sector to logarithm space to generate an R vector, a G vector and a B vector; apply a median filter to the R vector, the G vector and the B vector to identify color gradients along the sector line; compare the color gradients to a gradient threshold; select gradients that do not exceed the threshold; using the selected gradients, estimate parameters of a model of a lens which produced the image; and using the parameters, update the model of the lens and correct the image. 
     
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         [0009]    The foregoing and other features and advantages will be apparent from the more particular description of preferred embodiments, as illustrated in the accompanying drawings in which like reference characters refer to the same parts throughout the different views. The drawings are not necessarily to scale. In the drawings, the sizes of features may be exaggerated for clarity. 
           [0010]      FIGS. 1A and 1B  are schematic diagrams of basic components of an imaging system. 
           [0011]      FIG. 2  is a schematic diagram of a typical pixel in a sensor array. 
           [0012]      FIG. 3  contains a schematic plan view diagram of a two-dimensional sensor array, according to exemplary embodiments. 
           [0013]      FIG. 4  is a schematic logical flow diagram which illustrates the process of correcting for lens shading or vignetting in image data, according to some exemplary embodiments. 
           [0014]      FIG. 5  contains a schematic functional block diagram of an image sensor according to exemplary embodiments. 
       
    
    
     DETAILED DESCRIPTION 
       [0015]      FIGS. 1A and 1B  are schematic diagrams of basic components of an imaging system. Specifically,  FIG. 1A  is a top view of the illustrated imaging system components, and  FIG. 1B  is a side view of the components. The components include a lens  10  and a sensor array  12 . The center  14  of lens  10  is in alignment with a center pixel  16  of sensor array  12 . Pixels in sensor array  12  can be located according to a Cartesian coordinate system defined by mutually orthogonal x and y axes, such that each pixel  23  in sensor array  12  is associated with a Cartesian coordinate pair indicted by (x, y). Specifically, center pixel  16  of sensor array  12  can be identified by the coordinate pair (x center , y center ). 
         [0016]    As noted above, in general, pixel  23  is associated with a Cartesian coordinate pair referred to as (x, y). Referring to  FIG. 1B , light ray traces  25  from a point  24  in an object or scene being imaged pass through lens  10  and are joined together at pixel  23  of sensor array  12 . Generally, lens shading or vignetting at a pixel having coordinates (x, y) is related to the distance in sensor array  12  between center pixel  16  having coordinates (x center , y center ) and pixel  23  having coordinates (x, y). 
         [0017]      FIG. 2  is a schematic diagram of a typical pixel  23  in sensor array  12 . Referring to  FIG. 2 , each pixel  23  in sensor array  12  is actually made up of, in one particular exemplary embodiment, four optical detectors  26 A,  26 B,  26 C,  26 D disposed in a 2×2 cell. Each 2×2 cell in sensor array  12  is treated as a single pixel  23 . Within the 2×2 cell, each detector  26  is associated with a specific filter color. In this specific exemplary illustration, detector  26 A is filtered by a blue filter B, detector  26 B is filtered by a green filter Gr, detector  26 C is filtered by another green filter Gb, and detector  26 D is filtered by a red filter R. Each detector  26  generates an intensity value for pixel  23 . Specifically, detector  26 A generates a blue intensity value B for pixel  23 , detector  26 B generates a first green intensity value Gr for pixel  23 , detector  26 C generates a second green intensity value Gb for pixel  23 , and detector  26 D generates a red intensity value R for pixel  23 . In this exemplary illustration, the green intensity value G for pixel  23  is the average of the first and second green intensity values Gr and Gb, that is, G=(Gr+Gb)/2. Therefore, each pixel  23  in sensor array  12  includes red, green and blue intensity values R, G, B. 
         [0018]      FIG. 3  contains a schematic plan view diagram of two-dimensional sensor array  12 , according to exemplary embodiments. Sensor array  12  includes a two-dimensional array of pixels  23  which include data that are representative of an image of an object or scene. Each pixel  23  includes data for at least three color intensities. For example, each pixel includes intensity data for red, green and blue (R, G, B) intensities, as described above in detail. It should be noted that throughout this disclosure, the colors red, green, and blue (R, G, B) are used only for exemplary illustration. This disclosure is also applicable to other combinations of colors, such as cyan, yellow, magenta and red, green, blue, white color combinations, among others. 
         [0019]    In general, the data associated with pixels  23  of sensor array  12  represent an image with a lens shading or vignetting artifact. According to exemplary embodiments, the image data of pixels  23  are corrected and/or updated to substantially reduce or eliminate the lens shading or vignetting artifact. In addition, a lens model applied to the data to identify the lens shading or vignetting artifact is also corrected/updated. 
         [0020]    Referring to  FIG. 3 , as described above in detail, each pixel  23  is associated with a Cartesian coordinate pair (x, y), defined by the mutually orthogonal x and y axes. Sensor array  12  also includes a center pixel  16  whose Cartesian coordinate pair is referred to herein as (x center , y center ). According to exemplary embodiments, in the process of correcting for the lens shading or vignetting artifact, the coordinates of each pixel  23  are converted to polar coordinates such that the position of each pixel  23  is defined by a polar coordinate pair (r, θ), where r is the distance of pixel  23  from center pixel  16 , which is defined as an origin pixel for the polar coordinate system, and θ is the angle which a line passing through origin pixel or center pixel  16  and pixel  23  forms with the horizontal or x axis. 
         [0021]    According to exemplary embodiments, a plurality of radial sector lines  17 , which extend radially from origin pixel or center pixel  16  and form an angle θ 1  with the horizontal, i.e., x, axis, is defined. A plurality of sectors  19 ,  21 ,  25  are disposed along each radial sector line  17 . Sectors  19 ,  21 ,  25  are groups of pixels  23  bounded as illustrated by radial boundary lines  27  and  29  and radially by a predefined radial sector thickness. Each sector  19 ,  21 ,  25  is defined by a certain predetermined distance R i  from center pixel  16 . For example, sector  19  is defined as being a distance R 1  from center pixel  16 , and adjacent sector  21  is defined as being a distance R 2  from center pixel  16 . 
         [0022]    The size of each sector  25 , and, therefore, the number of pixels  23  in each sector  25 , is determined by the radial thickness of each sector  25 , which is determined by the distance R i  of sectors  25  from center pixel  16 , as well as the angle θ 2  separating radial boundary lines  27  and  29 . Either or both of these parameters of the sectors, i.e., the radial thickness and the angle θ 2 , can be selected based on the desired precision of the image correction being performed, according to some embodiments. 
         [0023]    Referring to  FIG. 3 , it will be understood that while only one radial sector line  17  is illustrated for the purpose of clarity of the illustration, many radial sector lines  17  are defined according to exemplary embodiments, such that sectors  25  are defined over the entire sensor array  12 . Also, it will be understood that while only two adjacent sectors  19  and  21  and one general sector  25  are illustrated along radial sector line  17 , many adjacent sectors are actually defined, such that, in some exemplary embodiments, sectors  25  are defined extending radially and radially adjacent to each other from origin or center pixel  16  to the edge of sensor array  12 . 
         [0024]      FIG. 4  is a schematic logical flow diagram which illustrates the process of correcting for lens shading or vignetting in image data and correcting and updating a lens model, according to some exemplary embodiments. Referring to  FIG. 4 , as described above, the coordinates of each pixel  23  are converted to polar coordinates in step  100 , and the polar coordinates for each pixel  23  are stored. Next, in step  102 , sensor array  12  is partitioned into radial sectors  25 . As described in detail above, each sector  25  is defined along a radial sector line  17 . The size of each sector  25  is determined according to the desired precision. That is, high precision would require a relatively higher quantity of smaller sectors  25  than low precision would require. 
         [0025]    In step  104 , the lens model is applied. The lens model may be the model that was updated by the processing of a previous group of radial sectors  25  along a radial sector line  17 . That is, the lens shading is corrected in the image using the lens model estimated and updated during the processing of the disclosure in connection with a previous group of sectors  25  along radial sector line  17 , and then the lens model is updated using the residual shading developed during the processing of the present group of sectors  25 . 
         [0026]    In step  106 , the average R, G and B values for each sector  25  are computed and stored. The average R value for a sector is the average of the R values of all of the pixels in the sector. Similarly, the average G value for a sector is the average of the G values of all of the pixels in the sector, and the average B value for a sector is the average of the B values of all of the pixels in the sector. 
         [0027]    In step  108 , the average R, G and B values are converted to log space. Next, in step  110 , the color components of each sector  25  are computed in log space. That is, for example, the red color content is computed in color space by dividing the average red value R for a sector by the average green value G for the sector by subtracting the respective converted logarithmic values. That is, R/G=log (R)−log (G). Similarly, the blue color content is computed in color space by dividing the average blue value B for a sector by the average green value G for the sector by subtracting the respective converted logarithmic values. That is, B/G=log (B)−log (G). In some exemplary embodiments, sectors having values that are too low are concluded to be dark and are excluded. Similarly, sectors with color content values that are too high, i.e., saturated, are also excluded. The remaining color content values are used to generate radial color vectors, for example, a red radial color vector and a blue radial color vector, for each radial sector line  17 . 
         [0028]    Next, in step  112 , a median filter is applied to each radial color vector along the radial direction to remove noise from the data for each sector  25 . Next, in step  114 , color gradients between radially adjacent sectors  25  are obtained for each radial color vector by subtracting corresponding color content values of adjacent sectors  25 . According to some embodiments, gradients caused by lens shading or vignetting are distinguished from actual image contents, since color edges of objects will result in inaccurate estimation and correction. To accomplish this, since gradients caused by edges of actual objects are substantially larger than gradients caused by lens shading or vignetting, outlier gradients, i.e., gradients above a certain threshold or outside a certain predetermined range, are eliminated from the processing. This is described in detail below. The result of computing all of the gradients and excluding the outlier gradients is generation of a radial gradient map for the image. 
         [0029]    Next, in step  116 , according to exemplary embodiments, lens shading or vignetting model parameters are computed, i.e., estimated, in order to update the lens shading or vignetting model and to correct the image data to eliminate or reduce the vignetting artifact. In exemplary embodiments, this is accomplished by applying an optimization technique. Most commonly, color vignetting does not result in dramatic changes or substantial gradients. Therefore, according to some exemplary embodiments, a second-order, i.e., quadratic, polynomial is used to model the lens vignetting. According to other exemplary embodiments, the model can be expanded to higher-order polynomials. The quadratic polynomial can be of the form, 
         [0000]        Y=C−a   1   r−a   2   r   2    (1);
 
         [0000]    where C is a reference constant, e.g., the color intensity of center pixel  16 , r is the radius or distance from center pixel  16 , Y is the color content value, i.e., either blue (B) or red (R) in exemplary embodiments, and a 1  and a 2  are model parameters which are estimated according to exemplary embodiments. 
         [0030]    A first derivative with respect to radius or distance r is applied to (1) to obtain, 
         [0000]      − Y′=a   1 +2 a   2   r    (2);
 
         [0000]    which describes color content gradient with respect to r. These model parameters are estimated using the radial gradient map by applying, in some exemplary embodiments, linear regression. Referring to step  118 , once the model parameters a 1  and a 2  are obtained using linear regression, the residual, i.e., the shading of the image in the processing of the present group of radial sectors, is corrected and the lens vignetting model is updated by adding back (a 1 +a 2 r) to the model polynomial (1) above. The addition operation in log space is multiplication in linear space, so the ratio can be calculated by converting back from log space to linear space. 
         [0031]    For example, according to some exemplary embodiments, for a pixel with a radius (radial distance) r from center pixel  16 , with the model parameters a 1 and a 1 , the color Y=C−a 1 r−a 2 r 2  can be estimated, where C is the color at center pixel  16 . According to exemplary embodiments, to correct Y to equal C, then Y′=Y+a 1 r+a 2 r 2 =C. In log space, a 1 r+a 2 r 2  is added to correct the shading at the current pixel. In linear space, a gain, R′/G′ is multiplied, i.e., R′/G′=Gain×R/G. Log(R′/G′)=log(R/G)+log(Gain), where log(Gain)=a 1 r+a 2 r 2 . Then, Gain can be computed by Gain=exp(a 1 r+a 2 r 2 ). For example, to correct the red color shading, R′=Gain×R. To correct the blue shading, B′=Gain×B, 
         [0032]    As described above, according to exemplary embodiments, radial gradients are distinguished based on whether they are due to lens vignetting or actual object edges or boundaries. Because the color edges of objects can result in false estimation, according to exemplary embodiments, the boundaries of the model parameters can be defined as a 1min , a 1max , a 2min , a 2max , because color variance caused by vignetting is much smaller than that caused by actual object edges. With the model parameters a 1  and a 2  bounded, then the gradients can be bounded according to the following: 
         [0000]        G   min   =a   1min   +a   2min   r    (3); and
 
         [0000]        G   max   =a   1max   +a   2max   r    (4).
 
         [0000]    Then, the outliers, i.e., the gradients that are not due to vignetting, can be identified by applying the gradients G to the range G min &lt;G&lt;G min , and excluding any gradients that lie outside of the range. 
         [0033]    According to some exemplary embodiments, computation of gradients between multiple radial sectors  25  can be used to smooth out noise. For example, computation of gradients between two radial sectors separated by a radial distance can have the effect of smoothing noise. For example, given a vector [x0, x1 x2, x3], the normal gradient is x1−x0, x3−x2. Using two radial sectors  25  separated by a radial distance, the gradient can be computed as (x2−x1)/2, (x3−x1)/2. 
         [0034]    According to some exemplary embodiments, a time-division approach can be used to reduce the size of the buffer used to store data used in the computations according to the disclosure. That is, for example, two frames can be used to estimate the lens model. For example, the polar image can be portioned as [64,24], i.e., the resolution in radius r for the sectors  25  is 64, and the total 360-degree angle is partitioned into 24 angular regions, i.e., θ 2 =15 degrees for each sector line  17  (see  FIG. 3 ). Under this time-division approach, only the vectors for odd angles are processed, and, in the second frame, only even angles are processed. This reduces by half the required data storage load. 
         [0035]      FIG. 5  contains a schematic functional block diagram of an image sensor  200  according to exemplary embodiments. Referring to  FIG. 5 , image sensor  200  includes a pixel array or sensor array  205 , described above as sensor array  12 . Pixel array  205  interfaces via bit lines with readout circuitry  210 . Readout circuitry  210  provides data from pixel array  205  to function logic  215 . 
         [0036]    Pixel array  205  interfaces with and is controlled in accordance with the foregoing detailed description by processing circuitry  220 . To that end, processing circuitry  220  includes all of the circuitry required to carry out the operation of an imaging system according to the present disclosure. For example, processing circuitry  220  can include a processor  222 , one or more memory or storage circuits  224  and one or more input/output interface circuits  226 . 
         [0037]    It is noted that the forgoing disclosure refers primarily to color vignetting and correction for color vignetting. It will be understood from the foregoing that the disclosure is also applicable to intensity vignetting. For example, as described above in detail, for color vignetting correction, R/G and B/G are corrected. For intensity vignetting correction, processing is performed on G or Y=(R+2G+B)/4. The process is the same as that described above in detail in connection with color vignetting correction. 
       Combinations of Features 
       [0038]    In any of the embodiments described in detail and/or claimed herein, estimating the parameters can include applying linear regression. 
         [0039]    In any of the embodiments described in detail and/or claimed herein, the model is a polynomial model. 
         [0040]    In any of the embodiments described in detail and/or claimed herein, the model is a quadratic polynomial model. 
         [0041]    In any of the embodiments described in detail and/or claimed herein, the parameters are obtained from a first derivative of the quadratic polynomial. 
         [0042]    In any of the embodiments described in detail and/or claimed herein, the origin pixel is at an optical center of the image. 
         [0043]    In any of the embodiments described in detail and/or claimed herein, the processing circuit comprises a memory. 
         [0044]    In any of the embodiments described in detail and/or claimed herein, the processing circuit comprises a processor. 
         [0045]    While the present disclosure has shown and described exemplary embodiments, it will be understood by those of ordinary skill in the art that various changes in form and details may be made therein without departing from the spirit and scope of the present disclosure, as defined by the following claims.