Abstract:
A lens vignetting correction method for use in imaging systems such as digital cameras employs a polynomial correction function F=ar 2 +br 4 +c, wherein r is a distance to a center of correction. A calibration image is obtained using the imaging system, then the correction function applied to the calibration image is least-squares fit to determine the variable coefficients a, b and c. Subsequent raw images from this imaging system are corrected by applying the correction function thereto on a pixel-by-pixel basis. A recursive technique is used to obtain correction function values for given pixel locations from modification of values for preceding pixel locations.

Description:
TECHNICAL FIELD  
       [0001]     The present invention relates generally to digital data processing of pictorial information derived from digital still cameras, digital video camcorders and other camera-like image-sensing instruments, and in particular relates to image enhancement by means of transformations for scaling the pixel intensity values of a captured image as a function of pixel position.  
       BACKGROUND ART  
       [0002]     Vignetting is an optical phenomenon that occurs in imaging systems wherein the amount of light reaching off-center positions of a sensor (or film) is less than the amount of light reaching the center. This imaging defect causes the image intensity to decrease toward the edges of an image. The amount of vignetting depends upon the geometry of the lens and varies with focal length and f-stop. The effect is more apparent in lenses of lower f-stop (larger aperture), which are used especially in semi-pro/professional still cameras and video camcorders.  
         [0003]     A lens vignetting correction algorithm may be applied to a digital image (or digitally-scanned photographic image), or to a series of such images, in order to compensate for the positionally unequal intensity effect of vignetting upon the image. This type of image enhancement or correction generally involves some kind of bitmap-to-bitmap transformation in which image intensity or “gray scale” data for the various picture elements (pixels) are appropriately scaled according to pixel position. Such an algorithm can be implemented in an image processor integrated circuit for use in digital still cameras, digital video camcorders, or any other camera-like image-sensing instrument affected by vignetting.  
         [0004]     The particular algorithm and its implementation would preferably achieve optimal anti-vignetting without the need for much extra processing hardware (multipliers, look-up tables, etc.) and with adequate speed and efficiency (especially for real-time image processing). A reasonable set of parameters for the correction formula, and ready adaptability to a variety of lenses, is desirable.  
         [0005]     U.S. Pat. No. 6,747,757 to Enomoto describes correction of several types of image errors, including decreasing brightness at the edge of the image field. It is geared especially to film scanners. The correction equation is log E(r)=a 1 r+a 2 r 2 +a 3 r 3 . It lacks a recursive approach suitable for real-time digital cameras. Accordingly, it relies upon look-up tables to implement the compensation upon the image data.  
         [0006]     U.S. Pat. Nos. 6,577,378 and 6,670,988 to Gallagher et al. describe compensation of respective film and digital images for non-uniform illumination or light falloff at the focal plane, due to factors such as vignetting. A single light falloff correction parameter f 2  is determined, based upon an assumed cos 4  θ light falloff away from the optical axis. For a given camera type, lens, f-stop and focal length, and any flash device, the parameter f 2  can be found by a least-squares fit using a representative sample of pixel elements p ij  of a captured image of a scene. Correction parameters specific to each type of camera lens may be stored in a meta-database. With the parameter f 2  known, compensation values are applied to the individual pixels according to the light falloff compensation function for that parameter. Although only one parameter f 2  needs to be determined, the correction calculation itself contains trigonometric functions (cos, tan −1 ), exponentiation (power of 4), logarithms and division, and is therefore quite complex and in need of extensive processor hardware.  
         [0007]     U.S. Patent Application Publication No. 2004/0155970 of Johannesson et al. describes a vignetting compensation method for digital images. The method obtains a 5×5 matrix of coefficients k ij  (0≦i,j≦4) describing a polynomial surface function g(x,y) representing the relative brightness or vignetting gain at pixel positions (x,y). In particular, 
 
 g ( x,y )=Σ i Σ j   k   ij   x   i y j . 
 
 The matrix coefficients k ij  are estimated by a least-squares fit to the measured intensity data from an image taken of a gray test chart. These coefficients are then stored in a nonvolatile memory. After being normalized, the function g is inverted to create a (look-up) gain table whose pixel-by-pixel multiplication with a raw image will produce a vignette-corrected image. Thus, the method involves both a complexity of computation and the need for look-up tables in order to implement the correction with any reasonable degree of efficiency. 
 
       SUMMARY DISCLOSURE  
       [0008]     The present invention is a lens vignetting correction algorithm implemented in image processing hardware and associated software. The correction applied to the image data is a 4 th -order polynomial function in polar form with only the three even-order coefficients being non-zero. This function defines a radially symmetric curve that expresses the relative intensity correction to be applied to each pixel as a function of the radial distance of that pixel from a center. The center of the correction curve may vary with the optical alignment of each camera, or may be assumed to be very close to the center of the image. The variable coefficients, possibly camera-specific, may be found by fitting of the image data directly to the polynomial correction function, for example, by a least-squares fitting technique. The algorithm applies the polynomial correction function to the several pixels of the image using an iterative calculation that eliminates the need for extra hardware, such as multipliers or look-up tables. 
     
    
     BRIEF DESCRIPTION OF THE DRAWINGS  
       [0009]      FIG. 1  is a flow diagram of an embodiment of the vignetting correction method of the present invention. 
     
    
     DETAILED DESCRIPTION  
       [0010]     With reference to  FIG. 1 , the basic flow of a vignetting correction method in accord with the present invention begins with obtaining a calibration image (step  11 ) using an imaging system, such as digital still camera, digital video camcorder, or other imaging instrument. The calibration image may be that of a uniformly-lit gray surface. However, due to vignetting, the calibration image will exhibit a falloff in intensity away from the image center.  
         [0011]     Next, a least-squares fit of the correction function (step  13 ) is performed. Least-squares (or minimum mean square error) is just one possible technique for fitting the parameters to the imaging characteristics of the optical system. Other fitting techniques could also be used to obtain a set of “best” parameters depending on the optimality criterion used.  
         [0012]     For the present invention, the lens vignetting correction function is chosen to be a radially symmetric 4 th -order polynomial, 
 
 F ( x,y ) =ar   2   +br   4   +c,  
 
 where (x,y) is the variable pixel position, r is the radial distance from a center (x 0 , y 0 ) of correction, 
 
 r   2 =( x−x   0 ) 2 +( y−y   0 ) 2  
 
 and a, b, and c are variable coefficients corresponding to the particular imaging system. 
 
         [0013]     The center (x 0 , y 0 ) of correction is usually assumed to coincide with the center (0,0) of the image, so that r 2 =x 2 +y 2 . In most cases, the alignment between image and lens is such that this assumption will be sufficiently accurate. However, if alignment between image and lens is not guaranteed, we can generalize to any center (x 0 , y 0 ) of correction.  
         [0014]     Applying this correction function F(x,y) to a raw image, we get a corrected image: 
 
 P ( x,y )= P   0 ( x,y )× F ( x,y ), 
 
 where P 0 (x,y) is the raw captured image and P(x,y) is the corrected image. 
 
         [0015]     If a flat gray calibration object is imaged by the system, then the resulting raw calibration image can be used to find the coefficients, a, b and c, for that particular imaging system. That is, we can assume that applying a correction function F(x,y) with the appropriate coefficients, a, b and c, to the raw calibration image, should obtain a corrected image that is again a perfectly flat gray image, P(x i ,y i )=constant.  
         [0016]     Accordingly, we choose a fitting technique to obtain a set of coefficients, a, b, and c. Least-squares or minimum mean square error is one such fitting technique that could be used. Others may choose different criteria to obtain a set of “best” coefficients in relation to image data. Applying a least-squares fitting technique, we obtain a metric E 2 : 
 
 E   2 =(1 /n )Σ i   [P   0 ( x   i   ,y   i )− F ( x   i   ,y   i )] 2 , 
 
 and find coefficients, a, b and c, of F, such that 
 
{∂( E   2 )/∂ a =0, ∂( E   2 )/∂ b =0, ∂( E   2 )/∂ c =0}. 
 
 The threefold criteria may be resolved into a matrix equation P=RA, where:  
         P   =     [             P   0     ⁡     (       x   0     ,     y   0       )                   P   0     ⁡     (       x   1     ,     y   1       )               ⋯               P   0     ⁡     (       x   N     ,     y   N       )             ]       ,     A   =     [         a           b           c         ]       ,   and       
         R   =     [             r   ⁡     (       x   0     ,     y   0       )       2             r   ⁡     (       x   0     ,     y   0       )       4         1               r   ⁡     (       x   1     ,     y   1       )       2             r   ⁡     (       x   1     ,     y   1       )       4         1           ⋯       ⋯       ⋯               r   ⁡     (       x   N     ,     y   N       )       2             r   ⁡     (       x   N     ,     y   N       )       4         1         ]       ,       
 
 where the elements of the 1×N matrix P are the corresponding values P 0  of the calibration pixels (x i ,y i ) over the entire calibration image (or a representative sample thereof), where the elements of the 1×3 matrix A are the correction coefficients a, b and c which are sought, and where the elements of the 3×N matrix R are the radial distances r of the calibration pixels (x i ,y i ) from the center of correction, raised to respective 2 nd , 4 th  and 0 th  powers. The solution of this matrix equation is known to be: 
 
 A=R   +   P,  
 
 where R + =(R T R) −1 R T  is the Moore-Penrose matrix pseudoinverse of R, with R T  being the matrix transpose of R. R +  can be found in advance for a given set of (x i ,y i ) and a given center of correction. Hence, least-squares fitting of the calibration image to find the correction coefficients (a,b,c) resolves into a relatively simple matrix multiplication. 
 
         [0017]     If the alignment between imager and lens is not guaranteed, such that one cannot assume that the center of vignetting coincides with the center of the image, we can search for coefficients (a,b,c) for different “center” coordinates (x 0h ,y 0k ), where h and k are variable. The center coordinates that lead to a minimum mean square error (smallest E 2  value) is then considered to be the center of vignetting, and the coefficients (a,b,c) are those found for that particular center.  
         [0018]     After fitting the correction function F, we can use that function to correct subsequent raw images obtained by that imaging system. Accordingly, whenever a raw image is obtained (step  15  in  FIG. 1 ), we apply the correction function F to the raw image on a pixel-by-pixel basis (step  17 ). That is, corrected image P(x,y)=P 0 (x,y)×F(x,y), for each pixel location (x,y). This application of correction function F can be repeated (return step  19 ) each time a new raw image is obtained.  
         [0019]     In order to eliminate the need for extra processing hardware, such as multipliers, and to operate efficiently in real-time, the algorithm that implements the image correction employs a recursive technique for updating r 2 . This technique may proceed as follows: 
    a, b, c     x0h, y0k     R 2 [=(x0h) 2 +(y0k) 2 ]    stepX, stepY     dy:=y0k     r 2 :=R 2       for y=1 to height (step=stepY) 
        dx:=x0h     rx 2 =r 2       for x=1 to width (step=stepX) 
            rx 4 :=rx 2 *rx 2       F:=rx 2 *a+rx 4 *b+c     P(x, y):=P(x, y)*F     rx 2 :=rx 2 +stepX*stepX−2*stepX*dx     dx:=dx−stepX    
            end     r 2 :=r 2 +stepY*stepY−2*stepY*dy     dy:=dy−stepY    
        end 
 
 Step Y is usually 1, 2, or 4, depending on the imager interface. Step X is always 2 for Bayer pattern imager output. This means that all multiplications involving the updates of rx 2  and r 2  can be implemented by shift operations, which are inexpensive in hardware (or in software.) As seen here, the correction coefficients a, b and c are provided, along with the correction center coordinates (x0h, y0k) for those coefficients. The radial distance squared, R 2 , from this center is also provided for the top-left pixel position of the image. The correction of pixel intensity, P(x,y):=P(x,y)*F, proceeds row by row (incrementing row coordinate y by stepY) from top to bottom, and from left to right within rows (incrementing column coordinate x by stepX) until the entire image has been corrected. Extensive use of exponentiation for each pixel is not required when this recursive technique is used. 
   
 
         [0039]     The vignetting correction of color images may use separate correction coefficients for each color. Consider the case of a Bayer pattern where each pixel is defined by one of three primary colors (e.g., red, green, and blue), and that uses a two-field interlaced format. Odd-numbered rows or lines 1, 3, 5, . . . may form a field 0 made up of alternating green and red pixels, while even-numbered rows or lines 2, 4, 6, . . . may form a field 1 made up of alternating blue and green pixels. This format may be indicated to the processor, for example, by means of one or more control register bits. Pixels in this particular format are effectively grouped into 2×2-color cells, made up of the primary colors is some defined pattern. For example: 
        line 1, pixel 0 (L1P0): green pixel;     line 1, pixel 1 (L1P1): red pixel;     line 2, pixel 0 (L2P0): blue pixel; and     line 2, pixel 1 (L2P1): green pixel. 
 
 The green pixels may use one set of correction coefficients (a 0 , b 0 , c 0 ), the red pixels may use another set of correction coefficients (a 1 , b 1 , c 1 ), and the blue pixels may use yet a third set of correction coefficients (a 2 , b 2 , c 2 ) . The coefficients may need to be scaled to fit a specified format, such as 1 sign bit, 3 integer bits, and 8 fractional bits. A register may designate the scale factor to be used. In this example, scaling could set registers to A_L1P0=A_L2P1=a 0 -2 scale , B_L1P0=B_L2P1=b 0 -2 2·scale , C_L1P0=C_L2P1=c 0 ·2, A_L1P1=a 1 ·2 scale , B_L1P1=b 1 ·2 2·scale , C_L1P1=c 1 ·2, A_L2P0=a 2 ·2 scale , B_L2P0=b 2 ·2 2·scale , and C_L2P0=c 2 ·2, where the registers store scaled versions of the coefficients used for the various color pixels in the cells. When applying the recursive correction algorithm to the pixels of a particular color, the processor will access these stored scaled coefficients from the appropriate registers. The four pixels in each cell might also reference a slightly different center of correction. Similar adaptations can be made for other image interlace types.