Abstract:
In large arrays of image sensing devices, like CCDs, a small number of defective image elements (pixels) must be tolerated and the resulting image degradation should be eliminated through bad pixel correction. The disclosed invention provides a mechanism to effectively detect defective pixels “on the fly” in a Bayer RGB type color image sensor, optimized for low cost applications. It calculates a variable threshold based on signal changes on nearby pixels of the same color within a single row and checks if the signal change of the pixel under test exceeds said variable threshold. It further performs a plausibility check using nearby pixels of an other color in the same row.

Description:
BACKGROUND OF THE INVENTION  
       [0001]     (1) Field of the Invention  
         [0002]     The present invention relates to a defective pixel correction circuit for correcting a defective pixel in a solid imaging device and, more particularly, to charge coupled device (CCD) and CMOS imagers as used in digital cameras, employing a pixel correction circuit with reduced memory requirements.  
         [0003]     (2) Description of Prior Art  
         [0004]     In large arrays of image sensing devices, arranged in a matrix of rows and columns, like CCDs a small number of defective image elements (pixels), caused by a variety of manufacturing deficiencies, must be tolerated, especially for low cost applications. The reason may be, for example, manufacturing process defects, which may show up as dead spots (totally dark), hot spots (totally white) and weak spots. Such a defective pixel is generated not only as an initial defects, but it is also generated because of aging, as the solid-state image pick-up device is used for a long period time. Similar such defective pixels may change with operating conditions, like temperature of the sensor or supply voltage. The position of the defective pixel itself is fixed. Therefore, generally, the image signal from the defective pixel is often corrected by storing the position thereof in advance.  
         [0005]     More specifically, the defective pixel is detected at the time of delivery of the image-sensing device and thereafter periodically, the position information of the defective element is generated and stored periodically in a storing device. The image signal corresponding to the defective pixel is replaced by interpolation, using image signals from the pixels around the defective pixel. Such an interpolation is performed after the image-sensing device takes in the image signal and before processing the signal, by reading the position information of the defective pixel from the storing device and by interpolating the defective pixel by using the image pick-up signals one-dimensionally or two-dimensionally there around.  
         [0006]      FIG. 1  shows a model for a photo-sensing array.  FIG. 2  shows the principal concept of Bayer RGB layout, which has twice as many green pixels as there are of the two other colors red and blue. Interpreting neighboring pixels in two dimensions is visualized in  FIG. 3 .  
         [0007]     U.S. Pat. No. 5,805,216 (to Tabei, et al.) describes a circuit to correct a defective pixel in a solid imaging device (CCD). It calculates magnitudes of boundaries from signals of eight peripheral pixels and produces an interpolation signal to correct the defective pixel.  
         [0008]     U.S. Pat. No. 5,047,863 (to Pape, et al.) shows a defect correction circuit including inoperative pixel detection. As disclosed, when a threshold indicates an inoperative pixel, such pixel data is replaced by image pixel data from an operative pixel immediately prior to the defective pixel.  
         [0009]     U.S. Pat. No. 4,535,359 (to Fearnside.) discloses an apparatus for defect correction in solid-state imaging. The disclosure uses the fact, that edges of dead and hot pixels correspond with very high spatial frequency components, to detect such dead or hot pixels.  
       SUMMARY OF THE INVENTION  
       [0010]     In large arrays of image sensing devices, like CCDs, a small number of defective image elements (pixels), caused by a variety of manufacturing deficiencies, must be tolerated, especially for low cost applications. The resulting image degradation should however be eliminated through a “bad pixel correction” mechanism. Permanently storing the manufactures “bad pixel map” in an additional memory device is expensive and continuously maintaining such map is complex. Evaluating a larger two-dimensional array to detect defective pixels “on the fly” is complex and still requires storing a multiple of rows, being expensive as well.  
         [0011]     The herewith disclosed invention provides a mechanism to effectively detect defective pixels “on the fly” in a Bayer RGB type color image sensor. The presented invention is primarily intended to and is optimized for low cost applications. It calculates a variable threshold based on signal changes on nearby pixels of the same color within the same scanning row and checks if the signal change of the pixel under test exceeds said variable threshold. If yes, it further performs a plausibility check using nearby pixels of the other color in the same row.  
         [0012]     In actual photographs, it is unlikely to find excessive peeks of just 1 pixel wide within an otherwise smooth image environment; a bad pixel could be assumed and should be corrected. However, an image area where the slope of signal change is significant, a steep change is more likely associated with a real pixel. In this case it should not be taken as a bad pixel and not be corrected. Such steep change could surely be assumed as being caused by a real image change rather than by a bad pixel, if neighboring pixels of another color show a similar steep change. If that decision would not be correct and that pixel would be in fact a bad pixel, not treating is as bad pixel and therefore not correcting it, would hardly be visible to the human eye as it is hidden under the other steep signal changes. The herewith disclosed invention implements these clauses in decision-making circuits and methods.  
     
    
     BRIEF DESCRIPTION OF THE DRAWINGS  
       [0013]     In the accompanying drawings, forming a material part of this description, there is shown:  
         [0014]      FIG. 1  (Prior Art) demonstrates a large image sensor array  
         [0015]      FIG. 2  (Prior Art) shows the Bayer RGB layout  
         [0016]      FIG. 3  (Prior Art) visualizes a concept of interpolating a field of 5×5 pixels to find a bad pixel  
         [0017]      FIG. 4  shows one example, according to this invention, with 9 pixels, 4 before and 4 behind the pixel P=UI=P 5  under investigation.  
         [0018]      FIG. 5  visualizes the fact, that a sharp line in the image, normally affects more than just one color.  
         [0019]      FIG. 6  gives a coarse overview on the bad pixel correction method  
         [0020]      FIG. 7  gives a more detailed functional view on the method for bad pixel correction  
         [0021]      FIG. 8  shows the method close to a possible implementation.  
     
    
     DESCRIPTION OF THE PREFERRED EMBODIMENTS  
       [0022]     The objectives of this invention are to perform an effective bad pixel correction in a low cost application.  
         [0023]     In large arrays of image sensing devices, arranged in a matrix of rows and columns, like CCDs, a small number of defective image elements (pixels), caused by a variety of manufacturing deficiencies, must be tolerated, especially for low cost applications. The reason may be, for example, manufacturing process defects, which may show up as dead spots (totally dark), hot spots (totally white) and weak spots. Such a defective pixel is generated not only as an initial defect, but it is also generated because of aging, as the solid-state image pick-up device is used for a long period time. Similar, such defective pixels may change with operating conditions, like temperature of the sensor or its supply voltage. The position of the defective pixel itself is fixed. Therefore, the image signal from the defective pixel is often corrected by storing the position thereof in advance. However, permanently storing the manufacturer&#39;s “bad pixel map” in an additional memory device is expensive and continuously maintaining such map is complex and it does not cure the problems with aging and with those problems related to varying operating conditions. Additional “on the fly” correction mechanisms are normally implemented.  
         [0024]     The herewith disclosed invention relies exclusively on an “on the fly” correction method, (though it may be implemented as an addition to another method). And, as low cost implementation is a driving factor, it avoids storing multiple image lines for correction purposes; it relies on image sensor data available from the same scan line in an intelligent way. In a first step, it evaluates the signal change characteristic of the neighboring pixel, assuming that with a strong signal change in the neighboring pixels, a heavy change at the pixel under investigation could be expected and with only moderate changes in neighboring pixels only a smaller signal change could be expected for said pixel under investigation. A variable threshold is generated based on the before mentioned conclusions.  
         [0025]     A further plausibility check uses the fact, that a typical narrow image line hits more than just a single color. Therefore when there is coincidence of a strong signal change for the pixel under investigation, and for another color pixel in the near vicinity, this is probably related to the real image content. If however a strong signal change for said pixel under investigation is to be found in an area of otherwise moderate signal change where no other color&#39;s pixel shows strong signal change as well, the probability for a bad pixel being just detected is high. The methods and algorithms of the herewith disclosed invention efficiently implement these clauses.  
         [0026]      FIG. 4  shows an example of a single line with P−UI as the “pixel under investigation”. In this example it shows a chain of 9 pixels, with 4 pixels before (P 1  to P 4 ) and 4 pixels after (P 6  to P 9 ) said pixel under investigation P−UI; P 5  in the middle of said pixel chain is identical to said P−UI.  
         [0027]      FIG. 5  visualizes the fact, that an image line LINE with sharp edges, projected across the sensor array would normally hit multiple pixels of various colors in the near vicinity. R, G and B in  FIG. 5  represent the red, green and blue pixels of said image sensor. Therefore only in rare cases would a pixel, with a signal strongly differing from nearby pixels of the same color, stand alone, without at least one of the closest pixels of an other color, strongly differing from their nearby pixels of the same color as well. The fact just described is used for said plausibility check, assuming a pixel is bad only if none of the closest pixels of an other color shows a signal strongly differing from nearby pixels of that same other color.  
         [0028]     The principal method to achieve the objectives of this invention is illustrated in  FIG. 6 . First, provide, ( 60 ) a color image-sensing array with Bayer pattern RGB, a sensor read-out circuit and a memory to temporary store a small number of pixels on a single row, an evaluation circuits to compare values of neighboring pixels on the same row and a circuit to eventually replace said bad pixel with a newly generated value. Now, after reading in at least a few pixels, select a specific pixel as the pixel-under-investigation ( 61 ) and read its sensor value ( 62 ). Then determine a variable threshold from neighboring pixels of the same color within a single row ( 63 ). Check if the threshold is exceeded ( 64 ), if yes, the pixel may be bad, if no, pixel is good.  
         [0029]     In case of a possibly bad pixel, perform said plausibility check of the just detected bad pixel indication by examining the neighbor pixels of the other color on the same scan-line ( 65 ). If the pixel under investigation is indeed to be considered a bad pixel ( 66 ), replace it by the average of the sensor values of nearest pixels of the same color on the same scan line ( 67 ). Now proceed to the next pixel ( 68 ) and repeat the process.  
         [0030]     A more detailed description of a method to achieve the objectives of this invention is illustrated in  FIG. 7 . First, provide, ( 70 ) the means for a color image-sensing array with Bayer pattern RGB, a circuit to sequentially read the digital sensor value of a series of pixels, a memory circuit to temporarily store the read-out values of several pixels before and behind a specific pixel for investigation, an arrangement of circuits to determine a variety of aspects, when comparing values of neighboring pixels on the same row and a circuit to replace said specific pixel&#39;s value by a new value.  
         [0031]     Now, after reading in at least a few pixels, select a specific pixel as the pixel-under-investigation ( 71 ) and read its sensor value ( 72 ). Then determine the sensor signal change characteristics of the next neighbors with the same color on the same line ( 73 ) and calculate a variable threshold—a smaller threshold in case of moderate signal changes and a higher threshold in case of strong signal changes. Next, determine if the sensor signal change characteristic falls within an anticipated range by determining whether the signal of said pixel-under-investigation exceeds the maximum/minimum of said next neighbors of the same color on the same line by more than said variable threshold ( 74 ) or not. If ( 75 ) it falls within said anticipated range it is most likely a good pixel ( 712 ). If however it exceeds the anticipated threshold, the pixel under investigation may be a bad pixel ( 76 ).  
         [0032]     In case said pixel under investigation may be a bad pixel, perform said plausibility check ( 77 ) to find out if it is a true bad pixel. For this purpose, examine, if the neighbor pixels of an other color on the same scan-line, show strong signal changes as well. If neighbor pixels of an other color show strong change ( 78 ) as well, a strong signal change of the pixel under investigation could be anticipated and the pixel is more likely a good pixel ( 711 ). If however the neighbor pixels of an other color show only moderate change, a strong signal change of the pixel under investigation indicates a truly bad pixel ( 79 ). In this case replace the detected bad pixel by an average value of the nearest pixels of the same color before and behind said pixel under investigation ( 710 ).  
         [0033]     Continue with the next pixel ( 713 ).  
         [0034]     The following is one example for a detailed implementation of the algorithm for bad pixel detection and correction. The same method is also visualized in  FIG. 8 .  
         [0035]     Start with characterization of a specific pixel under investigation after a few pixels have been read in one sensor row ( 80 )( 81 ).  
         [0000]     Algorithm 1:  
         [0036]     To begin, prepare some constants are defined: offset, mindiff, correction factor f. 
        const f=1.0;     const offset=10;     const mindiff=8; 
 
 Determine max/min sensor values of neighboring pixels P 1 , P 3 , P 7 , P 9  having the same color as P−UI=P 5  ( 82 ). 
 
 Algorithm 2: 
       
 
         [0040]     Then the maximum value of pixels P 1 , P 3 , P 7 , P 9  is determined and the minimum value of pixels P 1 , P 3 , P 7 , P 9  is determined. 
        max=P 1 ;     if (max&lt;P 3 ) then max=P 3 ;     if (max&lt;P 7 ) then max=P 7 ;     if (max&lt;P 9 ) then max=P 9 ;     min=P 1 ;     if (min&gt;P 3 ) then min=P 3 ;     if (min&gt;P 7 ) then min=P 7 ;     if (min&gt;P 9 ) then min=P 9 ; 
 
 Evaluate the signal change characteristic of the neighboring pixel, assuming that with a strong signal change in the neighboring pixels, a strong change at the pixel under investigation could be expected and with only moderate changes in neighboring pixels only a smaller signal change could be expected for said pixel under investigation. 
 
 Algorithm 3: 
       
 
         [0049]     Calculate the absolute difference of the next neighbor pixels P 3 −P 7  ( 83 ). But if said absolute difference of pixels P 3 −P 7  is smaller than the constant mindiff, set said absolute difference to the value of mindiff ( 84 ), thus defining a variable threshold. ( 85 ) 
        next_neighbors_difference=abs(P 3 −P 7 );     if (next_neighbors_difference&lt;mindiff) then next_neighbors_difference=mindiff; 
 
 Algorithm 4: 
       
 
         [0052]     If the value of P−UI=P 5  exceeds the maximum of pixels P 1 , P 3 , P 7  and P 9  by more than the absolute difference of P 3 −P 7 , multiplied by f, or 
    If the value P−UI=P 5  falls below the minimum of pixels P 1 , P 3 , P 7  and P 9  by more than the absolute difference of P 3 −P 7 , multiplied by f, ( 86 )     then the pixel is a candidate for a bad-pixel. ( 87 )     bad-pixel-candidate-high ═(P 5 &gt;(max+f*next_neighbors_difference));     bad-pixel-candidate-low ═(P 5 &lt;(min-f*next_neighbors_difference)); 
 
 If a pixel is a candidate for a bad pixel ( 88 ), said plausibility check is done by looking at the intermediate color pixels ( 89 ). The correction of the pixel is done only, if in the intermediate color no peak in the intensity is visible. 
 
 Algorithm 5: 
   
 
         [0057]     If the pixel is a candidate for bad-pixel, being excessive high, 
        and P 4  is smaller or equal to P 2  plus offset     or P 4  is smaller or equal to P 8  plus offset     and P 6  is smaller or equal to P 2  plus offset     or P 6  is smaller or equal to P 2  plus offset 
 
 or if the pixel is a candidate for bad-pixel, being excessive low, 
    and P 4  is greater or equal to P 2  minus offset     or P 4  is greater or equal to P 8  minus offset     and P 6  is greater or equal to P 2  minus offset     or P 6  is greater or equal to P 2  minus offset 
 
 the pixel is a bad pixel 
        if (bad-pixel-candidate-high) then bad-pixel-candidat-hi=(((P 4 &lt;=P 2 +offset)||(P 4 &lt;=P 8 +offset)) &amp;&amp; ((P 6 &lt;=P 2 +offset)||(P 6 &lt;=P 8 +offset)));     if (bad_pixel_candidate-low) then bad-pixel-candidat-lo=(((P 4 &gt;=P 2 -offset)||(P 4 &gt;=P 8 -offset)) &amp;&amp; ((P 6 &gt;=P 2 -offset)||(P 6 &gt;=P 8 -off set))); 
 
 Finally a bad pixel correction is done, if said plausibility check results in the pixel under investigation being a true bad pixel ( 811 ), otherwise it is considered a good pixel ( 813 ). Said bad pixel is corrected by replacing its value with the average values of pixels P 3  and P 7 . ( 812 ) 
 
 Algorithm 6: 
   
 
         [0068]     If a pixel is a bad pixel, replace its value P−UI=P 5  with the average of P 3  and P 7 . 
    if (bad-pixel-candidate-high  1 I bad-pixel-candidate-low) then P 5 =(P 3 +P 7 )/2; 
 
 Continue with the next pixel as a new pixel under investigation ( 814 ) and repeat the same process. 
   
 
         [0070]     While the invention has been particularly shown and described with reference to the preferred embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made without departing from the spirit and scope of the invention.