Abstract:
A method of removing columnar streaks from a digital image of the type in which it is assumed that pixels in a predetermined region near a given pixel are strongly related to each other and employing gain and offset values to compute streak removal information, a test is performed for a strong relation between the pixels in a predetermined region near a given pixel and streak removal information is computed only if such a strong relationship exists, whereby image content that does not extend the full length of the image in the column direction will not be interpreted as a streak.

Description:
FIELD OF THE INVENTION 
     The invention relates generally to the field of image processing, and in particular to an image processing method for removing streaks in digital images. The invention is particularly useful in removing streaking in digital images that are acquired by a linear image sensing array, but may also be used to remove streaks in conventional film that are caused by the camera or processing equipment. 
     BACKGROUND OF THE INVENTION 
     Every detector in an electronic image sensor, such as a CCD image sensor, may have a different response function that relates the input intensity of light (or other electromagnetic radiation) to a pixel value in the digital image. This response function can change with time or operating temperature. Image sensors are calibrated such that each detector has the same response for the same input intensity (illumination radiance). The calibration is generally performed by illuminating each detector with a given radiance from a calibration lamp and then recording the signal from each detector to estimate the response function of the detector. The response function for each detector is used to equalize the output of all of the detectors such that a uniform illumination across all of the detectors will produce a uniform output. 
     FIG. 1 shows a schematic of an image acquired by a linear image sensing arrays. In such an image, if errors in estimating the response curve of a detector are different from the errors in the response curve of an adjacent detector, the detector responses will not be equalized and streaking 2 will appear in the image along the scan direction indicated by arrow A. Often to achieve a very long array, several image sensor chips are joined together to form a single linear image sensor. When calibration errors occur between chips, the streaking is generally referred to as banding, as illustrated at reference numeral 4. 
     Even when the detectors are calibrated to minimize the streaking in the image, some errors from the calibration process are unavoidable. Each detector is sensitive to a slightly different spectrum of light, but they are all calibrated using the same calibration lamp with a broad, non-uniform spectrum. Since the scene spectrum is unknown, the calibration process assumes the spectrum of the calibration lamp and the scene are identical. The spectrum of the calibration lamp will usually be somewhat different than the spectrum of the scene being imaged, hence calibration errors will occur. Calibration errors also occur because the calibration process includes an incomplete model of the complete optical process and because the response function for each detector changes over time and operating temperature. 
     Streaking can be seen in uniform areas of an image acquired by a linear detector and become very apparent when the contrast of the image is enhanced. Calibration differences between the red, green, and blue detectors of color imaging systems produce streaks of varying colors in the composite color image. These streaks not only reduce the aesthetic quality of digital images but can impact the interpretability of features in the images. Streaking also severely degrades the performance of pattern recognition and feature extraction algorithms. 
     Streaks can be attenuated by reducing the contrast of the image or by blurring the image in a direction perpendicular to the streaking, but these methods degrade the quality of the overall image. Previously developed algorithms designed to remove the streaks while preserving the contrast and sharpness of the image assume that the streaks do not change over time or that the pixels near each other are strongly correlated. In general these assumptions are not true. The responsivity of each detector, and hence the magnitudes of the streaks, changes over time, therefore methods that remove fixed patterns of streaks will not always work. U.S. Pat. No. 5,065,444, issued Nov. 12, 1991, to Garber discloses a method of removing streaks from digital images by assuming that pixels in a predetermined region are strongly correlated, examining the pixels in the region, computes the difference between the pixels in the region, thresholding the pixel differences lower than a predetermined value, computes a gain and offset value from the distribution of differences, and uses the gain and offset value to remove the streaking. Methods that assume a strong correlation between pixels that are near each other, such as the one disclosed by Garber will interpret scene variations as streaks and produce additional streaking artifacts in the image as a result of attempting to remove existing streaks. FIG. 2a shows an image having streaks 2 and linear features 6 that are in the same direction as the streaks. As shown in FIG. 2b, the correction of the streaks 2 using the method taught by Garber remove the streaks 2, but results in additional streaking artifacts 8. 
     There is a need therefore for an improved digital image processing method for removing streaks in images. 
     SUMMARY OF THE INVENTION 
     The object of the present invention is achieved in a method of removing columnar streaks from a digital image of the type in which it is assumed that pixels in a predetermined region near a given pixel are strongly related to each other and employing gain and offset values to compute streak removal information, by testing for a strong relation between the pixels in a predetermined region near a given pixel and computing streak removal information only if such a strong relationship exists, whereby image content that does not extend the full length of the image in the columnar direction will not be interpreted as a streak. 
     The method of the present invention adaptively removes streaking, as well as banding, in digital images without reducing the sharpness or contrast of the image. Streaking occurs in images output from linear scanners and is generally caused by differences in the responsivity of detectors or amplifiers. The method disclosed detects pixel locations in the image where pixel-to-pixel differences caused by streaking can be distinguished from normal variations in the scene data. A linear regression is performed between each pair of adjacent pixels in a direction perpendicular to the streaking at the detected locations. A statistical outlier analysis is performed on the predicted values to remove the pixels that are not from the streaking. A second linear regression is performed to calculate the slope and offset values. The slope is set to unity if it is not statistically different from unity, and the offset is set to zero if it is not statistically different from zero. The slope and offset values are then used to remove the streaking from the corresponding line of image data. 
     ADVANTAGEOUS EFFECT OF THE INVENTION 
     This invention adaptively removes streaking in a digital image by testing for a strong correlation between the pixels in a predetermined region and computing streak removal information only if such a strong relationship exists. This process will remove the residual streaks that appear even after a calibration is performed on the imaging sensor. This method does not reduce the contrast or the sharpness of the image. 
    
    
     BRIEF DESCRIPTION OF THE DRAWINGS 
     FIG. 1 illustrates the streaking artifact in an image; 
     FIGS. 2(a)-2(b) illustrate the artifacts produced by methods that assume that pixels in a predetermined region near a given pixel are strongly related to each other; 
     FIG. 3 is a diagram showing an image processing chain using the present invention; 
     FIG. 4, made up of FIG. 4(a) and FIG. 4(b), is a flow chart of the streak removal process according to the present invention; 
     FIG. 5 illustrates a digital mask that is used to test for pixels that are strongly related to each other in a predetermined region; and 
     FIG. 6 is a graph illustrating the linear regression with the two adjacent columns of image data, useful in describing the method of the present invention. 
    
    
     DETAILED DESCRIPTION OF THE INVENTION 
     The streak removal process of the present invention can be employed in a typical image processing chain, such as the one shown in FIG. 3. A digital sensor 10, e.g. a linear scanner used in a camera system or a photographic scanner, outputs a digital image 12. If the detectors have gone through a calibration process, then the digital image 12 may go through a detector equalization process 14 to produce an equalized image 16. Both the digital image 12 and the equalized image 16 will contain streaks 2 as shown in FIG. 1. The digital image 12 or the equalized digital image 16 is processed through the streak removal process 18 to produce a corrected digital image 20 that has the streaks removed. This corrected digital image 20 is then processed through the nominal image processing chain and enhancements 22 to produce the final processed image 24. Without the streak removal process 18, the image processing and enhancements 22 may actually reduce the quality of the final processed image 24, especially if the digital image 12 is low in contrast or if the image processing and enhancements 22 includes a feature extraction algorithm. 
     If the original image was a photographic image having streaks or scratches resulting from the photographic, for example the scratches seen in old movie film, the images may be scanned in a high quality scanner and the streaks or scratches removed by the method of the present invention. 
     For the discussion of this invention it will be assumed that the streaks occur in the columnar direction of the digital image 12. The pixel at column coordinate x and row coordinate y has a digital count value i(x,y). If d x  is the detector for column x, then the response curve for detector d x  in the digital sensor 10 can be modeled as a linear function of the input illumination radiance, thus 
     
         i(x,y)=a.sub.x I(x,y)+b.sub.x,                             (1) 
    
     where I(x,y) is the intensity of the illumination radiance at location (x,y) in the image, a x  is the gain for detector d x , and b x  is the bias for detector d x . 
     Streaks occur in the digital image 12 because adjacent detectors in the digital sensor 10 have different response curves. The difference Δ(x,y) between adjacent pixels is given by 
     
         Δ(x,y)=i(x,y)-i(x+1,y)=a.sub.x I(x,y)+b.sub.x -a.sub.x+1 I(x+1,y)-b.sub.x+1,                                       (2) 
    
     and is dependent on the detector response as well as the difference between the illumination radiance incident on the adjacent pixels. If the detectors d x  and d x+1  have the same response curves, i.e. if a x  =a x+1  and b x  =b x+1 , then 
     
         Δ(x,y)=i(x,y)-i(x+1,y)=a.sub.x  I(x,y)-I(x+1,y)!,    (3) 
    
     and the difference between i(x,y) and i(x+1,y) is proportional to the difference between the illumination radiance incident on the adjacent pixels, which is desired, and no streaks due to sensor calibration errors will be present. 
     If I(x,y)=I(x+1,y) in Eq. (2) then 
     
         Δ(x,y)=i(x,y)-i(x+1,y)= a.sub.x -a.sub.x+1 !I(x,y)+ b.sub.x -b.sub.x+1 !,                                             (4) 
    
     and the difference between i(x,y) and i(x+1,y) is entirely from the different response curves between detectors d x  and d x+1 . 
     If I(x+1,y) is substituted for I(x,y) using Eq. (1) then ##EQU1## If ##EQU2## then 
     
         i(x,y)=Δa.sub.x i(x+1, y)+Δb.sub.x             (6) 
    
     and i(x,y) is just a linear transformation of i(x+1,) with a slope Δa x  and offset Δb x . By determining Δa x  and Δb x , the streaking between columns x and x+1 can be removed if the pixel count values i(x+1,y) are replaced with i&#39;(x+1,y) where 
     
         i&#39;(x+1,y)≡Δa.sub.x i(x+1,y)+Δb.sub.x.    (7) 
    
     The difference between adjacent pixels is now ##EQU3## which is the desired result from Eq. (3), hence no streaks due to sensor calibration error will be present. 
     Methods that determine Δa x  and Δb x  by assuming that the illumination radiance is always approximately equal in a predetermined region near pixel i(x,y), e.g. I(x,y)≈I(x+1,y), such as the one disclosed in U.S. Pat. No. 5,065,444, will generate poor estimates of Δa x  and Δb x  where I(x,y)≢I(x+1,y) and artifacts will occur. 
     According to the present invention, a test is performed for a strong relationship in spatial features between pixels and computes Δa x  and Δb x  only from those pixels where I(x,y)≈I(x+1,y) thus preventing artifacts due to the processing to remove streaking from occurring. A schematic of the streak removal process 18 disclosed in this invention is shown in FIG. 4. First two adjacent columns of image data are selected 30. Next, a column of pixel value pairs representing the pixel values of the adjacent pixels of the two columns is formed 32. Next a pair of columns of local mean values representing the mean values of pixels in an N pixel window for each of the adjacent columns of image data is formed 34. The local means μ(x,y) and μ(x+1,y) are calculated using ##EQU4## where N is the window length. To determine if I(x,y)=I(x+1,y), a mask, such as the mask 35 shown in FIG. 5, is centered at pixel i(x,y) and convolved with the image. Pixels in the first and last ((N-1)/2 rows of the image will not be used to determine Δa x  and Δb x . 
     Next, a local difference metric M(x,y) is calculated 36 that measures the similarity between local pixel regions. A difference metric based on the difference between the mean reduced values is given by ##EQU5## 
     The local pixel regions are similar if M(x,y)&lt;T M , where T M  is the difference metric threshold. The optimal value for T M  will depend on the characteristics of the digital sensor 10. A maximum difference threshold, T, is defined by determining the largest magnitude difference of Δ(x,y) that is possible from calibration differences alone. 
     To determined the values of Δa x  and Δb x  in Eq. (7), two columns of pixel values i x  (n) and pixel values i x+1  (n), where n is a counting index, are generated 38 for each row x, where only the k values of i(x,y) and i(x+1,y) that satisfy the conditions M(x,y)&lt;T M  and |Δ(x,y)|&lt;T.sub.Δ are used. 
     Initial estimates of the slope and offset are determined by performing a linear regression between i x  (n) and i x+1  (n) to determine the regression line 39 in FIG. 6. The initial estimate of the slope, Δa&#39; x , is calculated 40 by ##EQU6## where k is the total number of elements in i x  (n). The initial estimate of the offset, Δb&#39; x , is calculated 42 by ##EQU7## 
     The slope Δa x  and offset Δb x  for Eq. (7) are determined by performing a second linear regression between i x  (n) and i x+1  (n) after the statistical outliers 43 in FIG. 6 have been removed from the estimates of Δa&#39; x  and Δb&#39; x . The standard error s e  of the linear regression is calculated 44. The statistical outliers 43 will be defined as points lying outside a boundary 45 that is dependent on the standard error of estimate s e , given by ##EQU8## where 
     
         ix(n)=Δa.sub.x i.sub.x+1 (n)+Δb.sub.x.         (14) 
    
     Values of i(x,y) that satisfy the condition |i x  (n)-i x  (n)|&lt;T S  are determined 46, these values are not statistical outliers. The outlier threshold T S  is proportional to s e  and is typically set equal to 3s e . Two new columns of pixel values, i x  (n) and its adjacent pixel i x+1  (n) are generated 48 for each row x, where only the j≦k. The slope Δa x  and offset Δb x  for Eq. (7) are now determined 50 by ##EQU9## 
     The final statistical tests performed 52 are to determine if the slope Δa x  is statistically different from unity and the offset Δb x  is statistically different from zero. These tests are performed to ensure that the difference in the response curves estimated for detectors d x  and d x+1  are statistically different. If they are not statistically different, then using the estimates for Δa x .sbsb.-- 1 and Δb x .sbsb.-- 0 may add streaking to the image rather than remove it, hence degrading the quality of the image rather than improving it. 
     A statistical hypothesis test is used to determine if the slope Δa x  is statistically different from unity. The t statistic is given by ##EQU10## where ##EQU11## 
     The t statistic is compared to the t distribution value tα/2 to determine if Δa x  is statistically different from unity. If t.sub.Δa.sbsb.x &lt;tα/2 then Δa x  is not statistically different from unity hence a value of 1 is used 54 for Δa x  in Eq. (7). The value used for tα/2 depends on the number of sample points j as well as the confidence level desired for the statistical test, which is given by 100(1-α)%. For a 95% confidence and j&gt;50, tα/2=1.96. 
     To determine if the offset Δb, is statistically different from zero, the t statistic is given by ##EQU12## If t.sub.Δb.sbsb.x &lt;tα/2 then Δb x  is not statistically different from zero hence a value of 0 is used 56 for Δb x  in Eq. (7). 
     Finally, the pixels i(x+1,y) in column x are modified by Eq. (7) to remove the steaks 58. The procedure outlined above is repeated for the next column of image data. This process is continued until all columns of the image data have been processed and the corrected digital image 20 is output. 
     A listing of a computer program written in the Fortran language running on a Convex computer for performing the method of the present invention is included as Appendix A. 
     The invention has been described with reference to a preferred embodiment. However, it will be appreciated that variations and modifications can be effected by a person of ordinary skill in the art without departing from the scope of the invention. 
     
                       APPENDIX A______________________________________*   Subroutines Called:*   Open.sub.-- Tiff.sub.-- Image*   Read.sub.-- Tiff.sub.-- Image*   Close.sub.-- Tiff.sub.-- image*   Write.sub.-- Tiff.sub.-- Image2**************************************************************Implicit NoneInclude   `/cm/include/imgio.inc`Integer*4 IMGioPtr, npixels, nlines, Bits, band, datatypeCharacter*50     inname, outnameCharacter*4     cstdev, cthresh, cc1, cmaxdiff, cconfa, cconfbReal*4    Input.sub.-- Image(:,:),Output.sub.-- Image(:,:),     metric.sub.-- image(:,:),stdevAllocatable     (Input.sub.-- Image, Output.sub.-- Image, metric.sub.-- image)Parameter (band=0)Integer*4 X, Y, z, carg, narg, iargc, L, K, i1, i2REAL*8    line1(2048), line2(2048), line1a(2048), line2a(2048),     metric(2048)REAL*8    sum, sum1, sum2, sum11, sum22, sum12, cnt, sd,     sumprod, maxdiffREAL*8    max, min, max0, min0, MSE, cl, s, si, confa, confb,     ta, tb, bias, gainREAL*8    slope(2048), offset(2048), diff, linecnt, mean1, mean2,     thresh************* Read command line arguments* carg = 1 narg = iargc() IF(carg .le. narg )THEN  CALL getarg(carg,inname)  carg = carg + 1  CALL getarg(carg,outname)  carg = carg + 1  CALL getarg(carg,ccl)   READ(ccl,*) cl  carg = carg + 1  CALL getarg(carg,cmaxdiff)   READ(cmaxdiff,*)maxdiff  carg = carg + 1  CALL getarg(carg,cstdev)   READ(cstdev,*)stdev  carg = carg + 1  CALL getarg(carg,cthresh)   READ(cthresh,*)thresh  carg = carg + 1  CALL getarg(carg,cconfa)   READ(cconfa,*) confa  carg = carg + 1  CALL getarg(carg,cconfb)   READ(cconfb,*)confb ELSE  WRITE(6,*)`Usage: remove.sub.-- cal input.sub.-- file output.sub.--filewindow.sub.-- size maxdiff + stdev.sub.-- coeff Metric.sub.-- thresholdslope.sub.-- confoffset.sub.-- conf`   goto 999 ENDIF  Write(6,*) `RUNNING REMOVE CALIBRATION NOISE   ROUTINE`  write(6,*) `*** Only works on 2k or smaller images ***`  Write(6,*) `raw input filename =`,inname  Write(6,*) `output filename =`,outname  Write(6,*) `window size =`,cl  Write(6,*) `maximum difference =`,maxdiff  Write(6,*) `outlier stdev coefficient =`,stdev  Write(6,*) `MSE threshold =`,thresh  Write(6,*) `slope confidence t value =`,confa  Write(6,*) `offset confidence t vatue =`,confb************* Read input imagery - pixel &amp; line sizes* CALL Open.sub.-- Tiff.sub.-- Image(inname,IMGioPtr,npixels,nlines,Bits) ALLOCATE (Input.sub.-- Image(npixels, nines)) ALLOCATE (Output.sub.-- Image(npixels, nlines)) ALLOCATE (metric.sub.-- Image(npixels, nlines)) CALL Read.sub.-- Tiff.sub.-- Image(IMGioPtr,Input.sub.-- Image,npixels,nlines,Band) CALL Close.sub.-- Tiff.sub.-- Image(IMGioPtr)************ Determine mean-square error between line segments after  bias is removed DO X = 1, npixels-1  DO Y = int(cl/2)+1, nlines-int(cl/2)   mean1=0   mean2=0   do z=-int(cl/2),int(cl/2)    meanl=mean1+input.sub.-- image(x,y+z)/cl    mean2=mean2+input.sub.-- image(x+1,y+z)/cl   enddo  MSE=0  do z=-int(cl/2),int(cl/2)   MSE=MSE+((input.sub.-- image(x,y+z)-mean1)-(input.sub.-- image(x+    1,y+z)-mean2))**2  enddo   metric.sub.-- image(x,y)=sqrt(MSE)  enddo enddo************* Determine slope and offset to remove streaks************ Use only those points that have a low MSE  (high correlation) between lines* DO X = 1, npixels-1  linecnt = 0  slope(x)= 1  offset(x)=0  DO Y = int(cl/2)+1, nlines-int(cl/2)   diff = input.sub.-- image(X+1,Y) - input.sub.-- image(X,Y)   IF(metric.sub.-- image(x,y).le.thresh.and.abs(diff).le.maxdiff)then    linecnt = linecnt + 1   line1(linecnt)=input.sub.-- image(x,y)   line2(linecnt)=input.sub.-- image(x+1,y)  ENDIF ENDDO if(linecnt.gt.2)then  sum1 = 0.0  sum2 = 0.0  sum12 = 0.0  sum22 = 0.0  sum11 = 0.0  DO Y = 1, linecnt   sum1 = sum1 + line1(y)   sum2 = sum2 + line2(y)   sum12 = sum12 + line1(y)*line2(y)   sum11 = sum11 + line1(y)**2   sum22 = sum22 + line2(y)**2  ENDDO  slope(x)=(linecnt*sum12-sum1*sum2)/(linecnt*sum22-sum2**2)  offset(x)=(sum1-stope(x)*sum2)/linecnt* calculate standard error  sum = 0.0  DO Y = 1, linecnt   sum = sum + (line1(y)-slope(x)*line2(y)-offset(x))**2 ENDDO sd = sqrt(sum/(linecnt-2))* throw away outliers to improve calculation  cnt = 0.0  DO Y = 1,linecnt   IF(abs(line1(y)-slope(x)*line2(y)-offset(x)).le.(stdev*sd))THEN    cnt = cnt + 1   line1a(cnt)=line1(y)   line2a(cnt)=line2(y)  ENDIF ENDDO if(cnt.gt.2)then  linecnt=cnt  sum1 = 0.0  sum2 = 0.0  sum12 = 0.0  sum22 = 0.0  sum11 = 0.0  DO Y = 1,cnt   sum1 = sum1 + line1a(y)   sum2 = sum2 + line2a(y)   sum12 = sum12 + line1a(y)*line2a(y)   sum11 = sum11 + line1a(y)**2   sum22 = sum22 + line2a(y)**2  ENDDO   slope(x)=(cnt*sum2-sum1*sum2)/(cnt*sum22-sum2**2)   offset(x)=(sum1-slope(x)*sum2)/cnt endif s=sqrt(abs(sum11-sum1**2/linecnt-slope(x)*(sum12-  sum1*sum2/linecnt))/(linecnt-2)) si=sqrt(abs(sum22-sum2**2/linecrt)) ta=si*abs(slope(x)-1)/s tb=si*abs(offset(x)-0)/s/sqrt(sum22/cnt) if(ta.lt.confa)then  slope(x)=1  offset(x)=(sum1-slope(x)*sum2)/cnt endif if(tb.lt.confb)offset(x)=0 endifENDDO************* remove calibration differences* DO Y = 1,nlines  output.sub.-- image(1,y)=input.sub.-- image(1,y) ENDDO bias=0 gain=1 DO X = 1, npixels-1  bias=bias+offset(x)  gain=gain*slope(x)  DO Y = 1, nlines   output.sub.-- image(x+1,y)=gain*input.sub.-- image(x+1,y)+bias  ENDDO ENDDO************* DRA to avoid clipping* min = 10000 max = -10000 DO Y = 1, nlines  DO X = 1, npixels   IF(output.sub.-- image(X,Y).lt.min)min = output.sub.-- image(X,Y)   IF(output.sub.-- image(X,Y).gt.max)max = output.sub.-- image(X,Y)  ENDDO ENDDO max0=2047 min0=0 DO Y = 1, nlines  DO X = 1, npixels   output.sub.-- image(X,Y) = NINT((max0-min0)*(output.sub.-- image(X,Y)-1min)/(max-min)+min0)  ENDDO ENDDO************* Write output image* Datatype = 7 write(6,*)` ` write(6,*)`Writing Output Imagery` write(6,*)` `  Call Write.sub.-- Tiff.sub.-- Image2( Outname, Output.sub.-- Image,NPixels, + Nlines, Datatype, Band, bits)   goto 999************* END - OF - ROUTINE*999 END______________________________________ 
    
     
         ______________________________________PARTS LIST______________________________________2   streaks4   banding6   scene variation8   image artifact10  digital sensor12  digital image14  detector equalization16  equalized image18  streak removal process20  corrected digital image22  image processing and enhancements24  final processed image30  selecting two adjacent columns of pixels step32  create two columns of adjacent pixel values step34  calculate local means step35  mask used for testing pixel relationship36  calculate local difference metric step38  remove pixel values from columns of pixel values that exceed    thresholds step39  line from linear regression40  determine initial estimate of slope step42  determine initial estimate of offset step43  statistical outliers44  calculate standard error of linear regression step45  statistical outlier boundary46  determine statistical outliers step48  remove statistical outliers from columns of pixel values step50  determine new estimate of slope and offset step52  determine t statistics for slope and offset step54  set slope to unity if not statistically different from unity step56  set offset to zero if not statistically different from zero step58  remove streaking using slope and offset values step______________________________________