Adaptive non-uniformity compensation algorithm

An adaptive method for removing fixed pattern noise from focal plane array (FPA) imagery. A set of correction terms is applied to the focused image from the FPA, and a filter is applied to the corrected, focused image. The set of correction terms is also applied to a blurred version of the FPA image, and the filter is applied to the corrected, blurred image. Fixed pattern noise errors are then calculated using the filtered imagery, and employed to update the correction terms. The updated correction terms are then used for processing the next image. In one embodiment, the filter is an anti-median filter. In another embodiment, the filter is an anti-mean filter.

TECHNICAL FIELD OF THE INVENTION 
This invention relates to processes for the removal of fixed pattern noise 
from sensor images. 
BACKGROUND OF THE INVENTION 
Focal plane arrays (FPAs) are used in various applications to capture 
images for subsequent processing. For example, air-to-air missiles have 
employed infrared sensors with FPAs to capture infrared images of a target 
being tracked. The focal plane array contains n.times.n infrared 
detectors. Each detector has a slightly different sensitivity to infrared 
radiation. This non-uniform sensitivity yields fixed pattern noise (FPN). 
FPN manifest itself in the image resulting in some pixels which are too 
bright and some which are too dark. 
Some missile sensor processors have employed algorithms to reduce FPN. 
Currently used algorithms have introduced significant scene artifacts in 
the output image which causes the missile tracker to mis-track the target. 
Efforts to reduce the scene artifact growth have resulted in insufficient 
fixed pattern noise reduction under certain simulated flight conditions. 
It would therefore be advantageous to provide a technique for removal of 
fixed pattern noise from sensor images without introducing significant 
scene artifacts. 
SUMMARY OF THE INVENTION 
An adaptive method for removing fixed pattern noise from focal plane array 
(FPA) imagery is described. The method comprises the following steps: 
obtaining respective focused and blurred digitized versions of an FPA 
image; 
applying a set of correction terms to the focused version of said FPA image 
to provide a corrected, focused FPA image; 
applying a filter to the corrected, focused version of said FPA image to 
obtain a filter focus image; 
applying said set of correction terms to said blurred version of the FPA 
image to provide a corrected, blurred version of said FPA image; 
applying said filter to the corrected, blurred image to provide a filter 
blur image; 
calculating fixed pattern noise errors using the filter focus image and the 
filter blur image; and 
employing said fixed pattern noise errors to update the correction terms 
for use in processing a next FPA image. 
In a first embodiment, the filter includes an anti-median filter which 
provides, for each pixel in an image, a corresponding anti-median value 
which is a measure of how the pixel differs from a median value of the 
pixel and a set of neighboring pixels. In a second embodiment, the filter 
includes an anti-mean filter which provides, for each pixel in an image, a 
corresponding anti-mean value which is a measure of how the pixel differs 
from a mean value of the pixel and a set of neighboring pixels.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 
FIG. 1 is a simplified schematic diagram of a missile tracking system with 
which the present invention is advantageously utilized. In general the 
tracking system includes a sensor 10 which produces an image. Typically, 
the sensor 10 includes optics 12 for collecting and focusing incident 
infrared energy from the ambient scene and target on the focal plane array 
14, which produces an infrared image of the scene/area being tracked. The 
sensor 10 includes an analog-to-digital (A/D) convertor 16, which converts 
each of the n.times.n detector outputs into digital (numeric) values or 
pixels. The digitized images are processed in accordance with the 
invention by the adaptive non-uniformity compensation (NUC) processor 20 
to provide corrected output images 30 in which scene artifacts have been 
reduced, and fixed pattern noise has been reduced or eliminated. The NUC 
processor 20 adjusts each pixel to compensate for the differing 
sensitivity of each detector in the FPA 14. The missile tracking system 
(not shown) then employs the corrected output images in the tracking of 
targets. 
FIG. 2 illustrates a sequence of images 1 . . . n+1, alternating between a 
focussed image and a blurred image. The sensor output yields alternating 
focused and blurred images. The focus and blurring is controlled by the 
sensor optics 12. One exemplary way to achieve the focus and blurred 
images is to use a filter wheel in the sensor optics 12. The filter wheel 
spins, and has separate optics/lens to provide the focus image and the 
blurred image. The blurred images are used as part of the NUC process to 
help discriminate scene and target from FPN. 
FIG. 3 is a top level flow diagram illustrative of a first embodiment of 
the adaptive NUC process of the invention. The input image 18, which is 
either a focused image 18A or a blurred image 18B, is received from the 
sensor 10. The correction terms 26 are applied to the input image at step 
22 to yield a corrected output image. The correction terms are initially 
zeros. The corrected output image 30 is passed both to the tracking system 
and to the NUC filter system 24. A different NUC filter is used for both 
the focused and blurred images. These two filters are described below. The 
filter system 24 updates the correction terms 26 based on the corrected 
output image. 
FIG. 4 illustrates the focused image processing for the filter system 24 of 
FIG. 3. The input focused image 18A is received from the sensor, and the 
correction terms 26 are applied at step 22 to yield the corrected focused 
image 30, as described above. The filter system 24 includes a focused 
image filter 24A, which includes the application of an anti-median filter 
24C at step 24A1 to the corrected focused image to yield an anti-median 
image 30A. The image 30A is stored in memory for use on the following 
blurred image 18B. 
FIG. 5 is a flow diagram showing the blurred image processing for the 
filter system 24 of FIG. 3. The blurred image 18B is received from the 
sensor 10, and the correction terms 26 are applied at step 22 to provide 
the corrected blur image 30B, an output useful for diagnostics. The filter 
system 24 includes a blurred filter 24B which includes the application of 
the same anti-median filter 24C used in the focused filter 24A to the 
corrected blurred image. As the anti-median filter is applied to the 
corrected blurred image, the output is compared to the filter output 
stored from the previous focus frame via a NUC comparison function 24B2, 
described more fully below. The output of the comparison function yields 
error terms which are used to update the correction terms for the next 
input focus image. 
FIGS. 6, 7A-7D and 8 illustrates the application of the anti-median filter 
24C. The input image 18 is n pixels by n pixels in size. Each pixel has a 
numerical value representing brightness/intensity. For each pixel X in the 
image, the brightness/intensity values of X and two of its neighboring 
pixels are used to calculate a median value. The median is calculated by 
sorting the three pixels in ascending order of brightness/intensity 
values, and selecting the middle one of the sorted list to be the median. 
Two extra rows and columns of alternating Maximum (M) and minimum (m) 
values are added to the image to handle the boundary conditions. The 
"anti-median" is a measure of how the pixel differs from the median. The 
"anti-median" is calculated by subtracting the pixel "X" from the median 
as in the following: 
EQU anti.sub.-- median(row, col)=median(row,col)-pixel(row,col)--col) 
The result of the filtering is an n.times.n set 18D of anti-median values 
for the image 18. 
There are four filter patterns, illustrated as patterns 1-4 in FIGS. 7A-7D, 
which are used in the filter system 24. In FIGS. 7A-7D, the shaded areas 
represents pixels that are used in the median calculation; the center 
pixel is marked as "X." The patterns are cycled, pattern 1 through pattern 
4, for each successive pair of focus/blur image frames, as illustrated in 
FIG. 8. Thus, for a first set of focus/blur images, i.e. images 1 and 2, 
filter pattern 1 is used, for the second set of images 3 and 4, filter 
pattern 2 is used, for the third set of images 5 and 6, filter pattern 3 
is used, and for the fourth set of images 7 and 8, filter pattern 4 is 
used, with the filter pattern cycle starting again with images 9 and 10. 
The comparison function 24B2 (FIG. 5) is now described. Each value (row, 
column) of the anti-median image (focus) is compared to the corresponding 
value of the anti-median image (blur) as follows. Test 1: check whether 
the sign (.+-.) of the anti-median focus is equal to the sign (.+-.) of 
the anti-median blur. Test 2: check whether or not the anti-median blur is 
greater than some fraction (say, 3/4) of the anti-median focus. If Test 1 
and Test 2 pass, then the error term is set equal to the anti-median 
focus; otherwise the error term is set to zero. 
The calculation of the anti-median can be positive or negative depending if 
the center pixel "X" is greater than or less than the median value. For 
example, assume that the median is 100 and the center pixel is 90. The 
anti-median would be equal to 100 minus 90, or positive 10. But if the 
center pixel was 115, then the anti-median would be equal to 100 minus 115 
or -15 which is a negative value. The errors and the correction terms can 
be positive or negative. Positive values are used to correct pixels which 
are too dark. Negative values are used to correct pixels which are too 
bright. 
The comparison function 24B2 may be written in the following manner: 
if sign(anti.sub.-- median.sub.-- blur(row,col)=sign(anti.sub.-- 
median.sub.-- focus(row,col) 
and abs(anti.sub.-- median.sub.-- blur(row,col) greater than (NUC.sub.-- 
FACTOR times (abs(anti.sub.-- median.sub.-- focus(row,col) 
then 
error.sub.-- term(row,col)=anti.sub.-- median.sub.-- focus(row,col) 
else 
error.sub.-- term(row,col)=0 
end if 
where NUC.sub.-- FACTOR is 3/4. 
The error terms are used to update the correction terms. Each Error 
Term(row,col) is multiplied by two and then added to the Correction 
Term(row,col) to yield updated Correction Terms. Because the Correction 
Terms are of higher precision than the image, the Correction Terms are 
divided by eight before they are added to the image. The "multiply Error 
Terms by two, divide by eight" functions result in a one quarter (1/4) 
correction of the estimated error. The updating of the Corrections terms 
is illustrated diagrammatically in FIG. 9, which shows the Error Terms (n 
values.times.n values) being multiplied by 2, and added to the existing 
set of Correction Terms (n values.times.n values), to yield an updated set 
of Correction Terms (n pixels.times.n pixels). 
The Correction Terms are accumulated over many frames and stored with eight 
times the precision of the image. Each Correction Term is divided by eight 
and then added to the input image pixel (row,col) to yield a new pixel 
(row,col) in the output image. This is illustrated diagrammatically in 
FIG. 10, where the updated set of Correction Terms (n values.times.n 
values) is divided by eight, and added to the input image (n 
pixels.times.n pixels) to yield the corrected output image. 
FIG. 11 illustrates an alternate embodiment of the invention. An alternate 
NUC processor 20' applies the correction terms to the input image, and 
employs a NUC-II filter system 24' in the calculation of the Correction 
Terms. As in the embodiment of FIG. 3, the input image 18 (focused or 
blurred) is received from the sensor 10. The Correction Terms are applied 
at 22 to the input image to yield a Corrected output image 30'. The 
Correction Terms are initially zero, and are applied in same manner as 
described above regarding the processor 20. The Corrected output image is 
passed to both the Tracking system and the NUC-II filter system 24'. A 
different NUC-II filter is used for focused and blurred images, as will be 
described in further detail below. The NUC filter system 24' updates the 
Correction Terms based on the Corrected output image 30'. 
FIG. 12 illustrates the focused image processing for the filter system 24' 
of FIG. 11. The input focused image 18A is received from the sensor, and 
the correction terms 26' are applied at step 22 to yield the corrected 
focused image 30', as described above. The filter system 24' includes a 
focused image filter 24A', which includes the application of an anti-mean 
filter 24C' at step 24A1' to the corrected focused image to yield an 
anti-mean image 30A'. The image 30A' is stored in memory for use on the 
following blurred image 18B. 
FIG. 13 is a flow diagram showing the blurred image processing for the 
filter system 24' of FIG. 11. The blurred image 18B is received from the 
sensor 10, and the correction terms 26' are applied at step 22 to provide 
the corrected blur image 30B', an output useful for diagnostics. The 
filter system 24' includes a blurred filter 24B' which includes the 
application of the same anti-mean filter 24C' used in the focused filter 
24A' to the corrected blurred image. As the anti-mean filter is applied to 
the corrected blurred image, the output is compared to the filter output 
stored from the previous focus frame via a NUC-II comparison function 
24B2', described more fully below. The output of the comparison function 
yields Error Terms which are used to update the Correction Terms for the 
next input focus image. 
FIGS. 14, 15A-15D and 16 illustrate the application of the anti-mean filter 
24C'. The input image 18 is n pixels by n pixels in size. For each pixel X 
in the image, the sum of X and its neighboring pixels indicated by the 
shaded regions in the filter mask is calculated. This sum is 
"Alpha-Trimmed" by subtracting the minimum and maximum pixel values 
contained within the filter mask. The mean is calculated by dividing the 
"Alpha-Trimmed" sum by number of pixels remaining in the sum. Filter mask 
pixels outside the image boundary are ignored in the calculation. This is 
illustrated by the location of the 13-pixel filter mask in FIG. 14, which 
illustrates the starting position for application to the n.times.n image. 
Because there are no values for seven of the mask pixels which are outside 
the image boundary, they must be ignored in the calculation of the mean. 
Thus, for this starting position of the filter mask, only six pixels of 
the filter mask cover the image, two are determined to be the minimum and 
maximum and are "Alpha-Trimmed," and the remaining four pixels are 
averaged together to determine a mean value. As the mask is passed across 
the image, all 13 mask pixels will lie within the image boundary and will 
be used in the calculation. Note that for the anti-median embodiment 
illustrated in FIG. 6, this boundary condition case is handled by having 
the extra Maximum (M) and minimum (m) values added around the outside 
boundary of the image. This cannot be done with the anti-mean filter 
because the Maximum (M) and minimum (m) values would bias the mean. 
The "anti-mean" is a measure of how the pixel differs from the mean. The 
"anti-mean" is calculated by subtracting the pixel "X" from the mean as in 
the following: 
EQU anti.sub.-- mean(row,col)=mean(row,col)--pixel(row,col) 
There are two filter patterns used for the NUC-II filter system 24', and 
are shown in FIGS. 15A and 15B. The shaded areas represent pixels that are 
used in the mean calculation, with the center pixel marked as an "X". FIG. 
16 shows the cycling of the filter patterns for each successive pair of 
focus/blur frames. 
The comparison function 242B' includes the following steps. Each value 
(row,col) of the anti-mean image (focus) is compared to the corresponding 
value of the anti-mean image (blur) as follows: Test 1: check whether the 
absolute value of the difference between the anti-mean focus and the 
anti-mean blur is less than or equal to 5. Test 2: determine if the 
absolute value of the anti-mean focus is less than the absolute value of 
the anti-mean blur. If Test 1 and Test 2 pass, then the error term is set 
to the anti-mean focus. If Test 1 passes but Test 2 fails, then the error 
term is set equal to the anti-mean blur. If Test fails then the error term 
is set to zero and Test 2 is irrelevant. 
The Error Terms are used to update the Correction Terms 26'. FIG. 17 
diagrammatically illustrates the updating of the Correction Terms. Each 
error term (row,-col) is multiplied by a factor (F) based upon its 
magnitude and then added to the Correction Term (row,col) to yield updated 
Correction Terms. 
The factor (F) is calculated in the following manner. The absolute value of 
the Error Term is used to determine the factor (F) by which to multiply 
the Error Term. Say, for example, that the Error Terms have an absolute 
value range of zero (0) through 100. If the absolute value of the Error 
Term is 0 or 1, the factor (F) is set to 1. If the absolute value of the 
Error Term is 2 through 7, the factor (F) is set to 2. If the absolute 
value of the Error Term is 8 through 33, the factor (F) is set to 4. If 
the absolute value of the Error Term is 34 and greater, the factor (F) is 
set to 8. Because the Correction Terms are of higher precision than the 
image, they are divided by eight before they are added to the image. 
Therefore a factor (F) multiplier of 1 yields a correction of 1/8 of the 
error, 2 is 2/8 or 1/44 is 4/8 or 1/2, and 8 is 8/8 or 1 which is a full 
correction. 
In accordance with an aspect of the invention embodied in the system of 
FIG. 11, FPN is removed from focal plane array imagery by comparing the 
output of an anti-mean filter that is applied to both focused and 
optically blurred images. The comparison is done in such a way that scene 
artifacts are not created and subtle scene content is not destroyed. 
Because an anti-mean filter is used, FPN is removed even in the presence 
of shading. Shading can be caused by optics, blue-sky, and/or dome 
heating. The embodiment of FIG. 3 effectively "shuts off" in the presence 
of shading due to the use of an anti-median filter which preserves edges 
within the shading gradients. 
It is understood that the above-described embodiments are merely 
illustrative of the possible specific embodiments which may represent 
principles of the present invention. Other arrangements may readily be 
devised in accordance with these principles by those skilled in the art 
without departing from the scope and spirit of the invention.