NASA CASE NO. LAR 14779-1 

PRINT FIG. 2 r' ii 

NOTICE 

The invention disclosed in this document resulted from research in aeronautical and 

space activities performed under programs of the National Aeronautics and Space 

Administration. The invention is owned by NASA and is, therefore, available for licensing 

in accordance with the NASA Patent Licensing Regulation (14 Code of Federal 

Regulations 1245.2). 

To encourage commercial utilization of NASA-owned inventions, it is NASA policy to grant 

licenses to commercial concerns. Although NASA encourages nonexclusive licensing to 

promote competition and achieve the widest possible utilization, NASA will provide the 

necessary incentive to the licensee to achieve early practical application of the invention. 

Address inquiries and all applications for license for this invention to NASA Patent 

Counsel, Langley Research Center, Code 143, Hampton, Virginia 23665-5225. 

Serial No.: 071885,714 

05-1 4-92 LaRC 

(NASA-Z~SP-LAR-~~~~W) WLTIRFSPOYSE N92-29951 

LVAGtR AtdP TMAGINC PHOCCSS FOR IMPROVED 

RFSf3LUTTJN Pdtrnt Aopl icdtion (NASA) 16 p 

Uncl ds 

G3/74 0111105 
NASA Case No. LAR 14779-1 

AWARDS ABSTRACT 

MULTIRESPONSE IMAGER AND IMAGING PROCESS 

FOR IMPROVED RESOLUTION 

image systems including television cameras, charge coupled device imagers, 

forward locking infrared sensors, and infrared detectors produce a video image that has 

a resolution limited by the sampling interval, i.e., the space between photodetectors, of 

the imager. Attempts to produce images of objects requiring a resolution greater than this 

sampling interval result in poor visual quality, aliasing, blurring, colored noise and ringing. 

Multiple representations to overcome this resolution limitation are often precluded since 

often only a single incident is available. 

The implementation of this invention consists of two elements, the image-gathering 

device and an image-restoration filter algorithm. The image-gathering device is 

implemented with an optical aperture control that allows one to change the spatialfrequency response of the image-gathering device for each one of the A successive 

image acquisitions, and the image-restoration algorithm unscrambles the aliased and 

blurred signal components in the presence of photosensor noise to produce an image 

with an increased resolution. The upper limit of the increase in resolution is 1Im times 

the sampling interval of the image-gathering device. The image-gathering device and the 

target should be stationary with respect to each other during the image-gathering process. 

Novel aspects of the invention include assigning successive image acquisitons, 

unique spatial-frequency responses and then employing an image-restoration algorithm 

to unscramble the image based on these unique responses. 

Inventors: Carl E. Fales Fredrich 0. Huck 

Address: 18 S. Madison Lane 83 Linda Drive 

Newport News VA 23606 Newport News, VA 23602 

SS: SS: 

Employer: NASA LaRC 

Initial Evaluator: Harry F. Benz 

Serial No.: 071885,714 

Filed: May 14, 1992 
LAR 1 4779- 1 -1 - PATENT APPLICATION 

MULTIRESPONSE IMAGE GATHERING AND PROCESSING 

FOR IMPROVED RESOLUTION 

Orinin of the Invention 

The invention described herein was made by employees of the United States 

Government and may be manufactured and used by or for the Government for 

governmental purposes without the payment of any royalties thereon or therefor. 

Background of the Invention 

- 1. Technical Field 

The present invention relates generally to image gathering and restoration. 

Multiresponse image gathering and digital restoration are combined for improved 

resolution. 

- 2. Discussion of the Related Art 

Image systems including television cameras, charge coupled device imagers, 

forward looking infrared sensors, and infrared detectors produce a video image that 

has a resolution limited to the sampling interval (space between photodetectors) of 

the imager. Hitherto, it has never been possible to produce images of the random 

objects commonly encountered in nature with a resolution that is finer than the 

sampling interval or, stated differently, with a spatial frequency that is higher than 

the sampling passband, or Nyquist frequency, of the imager. 

The performance of conventional imaging is, in general, critically constrained by its sampling passband, spatial-frequency response, and sensitivity. 

These constraints limit the resolution of the restored images to the sampling interval, and further degrade their visual quality by blurring, aliasing artifacts, colored 

noise, and ringing. In some applications, e.g., planetary exploration, military reconnaissance or medical diagnosis, image gathering may be limited to a single 

costly event. Accordingly, the image gathering constraints need to be rigorously 
LAR 1 4779-1 PATENT APPLICATION 

accounted for without undue regard given to the complexity of processing required 

to do so. 

Obiects of the Invention 

It is accordingly an object of the present invention to improve the spatial 

resolution of digitally restored images that can be attained by image gathering 

and digital restoration. The improvement is attained by unscrambling, within the 

presence of photosensor noise, the within sampling-passband and aliased signal 

components of several images, each acquired with a different spatial-frequency 

10 response. 

Summary of the Invention 

In this invention, the resolution is improved to be finer than the sampling 

interval of the imager or, stated differently, the spatial frequency is improved to be 

higher than the sampling passband of the imager. The imaging process consists of 

two steps. The first step consists of acquiring A images with the image-gathering 

device. Each one of the A images is obtained with a different spatial-frequency 

response relative to the sampling passband. Two conditions must be fulfilled during 

this step. One condition is that the image-gathering device and the target are 

20 stationary with respect to each other during the image-gathering process. The 

other condition is that the A different spatial-frequency responses extend beyond 

the sampling passband. It is a natural consequence of this image-gathering process 

that sufficiently sampled, i.e., within sampling-passband, signal components and 

insufficiently sampled, i.e., aliased, signal components are weighted differently and 

predictably. 

25 The second step is to unscramble the within sampling-passband and aliased 

signal components and to restore these components as an image. The unscrambling 

and restoration is done in a single process. The Wiener-matrix filter (described 

below) restores an image with a restoration that is finer than the sampling interval 

by a factor of up to 1/a or, stated differently, it restores an image with a spatial 
LAR 14779-1 -3- PATENT APPLICATION 

frequency that is higher than the sampling passband by a factor of up to 42. The 

Wiener-matrix filter accounts for the aliasing, blurring and noise of the imagegathering process to restore an image with the minimum possible mean square 

5 error between the restored image and the actual target. 

Brief Descrivtion of the Drawings 

FIG. l(a) is a representation of a two-dimensional narrow passband Wiener 

spectrum; 

10 FIG. l(b) is a representation of a two-dimensional, circularly symmetric and 

monotonically decreasing Wiener spectrum; 

FIG. 2 is a schematic representation of the multi-response imaging process 

according to the present invention; 

FIG. 3 is a representation of a multiresponse undersampled signal spectrum; 

FIGS. 4(a), 4(b) and 4(c) graph typical spatial frequency response relative 

to the sampling passband when A = 1, 4, and 9, respectively; 

FIGS. 5(a), 5(b) and 5(c) graph typical through put responses relative to 

the sampling passband when A = 1, 4, and 9, respectively, with an SNR = 256; 

FIG. 6(a) is a random polygon target with a mean spatial detail of p = 0.75; 

FIGS. 6(b), 6(c) and 6(d) are images of the target of FIG. 6(a) for A = 1, 

20 4, and 9 with SNR = 256; 

FIG. 7(a) is a resolution wedge target; and 

FIGS. 7(b), 7(c), and 7(d) are images of the target of FIG. 7(a) for A = 1, 

4, and 9 with SNR = 256. 

Detailed Descrivtion of the Invention 

25 

FIG. l(a) and (b) illustrate two examples of undersampling. Here (X, Y) are 

the rectangular lattice spacings, and (v, w) are spatial frequencies. The sampling 

passband is the dashed rectangular region bounded by the Nyquist frequencies, 

i.e., Ivl 5 1/2X, lwl 5 1/2Y. The solid oval boundary describes the extent 

of the signal Wiener spectrum, and the dashed region represents the portion of 

30 
LAR 1 4779- 1 PATENT APPLICATION 

this spectrum that has been aliased into the passband. Consider first the narrow 

bandpassed spectrum shown in FIG. l(a). There the aliased portion intuitively 

represents useful information: for, by a priori knowledge suggesting careful imaging 

and linear filtering, this part of the signal can be placed back to its original 

5 position and added to the portion within the passband to recapture the original 

signal, to the extent that the additive noise allows. This process places both the 

within-passband and the aliased regions of the signal on equal footing as useful 

information. Formulations that treat the aliased signal as noise would not capture 

the essence of this Wiener spectrum. Consider next the circularly symmetric 

and monotonically decreasing Wiener spectrum shown in FIG. l(b). There the 

10 overlapping portions of the within-passband and aliased signals interfere with each 

other. If the within-passband and aliased signals cannot be unscrambled, then the 

aliased portion intuitively represents additive noise, i.e., information that is not 

only lost forever but also contaminates the within-passband signal, consistent with 

formulations that treat the aliased signal as noise. 

15 However, this invention presents a method for unscrambling the withinpassband and aliased signal components and restoring an image with a resolution 

that is finer than the sampling interval of the imager or, stated differently, with a 

spatial frequency that is higher than the sampling passband of the imager. The 

imaging process consists of two steps. The first step consists of acquiring A images 

with the image-gathering device. Each one of the A images is obtained with a dif20 ferent spatial-frequency response, or modulation transfer function, relative to the 

sampling passband. Two conditions must be fulfilled during this step. One condition is that the image-gathering device and the target are stationary with respect 

to each other during the image-gathering process. The other condition is that the 

A different spatial-frequency responses extend beyond the sampling passband. It 

25 is a natural consequence of this image-gathering process that sufficiently sampled, 

i.e., within sampling-passband, signal components and insufficiently sampled, i.e., 

aliased, signal components are weighted differently and predictably. 

The second step is to unscramble the within sampling-passband and aliased 

signal components and to restore these components as an image. The unscrambling 

and restoration is done in single process. The Wiener-matrix filter (described 
LAR 1 4779- 1 PATENT APPLICATION 

below) restores an image with a restoration that is finer than the sampling interval 

by a factor of up to 1/a or, stated differently, it restores an image with a spatial 

frequency that is higher than the sampling passband by a factor of up to dx. The 

Wiener-matrix filter accounts for the aliasing, blurring and noise of the imagegathering process to restore an image with the minimum possible mean square 

error between the restored image and the actual target. 

FIG. 2 is a simplified schematic diagram of the multiresponse imagegathering and Wiener-matrix restoration process. Thus, as illustrated in FIG. 3, 

each signal s,(x, y) is sampled at exactly the same location (x, y) but is weighted 

differently at each spatial frequency (v, w). The different spatial-frequency responses for obtaining each of the A images is obtained by changing the size of an 

adjustable, varying optical aperture 10 of a single image gathering device 12 for 

each acquired image. The scene and the imager must be stationary with respect 

to one another. 

The image-gathering process transforms the continuous radiance field 

L(x, y) reflected or emitted by the target into d discrete sample signals, each 

acquired by a different optical response T,(x, y) on the same sampling lattice 111. 

- 

The optical response ra(x, y) is the convolution of the objective lens point spread 

function and the photosensor point spread function for the a'th optical aperture 

opening. 

Each acquired signal s,(x, y), a = 1,2.. ., A is given by 

where each successive signal s,(x, y), a = 1,2, .. . , A, is acquired with a different op 

tical response r,(x, Y), Ka is the steady-state gain of the linear radiance-to-signal 

conversion, na(x, y) is the additive discrete photodetector noise, and 1) 1 - I I I (x, y) - - 

is the uniform rectangular sampling lattice with a unit sampling interval in each 

dimension. Thus, each signal sa(x, y) is sampled at exactly the same location 

(2, y) but is weighted differently at each spatial frequency (v, w). As stated above 
LAR 1 4779- 1 -6- PATENT APPLICATION 

this process can be implemented by widening the objective lens aperture (i.e., 

increasing its diffraction limit) for each successive image. The Fourier transform 

of this process is 

or, more explicitly, 

(14 

where A equals the number of sidebands that the process restores. The associated 

h 

restoration passband for IBsI = 1 increases to 

h 

with area lBAl = Al&l = A. 

FIGS. 4(a), 4(b) and 4(c) depict typical spatial-frequency responses +, (v, w) 

6 

25 relative to sampling passband Bs in one dimension for convenience, wherein A is 

respectively equal to 1, 4 and 9. 

The image-restoration process restores a continuous image from the A 

discrete signals s,(x, y). The Fourier transform of the image is given by the 

expression 
LAR 14779-1 -7- PATENT APPLICATION 

The Wiener-matrix filter $a(v, w) which minimizes the mean square difference 

between L(z, y) and RA(z, y) is 

where 

For white noise with variance rrCa 

10 

[F(v,w)lap = [~;(V,W) %(v,w) +;(v,w)] *fi + (~a~Ll~~~)-~ 6(a,8)- 

Dropping for convenience the spatial-frequency dependence (v, w), the matrix 

. 

[T(v, w)] is given more explicitly by 

The corresponding minimum mean square restoration error power spectral density 

25 where 

A 

G(V, W) = C ia(v,w)8a(vlw) 

a=l 

is the throughput response for multiresponse imaging. 

FIGS. 5(a), 5(b) and 5(c) depict typical throughput responses F(v, w) where 

30 A = 1, 4 and 9, respectively, with SNR = 256. As shown, the throughput responses 

improve relative to the sampling passband as A increases. 
LAR 14779-1 -8- PATENT APPLICATION 

FIGS. 6(a) and ?(a) illustrate two targets - a random multiple polygon 

target and a resolution wedge target. The mean distance p between the edges of 

the polygons is equal to 0.75 relative to the unit sampling interval of the imager. 

FIGS. 6(b) and 7(b) illustrate images restored for a single (conventional) 

5 image-gathering event; FIGS. 6(c) and ?(c) illustrate images restored for A = 4 

image-gathering events; and FIGS. 6(d) and 7(d) illustrate images stored for A = 9 

image-gathering processes. The improvement in resolution shown in FIGS. 6(c), 

7(c), 6(d) and 7(d) over that shown in FIGS. 6(b) and 7(b) is a direct consequence 

10 of this invention. 

Many improvements, modifications and substitutions will be apparent to the 

skilled artisan without departing from the spirit and scope of the present invention 

as described herein and defined in the following claims. 
LAR 14779-1 

Y 

9' PATENT APPLICATION 

MULTIRESPONSE IMAGE GATHERING AND PROCESSING 

FOR IMPROVED RESOLUTION 

Abstract of the Disclosure 

The implementation of this invention consists of two elements, the imagegathering device and an image-restoration filter algorithm. The image- gathering 

device is implemented with an optical aperture control that allows one to change 

10 the spatial-frequency response of the image-gathering device for each one of the 

successive image acquisitions, and the image-restoration algorithm unscrambles 

the within-passband and aliased signal components in the presence of photosensor 

noise to produce an image with an increased resolution. The upper limit of the 

15 increase in resolution is 1/a times the sampling interval of the image-gathering 

device for A image-gathering events. The image-gathering device and the target 

must be stationary with respect to each other during the image-gathering process. 
FIG. I (a) FIG. I(b) 
------------- MULTIRESPONSE l MMlNG 1- - - 

IMAGE RESTORATION 

I r----------- 1 

FIG. 2 

IDISCR~ - ' RESTORED I 

_, 

IMAGE I IMAGE 'SIGNALSH . ! 4 i sa &,Y) I ! 

L -----a,,------ n 4 (xd)~ L - - - - - , -- - - J 

CONTINUOUS-+-'DI SCRETE.-+ONTINUOUS - 

WIENERMATRIX 

A FILTJR 

yra 13, 
FIG. 3 
FIG. 4 (c 

FIG. 5(a) FIG. 5(b) FIG. S(c) 
FIG, 6(a) FIG. 6( b) 

FIG. 6(c) FIG. 6 (d) 
FIG. 7(a) FIG. 7(b) 

FIG. 7(c ) FIG. 7(d )