Abstract:
A fusion night vision system corrects for parallax error by comparing an image from a first sensor with an image from a second sensor.

Description:
CROSS REFERENCE TO RELATED APPLICATIONS 
     The present application claims the benefit of U.S. provisional patent application Ser. No. 60/909,779, filed Apr. 3, 2007 the entire disclosure of which is incorporated herein by reference. 
    
    
     BACKGROUND OF THE INVENTION 
     Night vision systems include image intensification, thermal imaging, and fusion monoculars, binoculars, and goggles, whether hand-held, weapon mounted, or helmet mounted. Standard night vision systems are typically equipped with one or more image intensifier tubes to allow an operator to see visible wavelengths of radiation (approximately 400 nm to approximately 900 nm). They work by collecting the tiny amounts of light, including the lower portion of the infrared light spectrum, that are present but may be imperceptible to our eyes, and amplifying it to the point that an operator can easily observe the image through an eyepiece. These devices have been used by soldier and law enforcement personnel to see in low light conditions, for example at night or in caves and darkened buildings. A drawback to night vision goggles is that they cannot see through smoke and heavy sand storms and cannot see a person hidden under camouflage. 
     Infrared thermal imagers allow an operator to see people and objects because they emit thermal energy. These devices operate by capturing the upper portion of the infrared light spectrum, which is emitted as heat by objects instead of simply reflected as light. Hotter objects, such as warm bodies, emit more of this wavelength than cooler objects like trees or buildings. Since the primary source of infrared radiation is heat or thermal radiation, any object that has a temperature radiates in the infrared. One advantage of infrared sensors is that they are less attenuated by smoke and dust and a drawback is that they typically do not have sufficient resolution and sensitivity to provide acceptable imagery of the scene. 
     Fusion systems have been developed that combine image intensification with infrared sensing. The image intensification information and the infrared information are fused together to provide a fused image that provides benefits over just image intensification or just infrared sensing. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       For a better understanding of the invention, together with other objects, features and advantages, reference should be made to the following detailed description which should be read in conjunction with the following figures wherein like numerals represent like parts: 
         FIG. 1  is an isometric view of a fusion night vision system consistent with an embodiment of the invention. 
         FIG. 2  is a block diagram of the fusion night vision system of  FIG. 1 . 
         FIG. 3  is a plot of misregistration of pixels due to parallax as a function of distance to target for the fusion night vision system of  FIG. 1 . 
         FIG. 4  is a block diagram of a parallax correction circuit consistent with an embodiment of the invention. 
         FIG. 5  is a block diagram of a horizontal/vertical feature filter circuit consistent with an embodiment of the invention. 
         FIG. 6A  is a schematic of a mechanical range finder consistent with an embodiment of the invention. 
         FIG. 6B  is a switch state diagram for the range finder of  FIG. 6A  consistent with an embodiment of the invention. 
         FIG. 6C  is a coarse parallax correction look-up table consistent with an embodiment of the invention. 
         FIG. 7A  is an image of a scene from a first sensor of the fusion night vision system of  FIG. 1 . 
         FIG. 7B  is the output of the image of  FIG. 7A  after passing though a horizontal/vertical feature filter circuit consistent with an embodiment of the invention. 
         FIG. 7C  is the output of the image of  FIG. 7B  after passing though a binary filter circuit consistent with an embodiment of the invention. 
         FIG. 7D  is an image of a scene from a second sensor of the fusion night vision system of  FIG. 1 . 
         FIG. 7E  is the output of the image of  FIG. 7D  after passing though a horizontal/vertical feature filter circuit consistent with an embodiment of the invention. 
         FIG. 7F  is the output of the image of  FIG. 7E  after passing though a binary filter circuit consistent with an embodiment of the invention. 
         FIG. 7G  is a fused image viewable through an eyepiece of the fusion night vision system of  FIG. 1 . 
         FIG. 8  is a plot of number of matching points versus translation useful in the parallax correction circuit of  FIG. 4  consistent with an embodiment of the invention. 
         FIG. 9  is a fusion alignment flow chart consistent with an embodiment of the invention. 
     
    
    
     DETAILED DESCRIPTION 
       FIG. 1  is an isometric view and  FIG. 2  is a block diagram of a fusion night vision system  100  consistent with an embodiment of the invention. The night vision system electronics and optics may be housed in a housing  102  which may be mounted on a weapon  112  to aid in identifying a threat and aiming of the weapon. The night vision system  100  may have a first sensor  204  located behind first objective optics  104  and a second sensor  208  located behind second objective optics  106 . The first sensor  204  may be configured to image scene information in a first range of wavelengths (7,000 nm-14,000 nm) and the second sensor  208  may be configured to image scene information from a second range of wavelengths (400 nm to 900 nm). The first sensor  204  may be an uncooled microbolometer focal plane array sensitive to long wave infrared radiation and the second sensor may be a digital image intensification (DI 2 ) device such as the electron bombarded active pixel sensor (EBAPS) sensitive to shorter wavelength radiation. Each sensor  204 ,  208  may have a two-dimensional array of detector elements that is translated into electric impulses that are communicated to signal processing electronics  202 . The signal processing electronics  202  may then translate the electric impulses into data for a display  220  for viewing through an eyepiece optics  108 . Other sensor/detector technologies including cooled long wave or mid wave infrared focal plane array, digital image intensification tube, a charge-coupled device (CCD) or complementary metal oxide semiconductor (CMOS) imager, or short wave infrared InGaAs array may be used without departing from the invention. 
     The system may have one or more actuators  110  for controlling system operation. Although the objective optics are shown spaced in the horizontal direction, they may be spaced in the vertical direction or a combination of vertical and horizontal without departing from the invention. Although the fusion night vision device is shown as a monocular, it may be binocular or biocular without departing from the invention. Although the fusion night vision system is shown as being weapon mountable, it may be helmet mounted or handheld without departing from the invention. 
     The fusion night vision system  100  may have the optical axis OA 1  of the first objective optics  104  physically offset a fixed distance “D” from the optical axis OA 2  of the second objective optics  106 . The optical axes of the first and second objective optics  104 ,  106  are typically factory aligned such that the image from the first sensor is fused and is aligned with the image from the second sensor for viewing by an operator as a unified image in the display  220  through the eyepiece  108  when the object/target being viewed is at a predetermined distance, typically aligned at infinity. At distances different from the predetermined distance, parallax can cause a misalignment of the two images as viewed in the eyepiece. The parallax problem may exist if the objective optics  104 ,  106  are offset in the horizontal as well as the vertical directions. The eyepiece  108  may have one or more ocular lenses for magnifying and/or focusing the fused image. The display  220  may be a miniature flat panel display, for example an organic light emitting diode (OLED) microdisplay or a liquid crystal display (LCD). 
       FIG. 3  is a plot of misregistration of pixels due to parallax as a function of distance to target for the fusion night vision system  100 . The plot shows that for the system  100  with the optical axes OA 1 , OA 2  of objective optics  104 ,  106  spaced by 1.6″, the image from the first sensor will be misregistered with the image from the second sensor in the display  220  by more than ten (10) pixels at a distance to target of less than 25 meters. The system  100  may reduce the misregistration using a coarse correction based on the position of one or both of the movable focus rings  104 A,  106 A and a fine correction based upon an autoalignment circuit. The system  100  may shift the position of the image from each sensor an equal and opposite number of pixels prior to fusion and display in order to maintain boresight of the system  100  to the weapon  112 . Shifting one of the images one more, or one less, pixel than the other image prior to fusion and display may be done without departing from the invention. Alternatively, the system  100  may reduce misregistration using just the autoalignment circuit, i.e without the coarse correction. 
     Scene information from the first sensor  204  may enter an analog to digital conversion circuit  206  before entering signal processing electronics  202  and scene information from the second sensor  208  may enter the signal processing electronics  202  in digital form directly. Batteries  214  may provide power to a low voltage power supply and controller  212  which provides conditioned power distributed throughout the system. The signal processing electronics  202  may include a digital processing circuit  202 A, a display formatter circuit  202 B, and a parallax correction circuit  202 C. 
       FIG. 4  is a block diagram of the parallax correction circuit  202 C consistent with an embodiment of the invention. The parallax correction circuit  202 C may be part of the digital processing circuit  202 A. The first sensor  204  may be a microbolometer array with 320×240 pixels and generate an output  402  and the second sensor  208  may be an EBAPS device with 1280×1024 pixels and generate an output  404 . An interpolation circuit  406  may scale/interpolate the output from the first sensor  204  so it is approximately the same size in number of pixels as the output from the second sensor  208 . The output from the interpolation circuit  406  and the second sensor  208  may be directed to an alignment filter  408  and respective first and second pixel shift circuits  412 ,  414 . The alignment filter  408  may first pass the scene information from the first sensor  204  and the second sensor  208  through a horizontal/vertical feature filter circuit  408 A to define edges in the scene information and then through a second filter  408 B that converts the output of the horizontal/vertical feature filter circuit  408 A to a binary output. Edges may be defined as pixel intensity discontinuities or localized intensity gradients within an image. Edges may help characterize an object boundary and therefore may be useful for detection of objects in a scene. Known edge detection circuits are disclosed in Fundamentals of Digital Image Processing authored by Anil K. Jain and published by Prentice-Hall, Inc., and are incorporated herein by reference in their entirety. For each image originating from the first sensor  204  and the second sensor  208 , the horizontal/vertical feature filter circuit  408 A may produce an image of edges and the second filter  408 B may convert the image into a binary map of edges with each pixel location assigned a value of one (1) or zero (0) based upon the edge intensity. Any pixel at the output of the horizontal/vertical feature filter circuit  408 A with pixel value equal to or above a predetermined positive threshold (for example +16 with an 8-bit bipolar image) or equal to or below a predetermined negative threshold (for example −16 with an 8-bit bipolar image) may be assigned to a value of one (1) by the binary filter  408 B. Any pixel at the output of the horizontal/vertical feature filter circuit  408 A with pixel value between the predetermined positive and negative threshold may be assigned the value of zero (0) by the binary filter  408 B. The binary edge map of 1&#39;s and 0&#39;s produced at the binary filter output for each image originating from the first sensor  204  and the second sensor  208  is temporarily stored within the buffer memory  430  accessible by the alignment correlation circuit  410 . 
     The alignment correlation circuit  410  accesses the binary edge maps held within the buffer memory  430  and may use a coarse alignment estimate based on “rough” distance to target information to speed up processing. The “rough” distance to target information may be provided by a mechanical range finder  432  or other means including input by the user. 
       FIG. 5  is a block diagram of the horizontal/vertical feature filter circuit  408 A consistent with the invention. The horizontal/vertical feature filter circuit  408 A may include a multi-row, multi-column buffer  520  and a multi-row, multi-column convolver  522  and a multi-row, multi-column convolution kernel  524 . Other values in the convolution kernel  524  may be used without departing from the invention. Although the convolver is shown as being 5×5, other sized convolver with extent smaller or larger may be used without departing from the invention, for example a 3×3, 7×7 or 9×9 convolver may be used. Although the convolution kernel  524  as shown is designed to locate vertical features within an image (as may be used for systems with horizontal placement of the objective optics  104 ,  106 ) it should be readily apparent that by transposing the kernel 90° the filter will perform equally well in locating horizontal image features (as may be used for systems with vertical placement of the objective optics  104 ,  106 ) without departing from the invention. 
     A mechanical range finder may require the operator to focus one of the focus rings  104 A,  106 A on a target and a linear or rotational position sensor could be used to determine the distance to target (see  FIGS. 6A-6C  and discussed in further detail below). Alternatively, a mechanical circuit may include a linear or rotary potentiometer mechanically coupled to one of the focus rings  104 A,  106 A. In an alternative embodiment, the system may accept inputs from a user regarding the distance to target. The input may be received through a near/far actuator or a menu selection. The system may be designed so the operator selects the far mode when the object being viewed is greater than 100 meters away and the operator selects the near mode when the object being viewed is less than 100 meters away. Distances other than 100 meters may be chosen without departing from the invention. The fusion night vision system may also incorporate multiple distance choices, for example close, less than 25 meters; mid range, 25-50 meters; long range, 50-100 meters; real long range, greater than 100 meters without departing from the invention. 
     The alignment correlation circuit  410  may receive the distance to target information from the mechanical range finder  432  to establish a coarse estimate of the amount of image shift required. Using the coarse estimate as an initial starting point the alignment correlation circuit  410  may then compare the binary maps held within buffer memory  430 , corresponding to the location of object edges in the images produced by the first sensor  204  and the second sensor  208 , to determine a fine alignment estimate. The alignment correlation circuit  410  may start the comparison of the binary edge maps shifted relative to each other based on the coarse estimate and then translate them left or right until a best match is found. A best match may be found when the number of matching pixels within the edge maps is highest, i.e. the peak of the correlation function, for the pixel shift attempted. For each increment in image shift, the alignment correlation circuit  410  may calculate the corresponding value of the correlation function by summing over all pixel locations the result of performing a pixel-by-pixel logical “and” operation between the two binary maps under evaluation with the net result being a count of all pixel locations for which a value of one (1) is present in both binary edge maps. Alternatively, the alignment correlation circuit  410  may use only a central portion of the binary maps without departing from the invention. 
     For the example shown in  FIG. 8 , the alignment correlation circuit has determined that the image from the first sensor should be shifted left (minus) three (3) pixels relative to the image from the second sensor as indicated by the peak in the correlation function as plotted. It should be readily apparent to those skilled in the art that a correlation circuit based upon a logical “nor” operation would produce equally valid results without departing from the invention if the use of 0&#39;s and 1&#39;s within the binary edge maps were interchanged. 
     The output of the alignment correlation circuit  410  may instruct the first pixel shift circuit  412  to shift the interpolated output from the first sensor  204  to the left one (1) pixel and the second pixel shift circuit  414  to shift the output from the second sensor  208  to the right two (2) pixels where the sum total shift in pixels is equal to the sum of the coarse and fine alignment estimates. The outputs of the pixel shift circuits  412 ,  414  may be inputted into an image combiner  416  to combine the two images into a single fused image outputted to the display formatter  202 B. An image combiner  416  may be a circuit that adds or fuses, in analog or digital form, the image from the first sensor  204  with the image from the second sensor  208  or may employ a more complex circuit designed for image fusion such as the Acadia processor from Pyramid Vision. 
       FIG. 6A  is a schematic of a mechanical range finder,  FIG. 6B  is a switch state diagram, and  FIG. 6C  is a look up table consistent with the invention. Sensors SW 1 , SW 2 , for example Hall effect switches, may be located in the housing  102  adjacent a rotatable or translatable focus ring  104 A,  106 A that surrounds the objective optics  104 ,  106 . The user can rotate or translate the focus ring  104 A,  106 A clockwise or counter-clockwise from near N to far F as the user attempts to focus on a target. As the focus ring  104 A,  106 A is rotated, the state of the sensors SW 1 , SW 2  may be read by a processor  620 . The processor  620  may be part of the low voltage power supply and controller  212  or may be part of the signal processing electronics  202 . A series of magnets  652  in close proximity, or a single arcuate magnet, may be coupled to the focus ring  104 A,  106 A in an arcuate path. The magnets  652  may be located in holes formed in the focus ring  104 A,  106 A. The location and spacing of the sensors relative to the magnets may depend on the angular rotation of the focus ring  104 A,  106 A from near N to far F. The location of the sensors SW 1 , SW 2  and the magnet(s)  652  may also be swapped without departing from the invention. 
       FIG. 6C  is a coarse parallax correction look-up table consistent with the invention. As the distance to target changes, the processor  620  may provide the corresponding estimate for coarse alignment in pixels of image shift to the alignment correlation circuit  410 . For example, when a target is approximately 75 meters away, the range finder  432  may send a signal to the alignment correlation circuit  410  that the images need to be shifted by a nominal amount of five (5) pixels in total. 
       FIG. 7A  is an image of a scene from the first sensor of the fusion night vision system  100 ;  FIG. 7D  is an image of the same scene from the second sensor of the fusion night vision system  100 ; and  FIG. 7G  is a fused/combined image viewable through the eyepiece  108  of the fusion night vision system  100 . The fused image provides more scene information than either of the individual images. In  FIG. 7D , an antenna  702  is viewable with the second sensor  208  but not with the first sensor  204  and in the  FIG. 7A  the doorway of the trailer is viewable with the first sensor  204  but not the second sensor  208 .  FIG. 7G  shows the fused image in which the antenna  702  and the trailer doorway  704  are viewable.  FIG. 7B  shows the output of the horizontal/vertical feature filter circuit  408 A and  FIG. 7C  shows the output of the binary filter  408 B for the first sensor  204 .  FIG. 7E  shows the output of the horizontal/vertical feature filter circuit  408 A and  FIG. 7F  shows the output of the binary filter  408 B for the second sensor  208 . 
     Certain embodiments of the invention can be implemented in hardware, software, firmware, or a combination thereof. In one embodiment, the parallax correction circuit is implemented in software or firmware that is stored in a memory and that is executable by a suitable instruction execution system. If implemented in hardware, as in an alternative embodiment, the circuits can be implemented with any or a combination of the following technologies, which are well known in the art: a discrete logic circuit(s) having logic gates for implementing logic functions upon data signals, an application specific integrated circuit (ASIC) having appropriate combinational logic gates, a programmable logic device (PLD), a field programmable gate array (FPGA), etc. 
       FIG. 9  is a fusion alignment flow chart consistent with an embodiment of the invention. The first sensor acquires data representative of a scene at block  902  and the second sensor acquires data representative of the same scene at block  912 . The output of the first sensor is received by the first shift pixel circuit at block  920  and the horizontal/vertical feature filter circuit at block  904  and output of the second sensor is received by the second shift pixel circuit at block  922  and the horizontal/vertical feature filter circuit at block  914 . The horizontal/vertical feature filter circuit filters the data from each sensor to define edges and then outputs the data from the first sensor to the binary filter at block  906  and the data from the second sensor to the binary filter at block  916 . The binary filter converts each pixel to one of two unique values (e.g. 0 and 1). The outputs of the binary filter circuits (binary edge maps) are then inputted to the buffer memory at block  908 . The alignment correlation circuit at block  909  may then start the comparison by translating the binary edge maps temporarily stored within the buffer memory and corresponding to images from the first and the second sensors left and right about the coarse align position until a best match is found. The alignment correlation circuit then instructs the pixel shift circuits how many pixels to move the image and in which direction at block  920 ,  922 . After the first and second images are shifted (if necessary), the images are combined in a combiner at block  910  and outputted to the display formatter. 
     Although several embodiments of the invention have been described in detail herein, the invention is not limited hereto. It will be appreciated by those having ordinary skill in the art that various modifications can be made without materially departing from the novel and advantageous teachings of the invention. Accordingly, the embodiments disclosed herein are by way of example. It is to be understood that the scope of the invention is not to be limited thereby.