Abstract:
Methods and systems for providing an aligned bank of images (e.g. for a panoramic display obtained by a scanning or moving camera) are disclosed that allow updating of the bank of images as new images are obtained; display of the original images stored in the bank as a mosaic according to any desired zoom; determination and display of the best mosaic view for a desired zoom; recall of the original images from the bank; and production of a mosaic that is free of accumulated stitching errors without imposing limitations on the way the camera scans. The methods and systems herein disclosed further have the ability to anchor the bank of images to the actual ground coordinates. The key to the method is inclusion of an iterative self-correcting procedure that reduces the accumulated errors as new images are added to the bank.

Description:
FIELD OF THE INVENTION 
       [0001]    This invention generally relates to methods for creating properly aligned image mosaics, and more specifically to methods and systems for creating such mosaics and using them for ground coordinate acquisition of any point on an image and for object detection in the image. 
       BACKGROUND OF THE INVENTION 
       [0002]    For many applications involving the use of a camera, there is a need for a tool to inform the user of the camera&#39;s orientation. An important example of a case where such a tool is needed is when a camera is used to create a mosaic of images covering a wide area where the ground coordinates of every point on the mosaic must be acquired quickly and accurately. 
         [0003]    Many applications exist for building panoramic images. Some applications work offline, that is, the mosaic is created after all of the images are taken. This type of method is not always applicable, however, as the mosaic frequently must be created on-line from a live video source, and needs to be updated in real time as the video camera moves. In an online application of this type, the video can theoretically continue forever. 
         [0004]    Some applications do allow online creation of a panoramic view from multiple individual photographs or video captures. These applications are usually based on image matching algorithms that compute a matching transformation between two images.  FIG. 7  illustrates schematically such a procedure, in which matching transformation T 1,2  computes the match between images I 1  and I 2  ( FIG. 7   a ), in particular point P 2  of image I 2  corresponding to point P 1  of image I 1  ( FIG. 7   b ). The accuracy of such algorithms is limited, and certain patterns of camera motion can lead to large errors in the image location as matching errors are accumulated.  FIG. 2  illustrates such a case. The accumulated matching error between the first image acquired ( 20 ) and the final image acquired ( 29 ) can be very large as the composite error is due to errors arising from 10 separate image matching operations. 
         [0005]    The possibilities of overcoming this problem are limited, as the mosaic created by the image matching procedure preserves neither the original images from the video nor their associated transformations. 
         [0006]    There is thus a long-felt need for a tool that has the following characteristics: (1) it is capable of storing all of the original images collected for the creation of the mosaic in a bank that allows an online mosaic display of these images, and which enables updating of the bank of images and the mosaic display as new images are obtained; (2) the original images stored in the bank can be displayed as a mosaic according to any desired zoom; (3) if many images cover the same area, the images with the proper zoom can be selected for inclusion in the mosaic display, thus allowing the best mosaic view according to the selected zoom; (4) it should always be possible to recall from the bank and to view the original images covering any part of the mosaic; and (5) the mosaic produced by the tool should be free of accumulated stitching errors, yet imposing no limitation on the way the camera is scanning (as in  FIG. 2 ). 
         [0007]    Furthermore, for maximum utility, any tool with the above characteristics needs to be able to anchor the images in the bank to the ground in order to provide a rapid and simple way to obtain the ground coordinates of any point in the displayed mosaic or in any of the original images from which the mosaic was formed. 
         [0008]    Specifically, when the source of images is a video stream, there is a special need to anchor the last video frame, so that the user can know where the camera is pointing at any given moment. The user can then retrieve the ground coordinates of any point of interest in the video frame. 
         [0009]    Several systems exist for ground anchoring of aerial photos. The common method is to allow the user to mark manually several pairs of “tie points.” The first point in the pair is a point on the photo; the second point is marked on a map or orthophoto. A global ground to image transformation is computed, and using this transformation, every ground point can be converted to image point and vice versa. Some systems can create the tie-points automatically. One common way to do it is to use an image to ground matching algorithm (IGA), usually based on matching the current photo to an already anchored photo, usually an orthophoto ( FIG. 9 ). However, good performance by IGAs is not easy to achieve. They frequently fail, especially when trying to match images taken from significantly different view angles, and are in general not sufficiently reliable to be able to provide acceptable results for video anchoring. 
         [0010]    Many systems for ground anchoring of aerial photos are generally adapted for offline work. After the photos are taken, the anchoring procedure can be performed either manually or automatically. These systems, however, are not built for anchoring a continuously growing bank of images or mosaic. Specifically, they cannot provide a means for automatically anchoring the last video frame. 
       SUMMARY OF THE INVENTION 
       [0011]    The system and method disclosed in this patent are specifically designed to overcome these difficulties. This system allows incremental buildup of an image bank in which the images are anchored to ground. The method further compensates for the poor performance of the image to ground matching algorithm (IGA) by using the information about connection between the images in the bank and by introducing a special iterative self-correction technique to reduce accumulated errors. 
         [0012]    The methods introduced in this patent are designed to provide a new process for creating an aligned bank of images via an iterative self-correction technique, for anchoring the bank to ground, for detecting objects in the images and for creating a mosaic display of the images in the bank collected up to the time of reaction of the mosaic. The source of the images is an optical imaging device (e.g. a camera or video camera) which scans in an unconstrained pattern the area surrounding it or part thereof (e.g. scanning a landscape). Each newly obtained image is stored in an image bank along with additional information about the image&#39;s orientation. The bank is continuously updated as new images are added. 
         [0013]    Two types of aligned bank of images are introduced. The first type (A-Bank) is adapted for the case of a camera at a fixed position that can rotate or change zoom. Each new image is matched to images already in the bank using an image matching algorithm (IM). The bank keeps the images together with their absolute transformation A, e.g. the transformation between each image and a predefined fixed image. Reference is now made to  FIG. 1 , in which such a procedure is illustrated schematically for a bank of images  100 . A series of absolute transformations  1001 ,  1002 , and  1003  transform predefined fixed image  10  to images  11 ,  12 , and  13 , respectively. When the newly obtained image overlaps more than one image, an iterative self-correction procedure introduced here is used to reduce the errors of the absolute transformation of some images already in the bank ( FIGS. 2 ,  3 ). This special technique ensures the low error rate of matching errors in the bank. 
         [0014]    At any time, a ground anchoring of A-BANK procedure can be performed either manually or automatically to anchor the A-BANK to the ground so that the ground coordinates of every image point can be easily computed. Images added to the bank after performing the anchoring procedure will be anchored as well. Specifically, the image most recently entered into the bank is anchored to the ground, allowing real-time online video-to-ground anchoring of the object(s) being viewed by the camera, e.g., a landscape. 
         [0015]    A second bank type, G-BANK, is used in the case of a moving camera. The bank retains the images along with their ground to image transformation G. This type of bank is illustrated schematically in  FIG. 4 , in which ground to image transformations  4000  and  4001  are associated with images provided from camera positions  40  and  41 , respectively. Each new image is transferred to an image to ground matching algorithm (IGA) and the ground to image transformation G is computed. If the procedure fails, G is approximated by matching the new image to other images already in the bank. With a G-BANK, there is no need to perform the type of anchoring procedure used with an A-BANK because the bank is created and incremented while the image is anchored to ground. Specifically, the ground to image transformation of the image most recently entered into the bank is computed, allowing real-time online video-to-ground anchoring of the object(s) being viewed by the camera, e.g., a landscape. 
         [0016]    When the newly obtained image is successfully matched to ground by the IGA algorithm or when it overlaps more than one image, an iterative self-correction procedure for the G-BANK is used to reduce the errors of the ground to image transformation of images already in the bank ( FIGS. 4 ,  5 ). This special technique ensures a low rate of matching errors. 
         [0017]    The advantage of use of a G-BANK is clear, as it obviates the need for performing an anchoring procedure. Nevertheless, the advantage of the A-BANK is that it can be created and incremented without using an IGA, an algorithm that normally requires use of a Digital Elevation Model (DEM) that may not always be available. If needed, the bank can be anchored to ground at any time. 
         [0018]    The aligned image bank described in this patent can be used for object detection in images obtained by a rotating or a moving camera. Object detection algorithms usually obtain a base image and an additional list of reference images overlapping the base image. The algorithm obtains the relative transformation between each reference image and the base image as well ( FIG. 13 ) and returns a list of detected objects locations on the base image, optionally, with additional information such as object size and velocity (in image coordinates). A special case of object detection is a moving object detection algorithm. As the original images are retained in the bank and the transformation of each image is known, an aligned image bank described herein allows detection of moving objects in a live video of a moving or rotating camera, including cases in which the zoom is changing. 
         [0019]    Furthermore, detected objects can be displayed on the mosaic view of the bank, and in the case of a G-BANK or an anchored A-BANK, ground coordinates of said objects can be computed and displayed on their appropriate location on a map. Ground anchoring can be further used to transform detected object size and velocity in image coordinates to the real velocity and size in ground coordinates. 
         [0020]    Another advantage of using the aligned image bank for object detection involves the use of regions of parameters for detection. Many object detection algorithms allow the definition of regions on the base image, each region with its appropriate set of parameters for detection. This way, one region can be defined for sensitive detection while other region can be totally excluded from detection ( FIG. 14 ). In the case of a fixed camera, the user can define the regions on the video image of the camera. However, when the camera is moving or rotating to cover a wide area, there is no simple way known in the prior art for a user to define the regions. Using the mosaic view of the aligned bank introduced herein provides a solution to the problem. The user can define the regions on the mosaic view of the bank. This is a simple and intuitive process for the user. The regions&#39; projections onto the base image for detection are then computed and transferred to the object detection algorithm. 
         [0021]    A further advantage is achieved in the case of a G-BANK or an anchored A-BANK. It is possible to define said regions on a map with ground coordinates rather than defining them on the mosaic view, giving the user a better alternative to define the regions. The regions&#39; projections onto the base image for detection are then computed and transferred to the object detection algorithm. 
         [0022]    It is therefore an object of the invention disclosed herein to provide a method for creating an aligned bank of images from an image sequence source comprising the steps of: (a) obtaining an image sequence source (ISS) in which images are captured from a fixed location; (b) creating a bank adapted for (i) storage of images; and, (ii) storage of additional data for every image, said additional data comprising at least (i) an absolute transformation adapted to transform a point in said image to a corresponding location on a predefined fixed reference image; and (ii) the quality value of said transformation; (c) capturing a first image from said ISS; (d) storing said first image with a fixed predefined absolute transformation and quality value; (e) capturing a new image from said ISS; (f) computing said new image&#39;s absolute transformation by matching said new image to at least one overlapping image within said bank; (g) storing said new image, its absolute transformation and its quality value in the said bank; (h) performing an iterative self-correcting procedure by altering the absolute transformation of a set of images within said bank if said new image overlaps at least two previously captured images within said bank, thereby improving the quality value of said transformation; and, (i) repeating steps (e)-(h) for each new image. It is within the essence of the invention wherein an aligned bank of images is created that contains substantially all of the information in said images along with additional information comprising at least an absolute transformation and further wherein said quality values of said images stored within said aligned bank of images are reduced by using an iterative self-correction procedure. 
         [0023]    It is a further object of the invention to provide a method as defined above, wherein said method further comprises the additional step of displaying at least two stored images as a mosaic by aligning the displayed images using the images&#39; transformation matrix according to a predetermined algorithm. 
         [0024]    It is a further object of the invention to provide a method as defined above, wherein said method further comprises the additional step of using a Ground Anchoring of A-BANK (GAAB) algorithm to anchor said bank to the ground. 
         [0025]    It is a further object of the invention to provide a method as defined above, wherein said method further comprises the additional steps of (a) selecting any point from any image in said bank; and (b) calculating the ground coordinates of said point using a Ground Coordinates Extraction of an Anchored A-BANK (GCEAAB) algorithm. It is within the essence of the invention wherein the geographic coordinates of a pixel of any image in said aligned bank of images is extracted via an iterative self-correction procedure. 
         [0026]    It is a further object of the invention to provide a method as defined above, wherein said method further comprises the additional steps of (a) anchoring said bank to the ground by a GAAB algorithm; (b) selecting any point of any image in said mosaic view of said bank; and (c) using a GCEAAB algorithm to calculate the ground coordinates said point. It is within the essence of the invention wherein the geographic coordinates of said pixel in said mosaic view of said aligned bank of images is extracted. 
         [0027]    It is a further object of the invention to provide a method as defined above, wherein said method further comprises the additional steps of (a) obtaining an ISS adapted for producing a temporal sequence of images; (b) updating the aligned bank of images with new images from said ISS; and (c) calculating the ground coordinates of any point in the most recently inserted image by using a GCEAAB algorithm. 
         [0028]    It is a further object of the invention to provide a method as defined above, wherein said method further comprises the additional steps of (a) defining a number N of regions to be used (N=integer≧0); (b) defining said regions in reference to a data set chosen from the group consisting of (i) images within said bank and (ii) said mosaic view created from images within said bank; (c) defining a set of detection parameters for each of said regions; (d) selecting a base image for object detection from said bank; (e) projecting said list of regions onto said base images by using a predetermined algorithm; (f) retrieving from said bank a list of reference images overlapping said base image; (g) applying the absolute transformation of said base image and said reference images, thereby computing a relative transformation between each of said reference images and said base image; and, (h) applying an object detection algorithm to said base image, said list of regions, said list of reference images, said relative transformations and said detection parameters, thereby obtaining a list of detected objects. It is within the essence of the invention wherein said method uses said aligned bank of images with said iterative self-correction procedure in order to detect an object. 
         [0029]    It is a further object of the invention to provide a method as defined above, wherein said method further comprises the additional step of using the images&#39; transformation to compute the location of said detected object within said mosaic view of said bank. 
         [0030]    It is a further object of the invention to provide a method as defined above, wherein said claim further comprises the additional steps of (a) defining a number N of regions to be used (N=integer≧0); (b) defining said regions in reference to a data set chosen from the group consisting of (i) images in said bank, (ii) the mosaic view within said bank, and (iii) ground coordinates; (c) defining a set of detection parameters for each of said regions; (d) selecting a base image for object detection from said bank; (e) projecting said list of regions onto said base image by using a predetermined algorithm; (f) retrieving a list of reference images overlapping said base image from said bank; (g) applying the absolute transformation of said base image and said reference images, thereby computing a relative transformation between each of said reference images and said base image; (h) applying an object detection algorithm to said base image, said list of regions, said list of reference images, said relative transformations and said detection parameters, thereby obtaining a list of detected objects; (i) updating the said bank with M new images (M=integer≧0); and (j) repeating steps (d)-(i) until a “stop” command is received. It is within the essence of the invention wherein said method detects an object by using said aligned bank of images that contains substantially all of the information in said images along with additional information comprising at least an absolute transformation and further wherein the said quality values of said images within said aligned bank of images are reduced by using an interactive self-correction procedure. 
         [0031]    It is a further object of the invention to provide a method as defined above, wherein said method further comprises the additional step of using a GCEAAB algorithm in order to compute the ground coordinates of each of said detected objects. 
         [0032]    It is a further object of the invention to provide a method for creating a geographically anchored bank of images from an image sequence source, comprising the steps of: (a) obtaining an image source sequence (ISS) whereas said images are taken from a plurality of locations; (b) creating a bank adapted for (i) storage of images and (ii) storage of additional data for each image in said bank, said additional data comprising at least (i) a ground to image transformation adapted to transform three-dimensional coordinates to corresponding image coordinates; and (ii) the quality value of said transformation; (c) capturing a new image I 0  from said ISS; (d) attempting to calculate a ground to image transformation by applying an image to ground anchoring algorithm to said new image I 0 ; (e) repeating steps (c) and (d) until a ground to image transformation has been calculated successfully; (f) adding said new image I 0  and the computed ground to image transformation of I 0  to said bank; (g) capturing a new image I n  from said ISS; (h) attempting to calculate a ground to image transformation by applying an image to ground anchoring algorithm to said new image I n ; (i) calculating a ground to image transformation if said ground to image transformation is not known for new image I n  by matching said new image I n  to at least one overlapping image already in said bank; (j) adding said new image In and the computed ground to image transformation of In to said bank; (k) performing an iterative self-correction procedure, thereby improving the quality value of the transformations of said images in said set, said procedure comprising the step of altering said ground to image transformation of a set of images within said bank if at least one of the following conditions is met: (1) said new image overlaps at least two previously captured video frames within said bank, (2) a ground to image transformation of said new image was successfully computed by said image to ground anchoring algorithm; and (l ) repeating steps (g)-(k) until a “stop” command is received. It is within the essence of the invention wherein an aligned image bank that contains substantially all of the information in said images along with additional information comprising at least a ground to image transformation is created and further wherein said quality values of said images contained within said bank are reduced by use of an interactive self-correction procedure. 
         [0033]    It is a further object of the invention to provide a method as defined above, wherein said method further comprises the additional step of extracting the geographic coordinates of a pixel in any image in said bank by applying a ground intersection algorithm (GIA). 
         [0034]    It is a further object of the invention to provide a method as defined above, wherein said method further comprises the additional step of creating a virtual view of said observed landscape from any point of view desired by the operator by using a virtual view engine to apply said image to ground transformation matrices of said stored images. 
         [0035]    It is a further object of the invention to provide a method as defined above, wherein said method further comprises the additional steps of (a) selecting a point from any image in said bank in said virtual view; and, (b) calculating the ground coordinates of said point by applying a GIA. It is within the essence of the invention wherein said method uses said geographically anchored bank of images to extract the geographic coordinates of any selected pixel in said virtual view. 
         [0036]    It is a further object of the invention to provide a method as defined above, wherein said method further comprises the additional steps of (a) obtaining an ISS adapted for producing a temporal sequence of images; (b) updating the aligned bank of images with new images from said ISS; and, (c) calculating the ground coordinates of any point in the most recently inserted image by using a GIA algorithm. 
         [0037]    It is a further object of the invention to provide a method as defined above, wherein said method further comprises the additional steps of (a) choosing a number N of regions (N=integer≧0); (b) defining said N regions with respect to a data set chosen from the group consisting of (i) a set of images within said bank, (ii) said virtual view of said bank and (iii) ground coordinates; (c) defining a set of detection parameters for each of said N regions; (d) selecting a base image for object detection from said bank; (e) projecting said list of regions on said base image by using a predetermined algorithm; (f) retrieving from said bank a list of reference images overlapping said base image; (g) applying the ground to image transformation of said base image and said reference images, thereby computing a relative transformation between each of said reference images and said base image; (h) applying an object detection algorithm to said base image, said list of regions, said list of reference images, said relative transformations and said detection parameters, thereby obtaining a list of detected objects; (i) updating the said bank with M new images (M=integer≧0); and (j) repeating steps (d)-(i) until a “stop” command is received. It is within the essence of the invention wherein said method uses said geographically anchored aligned bank of images to detect an object. 
         [0038]    It is a further object of this invention to provide a system for producing an aligned bank of images, said system comprising: (a) a CSDI; (b) means for producing an ISS from images captured by said CSDI; and (c) software means for producing an aligned bank of images from said ISS according to the method as defined above. It is within the essence of the invention wherein said system uses an iterative self-correction procedure for producing said aligned bank of images from said ISS and further wherein substantially all of the information contained in said images along with additional information comprising at least an absolute transformation is retained in said aligned bank of images. 
         [0039]    It is a further object of this invention to provide a system for producing an aligned bank of images, said system comprising: (a) a CSDI adapted for capturing images from a plurality of locations; (b) means for producing an ISS from images captured by said CSDI; and (c) software means for producing a geographically anchored bank of images from said ISS according to the method as defined above. It is within the essence of the invention wherein said system uses an iterative self-correction procedure for producing said geographically anchored bank of images from said ISS comprising images captured from a plurality of locations and further wherein substantially all of the information contained in said images along with additional information comprising at least an absolute transformation is retained in said aligned bank of images. 
     
    
     
       BRIEF DESCRIPTION OF THE FIGURES 
         [0040]      FIG. 1  illustrates schematically an A-BANK: an aligned bank of images storing images together with their absolute transformation relative to a predefined fixed image. 
           [0041]      FIG. 2  illustrates schematically a case of large accumulated matching errors between the first and the last image in an A-BANK. 
           [0042]      FIG. 3  illustrates schematically the use of the iterative self correction procedure disclosed herein to reduce errors in an A-BANK for the case illustrated in  FIG. 2 . 
           [0043]      FIG. 4  illustrates schematically a G-BANK: an aligned bank of images storing images together with their ground to image transformation. 
           [0044]      FIG. 5  illustrates schematically a case of large accumulated matching errors in a G-BANK inducing inaccurate ground to image transformation of certain images. 
           [0045]      FIG. 6  illustrates schematically the use of the iterative self correction procedure disclosed herein to reduce errors in a G-BANK for the case illustrated in  FIG. 5 . 
           [0046]      FIG. 7  illustrates schematically a schematic system implementing an image matching algorithm (IM). 
           [0047]      FIG. 8  illustrates schematically a typical virtual view engine (VVE) and a geometrical interpretation of the process. 
           [0048]      FIG. 9  illustrates schematically a typical system implementing an image to ground anchoring algorithm (IGA). 
           [0049]      FIG. 10  illustrates schematically a geometric interpretation of a ground to image matrix solving algorithm (GISA). 
           [0050]      FIG. 11  illustrates a geometric interpretation of a ground intersection algorithm (GIA), computing the ground coordinates of a particular point in the image given the ground to image transformation and a digital elevation model (DEM). 
           [0051]      FIG. 12  illustrates schematically computation of the azimuth and elevation of a point for the process of ground anchoring of image bank. 
           [0052]      FIG. 13  illustrates schematically an object detection algorithm detecting objects on a base image relative to a set of reference images overlapping the base image. 
           [0053]      FIG. 14  illustrates schematically regions of detection parameters defined on the base image for object detection. 
           [0054]      FIG. 15  comprises a flowchart and schematic illustrations describing one step of the self correction procedure for an A-BANK. 
           [0055]      FIG. 16  comprises a flowchart and schematic illustrations describing one step of the self-correction procedure for a G-BANK. 
       
    
    
     DETAILED DESCRIPTION OF THE INVENTION 
       [0056]    It will be apparent to one skilled in the art that there are several embodiments of the invention that differ in details of construction, without affecting the essential nature thereof, and therefore the invention is not limited by that which is illustrated in the figures and described in the specification, but only as indicated in the accompanying claims, with the proper scope determined only by the broadest interpretation of said claims. 
         [0057]    As used herein, the term “digitized image” refers to any two-dimensional array that is intended to represent something (e.g., an object or a landscape) that exists in (three-dimensional) space. Similarly, as used herein, the term “capturing and storing a digitized image” (CSDI) refers to any means for producing a digitized image as defined above from a real object, scene, landscape, etc. (“capturing”) and transferring it to any kind of digital storage medium (“storing”). Further, as used herein, the term “image sequence source” (ISS) refers to any means for providing a sequence of images obtained by CSDI from which a sequence of images can be extracted. 
         [0058]    In the specific embodiments described in detail, the digitized images are photographs or video captures produced by a camera, video camera, etc., using visible light. These embodiments are provided by way of non-limiting example only, however, as it will be clear to one skilled in the art that the basic principles disclosed herein can be applied to any series of digitized images as defined above, by any means for CSDI. It is acknowledged and emphasized, however, that the method disclosed herein is completely general, and can be applied to any means of CSDI. For example, the method can be used for visual images produced under light of any frequency (IR, UV, radio, etc.) by any means used to produce such images (e.g. cameras such as IR, UV, and X-ray cameras, video cameras, radar, sonar, thermal sensors, etc.). Thus the use of expressions such as “camera” or “video camera” refer generically to means for producing CSDI as defined above and not to limit it to cameras and video cameras adapted for producing images under visible light or to other means for producing CSDI, even those that are not typically called “cameras” in colloquial parlance. While, in the specific embodiments described in detail, the ISS is a sequence of images captured in real-time from a video camera or a sequence of video images captured from a file containing a bank of stored images that have been previously captured from a video camera, these examples are given to illustrate the best modes contemplated for the invention are not in any sense intended to limit it to the specific embodiments described. 
         [0059]    As used herein, the term “relative transformation between two images” refers to any linear transformation T 12  that, given two images I 1  and I 2 , computes the location P 2  within image I 2  that is equivalent to an arbitrary point P 1  within the image I 1 . One form of this transformation, which will be used hereinafter, is the affine transformation (well-known in the prior art) having the form 
         [0000]    
       
      
       P 
       2 
       =T 
       12 
       ·P 
       1  
      
     
         [0000]    where P 1 =(x 1 , y 1 , 1) is a point in the first image, P 2 =(x 2 , y 2 , 1) is a point in the second image, and the affine relative transformation 
         [0000]    
       
         
           
             
               T 
               12 
             
              
             
                
               
                 
                   
                     
                       a 
                       00 
                     
                   
                   
                     
                       a 
                       01 
                     
                   
                   
                     
                       a 
                       02 
                     
                   
                 
                 
                   
                     
                       a 
                       10 
                     
                   
                   
                     
                       a 
                       11 
                     
                   
                   
                     
                       a 
                       12 
                     
                   
                 
                 
                   
                     0 
                   
                   
                     0 
                   
                   
                     1 
                   
                 
               
                
             
           
         
       
     
         [0060]    As used herein, the term “absolute transformation of an image” refers to the transformation A i  between a fixed predefined image I n  and an arbitrary image I i  defined by: 
         [0000]      A i =T n,i    
         [0061]    The absolute transformation of an image gives the position and orientation of any image in the bank relative to one fixed image. 
         [0062]    As used herein, the term “image matching algorithm” (IM) refers to any algorithm that takes two overlapping images I 1  and I 2  and calculates the relative transformation T 1,2  between them such that T 1,2 =IM(I 1 , I 2 ). Reference is now made to  FIG. 7   a,  which illustrates schematically an image matching algorithm (IM) for computing T 1,2 =IM(I 1 , I 2 ). 
         [0063]    As used herein, the term “image selection algorithm” (ISA) refers to any algorithm that obtains an arbitrary image I x  with its transformation A x , where A x  can be either an absolute transformation or a ground to image transformation, and selects from the image bank a list L=ISA(I x , A x ) of candidate images I i , each one with an associated transformation A i , such that 
         [0000]      L={(I 1 ,A 1 ),(I 2 ,A 2 ), . . . (I n ,A n )} 
         [0000]    where each image I i  in the list can be matched successfully to the image I x  using the image matching algorithm IM. 
         [0064]    As used herein, the term “Digital Elevation Model” (DEM) refers to any model that describes the elevation z of any ground coordinate (x,y) such that z=DEM(x,y). In the present invention, the DEM implemented is of a type well known in the art, namely, a simple grid model in which the z value is given for every coordinate (x i ,y i ) in which x i  or y i  or both is a multiple of a fixed number N (usually 25 meters). As implemented in the present invention, DEM(X,Y) is computed using a bilinear interpolation of the four nearest neighbors of (X,Y) on the grid. 
         [0065]    As used herein, the term “ground to image transformation” refers to any transformation G that transforms every point P on the ground to its matching location Q in the image such that Q=G(P). In the present invention, the ground to image transformation implemented is of a type based on those well-known in the art: G is a 4×4 matrix G that represents a projective transformation having the form 
         [0000]    
       
      
       P′=G·P  
      
     
         [0000]    where P=(x 1 , y 1 , z 1 , 1) are the ground coordinates, P′=(x′, y′, z′, 1), and 
         [0000]    
       
         
           
             G 
             = 
             
                
               
                 
                   
                     
                       a 
                       00 
                     
                   
                   
                     
                       a 
                       01 
                     
                   
                   
                     
                       a 
                       02 
                     
                   
                   
                     
                       a 
                       03 
                     
                   
                 
                 
                   
                     
                       a 
                       10 
                     
                   
                   
                     
                       a 
                       11 
                     
                   
                   
                     
                       a 
                       12 
                     
                   
                   
                     
                       a 
                       13 
                     
                   
                 
                 
                   
                     
                       a 
                       20 
                     
                   
                   
                     
                       a 
                       21 
                     
                   
                   
                     
                       a 
                       22 
                     
                   
                   
                     
                       a 
                       23 
                     
                   
                 
                 
                   
                     0 
                   
                   
                     0 
                   
                   
                     0 
                   
                   
                     1 
                   
                 
               
                
             
           
         
       
     
         [0066]    The transformed point P″ on the image is given by 
         [0000]    
       
         
           
             
               P 
               ″ 
             
             = 
             
               ( 
               
                 
                   
                     x 
                     ′ 
                   
                   
                     z 
                     ′ 
                   
                 
                 , 
                 
                   
                     y 
                     ′ 
                   
                   
                     z 
                     ′ 
                   
                 
               
               ) 
             
           
         
       
     
         [0067]    As used herein, the term “relative transformation computation algorithm” (RTC) refers to any algorithm that obtains the ground to image transformation G 1 , G 2  of two images I 1 , I 2  and computes an approximate relative transformation T 12 =RTC(G 1 , G 2 ) between the images. 
         [0068]    As used herein, the term “virtual view” refers to a simulated view created from an image, which represents a view that would have been obtained had the image been taken from a different viewpoint. 
         [0069]    As used herein, the term “virtual view engine” (VVE) refers to an algorithm that creates a virtual view from an existing image. Reference is made to  FIG. 8 , which illustrates a typical VVE  800 . A DEM ( 810 ) of the terrain is used, and a virtual view VV x  defined with a ground to image transformation G V  of the desired view V obtained from camera position  80 . A virtual view (from virtual camera position  80 ′) is created for the source image I x  having an image to image transformation G x , such that VV x =VVE(G V , I x , G x ). 
         [0070]    As used herein, the term “image to ground anchoring algorithm” (IGA) refers to any algorithm that obtains an arbitrary input image I and calculates the ground to image transformation G of the image I. Reference is made to  FIG. 9 , illustrating a typical IGA algorithm  900 . It is assumed that the algorithm has access to a database of images of the area of interest that are anchored to the ground (i.e. the ground to image transformations of these images are known). Usually, an orthophoto (anchored air photos rectified as orthogonal photo) is used. This algorithm obtains the image and the camera ground coordinates. As implemented in typical embodiments of the invention herein disclosed, the IGA additionally obtains the approximate pan/tilt/yaw/zoom of the imaging device. According to these values, a projective view matrix V is computed and a virtual view of the image database is created according to the matrix V using a VVE. The transformation T I,VV  between the input image I and the virtual view VV is computed by matching the image to the virtual view by using the image matching algorithm (IM). The final image to ground transformation is G=T 1   I,VV ·V. 
         [0071]    Typical IGA algorithms frequently fail to match successfully the real and virtual views. The main reason is that the virtual view can severely differ from the input image. Thus, ideally, the IGA algorithm should notify the operator of its success or failure. In the invention herein disclosed, a manual version of the IGA algorithm enables a user to provide the ground coordinates P i  of at least two points Q i  in the image; a list L of these pairs is then created such that L={(Q 1 , P 1 ), (Q 2 ,P 2 ), . . . }. The image to ground transformation is further computed using a ground to image matrix solving algorithm. 
         [0072]    As used herein, the term “ground to image matrix solving algorithm” (GISA) refers to any algorithm that obtains the camera location C of the image I and a list L of at least two elements such that L={(q 1 , p 1 ), (q 2 , p 2 ) . . . }, in which q i  is a point on image I and p i  is the matching ground coordinate. The algorithm computes the transformation G I =GISA(C,L) where G I  is the ground to image transformation of the image I. In a typical implementation of the GIA as used in the invention herein disclosed, G is represented as a ground to image transformation matrix and is computed as follows. 
         [0073]    G can be written as: 
         [0000]    
       
      
       G=A·T  
      
     
         [0000]    where 
         [0000]    
       
         
           
             T 
             = 
             
                
               
                 
                   
                     1 
                   
                   
                     0 
                   
                   
                     0 
                   
                   
                     
                       - 
                       
                         C 
                         x 
                       
                     
                   
                 
                 
                   
                     0 
                   
                   
                     1 
                   
                   
                     0 
                   
                   
                     
                       - 
                       
                         C 
                         y 
                       
                     
                   
                 
                 
                   
                     0 
                   
                   
                     0 
                   
                   
                     1 
                   
                   
                     
                       - 
                       
                         C 
                         z 
                       
                     
                   
                 
                 
                   
                     0 
                   
                   
                     0 
                   
                   
                     0 
                   
                   
                     1 
                   
                 
               
                
             
           
         
       
     
         [0000]    (C x , C y , C z  are the camera ground coordinates) and the unknown matrix 
         [0000]    
       
         
           
             A 
             = 
             
                
               
                 
                   
                     
                       a 
                       00 
                     
                   
                   
                     
                       a 
                       01 
                     
                   
                   
                     
                       a 
                       02 
                     
                   
                   
                     0 
                   
                 
                 
                   
                     
                       a 
                       10 
                     
                   
                   
                     
                       a 
                       11 
                     
                   
                   
                     
                       a 
                       12 
                     
                   
                   
                     0 
                   
                 
                 
                   
                     
                       a 
                       20 
                     
                   
                   
                     
                       a 
                       21 
                     
                   
                   
                     
                       a 
                       22 
                     
                   
                   
                     0 
                   
                 
                 
                   
                     0 
                   
                   
                     0 
                   
                   
                     0 
                   
                   
                     1 
                   
                 
               
                
             
           
         
       
     
         [0074]    With a list L of at least five elements, the matrix A can be solved with any linear equation solving technique known in the art, such as SVD. Reference is now made to  FIG. 10 , which provides a schematic illustration of a GISA ( 150 ) for an image I obtained from camera position C 150 . 
         [0075]    As used herein, the term “ground intersection algorithm” (GIA) refers to any algorithm that obtains an arbitrary input image I with its ground to image transformation matrix G and computes the corresponding ground coordinate P=(p x , p y , p z ) of any point Q=(q x , q y ) on the image I such that P=GIA(G, Q). 
         [0076]    A typical implementation of the GIA as used in the invention herein disclosed begins with calculation of the inverse G −1  of a given ground to image transformation G. The ground coordinate P is computed using the ground digital elevation model (DEM) according to the following procedure. A line L is computed, starting at the point P 1 =G −1 ·Q 1  and passing through the point P 2 =G −1 ·Q 2  where Q 1 =(0, 0, 0, 1) and Q 2 =(q x , q y , 1, 1). The ground coordinate P is found by searching along the line L with a predefined step size, starting at point P 1  and stopping at the point P(x,y,z,1) when p z ≦DEM(p x , p y ). A schematic illustration of the computation is illustrated in  FIG. 11  for a typical implementation of the GIA  160  for an arbitrary image I(G). 
         [0077]    As used herein, the term “aligned bank of images” refers to a bank able to store images with additional data including a transformation and a quality value of said transformation. Two types of aligned bank of images are A-BANK and G-BANK. In an A-Bank, for every image I i  in the bank, the additional data includes at least an absolute transformation A i , where A i  is the relative transformation between a fixed predefined image and the image I i . Reference is now made to  FIG. 1 , which illustrates an A-Bank storing a set of images together with the associated absolute transformation. In a typical implementation, the predefined fixed image is the first image added to the bank. 
         [0078]    In a G-BANK, for every image in the bank, the additional data includes at least an image to ground transformation G i . Reference is further made to  FIG. 4 , which illustrates a G-Bank  400  storing a set of images together with the associated ground to image transformations  4000  and  4001 . 
         [0079]    As used herein, the term “Ground Coordinate Extraction of an Anchored A-BANK” (GCEAAB) refers to any procedure computing the ground coordinates Pi of every point Q i  in any image in A-BANK, including the ground coordinates of every point in the most recently stored image when the number of images in the bank is incremented. A typical GCEAAB algorithm requires performance of a preliminary procedure of Ground Anchoring of A-BANK defined hereafter. 
         [0080]    As used herein, the term “Ground Anchoring of A-BANK” (GAAB) refers to any procedure that enables a GCEAAB algorithm further to compute the ground coordinates of every point in any image in an A-BANK, including the ground coordinates of every point in the most recently stored image when the number of images in the bank is incremented. 
         [0081]    An example of a typical implementation of GAAB and GCEAAB algorithms is now presented. In this example, it is assumed that all of the images in the bank were taken from a fixed camera location and the absolute transformation of each image in the bank was computed (A-BANK). 
         [0082]    The GAAB procedure starts with the user providing ground coordinates of several points in images in the bank as a list L of pairs of points, 
         [0000]      L={(P 1 , Q 1 , I 1 ), (P 2 , Q 2 , I 2 ), . . . , (P n , Q n , I n )} 
         [0000]    where Q i  is a point in image I i  in the bank, and P i  is the corresponding ground coordinate. Next, a modified list L′ is created such that 
         [0000]      L′={(P 1 , Q 1 ′, I 1 ), (P 2 , Q 2 ′, I 2 ), . . . , (P n , Q n ′, I n )} 
         [0000]    where Q i ′=A −1   i ·Q i , A −1   i  is the inverse absolute transformation of the image I i  and Q i ′ is the corresponding location of Q i  on the fixed predefined reference image defined for the bank. 
         [0083]    Using L′ computed by GAAB, The GCEAAB can be used to computes P x =GCEAAB(Q x , I x ) where P x  is the ground coordinates of any point Q x  in any image I x . 
         [0084]    A typical implementation of a GCEAAB algorithm comprises the following steps:
       a. Obtain a point Q x  in image I x .   b. Calculate Q x ′=A −1   x ·Q x .   c. Create a new list B, containing at least five elements from the list L′ having the smallest distance to Q x ′, e.g. the smallest value of ∥Q′ i -Q x ′∥.   d. Using GISA, compute the ground to image transformation G′=GISA(C, B), where C is the camera ground coordinates, such that Q i ′=G′(P i ). e   . Using GIA, compute the ground coordinate P x  of the image point Q x  such that P x =GIA(G′, Q′ x ).       
 
         [0090]    Reference is now made to  FIG. 12 , illustrating another implementation of GCEAAB comprising images taken from a fixed known camera location. The Cartesian ground coordinates P i  (relative to true north, indicated in the illustration by the vector N) are transformed to A i E i , the azimuth and elevation angles of the line connecting the camera to the ground point with coordinates P i . List L is then transformed to L′={(Q i ′, AE 1 ), (Q 2 ′, AE 2 ), . . . }. The connection between Q i ′ and A i E i  is modeled as A i E i =M×Q′ i , where M is a 2×2 matrix. Solving for M requires at least two pairs of points. More points can be used in the solution to reduce the error, however. In a typical implementation, the two points in the image for which ground coordinates are already known that are closest (in the image) to the point for which the ground coordinates are being calculated are used. Thereafter, M can be used to compute the azimuth and elevation of the line connecting the camera to the ground location of every point in any stored image. 
         [0091]    The AE coordinates thus derived are not the true ground coordinates. The preferred way to convert AE coordinates to ground coordinates is to use a DEM. Reference is now made to  FIG. 11 , illustrating the use of a DEM  160  to convert AE coordinates to true ground coordinates. A line of sight L from the position of the imaging device P 1  to the image point P is computed and the intersection of the line of sight with the ground is computed using the DEM. 
         [0092]    The above implementation of ground anchoring of an A-Bank is “manual” in the sense that the operator provides the initial list of coordinates. An automatic ground anchoring procedure for a set of images taken from a known fixed camera location is now described. This automatic ground anchoring procedure involves the matching of several images from the bank to an already anchored photo (usually an orthophoto) using an IGA. In a typical implementation of an automatic ground anchoring procedure, the list L defined above, is created automatically, instead of manually, using the IGA algorithm assuming the camera position and the approximate camera orientation for each image are known. 
         [0093]    The following procedure is then used:
       a. Create an empty list L.   b. For every image I i  in the bank, use the IGA compute the ground to image transform G i  using the known camera position.   c. If the IGA succeeded, select five points Q 1 , . . . Q 5  in the image I i  (one point can be the image center and the other four point can be vertices of a rectangle with a predefined size around the center), and compute the corresponding ground coordinates P 1  . . . P 5  using the GIA, such that P i =GIA(G i , Q i ).   d. Add the elements {P 1 , Q 1 , I 1 } . . . {P 5 , Q 5 , I 5 } to the list L.   e. Repeat steps b-d until all the images in the bank have been processed.       
 
         [0099]    As used herein, the term “Computing ground to image transformation of image in anchored A-BANK” refers to any procedure for computing the ground to image transformation G i  of any image in the bank. A typical implementation of this procedure comprises the following steps:
       a. Select 5 image points (e.g. the four image vertices and the center).   b. Compute the ground coordinates of the 5 points using GCEAAB.   c. Compute G i  using the GISA using the said 5 points and the camera location.       
 
         [0103]    As used herein, the term “quality value of a transformation” refers to a factor q i  about an absolute transformation A i  or ground to image transformation G i  attached to an image I i . Since each of these transformations can be derived from a chain of image matching operations, each of which has limited accuracy, the total error will be proportional to the length of the chain. The quality value q i  is thus a relative measure of this total absolute error. q i =0 represents the highest possible confidence; increasing values of q i  represent lower confidence as the expected error increases. Reference is now made to  FIG. 2 , which illustrates schematically several images  20  through  29  in A-Bank  200  with the quality value (q) of their associated absolute transformation  2000 - 2008 . Reference is further made to  FIG. 5 , which illustrates schematically G-Bank  500  comprising images  50  through  55  and the associated quality values of the corresponding image to ground transformations  5000  through  5004 . 
         [0104]    As used herein, the term “iterative self-correction” refers to a procedure for reducing cumulative errors in image transformations (i.e., reducing the quality values of transformed images) caused by the limited accuracy of the image to image matching algorithm. Iterative self-correction is performed as new images are added to the bank: corrections are made sequentially in the bank of images, improving the accuracy of the transformations of images already in the bank. 
         [0105]    As used herein, the term “object detection algorithm” (ODA) refers to any algorithm for object detection. In order to detect an object, ODA requires at least:
       1. a base image I x  for object detection;   2. a list of reference images L={(I 0 , T 0X ), (I 1 , T 1X ), . . . (I N , T NX )}, N≧1, where I 0  . . . I N  are N images overlapping I x , where T 0X  . . . T NX  are the images&#39; relative transformation to the image I x ; and   3. a list R={(R 0 ,P 0 ), . . . (R M ,P M )}, M≧0, where R 0  . . . R M  is a list of M regions on the image I x , each with a set of parameters P i  defining the parameters for detection in the region R i .       
 
         [0109]    The algorithm detects objects on the images according to the parameter list and returns 
         [0000]        O=ODA ( I   x   , L, R ) 
         [0000]    where O={O 0 , . . . O N } is a list of the detected objects and O i  is a vector of attributes of the object i including at least the target location on the base image I x  and optionally other attributes such as object size, velocity and type. 
         [0110]    Reference is now made to  FIG. 13 , which illustrates an object detection algorithm  910  detecting objects O i  on a base image I x  relative to a set of reference images L 1 -L n  overlapping the base image, and to  FIG. 14 , which illustrates regions of detection parameters defined on the base image  920  for object detection. For example, region  9201  may be a region of high sensitivity; region  9202  one of low sensitivity; and region  9203  excluded altogether from detection. 
         [0111]    There are many ODAs known in the prior art. One family is based on shape detection and hence can only work for one type of image. Many other ODAs are known in the art that are based on measurement of changes between the base image and the reference images. These ODAs require the presence of overlapping images as a reference as wells as the relative transformation between each reference image and the base image I x . Motion detection algorithms are a well known family of this kind of ODA. 
         [0112]    In the invention disclosed herein, an ODA typically implemented is a motion detection algorithm that uses a base image I x , a sequence L of reference images, and the associated relative transformations of the images in the sequence to I x . In addition, the algorithm uses a list R of regions defined as polygons within the base image. Associated with each region is a set of detection parameters that include the ranges of the object&#39;s size, speed, and contrast. 
         [0113]    This ODA thus can provide, within a single list R, some regions with very sensitive detection limits while others are set to exclude any possibility of object detection within their borders. The ODA returns a list O of detected moving objects. List O includes the position on the base image of each object detected. Optionally, the list can contain the speed and size of each detected object as well. 
         [0114]    With these definitions in hand, we proceed to the detailed description of the invention and its embodiments. 
         [0115]    As will be demonstrated in detail below, one important difference between the invention herein disclosed and those known in the prior art is the use of iterative self-correction in the construction of the aligned bank of images. Two versions of the iterative self-correction procedure are here described. The first is adapted for reducing errors in the absolute transformations of images in an A-Bank. The second version is for reducing errors in the ground to image transformations of images in a G-Bank. 
         [0116]    Reference is now made to  FIGS. 2 and 3 .  FIG. 2  shows a typical A-Bank  200  comprising images  20 - 29 , and the accumulation of errors in the absolute transformation (as shown by the increase in q) as new images are added to the bank via the corresponding absolute transformations  2000 - 2008 .  FIG. 3  illustrates the result of an iterative self-correction procedure (absolute transformations  2900 - 2903 ) performed on the A-Bank of  FIG. 2  upon addition of a new image  290  to the A-BANK. 
         [0117]    A preferred embodiment of iterative self-correction as applied to creation of an aligned image bank with iterative self-correction is now described in detail. This embodiment is of an A-BANK created by images captured from an image sequence source whose images were taken from a fixed position, that is, the only motions that the imaging device can undergo are rotation around any or all of the x, y, and z axes and/or changing zoom. 
         [0118]    In this embodiment, the A-BANK is created and the images obtained by the imaging device are added to the bank according to a method described in detail below. A mosaic display of the images collected in the bank can be created from the images in the bank. The procedure for ground anchoring the A-BANK can be performed at any time, and the ground coordinates of any point in any image stored in the bank or of any point on the mosaic view can be computed by GCEAAB. Specifically, the ground coordinates of points within the last stored image can be computed. This way, a video anchoring system is provided, as every successive image (e.g. video frame) is anchored to the ground and ground coordinates of points in the last image can be computed. 
         [0119]    In this embodiment, the method for creating the bank comprises the following steps:
       1. Obtaining an image sequence source (ISS). The images can originate, e.g., from a camera observing a landscape, the camera being mounted on a pedestal enabling rotation the camera with two or three degrees of freedom (pan/tilt/yaw). The camera can have a zoom control for zooming in and out. Optionally, the camera can report the approximate values for the zoom and the pan/tilt/yaw angles, improving the performance of the whole system.   2. Creating a bank able to store images with additional data. For every image I i  in the bank, the additional data includes: (a) absolute transformation A i ; (b) quality value q i  of the transformation; (c) the index of a second image I j  (j≠i) used for computation of A i ; and (d) image matching transformation T j,i  between the second image I j  and the current image. The retrieval capacities of the bank minimally comprise: (a) given an image I n  with an absolute transformation A n , it can retrieve all the images in the bank that can be matches to the image I n  using the image matching algorithm (IM); (b) given an area defined as an absolute transformation A area  and a preferred zoom factor Z, it can retrieve the minimum number of images needed to cover the defined area, preferring images with zoom factor as close to Z as possible.   3. Capturing the first digital image I 0  and storing it in the bank. The additional information for this image will be A i =I (i.e. the unit transformation); q i =0; and j i =0. This image is thus the predefined fixed image for the absolute transformations associated with the images to be added to the bank.   4. Capturing a new digital image I n  from the ISS and, by using the image matching algorithm IM, computing the relative transformation T n-1,n =IM(I n-1 , I n ) between the previously captured digital image I n-1  and the new digital image.   5. Computing the approximate absolute transformation A′ n  by combining said relative transformation T n-1,n  with said absolute transformation A′ n-1  such that A′ n =T n-1,n ·A n-1 .   6. Retrieving from the bank a list L=ISA(I n , A n ) of images by that can be used by the image matching algorithm to make a successful match to the image I n . This retrieval is done by using the image selection algorithm (ISA).   7. From the list L, selecting the database image I m  having the best quality value (i.e., the lowest value of q m ).   8. Using the image matching algorithm to compute the relative transformation T m,n  between the selected database image I m  and the new digital image I n .   9. Computing the final image transformation A n  for the new video frame I n , where A n =T m,n ·A m .   10. Setting the quality value of the new digital image I n  to q n =q m +1.   11. Storing the new digital image I n  along with its additional data, namely, absolute transformation=A n ; quality value=q n ; selected database image index=m.   12. Computing q min  and q max , which are the minimum and maximum quality value of the images in the list L. The condition q max -q min &gt;S, where S is a predetermined threshold, indicates a situation where accumulated matching errors can cause a significant matching error around the new digital image I. The situation is described in  FIG. 4 . In this case, the iterative self-correction procedure for an A-Bank is used.   13. For each new image from the image sequence, repeat the method starting at step 4.       
 
         [0133]    This method thus creates a self-updating and self-correcting image bank, where every image contains the appropriate absolute transformation, i.e. all information needed to display the bank as a mosaic, in which the accumulated errors are kept small. The sequence of steps outlined above can be run an indefinite number of times, until every new image desired has been added to the bank. To end the sequence, a “stop” command is provided, either from within the software or by the user. In this context, a “stop” command need not necessarily be a software command; it includes the possibility of the user ending data collection by such physical means as turning off the CSDI or by detaching it from the data analysis software. 
         [0134]    In an additional embodiment of the invention herein disclosed, the following procedure is used to create a mosaic view of the bank:
       1. Defining the area to be displayed. The area can be defined as an absolute transformation A display .   2. Defining the optimal zoom factor Z of the images to be used.   3. Retrieving the optimal images from the bank overlapping the transformation A display  with preferred zoom factor Z. There can be many images covering the same area; each one has a different zoom factor. In this case, those images having a zoom factor close to the optimal zoom should be preferred.   4. For each of the retrieved images I i , create a warped image I′ i  using the absolute transformation A i  such that every point Q i  in I i  will be transformed to Q′ i =A −1   i ·Q i      5. Display all warped images I′ i .       
 
         [0140]    The method disclosed herein includes the iterative self-correction procedure for an A-BANK to be used when the following conditions hold:
       a. A possible large accumulated error (as defined below) in the absolute transformation A i  of images in the bank has been detected.   b. The absolute transformation A n  computed for the newly stored image can be used to reduce said errors.       
 
         [0143]    One indication of such a situation is when the newly stored image has at least two overlapping images with large differences in their quality values. 
         [0144]    A preferred embodiment of the iterative self-correction procedure for an A-Bank is now described. The procedure comprises the following steps:
       1. Given the newly stored image I m  with its absolute transformation A m  and quality value q m , the ISA is used to retrieve from the bank a list E=ISA(I m , A m ) of images that can be used by the image matching algorithm (IM) to match the image I m .   2. For every image I i  in the list E having a quality factor q i &gt;q m +1:
           a. Using the IM algorithm, compute the relative transformation T m,i =IM(I m , I i ) between the newly stored image I m  and the image I i .   b. Compute an absolute transformation of the image I i  using the image I m  such that A i ′=T m,i ·A m .   c. Set j old =j i  which is the neighboring image used for the computation of the old absolute transformation A i .   d. Store the following new data for the image I i :   
               
 
         [0000]      A i =A′ i    
         [0000]      j i =m 
         [0000]        q   i   =q   m +1       e. Set m=i and i=j old  and repeat the procedure from step 2 if q i &gt;q m +1.         
         [0152]    Reference is now made to  FIG. 15 , which illustrates one iteration of the iterative self-correction procedure. A flow chart of the procedure (beginning with arbitrary I, q, and A, shown as I m , q m , and A m  respectively) is shown in  FIG. 15   a.  As shown in  FIG. 15   b,  before the self-correction step, image  3311  was matched to image  3310  having a quality factor q=11 and an absolute transformation A 3311 =T 3310,3311 ·A 3310 . When a new image  3312  is added to the bank, it is matched to image  3313 , giving it a quality factor q 12 =6 and an absolute transformation A 3312 =T 3313,3312 ·A 3313 . The self-correction procedure is then performed. After one iteration of the procedure, image  3311  is matched to  3312 , which improves its quality factor q from 11 to 8 (=q 3312 +1) ( FIG. 15   c ) and gives it an absolute transformation A 3311 =T 3312,3311 ·A 3312 . 
         [0153]    Thus, after the completion of the whole self-correction procedure, the newly calculated transformation A i  of the processed chain of images I i  will always have a smaller quality factor q i , indicating an improvement of the overall accuracy of the alignment of the images in the bank. 
         [0154]    Reference is now made to  FIGS. 2 and 3  showing an example of the effect of the self-correction procedure.  FIG. 2  illustrates a typical A-Bank  200  comprising images  20 - 29  where each image is matched to its predecessor ( 2000 - 2008 ); the accumulation of matching errors is shown by the increase in q. Specifically, the error between image  20  and  29  is unacceptably large.  FIG. 3  illustrates schematically a newly added image  290 , overlapping image  20  and  29 , which initializes the self-correction procedure. Image  290  is matched to image  20  via transformation  2920 . After the procedure, the matching transformation of images  26 - 29  is changed to  2901 - 2903  respectively and as shown, the quality value q of images  26 - 29  is reduced, indicating an overall improvement in the accuracy of the A-BANK. 
         [0155]    In one embodiment of the present invention, a manual version of GAAB is used. The procedure involves providing the coordinates of the position of the fixed camera and the ground coordinates of several points in images in the bank. According to this embodiment, a list L is created manually with the entries containing an index m of an image I m , a point Q i  in the image I m  and the ground coordinates P i  of that point. The list is transferred to the manual version of the ground anchoring of A-Bank algorithm. Thereafter, the GCEAAB algorithm is used to compute the ground coordinate of every point in any stored image. 
         [0156]    In an additional embodiment of the present invention, automatic GAAB is used. The automatic version of the ground anchoring of A-Bank algorithm is used. Thereafter, GCEAAB algorithm can be used to compute the ground coordinate of every point in any stored image. 
         [0157]    The GAAB procedure described herein can be performed any time. Once the procedure is finished, images added afterwards will not require repetition of the procedure. Obtaining the ground coordinates of point in these newly added images is possible as well using GCEAAB. Specifically, the ground coordinates of points on the last stored image can be computed. This way, a video anchoring system is achieved, since every video frame is added to the bank and the ground coordinates of points on the last video frame can be computed. In an alternative embodiment of the invention herein disclosed, in order to increase anchoring accuracy, some points from the newly covered area are added to the list L. This embodiment is particularly useful if a large new area was covered after the initial performance of the GAAB procedure. 
         [0158]    In an additional embodiment of the method herein disclosed, the mosaic display of the bank is combined with ground anchoring, a powerful orientation tool for the user. This combination gives the user the option to view the mosaic of a predefined area and to get the ground coordinates of any point of interest on the mosaic. When a user selects a point Q m  in the mosaic display, an image I i  containing the selected point is found and the corresponding point Q i  on the said image I i  is computed. The GCEAAB is then used to compute the ground coordinates P m  of the selected mosaic point Q m  such that P m =GCEAAB(Q i , I i ). 
         [0159]    An additional embodiment of the invention herein disclosed includes object detection for image stream source originating from a rotating and zooming camera. Given any ODA, object detection is performed on the A-BANK as follows:
       1. Choose a number N of regions. (N≧0)   2. Define a list R′ of N polygonal regions with associated detection parameters according to the method described in detail below.   3. Select a base image for detection I x  from the bank.   4. Project the list R′ onto the base image to get the list R of regions on said image according to the procedure described in detail below.   5. Retrieve from the bank a list I 0  . . . I N  of reference images overlapping I x  and the associated absolute transformations A 0  . . . A N .   6. Construct the list L={({I 0 , T 0 }, (I 1 , T 1 ), . . . (I N , T N )} of images with their relative transformations T i  to the base image (T i =A i A −1   x ).   7. Use the ODA to obtain a list of detected objects O=ODA(I x , L, R), including the locations of the objects on the base image and the associated additional information.   8. Update the A-BANK with M new images (M=integer≧0).   9. Repeat steps 3-8 until a “stop” command is received.       
 
         [0169]    Alternative embodiments of the invention include the additional steps of computing the ground coordinates of objects in the list following detection of the object if the bank was anchored to ground, and of displaying on a map the ground coordinates of the objects. As with the previous embodiments, the object detection sequence may be run once, or a plurality of times until a “stop” command (either from within the software, via a command from the user, or physically stopping image collection, as defined above) is received, or indefinitely. 
         [0170]    There are several options to define the list R of polygonal regions for detection and to project them on the base image:
       a) R can be defined on the base image I x .   b) Alternatively, it can be defined on other images within the bank. In this case, the regions are projected onto R x  as follows: assuming that R t =(V 0  . . . V N ) is a polygon with N vertices defined on the image I y  having an absolute transformation A y . The projected polygon R x  on image I x  is then given by R x =A x A y   −1 R y .   c) R can be defined on the mosaic view of the bank. In this case the regions are projected onto I x  according to the following procedure: assuming that R M ={V 0  . . . V N ) is a polygon with N vertices defined on the mosaic, the projected polygon is given by R x =A x R M .   d) If the bank was anchored to ground with GAAB, R can be defined according to ground coordinates. In this case the regions are projected to I x  according to the following procedure: assuming that R M ={V 0  . . . V N ) is a polygon with N vertices defined as ground coordinates, the ground to image transformation G x  of I x  is computed using the “computing ground to image transformation of image in an anchored A-BANK” procedure, and the projected polygon is thus R x =G x (R M ).       
 
         [0175]    Regions can be predefined once with any of the methods in the list. Thereafter, object detection can be performed on any image with the predefined regions in effect. Continuous object detection on a video sequence can be performed by adding the images from the video sequence to the bank (step 8 above) and performing steps 3-7 above when the selected base image in step 3 is set as the image most recently captured from the video sequence. 
         [0176]    In the special case in which the ODA is based on moving object detection, there should be some additional consideration given to choosing the list of reference images to use. Usually, in order to detect fast-moving objects, the time interval between the images in the list L of reference images should be short, while in contrast, to detect slow objects, the time interval between the said images should be long. The reference images should thus be selected accordingly. 
         [0177]    An embodiment of the aligned image bank with self-correction technique herein disclosed in which a geographically anchored aligned image bank is created from an image sequence source in which the imaging device is moving is now described. This embodiment of the invention is suitable, e.g., for an image sequence source in which the images are taken from cameras mounted on a vehicle, airborne platform, or handheld. 
         [0178]    In this embodiment, a G-BANK type of image bank is created. The G-BANK stores the images along with their ground to image transformation G as shown in  FIG. 4 . Each new image is transferred to an IGA and the ground to image transformation G is computed. If the IGA fails to produce an acceptable G, G is approximated by matching the new image to other images already in the bank. With a G-BANK, there is no need to perform the anchoring procedure as was necessary in the case of an A-BANK, because the G-BANK is created and incremented while already anchored to ground. 
         [0179]    When the newly obtained image is successfully matched to ground by the IGA or when it overlaps more than one image, an iterative self-correction procedure for the G-BANK is used to reduce the errors of the ground to image transformation of some images already in the bank. This special technique ensures a low rate of matching errors in the bank. 
         [0180]    In contrast to embodiments involving creation and use of an A-BANK, the G-BANK is anchored to ground from its creation. Images are not added to the bank until the first image is anchored. Since the method disclosed herein retains all of the original images and all of their associated information, images for which a successful match was not made can be set aside and added to the bank at any later time at which a successful match can be made. Another option is to add the images with an approximate image to ground transformation from any external source if given. 
         [0181]    Thus, in many cases, using a G-BANK is superior to using an A-BANK, since for a G-BANK, there is no need to perform an anchoring procedure and embodiments in which a G-BANK is used are suitable for moving cameras. On the other hand, an A-BANK can be created and incremented without using an IGA. IGAs usually require a DEM, which is not always available, and as demonstrated above, an A-BANK can be anchored to ground at any time should such anchoring be necessary. 
         [0182]    A mosaic view from any viewpoint can be created from the images in the bank. The desired viewpoint of the mosaic is defined as a view matrix V. The ground coordinates of any point in any stored image or any point on the mosaic view can be computed. Specifically, the ground coordinates of points on the most recently stored image can be computed. This way, a video anchoring system for a moving camera is achieved as every video frame is anchored to the ground and ground coordinates of any point on he last video frame can be computed. 
         [0183]    The embodiment for G-BANK disclosed herein is based on the IGA. The automatic IGA is not expected to succeed in all cases. Hence, the method described herein is especially adapted to overcome the possibility of IGA failure. 
         [0184]    A typical IGA requires approximate values for the zoom factor, pan/tilt/yaw angles, and the ground position of the camera. As the accuracy of these values increases, the success rate of the IGA increases as well as the overall performance of the method. 
         [0185]    Nevertheless, if the level of performance of an automatic IGA is not acceptable, then a manual version of the IGA can be used to anchor some of the images in the bank and the method will still function. 
         [0186]    For simplicity, it is assumed that images will not be stored in the bank until the first image is anchored successfully to the ground by the IGA; if necessary, however, this constraint can be removed by keeping the images in a temporary storage location and adding them to the bank at some time after the first image is anchored to the ground or adding the images as they are arriving using an approximate image to ground transformation from any external source if given. 
         [0187]    One preferred embodiment of the aligned image bank with self-correction method based on a G-BANK and usable when the image source is moving comprises the steps of:
       1. Obtaining an image sequences source (ISS). The images can originate from a CSDI on a fixed or moving installation, enabling the camera to observe a landscape.   2. Creating a database capable of storing images and their associated additional data. For every image I n  in the bank, the additional data includes:
           a. the image to ground transformation G i ;   b. the ground coordinates of the camera;   c. the quality value q i  about the value of G i . If G i  is computed directly by the IGA, q i =0; if q i  is computed by a chain of image matching with length n, q i =n; and   d. the index of a neighboring image j i  used for the computation of G i  and the image matching transformation T i,m  between the current image and the neighboring image m=j i .   
               
 
         [0194]    The image bank has the following retrieval capacities:
       a. Given an image I n  with ground to image transformation G n , it can retrieve all the images in the bank that can be matched to the image I n  using the image matching algorithm (IM);   b. Given a ground area defined as a polygon, and a preferred zoom factor Z, it can retrieve the minimum number of images needed for covering the defined ground area, preferring images with zoom factor as close to Z as possible.   2. Capturing a new image I n  from the image source and getting the approximate values for the camera position and the zoom and pan/tilt/yaw values.   3. The image and the approximate values for camera position, zoom and camera angles are fed to the IGA. The IGA attempts to anchor the image and to compute the ground to image transformation G n  for the image I n .   4. If no image was successfully matched by the IGA, new images are captured until the IGA succeeds for the first time.   5. If the IGA succeeds, the image I n  is added to the bank with the additional data: G n , q n =0, j i =−1.   6. If the IGA fails,
           i. Using the IM algorithm, the relative transformation T n-1,n  between the previously captured digital image I n-1  and the new digital image I n  is computed.   ii. The approximate image to ground transformation G n ′ is calculated by combining the relative transformation T n-1,n  with the ground to image transformation G n-1  such that G n ′=T n-1,n ·G n-1 .   iii. By using the image selection algorithm, the list L=ISA(I n , G′ n ) is created containing all the images in the bank that can be used by the image matching algorithm IM to make a successful match to image I n .   iv. From the list L, the image I m  having the lowest value of q m  (i.e. the image with the best transformation quality value) is selected.   v. The image matching algorithm is used to compute the relative transformation T m,n  between the selected image I m  and the new digital image I n .   vi. The final image transformation for the new digital image I n  is computed as G n =T m,n ·G m .   vii. The quality value of the new digital image I n  is set to q n =q m +1 and j n  is set to j n =m.   
           7. Storing the new digital image I n  with its new data, including G n , q n , and the selected image j n =m.   8. Retrieving from the bank a list L of images overlapping the image I n  according to the transformation G n .   9. Adding the new digital image I n  to the list L.   10. Computing q min  and q max  for the images in the list L.   11. The condition q max -q min &gt;S (S is a predefined number, usually 3) indicates a situation where accumulated matching errors can cause a significant matching error around the new digital image. The situation is described in  FIG. 6 . In this case, a self-correction procedure is used.   12. For each consecutive new image from the image sequence source, follow the procedure listed, starting from step 3.       
 
         [0215]    As with the previous embodiments, the sequence of steps can be repeated indefinitely until a “stop” command (as defined above) is received. 
         [0216]    The method disclosed herein includes the iterative self-correction procedure for to be used for G-BANK when the following conditions hold:
       a. A possible large accumulated error (as defined below) in the ground to image transformation G i  of images in the bank has been detected.   b. The absolute transformation G n  computed for the newly stored image can be used to reduce said errors.       
 
         [0219]    A preferred embodiment of the iterative self-correction procedure for use with a G-BANK is now described. The procedure comprises the following steps:
       1. Given the newly stored image I m  and using the ISA, a list L=ISA(I m , G m ) of images that can be successfully matched to the image I m  is built.   2. For every image I i  in the list L having a quality value q i &gt;q m +1,
           a. The IM is used to compute the relative transformation T m,i =IM(I m , I i ) between image I m  and image I i .   b. A new ground to image transformation of the image I i  is calculated using the image I m  such that G i ′=T m,i ·G m .   c. Set j old =j i  which is the neighboring image used for the computation of the old G i .   d. The following new data is stored for image I i : G i =G i ′; j i =m; q i =q m +1.   e. Set m=i and i=j old .   f. If q i &gt;q m +1, then repeat the procedure from step b.   
               
 
         [0228]    Reference is now made to  FIG. 16 , which provides a schematic illustration of one iteration of the self-correction procedure  340 .  FIG. 16   a  illustrates a flow chart of the procedure beginning from arbitrary I m , q m , and G m . As shown in  FIG. 16   b,  before the self-correction step, image  3411  was matched to image  3410 , having a quality value q 11 =11 and a ground to image transformation G 11 =T 10,11 ·G 10 . A new image  3412  is added to the bank and matched to image  3413  thus having a quality value q 12 =7 and an ground to image transformation G 12 =T 3413,3412 ·G 3413 . The self-correction procedure is then performed. After one iteration of the procedure ( FIG. 16   c ), image  3411  is matched to  3412  which improves its quality value q 11  from 11 to 8 and gives it a ground to image transformation G 3411 =T 3412,3411 ·G 3412 . At the conclusion of this procedure, the newly computed transformation G i  of the processed chain of images I i  will always have a smaller (i.e. better) quality value q i , indicating an improvement of the overall accuracy of the alignment of the images in the bank. 
         [0229]    Reference is now made to  FIGS. 5 and 6  showing an example of the effect of the self correction procedure used for a G_BANK.  FIG. 5  illustrates a situation with large accumulated errors. Image  50  was successfully anchored to ground with an IGA having q=0. Images  51 - 55  were not successfully matched to ground by the IGA and therefore each was matched to its predecessor ( 5000 - 5004 ) to approximate their G. The quality factor q of these images increases: image  55 , for example, has a quality value q=5. A new image  550  is obtained and is successfully anchored to ground with the IGA, which therefore gives it a quality value q=0 as shown in  FIG. 6 . The overlapping image  55  has a quality value q=5, therefore, the self-correction procedure is activated, reducing the quality factors of the images  54  and  55 . The effect of the self-correction procedure with the new matching order ( 5500 - 5501 ) is shown schematically in  FIG. 6 . 
         [0230]    For creation of a mosaic view of the bank as described above, a method comprising the following steps is disclosed:
       1. The ground area to be displayed is defined as a polygon of ground coordinates.   2. The optimal zoom factor Z is defined for the images to be used to construct the mosaic.   3. The desired viewpoint of the mosaic is defined as a view matrix V.   4. The optimal list L of images from the bank overlapping the ground area to be displayed with preferred zoom factor Z is retrieved. Note that there can be many images covering the same area, each one having a different zoom factor. In this case, the images having a zoom factor close to the optimal zoom should be used.   5. For each of the retrieved images I i  in the list L having a ground to image transformation G i , create a virtual view image I i ′ according to the view matrix V, using the VVE.   6. Display all of the virtual view images I i ′.       
 
         [0237]    The ground coordinates P i  of any point Q i  on any image I x  in the bank can be obtained. The preferred way to obtain them is by using the GIA such that P i =GIA(Q i , G x ), where G x  is the ground to image transformation of the image I x . 
         [0238]    Specifically, the ground coordinates of any point on the last stored image can be computed. This way, a video anchoring system is achieved, as every video frame is added to the bank and the ground coordinates of points on the last video frame can be computed. 
         [0239]    Alternatively, a point Q v  can be selected in the mosaic view of the G-BANK and the ground coordinates P v  of the point can be computed as follows: an image I i  containing the selected point Q v  is found and the corresponding point Q i  on the said image I i  is computed. The ground coordinates are then computed using a GIA such that: P v =GIA(Q i , G i ), where G i  is the ground to image transformation of the image I i . 
         [0240]    In a further embodiment of the invention herein disclosed, using the above method for creating an anchored aligned bank of images using a G-Bank, and given any ODA, a method for object detection for a moving camera is disclosed, comprising the following steps:
       1. Choose a number N of desired regions (N≧0).   2. Define a list R′ of N polygonal regions with associated detection parameters according to the method described in detail below.   3. Select the base image for detection I x  from the bank.   4. Project the list R′ onto the base image to get the list R of regions on said image according to the procedure described in detail below.   5. Retrieve from the bank a list I 0  . . . I N  of reference images overlapping I x  and the associated ground to image transformation G i .   6. Construct the list L={(I 0 , T 0 ), (I 1 , T 1 ), . . . (I N , T N )} of images with their relative transformations T i  to the base image, T i =RTC(G i , G x ).   7. Use the ODA to obtain a list of detected objects O=ODA(I x , L, R) including their location on the base image and the additional available information.   8. Update the G-BANK with M new images (M=integer≧0).   9. Repeat steps 3-8 until a “stop” command is received.       
 
         [0250]    In additional embodiments of the invention, the object detection method includes the additional steps of computing the ground coordinates of objects in the list O, and/or displaying the ground coordinates of the objects on a map or on any mosaic view of the bank. 
         [0251]    The list R of polygonal regions of detection parameters is defined according to one of the region detection methods listed here and is projected on the base image I x .
       a) R is defined relative to the base image I x  used for object detection.   b) R is defined relative to other images in the bank. In this case the regions are projected onto I x  according to the following procedure: assuming that R t =(V 0  . . . V N ) is a polygon with N vertices defined on the image I y  having an absolute transformation A y , then the projected polygon on image I x  is R x =RTC(G x , G y ).   c) R is defined on the mosaic view of the bank with image to ground transformation G v . In this case the regions are projected onto I x  according to the following procedure: assuming that R M =(V 0  . . . V N ) is a polygon with N vertices defined on the mosaic, then the projected polygon is R x =RTC(G x , G y ).   d) R is defined relative to ground coordinates. In this case the regions are projected onto I x  according to the following procedure: assuming that R M ={V 0  . . . V N ) is a polygon with N vertices defined on the mosaic, then the projected polygon is R x =G x ·R m .       
 
         [0256]    As in the case of a G-BANK, regions can be predefined once with any of the methods in the list. Thereafter, object detection can be performed on any image with the predefined regions in effect. Furthermore, continuous object detection on a video sequence can be performed by adding the images from the video sequence to the bank and performing steps 3-7 above where the selected base image in step 3 is set as the most recently captured image from the video sequence. 
         [0257]    The various embodiments of the methods disclosed above are easily incorporated into a system designed for producing an aligned bank of images by using an iterative self-correction procedure. It is within the scope of the present invention to disclose such a system. Such a system comprises any type of CSDI as defined above; means for producing an ISS from the images captured by said CSDI; and the software necessary for enabling said ISS to produce an aligned bank of images according to any of the embodiments of the method disclosed above. A preferred embodiment of the system incorporates software adapted to run the method according to the preferred embodiment of the method for a sequence of images obtained from a fixed location. A second embodiment of the system incorporates software adapted to run the method according to the preferred embodiment for a sequence of images obtained from a moving CSDI (i.e. a sequence of images obtained from a plurality of locations).