Abstract:
A panoramic imaging threat detection and alert system to automate the detection, localization, tracking and assessment of moving objects within a specified field of view. This system utilizes an array of large-scale imaging chips, an array of reflective lenses coded for computational imaging, passive distance measurement and high-speed processors to determine the characteristics of objects of interest. This system selects moving objects to further evaluate for threat assessment and communicates object size, speed, distance and acceleration to a designated threat assessment center or personnel for further action.

Description:
CROSS-REFERENCES TO RELATED APPLICATIONS 
       [0001]    This application claims priority to U.S. Provisional Application 60/939,319 filed on May 21, 2007, which itself claims priority to U.S. Provisional Application 60/917,049, filed on May 9, 2007. The foregoing applications are hereby incorporated by reference in their entirety as if fully set forth herein. 
     
    
     STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT 
       [0002]    Not Applicable 
       INCORPORATION-BY-REFERENCE OF MATERIAL SUBMITTED ON A COMPACT DISC 
       [0003]    Not Applicable 
       BACKGROUND OF THE INVENTION 
       [0004]    The world situation requires constant vigilance against a variety of threats. Social scientists may argue that as distances shrink and populations grow, conflict at many levels would be inevitable. Whatever the cause or the reason, most of a nation&#39;s assets now require monitoring as a protection against some kind of threat by internal, external, foreign or domestic human forces. The results are a current need to monitor borders, pipelines, reservoirs, ports (both air and water), buildings, energy stores (oil, gas, hydro, electric, etc.) and other natural or man-made constructions of national value. In fact, sometimes the protected entity will be just the population at a sporting event or at work, or just gathered in one place, such as commuting on a highway. 
         [0005]    The first efforts at monitored protection have been efforts such as the British effort at fixed installed cameras. The cameras provide remote observation but require a human effort at the monitored end of the system to detect an alert and enable prevention. Otherwise, the system only provides a recorded video history as a beginning of the effort to discover the identity or exact means used by the perpetrators. 
         [0006]    Another aspect of the problems of the fixed camera technology as currently used is the lack of sufficient coverage. Cameras with sufficient resolution to enable some forms of automatic alert or threat identification would have very limited fields of view. So, the numbers of cameras required for sufficient coverage would be extremely large. 
         [0007]    This large number of installed cameras brings yet another complexity: the required connection bandwidth to bring all of that video back to the head-end for monitoring by a now extremely large number of human monitors or a very large analytic computer to provide the automated alert functions. As the complexity and the number of areas to be monitored grow, a system is needed which will manage the complexity and decrease the burden on these human monitors and responders: a system with the technology and the tools to allow for more reliable threat detection and assessment. 
       BRIEF SUMMARY OF THE INVENTION 
       [0008]    The invention described herein is an affordable, massively parallel camera system that provides sufficient resolution over a complete panoramic view to support threat detection and assessment. This invention is a system of software programs, unique electronic hardware components and a system of fixed reflective lenses. This invention implements a known technology, a “staring array,” using new technology imaging and lens components incorporated into a hierarchical architecture of massively parallel sub-systems. This apparatus is capable of operating in several modes, including a fully automatic threat detection, assessment and alert configuration. In this mode the device can monitor large areas of coverage, conceivably up to a full 360° panorama. 
         [0009]    The implementation can be tailored to fit many situations, from providing complete coverage for a small office, to protecting large assets isolated in open terrain. More or fewer parallel sub-systems can be ganged together to create a camera system that will deliver automated threat detection and alert to properly configured reactive personnel. 
         [0010]    This is a very different supporting construction from traditional video monitoring systems. As confidence grows in the automated detection technology, the force-multiplying effects will enable reactive personnel to cover very large areas. By this means, the system will dramatically enhance the value of the resources spent where they are the most effective, on reactive forces directly countering the threats to our national assets. And the system will minimize the expenditures of resources where they are the least effective: monitoring personnel that can&#39;t possibly cover the large number of monitoring points required. 
     
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       FIG.  1   
       Massively Parallel Optical Automated Threat Detection and Assessment Architecture 
         [0011]      FIG. 1  illustrates the physical architecture of the Massively Parallel Optical Automated Threat Detection and Assessment System. 
           [0012]    Item  1  represents the Imaging Subsystem comprising the lens, the image detection chip, the high speed differential serial interface bus to the image processing FPGA, and the FPGA image processing unit with local RAM storage. The local RAM storage provides a circular buffer for multiple full panoramic image components from this Imaging Subsystem, as well as accommodating circular buffers for each of the 6 tracks that can be managed in the processing resources of each imager. 
           [0013]    Item  2  represents the high-speed differential serial bus that links the vector alert status to the Management unit (Item  5 ). 
           [0014]    Item  3  represents the high-speed differential serial bus to the hard drive unit between the FPGA processor and the hard drive units. 
           [0015]    Item  4  represents the hard disk resource that provides short-term video data storage for multiple Imaging Subsystems (nominally, a 200 GB storage unit will support up to 10 imaging sub-systems). 
           [0016]    Item  5  represents the management unit. The management units may be hierarchically organized to manage multiple Imaging Subsystems and report up and down the chain to effect timely alerts and to evaluate alert responses against a time-line decision table. The management units are not tasked with any image processing responsibilities, but rather they are the link between the automatic threat detection and the “Reactive Personnel” charged with neutralizing or compromising the threat. 
         FIG.  2   
       Processing Functionality within Each Imaging Subsystem 
         [0017]    In this implementation, each imaging chip (nominally a Micron 9 Mega-Pixel chip) feeds a portion of the process. Six of the 7 processing blocks are repeated for every imaging chip in the array. 
           [0018]    Block  7  is used to process the overlapping images to form a stitched image of the adjacent image chips. Block  7  is repeated for every additional image chip to extend the image to (potentially) a full panoramic image. 
           [0019]    Block  1 : The search process is performed in parallel over the full panoramic scene in sections by imaging chip at a configurable “search rate,” nominally  5  fps. This is the basic preparatory processing in spatial, temporal and frequency modes. In this block, circular buffers are constructed of configurable length to enable course (large block) comparisons for movement. When a motion detection is made, more fidelity of the detection is provided though Size Estimators and Velocity Estimators. Velocity Estimators from Computational Imaging use the spatially or wave-coded filters in the lens optical path. 
           [0020]    Block  2 : Detections are tracked at a “tracking rate,” nominally  30  fps. The detections are tracked using a finer-grain comparison block and validated using Computational Imaging to create an updated Velocity Vector Map. 
           [0021]    Block  3 : Tracks are constantly “Assessed” against a “Threat Matrix” as an aid to the decision process that can classify a “Track” as a Threat, which can cause an alert to be transmitted. The Threat Assessment rules are configurable and range from physical barrier transgression, excessive size and position, to ground coupling anomalies (e.g., where a man carrying a heavy load loses the bounce in his step). 
           [0022]    Block  4 : This process manages the Alert transmission and response conformation. This process follows a defined, configurable Alert Matrix to inform the proper responding entity and confirm a response within a time and encroachment barrier. 
           [0023]    Block  5 : The Imaging chip is a nominal multi-Mega-pixel CMOS imager. Current, typical imaging chip might be the Micron 9 Mega-Pixel imager with built-in image adapting technology. This chip adds a new instruction set that enables some of the image configuration that is traditionally done off-chip in software. 
           [0024]    Block  6  is a typical, high speed, differential serial bus between the image chip and the FPGA processor. There is a similar bus connecting the SATA disk drive to the FPGA. 
         FIG.  3   
       Visual Depiction of Process Used in Zooming Tilting and Panning by Skipping or Binning Rows from Imaging Chip in Full Panoramic View 
         [0025]      FIG. 3  depicts two panoramic views, each across multiple Mega-pixel Imaging Subsystems. 
           [0026]    Item  1  depicts how a “zoomed out” image is made from visible light gathered from all over the panoramic array. Rows and columns are selected and others are skipped or averaged and totaled to become the next row or column. This way a zoomed out or wide view transportable image (Item  2 ) is built using energy from all over the array. 
           [0027]    Item  3  depicts how a “zoomed in” image is built using contiguous rows and columns to build a transportable image (Item  4 ). 
           [0028]    Item  5  illustrates how tilt, while zoomed in, moves the contiguous transportable image up the large array to a different vertical perspective. 
         FIG.  4   
       Array of Imaging Structures and Detail of Ray Paths 
         [0029]      FIG. 4  illustrates the principal light gathering physical components. This nominal array is designed for a 140-foot radius trip-line. This means the array will resolve approximately ⅛ th  of an inch per pixel at 140 feet of range. 
           [0030]    Item  1  is a nominal configuration of reflective lenses and imaging chips. This nominal array will provide 360 degrees of azimuth coverage. The disk could be double-sided and then the vertical extent of the design would be 10 degrees instead of 5 degrees. 
           [0031]    Item  2  is a blow-up of a nominal reflective lens made up of four elements potentially being made out of plastic or molded glass and finished machined. 
           [0032]    Item  3  represents the imaging chip. Item  3  is oriented radially from the center of the disc and is in natural alignment with the other imaging chips on the disc. 
       
    
    
     DETAILED DESCRIPTION OF INVENTION 
       [0033]    The central feature of this apparatus is the replicable architecture of the individual lens, imaging chip and the portion of the processing architecture assigned to each lens/imaging chip unit (herein referred to as the “Imaging Subsystem”), creating a highly parallel structure of sub-systems. The architecture supports a family of cameras, each designed using more or fewer of the Imaging Subsystems, chosen so that resolution and field of view sufficient to the target application are delivered by the apparatus. Sufficient resolution and field of view are defined as that which is required to enable fully automated threat detection and the delivery of alerts to a responsive resource with enough temporal margin to enable interdiction or corrective action. 
         [0034]    Key to the utility of this invention is the low-cost reflective lens component of the Imaging Subsystem. Although the core catoptric lens has not changed since Isaac Newton, the implementation of the reflective lens in this apparatus is unique. Computational imaging is used to extend the depth of field of the reflective lenses and to create a depth map of the objects in the field of view. The depth map range information extends the motion detection processing to the creation of a velocity vector map. The velocity vector map, along with computation for estimations of size and ground-coupled stability, is the basis for a novel assessment technology that will categorize threats with an unprecedented level of confidence. [See  FIG. 2 ] 
         [0035]    This low-cost reflective lens technology is capable of working to the full capability of the current technology megaPixel imaging chip. Each lens is coupled to a CMOS imaging chip of significant resolution (&gt;9 MegaPixels). The current CMOS imaging chip has a large pixel count and is built with a new level of integrated image processing technology on the chip itself. The imaging chip feeds detected video to a new generation of FPGA elements that enable an extremely large computational machine with significant local storage to be packaged in a minimal physical area with a very low power requirement. [See  FIG. 1 ] 
         [0036]    The implementation of computational imaging using reflective lenses involves the unique insertion of the coded filter. In traditional refractive lenses, the coded aperture filter is a physical disk of opaque material inserted into the optical path, usually by placing the filter ahead or behind the lens itself. The holes in the filters must be larger than the diffraction limit and optimally placed to enable the efficient mathematical process of image feature enhancement, typically, depth of field extension and range mapping. But for a reflective lens, the filters can be built into the surface of the reflectors themselves. There are many means of implementation. Molded glass reflectors could be directly micro-machined on the surface to effect spatial of even wave-coded filters. Eventually, the ambition to lower cost will machine the filters directly into the surface of plastic molds that will accurately transfer the filter to each lens. 
         [0037]    The utility provided by an automation alert system is judged by the time of warning provided or the trip-line distance from the protected area. As a general statement, being able to sufficiently read a license plate is sometimes considered a threshold of image resolution. The consensus among public safety officials is that 500 feet is a minimum trip-line distance. To be effective, the system must detect threats and provide an alert with enough time allowed for a response before the threat has advanced to close the effective distance between the threat and the protected area to less than 500 feet. A resolution analysis shows us that the lens system must be able to resolve to at least ⅛ th  of an inch per pixel at 700 feet. If each imager provides 2,500 pixels, then approximately 180 lens-imager units will be required to provide a 180° panoramic view. Two such units placed back to back could provide 360° coverage. 
         [0038]    This system by design requires usable connective bandwidth, but does not attempt to deliver captured video images to a head-end for processing. All significant processing is done in the camera. Instead, this system delivers, for further analysis, identified threats not yet fully classified as imminently hostile; and alerts, generated when the threat has violated some established geographical boundary (failed to stay outside of the fence), failed some operational procedure (left a package near the gate), violated some restriction (vehicle too large), or broken some other specifically defined rule for the current application. 
         [0039]    When alerts are detected and the system is equipped with sufficient bandwidth, clear images of the qualifying alert assessments will be delivered instantly. The system will also make use of minimal bandwidth accurate delivery connections to deliver textual messages of the alert to reactive personnel. This means that guards along the perimeter might get exact physical locations and a text-based description of the violation and, if bandwidth permits, pictures of the incident. Because the system has significant local storage, low-latency notification to key reactive personnel can prompt remote viewers to use other or even local higher data rate connectivity to examine or review the full video history of the alert. 
         [0040]    The camera system is novel in the way it captures image data. The system employs a plural format image capture process. A set of video management tools has been created that simultaneously support multiple image data formats. The camera system offers remote viewers a standard 320 by 240 video image that can be controlled in tilt, pan and zoom. Without moving parts, the system has a capability to provide a nominal 10× optical zoom feature. Simultaneous with this steered-beam capture technology the system captures mega-pixel images that can be up to full panoramic in scope. If detections are made that don&#39;t fit well into the automated processes, or if remote human operators need to use the system for intelligence-gathering operations then, given a required minimum bandwidth, the system will allow multiple simultaneous remote viewers to tilt, pan and zoom over the full panoramic scene on a non-interfering basis. 
         [0041]    Unique to this imaging sub-system is the way in which a remote viewer is enabled to traverse around the imaging array by tilt, pan and zoom functions. The very large image array is made up of multiple individual imaging chips (imager), where each imager has a nominal image array size of 2,500 by 3,500 pixels, while a typical standard video image is 320 by 240 pixels. (Again this is nominal, the pixel arrays could be of arbitrary dimension). 
         [0042]    Using the unique capabilities of the CMOS image array to skip rows and columns or alternatively to “bin” rows and columns, wide angle views will be created by selecting the rows and columns of the image as a smooth distribution over the entire area of the array. [See  FIG. 3 ] That is, to create a 320 by 240 image with the widest aperture or the widest field of view (i.e. zoomed all the way out), the rows and columns of the outermost “ring” of the image array will be the outermost “ring” of the created image. Then approximately 10 rows and columns will be skipped (or binned and averaged) and another ring will be selected. This process will be repeated until approximately 320 columns and 240 rows are created for the product image. This technique actually changes the aperture angle in much the same way as a zoom lens does. 
         [0043]    To zoom in, the outermost ring of the created image is taken from a more interior ring of the image array and then fewer of the rows and columns are skipped or binned to create the next ring in the product image. In the limit of this selected component optical zoom, the 320 by 240 image is created using neighboring pixels in the image array. Of course, at the display monitor the presented image can be mapped from created pixel to multiple pixels for a “digital” zoom effect. 
         [0044]    To pan the image, the image array selection of rings is chosen around a different center in the image array. As the chosen row and column rings near the edge of an individual image array the images are created using the data from columns and rows of the neighboring image arrays, if necessary. Thus the pan capability is nearly a full 180°. Similarly, tilt can be realized by moving the center of the image selection grid up or down the array. 
         [0045]    Live viewers and those viewing archived data can indicate or mark a scene. Those scenes with marks will be treated by the system as “directed alerts” and can be revisited at any time to be examined in very high fidelity. The system will select the nearest large format images on either side of the mark-time and allow the viewer to see the images in broad, wide-angle format. The viewer can then zoom in to very fine detail and examine the features of the scene. No matter what the zoom setting of the video stream during capture, the full wide-angle resolution image is captured simultaneously with the video image, but at a lower frame rate (the detection rate). The low frame rate data is available to provide wide-angle reference to reveal relevant activity and also fine-image detail to reveal exact details of a scene. 
         [0046]    The deployed system can be managed remotely by several different system-wide management utilities. Thus the system has hierarchical management capabilities that allow the system to be used in geographical areas composed of many local management entities while allowing these local interests to locally manage sensitive data. All of the data can be routinely managed within local departments. But when or if there is a situation that covers a wider area of concern across multiple departments, the system will allow, with proper managed permissions, access across parallel entities. Thus, video and data on a fleeing suspect who drives from one town to the next can be passed on so that those ahead of him can be given a view of the live or recorded situation; and they may be properly warned or alerted to the situation before the suspect actually enters their region. 
         [0047]    In application, the camera system could be configured to deliver full fidelity images of up to 800×600 to local recording at 30 fps. Simultaneously, the system could be configured record a large format image from each of the imaging chips stitched together to form a full panoramic image. In addition the system may provide a standard video image to a remote live viewer at the bandwidth dependent rate. Remote viewers with appropriate management authorization will have the capability to adjust tilt, pan and zoom settings for both the live and recorded data. 
         [0048]    It is well known in the prior art how to use refractor lenses and a coded spatial filter in conjunction with high-pixel count imaging chips to implement a computational imaging system. It is not well known how to use a reflector lens with a spatial or wave-based filter machined or cast into the surface of the reflector as a basis for a computational imaging system. 
         [0049]    It is well known in the prior art of video recording how to collect video data via a camera. It is also well known in the prior art how to store this data locally and how to stream video back to a location using a wired or wireless capability. What is not well known is how to disseminate alert or warning information in an environment where only minimal wireless networks have coverage. 
         [0050]    It is also not well known how to store video images by storing a complex data structure that includes full panoramic images stored at a detection rate and larger images of each alert stored at the alert monitoring rate. Video transmitted to remote live monitors is usually sized and updated at a frame rate to fit the available bandwidth, but is not stored. A remote viewer that asks to review previously transmitted video can ask to have the video retransmitted and the system will recreate the video stream from the higher fidelity alert video locally stored. 
         [0051]    It is also well known in the prior art how to change the view of a camera by means of a mechanical tilt/pan unit and an optical lens for changing zoom. It is not well known how to implement a tilt/pan/zoom apparatus that does not require mechanical movement or devices. 
         [0052]    It is also not well known how to build a system of reflective lenses and imaging chips to form a panoramic video system that enables an actual tilt, pan and zoom functionality without any moving parts.