Patent Publication Number: US-8982209-B2

Title: Method and apparatus for operating a video system

Description:
RELATED APPLICATIONS 
     This application hereby claims the benefit of and priority to U.S. Provisional Patent Application No. 61/256,463, titled “METHOD AND APPARATUS TO LEVERAGE VIDEO ANALYSIS TO OPTIMIZE VIDEO COMPRESSION”, filed on Oct. 30, 2009, and which is hereby incorporated by reference in its entirety. 
     This application also hereby claims the benefit of and priority to U.S. Provisional Patent Application No. 61/256,476, titled “METHOD AND APPARATUS TO ENCODE AND STORE VARIABLE RESOLUTION IMAGES PLUS METADATA”, filed on Oct. 30, 2009, and which is hereby incorporated by reference in its entirety. 
     This application also hereby claims the benefit of and priority to U.S. Provisional Patent Application No. 61/256,506, titled “METHOD AND APPARATUS TO USE METADATA TO SUPPORT RAPID SEARCHING”, filed on Oct. 30, 2009, and which is hereby incorporated by reference in its entirety. 
     This application also hereby claims the benefit of and priority to U.S. Provisional Patent Application No. 61/256,535, titled “METHOD AND APPARATUS TO VARY THE FRAME RATE AS NEEDED TO CAPTURE ACTIVITY”, filed on Oct. 30, 2009, and which is hereby incorporated by reference in its entirety. 
     This application also hereby claims the benefit of and priority to U.S. Provisional Patent Application No. 61/256,553, titled “METHOD AND APPARATUS TO VARY THE KEY FRAME INTERVAL TO OPTIMIZE VIDEO STORAGE”, filed on Oct. 30, 2009, and which is hereby incorporated by reference in its entirety. 
     This application also hereby claims the benefit of and priority to U.S. Provisional Patent Application No. 61/256,569, titled “METHOD AND APPARATUS TO STORE A MASTER REFERENCE FRAME IN EACH FILE”, filed on Oct. 30, 2009, and which is hereby incorporated by reference in its entirety. 
    
    
     TECHNICAL BACKGROUND 
     Camera manufacturers have begun offering digital cameras in a wide variety of resolutions ranging up to several megapixels for video recording. These high resolution cameras offer the opportunity to capture increased detail, but potentially at a greatly increased cost in terms of central processing unit (CPU) power, bandwidth, and storage required for high resolution images 
     In order to build the most cost effective solutions for video applications, system designers must leverage available technology to capture and store optimal video evidence as opposed to simply recording video. In the past, video analysis algorithms, video compression algorithms, and video storage methods have all been designed and developed independently. 
     OVERVIEW 
     A method for operating a video source is provided. The method includes capturing video data comprising a plurality of frames of a scene, and identifying a first frame from the plurality of frames that includes a first aspect of an activity occurring within the scene. The method also includes subsequent to identifying the first frame, identifying at least a second frame from the plurality of frames that includes a second aspect of the activity occurring within the scene, and transferring the first frame and the second frame for delivery to storage. 
     A video system comprising a video capture device configured to capture video data comprising a plurality of frames of a scene, a memory configured to store video data, and a video processor coupled with the video capture device and the memory is provided. The video processor is configured to identify from the plurality of frames a first frame that includes a first aspect of an activity occurring within the scene, subsequent to identifying the first frame, identify at least a second frame from the plurality of frames that includes a second aspect of the activity occurring within the scene, and transfer the first frame and the second frame to the memory. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         FIG. 1  illustrates a block diagram of an example of a video system. 
         FIG. 2  illustrates a block diagram of an example of a video source. 
         FIG. 3  illustrates a block diagram of an example of a video processing system. 
         FIG. 4  illustrates a block diagram of an example of a video system. 
         FIG. 5  illustrates an example scene in a video capture system. 
         FIG. 6  illustrates an example scene in a video capture system. 
         FIG. 7  illustrates an example scene in a video capture system. 
         FIG. 8  illustrates an example method of compressing video data in a video system. 
         FIG. 9  illustrates an example method of displaying compressed video data in a video system. 
         FIG. 10  illustrates a flow chart of an example method of operating a video system. 
     
    
    
     DETAILED DESCRIPTION 
     The following description and associated drawings teach the best mode of the invention. For the purpose of teaching inventive principles, some conventional aspects of the best mode may be simplified or omitted. The following claims specify the scope of the invention. Some aspects of the best mode may not fall within the scope of the invention as specified by the claims. Thus, those skilled in the art will appreciate variations from the best mode that fall within the scope of the invention. Those skilled in the art will appreciate that the features described below can be combined in various ways to form multiple variations of the invention. As a result, the invention is not limited to the specific examples described below, but only by claims and their equivalents. 
     Just like different numbers of pixels per foot are required to provide sufficient detail depending on the scene and the purpose of the camera, different numbers of images per second are required to capture activity that occurs at different speeds. 
     Intelligent cameras, such as that illustrated in  FIG. 2 , should capture and analyze images at the maximum frame rate available from the camera, and then based on an analysis of the activity the camera should decide which images need to be retained. Since the camera will not store images at a consistent frame rate, each image should include its own timestamp with millisecond precision.
         When there is very little activity, the camera may drop down to a very low image rate—one frame per second or less.   To track normal movement of people walking through the camera view, a medium frame rate like 2-5 frames per second would be sufficient.   If a person is detected running through the camera view or if a vehicle drives through the camera view at high speed, the camera may need to retain several consecutive images at the maximum image refresh rate to capture sufficient images of the fast-moving subject.   For a camera that views a point-of-sale terminal, conveyor belt movement, merchandise scanning, and cash drawer activity would trigger higher frame rates, then the frame rate would slow down during low activity periods between transactions.       

     Disclosed herein is a video system. Video system  100  includes video source  102 , video processing system  104 , and video storage system  106 . Video source  102  is coupled to video processing system  104 , and video processing system  104  is coupled to video storage system  106 . The connections between the elements of video system  100  may use various communication media, such as air, metal, optical fiber, or some other signal propagation path—including combinations thereof. They may be direct links, or they might include various intermediate components, systems, and networks. 
     In some embodiments, a large number of video sources may all communicate with video processing system  104 , this results in bandwidth concerns as video processing system  104  may have an input port incapable of receiving full resolution, real time video from all of the video sources. In such a case, it is desirable to incorporate some video processing functionality within each of the video sources such that the bandwidth requirements between the various video sources and video processing system  104  are reduced. An example of such a video source is illustrated in  FIG. 2 . 
     In this example embodiment, video source  102  captures video data comprising a plurality of frames of a scene and transfers the video data to video processing system  104 . Video processing system  104  processes the video data to identify a first frame from the plurality of frames that includes a first aspect of an activity occurring within a scene. Example activities include people or objects moving within the scene. A first aspect may include a first view of a person or object within the scene. 
     Video processing system  104  then identifies a second frame from the plurality of frames that includes a second aspect of the activity occurring within the scene. For example, a person may have moved such that their face is now shown within the scene, and video processing system  104  recognizes this as a second aspect or view of the person and identifies the second frame that contains this view of the person. 
     Video processing system  104  then transfers the first frame and the second frame for delivery to storage, such as memory  106 . In this embodiment, video processing system  104  stores only frames that include the activity and that also include new aspects of the activity. This allows for storage of a much compressed encoded video. In other embodiments, video processing system  104  may store additional frames as required for smooth playback or other reasons. For example, video processing system  104  may be configured to store at least one frame per second regardless of whether any new aspects of the activity are detected. 
     At a later point in time, video processing system  104  retrieves the encoded video from storage (memory  106 ) for display on display  110 . There are a number of different ways that the encoded video may be displayed. For example, video processing system  104  may store a time stamp with each frame stored. Then, upon playback of the encoded video each frame is repeatedly shown on the display until a timer reaches the timestamp value of the next frame that was stored. This allows the encoded video to be displayed at a rate substantially equal to that at which the frames were captured. If the selected frames where displayed one after another, the display would very quickly go through the selected frames without allowing a user time to absorb the information (aspects) contained within the selected frames. 
     In other embodiments, variable speed playback may be offered to a user, allowing the user to fast-forward through some portions of the encoded video while slowing down playback for other portions of the encoded video. This variable speed playback may be accomplished by any of a wide variety of means, such as modifying the timer speed, skipping frames, or the like. 
     In some embodiments, video processing system  104  retrieves the encoded video from storage (memory  106 ) and creates synthetic frames to fill in gaps between the selected, stored frames. These synthetic frames may simply be copies of a previous frame used for display until the time arrives at which the next frame stored in memory  106  is to be displayed. See  FIG. 9  for a graphical example of how video processing system  104  may fill in frames that were not stored in memory  106 . 
       FIG. 2  is a block diagram of an example of a video source  200 , such as video source  102  from  FIG. 1 . Video source  200  includes lens  202 , sensor  204 , processor  206 , memory  208 , and communication interface  210 . Lens  202  is configured to focus an image of a scene on sensor  204 . Lens  202  may be any type of lens, pinhole, zone plate, or the like able to focus an image on sensor  204 . Sensor  204  then digitally captures video of the scene comprising a plurality of frames and passes the video frames to processor  206 . Processor  206  is configured to store some or all of the video in memory  208 , process the video, and send the processed (encoded) video to external devices  212  through communication interface  210 . In some examples, external devices  212  include video processing system  104  and video storage system  106 . 
     In this example embodiment, video source  200  is configured to process video data and to determine which frames of video data to store. Lens  202  and sensor  204  are configured to capture video data of a scene at a normal (or maximum) frame rate and transfer the first video data to processor  206 . Processor  206  processes the first video data to identify a first frame from the plurality of frames that includes a first aspect of an activity occurring within the scene. Processor  206  then identifies a second frame from the plurality of frames that includes a second aspect of the activity occurring within the scene, and video source  200  then transfers the first frame and the second frame through communication interface  210  for transfer to storage in one or more external devices  212 . 
     Many embodiments include a video processing system such as video processing system  104  from  FIG. 1 , processor  206  from  FIG. 2  and video processing system  410  from  FIG. 4 . Any of these video processing systems may be implemented on a computer system such as that shown in  FIG. 3 . Video processing system  300  includes communication interface  311 , and processing system  301 . Processing system  301  is linked to communication interface  311  through a bus. Processing system  301  includes  302  and memory devices  303  that store operating software. 
     Communication interface  311  includes network interface  312 , input ports  313 , and output ports  314 . Communication interface  311  includes components that communicate over communication links, such as network cards, ports, RF transceivers, processing circuitry and software, or some other communication devices. Communication interface  311  may be configured to communicate over metallic, wireless, or optical links. Communication interface  311  may be configured to use TDM, IP, Ethernet, optical networking, wireless protocols, communication signaling, or some other communication format—including combinations thereof. 
     Network interface  312  is configured to connect to external devices over network  315 . In some examples these network devices may include video sources and video storage systems as illustrated in  FIGS. 1 and 4 . Input ports  313  are configured to connect to input devices  316  such as a keyboard, mouse, or other user input devices. Output ports  314  are configured to connect to output devices  317  such as a display, a printer, or other output devices. 
     Processor  302  includes microprocessor and other circuitry that retrieves and executes operating software from memory devices  303 . Memory devices  303  include random access memory (RAM)  304 , read only memory (ROM)  305 , a hard drive  306 , and any other memory apparatus. Operating software includes computer programs, firmware, or some other form of machine-readable processing instructions. In this example, operating software includes operating system  307 , applications  308 , modules  309 , and data  310 . Operating software may include other software or data as required by any specific embodiment. When executed by processor  302 , operating software directs processing system  301  to operate video processing system  300  as described herein. 
     In this example embodiment, video processing system  300  receives video data comprising a plurality of frames of a scene from one or more video sources (input devices  316 ) through input ports  313 , and communication interface  311 . Processor  302  then identifies a first frame from the plurality of frames that includes a first aspect of an activity occurring within the scene. Processor  302  then identifies a second frame from the plurality of frames that includes a second aspect of the activity occurring within the scene. Processor  302  then transfers the first frame and the second frame to memory devices  303  for delivery to storage or to network interface  312  for storage on network  315 . 
       FIG. 4  illustrates a block diagram of an example of a video system  400 . Video system  400  includes video source  1   406 , video source N  408 , video processing system  410 , and video storage system  412 . Video source  1   406  is configured to capture video of scene  1   402 , while video source N  408  is configured to capture video of scene N  404 . Video source  1   406  and video source N  408  are coupled to video processing system  410 , and video processing system  410  is coupled to video storage system  412 . The connections between the elements of video system  400  may use various communication media, such as air, metal, optical fiber, or some other signal propagation path—including combinations thereof. They may be direct links, or they might include various intermediate components, systems, and networks. 
     In some embodiments, a large number of video sources may all communicate with video processing system  410 , this results in bandwidth concerns as video processing system  410  may have an input port incapable of receiving full resolution, real time video from all of the video sources. In such a case, it is desirable to incorporate some video processing functionality within each of the video sources such that the bandwidth requirements between the various video sources and video processing system  410  are reduced. An example of such a video source is illustrated in  FIG. 2 . 
     In some embodiments including a large number of video sources it may be advantageous to configure the video sources to determine a desired frame rate for the video data that they capture. In other embodiments, the video sources may transfer the raw video data to video processing system  410  for identification of frames showing different aspects of an activity and storage in video storage  412 . 
       FIG. 5  illustrates an example scene in a video capture system. In this example illustration of a video capture system,  FIG. 5(   a ) illustrates a first frame  500  of a scene containing a convenience store and a person  502 . In this example, the activity occurring within the scene is a person moving within the scene. First frame  500  captures a first aspect of the person moving within the scene, a view of the back of the person. 
     Later, as shown in  FIG. 5(   b ), second frame  504  is identified by a video processor as including a second aspect of the person  506  moving within the scene, a view of the face of the person. In this example, first frame  500  and second frame  504  are stored in a memory for later display. Any intervening frames are determined not to contain any additional aspects of the activity (the person moving within the scene) and so are not stored in the memory. Thus, only frames containing unique aspects of the activity occurring within the scene are stored in memory. 
       FIG. 6  illustrates an example scene in a video capture system. In this example illustration of a video capture system,  FIG. 6(   a ) illustrates a first frame  600  of a scene containing an apartment building and a car  602 . In this example, the activity occurring within the scene is a car moving within the scene. First frame  600  captures a first aspect of the car moving within the scene, a view of the car located on the right side of the scene. 
     Later, as shown in  FIG. 6(   b ), second frame  604  is identified by a video processor as including a second aspect of the car moving within the scene, a view of the car located on the left side of the scene. In this example, first frame  600  and second frame  604  are stored in a memory for later display. Any intervening frames are determined not to contain any additional aspects of the activity (the car moving within the scene) and so are not stored in the memory. Thus, only frames containing unique aspects of the activity occurring within the scene are stored in memory. 
     Other frames of the video data may capture the license plate of the car and the video processor would recognize the license plate as a third aspect of the activity within the scene and would store a third frame including the license plate in the memory. Still other aspects of this example scene may include people entering or leaving the car, and frames including these aspects of the activity would also be identified by the video processor. 
       FIG. 7  illustrates an example scene in a video capture system. In this example illustration of a video capture system,  FIG. 7(   a ) illustrates a first frame  700  of a scene containing a conveyor belt  702  and an object  704 . In this example, the activity occurring within the scene is object  704  on conveyor belt  702  moving within the scene. First frame  700  captures a first aspect of object  704  moving within the scene, a view of front surface of object  704 . 
     Later, as shown in  FIG. 7(   b ), second frame  706  is identified by a video processor as including a second aspect of object  704  moving within the scene, a view of a side of object  704  displaying a biohazard sticker  708 . The presence of biohazard sticker  708  is an additional aspect of object  704  and thus is identified by the video processor as a second aspect of the activity occurring within the scene. In this example, first frame  700  and second frame  706  are stored in a memory for later display. Any intervening frames are determined not to contain any additional aspects of the activity (object  704  moving within the scene) and so are not stored in the memory. Thus, only frames containing unique aspects of the activity occurring within the scene are stored in memory. 
       FIG. 8  illustrates an example method of compressing video data in a video system. In this example, video data comprising N frames  800 - 812  of a scene are processed by a video processor to identify frames containing unique aspects of one or more activities occurring within the scene. In this example, frame  1   800  includes a first aspect of an activity occurring within the scene and is selected for storage in memory along with an associated timestamp  814 . Frame  2   802  and frame  3   804  do not include unique aspects of the activity occurring within the scene and are not stored in memory. 
     Frame  4   806  includes a second aspect of the activity occurring within the scene and is selected for storage in memory along with an associated timestamp  816 . Frame  5  through frame M−1 do not include unique aspects of the activity occurring within the scene and are not stored in memory. Frame M  808  includes a third aspect of the activity occurring within the scene and is selected for storage in memory along with an associated timestamp  818 . Likewise, frame N  812  includes a fourth aspect of the activity occurring within the scene and is selected for storage in memory along with an associated timestamp  820 . 
     Frame M+1  810  does not include unique aspects of the activity occurring within the scene and is not stored in memory. Thus, frame  1   800 , frame  4   806 , frame M  808 , and frame N  812  are the only frames including unique aspects of the activity occurring within the scene and are the only four frames in this example stored in memory. In some embodiments, a single video data file may contain frames including multiple different activities. In such an example, all frames containing unique aspects of each of the activities are stored in memory. 
       FIG. 9  illustrates an example method of displaying compressed video data in a video system. In this example, the four frames stored in memory as illustrated in  FIG. 8  are now prepared for display. Frame  1   800  along with timestamp  1   814  is read from memory. Frame  1   800  is initially displayed on the display at time  1 , and a timer based on timestamp  1  is started. The video processing system continues to display frame  1   800  on the display until the timer reaches a time value corresponding to timestamp  4   816  corresponding to frame  4   806  at time  4 . At time  4  frame  4   902  is displayed and the timer continues to run. When the timer reaches a time value corresponding to timestamp M  818  for frame M  808  at time M, frame M  808  is displayed. Likewise, then the timer reaches a time value corresponding to timestamp N  820  for frame N  812  at time N, frame N  812  is displayed. 
     In some embodiments, the video processing system sends frame  1   800  three times to the display at times  1 ,  2 , and  3 . In other embodiments, the display is configured to continuously display a frame until it receives a new frame. In such an embodiment the video processor sends frame  1   800  once at time  1  and then sends frame  4   806  to the display at time  4 . 
       FIG. 10  illustrates a flow chart of an example method of operating a video system. In this example method, video data comprising a plurality of frames of a scene is captured by a video capture device, (operation  1000 ). A video processor identifies a first frame from the plurality of frames that includes a first aspect of an activity occurring within the scene, (operation  1002 ). 
     The video processor then identifies a second frame from the plurality of frames that includes a second aspect of an activity occurring within the scene, (operation  1004 ), and transfers the first frame and the second frame for delivery to storage, (operation  1006 ). 
     Sophisticated video surveillance systems should not simply record video. Systems should be designed to gather optimal visual evidence that can be used to solve crimes or investigate incidents. Systems should use video analysis to identify specific types of activity and events that need to be recorded. The system should then tailor the recorded images to fit the activity—providing just the right level of detail (pixels per foot) and just the right image refresh rate for just long enough to capture the video of interest. The system should minimize the amount of space that is wasted storing images that will be of little value. 
     In addition to storing video images, the system should also store searchable metadata that describes the activity that was detected through video analysis. The system should enable users to leverage metadata to support rapid searching for activity that matches user-defined criteria without having to wait while the system decodes and analyzes images. All images should be analyzed one time when the images are originally captured (before compression) and the results of that analysis should be saved as searchable metadata. 
     Modern video compression methods like H.264 involve detecting changes between images and then determining the optimal way to encode those changes. A large image is divided into a grid of small images called macroblocks, where a typical macroblock size is 8×8 pixels. When the pixels of a macroblock have changed from one image to the next, the compression algorithm can choose between two methods of encoding the macroblock:
         Inter-encoding means the compression algorithm finds a macroblock in the previous image (the reference image) that closely matches the macroblock to be encoded. The encoding for the new macroblock consists of a motion vector (which identifies the location of the referenced macroblock in the reference image) and a compressed representation of the difference between the encoded macroblock and the referenced macroblock.   Intra-encoding means the macroblock is encoded without reference to any other macroblock. This approach can be used if the algorithm is unable to identify a motion vector that provides more optimal encoding.       

     The main characteristic that distinguishes a more effective compression algorithm from a less effective one is the degree to which the algorithm always finds the optimal motion vector for each macroblock to be encoded. If the amount of time and CPU power available for searching is unlimited, then the encoder can simply perform an exhaustive evaluation of all macroblocks in the reference image in order to identify the motion vector that will produce the smallest encoding. More advanced algorithms can take it one step further by using multiple reference images before and sometimes even after the image being encoded. More advanced algorithms can also select fractional pixel displacements for motion vectors (because moving objects won&#39;t always exactly align with macroblock boundaries). 
     Using reference images after the image being encoded would not be an option if latency must be minimized when encoding live video images—for live video, the current image must be completely encoded and transmitted before the next image is captured. 
     In general, the CPU power currently available in digital cameras for surveillance and other applications is not sufficient for the most advanced video encoding methods. The camera must use a highly optimized method to locate acceptable motion vectors quickly. Even within cameras that implement the same encoding standard (like H.264), the quality of the implementation can vary significantly—some algorithms find more optimal motion vectors more quickly than other algorithms. 
     For any macroblock, the ideal motion vector would be one that identifies the location of an identical macroblock in the reference image. That can occur when an object has moved within the camera view but otherwise the pixels representing the object have not changed. In general, if you know that an object is moving within the camera view, then the macroblocks that made up that same object in the reference image should be among the most promising motion vectors. 
     For video surveillance, one is normally interested in moving objects—in particular we are interested in people and their activity. One is also interested in moving vehicles (because they are operated by people). Video analysis algorithms attempt to make sense out of the pixels in an image. These algorithms separate pixels into distinct objects (people, vehicles, etc.) and track the movement, behaviors, and interaction of objects. 
     It seems clear that if a video analysis algorithm has identified distinct objects and the movement of those objects from image to image, then that information should provide an ideal basis for selecting optimal motion vectors for image encoding. 
     In an example embodiment, a method for operating a video source is provided. The method includes capturing first video data of a scene including a moving object, and processing the first video data to determine a motion vector for the moving object. The method also includes capturing second video data of the scene including the moving object, and compressing the second video data using the motion vector producing compressed second video data. 
     Within a particular high resolution image, portions of the image may contain information that is relevant to surveillance (moving people and vehicles) while the majority of the image may depict relatively static portions of the scene that closely match thousands of previous images from the same camera. 
     One way to handle a fixed surveillance camera, such as that illustrated in  FIG. 2 , would be to store a complete image at maximum resolution only at a very long interval—maybe once every few hours for example—to capture the details of the static objects within the scene. These images could be stored separately and could be made available for reference when an operator is reviewing recorded video from the camera. Except for these reference images, all other images would use the following technique:
         Capture an image at maximum resolution   Identify the regions of the image that contain objects of interest where maximum resolution may provide value. Divide those regions into macroblocks and encode those macroblocks at full resolution. Macroblocks may be 8 pixels by 8 pixels in size in some examples, while other embodiments may user macroblocks of other sizes.   Scale the image down to a lower resolution image—e.g. one half or one quarter the size of the original. Divide the lower resolution image into macroblocks and encode those macroblocks. If a macroblock in the lower resolution image corresponds completely to macroblocks already encoded at a higher resolution, that macroblock can be excluded from the lower resolution image.       

     It would be possible to apply this technique repeatedly so that a single image might be encoded in three or more different resolutions as opposed to only two resolutions. 
     Various law enforcement organizations including the FBI have published guidelines for the video resolution needed for specific purposes. These guidelines are expressed in terms of pixels per foot.
         Overview detail is in the range of 20 to 30 pixels per foot. This is sufficient detail to track people and their movement but is not sufficient detail to recognize faces.   Forensic detail is sufficient detail to serve as legal evidence in order to identify a particular person or read a license plate. Most sources recommend at least 40 pixels per foot to recognize faces and at least 60 pixels per foot to read license plates.   High detail is sufficient to identify specific currency or casino chip values or small items being purchased at a point-of-sale terminal. High detail is 80 or more pixels per foot.       

     By combining knowledge about the role of the camera in the surveillance system with video analysis, an intelligent camera could encode images at variable resolution in order to provide the optimum resolution for the activity that is currently visible within the camera view.
         If a video analysis algorithm detects that an original high resolution image includes a clear view of a person&#39;s face and the image provides at least 40 pixels per foot in the area where the face is detected, then the camera could retain full resolution for the macroblocks required to encode the image of the face. The camera could also generate metadata (where metadata is additional data stored separately from video images, describing the contents of the video images) to indicate that the image includes a face encoded with forensic detail.   If a camera uses video analysis to identify and track individuals, then the camera could capture a limited number of images of each person&#39;s face at forensic detail. The camera could identify and eliminate duplicate images showing exactly the same viewing angle of the person&#39;s face, but could retain additional images if images are captured at different viewing angles as the person moves within the camera view.   If a camera uses video analysis to identify license plates within the camera view, then the camera could retain an image of the license plate at forensic detail whenever that much detail becomes available. After an image has been captured at sufficient detail to read the license plate, there is no need for the camera to store additional images of the same license plate at forensic detail. Other images of the same vehicle should provide sufficient detail to see the make and model of the vehicle, the movement of the vehicle, and any visible information about the occupants of the vehicle.   For a camera that views a point-of-sale terminal, the camera could retain high detail of items on the conveyor and of cash-handling activity when the cash drawer is open. The camera could retain forensic detail of customers, employees, and merchandise scanning and bagging activity. Since employees typically enter the scene and then remain for long periods, there would be no need for the camera to repeatedly capture forensic detail of the employee&#39;s face—the camera should only retain enough detail to positively identify the employee when they arrive on the scene. For any uninterrupted sequence of images of the same employee, the metadata should identify the images that contain forensic detail to identify that employee. The camera would retain overview detail during periods when there is no customer present, the cash drawer is closed, and there is no activity of any known significance.       

     During video review, images could be displayed at low resolution initially. The low resolution macroblocks would be decoded first. Higher resolution macroblocks would be decoded and then scaled down to match the lower resolution macroblocks. The operator could select a portion of the image to be viewed in more detail. The application would determine if higher resolution macroblocks exist for that area, and would decode those macroblocks at full resolution. The application could automatically adjust the zoom level so that the selected area could be shown at full resolution—i.e. each pixel in the decoded image would correspond to one pixel in the displayed image. 
     If requested, the application could highlight the portions of the image where higher resolution is available for display. The application could leverage metadata generated and stored during video encoding to highlight objects and specific types of activity in the displayed video. The operator could keep the overview image displayed in one window and use separate windows to display regions of interest at higher resolution. 
     In an example embodiment, a method for operating a video source is provided. The method includes capturing video data of a scene, and processing the video data to determine an area of interest within the scene. The method also includes dividing the video data into macroblocks, determining the identity of the macroblocks including the area of interest, and encoding the macroblocks including the area of interest at a first resolution. The method further includes encoding the remaining macroblocks at a second resolution, wherein the first resolution is greater than the second resolution. 
     If a system, such as that illustrated in  FIG. 1 , uses video analysis to generate metadata during video encoding, that metadata can support rapid scanning and searching for specific types of activity. For example, an operator might want to review video of a doorway and see every person that went through that door during a period of time. If the metadata contains a history of the times at which the camera captured forensic detail of a new person arriving in the scene, then the application could leverage the metadata to quickly display images of each person. Searching metadata would be many times faster (and more efficient with system resources) than decoding and analyzing video images to detect activity. For example, what if only one person goes through that door on average in a 10 hour period? Instead of decoding and analyzing 10 hours worth of video images to find the next event of interest, the system might only need to search through a few kilobytes of metadata. 
     In an example embodiment, a method for operating a video source is provided. The method includes capturing video data of a scene, detecting an event within the video data, and placing an event timestamp including the date and time of the event in metadata corresponding to the video data. 
     Modern video compression methods like H.264 produce two types of compressed images:
         I-Frames or intra-encoded frames are frames that can be decoded without referencing any other frame. I-Frames are also known as key frames.   P-Frames or predicted frames are frames that require one or more reference frames during decoding. P-Frames are also known as delta frames because they store the differences between frames.       

     A Group of Pictures or GOP is a sequence of images consisting of a key frame followed by the delta frames that depend on that key frame. The number of images in a GOP is the key frame interval. For example, if the key frame interval is 32, that means one out of every 32 images is a key frame. 
     It is common to use a fixed key frame interval, but an intelligent encoding algorithm could produce better results by selecting the optimum number of images to include in each GOP. When activity in the camera view is low, P-Frames will be small because very few macroblocks need to be encoded for each new image. In this situation, an intelligent camera, such as that illustrated in  FIG. 2 , can keep the video data rate as low as possible by continuing to extend the current GOP instead of encoding a key frame to start a new GOP. 
     The encoder should only encode a key frame to start a new GOP when there is a reason for doing that—the encoder should not adhere to a fixed key frame interval. Following are valid reasons for starting a new GOP:
         Starting a new video file. Each video file must start with a key frame if a goal is to be able to decode all images in a video file without referencing any other video file.   Random access performance. During video playback, long sequences of images in a GOP can decrease responsiveness when a user attempts to reposition to a particular image within a GOP. To decode any particular image, the decoder must start with the key frame at the beginning of the GOP and then decode each successive image until the target image has been decoded. The maximum time required for random access to a particular image will depend on the maximum number of macroblocks that must be decoded in order to decode the selected image. Every GOP requires decoding all of the macroblocks in the key frame. After that, the number of macroblocks to be decoded depends on the amount of activity (which determines the number of macroblocks per image) and the number of images. When there are few macroblocks per delta frame because of low activity, the GOP can be extended to a larger number of images without causing excessive random access times. Conversely, when there is very high activity it may be necessary to reduce the number of images in a GOP in order to maintain an acceptable maximum random access time.   Accumulation of encoding errors. The encoding algorithm uses lossy compression when encoding the differences between images. Over time, the image that results from decoding successive delta frames can diverge from the image that would result from decoding a new key frame. If an intelligent camera has sufficient CPU power, it would be possible to encode a key frame for each captured image and then measure the encoding error of the image that would result from decoding the delta frames vs. the image that would result from decoding the key frame. To avoid having to transmit a new key frame, the encoder could select the macroblocks with the worst errors and encode and transmit those (using intra-encoding at the macroblock level if needed). If a large number of macroblocks have errors exceeding a threshold level, then it is time to end the current GOP and encode a new key frame.       

     In an example embodiment, a method for operating a video source is provided. The method includes capturing video data of a scene comprising a plurality of frames, opening a file for the video data, and storing a key frame in the file corresponding to a first of the plurality of frames. For each of the remaining plurality of frames, the method encodes the frame into macroblocks, determines a quantity of total macroblocks encoded since a last key frame was stored, and determines a quantity of macroblocks having errors since the last key frame was stored. The method also includes storing a new key frame when the quantity of total macroblocks encoded since the last key frame was stored exceeds a total encoded threshold, and storing a new key frame when the quantity of macroblocks having errors since the last key frame was stored exceeds an error threshold. 
     For fixed cameras, such as that illustrated in  FIG. 2 , viewing a scene with consistent lighting, there may be a great deal of redundancy between key frames. Many of the macroblocks in each key frame will depict unchanging portions of the camera view. Repeatedly encoding those fixed portions of the scene wastes space. However, it is necessary to periodically encode a key frame in order to support efficient random access to any video image. 
     To reduce the video redundancy for fixed cameras, a possible approach would be as follows:
         Identify the unchanging portions of the camera view. Save macroblocks from one or more images to build up a reference image. Each macroblock within the reference image should be the most common view of that portion of the picture.   Each video file will start with a master reference image that uses intra-encoding. This master reference image is never displayed—it only serves as raw material for encoding other images.   The frame at the beginning of each group of pictures (GOP) will be encoded using inter-encoding from the reference frame instead of the typical intra-encoding used for key frames. That is, calculate and store the differences between the reference image and the current image in exactly the same way a delta frame would be encoded. If there are a lot of closely matching macroblocks between the reference image and the image being encoded, then the resulting compressed image should be significantly smaller than a key frame.       

     For any macroblock, the encoder always has the choice of using intra-encoding if the encoder is unable to find a suitable motion vector. Therefore, encoding the initial frame of a GOP using this approach should not create results that are worse than standard key frame encoding. Some macroblocks within the reference image could be flagged to indicate that the encoder should always use intra-encoding for that macroblock. For example, macroblocks in regions of the camera view that have constant activity might use that flag. 
     To improve encoding efficiency for the first frame of each GOP, the encoder could bypass motion vector searching and use the following simpler method. Encode each macroblock using two approaches and retain the best (smallest) of the two encoded results:
         1. Intra-encoding (the method normally used for all macroblocks in a key frame)   2. Inter-encoding relative to the master reference image, where the only motion vector evaluated is 0, 0 (no horizontal or vertical displacement).       

     If there is no significant change between a macroblock in the reference image and the macroblock being encoded, the encoder can skip the macroblock entirely. During decoding, when a macroblock is skipped in the encoded image the decoder would simply copy the macroblock from the master reference image. 
     In an example embodiment, a method for operating a video system to generate and store encoded video comprising a plurality of key frames and a plurality of delta frames is provided. The method includes capturing video data comprising a plurality of frames of a scene, and selecting a fundamental view of at least a portion of the scene contained in the plurality of frames. The method also includes generating the plurality of key frames from a first subset of the plurality of frames and the fundamental view of at least the portion of the scene, generating the plurality of delta frames from a second subset of the plurality of frames and the plurality of key frames, and transferring the encoded video for delivery to storage. 
     The above description and associated figures teach the best mode of the invention. The following claims specify the scope of the invention. Note that some aspects of the best mode may not fall within the scope of the invention as specified by the claims. Those skilled in the art will appreciate that the features described above can be combined in various ways to form multiple variations of the invention. As a result, the invention is not limited to the specific embodiments described above, but only by the following claims and their equivalents.