Abstract:
A non-frame-based motion detection method and apparatus for imagers requires only a few line buffers and little computation. The non-frame-based, low cost motion detection method and apparatus are well suited for “system-a-chip” (SOC) imager implementations.

Description:
FIELD OF THE INVENTION 
       [0001]    Embodiments of the invention relate to imagers and more particularly to motion detection for imagers. 
       BACKGROUND 
       [0002]      FIG. 1  shows an example surveillance system  10  having a video camera  12 , a controller  14 , an alarm system  16  and a recording device  18 . The camera  12  includes an imager for capturing a viewed scene and a motion detection feature, which outputs a motion signal to the controller  14  when motion is detected in the scene viewed by the camera  12 . Motion detection is a desirable feature in security applications. An immediate benefit of including a motion detection feature in the camera  12  is that the feature helps save recording space by only activating the video recording device  18  (and/or only sending the detected moving region to an encoder for compression) when motion is detected. Another benefit of motion detection is that it allows the camera  12  to send an alert to the security personal in the viewing room causing them to focus on the specific area where motion has been detected. This is especially useful when there are many monitors in the viewing room that are connected to many different cameras, such as e.g., in a casino viewing room. The alarm system  16  could also be triggered by the controller  14  when motion is detected by the camera  12 . 
         [0003]    Conventional motion detection algorithms use temporal processing that compares an image of the current frame with a reference frame image and then counts the number of pixels that are different between the frames. This type of motion detection technique, however, requires a large frame buffer and, therefore, cannot easily be implemented in system-on-a-chip (SOC) imager systems, which have limited memory resources and general circuit area limitations. Moreover, the large frame memory adds cost to the imager, which is also undesirable. 
         [0004]    Accordingly, there exists a need for an improved method and system for motion detection within an imager. There further exists a need for a motion detection system and method that may be implemented in a system-on-a-chip imager system. 
     
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         [0005]      FIG. 1  illustrates an example surveillance system. 
           [0006]      FIG. 2  illustrates an imager (e.g., a system-on-a-chip imager) having non-frame-based motion detection logic/circuitry in accordance with an example embodiment disclosed herein. 
           [0007]      FIG. 3  illustrates an example of detecting motion in a particular area within a frame using weighting. 
           [0008]      FIG. 4  illustrates an example of detecting motion using sub-sampling. 
           [0009]      FIG. 5  shows a processor system incorporating at least one imaging device constructed in accordance with an embodiment disclosed herein. 
       
    
    
     DETAILED DESCRIPTION 
       [0010]    Referring to the figures, where like reference numbers designate like elements,  FIG. 2  illustrates an imager (e.g., a system-on-a-chip imager)  100  having non-frame-based motion detection circuit  120  in accordance with an example embodiment disclosed herein. Since motion detection is not frame-based, a large and expensive frame buffer memory is not required. This is beneficial because it reduces memory requirements, used circuit area and the cost of the imager  100 . Accordingly, the imager  100  is suitable for a system-on-a-chip implementation. In one embodiment, the imager  100  is implemented in a video camera  12  ( FIG. 1 ) such that the camera  12  has non-frame-based motion detection as described below. It should be appreciated that the imager  100  can also be implemented in a digital still camera (or other camera), which can be operated to take multiple images over a period of time (e.g., one or more images per second). 
         [0011]    The imager  100  comprises an image sensor  110  and the motion detection circuit  120 . The image sensor  110  outputs digital data (e.g., pixel data) representing the image captured by the sensor  110  to the motion detection circuit  120 . The image sensor  110  can be a CMOS image sensor, CCD image sensor or other suitable imaging device capable of imaging a scene and outputting a digital representation (i.e., digital image data) of the imaged scene. The digital representation is typically processed by an image processor within the imager before being output as a digital image. CMOS imagers, e.g., are generally known as discussed, for example, in U.S. Pat. No. 6,140,630, U.S. Pat. No. 6,376,868, U.S. Pat. No. 6,310,366, U.S. Pat. No. 6,326,652, U.S. Pat. No. 6,204,524 and U.S. Pat. No. 6,333,205, assigned to Micron Technology, Inc. Suitable CCD and other imagers are also known in the art. 
         [0012]    The motion detection circuit  120  includes four line buffers  122 ,  124 ,  126 ,  128  instead of a frame buffer memory used in prior art motion detection techniques. The line buffers  122 ,  124 ,  126 ,  128  are respectively associated with a current weighted column average (WCA) process  132 , a reference WCA process  134 , a reference weighted row average (WRA) process  136  and a current WRA process  138 . Additional processing included in the motion detection circuit  120  include: two absolute difference processes  142 ,  152 , threshold processes  144 ,  154 , consolidation processes  146 ,  156 , a Cartesian product process  160 , a horizontal camera motion compensation process  148  and a vertical camera motion compensation process  150 , which are described in more detail below. The processes  132 ,  134 ,  136 ,  138 ,  142 ,  144 ,  146 ,  148 ,  150 ,  152 ,  154 ,  156 ,  160  can be performed in logic circuit  121  or a processor included in the motion detection circuit  120 . 
         [0013]    As set forth above, the illustrated embodiment eliminates the frame buffer used for storing the reference frame used in prior motion detection systems. Certain information from the reference frame, however, is still needed because motion detection is essentially a temporal process requiring comparison of current information to reference information. Accordingly, the illustrated embodiment represents the current image data and reference data in a compact way so that the amount of data is reduced while the key information needed for motion detection is retained. As can be seen, at most, only four line buffers  122 ,  124 ,  126 ,  128  are required to implement the disclosed motion detection processing. 
         [0014]    This reduction in memory results from the use of weighted row averages and weighted column averages to represent the reference and current image frames in a compact manner. This way, two-dimensional image data (i.e., row by column data) is reduced to two lines of one-dimensional data (i.e., row only and column only). Instead of using a frame buffer to store an array of reference frame data, the illustrated embodiment uses only two line buffers  124 ,  126  to store separate one-dimensional lines of reference data. Likewise, only two line buffers  122 ,  128  are required to store the one-dimensional current image weighted row and weighted column averages. At the beginning of the disclosed motion detection procedure only the weighted row and column average data of the reference frame are computed and stored. The stored reference weighted row and weighted column average data can be updated as needed during the procedure. 
         [0015]    Let f(x,y) denote a video frame. The weighted row average, denoted by A row (f)(y), is defined in equation (1) as follows: 
         [0000]    
       
         
           
             
               
                 
                   
                     
                       
                         
                           A 
                           row 
                         
                          
                         
                           ( 
                           f 
                           ) 
                         
                       
                        
                       
                         ( 
                         y 
                         ) 
                       
                     
                     = 
                     
                       
                         
                           ∑ 
                           
                             x 
                             = 
                             0 
                           
                           
                             x 
                             = 
                             
                               width 
                               - 
                               1 
                             
                           
                         
                          
                         
                           
                             w 
                             x 
                           
                           · 
                           
                             f 
                              
                             
                               ( 
                               
                                 x 
                                 , 
                                 y 
                               
                               ) 
                             
                           
                         
                       
                       
                         
                           ∑ 
                           
                             x 
                             = 
                             0 
                           
                           
                             x 
                             = 
                             
                               width 
                               - 
                               1 
                             
                           
                         
                          
                         
                           w 
                           x 
                         
                       
                     
                   
                   , 
                 
               
               
                 
                   ( 
                   1 
                   ) 
                 
               
             
           
         
       
     
         [0016]    where “width” is the number of columns in the image frame. The weighted column average, denoted by A col (f)(x), is defined in equation (2) as follows: 
         [0000]    
       
         
           
             
               
                 
                   
                     
                       
                         
                           A 
                           col 
                         
                          
                         
                           ( 
                           f 
                           ) 
                         
                       
                        
                       
                         ( 
                         x 
                         ) 
                       
                     
                     = 
                     
                       
                         
                           ∑ 
                           
                             y 
                             = 
                             0 
                           
                           
                             y 
                             = 
                             
                               height 
                               - 
                               1 
                             
                           
                         
                          
                         
                           
                             v 
                             y 
                           
                           · 
                           
                             f 
                              
                             
                               ( 
                               
                                 x 
                                 , 
                                 y 
                               
                               ) 
                             
                           
                         
                       
                       
                         
                           ∑ 
                           
                             y 
                             = 
                             0 
                           
                           
                             y 
                             = 
                             
                               height 
                               - 
                               1 
                             
                           
                         
                          
                         
                           v 
                           y 
                         
                       
                     
                   
                   , 
                 
               
               
                 
                   ( 
                   2 
                   ) 
                 
               
             
           
         
       
     
         [0017]    where “height” is the number of rows in the image frame. 
         [0018]    Typically, the weights w x  and v y  are set to 1 such that the values of all pixels in the frame are used to compute the weighted row and column averages. The advantage of using weighted averages is that they provide the flexibility of specifying a particular area of interest to monitor for motion detection. For example, an area of interest can be defined by setting the weights for pixels within the area of interest to ‘1’ and setting the weights of the remaining pixels of the frame to ‘0’.  FIG. 3  illustrates an example frame  200  in which the left-most upper quadrant is an area of interest  210  is to be monitored for motion detection. As set forth above, weights w x  and v y  are set to 1 for pixels within the area of interest  210 , while the weights w x  and v y  for all other pixels are set to 0. 
         [0019]    In another embodiment, sub-sampled image data (without actually sub-sampling the image) can be used in the motion detection process. To sub-sample the image data, non-zero weights w x  and v y  are used for a sample of pixels  242  (shown as shaded pixels for illustrative purposes only) in the image frame  240  while all of the weights w x  and v y  for the remaining pixels  244  are set to ‘0’ (as shown in  FIG. 4 ). It should be appreciated that sub-sampling can further reduce the amount of required line buffer memory by at least a half. 
         [0020]    The weighted column averages are used to detect horizontal motion, if any. Let f c (x,y) denote the current frame, and f r (x,y) denote the reference frame. Then the current frame weighted column average is A col (f)(x), which is computed in the current WCA process  132  using equation (2), and the reference frame weighted column average is A col (f r )(x), which is computed in the reference WCA process  134  using equation (2). The computed current frame weighted column average is stored in line buffer  122  while the computed reference frame weighted column average is stored in line buffer  124 . That is, the entire frames are not stored in the circuit  120 . 
         [0021]    Before detecting horizontal motion, it is desirable to estimate and compensate for any horizontal camera movement because the motion detection feature disclosed herein is interested in object motion within the imaged scene, not camera motion. The horizontal camera motion is estimated in the horizontal camera motion compensation process  148 . In process  148 , the SAD (sum of absolute difference) defined in equation (3) below is minimized with respect to the parameter s within a certain range. Although not to be taken as limiting the disclosed embodiment, in a desired embodiment, the range for s is [−16,+16]. 
         [0000]    
       
         
           
             
               
                 
                   
                     S 
                      
                     
                         
                     
                      
                     A 
                      
                     
                         
                     
                      
                     
                       
                         D 
                         H 
                       
                        
                       
                         ( 
                         s 
                         ) 
                       
                     
                   
                   = 
                   
                     
                       ∑ 
                       
                         x 
                         = 
                         0 
                       
                       
                         x 
                         = 
                         
                           width 
                           - 
                           1 
                         
                       
                     
                      
                     
                        
                       
                         
                           
                             
                               A 
                               col 
                             
                              
                             
                               ( 
                               
                                 f 
                                 c 
                               
                               ) 
                             
                           
                            
                           
                             ( 
                             x 
                             ) 
                           
                         
                         - 
                         
                           
                             
                               A 
                               col 
                             
                              
                             
                               ( 
                               
                                 f 
                                 r 
                               
                               ) 
                             
                           
                            
                           
                             ( 
                             
                               x 
                               + 
                               s 
                             
                             ) 
                           
                         
                       
                        
                     
                   
                 
               
               
                 
                   ( 
                   3 
                   ) 
                 
               
             
           
         
       
     
         [0022]    The minimization can be achieved by performing an exhaustive search of s within its range. The value of s that minimizes the SAD H (S) in equation (3), denoted herein as s 0 , represents the horizontal camera motion. To compensate for this horizontal camera motion, the reference frame weighted column average stored in line buffer  124  is shifted by s 0 , i.e., A col (f r )(x+s 0 ). The shifted reference frame weighted column average can be stored in line buffer  124 , replacing the prior reference frame weighted column average, for subsequent use as described below. Alternatively, parameter so can be stored or passed onto subsequent processing without having to replace the reference frame weighted column average already stored in line buffer  124 . It should be appreciated that the horizontal camera motion compensation process  148  (and the vertical camera motion compensation process  150 ) can be skipped if the imager  100  is securely mounted, or otherwise not subject to any motion, if desired. 
         [0023]    As discussed below in more detail, the illustrated embodiment detects horizontal motion by generating a horizontal motion mask. Once the horizontal camera motion has been compensated for in process  148 , the horizontal motion detection process continues at absolute difference process  142 . Absolute difference process  142  computes the absolute difference D H (x) between the weighted column average of the current frame, A col (f c )(x), and the horizontal camera motion compensated weighted column average of the reference frame, A col (f r )(x+s 0 ), as shown below in equation (4): 
         [0000]        D   H ( x )=| A   col ( f   c )( x )− A   col ( f   r )( x+s   0 )|  (4) 
         [0024]    The absolute difference D H (x) is input by threshold process  144 , which thresholds the difference D H (x) to create a thresholded absolute difference M H (x) according to equation (5): 
         [0000]    
       
         
           
             
               
                 
                   
                     
                       M 
                       H 
                     
                      
                     
                       ( 
                       x 
                       ) 
                     
                   
                   = 
                   
                     { 
                     
                       
                         
                           
                             1 
                             , 
                           
                         
                         
                           
                             
                               if 
                                
                               
                                   
                               
                                
                               
                                 
                                   D 
                                   H 
                                 
                                  
                                 
                                   ( 
                                   x 
                                   ) 
                                 
                               
                             
                             &gt; 
                             
                               T 
                               H 
                             
                           
                         
                       
                       
                         
                           
                             0 
                             , 
                           
                         
                         
                           otherwise 
                         
                       
                     
                   
                 
               
               
                 
                   ( 
                   5 
                   ) 
                 
               
             
           
         
       
     
         [0025]    The threshold T H  can be determined before hand based on common motion detection principles or it may be an application specific threshold T H . It should be appreciated that how the threshold T H  is set does not limit the disclosed embodiments. Once the thresholded absolute difference M H (x) is computed, the consolidation process  146  applies morphology-like operators such as closing and opening to respectively fill out small gaps and eliminate small segments in M H (x) to generate the final horizontal motion mask {tilde over (M)} H (x) that consists of significant non-zero connected components. 
         [0026]    The weighted row averages discussed above are used to detect vertical motion, if any. Let A row (f c )(y) and A row (f r )(y) be the weighted row averages for the current frame and the reference frame, respectively, which are respectively computed in processes  138  and  136  using equation (1) discussed above. The computed current frame weighted row average is stored in line buffer  128  while the computed reference frame weighted row average is stored in line buffer  126 . 
         [0027]    Similar to the horizontal motion detection, it is desirable to estimate and compensate for any vertical camera motion before performing vertical motion detection. The vertical camera motion is estimated in the vertical camera motion compensation process  150 . In process  150 , the SAD (sum of absolute difference) defined in equation (6) below is minimized with respect to the parameter t within a certain range. Although not to be taken as limiting the disclosed embodiment, in a desired embodiment, the range for t is [−16,+16]. 
         [0000]    
       
         
           
             
               
                 
                   
                     S 
                      
                     
                         
                     
                      
                     A 
                      
                     
                         
                     
                      
                     
                       
                         D 
                         V 
                       
                        
                       
                         ( 
                         t 
                         ) 
                       
                     
                   
                   = 
                   
                     
                       ∑ 
                       
                         y 
                         = 
                         0 
                       
                       
                         y 
                         = 
                         
                           height 
                           - 
                           1 
                         
                       
                     
                      
                     
                        
                       
                         
                           
                             
                               A 
                               row 
                             
                              
                             
                               ( 
                               
                                 f 
                                 c 
                               
                               ) 
                             
                           
                            
                           
                             ( 
                             y 
                             ) 
                           
                         
                         - 
                         
                           
                             
                               A 
                               row 
                             
                              
                             
                               ( 
                               
                                 f 
                                 r 
                               
                               ) 
                             
                           
                            
                           
                             ( 
                             
                               y 
                               + 
                               t 
                             
                             ) 
                           
                         
                       
                        
                     
                   
                 
               
               
                 
                   ( 
                   6 
                   ) 
                 
               
             
           
         
       
     
         [0028]    The minimization can be achieved by performing an exhaustive search of t within its range. The value of t that minimizes the SAD V (t) in equation (6), denoted herein as t 0 , represents the vertical camera motion. To compensate for this vertical camera motion, the reference frame weighted row average stored in line buffer  126  is shifted by to, i.e., A row (f r )(y+t 0 ). The shifted reference frame weighted row average can be stored in line buffer  126 , replacing the prior reference frame weighted row average, for subsequent use as described below. Alternatively, parameter t 0  can be stored or passed onto subsequent processing without having to replace the reference frame weighted row average already stored in line buffer  126 . 
         [0029]    As discussed below in more detail, the illustrated embodiment detects vertical motion by generating a vertical motion mask. Once the vertical camera motion has been compensated for in process  150 , the vertical motion detection process continues at absolute difference process  152 . Absolute difference process  152  computes the absolute difference D V (y) between the weighted row average of the current frame, A row (f c )(y), and the vertical camera motion compensated weighted row average of the reference frame, A row (f r )(y+t 0 ), as shown below in equation (7): 
         [0000]        D   V ( y )=| A   row ( f   c )( y )− A   row ( f   r )( y+t   0 )   (7) 
         [0030]    The absolute difference D V (y) is input by threshold process  154 , which thresholds the difference D V (y) to create a thresholded absolute difference M V (y) according to equation (8): 
         [0000]    
       
         
           
             
               
                 
                   
                     
                       M 
                       V 
                     
                      
                     
                       ( 
                       y 
                       ) 
                     
                   
                   = 
                   
                     { 
                     
                       
                         
                           
                             1 
                             , 
                           
                         
                         
                           
                             
                               if 
                                
                               
                                   
                               
                                
                               
                                 
                                   D 
                                   V 
                                 
                                  
                                 
                                   ( 
                                   y 
                                   ) 
                                 
                               
                             
                             &gt; 
                             
                               T 
                               V 
                             
                           
                         
                       
                       
                         
                           
                             0 
                             , 
                           
                         
                         
                           otherwise 
                         
                       
                     
                   
                 
               
               
                 
                   ( 
                   8 
                   ) 
                 
               
             
           
         
       
     
         [0031]    The threshold T V  can be determined before hand based on common motion detection principles or it may be an application specific threshold T V . It should be appreciated that how the threshold T V  is set does not limit the disclosed embodiments. Once the thresholded absolute difference M V (y) is computed, consolidation process  156  applies morphology-like operators such as closing and opening to respectively fill out small gaps and eliminate small segments in M V (y). Then, a step of connecting all of the non-zero connected components in M V (y) is performed to generate the final vertical motion mask {tilde over (M)} V (y) that consists of one large non-zero connected component. 
         [0032]    The Cartesian product process  160  inputs the horizontal and vertical motion masks {tilde over (M)} H (x), {tilde over (M)} V (y) and computes and outputs a two-dimensional (2D) motion mask M 2D (x,y). The 2D motion mask M 2D (x,y) is computed using the Cartesian product of the horizontal motion mask {tilde over (M)} H (x) and the vertical motion mask {tilde over (M)} V (y) as set forth in equation (9): 
         [0000]        M   2D ( x,y )= {tilde over (M)}   H ( x )× {tilde over (M)}   V ( y )   (9) 
         [0033]    The 2D motion mask M 2D (x,y) indicates where motion occurred within a frame, with a e.g., a value of ‘1’ indicating motion and value of  0  indicating no motion. The 2D motion mask M 2D (x,y) may be output from the imager  100  ( FIG. 1 ) and used by a controller  14  ( FIG. 1 ) or other system component to e.g., trigger an alarm, turn on a recording device, etc. based on the detected motion. 
         [0034]    When the motion detection process starts, the data from the first captured frame is used as the reference data described above. As can be appreciated, the reference information will need to be updated periodically and/or aperiodically. The rules for updating the reference information may include: a) updating the reference information every T minutes (i.e., periodic update), where T is application specific; and/or b) updating the reference information when the detected motion area becomes greater than a predefined portion of the whole frame area (e.g., ⅔ of the whole frame area) (i.e., an aperiodic update). Updating the reference information merely requires overwriting the prior reference data with the newly calculated weighted row and column averages. 
         [0035]    In comparison to the prior art frame-based motion detection algorithms, the disclosed non-frame-based motion detection method and apparatus only requires a few line buffers (four line buffers or less if sub-sampling is used) of memory instead of a whole frame buffer memory. Moreover, because the above-mentioned processing is performed on one-dimensional data, the computations of the disclosed motion detection method is very fast and inexpensive to implement in comparison to other motion detection systems. 
         [0036]      FIG. 5  shows a processor system  500  incorporating at least one imager  100  constructed and operated in accordance with an embodiment disclosed herein. In a surveillance video system embodiment, the processor system  500  could, for example be a video camera comprising a view finder  534  and a lens system  538  for focusing an image on the pixel array of the imager  100 . The video camera could be activated to continuously image scenes and to perform motion detection as described herein. In another embodiment, the processor system  500  could, for example be a digital still camera comprising a shutter release button  532 , the view finder  534 , a flash  536  and the lens system  538  for focusing an image on the pixel array of the imager  100 . In a handheld video system embodiment, the system  500  would be a video camera with the addition of a start/stop record button instead of the shutter release button  532 . 
         [0037]    The system  500  generally also comprises a central processing unit (CPU)  502 , for example, a microprocessor for controlling functions and which communicates with one or more input/output devices (I/O)  504  over a bus  520 . The CPU  502  also exchanges data with random access memory (RAM)  514  over the bus  520 , typically through a memory controller. The camera system  500  may also include peripheral devices such as a removable memory  506 , which also communicates with CPU  502  over the bus  520 . In the case of a computer system, the system  500  could also include a CD ROM drive  512 . Other processor systems which may employ imagers  100  containing motion detection as described herein, besides still and video cameras, include computers, PDAs, cellular telephones, scanners, machine vision systems, and other systems requiring imaging applications in response to motion detection. 
         [0038]    The above description and drawings illustrate various embodiments It should be appreciated that modifications, though presently unforeseeable, of these embodiments that can be made without departing from the spirit and scope of the invention which is defined by the following claims.