Abstract:
Surveillance systems often encounter great challenges from lighting variations, especially for those inspecting outdoor environments. To construct a surveillance system robust to various background scene changes, including lighting variations, a strategy of background model learning is widely adopted. Based on this strategy, many approaches have been proposed in decades to represent background scenes by statistical models and to adapt background changes over time into the models. However, the focus of most background model learning research is put on adaptation of scene vibrations in to background, as well as of gradual lighting variations. For the background model adaptation to drastic lighting changes, many background model learning approaches are often inefficient. As a result, false alarms in foreground detection are issued under such quick lighting changes. To suppress this kind of false alarms, a new system design of background model learning is proposed.

Description:
REFERENCES 
       [0000]    
       
         C. Stauffer and W. Grimson, “Adaptive background mixture models for real-time tracking,” in  Proc. IEEE Conf. Computer Vision and Pattern Recognition , vol. 2, 1999, pp. 246-252. 
       
     
       BACKGROUND OF THE INVENTION 
       [0002]    The present invention relates generally to a video surveillance system using background subtraction to identify foreground abnormalities. Particularly, the present invention relates to the learning of a background model for a scene of interest that is robust to drastic lighting changes. 
         [0003]    For video surveillance systems using background subtraction to identify foreground abnormalities, the computation of a proper background model over time plays an important role that dominates the system performance. For applications of long-term video surveillance, the computation of an up-to-date background model is needed to prevent incorrect detections of background scene changes, e.g., lighting variations from dawn to night, as foreground alarms. To this end, periodic updating of background model estimates is often adopted by many surveillance systems to learn background scene changes. We regard the approach of periodic background model updating as background model learning. 
         [0004]    In last decades, many approaches addressing the problem of background model learning, e.g., C. Stauffer and W. Grimson, “Adaptive background mixture models for real-time tracking,” in  Proc. IEEE Conf. CVPR , vol. 2, pp. 246-252, 1999, have been proposed. For most background model learning systems, the frequencies of background model updating (for all pixel locations), alSo known as the model learning rates, have large effects on the system stability. (The definition of model learning rate is inherited from C. Stauffer and W. Grimson, “Adaptive background mixture models for real-time tracking,” in Proc. IEEE Conf. CVPR, vol. 2, pp. 246-252, 1999.) Through periodic background model updating, various background scene changes, including lighting variations, resting objects, etc., will be adapted into a background model. The more frequently a background model is updated; the more scene variations are adapted into the learnt model, which results in a surveillance system (based on background subtraction) being more robust to background interferences and less sensitive to foreground abnormalities. As a result, applying high learning rates to a background model learning system will result in insensitive foreground detections to intruders in slow motion, because the system tends to incorporate slowly-moving intruders into its background model via frequent model update. 
         [0005]    As most background model learning systems need to operate at moderate, but not high, learning rates to fit general surveillance conditions, gradual (and perhaps is quick) lighting variations may thus be captured by the background models computed from these systems. However, for some over-quick, also regarded as drastic, lighting changes induced by, e.g., sudden sunshine varying, such systems may then become inefficient. Many false detections of foreground regions resulted from drastic lighting changes will hence be generated. The lack of efficient and effective ways in handling drastic lighting changes for general background model learning approaches motivates this invention. 
         [0006]    To enhance the background model adaptation to lighting changes for general background model learning systems, a new system design is proposed by acquiring a lighting change processing unit as a post-processing module for these systems to revise their foreground region detection results under drastic lighting changes and to guide the dynamic adjustments of their model learning rates. With the proposed post-processing module, the capability of model adaptation to drastic lighting changes for general background model learning systems can be largely improved, without the need of manual tuning of model learning rates. 
         [0007]      FIG. 1  gives two examples of quick and over-quick (drastic) lighting changes. The image sequence shown in  FIGS. 1  ( a )-( c ) records a laboratory with a CRT monitor displaying rolling interfaces and with quick lighting changes, wherein  FIGS. 1  ( a )-( c ) show two images of image sequence I A  recorded at 20 fps experiencing quick lighting changes. In this image sequence, it takes about three seconds to increase the average intensity by ˜20%. This kind of quick variation in image brightness is able to be adapted by common background model learning approaches, e.g., C. Stauffer and W. Grimson, “Adaptive background mixture models for real-time tracking,” in  Proc. IEEE Conf. CVPR , vol. 2, pp. 246-252, 1999, by applying a higher learning rate (than the default value suggested in the paper). See  FIG. 2  ( a ) for the simulated foreground region detection results for image frame I t   A  by applying Stauffer and Grimson&#39;s method. On the other hand, for the case of over-quick lighting changes shown in  FIGS. 1  ( d )-( e ), which show two images of image sequence I B  recorded at 15 fps experiencing over-quick (drastic) lighting chances, similar increases of image intensity are observed in less than one second for an outdoor environment. As shown in  FIG. 2  ( b ) which shows foreground region detection results for the image frame I t   B  over-quick lighting, many false detection results in foreground are received under such a condition by using the same approach with an identical learning rate setting. Instead, by applying the proposed background model learning system according to one embodiment of this invention, most false positives of foreground region detection are eliminated, as shown in  FIG. 2  ( c ). 
       SUMMARY OF THE INVENTION 
       [0008]    The proposed invention addresses a new design of a background model learning system for lighting change adaptation that can be utilized in video surveillance applications. Specifically, the proposed system adopts any one of general background modeling learning approaches as its basic processing units for background model estimation and foreground region detection, and then acquires a lighting change processing unit as a post-processing module for revising the foreground region detection. The lighting changes processing unit is designed for reducing false foreground detections resulting from gradual, quick and drastic lighting changes. The revised foreground detection results by the lighting change processing unit can then be applied to next processing of, e.g., foreground object analysis, and sent back to the basic unit of background model estimation for dynamic adjustments of model learning rates. 
     
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         [0009]      FIGS. 1  ( a ), ( b ) and ( c ) show two snapshots of an image sequence in quick lighting changes and their image intensity variations via frame differencing; 
           [0010]      FIGS. 1  ( d ), ( e ) and ( f ) show two snapshots of an image sequence in over-quick lighting changes and their image intensity variations via frame differencing; 
           [0011]      FIGS. 2  ( a ) and ( b ) show the foreground region detection results for the two image sequences in quick and over-quick lighting changes, respectively, by using Stauffer and Grimson&#39;s background model learning method with an identical model learning rate; 
           [0012]      FIG. 2  ( c ) shows the improved foreground region detection results for the image sequence in over-quick lighting changes by using the proposed system based on the same background model learning method and learning rate setting as  FIGS. 2  ( a ) and ( b ). 
           [0013]      FIG. 3  shows a block diagram of the proposed background model learning system for lighting change adaptation, according to one embodiment of this invention. 
       
    
    
     DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 
       [0014]    Reference will now be made in detail to the present embodiments of the invention, examples of which are illustrated in the accompanying drawings. Wherever possible, the same reference numbers are used in the drawings and the description to refer to the same or like parts. It is also to be understood that the detail description of the present embodiments is by examples and is intended to provide further explanation of the invention as claimed. 
         [0015]      FIG. 3  shows a block diagram of the proposed background model learning system for lighting change adaptation according to one embodiment of the invention. The proposed background model learning system  100  includes a background model estimation unit  110 , a foreground map construction unit  120 , a lighting change processing unit  130  and a foreground object analysis unit  140 . The background model estimation unit  110  is in charge of the computation of a background model from an image sequence captured by, e.g, a static camera for a scene of interest, and can be realized by various approaches in literature, e.g., C. Stauffer and W. Grimson, “Adaptive background mixture models for real-time tracking,” in  Proc. IEEE Conf. CVPR , vol. 2, pp. 246-252, 1999, or D.-S. Lee, “Adaptive mixture learning in a dynamic system,” U.S. Pat. No. 7,103,584 B2, 2006. 
         [0016]    The estimated background model from the background model estimation unit  110  is then sent to the foreground map construction unit  120  to derive a reference foreground map that differentiates foreground regions from background ones via, e.g., background subtraction. Next, the lighting change processing unit  130  refines the reference foreground map derived by the foreground map construction unit  120  for eliminating false foreground regions induced by (drastic) lighting changes, and generates a revised foreground map as its output. The revised foreground map is then to applied to the next unit of foreground object analysis  140 , and sent back to the lighting change processing unit  130  and to the background model estimation unit  110  as feedbacks via the feedback links  150  and  160 , respectively. For the foreground object analysis unit, different algorithms, e.g., connected component analysis, can be adopted for different applications in system design. 
         [0017]    Specifically, the input of the background model estimation unit  110  is an image sequence I 0 , . . . , I t , fed in frame by frame from the input link  170  up to the current time instance t. The image frame I t ={I t,x |∀x} of the current time instance t contains a plural number of pixel intensities I t,x s whose format is, e.g., in grayscale for our discussion of a preferred embodiment. However, extensions of system embodiments from gray-scale images to color images in RGB, in YUV444/422/420, in Bayer-pattern-like format or in other color formats would be straightforward. Based on the input images, a background model denoted by a set B t ={B t,x |∀x} up to the current time instance t is computed by the background model estimation unit  110 , wherein each set element B t,x εB t  is a background model estimate for the pixel located at x. The core computation of the background model estimation unit  110  can be embodied by any one of general background model learning approaches. Without loss of generality, let a t,x  denote a model learning rate that controls the frequency of background model updating for a model estimate B t,x  for a chosen background model learning approach. As will be elaborated later, dynamic adjustments of the model learning rates a t ={a t,x |∀x} can be realized by referencing the output of the lighting change processing unit  130  via the feedback link  160 . 
         [0018]    In the foreground map construction unit  120 , a reference foreground map that differentiates foreground regions from background scenes is derived based on background subtraction from the background model B t  generated by the background model estimation unit  110 . More precisely, the reference foreground map of the time instance t is denoted by a set F t ={F t,x |∀x}, where every set element F t,x  can be a binary value (F t,x ε{0, 1}) indicating the background (as 0) and foreground (as 1) label for a pixel at x, or can be a real value (F t,x ε         ) representing the possibility of being foreground for a pixel at x. Such an output of reference foreground map F t  can be generated by most surveillance systems based background subtraction. 
         [0019]    The lighting change processing unit  130  receives the reference foreground map F t  as one of its inputs and generates a revised foreground map F t′  that contains less false detections of foreground regions resulting from lighting changes. The idea behind the lighting change processing unit  130  is simple yet effective. While image intensity variation of over-quick lighting change may seem to be large among temporally distant image frames, it may be small between two consecutive frames if the frame rate of recording is high enough. The small and smooth change of image brightness between consecutive image frames provides a cue for eliminating false alarms in foreground detection for drastic, but not abrupt, lighting changes. (Note that the abrupt changes in background are regarded as salient deviations between two consecutive image frames, due to, e.g., light on/off.) For example, by thresholding the differences between corresponding pair of pixels, each from two consecutive frames, at a proper level, such false alarms can often be reduced. 
         [0020]    Accordingly, the lighting change processing unit  130  performs the following computations for the case of the input reference foreground map F t  in binary. First, the thresholding of intensity difference for every pixel pair is carried out by 
         [0000]    
       
         
           
             
               D 
               
                 t 
                 , 
                 x 
               
             
             = 
             
               { 
               
                 
                   
                     1 
                   
                   
                     
                       
                          
                         
                           
                             I 
                             
                               t 
                               , 
                               x 
                             
                           
                           - 
                           
                             I 
                             
                               
                                 t 
                                 - 
                                 N 
                               
                               , 
                               x 
                             
                           
                         
                          
                       
                       &gt; 
                       
                         T 
                         d 
                       
                     
                   
                 
                 
                   
                     0 
                   
                   
                     
                       otherwise 
                       , 
                     
                   
                 
               
             
           
         
       
     
         [0000]    where I t,x  and I t-N,x  are two pixel intensities of the time instances t and t−N (with N=1 for example), respectively, at the same pixel location x, T d (=10) is a given threshold, and D t,x  is a binary indicator. Thus, a frame difference map D t ={D t,x |∀x} can be derived by the frame differencing sub-unit  131 . Then, a revised foreground map F t′  being less affected by lighting changes can be generated by a map combination of 
         [0000]        F   t′   =F   t  AND ( F   t′-1  OR  D   t ) 
         [0000]    in the foreground map revising mill  132 , where F t′-1  is the revised foreground map obtained at the previous time instance t−1 from the feedback link  150 . For the initialization of the map combination in a recursive formulation, the map F t′-0  is assigned to a set of zeros. Note that the OR operation in the map combination is designed for temporal accumulation of foreground regions, which is useful for detecting objects in slow motion. The revised foreground map F t′  derived by the foreground map revising sub-unit  132  acts as the final output of the lighting change processing unit  130  at the current time instance t, and is transmitted to the foreground object analysis unit  140  for further analysis. Meanwhile, the map F t′  is also sent back to the background model estimation unit  110  as one of its inputs via the feedback link  160 . 
         [0021]    The background model estimation unit  110  can now (for the current time instance t) apply the revised foreground map F t′-1  of the previous time instance t−1 to the dynamic adjustments of the model learning rates a t,x s for (drastic) lighting change adaptation into the background model B t . Specifically, denoted by In F t′-1 −F t-1  is a map fed in from the feedback link  160  to the background model estimation unit  110 . The pixels of P={x|F t′-1,x −F t-1,x &lt;0} represent the image regions of lighting changes detected by the lighting change processing unit  130 . The corresponding learning rates of these pixels, i.e., {a t,x |xδP}, need to be increased in the background model estimation unit  110  for speedup the background model adaptation to lighting changes. A simple way of doing so is to set 
         [0000]    
       
         
           
             
               α 
               
                 t 
                 , 
                 x 
               
             
             = 
             
               { 
               
                 
                   
                     
                       α 
                       H 
                     
                   
                   
                     
                       
                         if 
                          
                         
                             
                         
                          
                         x 
                       
                       ∈ 
                       P 
                     
                   
                 
                 
                   
                     
                       α 
                       L 
                     
                   
                   
                     
                       otherwise 
                       , 
                     
                   
                 
               
             
           
         
       
     
         [0000]    where a H  and a L  are two pre-defined rates for fast and normal background model learning, respectively. With the above dynamic adjustments of model learning rates, lighting changes can be adapted into the estimated background model B t , in a quicker pace. False alarms caused by lighting changes will hence be reduced in the generation of the next reference foreground map F t . 
         [0022]    Finally, the foreground object analysis unit  140  extracts foreground objects (if any) based on the current revised foreground map F t′ . For example, foreground objects (if any) may be extracted via connected component analysis or other object grouping methods. Above all, since one embodiment of this invention is the acquisition of a lighting change processing unit as a post-processing module for general-purpose background model estimation approaches, it can thus be easily integrated into existing background modeling systems to enhance their system robustness to lighting changes, without changing system architectures largely. 
         [0023]    In another embodiment, the lighting change processing unit  130  may receive a to reference foreground map F t  in real value that denotes the possibilities of being foreground for map pixels. For this case, the frame differencing sub-unit  131  may compute a frame difference map D t  in real value by applying 
         [0000]        D   t,x =ƒ(| I   t,x   −I   t-N,x |),
 
         [0000]    where D t,x εD t  denotes a real-valued pixel of the frame difference map D t  at the pixel location x of the current time instance t and ƒ(•)ε          is a transfer function that maps the absolute pixel difference to the possibility of being in changing for a pixel. The function ƒ(•) can actually be embodied by a sigmoid function or other types of transfer functions. Then, the foreground map revising sub-unit  132  can utilize min/max operations to revise the reference foreground map F t  and generate a revised foreground map F t′  in real value by 
         [0000]        F   t′ =min( F   t ,max( F   t′-1   ,D   t )), 
         [0000]    where the min/max operations are applied to the integration of the three input maps, F t , F t′-1 , and D t , in real value. Similarly, the set of pixels in lighting change for the dynamic adjustments of model learning rates in the background model estimation unit  110  can also be given by P={x|F t′-1,x −F t-1,x &lt;T L }, where T L (=0) is a preset threshold. 
         [0024]    Although the present invention has been described in considerable detail with reference to certain embodiments thereof, other embodiments are possible. Therefore, the spirit and scope of the appended claims should not be limited to the description of the embodiments contained herein. It will be apparent to those skilled in the art that various modifications and variations can be made to the structure of the present invention without departing from the scope or spirit of the invention. In view of the foregoing, it is intended that the present invention cover modifications and variations of this invention provided they fall within the scope of the following claims.