Abstract:
A method for surveilling a monitored environment includes classifying an individual detected in the monitored environment according to a role fulfilled by the individual within the monitored environment, generating a trajectory that illustrates movements and locations of the individual within the monitored environment, and detecting when the trajectory indicates an event that is inconsistent with an expected pattern for the role.

Description:
CROSS REFERENCE TO RELATED APPLICATIONS 
       [0001]    This application is a continuation of co-pending U.S. patent application Ser. No. 13/975,672, filed Aug. 26, 2013, which is herein incorporated by reference in its entirety. 
     
    
     BACKGROUND OF THE INVENTION 
       [0002]    The present invention relates generally to security and surveillance and relates more specifically to methods for classifying and tracking surveillance targets. 
         [0003]    Many venues such as retail environments, factories, offices, transportation hubs, and the like employ surveillance systems in order to ensure customer safety, national security, and/or operational efficiency. Many conventional surveillance systems distribute large sets of surveillance devices (e.g., cameras, sensors, and the like) within a monitored area in order to detect illicit, unsafe, and/or unauthorized events. These devices generate surveillance data in the forms of streams and/or data points that are typically forwarded to a central location for review (e.g., by a human operator). 
         [0004]    The computational cost of processing all of the surveillance data, however, can be prohibitive. For instance, if the surveillance system deploys a plurality of devices, all of which are continuously generating surveillance data, a potentially enormous amount of surveillance data will be generated. It may be difficult, if not impossible, for a human operator to efficiently review all of this data and therefore to respond in a timely manner to an event. 
         [0005]    Moreover, events that may be normal for one type of person may be abnormal for another type of person. For instance, in a bank environment, it might be “normal” (i.e., likely not worthy of a security alert) for a teller or cashier to venture behind the counter, but “abnormal” (i.e., potentially worthy of a security alert) for a customer to do the same. Many conventional surveillance systems, however, have difficulty distinguishing between normal and abnormal events based on the type (or role) of the person involved in the events. 
       SUMMARY OF THE INVENTION 
       [0006]    A method for surveilling a monitored environment includes classifying an individual detected in the monitored environment according to a role fulfilled by the individual within the monitored environment, generating a trajectory that illustrates movements and locations of the individual within the monitored environment, and detecting when the trajectory indicates an event that is inconsistent with an expected pattern for the role. 
     
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         [0007]    So that the manner in which the above recited features of the present invention can be understood in detail, a more particular description of the invention may be had by reference to embodiments, some of which are illustrated in the appended drawings. It is to be noted, however, that the appended drawings illustrate only typical embodiments of this invention and are therefore not to be considered limiting of its scope, for the invention may admit to other equally effective embodiments. 
           [0008]      FIG. 1  is a block diagram illustrating one embodiment of a system for role-based tracking and surveillance, according to the present invention; 
           [0009]      FIG. 2  is a flow diagram illustrating one embodiment of a method for performing role-based tracking and surveillance, according to the present invention; and 
           [0010]      FIG. 3  is a high-level block diagram of the role-based tracking and surveillance method that is implemented using a general purpose computing device. 
       
    
    
     DETAILED DESCRIPTION 
       [0011]    In one embodiment, the invention is a method and apparatus for role-based tracking and surveillance. Embodiments of the invention automatically detect abnormal behavior of specific individual types or roles based on surveillance data generated by a plurality of surveillance devices. Within the context of the present invention, a “role” refers to the social and/or professional capacity (including the behaviors, rights, and obligations associated with that capacity) within which an individual finds him or herself in a monitored environment. For instance, in a retail environment, an individual may fulfill the role of “customer,” “cashier,” “manager,” “security personnel,” or the like. Abnormal events can be automatically detected by comparing a detected individual&#39;s observed actions with historical patterns (e.g., predetermined “normal” or allowed conduct and activities) for the role according to which the individual is classified. The disclosed system supports even large-scale surveillance systems because it does not require or rely on the identification of the detected individual or on pre-defined behavior types. Although the invention is described within the exemplary context of human tracking and surveillance, the same techniques described herein may be used to track and monitor non-human surveillance targets (e.g., animals, vehicles, etc.). Moreover, although the invention is described within the exemplary context of security applications, the disclosed techniques could also be implemented to detect non-security related events. 
         [0012]      FIG. 1  is a block diagram illustrating one embodiment of a system  100  for role-based tracking and surveillance, according to the present invention. In one embodiment, the system  100  includes two main components: plurality of sensors  102   1 - 102   n  (hereinafter collectively referred to as “sensors  102 ”) and a behavior assessor  104 . The sensors  102  and the behavior assessor  104  cooperate to automatically classify individuals according to role and to detect behaviors that deviate from behavior expected of the roles. 
         [0013]    The sensors  102  provide surveillance data for processing by various components of the behavior assessor  104 . These sensors  102  may include one or more of: imaging sensors (e.g., still cameras, video cameras, thermographic cameras, or the like), biometric sensors (e.g., fingerprint sensors, ocular sensors, voice sensors, or the like), or electronic scanners (e.g., for scanning machine-readable tags or labels, such as radio frequency identification tags, bar codes, or the like). The sensors  102  collect surveillance data from various physical locations within a monitored environment. For instance, any one or more of the sensors  102  may be positioned to collect surveillance data at the entrances and exits of the monitored environment, from specific locations within the monitored environment (e.g., restricted access or high-traffic areas), or from any other location. 
         [0014]    The behavior assessor  104  receives surveillance data from the sensors  102  and generates an alert when analysis of the surveillance data indicates the occurrence of an abnormal event. To this end, the behavior assessor  104  comprises at least: a plurality of feature detectors  106   1 - 106   n  (hereinafter collectively referred to as “feature detectors  106 ”), a feature fusion engine  108 , a global classifier  110 , a global tracker  112 , a pattern matcher  114 , and a pattern builder  116 . Any of the behavior assessor components  106 - 116  may comprise a processor configured to perform specific functions related to role-based tracking and surveillance. In addition, the behavior assessor  104  is in communication with a pattern database  118 . Although the pattern database  118  illustrated in  FIG. 1  is depicted as remote storage (i.e., separate from the behavior assessor  104 ), in alternative embodiments, the pattern database may be integrated with the behavior assessor  104  as local storage. 
         [0015]    The feature detectors  106  receive raw surveillance data from the sensors  102  and extract features from the surveillance data. The features comprise characteristics depicted in the surveillance data that may aid in identifying the role of and/or tracking an individual present in the monitored environment. For instance, a feature detector  106  that receives thermographic images from a thermographic camera may extract a thermal marker or pattern from the thermographic images that indicates a particular role. Although  FIG. 1  illustrates a one-to-one correspondence between sensors  102  and feature detectors  106 , the system  100  is not so limited. For instance, a given feature detector  106  might receive surveillance data from a plurality of sensors  102  of the same type (e.g., a plurality of video cameras). Alternatively, a given sensor  102  may provide surveillance data to a plurality of feature detectors  106  that each implement different algorithms for feature extraction. 
         [0016]    The feature fusion engine  108  receives the features extracted by the feature detectors  106  and correlates the features into groups that pertain to common individuals or roles. For instance, the feature fusion engine  108  may determine that a thermographic marker extracted by a first feature detector  106 , a facial image extracted by a second feature detector  106 , and a radio frequency identification tag extracted by a third feature detector  106  all depict the same person or role. Having a plurality of extracted features related to a given individual may increase the probability of correctly identifying the individual&#39;s role and improve the ability of the system  100  to track the individual throughout the monitored environment. 
         [0017]    The global classifier  110  receives the groups of features generated by the feature fusion engine  108  and assigns a class to each of the groups of features. In one embodiment, the class assigned to a group of features identifies a characteristic of the individual to whom the group of features pertains. In one particular embodiment, the global classifier  110  classifies groups of features based on the roles of the associated individuals in the monitored environment. For example, individuals in a retail environment may fulfill a set of roles including “customer,” “cashier,” “manager,” “security personnel,” or the like; thus, the groups of features extracted from the retail environment may be classified according to these roles. The global classifier  110  identifies the role/class that is best implied by a given group of features. 
         [0018]    The global tracker  112  receives the classified groups of features and uses them to track the associated individuals or roles throughout the monitored environment. Thus, the global tracker  112  associates a trajectory with each group of features that records the movements and locations of the associated individual or role within the monitored environment. 
         [0019]    The pattern matcher  114  receives the trajectories and associated classes from the global tracker  112  and uses this information to determine when an event has occurred that represents a potential abnormality (e.g., a breach of security). In one embodiment, the pattern matcher  114  detects such events by comparing a given trajectory to an historical behavior pattern for the class associated with the trajectory. For instance, if the pattern matcher  114  receives a trajectory associated with the class of “customer,” the pattern matcher  114  would compare the trajectory to an historical behavior pattern for the class of “customer.” In one embodiment, the historical behavior patterns for various classes are stored in the pattern database  118 . 
         [0020]    The pattern builder  116  also receives the trajectories and associated classes from the global tracker  112 , but uses this information to learn and build behavior patterns associated with the various classes indicated by the groups of features. The learned patterns are stored in the pattern database  118  for use by the pattern matcher  114  as described above. 
         [0021]    Although the system  100  is illustrated as comprising a plurality of individual components that perform discrete functions, it will be appreciated that any two or more of the illustrated components may be combined in a single component that performs multiple functions. Additionally, although the system  100  is illustrated as a contained system, it will be appreciated that the various components of the system  100  may be physically distributed throughout the monitored environment (although still contained within the physical boundaries of the monitored environment), and some of the components may even be located off-site (i.e., outside the physical boundaries of the monitored environment). To this end, the various components of the system  100  may include a combination of wireless and physically connected devices. 
         [0022]      FIG. 2  is a flow diagram illustrating one embodiment of a method  200  for performing role-based tracking and surveillance, according to the present invention. The method  200  may be performed, for example, by the system  100  illustrated in  FIG. 1 . As such, reference is made in the discussion of the method  200  to various elements depicted in  FIG. 1 . However, it will be appreciated that the method  200  may also be performed by systems having alternate configurations. 
         [0023]    The method  200  begins at step  202  and proceeds to step  204 , where the sensors  102  collect surveillance data from a monitored environment. For instance, the surveillance data may include substantially real-time data collected by one or more of the sensors  102  that allows the system  100  to uniquely identify individuals within the monitored environment. Within the context of the present invention, the ability to “uniquely identify” and individual does not necessarily imply that the individual&#39;s identity (e.g., name) is recognized or revealed. Rather, embodiments of the present invention associate sets of features with a common individual as a means of recognizing the individual (and distinguishing the individual from others) when he or she moves to different locations within a monitored environment. As discussed above, the surveillance data may include images, biometric data, electronic data, or other types of information collected from the monitored environment. For instance, the images may include still and/or video images, which would allow the system  100  to identify and track individuals by their appearances. The biometric data may include fingerprints, ocular features, gait, or the like, which would allow the system  100  to identify and track individuals by their individual features. The electronic data may include bar codes, radio frequency identification tags, or the like, which would allow the system  100  to identify and track individuals by electronic and/or electromagnetic signals. 
         [0024]    In step  206 , the feature detectors  106  detect and extract features from the surveillance data collected in step  204 . In one embodiment, the extracted features include information that helps to identify the roles of individuals present in the monitored environment distinguish among individuals present in the monitored environment. For instance, features extracted from a still image of an individual might include a uniform or a badge that indicates the individual&#39;s role. Features extracted from a radio frequency identification tag attached to an individual might include an access authorization or license. Features extracted from a thermographic image might include thermal markers (e.g., patterns that may be recognized by a pattern recognition algorithm) embedded in an individual&#39;s clothing. Because thermal markers are substantially invisible to the human eye, they may be especially useful for marking undercover security personnel, who may need to access restricted locations without explicitly broadcasting their roles. Thermal markers are also less sensitive to environmental conditions that would otherwise affect tracking using conventional visual markers (e.g., low-light conditions). 
         [0025]    In step  208 , the feature fusion engine  108  correlates the extracted features into groups. A group of features created by the feature fusion engine  108  includes features that are believed to identify a common individual (e.g., a thermographic marker and a facial image associated with the same person). 
         [0026]    In step  210 , the global classifier  110  attempts to classify the groups of features. In one particular embodiment, the global classifier  110  attempts to recognize the role associated with each group of features. For example, in a retail environment, a given group of features might indicate that the individual from whom the features come is a customer, a cashier, a manager, a security guard, or the like. In a sport event environment, a given group of features might indicate that the individual is a player, a coach, a spectator, a referee/umpire, a security guard, a vendor, or the like. In some cases, the global classifier  110  may not be able to classify a given group of features. In such an event, the global classifier  110  may wait to receive additional information or features from the feature fusion engine  108 , in case the additional information helps to resolve the classification. 
         [0027]    In step  212 , the pattern matcher  114  matches the classes to historical behavior patterns associated with the classes. The historical behavior pattern for a given class represents movements and activities that are considered “normal” (i.e., likely not worthy of an alert) for the role associated with the given class. In one embodiment, the historical behavior patterns are retrieved from the pattern database  118 . 
         [0028]    In step  214 , the global tracker  112  tracks the movements of the individuals associated with the groups of features throughout the monitored environment. This creates a series of trajectories that illustrate the individuals&#39; real-time movements and behaviors in the monitored environment. In one embodiment, the individuals are tracked only for as long as they remain within the monitored environment; once an individual exits the monitored environment (which may be indicated by the global tracker&#39;s inability to detect the individual&#39;s presence), the global tracker  112  ceases to create a trajectory for the individual. 
         [0029]    In step  216 , the pattern matcher  114  compares the trajectories to the historical behavior patterns for the corresponding classes. In particular, the pattern matcher  114  observes the trajectories in order to confirm that the trajectories are substantially consistent with the expected historical behavior patterns to which they are matched. 
         [0030]    In step  218 , the pattern matcher  114  determines whether any of the trajectories have deviated from the historical behavior patterns to which they are matched. In one embodiment, deviations are detected by computing a measure of similarity or dissimilarity between the trajectories and the associated historical behavior patterns. In one embodiment, the pattern matcher  114  is tolerant to a threshold deviation; however, any deviation beyond the threshold is considered abnormal. 
         [0031]    If the pattern matcher  114  concludes in step  218  that a trajectory has deviated from an historical behavior pattern to which it is matched, then the pattern matcher  114  issues an alert in step  220 . In one embodiment, issuing the alert includes sending a message including details of the deviation (e.g., the class of the individual associated with the deviation, the action that deviated from the historical behavior pattern, the location at which the deviation occurred, etc.) to a central control location for review. In another or further embodiment, issuing the alert includes activating an alarm (e.g., an audible and/or visible alarm) or taking other cautionary or corrective measures (e.g., activating locks in restricted areas). Once an alert has been issued, the method  200  returns to step  214 . The global tracker  112  and the pattern matcher  114  continue to track individuals and to compare their movements to historical behavior patterns until the individuals exit the monitored environment. 
         [0032]    If the pattern matcher  114  concludes in step  218  that no trajectory has deviated from the historical behavior pattern to which it is matched, then the method  200  returns to step  214 . The global tracker  112  and the pattern matcher  114  continue to track individuals and to compare their movements to historical behavior patterns until the individuals exit the monitored environment. 
         [0033]    As discussed above, role-based tracking and surveillance according to the present invention relies in part on a database of historical behavior patterns for the various roles with which an individual in a monitored environment may be associated (e.g., pattern database  118 ). Although behavior patterns may be pre-programmed into the system  100 , the system  100  is also capable of learning new roles and associated behavior patterns. In addition, the system  100  may learn new features and/or behavior patterns associated with previously established roles. Ongoing learning allows the system  100  to perform more accurate tracking and surveillance by remaining up-to-date and by building a customized knowledge base for the monitored environment in which the system  100  is employed. 
         [0034]    Although the present invention is largely described within the context of the access control, it is noted that the methods disclosed herein may be advantageously deployed in other contexts. For instance, other embodiments, the present invention may be used to determine whether actions taken by a particular individual are consistent with his or her role. As a specific example, embodiments of the present invention may be employed to ensure security and adherence to safety protocols by monitoring employees in a workplace (e.g., a factory or laboratory). 
         [0035]    In further embodiments, the ability to detect different behavior patterns may support the provision of differentiated or customized services. For instance, different sub-types of customers may be detected in a retail environment, and customized goods or services could be offered to the customers based on sub-type. 
         [0036]      FIG. 3  is a high-level block diagram of the role-based tracking and surveillance method that is implemented using a general purpose computing device  300 . In one embodiment, a general purpose computing device  300  comprises a processor  302 , a memory  304 , a tracking and surveillance module  305  and various input/output (I/O) devices  306  such as a display, a keyboard, a mouse, a stylus, a wireless network access card, an Ethernet interface, and the like. In one embodiment, at least one I/O device is a storage device (e.g., a disk drive, an optical disk drive, a floppy disk drive). It should be understood that the tracking and surveillance module  305  can be implemented as a physical device or subsystem that is coupled to a processor through a communication channel. 
         [0037]    Alternatively, the tracking and surveillance module  305  can be represented by one or more software applications (or even a combination of software and hardware, e.g., using Application Specific Integrated Circuits (ASIC)), where the software is loaded from a storage medium (e.g., I/O devices  306 ) and operated by the processor  302  in the memory  304  of the general purpose computing device  300 . Thus, in one embodiment, the tracking and surveillance module  305  for role-based tracking and surveillance, as described herein with reference to the preceding figures, can be stored on a computer readable storage medium or device (i.e., a tangible or physical article such as RAM, a magnetic or optical drive or diskette, and the like, rather than a propagating signal). 
         [0038]    It should be noted that although not explicitly specified, one or more steps of the methods described herein may include a storing, displaying and/or outputting step as required for a particular application. In other words, any data, records, fields, and/or intermediate results discussed in the methods can be stored, displayed, and/or outputted to another device as required for a particular application. Furthermore, steps or blocks in the accompanying figures that recite a determining operation or involve a decision, do not necessarily require that both branches of the determining operation be practiced. In other words, one of the branches of the determining operation can be deemed as an optional step. 
         [0039]    While the foregoing is directed to embodiments of the present invention, other and further embodiments of the invention may be devised without departing from the basic scope thereof. Various embodiments presented herein, or portions thereof, may be combined to create further embodiments. Furthermore, terms such as top, side, bottom, front, back, and the like are relative or positional terms and are used with respect to the exemplary embodiments illustrated in the figures, and as such these terms may be interchangeable.