Source: https://patents.google.com/patent/US8548198B2/en
Timestamp: 2019-07-18 13:25:23
Document Index: 376329527

Matched Legal Cases: ['ART 325', 'ART 325', 'ART 325', 'ART 325', 'ART 325', 'ART1', 'ART2', 'ART 325', 'ART3', 'ART 325', 'ART 325', 'ART 325', 'ART 325', 'ART 325', 'ART 325', 'ART 325', 'ART 325', 'ART 325', 'ART 325', 'ART 325']

US8548198B2 - Identifying anomalous object types during classification - Google Patents
Identifying anomalous object types during classification Download PDF
US8548198B2
US8548198B2 US13/622,281 US201213622281A US8548198B2 US 8548198 B2 US8548198 B2 US 8548198B2 US 201213622281 A US201213622281 A US 201213622281A US 8548198 B2 US8548198 B2 US 8548198B2
US13/622,281
US20130022242A1 (en
2009-08-31 Priority to US12/551,276 priority Critical patent/US8270733B2/en
2012-09-18 Application filed by Behavioral Recognition Systems Inc filed Critical Behavioral Recognition Systems Inc
2012-09-18 Priority to US13/622,281 priority patent/US8548198B2/en
2013-01-24 Publication of US20130022242A1 publication Critical patent/US20130022242A1/en
2013-10-01 Publication of US8548198B2 publication Critical patent/US8548198B2/en
2014-12-22 Assigned to BEHAVIORAL RECOGNITION SYSTEMS, INC. reassignment BEHAVIORAL RECOGNITION SYSTEMS, INC. ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: COBB, WESLEY KENNETH, FRIEDLANDER, DAVID, GOTTUMUKKAL, RAJKIRAN KUMAR, SEOW, MING-JUNG, XU, GANG
This application is a continuation of co-pending U.S. patent application Ser. No. 12/551,276, filed Aug. 31, 2009. The aforementioned related patent application is herein incorporated by reference in its entirety.
Embodiments of the invention provide techniques for classifying objects using pixel-level micro-features extracted from image data. More specifically, embodiments of the invention relate to techniques for identifying anomaly object types during classification.
However, such surveillance systems typically require that the objects which may be recognized by the system to be defined in advance. Thus, in practice, these systems rely on predefined definitions for objects to evaluate a video sequence. In other words, unless the underlying system includes a description for a particular object, i.e., has been trained, the system is generally incapable of recognizing that type of object. This results in surveillance systems with recognition capabilities that are labor intensive and prohibitively costly to maintain or adapt for different specialized applications. Accordingly, currently available video surveillance systems are often unable to identify objects, events, behaviors, or patterns as being “normal” or “abnormal’ by observing what happens in the scene over time; instead, such systems rely on static object definitions.
Embodiments of the invention relate to techniques for a micro-feature classifier component to discover object type clusters using pixel-level micro-features extracted from image data and identify anomaly object types. The object type clusters are discovered and updated by a self-organizing map and adaptive resonance theory (SOM-ART) network. The SOM-ART network is produced and updated based on the pixel-level micro-features independent of any object definition data, i.e., without training. Therefore, unsupervised learning, cluster discovery, and object type classification may be performed in parallel.
One embodiment of the invention includes a computer-implemented method for identifying anomaly object types during classification of image data captured by a video camera. The method may generally include receiving a micro-feature vector including multiple micro-feature values, where each micro-feature value based on at least one pixel-level characteristic of a foreground patch that depicts a foreground object within the image data. The method may also include classifying the foreground object as depicting a first object type corresponding to a first object type cluster of the object type clusters based on the micro-feature vector, computing a probability density function for the object type clusters, computing a probability density value for the micro-feature vector, and evaluating a rareness measure of the micro-feature vector. The rareness measure estimates a likelihood of observing the micro-feature vector, based on the probability density function and the probability density value. That is, the rareness measure provides a percentile ranking of just how atypical the micro-feature vector is, relative to previously observed micro feature vectors in the scene. The method may also include identifying the foreground object as an anomaly object type when the rareness measure is below a specified threshold.
Another embodiment of the invention includes a computer-readable storage medium containing a program which, when executed by a processor, performs an operation for identifying anomaly object types during classification of image data captured by a video camera. The operation may generally include receiving a micro-feature vector including multiple micro-feature values, where each micro-feature value based on at least one pixel-level characteristic of a foreground patch that depicts a foreground object within the image data and classifying the foreground object as depicting a first object type corresponding to a first object type cluster of the object type clusters based on the micro-feature vector. The operation may also include computing a probability density function for the object type clusters, computing a probability density value for the micro-feature vector, and evaluating a rareness measure of the micro-feature vector. The rareness measure estimates a likelihood of observing the micro-feature vector, based on the probability density function and the probability density value. The operation may also include identifying the foreground object as an anomaly object type when the rareness measure is below a specified threshold.
Still another embodiment includes a system having a video input source configured to provide image data. The system may also include a processor and a memory containing a program, which, when executed on the processor is configured to perform an operation for identifying anomaly object types during classification of image data captured by a video camera. The operation may generally include receiving a micro-feature vector including multiple micro-feature values, where each micro-feature value based on at least one pixel-level characteristic of a foreground patch that depicts a foreground object within the image data and classifying the foreground object as depicting a first object type corresponding to a first object type cluster of the object type clusters based on the micro-feature vector. The operation may also include computing a probability density function for the object type clusters, computing a probability density value for the micro-feature vector, and evaluating a rareness measure of the micro-feature vector. The rareness measure estimates a likelihood of observing the micro-feature vector, based on the probability density function and the probability density value. The operation may also include identifying the foreground object as an anomaly object type when the rareness measure is below a specified threshold.
FIG. 4C illustrates a conceptual diagram of a probability density function for the object type clusters, according to one embodiment of the invention.
FIG. 4D illustrates another method for performing object type classification using the SOM-ART network, according to one embodiment of the invention.
Techniques are disclosed for identifying anomaly object types during classification of foreground objects extracted from image data. A self-organizing map and adaptive resonance theory (SOM-ART) network is used to discover object type clusters and classify objects depicted in the image data based on pixel-level micro-features that are extracted from the image data. The extracted micro-features are heuristic features of foreground patches depicting objects in frames of video. The extracted micro-features may be represented as a micro-feature vector input to the micro-feature classifier. The micro-feature classifier may learn a set of distinct object types, over time, through observing different micro-feature vectors. In one embodiment, a SOM-ART network is included within the micro-feature classifier to process the pixel-level micro-features to adaptively learn and organize the micro-features into object type clusters. The training of the SOM-ART network is unsupervised, i.e., performed independent of any training data that defines particular objects, allowing a behavior-recognition system to perform unsupervised learning and object classification in parallel without being constrained by specific object definitions.
In one embodiment, the primitive event detector 212 may be configured to receive the output of the computer vision engine 135 (i.e., the video images, the micro-feature vectors, and context event stream) and generate a sequence of primitive events—labeling the observed actions or behaviors in the video with semantic meaning. For example, assume the micro-feature classifier 221 has classified a foreground object as being a member of an object type cluster including vehicles based on the context event stream and/or micro-feature vectors received from the computer vision engine 135. The primitive event detector 212 may generate a semantic symbol stream that is output to the semantics component 242, providing a simple linguistic description of actions engaged in by the foreground object. For example, a sequence of primitive events related to observations of the computer vision engine 135 occurring at a parking lot could include language semantic vectors representing the following: “vehicle appears in scene,” “vehicle moves to a given location,” “vehicle stops moving,” “person appears proximate to vehicle,” “person moves,” person leaves scene” “person appears in scene,” “person moves proximate to vehicle,” “person disappears,” “vehicle starts moving,” and “vehicle disappears.” As described in greater detail below, the primitive event stream may be supplied to excite the perceptual associative memory 230.
FIG. 3A illustrates an example of the micro-feature classifier component 221 of the video analysis system shown in FIG. 2, according to one embodiment of the invention. The micro-feature classifier component 221 receives a micro-feature vector 300 that is produced by the context processor component 220 and outputs an identified object type 345. The context processor component 220 processes the image data and produces pixel-level characteristic(s) that are used to generate the micro-feature vector 300. Micro-feature values are computed by the context processor component 220 and output as elements of the micro-feature vectors 300. Examples of micro-feature values include values representing the foreground patch's hue entropy, magnitude-saturation ratio, orientation angle, pixel area, aspect ratio, groupiness (based on the pixel-level spatial distribution), legged-ness, verticality (based on per-pixel gradients), animateness, periodicity of motion, etc. Valid micro-feature values may range in value from 0 to 1 (inclusive) and −1 may be used to represent an invalid micro-feature value that should not be used for classification. The micro-feature values may be represented in a floating point format.
The micro-feature classifier 221 includes a learning component 310, a SOM-ART network component 340, and a classification component 320. The SOM-ART network component 340 includes a SOM 315 and an ART 325. The SOM-ART network component 340 provides a specialized neural network configured to create object type clusters from a group of inputs, e.g., micro-feature vectors. Each element (micro-feature value) of the micro-feature vector is a dimension of an input to the SOM 315. The SOM 315 receives the N dimension micro-feature vector and produces a reduced dimension geometric representation of M dimensions, where M is less than or equal to N. Reducing the number of dimensions also reduces the time duration for self-training since the amount of input data needed to represent the distribution of the clusters is reduced. For example, when N is 6, the number of different input values is 86=262,144 (assuming an 8 bit micro-feature value). Reducing the number of dimensions from 6 to 5, reduces the number of different input values from 262,144 to 32,768.
The ART 325 is updated by the learning component 310 to manage the object type clusters, creating new clusters, removing clusters, and merging clusters. The ART 325 is used by the classification component 320 to classify each foreground object depicted by a micro-feature vector as one of the learned object types or as an unrecognized (unknown) object type. The learning component 310 is not limited to specific pre-defined object types since the learning component 310 is configured to perform unsupervised learning to automatically find what and how many object types may exist in the image data. This unsupervised learning and object type cluster discovery are adaptive since the knowledge about existing classes of objects is dynamically updated as new object types appear and may occur in parallel with the classification functions performed by the classification component 320. Therefore, the micro-feature classifier 221 is suitable for real-time video surveillance applications. When the image data represents a new scene or at startup, the learning component 310 estimates the SOM 315 topology based on the number of input micro-feature vectors. A principal component analysis (PCA) technique may be used to initialize the SOM 315. The learning component 310 may process a first batch of micro-feature vectors to produce a mature SOM 315 before the classification component 325 processes any of the micro-feature vectors in the first batch. The size of the first and following batches may be adjusted to fit the incoming micro-feature data stream rate.
The learning component 310 may update the SOM-ART component 340 incrementally, i.e. as each micro-feature vector is received. Alternatively, the learning component 310 may collect a batch (predetermined number) of micro-feature vectors and update the SOM-ART component 340 periodically. The learning component 310 organizes the micro-feature vector elements in the SOM 315 neurons and then updates object type clusters in the ART neurons incrementally or periodically. The classification component 320 includes an anomaly detection component 322 and processes each micro-feature vector that is received by comparing the vector's micro-feature values with the existing object type clusters in the SOM-ART component 221. The anomaly detection component 322 is configured to compute a probability density function based on the existing clusters in the ART 325 and compute a probability density value for the micro-feature vector. The distance measure between the received micro-feature vector and the object type clusters in the SOM-ART (component 340) are used to characterize the accuracy of each cluster identified by the classification component 320 as a potential match. The object whence the micro-feature vector has been derived will be classified by the classification component 320 as either one of the learned object types or as an anomaly object type based on the probability density function and probability density value.
As is known, the vigilance parameter has considerable influence on an ART 325: higher vigilance produces many, fine-grained clusters, while lower vigilance results in more-general clusters. Further, the inputs may be a binary values (generally referred to as an ART1 network), or may be continuous values (generally referred to as an ART2 network). Other variations of the ART 325 include ART3, ARTMAP, and FUZZY ART networks.
As clusters emerge in the ART 325, the ART 325 may be evaluated to determine whether an unusual event has occurred, based on a set of alert rules. For example, consider the scenario when the ART 325 is configured to categorize objects of the “person” class, based on the position at which a person appears (or disappears) from the scene. In such a case, each cluster describes a prototypical position of where a person may appear—and a mean and variance from that prototypical position (e.g., to 2.5 standard deviations in the X and Y directions). In this scenario, an alert rule may specify that whenever the ART 325 generates a new cluster based on a set of parsed input values, an alert should be generated. Further, as the decision of the ART 325 to create a new cluster is dependent on whether a given input sufficiently resembles one of the current clusters, in one embodiment, the ART 325 is allowed to “mature” over specified period of time prior to any alerts being generated. That is, until the computer vision engine 135 has observed a sufficient number of persons, new clusters may be created with a relatively high frequency. Conversely, after prolonged observation, the relative frequency of new clusters should decline—making the event of a new cluster more unusual. Once the ART 325 has matured, objects with low computed accuracy values may be identified as anomaly object types and trigger an alert based on an alert rule. Of course, one of skill in the art that the alert rules may be based on a broad variety of triggering conditions based on the state of the ART 325—and that the actual alert rules may be tailored for the ART 325 and by the needs of a particular case.
FIG. 4A illustrates a method for performing unsupervised learning using the SOM-ART network component 340, according to one embodiment of the invention. At step 400 a micro-feature vector for a foreground patch is received by the micro-feature classifier 221. At step 405 the learning component 310 determines if a batch of micro-feature vectors has been received, and, if not, then at step 410 the learning component 310 waits to receive another micro-feature vector before repeating step 405. If, at step 405 the learning component 310 determines that a batch of micro-feature vectors have been received, then at step 415 the learning component 310 organizes the micro-classifier feature in the SOM 315 based on the micro-feature vectors in the batch. At step 420 the learning component 310 updates the object type clusters in the SOM 315 and ART 325 based on the micro-feature vectors in the batch. Importantly, nodes in the SOM 315 are not updated with the invalid micro-feature vector values.
FIG. 4B illustrates a method for performing object type classification using the SOM-ART network component 340, according to one embodiment of the invention. At step 430 a micro-feature vector for a foreground patch is received by the micro-feature classifier 221. At step 435 the classification component 320 compares the micro-feature vector with the object type clusters that have been discovered for the scene and represented by the SOM-ART component 340. At step 440 the classification component 320 determines if the micro-feature vector matches an object type cluster. If, in step 440 a match is found, then at step 450 the identified object type of the cluster is output for the matched micro-feature vector. If, at step 440 a match is not found, then at step 445 the classification component 320 indicates that a match was not identified for the micro-feature vector. When the anomaly detection component 322 is included in the classification component 320, the object type output for rare micro-feature vector that is unmatched, not accurately matched, or sometimes accurately matched micro-feature vector may be classified as the anomaly object type. An alternate method for performing object type classification using the anomaly detection component 322 is described in conjunction with FIG. 4D.
FIG. 4C illustrates a conceptual diagram of a probability density function for the object type clusters, according to one embodiment of the invention. The anomaly detection component 322 computes the probability density function for the object type clusters in the ART 325. The horizontal axis is the object type cluster distribution in the x dimension and the vertical axis is the probability density of the object type clusters. A first object type cluster 451 is positioned at cluster center 452 and a second object type cluster 457 is positioned at cluster center 456. The first object type cluster 451 may be identified by the classification component 320 as a match for a first micro-feature input vector 462 based on the position of the first micro-feature input vector 462 along the x axis. More specifically, the first object type cluster 451 is identified when the normalized Euclidean distance between the cluster center 452 and the position of the first micro-feature input vector 462 is less than a threshold value. Similarly, the first object type cluster 457 may also be identified by the classification component 320 as a match for a second micro-feature input vector 460 based on the position of the second micro-feature input vector 460 along the x axis.
The anomaly detection component 322 computes a probability density value 464 for the first micro-feature input vector 462. The probability density value 464 is very close to the probability density value for cluster 451, resulting in a high accuracy value that indicates the first object type cluster 451 is a match for the first micro-feature input vector 462. The anomaly detection component 322 also computes a probability density value 466 for the second micro-feature input vector 460. The probability density value 466 is not very close to the probability density value for cluster 451, resulting in a low accuracy value that indicates the first object type cluster 451 is not a match for the second micro-feature input vector 460.
FIG. 4D illustrates another method for performing object type classification using the SOM-ART network 340, according to one embodiment of the invention. At step 430 a micro-feature vector for a foreground patch is received by the classification component 320. At step 470 the classification component 320 computes the normalized Euclidean distance between the micro-feature vector and the object type clusters that have been discovered for the scene and represented by the SOM-ART component 340. At step 471 the classification component 320 identifies any object type clusters that match the micro-feature vector. At step 472 the anomaly detection component 322 computes the probability density function based on the object type clusters that have been discovered for the scene and represented by the SOM-ART component 340. At step 474 the computes a probability density value for the micro-feature vector. At step 476 the anomaly detection component 322 determines a rareness measure for the micro-feature vector. That is, the anomaly detection component 322 estimates a measure of the likelihood of observing the particular micro-feature vector, based on the probability density function and the probability micro-feature vector. In other words, the rareness measure may be used to estimate a percentile ranking of just how atypical the micro-feature vector is, relative to previously observed micro feature vectors in the scene.
At step 480 the classification component 320 determines if the micro-feature vector is rare or not for the foreground object represented by the micro-feature vector. If, in step 480 the micro-feature vector is not rare, then at step 490 the identified object type of the cluster is output for the matched micro-feature vector. If, at step 480 the micro-feature vector is found to be rare, then at step 485 the classification component 320 indicates that the anomaly object type was identified for the micro-feature vector. A threshold value may be used to determine if the micro-feature vector is rare or not.
Advantageously, embodiments of the invention may be used as part of a computer vision engine to identify unusual events as they are observed to occur in a sequence of video frames. Importantly, what is determined as unusual need not be defined in advance, but can be determined over time by observing a stream of primitive events and a stream of context events. The SOM-ART network may generate object type clusters from the set of micro-feature vectors that are supplied to the SOM-ART network. Each cluster represents an observed statistical distribution of a particular thing or event being observed in the SOM-ART network. Importantly, the computer vision engine requires no training using predefined object definitions in order to perform the cluster discovery and object type classification. The SOM-ART network is adaptive and able to learn by discovering the object type clusters and classifying objects in parallel.
1. A computer-implemented method for characterizing one or more objects depicted in frames of video captured by a video camera, the method comprising:
for each successive video frame:
identifying one or more foreground objects depicted in the video frame, and for each foreground object:
determining one or more appearance characteristics of the foreground object from pixels of the video frame depicting the foreground object, and
determining one or more kinematic characteristics of the foreground object from the pixels of the video frame depicting the foreground object; and
clustering the determined appearance characteristics and the determined kinematic characteristics of each of the foreground objects to determine one or more object type clusters for classifying objects depicted in the frames of video.
2. The method of claim 1, further comprising, generating, for each identified foreground object, for each frame, a micro-feature vector from the determined appearance characteristics and the determined kinematic characteristics of the foreground object.
3. The method of claim 2, wherein clustering the determined appearance characteristics and the determined kinematic characteristics of each of the foreground objects comprises processing each of the micro-feature vectors by a self-organizing map adaptive resonance theory (SOM-ART) network.
determining that at least a first object type cluster has matured; and
classifying a first foreground object depicted in a subsequent video frame captured by the video camera as depicting a first object type corresponding to the first object type cluster.
5. The method of claim 1, wherein at least one of the appearance characteristics is derived from a color characteristics of the pixels of the video frame depicting the foreground object.
6. The method of claim 1, wherein at least one of the kinematic characteristics corresponds to a coordinate position of the foreground object in the frame of video.
7. The method of claim 1, wherein at least one of the kinematic characteristics corresponds to either an acceleration or velocity of the foreground object over a plurality of the video frames.
8. A computer-readable storage medium containing a program which, when executed by a processor, performs an operation for characterizing one or more objects depicted in frames of video captured by a video camera, the operation comprising:
9. The computer-readable storage medium of claim 8, wherein the operation further comprises, generating, for each identified foreground object, for each frame, a micro-feature vector from the determined appearance characteristics and the determined kinematic characteristics of the foreground object.
10. The computer-readable storage medium of claim 9, wherein clustering the determined appearance characteristics and the determined kinematic characteristics of each of the foreground objects comprises processing each of the micro-feature vectors by a self-organizing map adaptive resonance theory (SOM-ART) network.
12. The computer-readable storage medium of claim 8, wherein at least one of the appearance characteristics is derived from a color characteristics of the pixels of the video frame depicting the foreground object.
13. The computer-readable storage medium of claim 8, wherein at least one of the kinematic characteristics corresponds to a coordinate position of the foreground object in the frame of video.
14. The computer-readable storage medium of claim 8, wherein at least one of the kinematic characteristics corresponds to either an acceleration or velocity of the foreground object over a plurality of the video frames.
a memory containing a program, which, when executed on the processor is configured to perform an operation for characterizing one or more objects depicted in frames of video captured by a video camera, the operation comprising:
identifying one or more foreground objects depicted in the video frame,
and for each foreground object:
16. The system of claim 15, wherein the operation further comprises, generating, for each identified foreground object, for each frame, a micro-feature vector from the determined appearance characteristics and the determined kinematic characteristics of the foreground object.
17. The system of claim 16, wherein clustering the determined appearance characteristics and the determined kinematic characteristics of each of the foreground objects comprises processing each of the micro-feature vectors by a self-organizing map adaptive resonance theory (SOM-ART) network.
19. The system of claim 15, wherein at least one of the appearance characteristics is derived from a color characteristics of the pixels of the video frame depicting the foreground object.
20. The system of claim 15, wherein at least one of the kinematic characteristics corresponds to a coordinate position of the foreground object in the frame of video.
21. The system of claim 15, wherein at least one of the kinematic characteristics corresponds to either an acceleration or velocity of the foreground object over a plurality of the video frames.
US13/622,281 2009-08-31 2012-09-18 Identifying anomalous object types during classification Active US8548198B2 (en)
US12/551,276 US8270733B2 (en) 2009-08-31 2009-08-31 Identifying anomalous object types during classification
US13/622,281 US8548198B2 (en) 2009-08-31 2012-09-18 Identifying anomalous object types during classification
US12/551,276 Continuation US8270733B2 (en) 2009-08-31 2009-08-31 Identifying anomalous object types during classification
US20130022242A1 US20130022242A1 (en) 2013-01-24
US8548198B2 true US8548198B2 (en) 2013-10-01
ID=43625015
US12/551,276 Active 2030-12-09 US8270733B2 (en) 2009-08-31 2009-08-31 Identifying anomalous object types during classification
US13/622,281 Active US8548198B2 (en) 2009-08-31 2012-09-18 Identifying anomalous object types during classification
US (2) US8270733B2 (en)
US9111148B2 (en) * 2012-06-29 2015-08-18 Behavioral Recognition Systems, Inc. Unsupervised learning of feature anomalies for a video surveillance system
2009-08-31 US US12/551,276 patent/US8270733B2/en active Active
2012-09-18 US US13/622,281 patent/US8548198B2/en active Active
US8270733B2 (en) 2012-09-18
US20110052068A1 (en) 2011-03-03
US20130022242A1 (en) 2013-01-24
Nascimento et al. 2009 Trajectory classification using switched dynamical hidden Markov models
KR20060012570A (en) 2006-02-08 Video scene background maintenance using change detection and classification
US20130241730A1 (en) 2013-09-19 Alert volume normalization in a video surveillance system
Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:COBB, WESLEY KENNETH;FRIEDLANDER, DAVID;GOTTUMUKKAL, RAJKIRAN KUMAR;AND OTHERS;SIGNING DATES FROM 20090828 TO 20090829;REEL/FRAME:034568/0337
Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:BEHAVIORAL RECOGNITION SYSTEMS, INC.;REEL/FRAME:034881/0424