Patent Description:
Many security systems observe areas where crowds form and it is necessary in many circumstances to estimate the size of the crowd for monitoring foot-traffic through the area or for providing services to the area to accommodate the crowd or for other reasons. There are numerous techniques that estimate crowd levels to discover the number of humans in the crowd. The techniques range from simple pixel level techniques such as background subtraction based blob counting to complex pattern recognition techniques such as body part detection and combined head pattern generation. Each technique has its own advantages and disadvantages.

Document D1 :<CIT> (<CIT>) discloses a method for crowd counting based on a division of an image in small areas and combining the estimation from all areas.

A single crowd estimation technique may not be suitable for all environmental and crowd conditions. For example, background subtraction techniques have inferior performance when there is an overlap of humans (i.e., an occlusion). Similarly, body part recognition is also affected in cases of occlusions at high crowd densities, thereby reducing the accuracy of the technique. However, combined head pattern techniques are observed to perform better at high crowd densities due to the underlying concept of learning combined head patterns, yet they tend to have lower accuracies at sparse crowd levels or low crowd densities.

Thus, what is needed is a method and system for real-time crowd estimation which provides improved accuracy in a variety of crowd conditions and crowd locations. Furthermore, other desirable features and characteristics will become apparent from the subsequent detailed description and the appended claims, taken in conjunction with the accompanying drawings and this background of the disclosure.

According to at least one embodiment of the present disclosure, a method for crowd level estimation is provided. The method is defined by claim <NUM>.

According to another embodiment of the present disclosure, a system for crowd level estimation is provided. The system is defined by claim <NUM>.

In accordance with a further embodiment of the present disclosure, a computer readable medium is provided. The computer readable medium is defined by claim <NUM>.

The accompanying figures, where like reference numerals refer to identical or functionally similar elements throughout the separate views and which together with the detailed description below are incorporated in and form part of the specification, serve to illustrate various embodiments and to explain various principles and advantages in accordance with a present embodiment.

Skilled artisans will appreciate that elements in the figures are illustrated for simplicity and clarity and have not necessarily been depicted to scale.

Referring to <FIG>, an illustration <NUM> depicts a crowd <NUM> at a location <NUM> and a camera <NUM> arranged to capture images of the crowd <NUM> at the location <NUM>. <FIG> depicts an illustration <NUM> of media <NUM> capturing images <NUM>, <NUM>. The images <NUM> are images of a high crowd level and the images <NUM> are images of a low crowd level.

Referring to <FIG>, a diagram <NUM> depicts a system for crowd estimation in accordance with an example. The system includes an input module <NUM> for receiving an image of the crowd <NUM>. In accordance with the example a plurality of crowd estimation technique calculators <NUM> receive the image of the crowd <NUM> from the input module <NUM> and estimate crowd counts <NUM> therefrom. An equal plurality of performance modeling modules <NUM> are coupled to each of the crowd estimation technique calculators <NUM> for modeling each of the plurality of crowd estimation techniques based on an accuracy thereof at different crowd levels and/or at different locations.

A crowd estimation technique integration module <NUM> selects one or more of the plurality of crowd estimation techniques in response to the performance modeling thereof and an estimated crowd level and/or an estimated location. The crowd estimation technique integration module <NUM> then estimates the crowd count of the crowd in the received image in accordance with the selected one or more of the plurality of crowd estimation techniques and outputs a final crowd count <NUM>.

Accordingly, performance models of individual crowd estimation techniques are created at different crowd levels by using incoming image frames to generate the estimated crowd counts for the different crowd estimation techniques. Then, a crowd level estimation method determines which estimated crowd count to select or assign a high confidence value. In accordance with one example the input module <NUM> can receive the image of the crowd and determine a region of interest within the image of the crowd. The crowd estimation technique integration module <NUM> then estimates one or both of the crowd level of the crowd in the region of interest within the image of the crowd or the location of the crowd in the region of interest within the image of the crowd.

In the invention, the input module <NUM> receives the image of the crowd and divides the image into a plurality of sub-regions. The input module could divide the image of the crowd into the plurality of sub-regions in consideration of a view point of the camera <NUM> which has captured the image or in respect of other criteria. The crowd estimation technique integration module <NUM> then selects one or more of the plurality of crowd estimation technique calculators <NUM> for each of the plurality of sub-regions in response to the performance modeling of the one or more of the plurality of crowd estimation techniques by the corresponding one of the plurality of performance modeling modules <NUM> and an estimated crowd level and/or an estimated location for the one of the plurality of sub-regions. The crowd estimation technique integration module <NUM> then estimates the crowd count of the crowd in each of the plurality of sub-regions in accordance with the selected one or more of the plurality of crowd estimation techniques for that one of the plurality of sub-regions. Finally, the crowd estimation technique integration module <NUM> combines the estimated crowd counts for each of the plurality of sub-regions to obtain the final crowd count <NUM> of the crowd in the received image.

In accordance with the invention the plurality of performance modeling modules <NUM> could assign a real-time confidence value to each of the plurality of crowd estimation techniques in accordance with the performance modeling thereof. The system may then include a confidence value observer <NUM> coupled to the crowd estimation technique integration module <NUM> for removing one of the plurality of crowd estimation technique calculators <NUM> from selection when the real-time confidence value of the one of the plurality of crowd estimation techniques falls below a confidence value threshold.

The crowd estimation technique integration module <NUM> could further select multiple ones of the plurality of crowd estimation technique calculators <NUM> and combine the crowd estimation results (crowd counts) <NUM> from the multiple crowd estimation technique calculators <NUM> to estimate the crowd count of the crowd in the received image. In accordance with the present embodiment, the crowd estimation technique integration module <NUM> can dynamically combine the crowd estimation results <NUM> from the multiple crowd estimation technique calculators <NUM> in accordance with the real-time confidence value thereof to estimate the final crowd count <NUM> of the crowd in the received image of the crowd <NUM>. The crowd estimation results <NUM> can be combined in accordance with an inverted weighted sum approach or in accordance with a normalized weighted sum approach.

A further enhancement of the system depicted in the diagram <NUM> could involve adding a foreground measurement module <NUM> coupled between the input module <NUM> and the crowd estimation technique integration module <NUM> to measure a crowd level in a foreground of the image of the crowd to provide an estimated crowd level for use by the crowd estimation technique integration module <NUM> when selecting the one or more of the plurality of crowd estimation technique calculators <NUM>.

Referring to <FIG>, a diagram <NUM> depicts a system for crowd level estimation in accordance with a second aspect of the present embodiment. The system depicted in the diagram <NUM> implements performance modeling of crowd estimation techniques by the performance modeling modules <NUM> for each of the one or more crowd estimation technique calculators <NUM> determining a plurality of performances of the corresponding crowd estimation technique calculator <NUM> at multiple crowd levels (e.g., HIGH crowd levels, LOW crowd levels) and modeling the performance of the crowd estimation technique calculator <NUM> in response to the plurality of performances of the crowd estimation technique calculator <NUM> at the multiple crowd levels.

This performance modeling operation of the performance modeling modules <NUM> is shown in a flowchart <NUM> of <FIG>. Each of the performance modeling modules <NUM> collect images at different crowd levels (Step <NUM>) and categorize those images into low crowd images and high crowd images (Step <NUM>). Each performance modeling module <NUM> then models the performance of the corresponding crowd estimation technique calculator <NUM> in response to the plurality of performances of the crowd estimation technique calculator <NUM> at the different crowd levels (Step <NUM>).

Referring back to <FIG>, the performance modeling modules <NUM> can also determine a plurality of performances of the corresponding crowd estimation technique calculator <NUM> at locations of interest and model the performance of the crowd estimation technique calculator <NUM> in response to the plurality of performances of the crowd estimation technique calculator <NUM> at the locations of interest.

In accordance with the present embodiment, a performance modeling module <NUM> may model the performance of a corresponding crowd estimation technique calculator <NUM> by determining an error distribution <NUM> of the plurality of performances of the crowd estimation technique, such as by determining an error of crowd counting for each of the plurality of performances and/or by determining a standard deviation of the error distribution for each of the plurality of performances of the crowd estimation technique, as an indicator of performance of the crowd estimation technique calculator <NUM>.

Referring to <FIG> and <FIG>, graphs <NUM>, <NUM> depict error distribution for crowd estimation in accordance with the second aspect of the present embodiment. The graph <NUM> depicts a graph of error distribution for crowd estimation of high crowd level crowds and the graph <NUM> depicts a graph of error distribution for crowd estimation of low crowd level crowds. Validation of an accuracy of the crowd estimation technique calculator <NUM> with image samples at different crowd levels is used by the performance modeling module <NUM> in accordance with the present embodiment to generate the error distribution <NUM> at the considered crowd levels, the error referring to the deviation in the crowd estimation from the actual number of people.

The standard deviation (σ) of the error distribution <NUM> indicates the suitability of the crowd estimation technique calculator <NUM>. When the count estimate error is less, σ is small. For low crowd level σ as shown in the graph <NUM>, the error distribution indicates the crowd estimation technique calculator <NUM> has less error for low crowd levels as compared to high crowd levels (i.e., as shown in the distribution graph <NUM>). The calculation of the standard deviation is shown in Equation (<NUM>) below. <MAT> where M is the number of samples, xi is the error of the ith sample. The Equation (<NUM>) shows that if σlow « σhigh, the particular crowd estimation technique calculator <NUM> being performance modeled by the corresponding performance modeling module <NUM> is suitable for low crowd level estimation.

Referring back to <FIG>, the performance modeling module <NUM> may alternatively model the performance of the corresponding crowd estimation technique calculator <NUM> by determining an accuracy metric for the plurality of performances of the crowd estimation technique, wherein the accuracy metric may include a F-score and wherein the performance modeling module <NUM> determines the accuracy metric for the plurality of performances of the corresponding crowd estimation technique calculator <NUM> by determining a variance of the F-score <NUM> for the plurality of performances of the crowd estimation technique. F-score is a measure of performance based on the number of humans not detected and other regions falsely detected as humans. The performance modeling module <NUM> may determine the variance of the F-score (F-score distribution) <NUM> with respect to a mean of F-scores for the multiple performances of the crowd estimation technique calculator <NUM> and then determine an indicator of performance of the crowd estimation technique calculator <NUM> in response to both the variance of the F-score for the multiple performances of the crowd estimation technique calculator <NUM> and the F-score distribution with respect to the mean of F-scores for the multiple performances of the crowd estimation technique calculator <NUM>.

<FIG> depicts a graph <NUM> showing F-score variance at a first crowd level <NUM> and a second crowd level <NUM>. The F-score for samples at different crowd levels is used to find the F-Score variance at these crowd levels. The variance V(Fcr) of F-scores at a particular crowd level can be calculated from Equation (<NUM>) below. <MAT> where cr is low or high crowd level, M is the number of samples, µ is a mean of F-scores and <MAT> is the F-score for the ith sample. If V(Flow) «V(Fhigh) and µlow » µhigh, the particular crowd estimation technique calculator <NUM> being performance modeled by the corresponding performance modeling module <NUM> is suitable for low crowd level estimation.

Referring to <FIG>, a diagram <NUM> depicts an exemplary system for crowd level estimation in accordance with a third aspect of the present embodiment. In accordance with this third aspect, a crowd level estimation module <NUM> provides an estimated crowd level to the crowd estimation technique integration module <NUM> for use in selecting a most appropriate one of the crowd estimation technique calculators <NUM>. The crowd level estimation module <NUM> can estimate a crowd level of the crowd in the input image received by the input module <NUM> in response to a crowd density level. This could be accomplished by focusing on a region of interest within the image. The input module <NUM> could receive the input image of the crowd and determine the region of interest within the input image of the crowd. Then, the crowd level estimation module <NUM> could estimate the crowd level of the crowd within the region of interest of the input image in response to the crowd density level at that region of interest.

The crowd level estimation module <NUM> may include a spatial pixel variation model building unit <NUM> for modeling spatial variations of each of a plurality of crowd levels in response to pixel density variations thereof to generate multiple models of crowd level spatial variations. The crowd level estimation module <NUM> can then estimate the crowd level for automatic crowd estimation technique switching <NUM> by determining a similarity of the input image of the crowd to each of the models of crowd level spatial variations built by the spatial pixel variation model building unit <NUM> and estimating the crowd level of the crowd in the input image in response to a most similar one of the models of crowd level spatial variations.

In regards to determining the most similar one of the models of crowd level spatial variations, the crowd level estimation module <NUM> can estimate the crowd level of the crowd in the input image in response to a probability density function of a similarity of the input image of the crowd and each of the plurality of models of crowd level spatial variations. More specifically, the crowd level estimation module estimates the crowd level of the crowd in the input image in response to a best fit model of the plurality of models of crowd level spatial variations as determined by the probability density function of the similarity of the input image of the crowd and each of the plurality of models of crowd level spatial variations.

The spatial pixel variation model building unit <NUM> can generate the plurality of models of crowd level spatial variations in response to one or more of a grayscale crowd histogram or a red-green-blue (RGB) crowd histogram <NUM>, a crowd local binary pattern <NUM> or a crowd texture <NUM>. The automatic crowd estimation technique switching <NUM> of the crowd level estimation module <NUM> can switch crowd estimation techniques in response to an estimated discrete level of the crowd in the input image.

Thus, crowd levels such as low crowd levels and high crowd levels are estimated to select or to assign higher confidence values to crowd estimation technique calculators <NUM> which perform better at the estimated crowd level. The crowd level estimation module <NUM> is accomplished by first spatial pixel variation model building by the spatial pixel variation model building unit <NUM> and then automatic crowd estimation technique switching <NUM>.

Referring to <FIG>, a flow chart <NUM> and illustrations <NUM> depict the operation of the crowd level estimation module <NUM> where estimation is based on modeling spatial variations of crowd levels by the spatial pixel variation model building unit <NUM>. The flowchart <NUM> depicts the spatial variation modeling process in accordance with the present embodiment. At a location of interest (Step <NUM>), the camera <NUM> acquires images of the crowd <NUM> (Step <NUM>). Training images of the crowd are extracted for required crowd levels (e.g., high crowd level or low crowd level) (Step <NUM>). The spatial pixel variations are then extracted from the training images (Step <NUM>) and spatial pixel variation models are developed for the required crowd levels (Step <NUM>).

Turning to the illustrations <NUM>, each of the steps of the flowchart is shown pictorially. At an illustration <NUM>, the camera <NUM> monitoring the location (location of interest) <NUM> is selected. In an illustration <NUM>, the video (media <NUM>) is recorded from the location of interest <NUM> covering different crowd levels ranging from high crowd levels in images <NUM> to low crowd levels in images <NUM>.

The illustrations <NUM> correspond to Step <NUM> in the flowchart <NUM> where training images <NUM> for high crowd levels and training images <NUM> for low crowd levels are extracted. In this manner, training images (image frames) <NUM>, <NUM> with different crowd levels covering a 'no person case' to 'a fully crowded case' are extracted from the video (video clip) <NUM> recorded at Step <NUM> in the flowchart <NUM>.

At the next Step <NUM>, spatial pixel variations are extracted. A histogram approach for extracting spatial pixel variation is provided as an example in the illustration <NUM>. A grey scale histogram of an image is a frequency representation of the pixel intensities grouped at discrete pixel intensity levels called bins. Grey scale histograms <NUM>, <NUM> of all the extracted image frames <NUM>, <NUM> are recorded with <NUM> bins. The image-histogram pairs are grouped into high crowd level image frames <NUM> and histograms <NUM> and low crowd level images <NUM> and histograms <NUM> based on the number of humans in the images <NUM>, <NUM>.

At each crowd level, a bin-wise frequency averaging is performed considering all the image-histogram pairs. The averaging forms histogram models <NUM>, <NUM> for each crowd level as pictured in the illustration <NUM>. In operation, incoming image images (imageframes) <NUM>, <NUM> are compared against these histogram models <NUM>, <NUM> to estimate a crowd level for each image frame.

Referring to <FIG>, a flow chart <NUM> and illustrations <NUM> depict the operation of the crowd level estimation module <NUM> where estimation is based on automatic crowd estimation technique switching by the automatic crowd estimation technique switching <NUM> (<FIG>). The flowchart <NUM> depicts the automatic crowd estimation technique switching process in accordance with the present embodiment. At a location of interest (Step <NUM>), the camera <NUM> acquires a live stream video of images of the crowd <NUM> (Step <NUM>). The spatial pixel variations are then extracted from the acquired images (Step <NUM>) and crowd level estimation is performed by probability calculation based on similarity determination (Step <NUM>). Processing then selects or integrates the appropriate crowd estimation technique calculator <NUM> (Step <NUM>).

Turning to the illustrations <NUM>, each of the steps of the flowchart are shown pictorially. At an illustration <NUM>, the camera <NUM> initiates a live video stream of the location of interest <NUM>. In an illustration <NUM>, the automatic periodic capture of image frames of the crowd <NUM> is initiated for the location of interest <NUM>. The user can define an appropriate time interval for image frame capture <NUM>.

At the next Step <NUM>, spatial pixel variations are extracted. An exemplary histogram approach for extracting spatial pixel variation <NUM> extracts a grey scale histogram <NUM> of an image as a frequency representation of the pixel intensities grouped at discrete pixel intensity levels.

At the next Step <NUM>, the histogram <NUM> is compared against all the histogram models <NUM>, <NUM> generated in the illustration (model building stage) <NUM> (<FIG>). The histogram <NUM> is compared to the histogram model <NUM> at the illustration <NUM> and compared to the histogram model <NUM> at the illustration <NUM>. The comparison is performed by calculating similarity scores between the histogram <NUM> of the incoming image frame and the histogram models <NUM>, <NUM>. Examples of the similarity calculation methods include the correlation method, the Bhattacharya distance method, the Chi-square method and the intersection method. The similarity calculation results in each similarity method acting as a classifier on whether the incoming image frame resembles a high crowd level (the illustration <NUM>) or a low crowd level (the illustration <NUM>).

For example, using four different pixel variation modeling methods with each method outputs histogram model <NUM> or <NUM> being compared against that of an incoming image frame <NUM> by four different similarity calculations would result in sixteen classifications. A Probability Density Function (PDF) can be constructed based on these sixteen classifications as shown in Equations (<NUM>) and (<NUM>). <MAT> <MAT>.

At Step <NUM>, an incoming image frame <NUM> is classified to a particular crowd level based on the highest probability calculated at step <NUM>. For crowd estimation technique selection at Step <NUM>, the crowd estimation technique calculator <NUM> with the lowest σ or V(Fcr) with a high Fcr at the estimated crowd level is selected.

For crowd estimation technique integration at Step <NUM>, the final count estimate (Fcount) is calculated using Equation (<NUM>). <MAT> where i = <NUM> to N crowd estimation techniques and Zri is a re-weighted confidence value calculated by Equation (<NUM>). <MAT> where Znormi is the normalized confidence value in the range [<NUM>,<NUM>] calculated using Equation (<NUM>). <MAT> where, Z can either be a set of σ or a set of V(Fcr) generated for all the crowd estimation technique calculators <NUM>. For example, where results from a first crowd estimation technique calculator <NUM> (e.g., a combined head pattern estimation technique) and a second crowd estimation technique calculator <NUM> (e.g., an individual head pattern estimation technique) can be represented as Count<NUM> and Count<NUM>, Equation (<NUM>) shows mean averaging of the confidence values. <MAT> where Z<NUM> >> Z<NUM> and Z<NUM> + Z<NUM> = <NUM> (for example Z<NUM> could be <NUM> and Z<NUM> could be <NUM>).

Referring to <FIG>, a flowchart <NUM> depicts a method for crowd level estimation in accordance with the present embodiment. When an input image of a crowd is received (Step <NUM>), each crowd estimation technique is applied to the image (Step <NUM>) and crowd counts are calculated for each of the crowd estimation techniques (Step <NUM>). At the same time, spatial pixel variations are extracted from the received input image (Step <NUM>). The extracted spatial pixel variations are compared against spatial pixel models to find the highest similarity as described above (Step <NUM>). The crowd level is determined from the comparisons in Step1010 and confidence values are assigned (Step <NUM>). The counts calculated for each of the crowd estimation techniques in Step <NUM> are then integrated with the crowd level/confidence values determined/assigned in Step <NUM> (Step <NUM>) to estimate the final crowd count (Step <NUM>).

Methods in accordance with the present embodiment can also be used to select the best performing crowd estimation technique. In this case, the incoming image frame is not processed by all crowd estimation techniques; only the selected techniques will process the incoming image frame. Referring to <FIG>, a flowchart <NUM> depicts this selection process. The input image is received (Step <NUM>) and spatial pixel variations are extracted (Step <NUM>). The spatial pixel variations are compared against all spatial pixel models (Step <NUM>) to determine the crowd level and select the crowd estimation technique for that crowd level (Step <NUM>). The selected crowd estimation technique is applied (Step <NUM>) to estimate the final crowd count (Step <NUM>).

Thus, it can be seen that the present embodiment provides methods and systems for real time robust and optimized crowd estimation. When analyzed closely, it is possible to identify and/or model multiple techniques which can complement each other. In accordance with present embodiments, methods and systems to automatically switch between these crowd estimation techniques depending on a current crowd level (low crowd level, high crowd level) and other parameters tap these advantages to provide optimized real time crowd estimation with improved accuracy in a variety of crowd conditions and crowd locations.

In the aforementioned embodiment, the functions of the system for crowd estimation depicted as the diagrams <NUM>, <NUM>, and <NUM> may be implemented, for example, by a processor included in a computer device operating in accordance with a program. <FIG> depicts a configuration example of the computer device according to the present embodiment. The computer device <NUM> includes a processor <NUM> and a memory <NUM>. The memory <NUM> includes a volatile memory and/or a non-volatile memory. The memory <NUM> stores a software (computer program) to be executed on the processor <NUM> in, for example, the non-volatile memory. The processor <NUM> is, for example, a Central Processing Unit (CPU) or the like, and the control and the operations executed by the computer device <NUM> are achieved by, for example, the processor <NUM> operating in accordance with the computer program loaded from the memory <NUM>. The processor <NUM> may load the computer program from an external memory of the computer device <NUM> and execute the loaded computer program instead of loading the computer program from the memory <NUM> in the computer device <NUM>.

The above computer program can be stored and provided to the computer device using any type of non-transitory computer readable media. Non-transitory computer readable media include any type of tangible storage media. Examples of non-transitory computer readable media include magnetic storage media (such as floppy disks, magnetic tapes, hard disk drives, etc.), optical magnetic storage media (e.g. magneto-optical disks), CD-ROM (compact disc read only memory), CD-R (compact disc recordable), CD-R/W (compact disc rewritable), and semiconductor memories (such as mask ROM, PROM (programmable ROM), EPROM (erasable PROM), flash ROM, RAM (random access memory), etc.). The program may be provided to a computer using any type of transitory computer readable media. Examples of transitory computer readable media include electric signals, optical signals, and electromagnetic waves. Transitory computer readable media can provide the program to a computer via a wired communication line (e.g. electric wires, and optical fibers) or a wireless communication line.

Claim 1:
A method for estimating a crowd count of a crowd comprising:
modeling performance of each of a plurality of crowd estimation techniques at different crowd levels and/or at different locations, by determining at least one of an error of crowd counting and a standard deviation of an error distribution of crowd counting, the plurality of crowd estimation techniques being techniques for estimating a crowd density of the crowd and discovering the number of humans in the crowd;
receiving an image of the crowd;
dividing the image of the crowd into a plurality of sub-regions,
selecting one or more of the plurality of crowd estimation techniques in response to the performance modeling of the one or more of the plurality of crowd estimation techniques and an estimated crowd level and/or an estimated location,
wherein selecting one or more of the plurality of crowd estimation techniques comprises selecting one or more of the plurality of crowd estimation techniques for each of the plurality of sub-regions in response to the performance modeling of the one or more of the plurality of crowd estimation techniques and an estimated crowd level and/or an estimated location for the one of the plurality of sub-regions; and
estimating a crowd count of the crowd in the received image in accordance with the selected one or more of the plurality of crowd estimation techniques,
wherein estimating the crowd count of the crowd in the received image comprises:
estimating the crowd count of the crowd in each of the plurality of sub-regions in accordance with the selected one or more of the plurality of crowd estimation techniques for that one of the plurality of sub-regions; and
combining the estimated crowd counts for each of the plurality of sub-regions to obtain the estimated crowd count of the crowd in the received image.