Method and system for queue length analysis

A system and method for analyzing queues in frames of video enables operators to preferably draw three regions of interest overlaid upon the video as short, medium, and long queue regions that form a notional queue area within the video. The regions are drawn with knowledge of, or in anticipation of, foreground objects such as individuals and vehicles waiting for service in a queue. Examples include retail point of sale locations or for automated teller machine (ATM) transactions. In conjunction with a video analytics system that analyzes the movement of the foreground objects relative to the queue regions, the system determines the number of objects occupying each queue region, length of the queue, and other queue-related statistics. The system can then create reports and send messages that include the queue analysis results for directing operators to change their staffing resources as part of a real-time queue servicing and optimization response.

BACKGROUND OF THE INVENTION

Video security systems have been traditionally used to help protect people, property, and reduce crime for homeowners and businesses alike and have become an increasingly cost-effective tool to reduce risk. Modern systems with video analytics capabilities provide the ability to detect and track individuals and objects within monitored scenes. These systems can provide both live monitoring of individuals, and forensic analysis of saved security video data to spot trends and search for specific behaviors of interest.

More recently, these video systems have been used to track usage and facilitate resource management, in general. For example, of increasing interest is the ability to identify and analyze a notional queue of objects. Examples here might be a line of individuals queueing at a point of sale location or a line of cars at a drive up window.

A number of solutions exist for analyzing queues. In one, areas of interest are defined within a frame of video to provide an estimate of the number of individuals in the area. Another solution defines an area within a scene of video to detect a queue of vehicles in the scene, where the region definition is calibrated in conjunction with radar-based sensors mounted in traffic lanes. Yet another solution defines separate regions within the scene and estimates wait times for objects in the queue relative to a difference in service times of two or more events associated with objects within the regions. In yet another example of analyzing queues, a system divides scene into slots, where each slot is approximately the size of an individual. The system detects a queue within the video based on motion of the individuals across the slots and counts the individuals that occupy the slots. Finally, still another system estimates wait times for individuals performing retail transactions in a notional transaction queue. The system first identifies each individual and the items they present for transaction at a point of sale location. Then, the system determines the time it takes to transact each item, and estimates the total service time for an individual as the aggregate of the transaction processing times for their items.

SUMMARY OF THE INVENTION

Current systems and methods for analyzing queues have problems. The current solutions that estimate the number of individuals in a queue and their wait times provide inaccurate estimates when other foreground objects occlude the individuals in a scene and when a group of individuals arrive or converge within a scene in a short period of time. Solutions that rely on radar data from sensors in conjunction with video data to determine vehicles in a queue are complicated and prone to error. This is because these systems require calibration between the radar sensors and the video camera taking the video data footage and require measuring the speed of the vehicles.

In other examples, dividing scenes of video into human-sized slots requires careful selection of the video camera angle when capturing the scene and is prone to error as the distance between individuals in the scene and the video camera increases. This solution also suffers from the same occlusion and grouping issues. Finally, solutions that provide an estimate of wait times for individuals based on the aggregate of the estimate of wait times of their transacted items have difficulty identifying the number of items each individual presents at the point of sale. The items can be held within a person's hand, shopping basket or cart, in examples. As a result, these solutions typically have difficulty distinguishing between items. This impacts the transaction wait time estimate of the items, and therefore the overall wait time estimate of the individual.

The present invention takes a different approach by defining queue regions in a queue area. The present invention determines objects in each queue region based on spatial overlap between the objects and the queue regions. To avoid the pitfalls of current estimation solutions, an operator's prior knowledge of camera positioning and angle for capturing the video data of the scene can be used in the definition of the queue regions.

Moreover, the present invention enables operators to define preferably two, three or more queue regions forming the queue area. Operators draw the queue regions over the video data. The queue regions are associated with short, medium, and long queue lengths. The operators can then define event triggers for critical events associated with each of the queue regions.

Users that can benefit from such a system include establishments that provide retail point of sale transactions and businesses that provide drive-through or drive-up window customer service, such as banks with Automated Teller Machines (ATM) and fast food restaurants, in examples. Because the system creates reports and sends electronic messages such as audio messages that include the queue analysis results, in response to events that satisfy defined event triggers associated with the queue regions, the system can be utilized as part of a real-time retail staffing and resource optimization response.

In general, according to one aspect, the invention features a method for monitoring queues in a video analysis system. The method comprises generating video data of a monitored area and analyzing objects relative to a queue area within the video data to determine if the objects belong to one or more queue regions forming the queue area, and to determine a queue length.

Determining the queue length includes successively determining if each of the queue regions is occupied, in one implementation. The method enables drawing of the queue regions over the video data. In examples, the queue regions are rectangular or trapezoidal.

Preferably, the method defines a short queue region, a medium queue region, and a long queue region of the queue regions. Objects are determined to have entered the queue area by determining if the objects intersect with the queue area by a minimum queue area intersection amount. The method can also determine whether each object occupies the queue area by determining that each object intersects with the queue area by a minimum queue area intersection amount for a predetermined period of time.

The objects are preferably determined to belong to the one or more queue regions forming the queue area by determining areas of intersection of the objects upon the queue regions, and marking each object as belonging to one or more of the queue regions. The method marks each object as belonging to one or more of the queue regions if the area of intersection between each object and a queue region, known as a marked area of intersection, is at least equal to a minimum queue region intersection threshold.

The queue length is preferably determined by calculating a union, for each of the queue regions, of the marked areas of intersection, and comparing the union of the marked areas of intersection of the objects belonging to each of the queue regions, to a minimum occupancy area for each of the queue regions. The method determines a number of objects that are within the queue area by counting the objects that belong to the one or more queue regions forming the queue area.

In general, according to another aspect, the invention features a video analysis system for monitoring queues. The system comprises at least one video camera generating video data of a monitored area and a video analytics system that analyzes objects relative to a queue area within the video data to determine if the objects belong to one or more queue regions forming the queue area, and to determine a queue length.

The system can further include a security system workstation enabling definition of the queue regions forming the queue area. The security system workstation includes a display, a user interface application that enables access to the video data via the video analytics system, one or more user input devices, and a drawing tool for defining the queue regions, wherein the queue regions are drawn over the video data. In examples, the queue regions are rectangular or trapezoidal in shape.

The video analytics system typically determines if the objects belong to a short queue region, a medium queue region, and a long queue region. The video analytics system also determines the queue length by successively determining if each of the queue regions is occupied, and determines if the objects have entered the queue area, by determining if the objects intersect with the queue area by a minimum queue area intersection amount.

Additionally, the video analytics system can determine that each object occupies the queue area by determining that each object intersects with the queue area by a minimum queue area intersection amount for a predetermined period of time.

Further still, the video analytics system can determine whether objects belong to the one or more queue regions forming the queue area by determining areas of intersection of the objects upon the queue regions, and marking each object as belonging to one or more of the queue regions. The video analytics system marks each object as belonging to one or more of the queue regions if the area of intersection between each object and a queue region, known as a marked area of intersection, is at least equal to a minimum queue region intersection threshold.

In yet another example, the video analytics system determines the queue length by calculating a union, for each of the queue regions, of the marked areas of intersection, and comparing the union of the marked areas of intersection of the objects belonging to each of the queue regions, to a minimum occupancy area for each of the queue regions.

In general, according to yet another aspect, the invention features a method for determining occupancy of objects in a area, such as a queue, within a scene of video data using a video analysis system. The method defines queue or other regions forming the queue or other type of area, and determines that each object occupies the queue area by determining that each object intersects with the queue area by a minimum queue area intersection amount for a predetermined period of time.

Then, the method determine can areas of intersection of the objects upon the queue regions, and marks each object as occupying one or more of the queue regions, if the area of intersection between each object and a queue region, known as a marked area of intersection, is at least equal to a minimum queue region intersection threshold.

Additionally, the method can determine length of the queue area by first calculating a union, for each of the queue regions, of the marked areas of intersection, and then comparing the union of the marked areas of intersection of the objects occupying each of the queue regions, to a minimum occupancy area for each of the queue regions.

According to another feature, the method can accomplish defining the queue regions forming the queue area using a video analytics system of the video analysis system. In examples, defining the queue regions forming the queue area comprises defining a short queue region, a medium queue region, and a long queue region of the queue regions. The queue regions can be rectangular or trapezoidal, in examples.

In general, according to an additional aspect, the invention features a method of operation of a finite state machine of a video analytics system for determining whether an object has entered or exited queue regions forming a queue area, for example, across frames of video data. Firstly, the method assigns the object as initially being in an unknown state, and identifies a tracking mask associated with the object in a current frame of the video data.

Secondly, the method determines that the object remains in the unknown state when the object does not have a bounding box in a next frame of the video data. Thirdly, the method determines that the object has transitioned to a state indicating that the object has exited the queue or other type of regions, when the object has a tracking mask in the next frame of video data that does not overlap with any queue regions by a predetermined amount.

Finally, the method determines that the object has transitioned to a state indicating that the object has entered an identified queue region, or other type of region, of the queue regions, when the object has a tracking mask in the next frame of video data that overlaps with the identified queue region of the queue regions by the predetermined amount.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

FIG. 1shows a first example video security analysis system100monitoring a retail point of sale (POS) area102within a room110. The video analytics system100includes components that communicate over a data network136, which could be a dedicated security network, such as video cameras103, a metadata database162, a network video recorder130, a video analytics system132, a security system workstation120, and a speaker system164. A control system114controls the components over the security network136. The video analytics system132includes non-transitory memory118and an operating system158that runs on top of a central processing unit (CPU)116.

In operation, the video analytics system132analyzes objects relative to a queue area134within the video data to determine if the objects belong to one or more queue regions forming the queue area134. And this information is used to determine a queue length.

The network video recorder130records video data from the video cameras103. The video data usually includes metadata, such as time stamp information for each frame of the video data. Additionally, metadata database162can save the recorded video data, detected object data, and queue length event trigger data, in examples. The video analytics system132receives live video data over the security network136from the video cameras103, and receives historical video data over the security network136from either the database162or the network video recorder130, in examples.

The security system workstation120includes a user interface application123and a drawing tool122. Operators interact with the user interface application123and the drawing tool122via user input devices126such as a keyboard and mouse, and a touchscreen of a display124, in examples. Using the drawing tool122and the display124, the operator interacts with the video analytics system132to define regions of interest upon the video data, such as a notional queue area134and its queue regions. In one example, the operator defines the boundaries of the queue area134in response to anticipated traffic patterns of individuals112waiting for service within the retail point of sale area102.

To setup the system100, an operator positions one or more video cameras103over or outside the retail POS area102. This enables the field of view104of the video camera103to include foreground objects such as individuals112-1located in or near a queue area134within the retail POS area102. The field of view104also often includes a point of sale terminal106on top of a desk140located near the queue area134. This allows the video camera to capture the individuals112-1as they wait and/or perform transactions within the queue area134. If possible, operators also position the video camera103such that individuals112-2and112-3located well outside the queue area134are excluded from the field of view104.

Using the user interface application123, the operator can define event triggers associated with movement of objects relative to the queue area134and specifically the queue regions. The video analytics system132typically stores the event triggers as metadata associated with the video. The video analytics system132stores the metadata within or in connection with each video frame, and to the metadata database162, in examples.

In response to events that occur within the frames of video data that satisfy the defined event triggers, the video analytics system132can generate messages that include information associated with the events that satisfy the event triggers. The video analytics system132includes the messages in a report178. Automatic messages can also be generated such as audio messages via speaker164or electronic messages sent from the control system114to the point of sale terminal106. Additionally, the messages can be sent to other systems on the security network136or to systems on other networks via a gateway.

FIG. 2shows a second example video security analysis system100monitoring an Automated Teller Machine (ATM) lane174at a bank176. A related example would be a drive-up window at a fast-food restaurant.

To illustrate an alternative configuration, in this example, the video camera103has an integrated video analytics system132that operates as a functional equivalent of the separate, remote, dedicated video analytics system132in the system ofFIG. 1.

Operators position one or more video cameras103to capture objects such as vehicles182within or near a queue area134of the ATM lane174. The field of view104also includes an ATM184located near the queue area134. This allows the video camera to capture the vehicles182and their individuals112as the individuals112perform ATM transactions within the queue area134.

Using the user interface application123, the operator can define event triggers associated with movement of the vehicles182and other foreground objects relative to the queue area134. The video analytics system132typically stores the event triggers as metadata within each video frame.

In response to events that occur within the frames of video data that satisfy the defined event triggers, the video analytics system132can generate messages that include information associated with the events that satisfy the event triggers. The video analytics system132includes the messages in a report178.

FIG. 3is a flow chart showing a setup method500for defining queue regions that form a queue area134within a frame of image data taken by a video camera103according to principles of the invention.

In step502, an operator mounts a video camera103outside of or overlooking a region of interest to be monitored. The region includes or is anticipated to include foreground objects arranged in a queue. The operator aims the video camera103to include the foreground objects in the field of view104of the video camera103in step504. Then, in step506, the operator connects the video camera103to the security video system100.

According to step508, on the security system workstation120, the operator opens the drawing tool122. In step510, the drawing tool122loads the frame of image data from the video camera103. In step512, using the drawing tool122, the operator preferably defines three queue regions as overlays upon the frame of image data. The regions form a queue area134within the region of interest.

FIG. 4shows three exemplary queue regions including a short queue region144-1, a medium queue region144-2, and a long queue region144-3. Using the drawing tool122, an operator draws the queue regions upon an image frame108of video data displayed on the display124. Each queue region can be drawn using a different shape and size as determined by the operator based on his/her analysis objectives. In examples, the queue regions can overlap, be superimposed upon or included within another, or be arranged adjacent to one another in a linear fashion. Preferably, the operator defines the queue regions in a manner that most resembles an anticipated notional queue of objects awaiting service within the scene.

As a result, the queue regions will have different shapes depending on camera position. For example with an overhead, look-down camera, the queue regions will often be rectangular, stretching in the direction of the queue. On the other hand, if the camera is located to the side or ahead or behind the queue, the queue regions might be trapezoidal due to the perspective of the camera.

Returning toFIG. 3, in step514, the video analytics system132creates a frame buffer for the queue regions that form the queue area134, mapping coordinates of the queue area134to pixel array coordinates. The operator can edit the overlay regions in step516, returning to step512to redefine the regions. When the operator is done drawing the regions, the method transitions to step518.

In step518, the operator defines event triggers of interest associated with movement of objects relative to the queue regions. In response to events that satisfy the event triggers, the system executes actions associated with the events, such as sending alert messages over the security network or generating audio messages using speaker164, for example. Finally, in step520, the operator submits the defined queue regions and event triggers to the video analytics system132.

FIG. 5Ashows a method600for a “live video” example for how the video analytics system132determines queue length within a scene of video data. The example is with respect to the retail POS area102ofFIG. 1.

In step602, the video analytics system132receives the next frame or frames of video or a combination of several frames from a video camera103pointed at a retail POS area102within a room110. In step604, the analytics system132analyzes the video to identify foreground objects such as individuals112. According to step606, the analytics system132assigns bounding boxes or other tracking mask128for the individuals in the current video frame108.

FIG. 6shows bounding boxes128that the analytics system132has generated around foreground objects such as individuals112in an exemplary video frame108. In the video frame108, only the heads of the individuals112can be seen because the video camera is mounted to overlook the retail POS area102. For this reason, the individuals112are represented as oval-shaped objects. Individuals112-1,112-2, and112-3are located within or near short queue region144-1, medium queue region144-2, and long queue region144-3, respectively. While the illustrated analytics system132encloses individuals112within a rectangular bounding box128, triangular-shaped regions and other forms of tracking masks can enclose or otherwise represent the space each individual occupies within the video frame108.

It can also be appreciated that the analytics system132generates tracking masks or notional bounding boxes128around other types of foreground objects, such as for the vehicles182waiting in line to perform transactions at the ATM184of bank176inFIG. 2.

Operators will typically define a minimum queue area intersection amount142for the queue area134. This is used to first determine if objects such as the individuals112are located within or near the queue area134. In examples, individuals112-4and112-3are located outside and inside the queue area134, respectively. This is because bounding box128-4for individual112-4does not overlap the queue area134by at least the minimum queue area intersection amount142, and because bounding box128-3for individual112-3does overlap the queue area134by at least the minimum queue area intersection amount142.

Returning toFIG. 5A, in step608, the method determines if the current bounding box128for an individual112has entered the queue area134. The individual112has entered the queue area134if its bounding box128intersects with the queue area by at least the minimum queue area intersection amount142. If the individual112has entered the queue area134, the method transitions to step612to determine if objects entering the queue area remain within the queue area for a minimum occupancy period and are therefore not transient. Otherwise, the method transitions to step610.

FIG. 7A-7Cillustrate the determination of the minimum occupancy period calculated in step612.FIG. 7Ashows multiple individuals112located within a queue area134of a video frame108of a scene, with individual112-2located mostly within medium queue region144-2and completely located within the queue area134. The video frame108is the first frame in a series of consecutive frames. The operator defines a minimum consecutive number of frames or video run time148to assist in the calculation of the queue area minimum occupancy period, such as the number of frames corresponding to 5 or more seconds of video run time.

FIG. 7Bshows the subsequent frame of video of the same scene. The individuals112have not changed their positions within the scene with the exception of individual112-2, who is now located partially within medium queue region144-2and partially outside the queue area134. This is likely associated with the individual112-2starting to leave the queue area134.

FIG. 7Cshows still a further subsequent frame of video108of the same scene. The individuals112have not changed their positions within the scene, again with the exception of individual112-2, who is now located completely outside medium queue region144-2and mostly outside the queue area134. This shows that while all other individuals112have remained within the queue area134over the minimum consecutive number of frames148(frames of 5 seconds of runtime, in this example) of video, individual112-2has continued moving away from and is leaving the queue area134.

Returning toFIG. 5A, upon completion of step612, the method transitions to step614if the current bounding box128was determined to have entered the queue area134and remained within the queue area134for at least the minimum consecutive number of frames148. Otherwise, the method transitions to step610.

Step610is reached when the bounding box128associated with an object was determined to be effectively located outside of the queue area134. As a result, step610removes the bounding box128from the queue length analysis, and transitions to step616to look for more bounding boxes128within the video data.

Step614is reached when each bounding box128associated with an object was determined to be within the queue area134. In step614, the analytics system132concludes that the foreground object associated with the bounding box128occupies one or more queue regions and includes the bounding box128as part of the analysis for determining the queue length. The method then transitions to step616to look for more bounding boxes128.

If there are more bounding boxes128to process in step616, the method transitions to step618to go to the next bounding box128. Otherwise, the method transitions to step620. Upon completion of step618, the method transitions to the beginning of step608to determine if the next bounding box128has entered the queue area134.

In step620, the method determines intersections of the bounding boxes128collected in step614with the short, medium, and long queue regions144-1,144-2, and144-3, respectively, to infer the length of the queue area134.

In step622, the analytics system132identifies a minimum intersection threshold146-1,146-2, and146-3for each of the short144-1, medium144-2, and long144-3queue regions, respectively.

FIG. 9Ashows example minimum queue region intersection thresholds146defined for each of the queue regions forming the queue area134. If a bounding box128intersects with a queue region by at least an amount equal to that region's minimum queue region intersection threshold146, the analytics system132marks the object associated with the bounding box128as “belonging to” that region. This is important because the analytics system132determines the number of objects or individuals within each queue region by counting the number of bounding boxes128determined to “belong” within that queue region.

In the example, bounding boxes128-1,128-2, and128-3intersect with the short queue region144-1, medium queue region144-2, and long queue region144-3, respectively. Bounding box128-1intersects with the short queue region144-1by at least the minimum short queue region intersection threshold146-1. Bounding box128-2intersects with the medium queue region144-2by at least the minimum medium queue region intersection threshold146-2. However, bounding box128-3does not intersect with the long queue region144-1by at least the minimum long queue region intersection threshold146-3. As a result, the analytics system132concludes that the object associated with bounding box128-1belongs to short queue region144-1, the object associated with bounding box128-2belongs to medium queue region144-2, and the object associated with bounding box128-3does not belong to any region.

Returning toFIG. 8, in step624, for each of the collected bounding boxes128, the analytics system132marks each object as “belonging” to a queue region if the amount of its intersection of its bounding box128with a queue region exceeds the minimum intersection threshold146for that queue region. In step626, the method saves the “belonging” or queue region membership information for each of the bounding boxes as metadata within the frame of video data and to the metadata database162, in examples. Then, in step628, the analytics system132marks the area of intersection of each collected bounding box128upon the queue region(s) as a first step in determining the queue length.

FIG. 9Bprovides an example for how the analytics system132calculates the queue length. First, the analytics system132marks an area of intersection of each collected bounding box128upon the queue region(s). Then, the analytics system132calculates a separate union of the marked areas of intersection152for all bounding boxes128belonging to each of the short144-1, medium144-2, and long144-3queue regions.

Then, the union of the marked areas of intersection152for each of the queue regions is compared to an operator-defined minimum occupancy area188for each of the queue regions. Preferably, the short144-1, medium144-2, and long144-3queue regions can each have separate minimum short188-1, medium188-2, and long188-3occupancy areas.

In the example, the minimum occupancy area of the short region188-1covers the smallest area of the occupancy areas. However, the minimum occupancy areas188for each of the regions can be of any area that is less than the area of its respective queue region. The marked areas of intersection152for objects belonging to the short queue region144-1include marked areas of intersection152-5through152-10and152-11a, associated with bounding boxes128-5through128-11. In a similar fashion, the marked areas of intersection152for objects belonging to the medium queue region144-2include marked areas of intersection152-11b,152-12, and152-13, associated with bounding boxes128-11,128-12, and128-13. Though no objects/bounding boxes belong to the long queue region144-3, the analysis is the same for the long queue region144-3.

Returning toFIG. 8, in step630, the analytics system132first analyzes the short queue region144-1. For the short queue region144-1, the analytics system132calculates the union of marked areas of intersection152of the bounding boxes128belonging to the short queue region144-1, and identifies the minimum occupancy area188-1of the short queue region144-1in step632

In step634, the analytics system132determines if the union of the marked areas of intersection152of the bounding boxes128belonging to the short queue region144-1is less than the minimum occupancy area188-1of the short queue region144-1. If this statement is true, the analytics system132marks the queue as empty in step636, and transitions to step658to bypass analysis of the remaining queue regions. Otherwise, the method transitions to step638and marks the queue as not empty.

Then, for the medium queue region144-2, the method calculates the union of the marked areas of intersection152of the bounding boxes128belonging to the medium queue region144-2, according to step640. The method identifies the minimum occupancy area188-2of the medium queue region144-2in step642.

According to step644, the analytics system132determines if the union of the marked areas of intersection152of the bounding boxes128belonging to the medium queue region144-2is less than the minimum occupancy area188-2of the medium queue region144-2. If this statement is true, the analytics system132marks the queue as short in step646, and transitions to step658to bypass analysis of the remaining queue regions. Otherwise, the method transitions to step648.

In step648, for the long queue region144-3, the analytics system132calculates the union of marked areas of intersection152of the bounding boxes128belonging to the long queue region144-3, and identifies the minimum occupancy area188-3of the short queue region144-3in step650.

Then, in step652, the analytics system132determines if the union of the marked areas of intersection152of the bounding boxes128belonging to the long queue region144-3is less than the minimum occupancy area188-3of the long queue region144-3. If this statement is true, the analytics system132marks the queue as medium in step654, and transitions to step658. Otherwise, the method transitions to step656to mark the queue as long, and transitions to step658.

Step658clears the marked areas of intersection152within the queue regions, and transitions to step660. This resets buffers to enable calculation of the queue length for the next or subsequent frame108of video data.

In step660, the analytics system132saves the per-region object membership information for the current frame of video data and queue length event trigger information within the frame of video data108and to the metadata database162. This enables the generation of queue-related statistics associated with the queue regions. In one example, an operator can determine queue utilization as a function of queue length across a range of video data frames, by calculating the amount of time that each queue area134was of a particular queue length. Returning toFIG. 5A, upon completion of step620, the method transitions to step622. In response to changes in the queue length and other occurrences that satisfy the event triggers, the video analytics system132sends audio messages to the loudspeaker164and electronic alert messages over the security network136to the security system workstation120, in one example. The audio messages and electronic alert messages might indicate the need to change retail staffing assignments to address changes in the length of the queue134.

FIG. 5Bshows an exemplary method700for processing of historical video data footage of the ATM lane174inFIG. 2, to show one specific example. The example infers movement of vehicles182relative to a queue area134within the ATM lane174.

In step702, via the user interface application123on the security system workstation120, an operator selects a time range of historical video to obtain from the database162to analyze peak wait times at a drive-up ATM lane174of a bank176. In step704, the operator defines event triggers associated with determining peak wait times at the ATM184. Then, in step706, from the database162and/or network video recorder130, the operator selects the next frame of previously recorded video data from the selected time range.

According to step708, the analytics system132loads all metadata including bounding boxes128for foreground objects such as vehicles182and/or individuals112in the current video frame108. The metadata including the bounding boxes128were generated previously by the analytics system132during live processing of the video data, and were saved within the video data and/or metadata database162for future forensics-based usage. The method then transitions to step710.

In step710, the method determines if the current bounding box128for a vehicle182has entered the queue area134. The vehicle182has entered the queue area134if its bounding box128intersects with the queue area134by at least the minimum queue area intersection amount142. If the vehicle182has entered the queue area134, the method transitions to step712to determine if objects entering the queue area remain within the queue area for a minimum occupancy period and are therefore not transient. Otherwise, the method transitions to step714.

Upon completion of step712, the method transitions to step720if the current bounding box128was determined to have entered the queue area, and remained within the queue area for at least the minimum consecutive number of frames148. Otherwise, the method transitions to step714.

Step714is reached when the bounding boxes128associated with an object were determined to be effectively outside of the queue area134. As a result, step714removes the bounding box128from the queue length analysis, and transitions to step718to look for more bounding boxes128within the video data.

Step720is reached when the bounding box128associated with each object was determined to be within the queue area134. In step720, the analytics system132concludes that the foreground object associated with the bounding box128occupies one or more queue regions and includes the bounding box128as part of the analysis for determining the queue length. In this example, the foreground objects are vehicles182. The method then transitions to step718to look for more bounding boxes128.

If there are more bounding boxes128to process in step718, the method transitions to step716to go to the next bounding box128. Otherwise, the method transitions to step620. Upon completion of step716, the method transitions to the beginning of step710to determine if the next bounding box128has entered the queue area134.

In step620, the method determines intersections of the bounding boxes128collected in step720with the short, medium, and long queue regions144-1,144-2, and144-3, respectively, to infer the length of the queue area134.

Returning toFIG. 5B, the method transitions to step724. In step724, the method saves metadata created in response to changes in the queue length and other occurrences that satisfy the defined event triggers. In step726, if there are more frames to process, the method transitions back to step706to select the next frame of historical footage to process. Otherwise, the method transitions to step728.

In step728, in response to changes in the queue length and other occurrences that satisfy the defined event triggers in the saved metadata over the selected time range, the analytics system132generates a report178, and include the report178within an electronic message sent over the security network136to the security system workstation120for law enforcement and loss prevention personnel.

The present invention also utilizes a finite state machine (FSM) to reduce ephemeral motion of foreground objects such as individuals122and vehicles182across frames of video data. This enables more accurate calculations for determining whether the objects have entered or exited a queue region of the queue area134.

As the analytics system132processes one frame of video data to another, the FSM determines whether each object remains in its current state or transitions to another state. All objects are initially in an UNKNOWN state. States also include OUT-OF-REGIONS, and IN-REGION-N, where N is a unique number assigned to each of the queue regions for the queue area134.

An objects transitions from a current state S1to a next state S2, using the notation “S1→S2” according to the exemplary state transition table below. In the description, each tracking mask128associated with an object is determined to “sufficiently” overlap with a queue region by a predetermined amount defined by an operator. The exemplary state transition table is included herein below:

UNKNOWN→UNKNOWN when an object does not have an associated bounding box128or tracking mask in the next frame

UNKNOWN→OUT-OF-REGIONS when object has a bounding box128in the next frame that does not overlap sufficiently with any queue regions

UNKNOWN→IN-REGION-1when object has a bounding box128in the next frame that overlaps sufficiently with a queue region “1”

OUT-OF-REGIONS→OUT-OF-REGIONS when an object has a bounding box128in the next frame that does not overlap sufficiently with any queue regions

OUT-OF-REGIONS→UNKNOWN when an object does not have a bounding box128in the next frame

OUT-OF-REGIONS→IN-REGION-1when an object has a bounding box128in the next frame that overlaps sufficiently with a queue region “1”

IN-REGION-1→IN-REGION-2when an object has a bounding box128in the next frame that overlaps sufficiently with a queue region “1”

IN-REGION-1→UNKNOWN when an object does not have a bounding box128in the next frame

IN-REGION-1→OUT-OF-REGIONS when an object has a bounding box128in the next frame that does not overlap sufficiently with any queue regions

IN-REGION-N→IN-REGION-N when an object has a bounding box128in the next frame that overlaps sufficiently with queue region “N”

IN-REGION-N→UNKNOWN when an object does not have a bounding box128in the next frame

IN-REGION-N→OUT-OF-REGIONS when an object has a bounding box128in the next frame that does not overlap sufficiently with any queue regions