Source: https://patents.com/us-10402659.html
Timestamp: 2019-10-21 05:43:35
Document Index: 655289767

Matched Legal Cases: ['Application No. 2016203571', 'Application No. 2016203571', 'Application No. 2016203579', 'Application No. 2', 'Application No. 16171901', 'Application No. 10201604368']

US Patent # 1,040,2659. Predicting external events from digital video content - Patents.com
United States Patent 10,402,659
Bataller; Cyrille (Mougins, FR), Astrom; Anders (Villa Marina, SG)
Family ID: 1000004254120
16/003,518
US 20180293443 A1 Oct 11, 2018
15167701 May 27, 2016 10007849
Current CPC Class: H04N 7/18 (20130101); G06K 9/00718 (20130101); G06K 9/00255 (20130101); G06T 7/0002 (20130101); G06K 9/00778 (20130101); H04N 7/181 (20130101); G06K 9/00295 (20130101); G06K 9/00765 (20130101); G06K 9/00744 (20130101); G06T 7/80 (20170101); G06F 16/51 (20190101); G06T 7/292 (20170101); G06T 15/205 (20130101); G06K 9/6263 (20130101); G06K 9/00771 (20130101); G06K 9/6202 (20130101); G06T 19/006 (20130101); G06T 7/75 (20170101); G06T 7/13 (20170101); G06T 2207/20061 (20130101); G06T 2207/30168 (20130101); G08B 13/19602 (20130101); G08B 13/19608 (20130101); G06K 2009/00738 (20130101); G06T 2207/10016 (20130101); G06T 2207/30201 (20130101); G06T 2207/30232 (20130101); G06T 2207/30242 (20130101)
Current International Class: G06K 9/00 (20060101); G06T 7/292 (20170101); G06T 7/13 (20170101); G06T 7/73 (20170101); G06T 7/80 (20170101); G06K 9/62 (20060101); G06T 19/00 (20110101); G06T 15/20 (20110101); H04N 7/18 (20060101); G06F 16/51 (20190101); G06T 7/00 (20170101); G08B 13/196 (20060101)
Field of Search: ;382/100,103,106-107,155,162,168,173,181,190,209,224,232,238,254,274,276,286-295,305,312,297 ;340/540 ;707/706 ;600/301 ;706/21 ;705/3
7418431 August 2008 Nies
2008/0201116 August 2008 Ozdemir et al.
2010/0207762 August 2010 Lee et al.
2010/0299116 November 2010 Tomastik et al.
2011/0228092 September 2011 Park
2012/0256745 October 2012 Piett
2013/0054552 February 2013 Hawkins
2015/0025329 January 2015 Amarasingham
2016/0034813 February 2016 Hsu et al.
2131328 Dec 2009 EP
WO 2003/067884 Aug 2003 WO
WO 2007/062044 May 2007 WO
Australian Office Action for Application No. 2016203571, dated Jan. 10, 2017, 2 pages. cited by applicant .
Australian Office Action for Application No. 2016203571, dated Sep. 2, 2016, 4 pages. cited by applicant .
Australian Office Action in Application No. 2016203579, dated Jun. 22, 2016, 5 pages. cited by applicant .
Canadian Office Action for Application No. 2,931,743, dated Feb. 16, 2017, 4 pages. cited by applicant .
European Extended Search Report for Application No. 16171901.8, dated Oct. 18, 2016, 10 pages. cited by applicant .
Singapore Search Report for Application No. 10201604368T, dated Nov. 11, 2016, 5 pages. cited by applicant.
This application is a continuation (and claims the benefit of priority under 35 U.S.C. .sctn. 120) of U.S. application Ser. No. 15/167,701, filed May 27, 2016, now allowed, which claims the benefit of U.S. Provisional Application Ser. No. 62/167,930, filed May 29, 2015, and titled "Video Analytics of Video Information." Both of these prior applications are incorporated by reference in their entirety.
1. A system, comprising: one or more computers; and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising: detecting events shown within digital video content captured by one or more video cameras, the detected events being associated with corresponding event parameters and detection times within a first time period, including: applying one or more video analytics to portions of the digital video content, the one or more applied video analytics comprising an object classification process and at least one of an image acquisition process, an object detection process, an object recognition process, an event detection process, or an object tracking process; and detecting, based on an outcome of applying the one or more video analytics to the portions of the digital video content, at least one of the events; obtaining first data that identifies at least one external event, the obtained data comprising observed values of an external event parameter that characterize the external event during the first time period, wherein the first data is not derived from the digital video content and wherein the at least one external event is not shown within the digital video content; establishing a predictive model; detecting an additional event shown within the digital video content, the additional event being associated with a corresponding additional event parameter and a second detection time; in response to the detection of the additional event, applying the predictive model to a value of the additional event parameter; determining an expected value of the external event parameter at the second detection time, based on an outcome of applying the predictive model to the value of the additional event parameter; and transmitting data identifying the expected value of external event parameter to a communications device, the communications device configured to present a representation of the expected value of the external event parameter to a user through a corresponding interface.
2. The system of claim 1, wherein the predictive model correlates the values of the event parameters that characterize a portion of the detected events during the first time period.
3. The system of claim 1, wherein: the one or more computers further perform the operation of receiving a portion of the video content from the one or more video cameras; and the one or more video cameras comprise at least one of a pan-zoom-tilt video camera or a camera having a fixed field-of-view.
4. The system of claim 3, wherein detecting the additional event comprises: receiving an additional video content from the one or more video cameras; and detecting the additional event within the additional video content captured by the one or more video cameras.
5. The system of claim 1, wherein: detecting the event comprises, based on an outcome of applying the one or more video analytics to the portions of the digital video content, establishing the value of at least one of the event parameters; and detecting the additional event comprises, based on an outcome of applying the one or more video analytics to the portions of the digital video content, detecting the additional event and establishing the value of the additional event parameter.
6. The system of claim 1, wherein the establishing the predictive model comprises: applying a machine learning algorithm to data identifying the values of the external event parameters and the values of the event parameters that characterize the portion of the detected events during the first time period; and establishing a correlation between the values of the external event parameters and the event parameters, based on an outcome of the machine learning algorithm.
7. The system of claim 1, wherein the one or more computers further perform the operations of: obtaining second data specifying an observed value of the external event parameter at the second detection time; determining an existence of a variance between the actual and expected values of the external event parameters; and modifying the predictive model in accordance with the determined variance.
8. The system of claim 1, wherein the one or more computers further perform the operations of: establishing a plurality of predictive models that correlate the values of the external event parameters to values of corresponding ones of the event parameters; applying each of the predictive models to the value of the additional event parameter; determining a plurality of expected values of the external event parameter at the second detection time, based on an outcome of corresponding ones of the predictive models; obtaining second data specifying an observed value of the external event parameter at the second detection time; determining variances between the observed value and corresponding ones of the expected values; and selecting one or more of the event parameters as predictors of the observed value of the external event parameter based on the determined variances.
9. The system of claim 1, wherein: the external event comprises at least one of a number of customer inquiries or orders, a total number of individuals within a restricted area, or a total number of vehicles disposed within a restricted area; the detected events comprise at least one of queued individuals, individuals entering a restricted area through a corresponding entrance, or vehicles entering the restricted area through a corresponding entrance; and the event parameters comprise at least one of a number of the queued individuals, a number of the individuals that enter the restricted area through the corresponding entrance, or a number of the vehicles that enter the restricted area through the corresponding entrance.
11. The system of claim 1, wherein the one or more computers further perform the operations of: establishing an existence of a time-varying pattern among the event parameters of the events detected during the first time period, wherein establishing the existence of the time-varying pattern comprises: applying at least one of a machine learning algorithm or a data mining algorithm to data identifying the detected events and the values of the event parameters; and establishing the existence of the time-varying pattern based on an outcome of the at least one machine learning algorithm or data mining algorithm; based on the time-varying pattern, generating data identifying expected occurrences of one or more of the events during a second time period that includes the second detection time, the second time period occurring after the first time period determining that the additional event represents a deviation from the expected occurrences during the second time period; and in response to the determination, transmitting data identifying the deviation to a communications device, the communications device being configured to present, to the user within a corresponding interface, a notification that includes at least a portion of the transmitted data.
12. The system of claim 11, wherein the detected additional event corresponds to at least one of a detected presence of an unexpected object within a portion of the digital video content, a detected removal of an expected object within a portion of the digital video content, or a detection of smoke or flames within a portion of the digital video content.
13. The system of claim 1, wherein the operations further comprise: applying the predictive model to values of event parameters that characterize a portion of the detected events during the first time period; based on an outcome of applying the predictive model to the values of the event parameters, generating data that identifies expected occurrences of one or more of the events during a third time period, the third time period occurring after the first time period; detecting a second additional event within the captured video content, the second additional event being associated with a third detection time that occurs within the third time period; determining that the second additional event represents a deviation from the expected occurrences during the third time period; and in response to the determination, transmitting data that identifies the deviation to a communications device, the communications device being configured to present, to the user within a corresponding interface, a notification that includes a representation of the deviation.
14. The system of claim 13, wherein the generating comprises: based on the outcome of the predictive model, establishing an existence of a time-varying pattern among the values of the event parameters of the events detected during the first time period, including: applying at least one of a machine learning algorithm or a data mining algorithm to data identifying the values of the event parameters and the detected events; and establishing the existence of the time-varying pattern based on an outcome of the at least one machine learning algorithm or data mining algorithm; and generating the expected occurrences of one or more of the events during the third time period in accordance with the time-varying pattern.
15. The system of claim 13, wherein the detected second additional event corresponds to at least one of a detected presence of an unexpected object within a portion of the digital video content, a detected removal of an expected object within a portion of the digital video content, or a detection of smoke or flames within a portion of the digital video content.
16. A computer-implemented method, comprising: detecting, by at least one processor, events shown within digital video content captured by one or more video cameras, the detected events being associated with corresponding event parameters and detection times within a first time period, including: applying one or more video analytics to portions of the digital video content, the one or more applied video analytics comprising an object classification process and at least one of an image acquisition process, an object detection process, an object recognition process, an event detection process, or an object tracking process; and detecting, based on an outcome of applying the one or more video analytics to the portions of the digital video content, at least one of the events; obtaining, by the at least one processor, first data that identifies at least one external event, the obtained data comprising observed values of an external event parameter that characterize the external event during the first time period, wherein the first data is not derived from the digital video content, and wherein the at least one external event is not shown within the digital video content; establishing, by the at least one processor, a predictive model; detecting, by the at least one processor, an additional event shown within the digital video content, the additional event being associated with a corresponding additional event parameter and a second detection time; in response to the detection of the additional event, applying, by the at least one processor, the predictive model to a value of the additional event parameter; determining, by the at least one processor, an expected value of the external event parameter at the second detection time, based on an outcome of applying the predictive model to the value of the additional event parameter; and transmitting, by the at least one processor, data identifying the expected value of external event parameter to a communications device, the communications device configured to present a representation of the expected value of the external event parameter to a user through a corresponding interface.
17. The method of claim 16, wherein the predictive model correlates the values of the event parameters that characterize a portion of the detected events during the first time period.
18. The method of claim 16, further comprising receiving a portion of the video content from the one or more video cameras, wherein the one or more video cameras comprise at least one of a pan-zoom-tilt video camera or a camera having a fixed field-of-view.
19. A computer program product, encoded on one or more non-transitory computer storage media, comprising instructions that when executed by one or more computers cause the one or more computers to perform operations comprising: detecting events shown within digital video content captured by one or more video cameras, the detected events being associated with corresponding event parameters and detection times within a first time period, including: applying one or more video analytics to portions of the digital video content, the one or more applied video analytics comprising an object classification process and at least one of an image acquisition process, an object detection process, an object recognition process, an event detection process, or an object tracking process; and detecting, based on an outcome of applying the one or more video analytics to the portions of the digital video content, at least one of the events; obtaining first data that identifies at least one external event, the obtained data comprising observed values of an external event parameter that characterize the external event during the first time period, wherein the first data is not derived from the digital video content and wherein the at least one external event is not shown within the digital video content; establishing a predictive model; detecting an additional event shown within the digital video content, the additional event being associated with a corresponding additional event parameter and a second detection time; in response to the detection of the additional event, applying the predictive model to a value of the additional event parameter; determining an expected value of the external event parameter at the second detection time, based on an outcome of applying the predictive model to the value of the additional event parameter; and transmitting data identifying the expected value of external event parameter to a communications device, the communications device configured to present a representation of the expected value of the external event parameter to a user through a corresponding interface.
FIG. 1 illustrates an example system 100 that correlates events detected within digital video content to external events not shown within the digital video content. In some aspects, system 100, includes one or more video cameras, e.g., a video camera 102, a client device 110, and a video processing system 120, which may be interconnected through any appropriate combination of communications networks, e.g., a wireless local area network (LAN), e.g., a "WiFi" network, a RF network, a Near Field Communication (NFC) network, a wireless Metropolitan Area Network (MAN) connecting multiple wireless LANs, and a wide area network (WAN), e.g., the Internet.
For example, based on an application of the one or more processes for detecting, classifying, recognizing, and/or tracking objects to digital video content 130A, video analytics module 122 may detect objects, such as individuals, vehicles, or luggage, occurring within frames of digital video content 130A and further, monitor a trajectory, speed, and travel time of these detected objects as they move within successive frames of digital video content 130A. Further, and based on an application of the one or more object-counting processes to digital video content 130A, video analytics module 122 may determine or "count" numbers of these objects occurring within the frames of digital video content 130A, and additionally or alternatively, a number of these objects that cross virtual lines disposed within the frames of digital video content 130A. Additionally, video analytics module 122 may apply the facial- and textual-recognition algorithms to identify faces of individuals and textual content, such a license plate, within the digital video content 130A. In other implementations, video analytics module may apply any additional or alternate set of video analytics that would be appropriate to the stored digital video content and to video processing system 120.
In additional implementations, one or more of the events detected by video analytics module 122 may reflect or be indicative of various real-world conditions or events, e.g., external events. For example, the determined numbers of customers queued at entrance 103 at various times during the temporal window associated with a sporting event on May 19, 2016, may be related to or indicative of external events that include, but are not limited to, a total attendance at the May 19.sup.th sporting event, a total number of customers served by and/or a total concession revenue of by the stadium's concession stands during the May 19.sup.th sporting event, and a total parking revenue collected from customers during the May 19.sup.th sporting event.
In certain aspects, video camera 102 and the one or more additional video cameras may be incapable of capturing these external events, and as such, video processing system 120 may be incapable of characterizing the external events through the video-analytical processes described above. For example, the external event may represent a total attendance at a sporting event held at the stadium on May 19.sup.th, and video processing system 120 may be unable to estimate accurately the total attendance the sporting event based solely on the digital video content captured by video camera 102 and the additional video cameras.
Video processing system 120 may receive the data identifying the external events and the external event parameter values, and may store portions of the received data within structured data records of a corresponding data repository, e.g., external event data storage 123. For example, and as described above, video processing system 120 may receive data identifying the total attendance established for the one or more prior sporting events held at the stadium, including the May 19.sup.th sporting event, and may store portions of the received data within corresponding structured data records of external event data storage 123, along with data identifying the dates and/or times of each of the sporting events.
By way of example, additional event data 130E may specify that 445 customers are queued at entrance 103 at a corresponding detection time prior to the May 20.sup.th sporting event at the stadium. Based on an application of the predictive model to portions of additional even data 130E (e.g., the 445 queued customers at the corresponding detection time), event prediction module 128 may predict an expected total attendance of 44,895 for the May 20.sup.th sporting event, and event prediction module 128 may populate outcome data 130F with the expected total attendance.
Video processing system 120 may, in certain aspects, transmit portions of output data 130F to communications device 110 across any of the communications networks described above. For example, the transmitted portions of output data 130F may identify the expected value of the external event parameter, e.g., the total attendance of 44,895 for the May 20.sup.th sporting event, and communications device 110 may execute one or more applications programs that present a representation of the expected total attendance to a user 101 through a corresponding interface. For example, communications device 110 may present the representation of the expected total attendance to user 101 through a portion of a web page or other graphical user interface (GUI), and additionally or alternatively, through a voice-user interface (VUI) that may leverage a virtual digital assistance functionality of communications device 110.
Additionally, in some implementations, communications device 110 may additionally perform operations upon receipt of data identifying the expected value of the external event parameter. By way of example, the expected value of the external event parameter, e.g., the expected total attendance of 44,895 for the May 20.sup.th sporting event, may exceed an anticipated value and may thus be inconsistent with a prior allocation of resources. In certain aspects, communications device 110 may execute one or more application programs that, either independently or in conjunction with other external computing systems (e.g., associated with various vendors), adaptively and automatically adjust the allocation of resources to match the expected attendance. For instance, the executed application programs may perform operations that re-allocate human resources, such as stadium attendant, security, and/or parking staff, to various locations within the stadium to reflect the expected total attendance, and further, may re-allocate a distribution of supplies to various concession stands and retail outlets within the stadium to reflect the expected total attendance.
Previous Patent US 10,402,658 | Next Patent US 10,402,660