EXHIBIT 2

Case 2:23-cv-00617-RWS-RSP Document 1-3 Filed 12/19/23 Page 1 of 16 PageID #: 40
US010701321B2 

( 12 ) United States Patent 

Westmacott et al . ( 10 ) Patent No .: US 10,701,321 B2 

( 45 ) Date of Patent : Jun . 30 , 2020 

( 54 ) SYSTEM AND METHOD FOR DISTRIBUTED 

VIDEO ANALYSIS 

10,108,157 B1 * 10/2018 Raghavan 2010/0002070 A1 * 1/2010 Ahiska 

2011/0080495 Al * 4/2011 Busse ( 71 ) Applicant : Sensormatic Electronics , LLC , Boca 

Raton , FL ( US ) 2012/0140067 A1 * 6/2012 Crossen 

G06K 9/00771 

GO8B 13/19691 

348/36 

G06K 93233 

348 / 222.1 

HO4N 7/0125 

348/143 

G06K 9/00221 

382/159 

HO4L 63/08 

726/5 

HO4N 7/181 

348/159 

( 72 ) Inventors : Ian Christopher Westmacott , 2014/0044348 Al * 2/2014 Chen 

Tewksbury , MA ( US ) ; Steve Carney , Foxborough , MA ( US ) 2014/0282934 A1 * 9/2014 Miasnik 

2014/0333776 A1 * 11/2014 Dedeoglu ( 73 ) Assignee : Sensormatic Electronics , LLC , Boca 

Raton , FL ( US ) 

* cited by examiner ( * ) Notice : Subject to any disclaimer , the term of this 

patent is extended or adjusted under 35 

U.S.C. 154 ( b ) by 106 days . Primary Examiner — Richard T Torrente 

( 74 ) Attorney , Agent , or Firm - HoustonHogle LLP 

( 21 ) Appl . No .: 14 / 634,428 

( 22 ) Filed : Feb. 27 , 2015 

( 65 ) Prior Publication Data 

US 2016/0253883 A1 Sep. 1 , 2016 

( 51 ) Int . CI . H04N 7/18 ( 2006.01 ) 

( 52 ) U.S. CI . CPC H04N 7/181 ( 2013.01 ) 

( 58 ) Field of Classification Search 

CPC HO4N 7/181 

See application file for complete search history . 

( 57 ) ABSTRACT 

A system and method for distributed analysis of image data 

in a video surveillance system can be deployed in house holds , businesses , and within corporate entities , in examples . The image data are processed on local video analytics 

systems located within the networks of the businesses or on 

remote video analytics systems hosted by a cloud service . To 

limit the data traffic imposed by the image data on the 

network , the system divides the image data processing into 

separate object detection and analysis functions . The system 

can also integrate the object detection function within the 

surveillance cameras or on a local gateway . This can sig nificantly reduce the data traffic sent over networks as 

compared to current video surveillance systems and methods 

since only image data containing object of interest needs to 

be sent . 

( 56 ) References Cited 

U.S. PATENT DOCUMENTS 

7,257,273 B2 

8,824,750 B2 

8/2007 Li et al . 9/2014 Jankowski et al . 23 Claims , 7 Drawing Sheets 

Biz Corp. 

Gateway System 

110-2 100 Cam Corp. 

Gateway System 110-3 -144 

remote database 142 

network cloud 60 Remote video analytics 

system 122 video control system 114 1 

object analysis system 123-2 

ACME Corp. Local Cloud service 180 Video Analytics System 112 Gateway System 110-1 104-1 

-118 object analysis 

system 123-1 116 116 

vehicle 182 104-2 117 local database 132 118 Network Video Recorder 130 

frame capture -118 System 106-1 

object detection system 108-1 frame capture 134 system 106-2 103-1 

object detection 

system 108-2 room 170 

103-2 building 102 

Case 2:23-cv-00617-RWS-RSP Document 1-3 Filed 12/19/23 Page 2 of 16 PageID #: 41
Biz Corp. 

Gateway System 110-2 

100 

Cam Corp. Gateway System 110-3 

www.no 

wewnia 

movement 

-144 

. www 

U.S. Patent 

www www 

remote database 142

. www 

www 

w 

network cloud60 

Remote videoanalytics 122 system 

. www 

video control 114 system 

analysis object 123-2 system 

Jun . 30 , 2020 

1 

. www 

ACME Corp. 

1 

Cloud service 180

Fouwen 

Local Video Analytics 112 System 

104-1 

Gateway System110-1 

118 

analysis object 123-1 system 

116 

-116 

Sheet 1 of 7 

vehicle 182

1-104-2 

local database 132

-118 

Network Video Recorder 130 

capture frame 106-1 system 

118 

detection object 108-1 system 

frame capture 106-2 system 

134 

103-1 

detection object 108-2 system 

room 170

US 10,701,321 B2 

103-2 

102 building 

FIG .1

Case 2:23-cv-00617-RWS-RSP Document 1-3 Filed 12/19/23 Page 3 of 16 PageID #: 42
Surveillance camera103-1 

110-1 gateway system 

local videoanalytics system 112

local database 132 

remote database 

frame capture 106-1 system 

1421 

remote video analytics system 122 

detection object 108-1 system 

U.S. Patent 

Cloud service 180

L. 

W 

202. Capture of data aimage monitored area outside abuilding for aspecified time interval andencode asframes ofvideo data 

206. sendthe imagedata tothe gateway system forfurther processing 

204. detectinterest objects ofwithin thevideo and frames ,extract image data fromthe video frames which include thedetected objects interest of

208. determine which oftheimage locally which analyze and todata analyze remotely 

210. sendimage datatolocal analytics system for analysis 

Jun . 30 , 2020 

212. access reference images, image processing functions ,patterns neuraletwork functions deepclassification objectfunctions ,andlearning 

213. analyze theimage databycomparing tohereference images, byrecognizing patterns inthemage data fromthe patterns ,andbyppling theimage 

functions neuraletwork processing functions ,the

classification object learning functions ,andthe deep

functions toheimage data

Sheet 2 of 7 

214. savemetadata fromanalysis oftheimage data

216. sendimage datato remote analytics analysis forsystem 

218. access reference images, image processing functions ,patterns neuraletwork functions deepclassification functions ,andobject learning 220. analyze theimage databycomparing tohereference images, byapplying recognizing patterns inthemage datafrom patterns ,andbythe neuraletwork functions ,deeplearning andtheobject 

classification functions toheimage data

D 

222. sendmetadata fromanalysis oftheimage 

data 

US 10,701,321 B2 

224. savemetadata fromimage analysis data ofthe

FIG .2

Case 2:23-cv-00617-RWS-RSP Document 1-3 Filed 12/19/23 Page 4 of 16 PageID #: 43
100 

144 

U.S. Patent 

network cloud 60

remote database 142

1 

Remote videoanalytics system 122

object analysis 123-2 system 

... 

Jun . 30 , 2020 

Small OfficeNetwork 

Cloud service 180

1 

134 

104-14 

-118 

103-1 

116 

-116 

System Gaway 110-1 

vehicle 182

Sheet 3 of 7 

-104-2 

capture frame 106-1 system 

detection object 108-1 system 

ATM 124 

-118 

110-2 System Gaway 

capture frame 106-2 system 

detection object 108-2 system 

US 10,701,321 B2 

102 building 

103-2 

FIG .3

Case 2:23-cv-00617-RWS-RSP Document 1-3 Filed 12/19/23 Page 5 of 16 PageID #: 44
Surveillance camera 103-1 

local database 132 

remote || database 

frame capture 106-1 system 

detection object 108-1 system 

gateway system110-1 

remote video analytics system 122 

1421 

U.S. Patent 

Cloud service 180www 

ww 

602. Capture of aimage data monitored area outside abuilding for aspecified time interval andencode as frames ofvideo data 

606. sendthe imagedata tothe gateway systemfor further processing 

604. detect interest ofobjects video wthin theframes ,andextract image data fromthe video frameswhich include thedetected objects of interest 

608. sendimage datato remote analytics analysis systemfor 

Jun . 30 , 2020 

610. access reference images ,imageneural functions ,patterns processing network functions ,deeplearning functions classification object and

612. analyze theimage databycomparing tohereference from patterns theimage datarecognizing in,byimages functions ,applying theimage processing patterns ,andbyneuraletwork functions ,thedeep learning andthe

functions toheimage dataobject classification 

Sheet 4 of 7 

614. sendmetadata fromanalysis oftheimage data

metadata from616. saveanalysis image dataofthe 

US 10,701,321 B2 

FIG .4

Case 2:23-cv-00617-RWS-RSP Document 1-3 Filed 12/19/23 Page 6 of 16 PageID #: 45
Biz Corp 

Camorp 

Gateway System 110-2 

detection object 108-2 system 

Gateway System 110-3 

100 

detection object 108-3 system 

U.S. Patent 

-144 

remote database 142

60 network cloud

video control 114 system 

Remote video analytics 122 system 

analysis object123-2 system 

Jun . 30 , 2020 

. 

ACME Corp 

www 

Cloud service 180

104-1 

System Gateway 110-1 

detection object 108-1 system 

-118 

116 

-116 

Sheet 5 of 7 

104-2 

134 

vehicle 182

local database 132

Network Video Recorder 130 

capture frame 106-1 system 

ATM 124 

-118 

frame capture system 106-2

103-1 

170 room 

US 10,701,321 B2 

103-2 

102 building 

FIG .5

Case 2:23-cv-00617-RWS-RSP Document 1-3 Filed 12/19/23 Page 7 of 16 PageID #: 46
Surveillance camera 103-2 

system 110gateway 

local database 132 

remote database 142 

frame capture 106-2 system 

remote video analytics system 122 

detection object 108-1 system 

U.S. Patent 

Cloud service 1801 

U 

304. sendall captured frames interval timespecified for

302. Capture of adata image monitored area within abuilding for aspecified time interval andencode asframes ofvideo data 

306. detectobjects ofinterest within thevideo frames, andvideo fromthe extract image dataframes whichinclude thedetected objects ofinterest 

308. sendthe imagedata toremte videoanalytics systemfor 

further processing 

Jun . 30 , 2020 

310. access reference images ,image neural functions patterns processing ,network functions ,deeplearning functions classification objectand 

Sheet 6 of 7 

312. analyze theimage dataþycomparing tohereference images, bythe imagedata frompatterns ,andbyrecognizing patterns innetwork functions processing ,neural applying theimage functions classification tofunctions ,andthe objectearning thedeep 

image datathe 

314. sendmetadata fromanalysis oftheimage 

data 

metadata from save 316,

analysis oftheimage data 

US 10,701,321 B2 

FIG .6

Case 2:23-cv-00617-RWS-RSP Document 1-3 Filed 12/19/23 Page 8 of 16 PageID #: 47
110-1 system gateway 

Network video recorder 130

Video contro 114 system 

local database 132 

remote video analytics ilsystem 122 

remote database 

142 1

detection object 108-1 system 

U.S. Patent 

Cloud service 1801

W 

404. sendall recorded videointerval specified time data for

402. Userselects asetof recorded video datafor monitored aarea

outside abuilding forinterval timespecified 

406.detect objects ofinterest within thevideo frames, andextract fromthe image datavideo frameswhich include thedetected objects ofinterest 

Jun . 30 , 2020 

408. sendthe imagedata toremte videoanalytics systemforurther processing 

410. access reference images, image network functions patterns neural processing ,functions ,deeplearning andobject 

functions classification 

412. analyze theimage databycomparing tohereference images ,bydata frompatterns ,andbyppling recognizing patterns inthemage network deepprocessing functions ,neural theimage learning functions ,andthe objectclassification functions tomagedta 

Sheet 7 of 7 

414. sendmetadata fromanalysis oftheimage data

416 ,sendmetadata fromdata oftheimage analysis 

US 10,701,321 B2 

FIG .7

Case 2:23-cv-00617-RWS-RSP Document 1-3 Filed 12/19/23 Page 9 of 16 PageID #: 48
US 10,701,321 B2 

1 2 

SYSTEM AND METHOD FOR DISTRIBUTED this requires that the video analytics systems be high 

VIDEO ANALYSIS performance or specialized computer systems . This typically increases the cost of the video analytics systems , which 

BACKGROUND OF THE INVENTION correspondingly increases the cost of the overall video 

5 surveillance systems . Moreover , the analysis methods are 

Video surveillance systems are increasingly deployed constantly being improved and so the systems must be 

within public and private institutions , such as households , periodically updated . commercial buildings , businesses , retail establishments , The present invention concerns a model for the processing schools , hospitals and government buildings , to list a few of the image data by separating the analysis into separate examples . These surveillance systems are often used for a 10 object detection and object analysis functions or systems . number of different tasks , such as intelligence gathering by The object detection system detects objects of interest and 

governments and businesses , prevention or investigation of classes of objects of interest within the image data . The 

crime by law enforcement or retail establishments , and object analysis system then analyzes the image data includ 

monitoring the usage of resources or structures such as ing the detected objects of interest , in one example , to buildings , bridges or roads , to list a few examples . 15 recognize ( e.g. identify ) the detected objects of interest . This 

A number of components are common to many of these division can be used to lower the volume of image data 

video surveillance systems . Surveillance cameras capture transmitted over the intervening networks by locating the 

image data that are typically then sent over one or more data object detection closer to the cameras and then sending only networks , such as governmental , service provider , enterprise the “ interesting ” portions of the image data to the more 

or home networks . This image data are typically stored on 20 remote object analysis system . network video recorders ( NVRs ) . Video analytics systems The object analysis system can analyze the image data 

are often included to analyze the image data . Finally , control using many different analysis methods . For this purpose , the 

systems are often used to control the surveillance systems object analysis system typically first accesses information 

and coordinate the operation of the components . from a database such as reference images , image processing These video analytics systems have the capability of 25 functions , patterns , neural network functions , deep learning automatically analyzing image data to detect and determine functions , and object classification functions . temporal and spatial events . Minimally , these systems track Then , the object analysis system can analyze the image objects against fixed background scenes . More sophisticated data by comparing the image data to the reference images to 

functions include object detection to determine the presence identify the objects of interest within the images , by recog of an object or a type of object . Even higher level functions 30 nizing patterns in the image data from the patterns , and / or by include object analysis , such as object identification . applying the image processing functions , the neural network 

The methods employed by video analytics systems for functions , the deep learning functions , and / or the object detecting and analyzing classes of objects in image data classification functions to the image data , in examples . The 

have become more accurate in recent years , and are , as a image processing functions can include filtering , image / 

consequence , becoming more and more important in video 35 contrast enhancement , compression , and deblurring of the surveillance systems . Two common examples of object image data , in examples . analysis within image data are facial recognition of indi In one example , the object detection system is integrated viduals and license plate recognition , or number plate rec within the surveillance cameras or a gateway system , and the 

ognition . Some of these video analytics systems are sophis object analysis system is then located in a more remote video ticated enough to identify a particular person from their 40 analytics system . facial characteristics , and can identify a particular license As a result , only image data that includes the detected 

plate by reading and recognizing the text and numbers objects of interest needs to be sent over the data network for 

within the license plate , in examples . object analysis by the local or remote video analytics systems . This saves on bandwidth as compared to current 

SUMMARY OF THE INVENTION 45 video surveillance systems , which typically send all image data including empty scenes and image data not including Current video surveillance systems have problems . Video objects of interest over the data network . surveillance systems can increase the volume of data traffic Finally , the object detection system can further limit the 

sent over the networks . This increased data traffic volume image data sent over the data network to achieve even 

can limit the available bandwidth of the networks , which can 50 greater bandwidth savings . In one example , the object 

cause performance problems for components of the video detection system can create cropped images from the image surveillance systems and other nodes that communicate over data that include only the detected objects of interest , and 

the networks . send only the cropped images over the data network for 

One reason for the increased data traffic created by current processing by the video analytics system . video surveillance systems is that the surveillance cameras 55 In general , according to one aspect , the invention features 

typically send their image data over the networks for further a video surveillance system that comprises one or more processing . For example , the image data are typically sent surveillance cameras that capture image data , an object by surveillance cameras over the network to video analytics detection system that detects objects of interest within the 

systems located within the network . Another reason for the image data , and a video analytics system that receives the 

increased data traffic over the networks is that recent 60 image data , transmitted over a network , from the object advances in video technology have resulted in an increase detection system The image data received by the video the resolution of the image data that the surveillance cam analytics system correspond to objects of interest detected 

eras capture and transmit . by the object detection system , and the video analytics Another issue with current video surveillance systems is system then analyzes the objects of interest . cost . The video analysis methods employed by video ana- 65 In this way , the object detection is separated from object lytics systems for analyzing the image data are typically analysis . Thus , only a portion of the image data generated by 

memory intensive and processor intensive tasks . As a result , the surveillance cameras needs to be sent over the network . 

Case 2:23-cv-00617-RWS-RSP Document 1-3 Filed 12/19/23 Page 10 of 16 PageID #: 49
US 10,701,321 B2 

3 4 

In some embodiments , the object detection system creates FIG . 5 is a schematic block diagram of a third embodi 

cropped images from the image data that include the ment of n distributed video surveillance system , and 

detected objects of interest and sends the cropped images to FIGS . 6 and 7 are diagrams that show exemplary mes 

the video analytics system to further reduce bandwidth . In saging between the major components of the system of FIG . other cases , entire frames of image data are provided , also 5 5 , where FIG . 6 shows messaging associated with process metadata may be included to identify the portion of the ing of live image data , and where FIG . 7 shows messaging frames containing the detected objects . associated with forensic processing of image data recorded 

In some cases , objects of interest detected by the object by a network video recorder . detection system are ranked by the object detection system according to factors including close proximity of the objects 10 DETAILED DESCRIPTION OF THE 

of interest to the one or more surveillance cameras , picture PREFERRED EMBODIMENTS 

quality of the objects of interest , and / or contrast of the objects of interest . The invention now will be described more fully herein The video analytics system may include an object analysis after with reference to the accompanying drawings , in which 

system that compares the image data corresponding to 15 illustrative embodiments of the invention are shown . This 

objects of interest detected by the object detection system to invention may , however , be embodied in many different 

reference images to identify the objects of interest . At the forms and should not be construed as limited to the embodi 

same time , the object detection system may be integrated ments set forth herein ; rather , these embodiments are pro within the one or more surveillance cameras . vided so that this disclosure will be thorough and complete , The video analytics system may be a node on an enter- 20 and will fully convey the scope of the invention to those 

prise network . On the other hand , it could alternatively or skilled in the art . 

additionally be a cloud analytics system . As used herein , the term “ and / or ” includes any and all 

In some embodi its , a gateway system is included that combinations of one or more of the associated listed items . provides access to the video analytics system over a public Further , the singular forms and the articles “ a ” , “ an ” and 

network . The object detection system might be integrated 25 “ the ” are intended to include the plural forms as well , unless 

within the gateway system . The gateway system can be a expressly stated otherwise . It will be further understood that 

node on the local network or integrated within the cameras . the terms : includes , comprises , including and / or comprising , In general , according to one aspect , the invention features when used in this specification , specify the presence of 

a method for analyzing video in a video surveillance system . stated features , integers , steps , operations , elements , and / or This method comprises capturing image data from one or 30 components , but do not preclude the presence or addition of 

more surveillance cameras , detecting objects of interest one or more other features , integers , steps , operations , within the image data via an object detection system . The elements , components , and / or groups thereof . Further , it will 

object detection system transmits the image data over a be understood that when an element , including component network , the image data transmitted by the object detection or subsystem , is referred to and / or shown as being connected 

system corresponding to objects of interest detected by the 35 or coupled to another element , it can be directly connected 

object detection system . Finally , a video analytics system or coupled to the other element or intervening elements may receives the image data from the object detection system and be present . analyzes the objects of interest . FIG . 1 shows an embodiment of a distributed video 

The above and other features of the invention including surveillance system 100. The system 100 is deployed within various novel details of construction and combinations of 40 large businesses , large corporate entities , or government parts , and other advantages , will now be more particularly entities , in examples . Multiple exemplary businesses Biz 

described with reference to the accompanying drawings and Corp. ( Biz ) , Cam Corp. ( Cam ) , and ACME Corp ( “ ACME ” ) pointed out in the claims . It will be understood that the are shown . Detail is included for network 134 of only one 

particular method and device embodying the invention are ( ACME ) , for clarity , however . shown by way of illustration and not as a limitation of the 45 The illustrated system 100 includes a local video analytics invention . The principles and features of this invention may system 112 that is a node on the business's enterprise be employed in various and numerous embodiments without network 134. The local video analytics system 112 enables 

departing from the scope of the invention . processing of the image data from the surveillance cameras 

103 within the enterprise network 134. In addition , the 

BRIEF DESCRIPTION OF THE DRAWINGS 50 businesses such as ACME can also use remote video ana 

lytics systems 122 hosted by cloud services 180 to analyze In the accompanying drawings , reference characters refer the image data from the surveillance cameras 103 . to the same parts throughout the different views . The draw In some implementations , a third - party company operates ings are not necessarily to scale ; emphasis has instead been cloud service 180. The cloud service 180 can also be a public placed upon illustrating the principles of the invention . Of 55 or private cloud service , in different examples . Public clouds the drawings : differ from private clouds with respect to security and 

FIG . 1 is a schematic block diagram of an embodiment of sharing of the resources that they provide to their clients . a distributed video surveillance system according to the Public clouds generally share their datacenter resources 

invention ; among multiple clients . In contrast , private clouds typically FIG . 2 is a diagram that shows exemplary messaging 60 dedicate specific resources to individual clients . between the major components of the system of FIG . 1 for The cloud service 180 is typically operated by a different processing of image data ; business entity than the business entities that operate busi 

FIG . 3 is a schematic block diagram of a second embodi nesses Cam , Biz , and ACME . The businesses are clients of 

ment of the distributed video surveillance system ; the cloud service 180. This allows the businesses Cam , Biz , FIG . 4 is a diagram that shows exemplary messaging 65 and ACME to leverage the expertise of the cloud service and 

between the major components of the system of FIG . 3 for its ability to provide constant performance updates and 

processing of image data ; improvements . 

Case 2:23-cv-00617-RWS-RSP Document 1-3 Filed 12/19/23 Page 11 of 16 PageID #: 50
US 10,701,321 B2 

5 6 

The cloud service 180 includes components such as a detected objects of interest . In examples , objects of interest 

remote database 142 and a remote video analytics system include vehicles 182 and individuals 118 , in examples . Of 

122. The remote video analytics system 122 includes an particular interest are the faces of the individuals 118 and 

object analysis system 123-2 . The remote database 142 and license plates 116 of the vehicles 182. In other examples , the 

the remote video analytics system 122 are nodes of cloud 5 objects of interest include animals , retail products or product service network 144 . packaging , smoke , flames , explosives , visible traces of 

The business clients ACME , Biz , and Cam access the vapors , and atmospheric conditions such as rain , mist , and 

components of the cloud service 180 via their respective fog . gateway systems 110. Each of the gateway systems 110 has In one implementation , the object detection system 108-1 

a physical or logical interface that connects to an associated 10 can create cropped images extracted from the frames . The enterprise network , such as ACME's enterprise network cropped images include the detected objects of interest . 134 , and an interface that connects to a network cloud 60 . In another example , the object detection system 108-1 

The network cloud 60 is preferably a public / service provider may optionally choose to send only some frames that 

network such as the Internet , but can also be a private include an indicative pattern or object of interest . Because a 

network built for each specific enterprise client , in examples . 15 face or license plate , or other object class of a detected object Components of the system 100 that are deployed within of interest may be present in the image for a length of time , ACME are preferably installed as nodes on ACME's enter typically there will be many frames within the image data 

prise network 134. The components include surveillance that include an object of interest . cameras 103-1 and 103-2 , gateway system 110-1 , video The object detection system 108-1 could be also opti control system 114 , local video analytics system 112 , net- 20 mized to send a limited number of frames of image data , or 

work video recorder 130 , and local database 132. The local even just one frame of image data . This is especially true if video analytics system 112 includes an object analysis the object analysis system 123-1 can sufficiently identify 

system 123-1 . Operators of the video surveillance system objects from such a limited number of frames . 100 such as on - site security and surveillance personnel can Further , in some implementations , the object detection 

use the video control system 114 to control the system's 25 system generates and sends metadata along with the image components . data that includes its identification of a type of the object of 

The surveillance cameras 103 capture images and / or interest and / or the location and pixel area of the detected 

video of monitored areas , such as a room 170 within a object ( S ) . building 102 or premises adjacent to a building 102. The Finally , in another example , the object detection system images and / or video , also known as image data , can be 30 108-1 can rank the frames or portions of the frames image formatted or presented in different ways . In examples , the data corresponding to objects of interest detected by the 

image data can be encoded into different formats such as object detection system 108-1 according to different factors . MPEG4 and H.264 , JPEG or left as “ raw ” composite image These factors include close proximity of the objects of 

data , in examples . interest to the one or more surveillance cameras 103 , picture Surveillance camera 103-1 is positioned to capture image 35 quality of the objects of interest , and / or image contrast of the 

data of an area adjacent to building 102. The building 102 portion of the image data that contains the objects of interest . has entry / access points 117 such as doors from which The rankings are typically saved in the metadata within or 

employee individuals 118 may enter or leave building 102 . with the image data . Then , the object analysis system 123-1 

Individuals 118 in vehicles 182 such as service personnel can read the rankings within the metadata to decide whether 

and employees drive by the building 102 and park the 40 or not to analyze the image data . vehicles 182 in parking lots , in examples . The vehicles 182 According to step 206 , the object detection system 108-1 include license plates 116. The entry / access points 117 and then sends the image data , such as cropped or entire images , the vehicles 182 are included within the field of view 104-1 to the gateway system 110-1 for further processing . of surveillance camera 103-1 . Surveillance camera 103-1 In step 208 , the gateway system 110-1 can determine also integrates a frame capture system 106-1 and an object 45 which of the image data to analyze on the local video 

detection system 108-1 . analytics system 112 and which of the image data to analyze Surveillance camera 103-2 is positioned within building on the remote video analytics system 122 , if it is present and 

102. The building 102 includes a room 170 with individuals available . 118. The room 170 and the individuals 118 are included In step 210 , the gateway system 110-1 can send at least 

within the field of view 104-2 of surveillance camera 103-2 . 50 some of the image data over the enterprise network 134 to 

Surveillance camera 103-2 also integrates frame capture the local video analytics system 122 . system 106-2 and object detection system 108-2 . In step 212 , the local video analytics system 112 might FIG . 2 shows exemplary messaging between major com access reference images , image processing functions , pat ponents of the distributed video surveillance system of FIG . terns , neural network functions , deep learning functions , 1 associated with “ live ” processing of image data for large 55 and / or object classification functions from the local database business ACME . The major components include the surveil 132. Then , in step 212 , the local video analytics system 122 

lance camera 103-1 , the gateway system 110-1 , the local analyzes the image data via its object analysis system 123-1 . video analytics system 112 , the local database 132 , and the The local video analytics system 112 analyzes the image cloud service 180 . data by comparing the image data to the reference images , In step 202 , the frame capture system 106-1 of the 60 by recognizing patterns in the image data from the patterns , surveillance camera 103-1 captures image data of the moni and / or by applying the image processing functions , the 

tored area outside the building 102 for a specified time neural network functions , the deep learning functions , and / interval . The frame capture system 106-1 then encodes the or the object classification functions to the image data , in 

image data into frames of video data . examples . In step 204 , the object detection system 108-1 detects 65 In one specific example , the local video analytics system objects of interest within the frames , and possibly extracts 112 analyzes the image data by comparing the image data to 

video frames or portions of the framess that include the the reference images to identify the objects of interest within 

Case 2:23-cv-00617-RWS-RSP Document 1-3 Filed 12/19/23 Page 12 of 16 PageID #: 51
10 

15 

US 10,701,321 B2 

7 8 

the images . Reference images can include images of current The SOHO typically might not possess the processing and former employee individuals 118 of ACME , and images capability or could support the cost associated with includ 

of their license plates and / or vehicles , in examples . ing a local video analytics systems 112 within the network 

In step 214 , the local video analytics system 122 saves 134. At the same time , the small office or household typi metadata from analysis of the image data for the identified 5 cally requires only periodic or occasional access to image objects of interest with the image data to the local database data analysis services provided by video analytics systems . 132 . As a result , SOHO entities generally use a remote video 

In another specific example , the object detection system analytics system 122 hosted by a cloud service 180 to 

108-1 determines if the image data includes an object of analyze the image data sent from the surveillance cameras 

interest , such as a face of an individual 118 , and the local or 103 . remote video analytics system 112 , 122 analyze the image A third - party company typically operates the cloud ser 

data including the face of the individual 118 to determine the vice 180 , which would often be a public cloud in this 

identity of the individual 118 . example The SOHO is a client of the cloud service 180 . According to step 216 , the gateway system 110-1 can send The cloud service 180 here also includes components 

at least some of the image data over the network cloud 60 to such as the remote database 142 and the remote video 

the remote video analytics system 122 for analysis of the analytics system 122. The remote video analytics system image data . 122 includes an object analysis system 123-2 . The remote 

In response to this event , in step 218 , the remote video database 142 and the remote video analytics system 122 are 

analytics system 122 accesses reference images , image 20 nodes of remote network 144 . processing functions , patterns , neural network functions , Components of the system 100 that are deployed within 

deep learning functions , and / or object classification func the SOHO are preferably installed as nodes on its network 

tions from the remote database 142. Then , in step 220 , the 134. The components include the surveillance cameras 

remote video analytics system 122 analyzes the image data 103-1 and 103-2 and the gateway system 110-1 . Surveil via its object analysis system 123-2 . 25 lance camera 103-1 includes the gateway system 110-1 , Here also , in examples , the remote video analytics system frame capture system 106-1 , and object detection system 122 analyzes the image data by comparing the image data to 108-1 . Surveillance camera 103-2 includes gateway system the reference images , by recognizing patterns in the image 110-2 , frame capture system 106-2 , and object detection 

data from the patterns , and / or by applying the image pro system 108-2 . cessing functions , the neural network functions , the deep 30 Surveillance camera 103-1 is positioned to capture image learning functions , and / or the object classification functions data of an area adjacent to building 102. The building 102 

to the image data . has tenants such as a bank and includes an Automated Teller 

In one specific example , the remote video analytics sys Machine ( ATM ) 124 in the illustrated example . Individuals 

tem 122 analyzes the image data by comparing the image 118 in vehicles 182 access the ATM 124 to perform financial 

data to the reference images to identify the objects of interest 35 transactions . The vehicles 182 include license plates 116 . within the images . The ATM 124 and the vehicles 182 are included within the 

Cloud services often provide access to multiple public and field of view 104-1 of surveillance camera 103-1 . subscription - based remote databases 142. These can include Surveillance camera 103-2 is positioned within building 

government and law enforcement databases that include 102. The building 102 includes a room 170 with individuals 

reference images of known criminals , and images associated 40 118. The room 170 and the individuals 118 are included 

with border patrol and surveillance of airports , in examples . within the field of view 104-2 of surveillance camera 103-2 . In one example , the object detection system 108-1 deter Surveillance camera 103-2 also integrates frame capture mines if the image data includes an object of interest , such system 106-2 and object detection system 108-2 . as a product on a shelf of a large supermarket , and the SOHO and small retail establishments typically have 

remote video analytics system 122 analyzes the image data 45 limited available bandwidth resources in their networks 134 . including the product to determine the type of product , and As in the embodiment of FIG . 1 , the surveillance cameras 

to determine information about the product within its prod 103 integrate the object detection systems 108 to minimize 

uct label . Reference images taken of the product shelf at the amount image data that is sent over the network 134 to 

different times in the past can provide information for the cloud service . Moreover , the gateway system 110 is restocking and inventory management , and for marketing 50 included within the surveillance cameras 103 to save on 

and sales purposes to determine how long products remain cost . Although in other examples , it can be a standalone 

on shelves , in examples . system that is a node on the local network 134 that aggregate In step 222 , the remote video analytics system 122 sends image data from multiple cameras . metadata for the identified objects of interest over the FIG . 4 shows exemplary messaging between major com 

network cloud 60 to the gateway system 110-1 . The gateway 55 ponents of the distributed video surveillance system of FIG . system , in step 224 , then saves the metadata to the local 3 associated with “ live ” processing of image data for a small 

database 132. In this way , information generated by the business , SOHO , or small retail establishment . The major cloud system is made available back to the enterprise . components include surveillance camera 103-1 , gateway FIG . 3 shows a second embodiment of a distributed video system 110-1 , local database 132 , and cloud service 180 . surveillance system 100. The system 100 here might be 60 In step 602 , the frame capture system 106-1 of the 

deployed within a household , small office / home office surveillance camera 103-1 captures image data of a moni 

( SOHO ) , small retail establishment , or bank branch , in tored area outside a building 102 for a specified time 

examples . The network 134 is additionally labeled as “ Smal interval . The frame capture system 106-1 then encodes the 

10fficeNetwork ” to emphasize the nature of the deployment image . associated with the video surveillance system 100. In 65 In step 604 , the object detection system 108-1 detects 

examples , the network 134 can be a wireless network , or objects of interest within the video frames , and extracts 

wired local area network ( LAN ) . image data from the video frames that include the detected 

Case 2:23-cv-00617-RWS-RSP Document 1-3 Filed 12/19/23 Page 13 of 16 PageID #: 52
US 10,701,321 B2 

9 10 

objects of interest . In examples , objects of interest include FIG . 6 shows exemplary messaging between major com 

vehicles 182 and individuals 118 . ponents of the distributed video surveillance system of FIG . In one example , the object detection system 108-1 can 5 associated with “ live ” processing of image data for a 

create cropped frames extracted from the image data . The medium - sized business , ACME . The major components 

cropped frames include the detected objects of interest . 5 include surveillance camera 103-2 , gateway system 110-1 , According to step 606 , the object detection system 108-1 local database 132 , and cloud service 180 . then sends the image data , such as cropped frames or In step 302 , the frame capture system 106-1 of the 

complete frames or image data from a series of frames , to the surveillance camera 103-1 captures image data of a moni 

gateway system 110-1 within the surveillance camera 103-1 . tored area within a building 102 for a specified time interval . 

Then , in step 608 , the gateway system 110-1 sends the image 10 The frame capture system 106-1 then encodes the image data into frames of video data . data including the detected objects of interest over the In step 304 , the surveillance camera 103-2 sends all network cloud 60. The image data are sent over the network captured frames for the specified interval to the gateway cloud 60 to the remote video analytics system 122 of the system 110-1 . Then , according to step 306 , the object cloud service 180 for processing . 15 detection system 108 of the gateway system 110 detects In step 610 , the remote video analytics system 122 objects of interest within the video frames , and extracts 

accesses reference images , image processing functions , pat image data from the video frames that include the detected 

terns , neural network functions , deep learning functions , and objects of interest . object classification functions from the remote database 142 . According to step 308 , the gateway system 110-1 then 

Then , in step 612 , the remote video analytics system 122 20 sends the image data including the detected objects of 

analyzes the image data via its object analysis system 123-1 . interest over the network cloud 60. The image data are sent 

The analysis includes comparing the image data to the over the network cloud 60 to the remote video analytics reference images , by recognizing patterns in the image data system 122 of the cloud service 180 for processing . from the patterns , and / or applying the image processing In step 310 , the remote video analytics system 122 

functions , the neural network functions , the deep learning 25 accesses reference images , image processing functions , pat functions , and / or the object classification functions to the terns , neural network functions , deep learning functions , image data , in examples . and / or object classification functions from the remote data 

In one example , the remote video analytics system 122 base 142 . analyzes the image data by comparing the image data to Then , in step 312 , the remote video analytics system 122 

reference images to identify the objects of interest within the 30 analyzes the image data via its object analysis system 123-2 . images . The remote system may store metadata and image The analysis includes comparing to the reference images , data for the client in its remote database , or in step 614 , the recognizing patterns in the image data from the patterns , remote video analytics system 122 may send the metadata and / or applying the image processing functions , the neural 

for the identified objects of interest in the image data to the network functions , the deep learning functions , and / or the 

local gateway system 110-1 . Finally , in step 616 , the gate- 35 object classification functions to the image data , in 

way system 110-1 saves the metadata for the identified examples . objects of interest in the image data to a local database . In one example , the remote video analytics system 122 

FIG . 5 shows a third embodiment of a distributed video analyzes the image data by comparing the image data to 

surveillance system 100. The system 100 might be deployed reference images to identify the objects of interest within the 

within medium - size businesses and corporate entities , in 40 images . In step 314 , the remote video analytics system 122 

examples . Multiple exemplary businesses Biz Corp. ( Biz ) , sends metadata for the identified objects of interest in the 

Cam Corp. ( Cam ) , and ACME Corp ( “ ACME " ) are shown . image data to the gateway system 110-1 . Finally , in step 316 , Detail is only included for network 134 of ACME , to avoid the gateway system 110-1 saves the metadata for the iden cluttering the drawing . tified objects of interest in the image data to the local 

In this example , the businesses can require 24 - hour access 45 database 132 . to image data analysis services provided by video analytics FIG . 7 shows exemplary messaging between major com 

systems , but cannot justify the cost of including the local ponents of the distributed video surveillance system of FIG . video analysis system 112 as a component within their 5 associated with forensic processing of image data for a 

network 134. As a result , medium size businesses might use medium - sized business , ACME . The major components 

a remote video analytics system 122 hosted by a cloud 50 include video control system 114 , network video recorder 

service 180 to analyze the image data from the surveillance 130 camera 103-2 , gateway system 110-1 , local database 

cameras 103 . 132 , and cloud service 180 . A third - party company operates cloud service 180. The In step 402 , a user such as a security or surveillance 

cloud service 180 can be a public or private cloud service , operator accesses the video control system 114 to select a set 

in examples . The cloud service 180 is typically operated by 55 of recorded video data of a monitored area outside a building 

a different business entity than the business entities that 102. The set of recorded video data is associated with a 

operate businesses Cam , Biz , and ACME . The businesses specific time interval selected by the operator . The video 

are clients of the cloud service 180 . control system 114 then requests the selected set of recorded 

Here again , the cloud service 180 includes components video data from the video recorder 130 . such as a remote database 142 and a remote video analytics 60 In step 404 , the video recorder 130 sends all recorded 

system 122. The remote video analytics system 122 includes video data for the specified time interval to the gateway 

an object analysis system 123-2 . The remote database 142 system 110-1 . and the remote video analytics system 122 are nodes of Then , according to step 406 , the object detection system 

remote network 144 . 108 detects objects of interest within the video frames , and The gateway systems 110-1 , 110-2 , and 110-3 integrate 65 extracts image data from the video frames that include the 

the object detection systems 108-1 , 108-2 , and 108-3 , detected objects of interest . In examples , objects of interest respectively . include vehicles 182 and individuals 118 . 

Case 2:23-cv-00617-RWS-RSP Document 1-3 Filed 12/19/23 Page 14 of 16 PageID #: 53
cameras . 

US 10,701,321 B2 

11 12 

In one example , the object detection system detects metadata with the image data and an object analysis objects of interest within the video data by first scanning the system of the video analytics system reads the rankings pixels , macroblocks , or regions of the video data to detect within the metadata and decides whether or not to 

patterns indicative of the object class to be recognized . For analyze the image data . facial detection , the pattern may be eyes , nose and mouth 5 2. The system of claim 1 , wherein the object detection features arrayed in a particular orientation . For license plate system creates cropped images from the image data that detection , the pattern may be a rectangular area including include the detected objects of interest and sends the alphanumeric features . cropped images to the video analytics system . According to step 408 , the gateway system 110-1 then 

sends the image data including the detected objects of 10 video data and the object detection system provides frames 3. The system of claim 1 , wherein the image data include 

interest over the network cloud 60. The image data are sent from the video data that include the detected objects of over the network cloud 60 to the remote video analytics interest and sends the frames to the video analytics system system 122 of the cloud service 180 for processing . In step 410 , the remote video analytics system 122 as the image data . 

accesses reference images , image processing functions , pat- 15 4. The system of claim 1 , wherein the video analytics 

terns , neural network functions , deep learning functions , system includes an object analysis system that analyzes the 

and / or object classification functions from the remote data image data corresponding to objects of interest detected by 

base 142 . the object detection system by comparing the image data to 

Then in step 412 , the remote video analytics system 122 reference images to identify the objects of interest , by analyzes the image data via its object analysis system 123-2 . 20 recognizing patterns in the image data , and / or by applying The analysis includes comparing to the reference images , image processing functions , neural network functions , deep recognizing patterns in the image data from the patterns , learning functions , and / or object classification functions to 

and / or applying the image processing functions , the neural the image data . network functions , the deep learning functions , and / or the 5. The system of claim 1 , wherein the object detection 

object classification functions to the image data , in 25 system is integrated within the one or more surveillance 

examples . In one example , the remote video analytics system 122 6. The system of claim 1 , wherein the image data corre 

analyzes the image data by comparing the image data to sponding to objects of interest detected by the object detec 

reference images to identify the objects of interest within the tion system are compared by the video analytics system to images . In step 414 , the remote video analytics system 122 30 reference images to identify the objects of interest . sends metadata for the identified objects of interest in the 7. The system of claim 6 , further comprising a database 

image data to the gateway system 110-1 . Finally , in step 316 , that receives metadata associated with the identified objects the gateway system 110-1 saves the metadata for the iden of interest from the video analytics system . tified objects of interest in the image data to the local 8. The system of claim 1 , further comprising a gateway database 132 . 35 system that provides access to the video analytics system While this invention has been particularly shown and over a public network . described with references to preferred embodiments thereof , 9. The system of claim 8 , wherein the object detection 

it will be understood by those skilled in the art that various system is integrated within the gateway system . changes in form and details may be made therein without 10. The system of claim 1 , wherein the one or more departing from the scope of the invention encompassed by 40 surveillance cameras each include a gateway system that 

the appended claims . provides access to the video analytics system over a public What is claimed is : network . 1. A video surveillance system , comprising : 11. The system of claim 1 , wherein the video analytics 

one or more surveillance cameras that capture image data ; system receives the image data from the object detection 

an object detection system that detects objects of interest 45 system after being transmitted over an enterprise network . within the image data and extracts portions of frames 12. The system of claim 1 , wherein the video analytics that include the detected objects of interest and sends system receives the image data from the object detection 

the extracted portions of the frames ; and system after being transmitted over a cloud network . a video analytics system that receives only the extracted 13. A method for analyzing video in a video surveillance 

portions of the frames of the image data , transmitted 50 system , the method comprising : 

over a network , from the object detection system , the capturing image data from one or more surveillance 

image data received by the video analytics system cameras ; corresponding to objects of interest detected by the detecting objects of interest within the image data via an object detection system , the video analytics system object detection system ; 

analyzing the objects of interest ; the object detection system extracting portions of frames 

wherein the video analytics system receives the image that include the detected objects of interest and trans data from multiple object detection systems after being mitting the extracted portions of the frames of the 

transmitted over the network and the video analytics image data over a network , the image data transmitted 

system is operated by a different business entity than by the object detection system corresponding to objects 

the one or more surveillance cameras ; and of interest detected by the object detection system ; and wherein the image data corresponding to objects of inter a video analytics system , which is operated by a different 

est detected by the object detection system are ranked business entity than the one or more surveillance cam 

by the object detection system according to factors eras , only receiving the extracted portions of the frames 

including close proximity of the objects of interest to of the image data from the object detection system and 

the one or more surveillance cameras , picture quality of 65 analyzing the objects of interest ; 

the objects of interest , and / or contrast of the objects of the video analytics system further receiving image data 

interest to create a ranking and the ranking is saved in from other object detection systems ; 

55 

60 

Case 2:23-cv-00617-RWS-RSP Document 1-3 Filed 12/19/23 Page 15 of 16 PageID #: 54
5 

10 

15 

US 10,701,321 B2 

13 14 

the object detection system ranking the image data cor 22. A video surveillance system , comprising : 

responding to objects of interest detected by the object one or more surveillance cameras that capture image data , detection system according to factors including close each having an object detection system that detects proximity of the objects of interest to the one or more faces of individuals and license plates of vehicles surveillance cameras , picture quality of the objects of within the image data and creates cropped images from interest , and / or contrast of the objects of interest to the image data that include the detected faces of create a ranking and the ranking is saved in metadata 

with the image data ; and individuals and license plates of vehicles and send the 

an object analysis system of the video analytics system cropped images from the image data ; and 

reading the rankings within the metadata and deciding a video analytics system that receives the cropped images whether or not to analyze the image data . generated by the object detection systems of the sur 

14. The method of claim 13 , further comprising the object veillance cameras , transmitted over a network , the detection system creating cropped images from the image video analytics system analyzing the faces of individu data that include the detected objects of interest and sending als and license plates of vehicles in the cropped images the cropped images to the video analytics system . by identifying the faces of the individuals from refer 15. The method of claim 13 , further comprising the image ence images , recognizing patterns in the image data , data including video data and the object detection system providing frames from the video data that include the and by applying object classification functions to the 

detected objects of interest and sending the frames to the image data ; 

video analytics system as the image data . wherein the video analytics system is operated by a 

16. The method of claim 13 , wherein the video analytics 20 different business entity than the one or more surveil 

system analyzing the image data corresponding to objects of lance cameras ; and 

interest detected by the object detection system comprises wherein the image data corresponding to objects of inter comparing the image data to reference images to identify the est detected the object detection system are ranked objects of interest , by recognizing patterns in the image data , by the object detection system according to factors and / or by applying image processing functions , neural net including close proximity of the objects of interest to work functions , deep learning functions , and / or object clas 

sification functions to the image data . the one or more surveillance cameras , picture quality of 

17. The method of claim 16 , further comprising receiving the objects of interest , and / or contrast of the objects of 

metadata associated with the identified objects of interest interest to create a ranking and the ranking is saved in 

from the video analytics system . metadata with the image data and an object analysis 18. The method of claim 13 , further comprising integrat system of the video analytics system reads the rankings ing the object detection system within the one or more within the metadata and decides whether or not to 

surveillance cameras . analyze the image data . 19. The method of claim 13 , further comprising providing 23. The system of claim 22 , the object detection system of 

access to the video analytics system over a public network the surveillance cameras determines if the image data via a gateway system . 20. The method of claim 19 , further comprising integrat includes a product on a shelf of a large supermarket , and the 

video analytics system determines a type of product and ing the object detection system within the gateway system . 21. The method of claim 13 , further comprising the one or provides information for restocking and inventory manage 

more surveillance cameras each including a gateway system ment , and for marketing and sales purposes to determine 

that provides access to the video analytics system over a 40 how long the products remain on the shelf . public network . 

25 

30 

35 

Case 2:23-cv-00617-RWS-RSP Document 1-3 Filed 12/19/23 Page 16 of 16 PageID #: 55