System and method for image analysis based security system

A system and method for determining an object is disclosed. A security appliance, with a processor and memory is provided. The security appliance is configured to receive image of an object captured by a security device. The security device located in a first location. The security device also receives image of another object captured by another security device. The another security device located in a second location. The image of the object is processed to generate a first plurality of attributes for the object. The image of the another object is processed to generate a second plurality of attributes for the another object. The first plurality of attributes for the object and the second plurality of attributes for the another object are compared and based on the comparison, determine that the object and the another object are the same.

RELATED APPLICATIONS

TECHNICAL FIELD

The present invention relates generally to detection of an object by a security system and, more particularly, generating attributes related to the detected object.

DESCRIPTION OF RELATED ART

In the present-day environment, video or image based security systems are used to capture images based on various triggers. In some implementations, captured images are stored in a data store for review and playback. In some implementations, captured images may be sent to a user in real time. In some implementations the event of capture may be reported to a user.

In some examples, multiple devices may be capturing images. However, images captured by multiple devices may not be reviewed as a whole to develop a story or attributes about the detected object.

As more and more image capturing devices are deployed in a neighborhood, there is a need to piece together information gathered by a plurality of the image capturing device to develop a story or attribute about a detected object of interest. It is with these needs in mind, this disclosure arises.

SUMMARY OF THE INVENTION

In one embodiment, a method for determining an object is disclosed. A security appliance with process and memory is provided. Image of an object is received by the security appliance, image of the object captured by a first security device located in a first location. Image of another object is received by the security appliance, image of the another object captured by a second security device located in a second location. Image of the object is processed by the security appliance to generate a first plurality of attributes for the object. Image of the another object is processed by the security appliance to generate a second plurality of attributes for the another object. The first plurality of attributes for the object is compared with the second plurality of attributes for the another object and determine that the object and the another object are same based on the comparison.

In yet another embodiment, a system to determine an object is disclosed. A security appliance with process and memory is provided. Image of an object is received by the security appliance, image of the object captured by a first security device located in a first location. Image of another object is received by the security appliance, image of the another object captured by a second security device located in a second location. Image of the object is processed by the security appliance to generate a first plurality of attributes for the object. Image of the another object is processed by the security appliance to generate a second plurality of attributes for the another object. The first plurality of attributes for the object is compared with the second plurality of attributes for the another object and determine that the object and the another object are same based on the comparison.

This brief summary has been provided so that the nature of the disclosure may be understood quickly. A more complete understanding of the disclosure can be obtained by reference to the following detailed description of the preferred embodiments thereof in connection with the attached drawings.

DETAILED DESCRIPTION

The embodiments herein disclose a systems and methods for generating an address for an object, based on its location. Referring now to the drawings, where similar reference characters denote corresponding features consistently throughout the figures, various examples of this disclosure is described.

FIG. 1depicts an example network computing environment100. The network computing environment100may have a plurality of computing resources, for example, computing resources102A,102B and102C, all connected to internet104, over link106. In some examples, computing resources102A,102B and102C may be a cloud computing resource, for example, cloud computing resource provided by vendors like Amazon® or Microsoft®. In some examples, the computing resources102A,102B and102C may be a combination of standalone computing resources connected to the internet106and cloud computing resources.

FIG. 1also depicts security device A108A, security device B108B, security device C108C and security device D108D. The security device A108A, security device B108B, security device C108C and security device D108D are also configured to be coupled to the internet104over link106. In one example, the security device A108A is located in a first location110A, the security device B108B is located in a second location110B. The first location110A and the second location110B are located in a first neighborhood112A. In one example, the security device C108C is located in a third location110C, the security device D108D is located in a fourth location110D. The third location110C and the fourth location110D are located in a second neighborhood112B. Although only one security device is shown in each of the first location110A, second location110B, third location110C and fourth location110D, there might be multiple security devices in each of the first location, second location, third location and fourth location.

A security appliance114may be executed in computing resource102A. Additionally, one or more application servers may be executed in computing resource102B and102C. As an example, application server A116is executed on computing resource102B and application server B118is executed on computing resource102C. As one skilled in the art appreciates, application servers may be configured to provide one or more services.

In some examples, application servers may be configured as a map server, configured to provide a map associated with a location. In some examples, application servers may be configured as an image processor, capable of identifying various images, based on detecting one or more attributes of the image. In some examples, application servers may be configured as authentication servers, configured to authenticate a user to provide access to various services and functions. For example, selective access to the security appliance may be granted to one or more users, based on verification of credentials of a user by the authentication server. As one skilled in the art appreciates, these are only some of the examples of functions and features of application servers and application servers may be configured to provide various other services.

Now, referring toFIG. 2, example security appliance114of this disclosure will be described. The security appliance114includes a processor engine202, admin user interface204, subscriber user interface206, data store208, video receiver210, application programming interface (API)212and a communication engine214, all coupled over a bus216. Various function and features of the security appliance114will now be described. Detailed operation of the security appliance114will be later described with reference to additional examples and figures.

The processor engine202is configured to perform various arithmetic and logical functions of the security appliance114. The memory218is used to stored and retrieve various programs, sub-routines, including transient and permanent information used or created by the processor engine202. The data store208is used to store and retrieve various information generated, received or used by the security appliance114. In one example, the data store208may include a user data store220and a video data store224.

The admin user interface204is configured to present an admin user interface to receive one or more information from an admin user of the security appliance114. As one skilled in the art appreciates, an admin user may have certain privileges that may be different from a subscriber user. The subscriber user interface206is configured to present a subscriber user interface to receive one or more information from a subscriber user of the security appliance114. As one skilled in the art appreciates, a subscriber user may have certain privileges that may be different than an admin user. In one example, various information received from a subscriber user may be stored in the user data store220.

The video receiver210is configured to receive images or video into the security appliance. In one example, the images or video received may be stored in the video data store224. Video or images may be received from various sources, for example, from a video capturing device226, a video feed228or a smart video capturing device230. In some examples, a video capturing device226may capture a plurality of images, as a video chunk for a short duration of time, sometimes in the range of 30 to 60 seconds.

In some examples, the video capturing device226may push the captured video chunk to the security appliance114. In some examples, the video capturing device226may send the video chunk to a designated storage location for storage, for example, a storage location in a computing environment, as an example, a cloud storage device. In some examples, a link to the stored location may be sent to the security appliance114for retrieval. In some examples, the security appliance114may be configured to retrieve the details of the link and access the link to retrieve the stored video chunk at the storage location. In some examples, the security appliance114may periodically retrieve the stored video chunk from the storage location.

In yet another example, the storage device may be configured to periodically send video feeds228to the security appliance114. In yet another example, a smart video capturing device230may send or push the video chunk to the security appliance.

As one skilled in the art appreciates, various video chunks received by the security appliance114may conform to different protocols. The video receiver210is configured to decipher various video chunks and attributes related to the video chunks. The video and corresponding attributes are stored in the video data store224for further processing. Some of the attributes may include the identification of the security device, location of the security device, time stamp indicating when the video chunk was recorded and the likes.

The application programming interface212provides an interface to communicate with various external services, for example, services provided by one or more application servers. In one example, the application programming interface212may provide an interface to communicate with an object detection server232. In one example, the application programming interface212may provide an interface to communicate with a map server234. In yet another example, the application programming interface may provide an interface to communicate with a social media server236.

As one skilled in the art appreciates, the security appliance114may communicate with various external devices using one or more different communication protocols. The communication engine214is configured to communicate with external devices, using one or more protocols recognized by one or more external devices.

Having described an example security appliance114of this disclosure, now referring toFIG. 3, an example processor engine202is described. The processor engine202includes a processor302and memory218. The processor302is configured to perform various arithmetic and logical operations. The memory218is used to stored and retrieve various programs, sub-routines, including transient and permanent information used or created by the processor engine202. In one example, the processor engine202includes a login processor304, geo mapper306, an object detection engine308, a database engine310, an analytics engine312, and an AI engine314.

The login processor304is configured to receive various information, for example, user name and password for verification, verify the credentials and grant selective access to various functions and features of the security appliance114to a user. In one example, credential information may be received from a subscriber user, who interacts with the security appliance114using the subscriber user interface206. In another example, credential information may be received from an admin user, who interacts with the security appliance114using the admin user interface204.

The geo mapper engine306is configured to present a geo map of the location of the security device. For example, the geo mapper engine306may receive an address of the location of the security device from a user and based on the received address, retrieve a geo map of the location, for example, from a map server. The retrieved geo map of the location of the security device may be selectively presented to the user on a display device, by the security appliance114.

The object detection engine308is configured to analyze the received video chunks and detect one or more objects present in the received video chunk. Once an object is identified, one or more attributes of the object are detected. The object along with one or more detected attributes are stored in the data store. In some examples, the attributes of the object may be referred to as meta data for the object. The object along with the meta data for the object is stored in a data store, along with a time stamp corresponding to the video chunk and the security device that produced the video chunk.

The database engine310is configured to communicate with one or more data stores. In some examples, the database engine310may be configured to retrieve stored video chunks in a data store. In some examples, the data store may be local or internal to the security appliance114. In some examples, the data store may be external or remote to the security appliance114, for example, accessible over the link106. In some examples, the database engine310is configured to store the object and the meta data associated with the object in a data store. The data store may be internal to the security appliance114, for example, data store208or a data store external to the security appliance114. In some examples, the database engine310may associate one or more user information to the video chunk and the detected objects from the video chunk.

The analytics engine312is configured to analyze various detected objects and their attributes and develop a story regarding the detected object. For example, the movement of the object within the neighborhood. The analytics engine312may also develop statistics related to detected objects, incidents observed in a given neighborhood over time and the likes.

The AI engine314may be configured to analyze various activities detected by the security appliance for a given neighborhood, generate historic data of activities detected for a given neighborhood, generate indicators of likely future events in a given neighborhood or other neighborhoods. For example, break-ins may happen in one neighborhood on certain days or times of the week and a similar break-ins may happen in another neighborhood on certain other days or times. Based on the analysis of the historic information, the AI engine314may predict a likely future event in another neighborhood, based on activities in one neighborhood.

Now referring toFIG. 4Aan example legacy security device400is described. In one example, the legacy security device400may correspond to one or more of the security devices108A,108B,108C and108D, previously described with reference toFIG. 1. Now, referring back toFIG. 4A, the legacy security device400includes a camera402, processor404, a memory406, a data store408, a camera controller410, a video capture engine412, and a communication interface414, all coupled over a bus416. The camera402is configured to take images of the surroundings viewable by the camera402. The processor404is configured to perform various arithmetic and logical functions of the legacy security device400. The memory406is used to stored and retrieve various programs, sub-routines, including transient and permanent information used or created by the processor404. The data store408is used to store and retrieve various information generated, received or used by the legacy security device400.

The camera controller410is configured to selectively control various functions of the camera402. In one example, a sensor (not shown) may be disposed in the legacy security device400, for example, in the housing of the camera402, to detect any movement in the view of the camera402and send a trigger signal to the camera controller408. Based on the received trigger signal, the camera controller408may selectively turn on the camera402to capture any images visible to the camera402. The images captured by the camera402is processed by the video capture engine412. The processed video is then stored in the data store408for further action. In one example, processing of the video may include one or more of encoding the video in a known or proprietary format, enhancing the video, compressing the video and encrypting the video. In one example, the communication interface414may be used to communicate with external devices. In some examples, an alert signal may be sent to a user, by the communication interface414.

In some examples, the legacy security device400may be coupled to a digital video recorder (DVR) (not shown) which may be configured to communicate with the security device400and store one or more of the video images captured by the security device400. In some examples, the legacy security device400may be configured to communicate over the internet and store one or more of the video images in a storage device accessible over the internet. In some examples, the DVR may be configured to communicate over the internet and store one or more of the video images in a storage device accessible over the internet.

In some examples, video images stored in the storage device accessible over the internet may be selectively retrieved by the security appliance114. In some examples, the security appliance114may be provided with access information to access the stored video images from the DVR. In some examples, the stored video images in the legacy security device400may be accessible by the security appliance114. In some examples, a user may selectively upload the stored video images to the security appliance114.

Having described a legacy security device400, a smart security device420will now be described with reference toFIG. 4B. Now referring toFIG. 4Aan example smart security device420is described. In one example, the smart security device420may correspond to one or more of the security devices108A,108B,108C and108D, previously described with reference toFIG. 1. Now, referring back toFIG. 4B, the smart security device420is similar to legacy security device400in that the smart security device402includes a camera402, processor404, a memory406, a data store408, a camera controller410, a video capture engine412, and a communication interface414, all coupled over a bus416. In addition, the smart security device420includes a video stream processor422, an object engine424and a SA interface426. These elements will now be described.

The smart security device420functions similar to legacy security device400in that video images are captured and stored in the data store408. The video stream processor422retrieves the stored video images and identifies one or more objects in the video images and sends the objects for further processing by the object engine424. The object engine424identifies various attributes of the object and creates meta data associated with the detected object. The detected object along with the meta data is stored in the data store424. In one example, the object engine424classifies one or more detected objects as known objects, based on observing the presence of the detected object in multiple video streams over time. Detected objects that do not occur frequently may be classified as an unknown object. In one example, the unknown object and associated meta data may be sent to the security appliance, by the smart security device.

The SA interface426is configured to communicate with the security appliance114. In one example, the security appliance114may send a command to the smart security device to initiate capture of the images by the camera402. For example, if there are multiple smart security devices in a neighborhood, and one of the smart security device sends a message to the security device that an unknown object was detected, the security appliance114may selectively enable other smart security devices in the neighborhood to initiate capture of the video images by their camera. Objects detected by these other smart security devices may be classified as known or unknown object. If multiple smart security devices detect the same object, based on a time of detection and location of the smart security device, a possible path the object moved in the neighborhood is determined.

In some examples, objects classified as unknown may be sent to a user to classify. In some examples, the smart security device420may send the objects classified as unknown to the user for classification. In some examples, security appliance114may send the objects classified as unknown to the user for classification. Based on user response, the objects classified as unknown may be reclassified as a known object. In some examples, updated classification of the object may be stored in the smart security device420. In some examples, the updated classification of the object may be stored in the security appliance114.

In some examples, the security appliance114may send one or more objects it has received from security devices in the neighborhood to the smart security device420for identification. The smart security device420may compare the received object and compare the received objects with objects stored in the data store408for a comparison. If the received object matches one or more of the stored objects that are classified as known, the smart security device420responds to the security device114, indicating the received object is a known object.

Now referring toFIGS. 5A and 5B, example database schema for use with this disclosure is described. Now, referring toFIG. 5A, table502, various attributes associated with a user is shown. Table504shows various attributes of a camera (sometimes referred to as a security device associated with the user. Table506shows various attributes of an incident associated with the user. Table508shows various attributes associated with a video chunk is described. In one example, the camera is associated with the video chunk. In one example, the incident is associated with the video chunk.

Now, referring to table502, some of the attributes of interest for user are location510, camera list512, incident list514, neighbor list516and familiar object list516. Location510indicates the location of the user. In one example, the location of the user may include a street address. In one example, the street address may be mapped to a geo location or geo coordinates, like latitude and longitude. Camera list512indicates one or more cameras associated with the user. Incident list514indicates incidents reported by the user. Neighbor list516indicates neighbors associated with the user. As one skilled in the art appreciates, a neighbor will be another user, with associated user, camera, incident and video chunk tables. Familiar object list518corresponds to objects determined to be known object based on classification of objects captured by one or more cameras associated with the user.

Now, referring to table504, some of the attributes of interest for a camera are described. Streaming URL520, which corresponds to a storage location where video chunks from the camera are stored. In one example, the streaming URL520may be used by the security appliance to selectively retrieve stored video chunks. As previously described, in some examples, the storage location may be local to the security device and in some examples, the storage location may be external to the security device. Camera UID522corresponds to a unique identification for the camera. Camera privacy524corresponds to a privacy setting for the camera. A user may selectively assign a privacy setting for the camera, for example either as a “public” camera or a “private” camera. If the privacy setting of a camera is set as “public”, then, the video chunks captured by the camera may be accessible to the security appliance. If the privacy setting of the camera is set as “private”, then, the video chunks captured by the camera may not be accessible to the security appliance. In some examples, further granularity in the privacy settings may be provided. One example of further granularity may include assigning a “protected” category, wherein the video cannot be shared with neighbors, but can be processed by the security appliance. Another example of further granularity may include assigning a “Time-limited public”, meaning the security appliance can retain the video only for a limited period of time. In one example, a user may be provided an option to select the time period. Camera type526may indicate the type of camera, for example, a legacy security device or a smart security device. Username528and password530are associated with the access control credentials to access the camera. Chunk list532corresponds to the list of video chunks created and stored by the camera.

Now referring to table506, some of the attributes of interest for incident are described. Title534corresponds to a title of the incident. In one example, this is created by the user. Description536corresponds to a brief description of the incident. In one example, this is created by the user. Camera details538corresponds to the camera that captured the incident. Object list540corresponds to the list of objects observed in the video chunk that corresponds to the incident. Date from542and date to544correspond to a time window during which the incident took place. For example, if a package was delivered at 2:00 PM on Jan. 10, 2019 and was noticed missing at 6:00 PM on Jan. 10, 2019, the date from542would correspond to 2:00 PM on Jan. 10, 2019 and date to544would correspond to 6:00 PM on Jan. 10, 2019. Report546corresponds to the report associated with the incident.

Now, referring to table508, some of the attributes of interest for video chunk are described. Chunk path548refers to the path, address or url to the stored video chunk. Timestamp550corresponds to the time associated with when the video chunk was captured. Objectlist552corresponds to the objects detected in the video chunk. Processed path554corresponds to the path, address or url to the processed video chunk. As previously described, the video chunk is processed or analyzed for detecting objects and detected objects along with the meta data associated with the detected object are stored. Processed path554corresponds to the path, address, or url that links to the processed video chunk. Status556corresponds to whether the video chunk has been processed or not. Chunk duration558corresponds to the duration of the video chunk. As previously described, duration of a video chunk may be of the order of about 30 seconds to about 60 seconds.

Now, referring toFIG. 5B, table560described various attributes associated with a neighborhood. Name562refers to a name of the neighborhood. Name of the neighborhood may be set by the admin user of the security appliance. Geographic boundaries564corresponds to the meet and bound of the neighborhood. In some examples, various streets in the neighborhood may be used to draw a geographic boundary. In some examples, a pre-defined area may be used to define the geographic boundaries. In some examples, a range of latitude and longitude may be used to define a geographic boundary for the neighborhood. User list566corresponds to the users located within the geographic boundary of the neighborhood. Neighborhood watch info568corresponds to names of people associated with the neighborhood watch program for the neighborhood. Familiar objects list570corresponds to list of objects that have been classified as known or friendly objects within the neighborhood.

Now, referring toFIG. 5C, table580, various attributes associated with an object is described. Referring to table580, various attributes associated with an object may include object ID582, object classification584, object identification586, timestamp588, camera type590, camera location592, object characteristics594, linked cameras596, and object history598. In some examples, various attributes associated with an object may be referred to as meta data associated with the object.

Object ID582may correspond to a unique identifier for the object. Object ID582may be numeric, alphabet or a combination of alpha-numeric number. Object classification584identifies a class or group to which the object belongs to. For example, based on the analysis of the video chunk, one or more objects may be identified. Each identified objects is given an object ID. Next, the object is analyzed to determine which group it belongs to. For example, a person, vehicle, animal etc. Based on the analysis, Object classification584is updated to indicate the group to which the object belongs. Timestamp588refers to the time at which the object was detected. Camera type590corresponds to the type of camera where the object was captured. Camera location592corresponds to the location details of the camera that captured the image of the object.

Object characteristics594corresponds to various characteristics of the object. In one example, the object characteristics594may be different, based on the object classification584. Some of the object characteristics594may be whether the object is a friendly, unfriendly or unknown object. In case of a vehicle, the object characteristics594may be the color of the vehicle, license plate number and the likes. In one example, objects that are friendly, may be associated with corresponding neighborhood table shown inFIG. 5B, for example, as part of the familiar object list.

Linked cameras596corresponds to a list of other cameras that are linked or associated with the camera where the object was captured. For example, if there are multiple cameras in a location and an object was captured in one camera and later classified, it may be beneficial to associate the object with other cameras in the same location.

Object history598corresponds to history associated with the identified object, for example, if the object was a subject or target of prior incidence analysis.

Now, referring toFIGS. 6A and 6B, various queries and activities supported by the security appliance114is described. Table602shows queries related to a user. Table604shows queries related to a camera. Table606shows queries related to the incidents. Table608shows queries related to video chunks. Table610shows queries related to neighborhood. Table612shows queries related to business logics. Table614shows various queries, their corresponding input and output. For example, column616shows various queries. Column618shows input for the query and column620shows output for query if applicable. For example, referring to row622, for “Create Incidents” query, the input is “incident details”. The “incident details” may correspond to one or more of the attributes described with reference to table506inFIG. 5A. For example, incident details may include title, date, time range, description, cameras and the like.

Now, referring toFIG. 7A, an example sequence flow diagram700is described. The sequence flow diagram700illustrates various communication and actions between the security appliance114, smart security device420and a subscriber user702associated with the smart security device. In one example, the smart security device420upload object list (704) to the security appliance114. The object list may contain lists of objects discovered by the smart security device402. The object may be of many types, for example, a facial image, a vehicle, a license plate, a car and the likes. The security appliance114updates the objects table in the database and associates the objects to the corresponding smart security device (706). Next the stored objects in the database are read by the security appliance114and user associated with the smart security device420is identified (708). Next, the security appliance114notifies the identified user to review the objects for classification (710). The identified user may be notified by one or more methods, for example, via e-mail, text message or via a smart phone application. The identified user reviews the objects and classify the objects, for example, as friendly object (or known object), unfriendly object and unknown object (712). In one example, the identified user is presented with each of the objects and a list of classification to be chosen by the identified user. The security appliance114receives the classification and updates the database with the classification for the objects (714). The security appliance114then reads the classified objects (716) and sends the classified objects and their classification to the smart security device (718). In one example, the classified objects and their classification are sent to all the smart security devices associated with the identified user.

Now, referring toFIG. 7B, another example sequence flow diagram720is described. The sequence flow diagram720illustrates various communication and actions between the security appliance114, smart security device420, a subscriber user702associated with the smart security device, neighborhood smart security devices722and neighborhood users724. The sequence flow diagram720illustrates an example smart security device initiated incident management.

The smart security device420sends a message to the security appliance114to create an incident (726). As one skilled in the art appreciates, the smart security device420can scan the video chunks and based on its analysis can initiate an incident, for example when an unknown or unfriendly object is identified. The message in one example includes camera ID, objects of interest and a timestamp.

The security appliance114updates the incident table in the database (728). The security appliance114then reads the incident details, identifies the user702associated with the smart security device420(730). The security appliance114then notifies the identified user about the new incident (732). In one example, the notification may include details of the incident and a request to the user702to upload any video images that may be relevant to the incident stored in legacy security devices, for example, based on the timestamp of the incident.

The security appliance114also notifies other subscribers or users in the neighborhood regarding the new incident (734). In one example, the notification may include details of the incident and a request to the users in the neighborhood to upload any video images that may be relevant to the incident stored in legacy security devices, for example, based on the timestamp of the incident.

The subscriber user and users in the neighborhood upload any video images of interest to the security appliance114(736). The security appliance114updates the database with the received video images (738).

In one example, the security appliance114also sends a message to other smart security devices in the neighborhood about the incident and request upload of objects relevant to the incident (732a). Smart security devices in the neighborhood send objects relevant to the incident to the security appliance114(736a). Received information from the smart security devices in the neighborhood are stored in the data store (738a).

Users may upload any additional video images, when the incident is still active (740). The incident age may be set to be a predefined time period, for example, one week (742). Various video images and objects received by the security appliance114is analyzed by the security appliance and an incident report is generated (744). In one example, the security appliance identifies set of objects detected by the user camera as well as cameras in the neighborhood. Based on the detected objects, a geospatial and temporal analysis is performed to determine movement of the detected object in the neighborhood, identification of unfamiliar objects and detection of anomalous behavior. Based on the analysis, the security appliance generates a report. In one example, the generated report may include one or more images, indicative of the evidence for the incident. The generated incident report is sent to the user associated with the smart security device402that initiated the incident (746).

In one example, the incident report may include time stamped video footage containing all objects of interest from the subscriber user's smart security device, time stamped video footage containing same objects of interest from the smart security devices in the neighborhood, time stamped video footage containing the same objects of interest from legacy security devices within the neighborhood, best fit track (or movement) of the object of interest across the neighborhood, and meta data pertaining to the object of interest (for example, license plate number, color/make/model of vehicle, build/height of a person and the like).

Now, referring toFIG. 8, another example sequence flow diagram800is described. The sequence flow diagram800illustrates various communication and actions between the security appliance114, an incident user802, neighborhood smart security devices722and neighborhood users724. The sequence flow diagram720illustrates an example user initiated incident management.

Now, referring to flow diagram800, an incident user802initiates an incident for processing by the security appliance114(804). In one example, the incident user802uploads a video image corresponding to the incident. The video image may be captured by a legacy security device. The security appliance114updates the incidents table in the database (806). The security appliance114reads the created incident and queries the subscriber table to identify the incident user (808). Then, the security appliance114determines the geographic area around the subscriber's neighborhood and retrieves a list of smart security devices in the subscriber's neighborhood.

The security appliance114then notifies all the subscribers or users in the neighborhood about the incident (810). In one example, the notification may include details of the incident and a request to the users in the neighborhood to upload any video images that may be relevant to the incident stored in legacy security devices, for example, based on the timestamp of the incident.

The users in the neighborhood upload any video images of interest to the security appliance114(812). The security appliance114updates the database with the received video images (814).

In one example, the security appliance114also sends a message to the smart security devices in the neighborhood about the incident and request upload of objects relevant to the incident (810a). Smart security devices in the neighborhood send objects relevant to the incident to the security appliance114(812a). Received information from the smart security devices in the neighborhood are stored in the data store (814a).

Users may upload any additional video images, when the incident is still active (816). The incident age may be set to be a predefined time period, for example, one week (818). Various video images and objects received by the security appliance114is analyzed by the security appliance and an incident report is generated (820).

In one example, objects received by the security appliance114is compared with a list of objects that have been associated with incidences that were reported and analyzed previously. In some examples, the list of objects from earlier reported incidences may be referred to as a suspect object list. In some examples, the objects may be auto detected by the security appliance, without any human interactions, In some examples, the objects may have to be presented to a user to help characterize various attributes of the object and identify if the object is of interest in the reported incidence. Once one or more objects of interest are identified, the security appliance114can check whether other security devices in the neighborhood have captured same object.

As one skilled in the art appreciates, once an object has been classified, for example, as a person or a vehicle, further analysis may be performed. For example, if the object is a person, face of the person can be extracted and features of the face (sometimes referred to as “faceprint”) may be mapped and compared with other objects that were identified and classified in other security devices. In one example, there may be a list of known suspect person table, with extracted features of the face. This suspect person table may be searched for a possible match. In some examples, if there is a match, then, the image is discarded and a reference identifier of the suspect person may be used. If no match is found, then, the faceprint of the person may be stored for further classification. Over time, if the image of the person is captured at a given location in multiple occasions, the person may be classified as a known or friendly person, associated with that location. In one example, the person is also associated with the neighborhood.

If the object is classified as a vehicle, then, meta data associated with the vehicle may be selectively extracted, by analyzing the object. Extracted meta data for the vehicle may be stored in the object attribute table. Over time, if the vehicle is captured at a given location in multiple occasions, the vehicle may be classified as a known or friendly vehicle, associated with that location. In one example, the vehicle is also associated with the neighborhood.

Based on geospatial and temporal analysis, likely participation of the object of interest in the reported incidence is determined. Thereafter, an incident report is generated.

The generated incident report is sent to the incident user that initiated the incident (822). In one example, the incident report may include time stamped video footage containing all objects of interest from the incident user's security device, time stamped video footage containing same objects of interest from the smart security devices in the neighborhood, time stamped video footage containing the same objects of interest from legacy security devices within the neighborhood, best fit track (or movement) of the object of interest across the neighborhood, and meta data pertaining to the object of interest (for example, license plate number, color/make/model of vehicle, build/height of a person and the like).

Now, referring toFIGS. 8A, 8B and 8C, an example incidence analysis will be described. Referring toFIG. 8A, an example neighborhood830is shown, with a street map and location of various devices, for example, cameras in the neighborhood830. For example, cameras C1832and C6834are located along the “Delta Road”, camera C2836is located along the “Oldfield Way”, camera C3838is located on “Fowler Way”, cameras C4840and C5842are located on “Addison Park Ln”. Delta Road, Oldfield Way and Fowler Way all connect to Main Street. Ruby Way connects Delta Road and Oldfield Way. Countryforge Ln connects Oldfield Way and Fowler Way.

In one example, when an object is detected by one camera, it may send a signal to other neighborhood smart security devices to turn on the camera and capture the images. For example, when camera C1detects an object or movement, it may send a signal to camera C2and camera C6to turn on the camera and capture the images. When camera C2detects an object or a movement, camera C2may send a signal to camera C3, camera C4and camera C5to turn on the camera and capture images. In one example, the signal will turn on the camera and capture images for a defined period of time, In one example, the defined period of time may be based on the distance between the location of the cameras and an estimated time it would take for a moving object to travel from one location to another location.

In one example, owner of camera C1reports an incident, for example, tampering of his mailbox, on the night of Feb. 2, 2019. Objects captured by the camera C1is reviewed for a defined time range. For example, two objects, object ID 215142 and object ID 215143 are identified. Based on the analysis of stored objects, an object tracking table for each of the identified objects are created. For example, referring toFIG. 8B, an object tracking table850is shown, for object with an object ID of 215142. And, referring toFIG. 8C, another object tracking table860is shown, for object with an object ID of 215143.

Now, referring toFIG. 8B, table850, referring to rows852and854, we notice that Camera C1captured the object ID of 215142 at a time of “2 Feb. 2019 9:00 PM”. Referring to rows856and858, we notice that Camera C6captured the object ID of 215142 at a time of “2 Feb. 2019 8:59 PM”. Referring to rest of the rows of the object tracking table850, we notice none of the other cameras (C2-C5) in the neighborhood830detected same object. In one example, various attributed of object ID 215142 may be stored in an object attribute table as previously described with reference toFIG. 5C, table580. In one example, the object ID of 215142 may correspond to a friendly object, for example, a vehicle known to be parked at the location corresponding to Camera1. Based on this analysis, object ID 215142 is concluded as an object not related to the reported incident. Further, based on the time stamp of camera C1and C6, a likely path for the object ID 215142 can be predicted. In this case, given the time stamp of Camera C6is prior to the time stamp of Camera C1, it may be concluded that object ID 215142 moved from the location of camera C6to the location of camera C1. It may be further concluded that the object ID 215142 entered Delta Road from Main Street.

Now, referring toFIG. 8C, table860, rows862and864, we notice that Camera C1captured the object ID of 215143 at a time of “2 Feb. 2019 11:02 PM”. Further, referring to rows866and868, we notice that camera C2captured the object ID of 215143 at a time of “2 Feb. 2019 11:03 PM”. Further, referring to rows878and880, we notice that camera C5captured the object ID of 215143 at a time of “2 Feb. 2019 11:05 PM”. However, referring to rows870-874and878-880, we notice that object ID of 215143 was not captured by camera C3, camera C4and camera C6. In one example, various attributed of object ID 215143 may be stored in an object attribute table as previously described with reference toFIG. 5C, table580. In one example, the object ID of 215143 is marked as an unknown object, based on the information stored in the object attribute table.

Now, a geospatial and temporal analysis of the data stored in the object tracking table860is performed. In one example, based on the address of the camera location, a corresponding geo location is retrieved. In one example, a request may be sent to a map server, with the address and receive corresponding geo location of the camera. In one example, the geo location may be the latitude and longitude of the location of the camera. In one example, distance between the cameras of interest may be calculated. In one example, a map server may be configured to provide a distance between various addresses. In some examples, the map server may be configured to provide the distance based on paths or roads that correspond to the address of the camera location. Next, based on the permitted speed limit in a neighborhood, an estimate of time to travel from one camera location to another camera location is calculated. Based on the time to travel from one camera location to another camera location, a possible route for the movement of the object is determined.

Now, referring toFIGS. 8A and 8C, a possible route for the movement of the object with an object ID of 215143 is determined. In one example, the distance between camera C1and camera C2is L1. Distance between camera C2and camera C5is L2. Based on the detection of the object with an object ID of 215143 by cameras C2and C5, a likely path for the movement of the object is camera C1location on Delta road to camera C2location on Oldfield way via Ruby Way and camera C5via Countryforge Ln. This likely path is further validated by comparing the difference in the timestamp of detecting the object ID 215143.

For example, there is a difference of one minute in timestamp between camera C1and camera C2. Based on the distance of L1and a time difference T1of one minute, a likely speed of travel of the object ID 215143 is calculated. Likely speed of travel S1may be calculated by dividing the time difference T1by the distance L1. Calculated speed S1is then compared with permitted speed limit P1for the neighborhood to see if the calculated speed S1is within a threshold value Q1of the permitted speed limit P1. In one example, threshold value Q1may be set to be within 10% of the permitted speed limit P1. In this example, the calculated speed S1is within the threshold value Q1of the speed limit P1. This conclusion validates that the object ID 215143 moved from location of camera C1to location of camera C2.

Similarly, there is a difference of two minutes in timestamp between camera C2and camera C5. Based on the distance of L2and a time difference T2of two minutes, a likely speed of travel of the object ID 215143 is calculated. Likely speed of travel S2may be calculated by dividing the time difference T2by the distance L2. Calculated speed S2is then compared with permitted speed limit P1for the neighborhood to see if the calculated speed S2is within a threshold value Q1of the permitted speed limit P1. In one example, threshold value Q1may be set to be within 10% of the permitted speed limit P1. In this example, the calculated speed S2is within the threshold value Q1of the speed limit P1. This conclusion validates that the object ID 215143 moved from location of camera C2to location of camera C5.

Referring back toFIG. 8A, we notice that there were two likely paths for the object ID 215143 to travel from location of camera C2to location of camera C5, either via Countryforge Ln or via Addison Park Ln. However, cameras C3and C4located on Addison Park Ln did not detect the object ID 215143 during the time frame of interest. Therefore, the likely movement of object ID 215143 is concluded to be via Countryforge Ln. Further, based on the movement of the object ID 215143 from location of camera C2to location of camera C5, it may also be concluded that the object ID 215143 proceeded on to Main Street.

As one skilled in the art appreciates, in some examples, more than one likely path may be predicted based on the analysis of the object tracking table. In such a scenario, a plurality of likely paths may be identified for the object of interest.

Now, referring toFIG. 9an example flow diagram900is described. In block S902, a security appliance with a processor and memory is provided. For example, security appliance114as described with reference toFIG. 2is provided.

In block S904, image of an object is received by the security appliance. The image is captured by a security device located in a first location. For example, image captured by security device A is received by the security appliance, as described with reference toFIG. 1. In one example, the image captured by security device C1is received by the security appliance, as described with reference toFIG. 8A.

In block S906, image of another object is received by the security appliance. The image is captured by another security device located in a second location. For example, image captured by security device B is received by the security appliance, as described with reference toFIG. 1. In one example, the image captured by security device C2is received by the security appliance, as described with reference toFIG. 8A.

In block S908, image of the object is processed by the security appliance to generate a first plurality of attributes for the object. For example, meta data of the object is created by analyzing the object. For example, meta data of the object may be created as described with reference toFIG. 5C.

In block S910, image of the another object is processed by the security appliance to generate a second plurality of attributes for the another object. For example, meta data of the another object is created by analyzing the object. For example, meta data of the object may be created as described with reference toFIG. 5C.

In block S912, the first plurality of attributes for the object and the second plurality of attributes for the another object is compared and based on the comparison, the object and the another object is determined to be the same. By comparing one or more attributes of the object and another object in the corresponding object table, the object and another object may be determined to be the same. For example, one or more object characteristics of the object may be compared to determine whether the object and another object are same.

The embodiments disclosed herein can be implemented through at least one software program running on at least one hardware device and performing various functions of the security appliance. Various functions of the security appliance as described herein can be at least one of a hardware device, or a combination of hardware device and software module. In some examples, one or more functions described with reference to the security appliance may be performed in the security device. For example, in some examples, analysis of the objects may be performed in the security device, for example, a smart security device. Based on the analysis of the objects, object attribute table may be generated by the security device. In some example, the generated object attribute table may be selectively accessible to the security appliance.

The hardware device can be any kind of device which can be programmed including e.g. any kind of computer like a server or a personal computer, or the like, or any combination thereof, e.g. one processor and two FPGAs. The device may also include means which could be e.g. hardware means like e.g. an ASIC, or a combination of hardware and software means, e.g. an ASIC and an FPGA, or at least one microprocessor and at least one memory with software modules located therein. Thus, the means are at least one hardware means, and at least one software means. The method embodiments described herein could be implemented in pure hardware or partly in hardware and partly in software. Alternatively, the invention may be implemented on different hardware devices, e.g. using a plurality of CPUs.