Using artificial intelligence to analyze sensor data to detect potential change(s) for risk and threat assessment and identification

In some aspects, a server receives a video stream from a security system and processes a frame from the video stream to create a processed frame. The server analyzes the processed frame using artificial intelligence and determines that the processed frame includes a change to a surface area of an object and determines details associated with the change. The server determines that the change satisfies one or more thresholds, such as a change threshold and a time threshold. The server adds annotations to the processed frame to create an annotated frame. The annotations include the change and at least a portion of the details associated with the change to the surface area of the object. The server sends, to a designated recipient, a notification that includes a link to view the annotated frame.

FIELD OF THE INVENTION

This invention relates to a system that uses artificial intelligence to analyze sensor data (e.g., images, audio data, and the like) to detect a potential threat, risk, crime, or a potential change to an object, including a change to state of the object, or a posture of a human being. If a change threshold is satisfied, the system may perform one or more actions, including, for example, notifying designated recipient(s).

DESCRIPTION OF THE RELATED ART

Various locations (e.g., locations inside a building and locations outside the building), such as a place of worship, a statue, and the like may be subject to damage by vandals. For example, the damage may include graffiti, defacement, or the like. Often, by the time the authorities (the police, an owner, a manager, a security company, or the like) have been notified, the vandal(s) have left the scene, making it difficult to find and apprehend the perpetrators.

SUMMARY OF THE INVENTION

This Summary provides a simplified form of concepts that are further described below in the Detailed Description. This Summary is not intended to identify key or essential features and should therefore not be used for determining or limiting the scope of the claimed subject matter.

In some aspects, a server receives a video stream from a system and processes a frame from the video stream to create a processed frame. The server analyzes the processed frame using artificial intelligence and determines that the processed frame includes a change to a state of an object and determines details associated with the change. The server determines that the change satisfies one or more thresholds, such as a change threshold and a time threshold. The server adds annotations to the processed frame to create an annotated frame. The annotations include the change and at least a portion of the details associated with the change to the state of the object. The server sends, to a designated recipient, a notification that includes a link to view the annotated frame.

DETAILED DESCRIPTION

The systems and techniques described herein monitor the output, e.g., the sensor data (e.g., video and/or audio data) of a security system and use artificial intelligence (AI) to analyze the sensor data to identify changes, determine whether the changes satisfy one or more thresholds, and if the changes satisfy the one or more thresholds, provide a notification to one or more recipients. The notification may include a link that enables the recipient(s) to view at least a portion of the video, listen to a portion of the audio data, or both. For example, in some cases, the notification may be sent to an owner of a business or to a security company monitoring the video stream and the owner or an employee of the security company may view at least a portion of the video stream, listen to the audio data, or both and determine whether to notify the police. In other cases, the notification may be sent directly to the police and the police may view the portion of the video stream, listen to the audio data, or both and determine whether to send police officers to the business, e.g., to apprehend those involved.

The video stream and/or audio data may be provided by one or more sensors (e.g., image sensors, microphones, and the like). The image sensors may have a field of view of one or more objects (defined herein as anything having a surface area), one or more people, or any combination thereof. The sensors of the security system may include microphones to provide audio data for analysis by the AI. The field of view may include (1) a portion of an interior surface of a building, a vehicle, or another interior object, (2) at least a portion of exterior surface(s) of an object, such as a building, (3) immovable objects (e.g., statue or the like), (4) movable objects, (5) one or more human beings, or any combination thereof. For exterior objects, the AI may be trained to recognize and ignore weather-related effects, such as snow, sleet, hail, rain, and the like, while flagging other changes for human review. For example, a truck splashing water on the exterior surface of a building, a person spilling a drink on an interior wall of a building, a bird defecating on a statue, or the like may cause the AI to detect and send a notification of the change to a human for review. In some cases, the AI may use a neural network algorithm, such as a convolutional neural network algorithm.

The artificial intelligence may perform change detection to one or more objects using several different techniques. An object is anything with a surface area. An object may include other objects. For example, a statue may include both the statute and a display case, pedestal, or the like. The artificial intelligence may identify a change to an object, such as something (e.g., graffiti) that has been added to an object (e.g., exterior of a building, statue, interior of a building or vehicle, or the like). As another example, the artificial intelligence may identify a change to an object, such as defacement of an object (e.g., removal of a portion of the object).

In some cases, the output of the security system may be modified prior to being analyzed by the artificial intelligence. For example, an image from one of the video streams may be modified by changing the size, changing the dimensions, changing the sharpness, changing the zoom, changing the brightness, changing the contrast, changing the saturation, other modifications, or any combination thereof to enable and increase an accuracy of computer vision change detection using the artificial intelligence. Audio data from microphones may be filtered to emphasize human voice frequencies and automatic speech recognition performed to determine whether the speech includes threatening words or phrases. In some cases, one large image may be split into two or more smaller images. For example, a static object, such as a statue, may be distinguished from moving objects, such as people, animals, and the like.

After identifying one or more changes in the images, the artificial intelligence may determine whether the changes are sufficient to satisfy a particular threshold. For example, if a person temporarily stands in front of an immovable object (e.g., exterior wall, statue, or the like), then the artificial intelligence may determine that no change is present because the threshold was not satisfied. As another example, if a person spray paints graffiti or defaces an immovable object (e.g., exterior wall, statue, or the like), then the artificial intelligence may determine that the threshold is satisfied (e.g., change to the object persist for more than a predetermined amount of time). If the threshold(s) are satisfied, one or more designated recipients (e.g., an owner, a manager, a security company employee, a police station, a website, a social media posting, or the like) may be provided a notification.

As a first example, a server includes one or more processors and one or more non-transitory computer-readable storage media to store instructions executable by the one or more processors to perform various operations. The operations include receiving a video stream comprising a plurality of frames from a security system. The operations include processing a frame from the plurality of frames to create a processed frame. For example, processing the frame from the plurality of frames to create the processed frame may include: adjusting a frame size of the frame, adjusting a sharpness of the frame, adjusting a zoom of the frame, adjusting a brightness of the frame, adjusting a contrast of the frame, adjusting a color saturation of the frame, adjusting one or more colors of the frame, converting the one or more colors of the frame to a grayscale, converting the frame into two or more portions of the frame, adjusting the resolution of the frame, stitching one or more frames into a single frame or any combination thereof. The operations include performing an analysis of the processed frame using an artificial intelligence module executed by the one or more processors. For example, the artificial intelligence module uses a deep learning neural network. The operations include determining, based on the analysis, that the processed frame includes a change to a state of an object. For example, the change to the state of the object may include a change in: a presence or absence of the object, an orientation of the object, a position of the object, a color of the object, a location of the object, a material of the object, a texture of the object, a shape of the object, a light on the object, a reflection of the object, a size of the object, a visibility of the object, a surface area of the object, or any combination thereof. Determining, based on the analysis, that the processed frame includes the change to the state of the object may include determining that a substance is present on at least a portion of the object. Determining, based on the analysis, that the processed frame includes the change to the surface area of the object may include determining that at least a portion of the object includes defacement. For example, the defacement may include (1) scratching at least the portion of the object, (2) removing at least the portion of the object from the object, or (3) both. The operations include determining, based on the analysis, details associated with the change to the state of the object. The operations include determining, based on the details associated with the change to the state of the object, that a particular threshold of a plurality of thresholds has been satisfied. For example, the particular threshold may include (1) a particular type of change and (2) a length of time that the particular type of change is present. The operations include annotating the processed frame to create an annotated frame that includes the change and at least a portion of the details associated with the change to the state of the object. The operations include sending a notification to a designated recipient, the notification including a link to view the annotated frame.

As a second example, a server includes one or more processors and one or more non-transitory computer-readable storage media to store instructions executable by the one or more processors to perform various operations. The operations include receiving a video stream from a security system. The operations include processing a frame from the video stream to create a processed frame. For example, processing the frame to create the processed frame may include: adjusting a frame size of the frame, adjusting a sharpness of the frame, adjusting a zoom of the frame, adjusting a brightness of the frame, adjusting a contrast of the frame, adjusting a color saturation of the frame, adjusting one or more colors of the frame, adjusting the resolution of the frame, stitching one or more frames into a single frame converting the one or more colors of the frame to a grayscale, converting the frame into two or more portions of the frame, or any combination thereof. The operations include performing an analysis of the processed frame using an artificial intelligence module. In some cases, the artificial intelligence module may use a deep learning neural network. For example, the analysis may identify in the processed frame at least one of: a posture of a person, a gesture performed by the person, a facial expression of the person, a weapon, or a state of an object. To illustrate, the gesture may include: a first gesture that indicates a presence of a threat or a second gesture performed near a particular sign. As another illustration, the posture associated with the person includes: holding both arms up, holding both hands behind the person's head, holding both hands behind the person's back, crouching, lying down, push, fist fight, kick, punch, slap, hackling, shoving, hands over a person's face, hands over another person's body, hands tied, legs tied, hiding behind an object, crouching behind an object, a presence of the person in an area that the person is not authorized to be present in, bent over, shaking, trembling, standing, seating, walking, running, snatching, an injured posture, or any combination thereof. As a further illustration, the facial expression may indicate: fear, surprise, anger, aggression, pain, hurt, distress, aggravation, or being bullied, or any combination thereof. The operations include determining, based on the analysis, that the processed frame includes one or more threats. The operations include determining, based on the analysis, details associated with individual threats of the one or more threats. For example, the details associated with individual threats may include: a locus comprising one of the person, or the object, a threat type, a threat level determined based at least in part on the facial expression of the person, a threatening gesture, or the weapon in the processed frame, a length of time individual threats have been detected, a consecutive number of frames that individual threats have been detected, a probability associated with the threat type, or any combination thereof. The operations include determining, based on the details associated with the one or more threats, that a particular threat threshold of a plurality of threat thresholds has been satisfied. For example, the particular threat threshold may include (1) a particular type of threat and (2) a length of time that the particular type of threat is present. The operations include annotating the frame to create an annotated frame that includes the one or more threats and at least a portion of the details associated with individual threats of the one or more threats. The operations include sending a notification to a designated recipient. The notification includes a link to view the annotated frame.

As a third example, a server may include one or more processors and one or more non-transitory computer-readable storage media to store instructions executable by the one or more processors to perform various operations. The operations include receiving audio data (e.g., a type of sensor data) from one or more microphones (e.g., a type of sensor) of a security system. The operations include processing the audio data to create processed audio. For example, processing the audio data to create the processed audio may include filtering the audio data to (1) include a frequency range associated with a human voice (2) exclude higher frequencies above the frequency range associated with the human voice, and (3) exclude lower frequencies below the frequency range associated with the human voice. The operations include performing an analysis of the processed audio using an artificial intelligence module. The analysis may identify, in the processed audio, at least one of: a sound, a word, or a phrase. For example, the sound may include an indication of pain. As another example, the word or phrase may indicate an argument, indicate cursing, indicates pleading, indicate an injury, indicate a threat, indicate bullying, indicate pain, indicate fear, includes a racial slur, or any combination thereof. As a further example, the word or phrase may have (1) a volume level greater than a first decibel threshold, indicating shouting or (2) a volume level less than a second decibel threshold, indicating whispering. The operations include determining, based on the analysis, that the processed audio includes one or more threats. The operations include determining, based on the one or more threats, that a particular threat threshold of a plurality of threat thresholds has been satisfied. For example, the processed audio may include a pitch of a voice changing from a first frequency to a second frequency where the pitch of the voice changing from the first frequency to the second frequency satisfies the particular threat threshold. The operations include sending a notification to a designated recipient. The notification includes a link to listen to the audio data or the processed audio.

FIG.1is a block diagram of a system100that includes an artificial intelligence (AI) to analyze the output of a security system, according to some embodiments. In the system100, a security system102is located at a location104to monitor an object, such as an interior of an object (e.g., vehicle) or an exterior of an object (e.g., building, a statue, an artifact, or another relatively unchanging object). The security system102may include one or more devices that include sensors capable of capturing, displaying, and/or transmitting a set of (e.g., one or more) images, including a screen capture. In some cases, the sensors may be capable of capable of capturing, displaying, and/or transmitting audio data.

The security system102may be connected to a server106via one or more networks108. At least a portion of the servers106may, in some cases, be cloud-based. For ease of understanding, various software modules and data are illustrated as being located at the location104or at the server106. However, it should be understood that the various software modules and data described herein may be distributed between the security system102(or a computing device at the location104) and the servers106. For example, one or more of the software modules and data illustrated as being hosted by the server106may be hosted by the security system102(or a computing device at the location104). In addition, in some cases, one or more of the functions described herein may be incorporated into a single board, embedded device, integrated circuit (e.g., a system-on-a-chip) or the like.

The security system102may be connected (e.g., wirelessly, wired, or a combination of both) to multiple sensors, such as a sensor110(1) to a sensor110(N) (N>0). For example, the sensors110may include one or more imaging devices (e.g., cameras) and other types of sensors such as microphones, motion detectors, and the like. In addition to capturing visible light, the sensors110may be capable of capturing other types of light (e.g., electromagnetic radiation) that are not visible to the human eye, such as infrared, ultraviolet, thermal imaging, and the like. Each of the sensors110may have a particular view (e.g., field of view). For example, the sensor110(1) may have a view112(1) and the sensor110(N) may have a view112(N). Each of the views110may be different from others of the views110. For example, the view112(1) may be different from the view112(N). In some cases, a portion of one view may overlap with a portion of another view. Each of the views112may include the state of one or more objects present in the location104, such as a representative object115. In some cases, in a retail business, one or more of the views112may include at least a portion of a person114(e.g., a first person) and at least of the object115. In some cases, one or more of the views112may include a person116(e.g., a second person).

In some cases, an output118(e.g., sensor data, such as one or more video streams, audio data streams, or the like from the sensors110) of the security system102may be sent to one or more servers106. For example, a place of worship (e.g., a church, a temple, a mosque, or the like), a government (e.g., city, county, state, federal) entity, a business, or another type of entity may have a subscription to a monitoring service (e.g., hosted by the servers106) that monitors the output118of the security system102to determine when a change is detected and perform one or more actions in response to detecting a change. Each video stream may include a sequence of images captured at a particular rate, such as, for example, 15, 30, 60 frames per second (FPS). Each image may include a timestamp or may include a timecode identifying a particular time when the image was captured. For example, the timestamp or timecode may include a date (e.g., year, month, day) and hours, minutes, seconds, and milliseconds. The output118may include digital audio data recorded by the sensors110at a particular sampling frequency.

The server106may include a processing module120that processes the sensor data in the output118to create processed data122(e.g., processed images, processed audio, or the like). The output118may be processed in such a way as to enable changes (e.g., graffiti, defacement, or the like) to be more easily detected by an artificial intelligence (AI) module124. Thus, the processed data122may be processed to be machine-readable (e.g., rather than human-readable) by the AI module124. The AI124may perform an analysis of the processed data122including performing change detection126and creating annotated images130in which the detected changes are identified (examples of annotated images are provided herein). The processing120may receive the images in the output118or retrieve the images from a storage location, such as a buffer, or other type of storage. The buffer may enable the AI module124to process the output118substantially in real-time, e.g., with a relatively small amount of latency (e.g., less than 1 second) without losing any portion of the output118.

The AI124may determine various details associated with one or more changes (e.g., 1 to M, M>0), such as a locus140of each change, a type142of each change, a change level144of each change, a length146of each change, and a probability148. The locus140may indicate what the change is associated with, e.g., the object115in the location104. The type142may indicate the type of change identified, e.g., graffiti, defacement, or the like. The change level144may indicate a level associated with a change. For example, the change level144may indicate how large the changed area is relative to the overall area of the object115. To illustrate, the larger the amount of change relative to the overall area of the object115, the greater the change level144. For example, a large amount of graffiti or defacement may have a higher change level144than a smaller amount of graffiti or defacement. The length146may indicate a length of time that a change has been identified. For example, the length146may include a number of images in which the change has been identified, a duration (e.g., the last time at which the change was identified minus the first time at which the change was identified), or both. For example, the AI124determine a change is present when the change is detected consecutively in at least a pre-determined number of images (e.g., at least 9,000 images at 30 fps=5 minutes) or for at least a predetermined amount of time (e.g., at least 300 seconds=5 minutes). The probability148may be associated with the type142of change, e.g., 98% probability that a permanent change (e.g., to a surface area of a building, statue, artifact, interior of a vehicle, or the like) occurred. The change that is detected by the AI124may include an item (e.g., an enclosure that includes a bomb or another type of device) placed near the object115.

If the number and/or type of changes detected by the change detection126satisfies one or more thresholds128, then the server106may perform one or more notification actions132. For example, the thresholds128may be based on the locus140, the type of change142, the level of change144, the length of the change146, the probability148, another factor, or any combination thereof. To illustrate, if a first change (e.g., graffiti to a building) and a second change (e.g., defacement of a statue or an artifact) are both present, then one of the thresholds128may be satisfied.

One of the notification actions132may include sending a notification134that includes a link136to one or more recipients138specified in the notification actions132. The link136may enable the recipients138to view and/or listen to at least a portion of the output118, the processed data122, the annotated images130, or any combination thereof. The recipients138may include law enforcement, such as police. The recipients138may include an owner (or manager) of the object115at the location104, a security company that monitors various locations including the location104, or other responsible individual. In response to receiving the notification134, the recipients138may view at least a portion of the output118, the processed data122, the annotated images130, or any combination thereof and determine whether to notify law enforcement (e.g., police). For example, in some areas, creating a false alarm by calling law enforcement when no crime is present may result in a fine being levied to a business. To avoid such situations, the recipients138may include the owner (or manager) or a security company to enable a human to view what is happening at the location104prior to notifying law enforcement. The notification134may take the form of a text message, an automated voice message, a posting to a social media site, posting to website, another type of notification message, or any combination thereof. The notification actions132may include sounding an alarm at the location104or at a remote location, such as at a security company monitoring the location104or at the home of an owner of manager of the object115at the location104.

The notification actions132may include sounding an alarm (e.g., at the location104, at a security company that is monitoring the location104, at the home of the owner of the business, or any combination thereof), playing an audio file at the location104(e.g., “Authorities have been alerted”), sending a message (e.g., text or voice message), a social media posting, website posting, sharing an image stream or a link to a live stream (e.g., the output118, the annotated images130, or both). The message, call, or social media post may include one or more of: a name of the location104, an address or other type of coordinates associated with the location104, the link136to a secure website that shows one or more images of the annotated images130, the output118, or both, instructions on how to access the secure website, other information, or any combination thereof. The streaming (e.g., the output118, the processed data122, the annotated images130, or the like) may be accessed via a password or via a unique universal resource locator (URL). The site where the streaming is available may enable a viewer to share the streaming with others (e.g., owner of business or security company may share with authorities, such as law enforcement).

The AI124may include one or more deep learning neural networks (e.g., convolution neural network) and may be trained using supervised learning, unsupervised learning, reinforcement learning, self-learning, or any combination thereof. The AI124may analyze an emotion conveyed by a facial expression of the person114, the person116, or both. For example, the AI124may take into account how long the person114has a particular expression, such as a shocked expression, a fearful expression, crying (e.g., one or more tears present), a distracted expression, blinks frequently or in a particular pattern (e.g., 3 short blinks followed by 3 long blinks followed by 3 short blinks, alternating long and short blinks, or the like).

In this way, the output118of the security system102may be analyzed using the AI124to determine whether one or more changes are present. If the AI124detects one or more changes and the changes satisfy the thresholds128, then the server106may perform the one or more notification actions132by, for example, sending the notification134including the link136to one or more recipients138specified in the notification actions132.

While the processing120, the AI124, the annotated images130, and the notification actions132are shown as being performed on the server106, in some cases, one or more of these functions may be performed at the location104. Thus, it should be understood that the functionality described herein may be divided in any number of ways between the location104and the server106. For example, the processing120may be performed at the location104and the processed images122may be sent as the output118to the server106. As another example, the processing120, the AI124, or both may be performed at the location104and the notification134may be sent by the security system102(or by a computing device at the location104) rather than the server106. In some cases, one or more of the functions described herein may be incorporated into an embedded controller, integrated circuit, system-on-a-chip (SOC), or the like.

While a single representative location104is shown inFIG.1for ease of understanding, it should be understood that the server106may receive multiple outputs from multiple security systems in multiple locations. Thus, the server106may monitor multiple locations substantially simultaneously. For example, in some cases, the server106may be associated with a security company that is monitoring multiple locations. The server106may send the notification134to an employee of the security company when one or more changes have been detected and the one or more changes satisfy at least one of the thresholds128.

Thus, a system may receive (or retrieve) multiple data streams from multiple sensors (e.g., video, audio, and the like) or as screenshot of one or multiple display devices. A processing module may process the data (e.g., video data, audio data) to create processed data. The AI module may analyze individual images of the processed images to determine whether one or more changes are present. If the AI detects one or more changes, the AI may determine whether the number of changes, a length of time for which each change has been present, or other change-related characteristics satisfy one or more thresholds. If the thresholds are satisfied, then a notification may be automatically (e.g., without human interaction) sent to one or more recipients, such as law enforcement, a business owner or manager, a security company, another recipient, or any combination thereof.

An object is anything that has surface area. The AI determines a change in the state of the object to determine a risk and/or threat. Some examples of objects include, but are not limited to: a drawer, a safe, a door, a window, one or more shelves, a physical shopping cart, a hand basket, one or more products, a wall, a floor, a sculpture or other type of art object, a statue, a pedestal, art work, a vehicle, a bag, or a backpack. An object may include one or more additional objects. For example, an object such as a showcase or a shelf may include additional objects, such as art works, jewelry or the like. As another example, an object, such as a pedestal may include an additional object, such as a sculpture.

Any perceived or actual physical change to the object may be referred to as a “Change in State of an Object”. The AI is used to monitor one or more objects to detect a change in the state of each object of the one or more objects, as shown in Table 1 below.

TABLE 1Change in State of ObjectExample(s)Presence of an objectIf a bag (or any other object) was detected in restricted area.Position or Orientation of theAn artwork on the wall has tilted hence its position has changedwhole or part of thefrom its normal state. A statue that used to face the camera hasmonitored object(s) changednow shifted its orientation or position. A piece of Jewelry movedfrom its position.Location of the whole or partA filled shopping cart (object) was detected going out (direction)of the monitored object(s)of the entrance door (location). A statue was on pedestal but nowchangedis on the ground. Cellphone inside the Safe was removed. Aperson sneaked the Jewelry item from shelf into their bag(located changed). A filled hand basket (Object) in supermarket isnot supposed to go out through entrance door (at undesiredlocation). A vehicle detected moving towards a crowd (location ofthe vehicle at a undesired location). Flash Mob picks up valuablesfrom the shelf and run (location of the valuable moved)Texture of the whole or partWhen an interior or external wall is vandalized with the paint itof the monitored object(s)will change the texture of the wall. When someone tries to usechangedflame burner to break into a safe it will change the texture of thesafe. When someone tries to break a wall it will change thetexture of the wall. A glass showcase was broken and now itstexture is differentMaterial of the whole or partA thief replaces silver jewelry with aluminum jewelry. The canvasof the monitored object(s)of the art work changed to cheap paper replica.changedPhysical form of the whole orA wall (object) was vandalized or a statue (object) waspart of the monitoredvandalized with paint, the physical form of the object hasobject(s) changedchanged.The wall used to be a brick wall but now it has graffiti. The statuewas white marble statue but due to vandalization it now hasmarks or symbols.Shape of the whole or of theThe statue (object) was broken (change in shape) and now themonitored object(s) changedshape of the statue is different then its normal state.Color of the whole or part ofThe floor (object) color has changed (Partial area) with bloodthe monitored object(s)stain (different color). An object was vandalized using color.changedOpen or Close or Empty orA safe door open or partially open; A cash register empty orFilled. Partially or Whole.partially empty. A safe empty or partially empty. A showcaseempty or partially empty. A shelf with valuables empty orpartially empty. Door to the safe room left partially open.Physical EnvironmentA thief is tampering the safe with flame burner, the model canchanged such as smoke, fire,detect smoke or fire (change in physical environment); Lightingenvironment lightingin a certain area of the showroom blows off or turns off (changein lighting in the physical environment)Change to an object in anyAll physical changes or perceived physical changes. All physicalmanneror perceived change to the state of the object

The length of time for which a change is present may include for example, a duration in an amount of time (e.g., milliseconds, seconds, minutes, or the like), the number of video frames in which the change is present, or the like.

Detecting a change that includes a threat may include determining a posture, determining a behavior, determining a facial expression or any combination thereof. Examples, include, but are not limited to: Push (A person pushing another person e.g. on subway rail tracks), Fist fight (One or more person Fist fighting), Kick (One or more person kicking), Punch (One or more person punching another person or persons), Slap (One or more person slapping another person or persons), Hackling (One or more person hacking another person or persons), Shoving (One or more person shoving an object or thing), Hands behind the head (A person putting their hands behind the head), Hands over face (A person hiding their face), Hands tied (A person whose hands are tied), Legs tied (A person whole legs are tied), Hiding or crouching behind an object (A person partially or fully hiding behind an object), Walking (A person walking or present in a location they should not be), Standing (A person standing or present in a location they should not be), Sitting (A person sitting or present in a location they should not be), Bent (A person bent), Lying on the floor (A person lying on the floor), Shaking or Trembling (A person shaking or trembling), Running (A person running away), Snatching (A person snatching an object), Bleeding (A person bleeding), Crying (A person crying), Pain (A person in pain), Hurt (A person is hurt), Distress (A person in distress), Aggravation (A person demonstrating aggravation), Bullying (A person bullying another person), Chaos.

A weapon is defined as anything perceived as capable of being used to hurt a human being. The AI is used to detect the presence of a weapon, such as, for example, any type of sharp object that can be used to hurt others, any type of heavy object that can be used to hurt others, any type of gun (e.g., hand gun, rifle, semi-automatic, automatic etc.), any type of knife, any type of baseball bat or cricket bat or rod or such things, any type of a blade, any type of a stick, or any type of light saber.

When the sensors include microphones, the audio data may be processed and the AI used to analyze the processed data. Argument (One or more person is arguing), Threatening (one or more person is threatening), Quarrelling (one or more person are quarrelling), Aggravation (one or more person are aggravating others), Bullying (one or more person are bullying), Whispering (one or more person are whispering), Yelling (one or more person are yelling), Shouting (one or more person are shouting), Cursing (one or more person are saying curse words), Racial slurs, Pain related sound by person, Pain related words by person (person saying “aaah” or another type of sound), Pleading related sound tone by person (Fearful and distress type sound tone), Pleading related words by person (person saying “please don't do this”, “please stop”, “help”), State of situation (person saying “I have been shot”, “There is debris all over”, “I have been stabbed”, “A person has been stabbed”, “A person is injured”), a combination of (1) a word (e.g., name calling), (2) a pitch of a voice (e.g., higher than normal), and (3) volume of voice (higher than normal (yelling) or lower than normal (whispering)). For audio, the processing120may use multiple frequency bands to identify the frequencies associated with the speech in the output118and create a filter to filter out non-speech frequencies. For example, a male voice typically occupies the frequencies 85 to 180 Hertz (Hz) while a female voice typically occupies the frequencies 165 to 255 Hz. The processing120may use at least 3 bands, e.g., (1) below 85 Hz, (2A) 85 Hz-170 Hz, (2B) 171 Hz-255 Hz, and (3) above 255 Hz. The processing120may analyze a first portion (e.g., less than a few seconds) of the audio data in the output118and determine the frequencies that the speech in the output118occupy. Based on this, the processing120may subsequently divide the audio data in the output120into three bands. For example, if the analysis of the output120indicates speech content between approximately 85 Hz-180 Hz, the processing120may create three bands, e.g., (1) below 85 Hz, (2) between 85 Hz-180 Hz, and (3) above 180 Hz. The processing120may modify a volume of bands (1) and (3) to attenuate unwanted signals, thereby improving the intelligibility of the voice band (e.g., between 85 Hz-180 Hz).

FIG.2AandFIG.2Billustrate annotated images of an interior of a building, according to some embodiments. The annotated images130ofFIG.1may include an annotated image130(S) ofFIG.2Acaptured at a time S and an annotated image130(T) ofFIG.2Bcaptured at a time T, where T>S. Thus, annotated image130(T) is subsequent to annotated image130(S). The artificial intelligence (AI)124ofFIG.1may add annotations212, such as change information, to the processed data122or the output118of the security system102, to create the annotated images130.

The annotated image130(S) ofFIG.2Aincludes the person114(e.g., first person), the person116(e.g., second person), and the object115(e.g., a point-of-sale terminal) located on a counter202. In the annotated image130(S), an annotation212(1) indicates that the person114has their hands up. The annotation212(1) provided by the AI124may include information that a change204(1) was detected, a type206(1) of change (e.g., a threat), e.g., the person114has their hands up, and a probability208(1) of the predicted threat. The annotation212(1) may be displayed with a visual linking (e.g., using lines, as shown inFIGS.2A and2B, or a similar technique) to the locus. In annotated image130(S), the annotation212(1) is associated with the person114(e.g., the locus) and specifically the first person's hands. In some cases, the probability208(1) may increase over time. For example, at a time T1, when the person114(e.g., first person) puts their hands up, the AI124may determine that no other change is present and may predict, based on past experience, that the person114(e.g., first person) is stretching their arms. However, if at a time Tx (x>0, such as 10 seconds) the person114(e.g., first person) still has their hands up, then the AI124may determine that a change is present. The probability208(1) of a change may increase proportional to the amount of time that the person114(e.g., first person) has their hands up. For example, after one second the probability may be 10%, after two seconds the probability may be 20% and so on, and the probability of a change at ten seconds (or more) may be 100%. In some cases, the probability of a change may be determined based on how many frames include the person114making a particular gesture. For example, inFIGS.2A and2B, T-S represents how long the person114has had their hands up. If T−S>=V, where V is a predetermined period of time, then the AI124may determine that a change is present. If T−S<V, then the AI124may determine that a change is absent (e.g., no change present).

The postures detected by the AI124may include, for example, crouching and lying down. Crouching includes adopting a position in which the knees are bent and the upper body is brought forward and down, e.g., to avoid detection.

InFIG.2B, the annotated image130(T) has multiple annotations212(1),212(2), and212(3). Annotated image130(T) is subsequent to annotated image130(S) and has the annotation212(1) indicating that the person114has their hands up and an annotation212(2) indicating that a change204(2) was detected with the person116. The annotation212(2) indicates that a type206(2) of the change is a presence of a weapon. In some cases, the AI124may further identify the type of weapon. For example, as shown inFIG.2B, the AI124has identified the weapon as a gun and indicated this in the annotation212(2). The AI124may identify in the annotation212(2) a probability208(2) associated with the identification of the type206(2), e.g., a weapon. In this example, the AI124indicates that a weapon in the form of a gun has been identified with the probability of 98%.

InFIG.2B, the AI124may identify multiple threats, including a first change associated with the person114(locus 1) and a second change associated with the person116(locus 2). Based on the thresholds128, the identification of two threats may be sufficient to satisfy one of the thresholds128, causing the server106ofFIG.1to automatically send the notification134including the link136to the recipients138. InFIG.2B, even if the person114has not had their hands up for very long (e.g., less than 1 second), the AI124may determine that one of the thresholds128is satisfied because the AI124detected the second threat, e.g., the weapon being held by the person116.

The AI124may provide an annotation212(3) indicating a third change204(3) (e.g., a threat) with a type206(3) of a partially or fully empty cash register (locus3), with a probability208(3) that this is a change of 95%. The third change204(3) is associated with an object in the location104ofFIG.1and may be a cash tray of the point-of-sale terminal115, a safe, a shelf that holds high value objects (e.g., jewelry) or the like. The thresholds128may indicate that if change204(3) and either change204(2) or204(1) are detected, to automatically (without human interaction) perform one or more of the notification actions132. The thresholds128may be initially created by an owner of a business or the security system provider and then modified (e.g., adjusted) by the AI124. For example, the AI124may be periodically (e.g., at a predetermined time interval, such as every P months, P>0) be retrained with additional data to further increase accuracy. By training the AI124using additional training data, the AI124may identify at least some additional thresholds to add to the thresholds128.

In some cases, the annotations212may indicate a length that each change (e.g., threat) has been present, e.g., either in terms of a number of images in which the change has been identified, a length of time during which the change has been identified, or both. In other cases, each of the annotations212may be added to the images after a predetermined length. For example, each of the annotations212may be added after the change was detected in at least X images (e.g., at a particular fps) or for at least Y seconds.

Thus, an AI module may identify one or more threats in an image and annotate the image to indicate the presence of a change (e.g., a threat), identify the type of change, provide a probability associated with the prediction of the change, and other change-related information. The change may be associated with a first person (e.g., employee), such as the first person putting their hands up in the air, the first person hiding behind a counter or other furniture, or the like. The change may be associated with a second person that is located near the first person, such as a second person with a weapon, a second person that is making a threatening gesture, or the like. The change may be associated with an object in the location, such as a cash drawer, a safe, a shelf that includes valuable objects (e.g., jewelry), a wall, a window, a statue, an artifact, or the like. Based on the number of threats detected, the type of threats, how long the threats have been present, and the like, a determination may be made whether one or more thresholds have been satisfied. If one of the thresholds has been satisfied, then a notification may be sent to one or more recipients to notify them of the detected threat. The recipients may include, for example, law enforcement, security company, an owner or a manager of a business, another type of recipients, or any combination thereof.

FIG.3AandFIG.3Billustrate images of an exterior of a building, according to some embodiments. The annotated images130ofFIG.1may include the annotated image130(S) ofFIG.3Acaptured at a time S and the annotated image130(T) ofFIG.3Bcaptured at a time T, where T>S. The artificial intelligence (AI)124ofFIG.1may add annotations302(e.g., change information) to the processed data122or to the output118of the security system102, to create the annotated images130.

InFIG.3A, the annotated image130(S) includes the object115, e.g., a building, such as a house of worship (e.g., a church, a temple, a mosque, or the like). The annotated image130(S) does not include any annotations because the AI124has not detected any changes to the exterior of the object115.

InFIG.3B, the annotated image130(T) includes the object115and an annotation302that indicates that the AI124has detected a change304to the object115. The annotation302indicates that a type206(4) of the change304is predicted to be graffiti, with a probability208(4) of 98%.

In identifying the change204(4), the AI124may determine that the object115appeared similar (e.g., less than a threshold amount of change) to its appearance in image130(S) for at least a first predetermined number of frames (or a first predetermined amount of time) prior to the change204(4). After detecting the change204(4) between the object115in image130(3) and in image130(T), the AI124may determine that the change exceeds a threshold amount of change and is persistent (e.g., not caused by weather, or another temporary factor) because the change is present for at least a second predetermined number of frames (or a second predetermined amount of time). The AI124may, in some cases, use optical character recognition (OCR) to determine that the change204(4) includes graffiti by recognizing the characters.

FIG.4AandFIG.4Billustrate annotated images that include an object (e.g., a statue or an artifact), according to some embodiments. The annotated images130ofFIG.1may include the annotated image130(S) ofFIG.4Acaptured at a time S and the annotated image130(T) ofFIG.4Bcaptured at a time T, where T>S. The artificial intelligence (AI)124ofFIG.1may add annotations312(e.g., change information) to the processed data122or to the output118of the security system102, to create the annotated images130.

InFIG.4A, the annotated image130(S) includes the object115, e.g., a statue or another type of object (e.g., non-building). The annotated image130(S) does not include any annotations because the AI124has not detected any changes to the object115.

InFIG.4B, the annotated image130(T) includes the object115and an annotation402(1) that indicates that the AI124has detected a change204(5) to the object115. The annotation402(1) indicates that a type206(5) of the change204(5) is predicted to be defacement, with a probability208(5) of 95%. The annotated image130(T) includes an annotation402(2) that indicates that the AI124has detected a change204(6) to the object115. The annotation402(2) indicates that a type206(6) of the change204(6) is predicted to be graffiti, with a probability208(6) of 98%.

In identifying the changes204(5),204(6), the AI124may determine that the object115appeared similar (e.g., less than a threshold amount of change) to its appearance in image130(S) for at least a first predetermined number of frames (or a first predetermined amount of time) prior to the changes204(5),204(6). After detecting the change202(4) between the object115in image130(3) and in image130(T), the AI124may determine that the change exceeds a threshold amount of change, e.g., the changes204(5),204(6) are significant, and are persistent (e.g., not caused by weather, or another temporary factor) because the change is present for at least a second predetermined number of frames (or a second predetermined amount of time).

Of course,FIGS.3A,3B,4A, and4Bare merely examples of annotations of objects (e.g., located inside a building or outside a building) and it should be understood that the AI124may be trained to detect a change to any type of object, including but not limited to the exterior of a building, statue, an artifact, art work (e.g., wall mural painted on the exterior of a building), or the like.

In the flow diagrams ofFIGS.5,6, and7each block represents one or more operations that can be implemented in hardware, software, or a combination thereof. In the context of software, the blocks represent computer-executable instructions that, when executed by one or more processors, cause the processors to perform the recited operations. Generally, computer-executable instructions include routines, programs, objects, modules, components, data structures, and the like that perform particular functions or implement particular abstract data types. The order in which the blocks are described is not intended to be construed as a limitation, and any number of the described operations can be combined in any order and/or in parallel to implement the processes. For discussion purposes, the processes500,600, and700are described with reference toFIGS.1,2A,2B,3A,3B,4A, and4Bas described above, although other models, frameworks, systems and environments may be used to implement this process.

FIG.5is a flowchart of a process that includes sending a notification to one or more recipients after determining a change (e.g., including detecting one or more threats), according to some embodiments. The process500may be performed by the server106ofFIG.1. Of course, in some cases, at least a portion of the functionality described as being performed by the server106may be located at the location104. For example, the processing120, the AI124, the change detection126, the thresholds128, the annotated friends130, the notification actions132, or any combination thereof may be performed at the location104by a computing device that is connected to the security system102or by the security system102itself.

At502, the server may receive or retrieve an output (e.g., sensor data, such as images and/or audio data) from a security system. At504, the server may process individual sensor data items (e.g., images, audio data, and the like) in the output (e.g., in each stream) to create processed data (e.g., processed images, processed audio, and the like). For example, inFIG.1, the server106may receive the output118from the security system102and use the processing module120to create the processed data122. The processed data122may be processed to increase machine readability to enable the AI124to detect threats in the processed data122(e.g., processed images, processed audio, or the like).

At506, the server may perform an analysis of the processed data using artificial intelligence (e.g., a deep learning neural network, such as a convolutional neural network). At508, the server may determine whether changes detected based on the analysis. If the server determines, at508, that no changes were detected based on the analysis, then the process may proceed back to502. If the server determines, at508that one or more changes were detected based on the analysis, then the process may proceed to510. For example, inFIG.1, the server106may perform an analysis of the processed data122using the AI124. Based on the analysis, the server106may determine whether the processed data122indicates one or more changes (e.g., the person114holding up their hands, a weapon in the possession of the person116, cash being removed from the point-of-sale terminal115, graffiti or defacement of the object115, or the like). If the server106determines that the AI124detected no changes, then the server106may continue to process the output118from the security system102and analyze the processed data122using the AI124.

At510, the server may determine whether the changes that were detected satisfy one or more thresholds. If the server determines, at510, that the changes that were detected fail to satisfy the one or more thresholds, then the process may proceed to502, where the server receives additional output. If the server determines, at510, that the changes that were detected satisfy one or more of the thresholds, then the process may proceed to512. For example, the server106may determine whether the changes betweenFIG.3AandFIG.3Bor betweenFIG.4AandFIG.4Bare significant (e.g., greater than a threshold amount) and are persistent (e.g., present for more than a threshold amount of time). If the change(s) are significant and persist for more than a threshold amount of time, then the sensor data is annotated, substantially in real time.

At512, the server may annotate one or more images (or other data) to include details associated with the change. At514, the server may send a notification to one or more specified recipients. At516, the server may include a link in the notification to enable the specified recipient(s) to access a portion of the output or a portion of the processed data (e.g., one or more of the annotated images or the unannotated images). For example, inFIG.2B, the AI124may create the annotated image130(T) that includes the annotations212(1),212(2), and212(3).FIG.3BandFIG.4Billustrate examples of annotations to objects. InFIG.1, after determining that the changes (e.g., threats) detected in the processed data122satisfy at least one of the thresholds128, the AI124may create the annotated images130and perform one or more of the notification actions132. For example, the server106may send the notification134to one or more of the recipients138specified in the notification actions132. In some cases, the server106may include the link136in the notification134to enable the recipients138to view at least a portion of the annotated images130, the output118, or both. If the output118includes audio data, the link136may enable the recipients138to listen to a portion of the output118or a portion of the processed (audio) data122.

Thus, a server may receive the output (e.g., sensor data, including multiple video streams) of a security system that includes multiple sensors. The server may process the output to improve their machine readability and create processed data (e.g., processed images, processed audio, and the like). The server may use an AI, such as a deep learning neural network, to analyze the processed data. If the AI detects one or more changes in the processed data, the AI may annotate the images to indicate the type of change, the locus of the change, how long the change has been present, other details associated with the change, and the like. The server may determine whether the change satisfies one or more thresholds (e.g., specified based on types of changes, number of changes, how long the changes are present, and the like). If the changes fail to satisfy the one or more thresholds, the server may continue to process the output of the security system. If the changes satisfy at least one of the thresholds, then the server may send a notification to one or more previously specified recipients and, in some cases, include a link in the notification to enable the specified recipients to view at least a portion of the annotated images. The advantage provided is that a person does not have to call the authorities (e.g., law enforcement), enabling 24-hour detection of changes, even when no humans are present. The system uses AI to automatically identify changes, determine whether the changes satisfy one or more thresholds, and when the changes satisfy at least one of the thresholds, the system may automatically send a notification to one or more recipients, such as someone responsible for monitoring the location in which the object is located, such as a security company that monitors the location, law enforcement, another recipient, or any combination thereof. By reducing the time between when a change occurs and when one or more recipients are notified of the change, perpetrators of the change can be quickly apprehended, sometimes before they have left the location or shortly after they have left the location.

FIG.6is a flowchart of a process400that includes adjusting various characteristics of an image, according to some embodiments. The process400may be performed by the processing module120ofFIG.1.

At602, the process may select a image from a stream of one or more images (e.g., video). At604, the process may adjust an image size of the image. At606, the process may adjust one or more of sharpness, zoom, brightness, contrast, saturation, colors, or any combination thereof associated with the image. At608, the process may perform a conversion of the image from a first format to a second format. For example, inFIG.1, each of the images of the output118may be captured in a particular size, such as 800×600, 1200×900, 1280×720, 1920×1080 pixels, or the like. To enable rapid processing of the output118, in some cases the size may be reduced. For example, the processing module120may reduce a size of images (e.g., from 1200×900 to 800×600) to enable faster processing of the processed data122. The processing module120may modify the sharpness of one or more images from the output118to enable the AI124to more easily detect threats. The processing module120may perform zooming (e.g., digital zoom) of individual images in the output118. For example, the processing module120may divide an individual image into three portions, a first portion that includes the person114(e.g., first person), a second portion that includes an object, such as the point-of-sale terminal115, and a third portion that includes the person116(e.g., second person). In this way, the AI124can analyze each of the three portions to determine whether there is a change associated with the person114, the point-of-sale terminal115, or the person116. The processing120may adjust the brightness, contrast, saturation, and number of colors to enable the AI124to more easily detect threats in the processed data122. In some cases, the selected image may be captured as or converted (by the processing120) to a format such as RGB (Red, Green, Blue), RGBA (Red, Green, Blue, Alpha), BGR (Blue, Green, Red), BGRA (Blue, Green, Red, Alpha), or grayscale.

Thus, images output by a security system may be processed to enable an AI to more easily detect threats. For example, the video processing may be used to make the processed data machine readable for inferencing and training.

FIG.7is a flowchart of a process700to train a machine learning algorithm to create the AI124, according to some embodiments. The process700may be performed by a server, such as the server106ofFIG.1.

At702, a machine learning algorithm (e.g., software code) may be created by one or more software designers. At704, the machine learning algorithm may be trained using pre-classified training data706(e.g., a portion of the conversation data136that has been pre-classified). For example, the training data706may have been pre-classified by humans, by machine learning, or a combination of both. After the machine learning has been trained using the pre-classified training data706, the machine learning may be tested, at708, using test data710to determine an accuracy of the machine learning. For example, in the case of a classifier, the accuracy of the classification may be determined using the test data710.

If an accuracy of the machine learning does not satisfy a desired accuracy (e.g., 95%, 98%, 99% accurate), at708, then the machine learning code may be tuned, at712, to achieve the desired accuracy. For example, at712, the software designers may modify the machine learning software code to improve the accuracy of the machine learning algorithm. After the machine learning has been tuned, at712, the machine learning may be retrained, at704, using the pre-classified training data706. In this way,704,708,712may be repeated until the machine learning is able to classify the test data710with the desired accuracy.

After determining, at708, that an accuracy of the machine learning satisfies the desired accuracy, the process may proceed to714, where verification data716(e.g., a portion of the conversation data136that has been pre-classified) may be used to verify an accuracy of the machine learning. After the accuracy of the machine learning is verified, at714, the machine learning130, which has been trained to provide a particular level of accuracy may be used as the AI124. For example, the AI124may be trained to predict a type of weapon, what a particular facial expression indicates, what a particular posture indicates, what a particular hand gesture indicates, what graffiti looks like, what defacement looks like, recognizing threatening words and phrases in audio data, not detecting weather-related changes as graffiti or defacement, not detecting animal defecation (e.g., bird poop, dog poop, or the like) as graffiti or defacement, and the like.

FIG.8illustrates an example configuration of a device800that can be used to implement the systems and techniques described herein, such as for example, the security system102, a computing device located at the location104, and/or the server106ofFIG.1. For illustration purposes, the device800is illustrated inFIG.8as implementing the server106ofFIG.1.

The device800may include one or more processors802(e.g., central processing unit (CPU), graphics processing unit (GPU), or the like), a memory804, communication interfaces806, a display device808, other input/output (I/O) devices810(e.g., keyboard, trackball, and the like), and one or more mass storage devices812(e.g., disk drive, solid state disk drive, or the like), configured to communicate with each other, such as via one or more system buses814or other suitable connections. While a single system bus814is illustrated for ease of understanding, it should be understood that the system buses814may include multiple buses, such as a memory device bus, a storage device bus (e.g., serial ATA (SATA) and the like), data buses (e.g., universal serial bus (USB) and the like), video signal buses (e.g., ThunderBolt®, DVI, HDMI, and the like), power buses, etc.

The processors802are one or more hardware devices that may include a single processing unit or a number of processing units, all of which may include single or multiple computing units or multiple cores. The processors802may include a graphics processing unit (GPU) that is integrated into the CPU or the GPU may be a separate processor device from the CPU. The processors802may be implemented as one or more microprocessors, microcomputers, microcontrollers, digital signal processors, central processing units, graphics processing units, state machines, logic circuitries, and/or any devices that manipulate signals based on operational instructions. Among other capabilities, the processors802may be configured to fetch and execute computer-readable instructions stored in the memory804, mass storage devices812, or other computer-readable media.

Memory804and mass storage devices812are examples of computer storage media (e.g., memory storage devices) for storing instructions that can be executed by the processors802to perform the various functions described herein. For example, memory804may include both volatile memory and non-volatile memory (e.g., RAM, ROM, or the like) devices. Further, mass storage devices812may include hard disk drives, solid-state drives, removable media, including external and removable drives, memory cards, flash memory, floppy disks, optical disks (e.g., CD, DVD), a storage array, a network attached storage, a storage area network, or the like. Both memory804and mass storage devices812may be collectively referred to as memory or computer storage media herein and may be any type of non-transitory media capable of storing computer-readable, processor-executable program instructions as computer program code that can be executed by the processors802as a particular machine configured for carrying out the operations and functions described in the implementations herein.

The device800may include one or more communication interfaces806for exchanging data via the network108. The communication interfaces806can facilitate communications within a wide variety of networks and protocol types, including wired networks (e.g., Ethernet, DOCSIS, DSL, Fiber, USB etc.) and wireless networks (e.g., WLAN, GSM, CDMA, 802.11, Bluetooth, Wireless USB, ZigBee, cellular, satellite, etc.), the Internet and the like. Communication interfaces806can also provide communication with external storage, such as a storage array, network attached storage, storage area network, cloud storage, or the like.

The display device808may be used for displaying content (e.g., information and images) to users. Other I/O devices810may be devices that receive various inputs from a user and provide various outputs to the user, and may include a keyboard, a touchpad, a mouse, a printer, audio input/output devices, and so forth.

The computer storage media, such as memory804and mass storage devices812, may be used to store software and data, including, for example, the AI124, the processed data122, the processing120, the notification actions130, other data816, and other software818. For ease of illustration, not all the software and data associated with the server106are shown.